{"id":3606,"date":"2020-10-16T20:58:09","date_gmt":"2020-10-16T12:58:09","guid":{"rendered":"http:\/\/www.sniper97.cn\/?p=3606"},"modified":"2020-10-16T20:58:09","modified_gmt":"2020-10-16T12:58:09","slug":"%e3%80%90%e6%b7%b1%e5%ba%a6%e5%ad%a6%e4%b9%a0%e7%ac%94%e8%ae%b0%e3%80%91attention%e5%92%8cself-attention%e7%9a%84%e8%81%94%e7%b3%bb%e5%92%8c%e5%8c%ba%e5%88%ab","status":"publish","type":"post","link":"http:\/\/www.sniper97.cn\/index.php\/note\/deep-learning\/base\/3606\/","title":{"rendered":"\u3010\u6df1\u5ea6\u5b66\u4e60\u7b14\u8bb0\u3011Attention\u548cself Attention\u7684\u8054\u7cfb\u548c\u533a\u522b"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">\u4ec0\u4e48\u662f\u6ce8\u610f\u529b\uff1f<\/h2>\n\n\n<p>\u5f53\u4eba\u7c7b\u89c2\u5bdf\u4e00\u5f20\u56fe\u7247\u6216\u8005\u4e00\u6bb5\u6587\u5b57\u65f6\uff0c\u53ef\u4ee5\u5c06\u91cd\u70b9\u653e\u5230\u67d0\u4e00\u90e8\u5206\uff0c\u4ece\u800c\u53ef\u4ee5\u66f4\u597d\u7684\u5904\u7406\u56fe\u4e2d\u6216\u6587\u5b57\u4e2d\u7684\u4fe1\u606f\u3002<\/p>\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"551\" height=\"391\" src=\"\/wp-content\/uploads\/2020\/10\/\u56fe\u7247.png\" alt=\"\" class=\"wp-image-3608\" srcset=\"http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247.png 551w, http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-300x213.png 300w\" sizes=\"(max-width: 551px) 100vw, 551px\" \/><\/figure>\n\n\n<p>\u5982\u679c\u6211\u4eec\u5728\u795e\u7ecf\u7f51\u7edc\u4e2d\u4e5f\u4f7f\u7528\u7c7b\u4f3c\u673a\u5236\uff0c\u53ef\u4ee5\u8ba9\u7f51\u7edc\u66f4\u597d\u7684\u83b7\u53d6\u4e0a\u4e0b\u6587\u4fe1\u606f\uff0c\u4ece\u800c\u83b7\u5f97\u66f4\u597d\u7684\u6548\u679c\u3002<\/p>\n\n\n<h2 class=\"wp-block-heading\">Attention<\/h2>\n\n\n<p> \u6700\u5148\u51fa\u81ea\u4e8eBengio\u56e2\u961f\u4e00\u7bc7\u8bba\u6587\uff1aNEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE \uff0c\u8bba\u6587\u57282015\u5e74\u53d1\u8868\u5728ICLR\u3002 <\/p>\n\n\n<p> Attention\u51fd\u6570\u7684\u672c\u8d28\u53ef\u4ee5\u88ab\u63cf\u8ff0\u4e3a\u4e00\u4e2a\u67e5\u8be2\uff08query\uff09\u5230\u4e00\u7cfb\u5217\uff08\u952ekey-\u503cvalue\uff09\u5bf9\u7684\u6620\u5c04 \u3002<\/p>\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"527\" height=\"258\" src=\"\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-3.png\" alt=\"\" class=\"wp-image-3611\" srcset=\"http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-3.png 527w, http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-3-300x147.png 300w\" sizes=\"(max-width: 527px) 100vw, 527px\" \/><\/figure><\/div>\n\n\n<p> \u5728\u8ba1\u7b97attention\u65f6\u4e3b\u8981\u5206\u4e3a\u4e09\u6b65\uff0c<\/p>\n\n\n<p>\u7b2c\u4e00\u6b65\u662f\u5c06query\u548c\u6bcf\u4e2akey\u8fdb\u884c\u76f8\u4f3c\u5ea6\u8ba1\u7b97\u5f97\u5230\u6743\u91cd\uff0c\u5e38\u7528\u7684\u76f8\u4f3c\u5ea6\u51fd\u6570\u6709\u70b9\u79ef\uff0c\u62fc\u63a5\uff0c\u611f\u77e5\u673a\u7b49\uff1b<\/p>\n\n\n<p>\u7b2c\u4e8c\u6b65\u4e00\u822c\u662f\u4f7f\u7528\u4e00\u4e2asoftmax\u51fd\u6570\u5bf9\u8fd9\u4e9b\u6743\u91cd\u8fdb\u884c\u5f52\u4e00\u5316\uff1b<\/p>\n\n\n<p>\u7b2c\u4e09\u6b65\u5c06\u6743\u91cd\u548c\u76f8\u5e94\u7684\u952e\u503cvalue\u8fdb\u884c\u52a0\u6743\u6c42\u548c\u5f97\u5230\u6700\u540e\u7684attention\u3002\u76ee\u524d\u5728NLP\u7814\u7a76\u4e2d\uff0ckey\u548cvalue\u5e38\u5e38\u90fd\u662f\u540c\u4e00\u4e2a\uff0c\u5373<strong>key=value<\/strong> \u3002<\/p>\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"542\" height=\"251\" src=\"\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-4.png\" alt=\"\" class=\"wp-image-3612\" srcset=\"http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-4.png 542w, http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-4-300x139.png 300w\" sizes=\"(max-width: 542px) 100vw, 542px\" \/><\/figure><\/div>\n\n\n<p> Attention\u673a\u5236\u53d1\u751f\u5728Target\u7684\u5143\u7d20Query\u548cSource\u4e2d\u7684\u6240\u6709\u5143\u7d20\u4e4b\u95f4\u3002<\/p>\n\n\n<p> \u6bd4\u5982entity1\uff0centity2\uff0centity3&#8230;.\uff0cattn\u4f1a\u8f93\u51fa[0.1\uff0c0.2\uff0c0.5\uff0c&#8230;.]\u8fd9\u79cd\uff0c\u544a\u8bc9\u4f60entity3\u91cd\u8981\u4e9b\u3002  <\/p>\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"348\" height=\"400\" src=\"\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-1.png\" alt=\"\" class=\"wp-image-3609\" srcset=\"http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-1.png 348w, http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-1-261x300.png 261w\" sizes=\"(max-width: 348px) 100vw, 348px\" \/><\/figure><\/div>\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"375\" height=\"424\" src=\"\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-2.png\" alt=\"\" class=\"wp-image-3610\" srcset=\"http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-2.png 375w, http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-2-265x300.png 265w\" sizes=\"(max-width: 375px) 100vw, 375px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">self attention<\/h2>\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"681\" height=\"436\" src=\"\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-5.png\" alt=\"\" class=\"wp-image-3613\" srcset=\"http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-5.png 681w, http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-5-300x192.png 300w\" sizes=\"(max-width: 681px) 100vw, 681px\" \/><\/figure><\/div>\n\n\n<p> self attention\u4f1a\u7ed9\u4f60\u4e00\u4e2a\u77e9\u9635\uff0c\u544a\u8bc9\u4f60&nbsp;entity1&nbsp;\u548centity2\u3001entity3&nbsp;&#8230;.\u7684\u5173\u8054\u7a0b\u5ea6\u3001entity2\u548centity1\u3001entity3&#8230;\u7684\u5173\u8054\u7a0b\u5ea6\u3002<\/p>\n\n\n<p>\u5b83\u6307\u7684\u4e0d\u662fTarget\u548cSource\u4e4b\u95f4\u7684Attention\u673a\u5236\uff0c\u800c\u662fSource\u5185\u90e8\u5143\u7d20\u4e4b\u95f4\u6216\u8005Target\u5185\u90e8\u5143\u7d20\u4e4b\u95f4\u53d1\u751f\u7684Attention\u673a\u5236\uff0c\u4e5f\u53ef\u4ee5\u7406\u89e3\u4e3aTarget=Source\u8fd9\u79cd\u7279\u6b8a\u60c5\u51b5\u4e0b\u7684\u6ce8\u610f\u529b\u8ba1\u7b97\u673a\u5236\u3002<strong>Q=K=V<\/strong>\u3002 <\/p>\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><img loading=\"lazy\" decoding=\"async\" width=\"417\" height=\"474\" src=\"\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-6.png\" alt=\"\" class=\"wp-image-3614\" srcset=\"http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-6.png 417w, http:\/\/www.sniper97.cn\/wp-content\/uploads\/2020\/10\/\u56fe\u7247-6-264x300.png 264w\" sizes=\"(max-width: 417px) 100vw, 417px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">\u603b\u7ed3<\/h2>\n\n\n<p>attention\u548cself attention \u5176\u5177\u4f53\u8ba1\u7b97\u8fc7\u7a0b\u662f\u4e00\u6837\u7684\uff0c\u53ea\u662f\u8ba1\u7b97\u5bf9\u8c61\u53d1\u751f\u4e86\u53d8\u5316\u800c\u5df2\u3002<\/p>\n\n\n<p>attention\u662fsource\u5bf9target\u7684attention\uff0c<\/p>\n\n\n<p>\u800cself attention \u662fsource \u5bf9source\u7684attention\u3002<\/p>\n\n\n<p>\u5176\u4ed6\u94fe\u63a5\uff1a<\/p>\n\n\n<p><a href=\"http:\/\/www.sniper97.cn\/index.php\/note\/deep-learning\/traslation\/3487\/\">\u5e26\u6ce8\u610f\u529b\u673a\u5236\u7684seq2seq<\/a><\/p>\n\n\n<p><a href=\"http:\/\/www.sniper97.cn\/index.php\/note\/deep-learning\/note-deep-learning\/3850\/\">self-attention\uff08Synthesizer\uff09<\/a><\/p>\n\n\n<p><a href=\"http:\/\/www.sniper97.cn\/index.php\/note\/deep-learning\/note-deep-learning\/4002\/\">\u4e3a\u4ec0\u4e48transformer(Bert)\u7684\u591a\u5934\u6ce8\u610f\u529b\u8981\u5bf9\u6bcf\u4e00\u4e2ahead\u8fdb\u884c\u964d\u7ef4\uff1f<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u4ec0\u4e48\u662f\u6ce8\u610f\u529b\uff1f \u5f53\u4eba\u7c7b\u89c2\u5bdf\u4e00\u5f20\u56fe\u7247\u6216\u8005\u4e00\u6bb5\u6587\u5b57\u65f6\uff0c\u53ef\u4ee5\u5c06\u91cd\u70b9\u653e\u5230\u67d0\u4e00\u90e8\u5206\uff0c\u4ece\u800c\u53ef\u4ee5\u66f4\u597d\u7684\u5904\u7406\u56fe\u4e2d\u6216\u6587 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[8,9],"tags":[30,37,39],"views":28149,"_links":{"self":[{"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/posts\/3606"}],"collection":[{"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/comments?post=3606"}],"version-history":[{"count":0,"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/posts\/3606\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/media?parent=3606"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/categories?post=3606"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.sniper97.cn\/index.php\/wp-json\/wp\/v2\/tags?post=3606"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}