{"id":796,"date":"2018-05-11T05:43:39","date_gmt":"2018-05-11T05:43:39","guid":{"rendered":"http:\/\/wordpress.cs.vt.edu\/optml\/?p=796"},"modified":"2018-05-11T05:43:39","modified_gmt":"2018-05-11T05:43:39","slug":"nesterov-accelerated-gradient-descent","status":"publish","type":"post","link":"https:\/\/wordpress.cs.vt.edu\/optml\/2018\/05\/11\/nesterov-accelerated-gradient-descent\/","title":{"rendered":"(Nesterov) Accelerated Gradient Descent"},"content":{"rendered":"

We define standard gradient descent as:
\n\"w^{k+1}<\/p>\n

Momentum adds a relatively subtle change to gradient descent by replace f with z. Where
\n\"z^{k+1}
\n\"w^{k+1}<\/p>\n

As such, alpha retains the role of representing the step size. However, we introduce a new constant, beta. Beta adds a decaying effect on momentum, and must be between 0 and 1, noninclusive. We briefly demonstrate the edge cases of beta below:
\n\"\beta
\n\"\beta<\/p>\n

We now introduce a change of basis which will be convenient for later. Consider a matrix form of gradient descent:
\n\"w^{k+1}<\/p>\n

Which has optimal solution
\n\"w^{\star}
\nwe consider the eigenvectors of A which is a natural space of viewing A where all dimensions act independently.<\/p>\n

Given:
\n$latex\u00a0A = Q \\text{ diag }(\\lambda_1 \\dots \\lambda_n)Q^T $
\n\"x^k<\/p>\n

We may derive this by:
\n\"w^{k+1}
\n\"w^{k+1}
\n\"Q^T\big(w^{k+1}
\n\"x^{k+1}<\/p>\n

\"w^{\star}
\n\"x^{k+1}
\n\"x^{k+1}
\n\"x^{k+1}
\n\"x^{k+1}
\n\"x^{k+1}
\n\"x^{k+1}
\n\"x_i^{k+1}
\n$latex\u00a0= \\sum_i^n x_i^0(1-\\alpha \\lambda_i)^k q_i $<\/p>\n

Thus, we now can view gradient descent in basis of the eigenvectors of A. We consider the contributions of error as
\n$latex\u00a0f(w^k)-f(w^\\star) = \\sum (1 – \\alpha \\lambda_i)^{2k} \\lambda_i [x_i^0]^2 $
\nFor convergence, we note that
\n$latex\u00a01 – \\alpha \\lambda_i $
\nmust be strictly less than 1.<\/p>\n

 <\/p>\n

Note: To be continued…<\/p>\n","protected":false},"excerpt":{"rendered":"

We define standard gradient descent as: Momentum adds a relatively subtle change to gradient descent by replace f with z. Where As such, alpha retains the role of representing the step size. However, we introduce a new constant, beta. Beta adds a decaying effect on momentum, and must be between 0 and 1, noninclusive. We … Continue reading (Nesterov) Accelerated Gradient Descent<\/span><\/a><\/p>\n","protected":false},"author":164,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-796","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9CQAE-cQ","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/posts\/796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/users\/164"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/comments?post=796"}],"version-history":[{"count":20,"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/posts\/796\/revisions"}],"predecessor-version":[{"id":816,"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/posts\/796\/revisions\/816"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/media?parent=796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/categories?post=796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/optml\/wp-json\/wp\/v2\/tags?post=796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}