We examine differentially non-public (DP) optimization algorithms for stochastic and empirical targets that are neither easy nor convex, and suggest strategies that return a Goldstein-stationary level with pattern complexity bounds that enhance on current works.
\nWe begin by offering a single-pass $(epsilon,delta)<\/annotation><\/semantics><\/math><\/span>(<\/span>\u03f5<\/span>,<\/span>\u03b4<\/span>)<\/span><\/span><\/span><\/span>-DP algorithm that returns an (<\/mo>\u03b1<\/mi>,<\/mo>\u03b2<\/mi>)<\/mo><\/mrow>(alpha,beta)<\/annotation><\/semantics><\/math><\/span>(<\/span>\u03b1<\/span>,<\/span>\u03b2<\/span>)<\/span><\/span><\/span><\/span>-stationary level so long as the dataset is of measurement \u03a9<\/mi>~<\/mo><\/mover>(<\/mo>1<\/mn>\/<\/mi>\u03b1<\/mi>\u03b2<\/mi>3<\/mn><\/msup>+<\/mo>d<\/mi>\/<\/mi>\u03f5<\/mi>\u03b1<\/mi>\u03b2<\/mi>2<\/mn><\/msup>+<\/mo>d<\/mi>3<\/mn>\/<\/mi>4<\/mn><\/mrow><\/msup>\/<\/mi>\u03f5<\/mi>1<\/mn>\/<\/mi>2<\/mn><\/mrow><\/msup>\u03b1<\/mi>\u03b2<\/mi>5<\/mn>\/<\/mi>2<\/mn><\/mrow><\/msup>)<\/mo><\/mrow><\/mrow>widetilde{Omega}left(1\/alphabeta^{3}+d\/epsilonalphabeta^{2}+d^{3\/4}\/epsilon^{1\/2}alphabeta^{5\/2}proper)<\/annotation><\/semantics><\/math><\/span>\u03a9<\/span><\/span><\/svg><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span><\/span>1\/<\/span>\u03b1<\/span>\u03b2<\/span>3<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span>d<\/span>\/<\/span>\u03f5<\/span>\u03b1<\/span>\u03b2<\/span>2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span>d<\/span>3\/4<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\/<\/span>\u03f5<\/span>1\/2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u03b1<\/span>\u03b2<\/span>5\/2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span>, which is \u03a9<\/mi>(<\/mo>d<\/mi><\/msqrt>)<\/mo><\/mrow>Omega(sqrt{d})<\/annotation><\/semantics><\/math><\/span>\u03a9<\/span>(<\/span>d<\/span><\/span><\/span><\/svg><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span> occasions smaller than the algorithm of Zhang et al. [2024] for this process, the place d<\/mi><\/mrow>d<\/annotation><\/semantics><\/math><\/span>d<\/span><\/span><\/span><\/span> is the dimension.\nWe then present a multi-pass polynomial time algorithm which additional improves the pattern complexity to \u03a9<\/mi>~<\/mo><\/mover>(<\/mo>d<\/mi>\/<\/mi>\u03b2<\/mi>2<\/mn><\/msup>+<\/mo>d<\/mi>3<\/mn>\/<\/mi>4<\/mn><\/mrow><\/msup>\/<\/mi>\u03f5<\/mi>\u03b1<\/mi>1<\/mn>\/<\/mi>2<\/mn><\/mrow><\/msup>\u03b2<\/mi>3<\/mn>\/<\/mi>2<\/mn><\/mrow><\/msup>)<\/mo><\/mrow><\/mrow>widetilde{Omega}left(d\/beta^2+d^{3\/4}\/epsilonalpha^{1\/2}beta^{3\/2}proper)<\/annotation><\/semantics><\/math><\/span>\u03a9<\/span><\/span><\/svg><\/span><\/span><\/span><\/span><\/span><\/span>(<\/span><\/span>d<\/span>\/<\/span>\u03b2<\/span>2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span>d<\/span>3\/4<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\/<\/span>\u03f5<\/span>\u03b1<\/span>1\/2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u03b2<\/span>3\/2<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span>, by designing a pattern environment friendly ERM algorithm, and proving that Goldstein-stationary factors generalize from the empirical loss to the inhabitants loss.<\/p>\n\u2020 Work partially finished throughout Apple internship<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"We examine differentially non-public (DP) optimization algorithms for stochastic and empirical targets that are neither easy nor convex, and suggest strategies that return a Goldstein-stationary level with pattern complexity bounds that enhance on current works. We begin by offering a single-pass (\u03f5,\u03b4)(epsilon,delta)(\u03f5,\u03b4)-DP algorithm that returns an (\u03b1,\u03b2)(alpha,beta)(\u03b1,\u03b2)-stationary level so long as the dataset is of […]<\/p>\n","protected":false},"author":2,"featured_media":2180,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[2158,2156,2160,2159,1252,393,2157],"class_list":["post-2178","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-complexity","tag-improved","tag-nonconvex","tag-nonsmooth","tag-optimization","tag-private","tag-sample"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/2178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2178"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/2178\/revisions"}],"predecessor-version":[{"id":2179,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/2178\/revisions\/2179"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/2180"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2178"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2178"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}$