Seminar: Mathematics of Learning - From Hilbert's 13th Problem to Fluctuating Hydrodynamics

Description

Neural networks (NN) constitute a class of functions that nonlinearly depend on a finite number of parameters which in turn are typically used for (optimal) approximation of an unknown function from a given set of function values.

In this seminar, we plan first to investigate the richness and approximation power of neural networks by considering topics like Hilbert's 13th problem, approximation error estimates, and discretization errror estimates for NN-discretizations of partial differential equations.

Furthermore we will adress the algebraic minimization problems for (optimal) NN-parameters. Popular numerical solution methods like stochastic gradient-type descent will finally lead us to novel reinterpretations of NN in terms of fluctuating hydrodynamics that are currently investigated in a MATH+ project of the organizers.

Participants are expected to have basic knowledge in numerical analysis as communicated, e.g. in the lectures Numerik I - III at FU Berlin.

Schedule

The introduction of the topics takes place on 11.04.2019.

Date Title Speaker Literature
23.05.2019 Hilbert's 13th problem Issagali [1][2]
06.06.2019 The Kolmogorov-Arnold representation theorem and extensions Merten [3][4][5]
13.06.2019 Towards neural networks: Constructive proofs of Kolmogorov-Arnold Boisserée [6][7][8][9]
20.06.2019 Towards neural networks: Approximation instead of representation Srinivasan [10]
27.06.2019 Iterative parameter computation by gradient-type methods Kostré [11][12][13]
04.07.2019 Fluctuating hydrodynamics: The Dean-Kawasaki model Kornhuber
11.07.2019 Neural networks as interacting particle systems Gräser

Organizational Matters

Literature

  1. A.G. Vitushkin: On Hilbert's thirteenth problem and related questions. Russian Mathematical Surveys, 59(1), 11-24 (2004)
  2. V. Brattka: From Hilbert’s 13th Problem to the theory of neural networks: constructive aspects of Kolmogorov’s Superposition Theorem. In Kolmogorov’s heritage in mathematics. Springer, Berlin, Heidelberg, 253-280 (2007)
  3. A. N. Kolmogorov: On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. Dokl. Akad. Nauk SSSR, 114:5,953–956 (1957)
  4. V. I. Arnol'd: On the representability of a function of two variables in the form \(\chi \phi(x) + \psi(y)\). Uspekhi Matematicheskikh Nauk 12.2, 119-121 (1957)
  5. D.A. Sprecher: On the structure of continuous functions of several variables. Transactions of the American Mathematical Society, 115, 340-355 (1965)
  6. J. Braun, M. Griebel: On a constructive proof of Kolmogorov’s superposition theorem. Constructive approximation, Springer (2009)
  7. D.A. Sprecher: A Numerical Implementation of Kolmogorov’s Superpositions. Neural Networks, 9(5), 765-772 (1996)
  8. D.A. Sprecher: A Numerical Implementation of Kolmogorov’s Superpositions II. Neural Networks, 10(3), 447–457 (1997)
  9. V. Kurkova: Kolmogorov's theorem is relevant. Neural Computation 3(4), 617-622, (1991)
  10. A. Pinkus: Approximation theory of the MLP model in neural networks. Acta Numerica, 143-195 (1999)
  11. A. Botev, H. Ritter, D. Barber: Practical Gauss-Newton optimisation for deep learning. arxiv:1706.03662, 2017
  12. Y. A. LeCun, L. and Bottou, G. B. Orr, K.-R. Müller: Efficient {BackProp}. In Neural networks: Tricks of the trade. Springer, 6-50 (1998) doi
  13. D. P. Kingma, J. Ba: Adam: A method for stochastic optimization. arxiv:1412.6980

Kontakt

Prof. Dr. Carsten Gräser graeser@mi.fu-berlin.de Arnimallee 6, Raum 121
Sekretariat Frau Engel: Arnimallee 6, Room 131
Prof. Dr. Ralf Kornhuber ralf.kornhuber@fu-berlin.de Arnimallee 6, Raum 130
Sekretariat Frau Engel: Arnimallee 6, Room 131