Do Kernel and Neural Embeddings Help in Training and Generalization?
| العنوان: | Do Kernel and Neural Embeddings Help in Training and Generalization? |
|---|---|
| المؤلفون: | Rahbar, Arman, 1992, Jorge, Emilio, 1992, Dubhashi, Devdatt, 1965, Haghir Chehreghani, Morteza, 1982 |
| المصدر: | Neural Processing Letters. 55(2):1681-1695 |
| مصطلحات موضوعية: | Gram matrix, Convergence, Kernel embedding, Neural Network |
| الوصف: | Recent results on optimization and generalization properties of neural networks showed that in a simple two-layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines the convergence of the optimization during training. Such analyses also provide upper bounds on the generalization error. We experimentally investigate the implications of these results to deeper networks via embeddings. We regard the layers preceding the final hidden layer as producing different representations of the input data which are then fed to the two-layer model. We show that these representations improve both optimization and generalization. In particular, we investigate three kernel representations when fed to the final hidden layer: the Gaussian kernel and its approximation by random Fourier features, kernels designed to imitate representations produced by neural networks and finally an optimal kernel designed to align the data with target labels. The approximated representations induced by these kernels are fed to the neural network and the optimization and generalization properties of the final model are evaluated and compared. |
| وصف الملف: | electronic |
| URL الوصول: | https://research.chalmers.se/publication/531293 https://research.chalmers.se/publication/531293/file/531293_Fulltext.pdf |
| قاعدة البيانات: | SwePub |
| تدمد: | 13704621 1573773X |
|---|---|
| DOI: | 10.1007/s11063-022-10958-8 |