Do Kernel and Neural Embeddings Help in Training and Generalization?
| العنوان: | Do Kernel and Neural Embeddings Help in Training and Generalization? |
|---|---|
| المؤلفون: | Rahbar, Arman, 1992, Jorge, Emilio, 1992, Dubhashi, Devdatt, 1965, Haghir Chehreghani, Morteza, 1982 |
| المصدر: | Neural Processing Letters. 55(2):1681-1695 |
| مصطلحات موضوعية: | Gram matrix, Convergence, Kernel embedding, Neural Network |
| الوصف: | Recent results on optimization and generalization properties of neural networks showed that in a simple two-layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines the convergence of the optimization during training. Such analyses also provide upper bounds on the generalization error. We experimentally investigate the implications of these results to deeper networks via embeddings. We regard the layers preceding the final hidden layer as producing different representations of the input data which are then fed to the two-layer model. We show that these representations improve both optimization and generalization. In particular, we investigate three kernel representations when fed to the final hidden layer: the Gaussian kernel and its approximation by random Fourier features, kernels designed to imitate representations produced by neural networks and finally an optimal kernel designed to align the data with target labels. The approximated representations induced by these kernels are fed to the neural network and the optimization and generalization properties of the final model are evaluated and compared. |
| وصف الملف: | electronic |
| URL الوصول: | https://research.chalmers.se/publication/531293 https://research.chalmers.se/publication/531293/file/531293_Fulltext.pdf |
| قاعدة البيانات: | SwePub |
| ResultId |
1 |
|---|---|
| Header |
edsswe SwePub edsswe.oai.research.chalmers.se.2cae1cc4.d242.464d.bd8f.3e4e23d9ac76 977 6 unknown 977.43310546875 |
| PLink |
https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsswe&AN=edsswe.oai.research.chalmers.se.2cae1cc4.d242.464d.bd8f.3e4e23d9ac76&custid=s6537998&authtype=sso |
| FullText |
Array
(
[Availability] => 0
)
Array ( [0] => Array ( [Url] => https://research.chalmers.se/publication/531293# [Name] => EDS - SwePub [Category] => fullText [Text] => View record in SwePub [MouseOverText] => View record in SwePub ) ) |
| Items |
Array
(
[Name] => Title
[Label] => Title
[Group] => Ti
[Data] => Do Kernel and Neural Embeddings Help in Training and Generalization?
)
Array ( [Name] => Author [Label] => Authors [Group] => Au [Data] => <searchLink fieldCode="AR" term="%22Rahbar%2C+Arman%22">Rahbar, Arman</searchLink>, 1992<br /><searchLink fieldCode="AR" term="%22Jorge%2C+Emilio%22">Jorge, Emilio</searchLink>, 1992<br /><searchLink fieldCode="AR" term="%22Dubhashi%2C+Devdatt%22">Dubhashi, Devdatt</searchLink>, 1965<br /><searchLink fieldCode="AR" term="%22Haghir+Chehreghani%2C+Morteza%22">Haghir Chehreghani, Morteza</searchLink>, 1982 ) Array ( [Name] => TitleSource [Label] => Source [Group] => Src [Data] => <i>Neural Processing Letters</i>. 55(2):1681-1695 ) Array ( [Name] => Subject [Label] => Subject Terms [Group] => Su [Data] => <searchLink fieldCode="DE" term="%22Gram+matrix%22">Gram matrix</searchLink><br /><searchLink fieldCode="DE" term="%22Convergence%22">Convergence</searchLink><br /><searchLink fieldCode="DE" term="%22Kernel+embedding%22">Kernel embedding</searchLink><br /><searchLink fieldCode="DE" term="%22Neural+Network%22">Neural Network</searchLink> ) Array ( [Name] => Abstract [Label] => Description [Group] => Ab [Data] => Recent results on optimization and generalization properties of neural networks showed that in a simple two-layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines the convergence of the optimization during training. Such analyses also provide upper bounds on the generalization error. We experimentally investigate the implications of these results to deeper networks via embeddings. We regard the layers preceding the final hidden layer as producing different representations of the input data which are then fed to the two-layer model. We show that these representations improve both optimization and generalization. In particular, we investigate three kernel representations when fed to the final hidden layer: the Gaussian kernel and its approximation by random Fourier features, kernels designed to imitate representations produced by neural networks and finally an optimal kernel designed to align the data with target labels. The approximated representations induced by these kernels are fed to the neural network and the optimization and generalization properties of the final model are evaluated and compared. ) Array ( [Name] => Format [Label] => File Description [Group] => SrcInfo [Data] => electronic ) Array ( [Name] => URL [Label] => Access URL [Group] => URL [Data] => <link linkTarget="URL" linkTerm="https://research.chalmers.se/publication/531293" linkWindow="_blank">https://research.chalmers.se/publication/531293</link><br /><link linkTarget="URL" linkTerm="https://research.chalmers.se/publication/531293/file/531293_Fulltext.pdf" linkWindow="_blank">https://research.chalmers.se/publication/531293/file/531293_Fulltext.pdf</link> ) |
| RecordInfo |
Array
(
[BibEntity] => Array
(
[Identifiers] => Array
(
[0] => Array
(
[Type] => doi
[Value] => 10.1007/s11063-022-10958-8
)
)
[Languages] => Array
(
[0] => Array
(
[Text] => English
)
)
[PhysicalDescription] => Array
(
[Pagination] => Array
(
[PageCount] => 15
[StartPage] => 1681
)
)
[Subjects] => Array
(
[0] => Array
(
[SubjectFull] => Gram matrix
[Type] => general
)
[1] => Array
(
[SubjectFull] => Convergence
[Type] => general
)
[2] => Array
(
[SubjectFull] => Kernel embedding
[Type] => general
)
[3] => Array
(
[SubjectFull] => Neural Network
[Type] => general
)
)
[Titles] => Array
(
[0] => Array
(
[TitleFull] => Do Kernel and Neural Embeddings Help in Training and Generalization?
[Type] => main
)
)
)
[BibRelationships] => Array
(
[HasContributorRelationships] => Array
(
[0] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Rahbar, Arman
)
)
)
[1] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Jorge, Emilio
)
)
)
[2] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Dubhashi, Devdatt
)
)
)
[3] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Haghir Chehreghani, Morteza
)
)
)
)
[IsPartOfRelationships] => Array
(
[0] => Array
(
[BibEntity] => Array
(
[Dates] => Array
(
[0] => Array
(
[D] => 01
[M] => 01
[Type] => published
[Y] => 2023
)
)
[Identifiers] => Array
(
[0] => Array
(
[Type] => issn-print
[Value] => 13704621
)
[1] => Array
(
[Type] => issn-print
[Value] => 1573773X
)
[2] => Array
(
[Type] => issn-locals
[Value] => swepub_free
)
[3] => Array
(
[Type] => issn-locals
[Value] => CTH_SWEPUB
)
)
[Numbering] => Array
(
[0] => Array
(
[Type] => volume
[Value] => 55
)
[1] => Array
(
[Type] => issue
[Value] => 2
)
)
[Titles] => Array
(
[0] => Array
(
[TitleFull] => Neural Processing Letters
[Type] => main
)
)
)
)
)
)
)
|
| IllustrationInfo |