Report
Large Language Models as Code Executors: An Exploratory Study
| العنوان: | Large Language Models as Code Executors: An Exploratory Study |
|---|---|
| المؤلفون: | Lyu, Chenyang, Yan, Lecheng, Xing, Rui, Li, Wenxi, Samih, Younes, Ji, Tianbo, Wang, Longyue |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computation and Language, Computer Science - Artificial Intelligence |
| الوصف: | The capabilities of Large Language Models (LLMs) have significantly evolved, extending from natural language processing to complex tasks like code understanding and generation. We expand the scope of LLMs' capabilities to a broader context, using LLMs to execute code snippets to obtain the output. This paper pioneers the exploration of LLMs as code executors, where code snippets are directly fed to the models for execution, and outputs are returned. We are the first to comprehensively examine this feasibility across various LLMs, including OpenAI's o1, GPT-4o, GPT-3.5, DeepSeek, and Qwen-Coder. Notably, the o1 model achieved over 90% accuracy in code execution, while others demonstrated lower accuracy levels. Furthermore, we introduce an Iterative Instruction Prompting (IIP) technique that processes code snippets line by line, enhancing the accuracy of weaker models by an average of 7.22% (with the highest improvement of 18.96%) and an absolute average improvement of 3.86% against CoT prompting (with the highest improvement of 19.46%). Our study not only highlights the transformative potential of LLMs in coding but also lays the groundwork for future advancements in automated programming and the completion of complex tasks. |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2410.06667 |
| رقم الانضمام: | edsarx.2410.06667 |
| قاعدة البيانات: | arXiv |
| ResultId |
1 |
|---|---|
| Header |
edsarx arXiv edsarx.2410.06667 1128 3 Report report 1128.01770019531 |
| PLink |
https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsarx&AN=edsarx.2410.06667&custid=s6537998&authtype=sso |
| FullText |
Array
(
[Availability] => 0
)
Array ( [0] => Array ( [Url] => http://arxiv.org/abs/2410.06667 [Name] => EDS - Arxiv [Category] => fullText [Text] => View record in Arxiv [MouseOverText] => View record in Arxiv ) ) |
| Items |
Array
(
[Name] => Title
[Label] => Title
[Group] => Ti
[Data] => Large Language Models as Code Executors: An Exploratory Study
)
Array ( [Name] => Author [Label] => Authors [Group] => Au [Data] => <searchLink fieldCode="AR" term="%22Lyu%2C+Chenyang%22">Lyu, Chenyang</searchLink><br /><searchLink fieldCode="AR" term="%22Yan%2C+Lecheng%22">Yan, Lecheng</searchLink><br /><searchLink fieldCode="AR" term="%22Xing%2C+Rui%22">Xing, Rui</searchLink><br /><searchLink fieldCode="AR" term="%22Li%2C+Wenxi%22">Li, Wenxi</searchLink><br /><searchLink fieldCode="AR" term="%22Samih%2C+Younes%22">Samih, Younes</searchLink><br /><searchLink fieldCode="AR" term="%22Ji%2C+Tianbo%22">Ji, Tianbo</searchLink><br /><searchLink fieldCode="AR" term="%22Wang%2C+Longyue%22">Wang, Longyue</searchLink> ) Array ( [Name] => DatePubCY [Label] => Publication Year [Group] => Date [Data] => 2024 ) Array ( [Name] => Subset [Label] => Collection [Group] => HoldingsInfo [Data] => Computer Science ) Array ( [Name] => Subject [Label] => Subject Terms [Group] => Su [Data] => <searchLink fieldCode="DE" term="%22Computer+Science+-+Computation+and+Language%22">Computer Science - Computation and Language</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Science+-+Artificial+Intelligence%22">Computer Science - Artificial Intelligence</searchLink> ) Array ( [Name] => Abstract [Label] => Description [Group] => Ab [Data] => The capabilities of Large Language Models (LLMs) have significantly evolved, extending from natural language processing to complex tasks like code understanding and generation. We expand the scope of LLMs' capabilities to a broader context, using LLMs to execute code snippets to obtain the output. This paper pioneers the exploration of LLMs as code executors, where code snippets are directly fed to the models for execution, and outputs are returned. We are the first to comprehensively examine this feasibility across various LLMs, including OpenAI's o1, GPT-4o, GPT-3.5, DeepSeek, and Qwen-Coder. Notably, the o1 model achieved over 90% accuracy in code execution, while others demonstrated lower accuracy levels. Furthermore, we introduce an Iterative Instruction Prompting (IIP) technique that processes code snippets line by line, enhancing the accuracy of weaker models by an average of 7.22% (with the highest improvement of 18.96%) and an absolute average improvement of 3.86% against CoT prompting (with the highest improvement of 19.46%). Our study not only highlights the transformative potential of LLMs in coding but also lays the groundwork for future advancements in automated programming and the completion of complex tasks. ) Array ( [Name] => TypeDocument [Label] => Document Type [Group] => TypDoc [Data] => Working Paper ) Array ( [Name] => URL [Label] => Access URL [Group] => URL [Data] => <link linkTarget="URL" linkTerm="http://arxiv.org/abs/2410.06667" linkWindow="_blank">http://arxiv.org/abs/2410.06667</link> ) Array ( [Name] => AN [Label] => Accession Number [Group] => ID [Data] => edsarx.2410.06667 ) |
| RecordInfo |
Array
(
[BibEntity] => Array
(
[Subjects] => Array
(
[0] => Array
(
[SubjectFull] => Computer Science - Computation and Language
[Type] => general
)
[1] => Array
(
[SubjectFull] => Computer Science - Artificial Intelligence
[Type] => general
)
)
[Titles] => Array
(
[0] => Array
(
[TitleFull] => Large Language Models as Code Executors: An Exploratory Study
[Type] => main
)
)
)
[BibRelationships] => Array
(
[HasContributorRelationships] => Array
(
[0] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Lyu, Chenyang
)
)
)
[1] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Yan, Lecheng
)
)
)
[2] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Xing, Rui
)
)
)
[3] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Li, Wenxi
)
)
)
[4] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Samih, Younes
)
)
)
[5] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Ji, Tianbo
)
)
)
[6] => Array
(
[PersonEntity] => Array
(
[Name] => Array
(
[NameFull] => Wang, Longyue
)
)
)
)
[IsPartOfRelationships] => Array
(
[0] => Array
(
[BibEntity] => Array
(
[Dates] => Array
(
[0] => Array
(
[D] => 09
[M] => 10
[Type] => published
[Y] => 2024
)
)
)
)
)
)
)
|
| IllustrationInfo |