Code Generation on MBPP

Metric: Accuracy (higher is better)

LeaderboardDataset

Loading chart...

Results

Hide extra data

Sort:

#	Model↕	Accuracy▼	Extra Data	Paper	Date↕	Code
1	EG-CFG (DeepSeek-V3-0324)	96.6	No	Execution Guided Line-by-Line Code Generation	2025-06-12	Code
2	QualityFlow (Sonnet-3.5)	94.2	No	QualityFlow: An Agentic Workflow for Program Syn...	2025-01-20	-
3	o1-mini + MapCoder (Hamming.ai)	93.2	Yes	MapCoder: Multi-Agent Code Generation for Compet...	2024-05-18	Code
4	MGDebugger (DeepSeek-V3-0324)	92.4	No	From Code to Correctness: Closing the Last Mile ...	2024-10-02	Code
5	GPT-4 + AgentCoder	91.8	No	AgentCoder: Multi-Agent-based Code Generation wi...	2023-12-20	Code
6	CodeSim (GPT4o)	90.7	No	CODESIM: Multi-Agent Code Generation and Problem...	2025-02-08	Code
7	Jiutian-大模型	90	No	-	-	-
8	GPT-3.5 Turbo (ChatGPT) + AgentCoder	89.9	No	AgentCoder: Multi-Agent-based Code Generation wi...	2023-12-20	Code
9	MapCoder (GPT-4o)	89.7	No	MapCoder: Multi-Agent Code Generation for Compet...	2024-05-18	Code
10	GPT-4 (ChatGPT Plus)	87.5	No	How Does Naming Affect LLMs on Code Analysis Tas...	2023-07-24	-
11	Claude 3 Opus	86.4	No	-	-	-
12	LPW (GPT-4o)	84.8	No	Planning-Driven Programming: A Large Language Mo...	2024-11-21	Code
13	AFlow(GPT-4o-mini)	83.4	No	AFlow: Automating Agentic Workflow Generation	2024-10-14	Code
14	GPT-3.5 Turbo (ChatGPT)	83.2	No	How Does Naming Affect LLMs on Code Analysis Tas...	2023-07-24	-
15	EG-CFG (DeepSeek Coder 1.3b Instruct)	83.2	No	Execution Guided Line-by-Line Code Generation	2025-06-12	Code
16	MapCoder (GPT-4)	83.1	No	MapCoder: Multi-Agent Code Generation for Compet...	2024-05-18	Code
17	o1-mini + Language Agent Tree Search (Hamming.ai)	82.3	No	Language Agent Tree Search Unifies Reasoning Act...	2023-10-06	Code
18	GPT-4 (Bing Chat)	82	No	How Does Naming Affect LLMs on Code Analysis Tas...	2023-07-24	-
19	GPT-3.5 Turbo + Language Agent Tree Search	81.1	No	Language Agent Tree Search Unifies Reasoning Act...	2023-10-06	Code
20	MGDebugger (CodeQwen1.5)	80.8	No	From Code to Correctness: Closing the Last Mile ...	2024-10-02	Code
21	Claude 3 Haiku	80.4	No	-	-	-
22	GPT-4 (Self-Debugging with unit tests + trace)	80.2	No	Teaching Large Language Models to Self-Debug	2023-04-11	Code
23	GPT-4 (few-shot)	80	Yes	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
24	Claude 3 Sonnet	79.4	No	-	-	-
25	Bard (PaLM 2/chat-bison-001)	76.2	No	How Does Naming Affect LLMs on Code Analysis Tas...	2023-07-24	-
26	GPT-3.5 Turbo (Self-Debugging with unit tests + trace)	72.8	No	Teaching Large Language Models to Self-Debug	2023-04-11	Code
27	Claude	71.4	No	How Does Naming Affect LLMs on Code Analysis Tas...	2023-07-24	-
28	code-davinci-002 175B (Self-Debugging with unit tests + trace)	70.8	No	Teaching Large Language Models to Self-Debug	2023-04-11	Code
29	GPT-3.5 Turbo (few-shot)	70.8	Yes	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
30	DeepSeek-Coder-Instruct 33B (few-shot)	70	No	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
31	GPT-3.5 Turbo + INTERVENOR	69.8	No	INTERVENOR: Prompting the Coding Ability of Larg...	2023-11-16	Code
32	code-davinci-002 175B + LEVER	68.9	No	LEVER: Learning to Verify Language-to-Code Gener...	2023-02-16	Code
33	code-davinci-002 175B + CodeT	67.7	No	CodeT: Code Generation with Generated Tests	2022-07-21	Code
34	GPT-3.5 Turbo (3-shot)	67.6	Yes	Teaching Large Language Models to Self-Debug	2023-04-11	Code
35	code-davinci-002 175B + Reviewer	66.9	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
36	code-davinci-002 175B + Coder-Reviewer	66.4	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
37	StarCoder2-15B	66.2	No	StarCoder 2 and The Stack v2: The Next Generation	2024-02-29	Code
38	DeepSeek-Coder-Base 33B (few-shot)	66	No	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
39	Code Llama - Python 70B (3-shot)	65.5	Yes	Code Llama: Open Foundation Models for Code	2023-08-24	Code
40	DeepSeek-Coder-Instruct 6.7B (few-shot)	65.4	No	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
41	code-davinci-002 175B + MBR-Exec	63	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
42	Code Llama 70B (3-shot)	62.4	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
43	Code Llama - Instruct 70B (3-shot)	62.2	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
44	code-davinci-001 175B + CodeT	61.9	No	CodeT: Code Generation with Generated Tests	2022-07-21	Code
45	code-davinci-002 175B (3-shot)	61.4	Yes	Teaching Large Language Models to Self-Debug	2023-04-11	Code
46	Unnatural Code Llama 34B (3-shot)	61.2	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
47	Mixtral 8x7B (3-shot)	60.7	No	Mixtral of Experts	2024-01-08	Code
48	DeepSeek-Coder-Base 6.7B (few-shot)	60.6	No	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
49	code-davinci-001 175B + MBR-Exec	58.2	No	Natural Language to Code Translation with Execut...	2022-04-25	Code
50	Code Llama - Instruct 34B (3-shot)	57	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
51	Code Llama - Python 34B (3-shot)	56.2	Yes	Code Llama: Open Foundation Models for Code	2023-08-24	Code
52	code-cushman-001 12B (CodeT)	55.4	No	CodeT: Code Generation with Generated Tests	2022-07-21	Code
53	Code Llama 34B (3-shot)	55	Yes	Code Llama: Open Foundation Models for Code	2023-08-24	Code
54	StarCoder 15.5B (Self-Debugging with unit tests + trace)	53.2	No	Teaching Large Language Models to Self-Debug	2023-04-11	Code
55	StarCoder 15.5B	52.7	No	StarCoder: may the source be with you!	2023-05-09	Code
56	GPT-3.5 Turbo	52.2	Yes	Code Llama: Open Foundation Models for Code	2023-08-24	Code
57	WizardCoder 15B	51.8	Yes	WizardCoder: Empowering Code Large Language Mode...	2023-06-14	Code
58	PaLM 2-S* (few-shot)	50	No	PaLM 2 Technical Report	2023-05-17	Code
59	CodeGen-Mono 16B + CodeT	49.5	No	CodeT: Code Generation with Generated Tests	2022-07-21	Code
60	Code Llama - Instruct 13B (3-shot)	49.4	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
61	DeepSeek-Coder-Instruct 1.3B (few-shot)	49.4	No	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
62	StarCoderBase 15.5B	49	No	StarCoder: may the source be with you!	2023-05-09	Code
63	Code Llama - Python 13B (3-shot)	49	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
64	Qwen2idae-16x14B (4-shot)	48.6	No	Parameter-Efficient Sparsity Crafting from Dense...	2024-01-05	Code
65	code-cushman-001 12B + MBR-Exec	48.3	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
66	Code Llama - Python 7B (3-shot)	47.6	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
67	Mistral 7B (3-shot)	47.5	No	Mistral 7B	2023-10-10	Code
68	CodeGen 16B + MBR-Exec	47.3	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
69	StarCoder 15.5B (3-shot)	47.2	No	Teaching Large Language Models to Self-Debug	2023-04-11	Code
70	PaLM Coder 540B	47	No	PaLM: Scaling Language Modeling with Pathways	2022-04-05	Code
71	Code Llama 13B (3-shot)	47	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
72	CodeGen 16B + Coder-Reviewer	46.2	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
73	DeepSeek-Coder-Base 1.3B (few-shot)	46.2	No	DeepSeek-Coder: When the Large Language Model Me...	2024-01-25	Code
74	GPT-3.5 Turbo (few-shot)	45.4	No	INTERVENOR: Prompting the Coding Ability of Larg...	2023-11-16	Code
75	Llama 2 70B (zero-shot)	45	No	Llama 2: Open Foundation and Fine-Tuned Chat Mod...	2023-07-18	Code
76	Code Llama - Instruct 7B (3-shot)	44.4	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
77	CodeGen 16B + Reviewer	44.1	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
78	phi-1.5-web 1.3B	43.5	No	Textbooks Are All You Need II: phi-1.5 technical...	2023-09-11	Code
79	Branch-Train-Merge 4x7B (top-2)	42.6	No	Branch-Train-MiX: Mixing Expert LLMs into a Mixt...	2024-03-12	Code
80	Code Llama 7B (3-shot)	41.4	No	Code Llama: Open Foundation Models for Code	2023-08-24	Code
81	Camelidae-8×34B (4-shot)	41.4	No	Parameter-Efficient Sparsity Crafting from Dense...	2024-01-05	Code
82	GPT-3.5 Turbo (0-shot)	39.8	No	INTERVENOR: Prompting the Coding Ability of Larg...	2023-11-16	Code
83	Branch-Train-MiX 4x7B (sampling top-2 experts)	39.4	No	Branch-Train-MiX: Mixing Expert LLMs into a Mixt...	2024-03-12	Code
84	LLaMA 65B (0-shot)	37.7	No	LLaMA: Open and Efficient Foundation Language Mo...	2023-02-27	Code
85	PaLM 540B	36.8	No	PaLM: Scaling Language Modeling with Pathways	2022-04-05	Code
86	SantaCoder 1.1B	35	No	StarCoder: may the source be with you!	2023-05-09	Code
87	InCoder 6.7B + CodeT	34.4	No	CodeT: Code Generation with Generated Tests	2022-07-21	Code
88	Llama 2 34B (0-shot)	33	No	Llama 2: Open Foundation and Fine-Tuned Chat Mod...	2023-07-18	Code
89	Llama 2 13B (0-shot)	30.6	No	Llama 2: Open Foundation and Fine-Tuned Chat Mod...	2023-07-18	Code
90	LLaMA 33B (0-shot)	30.2	No	LLaMA: Open and Efficient Foundation Language Mo...	2023-02-27	Code
91	InCoder 6.7B + MBR-Exec	26.7	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
92	InCoder 6.7B + Coder-Reviewer	26.1	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
93	InCoder 6.7B + Reviewer	24.4	No	Coder Reviewer Reranking for Code Generation	2022-11-29	Code
94	CodeGeeX-13B	24.4	No	CodeGeeX: A Pre-Trained Model for Code Generatio...	2023-03-30	Code
95	LLaMA 13B (0-shot)	22	No	LLaMA: Open and Efficient Foundation Language Mo...	2023-02-27	Code
96	Llama 2 7B (0-shot)	20.8	No	Llama 2: Open Foundation and Fine-Tuned Chat Mod...	2023-07-18	Code
97	InCoder 6.7B (0-shot)	19.4	No	InCoder: A Generative Model for Code Infilling a...	2022-04-12	Code
98	LLaMA 7B (0-shot)	17.7	No	LLaMA: Open and Efficient Foundation Language Mo...	2023-02-27	Code

#1EG-CFG (DeepSeek-V3-0324)SOTA
96.6
Accuracy· 2025-06-12
Execution Guided Line-by-Line Code Generation Code
#2QualityFlow (Sonnet-3.5)SOTA
94.2
Accuracy· 2025-01-20
QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks
#3o1-mini + MapCoder (Hamming.ai)SOTA
93.2
Accuracy· Extra Data· 2024-05-18
MapCoder: Multi-Agent Code Generation for Competitive Problem Solving Code
#4MGDebugger (DeepSeek-V3-0324)
92.4
Accuracy· 2024-10-02
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging Code
#5GPT-4 + AgentCoderSOTA
91.8
Accuracy· 2023-12-20
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation Code
#6CodeSim (GPT4o)
90.7
Accuracy· 2025-02-08
CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging Code
#7Jiutian-大模型
90
Accuracy
No paper
#8GPT-3.5 Turbo (ChatGPT) + AgentCoder
89.9
Accuracy· 2023-12-20
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation Code
#9MapCoder (GPT-4o)
89.7
Accuracy· 2024-05-18
MapCoder: Multi-Agent Code Generation for Competitive Problem Solving Code
#10GPT-4 (ChatGPT Plus)SOTA
87.5
Accuracy· 2023-07-24
How Does Naming Affect LLMs on Code Analysis Tasks?
#11Claude 3 Opus
86.4
Accuracy
No paper
#12LPW (GPT-4o)
84.8
Accuracy· 2024-11-21
Planning-Driven Programming: A Large Language Model Programming Workflow Code
#13AFlow(GPT-4o-mini)
83.4
Accuracy· 2024-10-14
AFlow: Automating Agentic Workflow Generation Code
#14GPT-3.5 Turbo (ChatGPT)
83.2
Accuracy· 2023-07-24
How Does Naming Affect LLMs on Code Analysis Tasks?
#15EG-CFG (DeepSeek Coder 1.3b Instruct)
83.2
Accuracy· 2025-06-12
Execution Guided Line-by-Line Code Generation Code
#16MapCoder (GPT-4)
83.1
Accuracy· 2024-05-18
MapCoder: Multi-Agent Code Generation for Competitive Problem Solving Code
#17o1-mini + Language Agent Tree Search (Hamming.ai)
82.3
Accuracy· 2023-10-06
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models Code
#18GPT-4 (Bing Chat)
82
Accuracy· 2023-07-24
How Does Naming Affect LLMs on Code Analysis Tasks?
#19GPT-3.5 Turbo + Language Agent Tree Search
81.1
Accuracy· 2023-10-06
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models Code
#20MGDebugger (CodeQwen1.5)
80.8
Accuracy· 2024-10-02
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging Code
#21Claude 3 Haiku
80.4
Accuracy
No paper
#22GPT-4 (Self-Debugging with unit tests + trace)SOTA
80.2
Accuracy· 2023-04-11
Teaching Large Language Models to Self-Debug Code
#23GPT-4 (few-shot)
80
Accuracy· Extra Data· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#24Claude 3 Sonnet
79.4
Accuracy
No paper
#25Bard (PaLM 2/chat-bison-001)
76.2
Accuracy· 2023-07-24
How Does Naming Affect LLMs on Code Analysis Tasks?
#26GPT-3.5 Turbo (Self-Debugging with unit tests + trace)
72.8
Accuracy· 2023-04-11
Teaching Large Language Models to Self-Debug Code
#27Claude
71.4
Accuracy· 2023-07-24
How Does Naming Affect LLMs on Code Analysis Tasks?
#28code-davinci-002 175B (Self-Debugging with unit tests + trace)
70.8
Accuracy· 2023-04-11
Teaching Large Language Models to Self-Debug Code
#29GPT-3.5 Turbo (few-shot)
70.8
Accuracy· Extra Data· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#30DeepSeek-Coder-Instruct 33B (few-shot)
70
Accuracy· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#31GPT-3.5 Turbo + INTERVENOR
69.8
Accuracy· 2023-11-16
INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair Code
#32code-davinci-002 175B + LEVERSOTA
68.9
Accuracy· 2023-02-16
LEVER: Learning to Verify Language-to-Code Generation with Execution Code
#33code-davinci-002 175B + CodeTSOTA
67.7
Accuracy· 2022-07-21
CodeT: Code Generation with Generated Tests Code
#34GPT-3.5 Turbo (3-shot)
67.6
Accuracy· Extra Data· 2023-04-11
Teaching Large Language Models to Self-Debug Code
#35code-davinci-002 175B + Reviewer
66.9
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#36code-davinci-002 175B + Coder-Reviewer
66.4
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#37StarCoder2-15B
66.2
Accuracy· 2024-02-29
StarCoder 2 and The Stack v2: The Next Generation Code
#38DeepSeek-Coder-Base 33B (few-shot)
66
Accuracy· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#39Code Llama - Python 70B (3-shot)
65.5
Accuracy· Extra Data· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#40DeepSeek-Coder-Instruct 6.7B (few-shot)
65.4
Accuracy· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#41code-davinci-002 175B + MBR-Exec
63
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#42Code Llama 70B (3-shot)
62.4
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#43Code Llama - Instruct 70B (3-shot)
62.2
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#44code-davinci-001 175B + CodeT
61.9
Accuracy· 2022-07-21
CodeT: Code Generation with Generated Tests Code
#45code-davinci-002 175B (3-shot)
61.4
Accuracy· Extra Data· 2023-04-11
Teaching Large Language Models to Self-Debug Code
#46Unnatural Code Llama 34B (3-shot)
61.2
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#47Mixtral 8x7B (3-shot)
60.7
Accuracy· 2024-01-08
Mixtral of Experts Code
#48DeepSeek-Coder-Base 6.7B (few-shot)
60.6
Accuracy· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#49code-davinci-001 175B + MBR-ExecSOTA
58.2
Accuracy· 2022-04-25
Natural Language to Code Translation with Execution Code
#50Code Llama - Instruct 34B (3-shot)
57
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#51Code Llama - Python 34B (3-shot)
56.2
Accuracy· Extra Data· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#52code-cushman-001 12B (CodeT)
55.4
Accuracy· 2022-07-21
CodeT: Code Generation with Generated Tests Code
#53Code Llama 34B (3-shot)
55
Accuracy· Extra Data· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#54StarCoder 15.5B (Self-Debugging with unit tests + trace)
53.2
Accuracy· 2023-04-11
Teaching Large Language Models to Self-Debug Code
#55StarCoder 15.5B
52.7
Accuracy· 2023-05-09
StarCoder: may the source be with you!Code
#56GPT-3.5 Turbo
52.2
Accuracy· Extra Data· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#57WizardCoder 15B
51.8
Accuracy· Extra Data· 2023-06-14
WizardCoder: Empowering Code Large Language Models with Evol-Instruct Code
#58PaLM 2-S* (few-shot)
50
Accuracy· 2023-05-17
PaLM 2 Technical Report Code
#59CodeGen-Mono 16B + CodeT
49.5
Accuracy· 2022-07-21
CodeT: Code Generation with Generated Tests Code
#60Code Llama - Instruct 13B (3-shot)
49.4
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#61DeepSeek-Coder-Instruct 1.3B (few-shot)
49.4
Accuracy· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#62StarCoderBase 15.5B
49
Accuracy· 2023-05-09
StarCoder: may the source be with you!Code
#63Code Llama - Python 13B (3-shot)
49
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#64Qwen2idae-16x14B (4-shot)
48.6
Accuracy· 2024-01-05
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks Code
#65code-cushman-001 12B + MBR-Exec
48.3
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#66Code Llama - Python 7B (3-shot)
47.6
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#67Mistral 7B (3-shot)
47.5
Accuracy· 2023-10-10
Mistral 7B Code
#68CodeGen 16B + MBR-Exec
47.3
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#69StarCoder 15.5B (3-shot)
47.2
Accuracy· 2023-04-11
Teaching Large Language Models to Self-Debug Code
#70PaLM Coder 540BSOTA
47
Accuracy· 2022-04-05
PaLM: Scaling Language Modeling with Pathways Code
#71Code Llama 13B (3-shot)
47
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#72CodeGen 16B + Coder-Reviewer
46.2
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#73DeepSeek-Coder-Base 1.3B (few-shot)
46.2
Accuracy· 2024-01-25
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Code
#74GPT-3.5 Turbo (few-shot)
45.4
Accuracy· 2023-11-16
INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair Code
#75Llama 2 70B (zero-shot)
45
Accuracy· 2023-07-18
Llama 2: Open Foundation and Fine-Tuned Chat Models Code
#76Code Llama - Instruct 7B (3-shot)
44.4
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#77CodeGen 16B + Reviewer
44.1
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#78phi-1.5-web 1.3B
43.5
Accuracy· 2023-09-11
Textbooks Are All You Need II: phi-1.5 technical report Code
#79Branch-Train-Merge 4x7B (top-2)
42.6
Accuracy· 2024-03-12
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM Code
#80Code Llama 7B (3-shot)
41.4
Accuracy· 2023-08-24
Code Llama: Open Foundation Models for Code Code
#81Camelidae-8×34B (4-shot)
41.4
Accuracy· 2024-01-05
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks Code
#82GPT-3.5 Turbo (0-shot)
39.8
Accuracy· 2023-11-16
INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair Code
#83Branch-Train-MiX 4x7B (sampling top-2 experts)
39.4
Accuracy· 2024-03-12
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM Code
#84LLaMA 65B (0-shot)
37.7
Accuracy· 2023-02-27
LLaMA: Open and Efficient Foundation Language Models Code
#85PaLM 540B
36.8
Accuracy· 2022-04-05
PaLM: Scaling Language Modeling with Pathways Code
#86SantaCoder 1.1B
35
Accuracy· 2023-05-09
StarCoder: may the source be with you!Code
#87InCoder 6.7B + CodeT
34.4
Accuracy· 2022-07-21
CodeT: Code Generation with Generated Tests Code
#88Llama 2 34B (0-shot)
33
Accuracy· 2023-07-18
Llama 2: Open Foundation and Fine-Tuned Chat Models Code
#89Llama 2 13B (0-shot)
30.6
Accuracy· 2023-07-18
Llama 2: Open Foundation and Fine-Tuned Chat Models Code
#90LLaMA 33B (0-shot)
30.2
Accuracy· 2023-02-27
LLaMA: Open and Efficient Foundation Language Models Code
#91InCoder 6.7B + MBR-Exec
26.7
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#92InCoder 6.7B + Coder-Reviewer
26.1
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#93InCoder 6.7B + Reviewer
24.4
Accuracy· 2022-11-29
Coder Reviewer Reranking for Code Generation Code
#94CodeGeeX-13B
24.4
Accuracy· 2023-03-30
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X Code
#95LLaMA 13B (0-shot)
22
Accuracy· 2023-02-27
LLaMA: Open and Efficient Foundation Language Models Code
#96Llama 2 7B (0-shot)
20.8
Accuracy· 2023-07-18
Llama 2: Open Foundation and Fine-Tuned Chat Models Code
#97InCoder 6.7B (0-shot)
19.4
Accuracy· 2022-04-12
InCoder: A Generative Model for Code Infilling and Synthesis Code
#98LLaMA 7B (0-shot)
17.7
Accuracy· 2023-02-27
LLaMA: Open and Efficient Foundation Language Models Code