Code Completion on SAFIM

Metric: Control (higher is better)

LeaderboardDataset

Loading chart...

Results

Sort:

#	Model↕	Control▼	Extra Data	Paper	Date↕	Code
1	deepseek-coder-33b-base	71.1	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
2	deepseek-coder-6.7b-base	65.79	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
3	CodeLlama-13b-hf	57.25	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
4	gpt-4-1106-preview	55.15	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
5	starcoderbase	54.46	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
6	deepseek-coder-1.3b-base	54.1	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
7	CodeLlama-34b-hf	53.98	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
8	CodeLlama-7b-hf	53.56	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
9	gpt-3.5-turbo-0301	37.48	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
10	codegen-16B-multi	35.74	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
11	codegen-6B-multi	34.8	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
12	codegen-2B-multi	32.89	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
13	incoder-6B	28.16	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
14	codegen-350M-multi	26.06	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
15	incoder-1B	22.89	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code

#1deepseek-coder-33b-baseSOTA
71.1
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#2deepseek-coder-6.7b-base
65.79
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#3CodeLlama-13b-hf
57.25
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#4gpt-4-1106-preview
55.15
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#5starcoderbase
54.46
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#6deepseek-coder-1.3b-base
54.1
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#7CodeLlama-34b-hf
53.98
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#8CodeLlama-7b-hf
53.56
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#9gpt-3.5-turbo-0301
37.48
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#10codegen-16B-multi
35.74
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#11codegen-6B-multi
34.8
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#12codegen-2B-multi
32.89
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#13incoder-6B
28.16
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#14codegen-350M-multi
26.06
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#15incoder-1B
22.89
Control· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code