Code Completion on SAFIM

Metric: Average (higher is better)

LeaderboardDataset

Loading chart...

Results

Sort:

#	Model↕	Average▼	Extra Data	Paper	Date↕	Code
1	deepseek-coder-33b-base	69.01	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
2	deepseek-coder-6.7b-base	63.4	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
3	starcoderbase	55.54	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
4	gpt-4-1106-preview	53.28	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
5	CodeLlama-13b-hf	52.78	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
6	deepseek-coder-1.3b-base	52.63	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
7	CodeLlama-34b-hf	49.66	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
8	CodeLlama-7b-hf	45	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
9	gpt-3.5-turbo-0301	40.86	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
10	incoder-6B	33.79	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
11	codegen-16B-multi	30.99	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
12	codegen-2B-multi	29.55	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
13	incoder-1B	29.27	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
14	codegen-6B-multi	28.71	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
15	codegen-350M-multi	22.94	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code

#1deepseek-coder-33b-baseSOTA
69.01
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#2deepseek-coder-6.7b-base
63.4
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#3starcoderbase
55.54
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#4gpt-4-1106-preview
53.28
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#5CodeLlama-13b-hf
52.78
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#6deepseek-coder-1.3b-base
52.63
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#7CodeLlama-34b-hf
49.66
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#8CodeLlama-7b-hf
45
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#9gpt-3.5-turbo-0301
40.86
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#10incoder-6B
33.79
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#11codegen-16B-multi
30.99
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#12codegen-2B-multi
29.55
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#13incoder-1B
29.27
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#14codegen-6B-multi
28.71
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#15codegen-350M-multi
22.94
Average· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code