Code Completion on SAFIM

Metric: Algorithmic (higher is better)

LeaderboardDataset

Loading chart...

Results

Sort:

#	Model↕	Algorithmic▼	Extra Data	Paper	Date↕	Code
1	deepseek-coder-33b-base	60.78	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
2	deepseek-coder-6.7b-base	54.74	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
3	starcoderbase	44.11	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
4	gpt-4-1106-preview	42.11	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
5	CodeLlama-13b-hf	41.41	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
6	deepseek-coder-1.3b-base	41.2	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
7	CodeLlama-34b-hf	38.55	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
8	CodeLlama-7b-hf	34.68	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
9	gpt-3.5-turbo-0301	31.24	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
10	codegen-16B-multi	25.94	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
11	incoder-6B	25.16	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
12	codegen-6B-multi	23.6	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
13	codegen-2B-multi	23.49	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
14	incoder-1B	21.06	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
15	codegen-350M-multi	16.3	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code

#1deepseek-coder-33b-baseSOTA
60.78
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#2deepseek-coder-6.7b-base
54.74
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#3starcoderbase
44.11
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#4gpt-4-1106-preview
42.11
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#5CodeLlama-13b-hf
41.41
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#6deepseek-coder-1.3b-base
41.2
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#7CodeLlama-34b-hf
38.55
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#8CodeLlama-7b-hf
34.68
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#9gpt-3.5-turbo-0301
31.24
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#10codegen-16B-multi
25.94
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#11incoder-6B
25.16
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#12codegen-6B-multi
23.6
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#13codegen-2B-multi
23.49
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#14incoder-1B
21.06
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#15codegen-350M-multi
16.3
Algorithmic· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code