Code Completion on SAFIM

Metric: API (higher is better)

LeaderboardDataset

Loading chart...

Results

Sort:

#	Model↕	API▼	Extra Data	Paper	Date↕	Code
1	deepseek-coder-33b-base	75.16	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
2	deepseek-coder-6.7b-base	69.68	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
3	starcoderbase	68.06	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
4	gpt-4-1106-preview	62.58	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
5	deepseek-coder-1.3b-base	62.58	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
6	CodeLlama-13b-hf	59.68	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
7	CodeLlama-34b-hf	56.45	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
8	gpt-3.5-turbo-0301	53.87	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
9	incoder-6B	48.06	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
10	CodeLlama-7b-hf	46.77	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
11	incoder-1B	43.87	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
12	codegen-2B-multi	32.26	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
13	codegen-16B-multi	31.29	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
14	codegen-6B-multi	27.74	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code
15	codegen-350M-multi	26.45	No	Evaluation of LLMs on Syntax-Aware Code Fill-in-...	2024-03-07	Code

#1deepseek-coder-33b-baseSOTA
75.16
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#2deepseek-coder-6.7b-base
69.68
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#3starcoderbase
68.06
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#4gpt-4-1106-preview
62.58
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#5deepseek-coder-1.3b-base
62.58
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#6CodeLlama-13b-hf
59.68
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#7CodeLlama-34b-hf
56.45
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#8gpt-3.5-turbo-0301
53.87
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#9incoder-6B
48.06
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#10CodeLlama-7b-hf
46.77
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#11incoder-1B
43.87
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#12codegen-2B-multi
32.26
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#13codegen-16B-multi
31.29
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#14codegen-6B-multi
27.74
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code
#15codegen-350M-multi
26.45
API· 2024-03-07
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks Code