Code Generation on RES-Q

Metric: pass@1 (higher is better)

LeaderboardDataset

Loading chart...

Results

Submit a result

Sort:

#	Model↕	pass@1▼	Extra Data	Paper	Date↕	Code
1	QurrentOS-coder + Claude 3.5 Sonnet	58	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
2	QurrentOS-coder + GPT-4o	46	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
3	QurrentOS-coder + GPT-4 Turbo	37	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
4	QurrentOS-coder + Claude 3 Opus	36	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
5	QurrentOS-coder + GPT-4	30	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
6	QurrentOS-coder + Gemini 1.5 Pro	30	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
7	QurrentOS-coder + DeepSeek-Coder-V2	29	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
8	QurrentOS-coder + Llama 3 70b	20	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code
9	QurrentOS-coder + Qwen-72B-Instruct	18	No	RES-Q: Evaluating Code-Editing Large Language Mo...	2024-06-24	Code

#1QurrentOS-coder + Claude 3.5 SonnetSOTA
58
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#2QurrentOS-coder + GPT-4o
46
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#3QurrentOS-coder + GPT-4 Turbo
37
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#4QurrentOS-coder + Claude 3 Opus
36
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#5QurrentOS-coder + GPT-4
30
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#6QurrentOS-coder + Gemini 1.5 Pro
30
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#7QurrentOS-coder + DeepSeek-Coder-V2
29
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#8QurrentOS-coder + Llama 3 70b
20
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code
#9QurrentOS-coder + Qwen-72B-Instruct
18
pass@1· 2024-06-24
RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale Code