Code Generation on PECC
Metric: Pass@3 (higher is better)
Metric: Pass@3 (higher is better)
| # | Model↕ | Pass@3▼ | Extra Data | Paper | Date↕ | Code |
|---|---|---|---|---|---|---|
| 1 | Claude 3 Haiku | 27.67 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |
| 2 | GPT-3.5 Turbo | 23.75 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |
| 3 | codechat-bison | 11.39 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |
| 4 | chat-bison | 8.48 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |
| 5 | Mixtral-8x7B-Instruct | 8.35 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |
| 6 | Phi-3-mini-128k-instruct | 7.18 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |
| 7 | WizardLM-2-7B | 3.72 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |
| 8 | Llama-3-8B-Instruct | 3.1 | No | PECC: Problem Extraction and Coding Challenges | 2024-04-29 | Code |