NaturalCodeBench

Introduced 2024-05-07

NaturalCodeBench (NCB) is a comprehensive code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks¹². It comprises 402 high-quality problems in Python and Java, meticulously selected from an online coding service, covering 6 different domains¹².

The seed problems of NCB are cleaned from the queries in coding online services, spanning across 6 domains: Artificial Intelligence, Data Science, Algorithm and Data Structure, Front-End, Software Engineering, and System Administration¹.

Here is a summary of the number of problems in each domain¹:

  • Software Engineering: 132 problems
  • Data Science: 100 problems
  • Algorithm and Data Structure: 95 problems
  • System Administration: 33 problems
  • Artificial Intelligence: 28 problems
  • Front-End: 14 problems

The development set of NCB, which contains 140 problems (70 in Python and 70 in Java), is released for research purposes¹. The data format includes a unique identifier for each question, the problem description, testcases, setup code, a reference solution, and the domain of the problem¹.

(1) GitHub - THUDM/NaturalCodeBench. https://github.com/THUDM/NaturalCodeBench. (2) NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval .... https://arxiv.org/abs/2405.04520. (3) 清华、智谱AI 团队推出代码评测基准 NaturalCodeBench .... https://blog.csdn.net/www3300300/article/details/138752139. (4) NaturalCodeBench: Examining Coding Performance .... https://www.x-mol.com/paper/1788507497390862336. (5) undefined. https://doi.org/10.48550/arXiv.2405.04520.