CodeFuseEval

CodeFuseEval is a Code Generation benchmark that combines the multi-tasking scenarios of CodeFuse Model with the benchmarks of HumanEval-x and MBPP. This benchmark is designed to evaluate the performance of models in various multi-tasking tasks, including code completion, code generation from natural language, test case generation, cross-language code translation, and code generation from Chinese commands, among others.

The evaluation of the generated codes involves compiling and running in multiple programming languages. The versions of the programming language environments and packages we use are as follows:

| Dependency | Version | | ---------- |----------| | Python | 3.10.9 | | JDK | 18.0.2.1 | | Node.js | 16.14.0 | | js-md5 | 0.7.3 | | C++ | 11 | | g++ | 7.5.0 | | Boost | 1.75.0 | | OpenSSL | 3.0.0 | | go | 1.18.4 | | cargo | 1.71.1 |