TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

SotA/Natural Language Processing/Code Generation

Code Generation

71 benchmarks1697 papers

Code Generation is an important field to predict explicit code or program structure from multimodal data sources such as incomplete code, programs in another programming language, natural language descriptions or execution examples. Code Generation tools can assist the development of automatic programming tools to improve programming productivity.

<span class="description-source">Source: Deep Learning for Source Code Modeling and Generation </span>

Image source: Measuring Coding Challenge Competence With APPS

Benchmarks

Code Generation on MBPP

Accuracy

Code Generation on CoNaLa

BLEUExact Match Accuracy

Code Generation on Django

AccuracyBLEU Score

Code Generation on WikiSQL

Execution AccuracyExact Match Accuracy

Code Generation on RES-Q

pass@1

Code Generation on APPS

Introductory Pass@1Interview Pass@1Competition Pass@1Competition Pass@anyPass@1Interview Pass@anyIntroductory Pass@anyCompetition Pass@5Interview Pass@5Introductory Pass@5Competition Pass@1000Interview Pass@1000Introductory Pass@1000

Code Generation on CodeContests

Test Set pass@1Val Set pass@1Test Set pass@5Val Set pass@5

Code Generation on CodeSearchNet - Java

Smoothed BLEU-4

Code Generation on CodeSearchNet - JavaScript

Smoothed BLEU-4

Code Generation on CodeSearchNet - Php

Smoothed BLEU-4

Code Generation on HumanEval

Pass@1

Code Generation on PECC

Pass@3

Code Generation on WebApp1K-React

pass@1

Code Generation on CodeSearchNet

Smoothed BLEU-4

Code Generation on CodeSearchNet - Go

Smoothed BLEU-4

Code Generation on CodeSearchNet - Python

Smoothed BLEU-4

Code Generation on CodeSearchNet - Ruby

Smoothed BLEU-4

Code Generation on CoNaLa-Ext

BLEU

Code Generation on WebApp1k-Duo-React

pass@1

Code Generation on DSEval-LeetCode

Pass Ratew/o Intactw/o PE

Code Generation on Turbulence

CorrSc

Code Generation on Livecodebench

Acc

Code Generation on Shellcode_IA32

BLEU-4Exact Match Accuracy

Code Generation on BigCodeBench-Complete

Pass@1

Code Generation on BigCodeBench-Instruct

Pass@1

Code Generation on CONCODE

Exact MatchBLEUCodeBLEU

Code Generation on CodeXGLUE - CodeSearchNet

Python/BLEUPython/CodeBLEUJava/EMPython/EMJava/BLEUJava/CodeBLEU

Code Generation on CodeXGLUE - CodeTrans

Accuracy (C#→Java)Accuracy (Java→C#)BLEU (C#→Java)BLEU (Java→C#)CodeBLEU (C#→Java)CodeBLEU (Java→C#)

Code Generation on HumanEval-ET

Pass@1

Code Generation on MBPP-ET

Pass@1

Code Generation on NLC2CMD

Accuracy

Code Generation on Verified Smart Contract Code Comments

BLEU score

Code Generation on Android Repos

Perplexity

Code Generation on FloCo

BLEUCodeBLEU

Code Generation on RESIDE TestSet-S

F1 (Hidden Test Set)

Code Generation on VerilogEval

Pass Rate

Code Generation on TACO-BAAI

easy pass@1