TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Models/CodeBERT

CodeBERT

Reported on 32 benchmarks across 5 tasks · 3 papers · 32 SOTA

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Computer Code18 results

  • Code SearchonCodeXGLUE - AdvTest
    MRR· 2021-02-09
    27.19
    best: 44.7 (CodeT5+ 770M)
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code SearchonCodeXGLUE - WebQueryTest
    Accuracy· 2021-02-09
    47.8
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code SearchonCodeXGLUE - WebQueryTest
    F1· 2021-02-09
    58.95
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Program SynthesisonManyTypes4TypeScript
    Average Accuracy· 2020-02-19
    61.72
    best: 71.27 (CodeTIDAL5)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Program SynthesisonManyTypes4TypeScript
    Average F1· 2020-02-19
    59.57
    best: 60.57 (GraphCodeBERT)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Program SynthesisonManyTypes4TypeScript
    Average Precision· 2020-02-19
    59.34
    best: 60.06 (GraphCodeBERT)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Program SynthesisonManyTypes4TypeScript
    Average Recall· 2020-02-19
    59.8
    best: 61.08 (GraphCodeBERT)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Code SearchonCodeSearchNet
    Go· 2020-02-19
    69.3
    best: 97.7 (cpt-code S)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Code SearchonCodeSearchNet
    JS· 2020-02-19
    74.8
    best: 86.5 (cpt-code M)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Code SearchonCodeSearchNet
    Java· 2020-02-19
    86.8
    best: 94.4 (cpt-code M)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Code SearchonCodeSearchNet
    Overall· 2020-02-19
    76
    best: 93.5 (cpt-code M)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Code SearchonCodeSearchNet
    PHP· 2020-02-19
    70.6
    best: 97.2 (cpt-code M)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Code SearchonCodeSearchNet
    Python· 2020-02-19
    84
    best: 99.9 (cpt-code M)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Code SearchonCodeSearchNet
    Ruby· 2020-02-19
    70.6
    best: 86.3 (cpt-code S)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Type predictiononManyTypes4TypeScript
    Average Accuracy· 2020-02-19
    61.72
    best: 71.27 (CodeTIDAL5)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Type predictiononManyTypes4TypeScript
    Average F1· 2020-02-19
    59.57
    best: 60.57 (GraphCodeBERT)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Type predictiononManyTypes4TypeScript
    Average Precision· 2020-02-19
    59.34
    best: 60.06 (GraphCodeBERT)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155
  • Type predictiononManyTypes4TypeScript
    Average Recall· 2020-02-19
    59.8
    best: 61.08 (GraphCodeBERT)
    SOTA
    CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesarXiv:2002.08155

Natural Language Processing14 results

  • Code GenerationonShellcode_IA32
    BLEU-4· 2022-02-08
    91.7
    SOTA
    Can We Generate Shellcodes via Natural Language? An Empirical StudyarXiv:2202.03755
  • Code GenerationonShellcode_IA32
    Exact Match Accuracy· 2022-02-08
    89.75
    SOTA
    Can We Generate Shellcodes via Natural Language? An Empirical StudyarXiv:2202.03755
  • Code GenerationonCodeXGLUE - CodeTrans
    Accuracy (C#→Java)· 2021-02-09
    58
    best: 66.9 (CodeT5)
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code GenerationonCodeXGLUE - CodeTrans
    Accuracy (Java→C#)· 2021-02-09
    59
    best: 65.9 (CodeT5)
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code GenerationonCodeXGLUE - CodeTrans
    BLEU (C#→Java)· 2021-02-09
    72.14
    best: 79.87 (CodeT5)
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code GenerationonCodeXGLUE - CodeTrans
    BLEU (Java→C#)· 2021-02-09
    79.92
    best: 84.03 (CodeT5)
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code GenerationonCodeXGLUE - CodeTrans
    CodeBLEU (C#→Java)· 2021-02-09
    79.41
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code GenerationonCodeXGLUE - CodeTrans
    CodeBLEU (Java→C#)· 2021-02-09
    85.1
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code RepaironCodeXGLUE - Bugs2Fix
    Accuracy (medium)· 2021-02-09
    5.2
    best: 13.87 (NSEdit)
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code RepaironCodeXGLUE - Bugs2Fix
    Accuracy (small)· 2021-02-09
    16.4
    best: 24.04 (NSEdit)
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code RepaironCodeXGLUE - Bugs2Fix
    BLEU (medium)· 2021-02-09
    91.07
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code RepaironCodeXGLUE - Bugs2Fix
    BLEU (small)· 2021-02-09
    77.42
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code RepaironCodeXGLUE - Bugs2Fix
    CodeBLEU (medium)· 2021-02-09
    87.52
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664
  • Code RepaironCodeXGLUE - Bugs2Fix
    CodeBLEU (small)· 2021-02-09
    75.58
    SOTA
    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and GenerationarXiv:2102.04664