OpenAPI completion refined
TextsMITIntroduced 2024-05-24
A human-refined dataset of OpenAPI definitions based on the APIs.guru OpenAPI directory.
The dataset was collected from the APIs.guru OpenAPI definitions directory. The directory contains more than 4,000 definitions in yaml format. Analysis of the repository revealed that about 75% of the definitions in the directory are produced by a handful of major companies like Amazon, Google, and Microsoft. To avoid the dataset bias towards a specific producer, the maximum number of definitions from a single producer was limited to 20. Multiple versions of the same API were also excluded from the dataset as they are likely to contain very similar definitions.
Benchmarks
Code Completion/Correctness, avg., %Code Completion/Correctness, max., %Code Completion/Validness, avg., %Code Completion/Validness, max., %OpenAPI code completion/Correctness, avg., %OpenAPI code completion/Correctness, max., %OpenAPI code completion/Validness, avg., %OpenAPI code completion/Validness, max., %