EnvBench

EnvBench is a comprehensive benchmark for automating environment setup - an important task in software engineering. We have collected the largest dataset to date for this task and introduced a robust framework for developing and evaluating LLM-based agents that tackle environment setup challenges.

Our benchmark includes:

• 994 repositories: 329 Python and 665 JVM-based (Java, Kotlin) projects
• Genuine configuration challenges: Carefully selected repositories that cannot be configured with simple deterministic scripts
• Evaluation metrics: Static analysis for missing imports in Python and compilation checks for JVM languages
• Baselines: Zero-shot baselines and agentic workflows tested with GPT-4o and GPT-4o-mini