wonderbread

A benchmark + dataset for evaluating multimodal models on business process management (BPM) tasks.

A dataset containing 2,928 human demonstrations of 598 workflows, each of which include a textual description of the workflow intent, a full screen recording of the human completing the workflow, an action trace detailing which screen elements were interacted with, and a manually-written step-by-step guide in the form of a "Standard Operating Procedure" (SOP). The demonstrations cover a wide range of tasks from 4 different websites interfaces sourced from the excellent WebArena benchmark. A benchmark with 6 novel BPM tasks that measure the ability of a model to generate accurate documentation, assist in knowledge transfer, and improve workflows. The tasks are designed to evaluate models on real-world applications of BPM tools, with a more broad focus than simple end-to-end automation. A fully automated evaluation harness for evaluating models on the introduced tasks. The harness is designed to be easy to use and can be run on any multimodal FM that can be prompted via API. The harness is open-source and available on our GitHub repository.