Papers With Code 2 | ML Benchmarks, SotA Results & Code

Olympic 2024 is a human-annotated dataset that contains 220 high-quality instance. Each instance consists of an input tuple (user instruction, response 1, response confidence of response 1, response 2, response confidence of response 2) and an output tuple (evaluation explanation, evaluation result). The evaluation result would be either ‘1’ or ‘2’, indicating that response 1 or response 2 is better. To ensure the quality of human annotations, we involve three experts to concurrently annotate the same data point during the annotation process.