NeedleBench

Introduction to the Needle In A Haystack Test

The Needle In A Haystack test, inspired by NeedleInAHaystack, is an evaluation method that randomly inserts key information into long texts to create prompts for large language models (LLMs). This test aims to determine whether LLMs can effectively extract key information from extensive texts, thereby assessing their capabilities in processing and understanding long documents.

Task Overview

Within the NeedleBench framework of OpenCompass, we have designed a series of increasingly challenging test scenarios to comprehensively evaluate the models' abilities in long text information extraction and reasoning:

• Single-Needle Retrieval Task (S-RT): Assesses an LLM's ability to extract a single key piece of information from a long text, testing its precision in recalling specific details within broad narratives. This task corresponds to the original Needle In A Haystack test setup.

• Multi-Needle Retrieval Task (M-RT): Explores an LLM's capability to retrieve multiple related pieces of information from long texts, simulating real-world scenarios of complex queries on comprehensive documents.

• Multi-Needle Reasoning Task (M-RS): Evaluates an LLM's long-text capabilities by extracting and utilizing multiple key pieces of information, requiring the model to have a comprehensive understanding of each key information fragment.

• Ancestral Trace Challenge (ATC): Uses the "relational needle" to test an LLM's ability to handle multi-layer logical challenges in real long texts. In the ATC task, a series of logical reasoning questions test the model's memory and analytical skills for every detail in the text. For this task, we remove the irrelevant text (Haystack) setting, designing all texts as critical information, requiring the LLM to accurately use all the content and reasoning in the text to answer the questions.

Refer to the OpenCompass Documentation for more details.