CValues

Introduced 2023-07-19

CValues is a Chinese human values evaluation benchmark designed to assess the alignment of Chinese Large Language Models (LLMs) with human values. Let me provide you with more details:

  1. Purpose and Context:

    • With the rapid evolution of large language models, there is a growing concern that they may pose risks or have negative social impacts.
    • CValues focuses on evaluating the alignment ability of Chinese LLMs in terms of both safety and responsibility criteria.
    • Previous work mainly assessed LLMs based on knowledge and reasoning abilities, but CValues specifically targets human values alignment, especially in a Chinese context.

  2. Data Collection:

    • The benchmark involves manually collecting adversarial safety prompts across 10 scenarios and inducing responsibility prompts from 8 domains using input from professional experts.

  3. Evaluation Methods:

    • Human Evaluation: Experts assess the alignment of Chinese LLMs with human values.
    • Automatic Evaluation: Multi-choice prompts are constructed for automatic assessment.

  4. Findings:

    • Most Chinese LLMs perform well in terms of safety.
    • However, there is room for improvement in terms of responsibility.
    • Both automatic and human evaluations are crucial for assessing human values alignment.

Source: Conversation with Bing, 3/18/2024 (1) [2307.09705] CValues: Measuring the Values of Chinese Large Language .... https://arxiv.org/abs/2307.09705. (2) VALUE - GitHub Pages. https://value-benchmark.github.io/. (3) Benchmarking 101: Definition, Types, Benefits and How to Use Them - Databox. https://databox.com/what-are-benchmarks. (4) Compare and Conquer: 12 Types of Benchmarking for Measuring ... - Databox. https://databox.com/benchmarking-types. (5) [2307.09705] CValues: Measuring the Values of Chinese Large Language .... https://ar5iv.labs.arxiv.org/html/2307.09705. (6) undefined. https://doi.org/10.48550/arXiv.2307.09705.