TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Datasets/SNIPS

SNIPS

SNIPS Natural Language Understanding benchmark

TextsUnknownIntroduced 2018-01-01

The SNIPS Natural Language Understanding benchmark is a dataset of over 16,000 crowdsourced queries distributed among 7 user intents of various complexity:

  • SearchCreativeWork (e.g. Find me the I, Robot television show),
  • GetWeather (e.g. Is it windy in Boston, MA right now?),
  • BookRestaurant (e.g. I want to book a highly rated restaurant in Paris tomorrow night),
  • PlayMusic (e.g. Play the last track from BeyoncĂ© off Spotify),
  • AddToPlaylist (e.g. Add Diamonds to my roadtrip playlist),
  • RateBook (e.g. Give 6 stars to Of Mice and Men),
  • SearchScreeningEvent (e.g. Check the showtimes for Wonder Woman in Paris). The training set contains of 13,084 utterances, the validation set and the test set contain 700 utterances each, with 100 queries per intent.

Source: https://paperswithcode.com/paper/snips-voice-platform-an-embedded-spoken/

Benchmarks

Image Classification/F1 MacroIntent Detection/AccuracyIntent Discovery/ARIOpen Intent Discovery/ACCOpen Intent Discovery/ARIOpen Intent Discovery/NMISlot Filling/F1Slot Filling/F1 (1-shot) avgSlot Filling/F1 (5-shot) avgText Clustering/ACCText Clustering/ARIText Clustering/NMIZero-Shot Learning/Accuracy

Related Benchmarks

SNIPS (25% known)/Intent Detection/F1SNIPS (50% known)/Intent Detection/F1SNIPS (75% known)/Intent Detection/F1Snips-SmartLights/Dialogue/Accuracy (%)Snips-SmartLights/Dialogue Understanding/Accuracy (%)Snips-SmartLights/Spoken Language Understanding/Accuracy (%)Snips-SmartSpeaker/Dialogue/Accuracy-EN (%)Snips-SmartSpeaker/Dialogue/Accuracy-FR (%)Snips-SmartSpeaker/Dialogue Understanding/Accuracy-EN (%)Snips-SmartSpeaker/Dialogue Understanding/Accuracy-FR (%)Snips-SmartSpeaker/Spoken Language Understanding/Accuracy-EN (%)Snips-SmartSpeaker/Spoken Language Understanding/Accuracy-FR (%)

Statistics

Papers
256
Benchmarks
13

Links

Homepage

Tasks

Image ClassificationIntent DetectionIntent DiscoveryOpen Intent DiscoveryOut of Distribution (OOD) DetectionSlot FillingText ClusteringZero-Shot Learning