Alaa Saade, Alice Coucke, Alexandre Caulier, Joseph Dureau, Adrien Ball, Théodore Bluche, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet
We consider the problem of performing Spoken Language Understanding (SLU) on small devices typical of IoT applications. Our contributions are twofold. First, we outline the design of an embedded, private-by-design SLU system and show that it has performance on par with cloud-based commercial solutions. Second, we release the datasets used in our experiments in the interest of reproducibility and in the hope that they can prove useful to the SLU community.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Dialogue | Snips-SmartSpeaker | Accuracy-EN (%) | 68.7 | Snips |
| Dialogue | Snips-SmartSpeaker | Accuracy-FR (%) | 75.1 | Snips |
| Dialogue | Snips-SmartSpeaker | Accuracy-EN (%) | 47.8 | |
| Dialogue | Snips-SmartSpeaker | Accuracy-FR (%) | 42.3 | |
| Dialogue | Snips-SmartLights | Accuracy (%) | 84.2 | Snips |
| Dialogue | Snips-SmartLights | Accuracy (%) | 79.3 | |
| Spoken Language Understanding | Snips-SmartSpeaker | Accuracy-EN (%) | 68.7 | Snips |
| Spoken Language Understanding | Snips-SmartSpeaker | Accuracy-FR (%) | 75.1 | Snips |
| Spoken Language Understanding | Snips-SmartSpeaker | Accuracy-EN (%) | 47.8 | |
| Spoken Language Understanding | Snips-SmartSpeaker | Accuracy-FR (%) | 42.3 | |
| Spoken Language Understanding | Snips-SmartLights | Accuracy (%) | 84.2 | Snips |
| Spoken Language Understanding | Snips-SmartLights | Accuracy (%) | 79.3 | |
| Dialogue Understanding | Snips-SmartSpeaker | Accuracy-EN (%) | 68.7 | Snips |
| Dialogue Understanding | Snips-SmartSpeaker | Accuracy-FR (%) | 75.1 | Snips |
| Dialogue Understanding | Snips-SmartSpeaker | Accuracy-EN (%) | 47.8 | |
| Dialogue Understanding | Snips-SmartSpeaker | Accuracy-FR (%) | 42.3 | |
| Dialogue Understanding | Snips-SmartLights | Accuracy (%) | 84.2 | Snips |
| Dialogue Understanding | Snips-SmartLights | Accuracy (%) | 79.3 |