Petr Lorenc, Tommaso Gargiani, Jan Pichl, Jakub Konrád, Petr Marek, Ondřej Kobza, Jan Šedivý
Conversational agents are usually designed for closed-world environments. Unfortunately, users can behave unexpectedly. Based on the open-world environment, we often encounter the situation that the training and test data are sampled from different distributions. Then, data from different distributions are called out-of-domain (OOD). A robust conversational agent needs to react to these OOD utterances adequately. Thus, the importance of robust OOD detection is emphasized. Unfortunately, collecting OOD data is a challenging task. We have designed an OOD detection algorithm independent of OOD data that outperforms a wide range of current state-of-the-art algorithms on publicly available datasets. Our algorithm is based on a simple but efficient approach of combining metric learning with adaptive decision boundary. Furthermore, compared to other algorithms, we have found that our proposed algorithm has significantly improved OOD performance in a scenario with a lower number of classes while preserving the accuracy for in-domain (IND) classes.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Intent Detection | BANKING-77 (50% known) | 1:1 Accuracy | 83.78 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | BANKING-77 (50% known) | F1-score | 84.93 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | OOS(25%known) | 1:1 Accuracy | 91.81 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | OOS(25%known) | F1-score | 85.9 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | BANKING-77 (75% known) | 1:1 Accuracy | 84.4 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | BANKING-77 (75% known) | F1-score | 88.39 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | BANKING77 (25%known) | 1:1 Accuracy | 85.71 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | BANKING77 (25%known) | F1-score | 78.86 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | OOS(75%known) | 1:1 Accuracy | 88.54 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | OOS(75%known) | F1-score | 92.21 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | OOS(50%known) | 1:1 Accuracy | 88.81 | Metric learning + Adaptive Decision Boundary |
| Intent Detection | OOS(50%known) | F1-score | 89.19 | Metric learning + Adaptive Decision Boundary |