Chakkrit Termritthikun, Ayaz Umer, Suwichaya Suwanwimolkul, Feng Xia, Ivan Lee
Recent advancements in deep convolutional neural networks have significantly improved the performance of saliency prediction. However, the manual configuration of the neural network architectures requires domain knowledge expertise and can still be time-consuming and error-prone. To solve this, we propose a new Neural Architecture Search (NAS) framework for saliency prediction with two contributions. Firstly, a supernet for saliency prediction is built with a weight-sharing network containing all candidate architectures, by integrating a dynamic convolution into the encoder-decoder in the supernet, termed SalNAS. Secondly, despite the fact that SalNAS is highly efficient (20.98 million parameters), it can suffer from the lack of generalization. To solve this, we propose a self-knowledge distillation approach, termed Self-KD, that trains the student SalNAS with the weighted average information between the ground truth and the prediction from the teacher model. The teacher model, while sharing the same architecture, contains the best-performing weights chosen by cross-validation. Self-KD can generalize well without the need to compute the gradient in the teacher model, enabling an efficient training system. By utilizing Self-KD, SalNAS outperforms other state-of-the-art saliency prediction models in most evaluation rubrics across seven benchmark datasets while being a lightweight model. The code will be available at https://github.com/chakkritte/SalNAS
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Saliency Detection | SALICON | AUC | 0.87 | SalNAS-XL + Self-KD |
| Saliency Detection | SALICON | CC | 0.909 | SalNAS-XL + Self-KD |
| Saliency Detection | SALICON | IG | 0.913 | SalNAS-XL + Self-KD |
| Saliency Detection | SALICON | KLD | 0.195 | SalNAS-XL + Self-KD |
| Saliency Detection | SALICON | NSS | 2.019 | SalNAS-XL + Self-KD |
| Saliency Detection | SALICON | SIM | 0.796 | SalNAS-XL + Self-KD |
| Saliency Detection | SALICON | sAUC | 0.749 | SalNAS-XL + Self-KD |
| Saliency Prediction | SALICON | AUC | 0.87 | SalNAS-XL + Self-KD |
| Saliency Prediction | SALICON | CC | 0.909 | SalNAS-XL + Self-KD |
| Saliency Prediction | SALICON | IG | 0.913 | SalNAS-XL + Self-KD |
| Saliency Prediction | SALICON | KLD | 0.195 | SalNAS-XL + Self-KD |
| Saliency Prediction | SALICON | NSS | 2.019 | SalNAS-XL + Self-KD |
| Saliency Prediction | SALICON | SIM | 0.796 | SalNAS-XL + Self-KD |
| Saliency Prediction | SALICON | sAUC | 0.749 | SalNAS-XL + Self-KD |
| Few-Shot Transfer Learning for Saliency Prediction | SALICON | AUC | 0.87 | SalNAS-XL + Self-KD |
| Few-Shot Transfer Learning for Saliency Prediction | SALICON | CC | 0.909 | SalNAS-XL + Self-KD |
| Few-Shot Transfer Learning for Saliency Prediction | SALICON | IG | 0.913 | SalNAS-XL + Self-KD |
| Few-Shot Transfer Learning for Saliency Prediction | SALICON | KLD | 0.195 | SalNAS-XL + Self-KD |
| Few-Shot Transfer Learning for Saliency Prediction | SALICON | NSS | 2.019 | SalNAS-XL + Self-KD |
| Few-Shot Transfer Learning for Saliency Prediction | SALICON | SIM | 0.796 | SalNAS-XL + Self-KD |
| Few-Shot Transfer Learning for Saliency Prediction | SALICON | sAUC | 0.749 | SalNAS-XL + Self-KD |