Jiaqi Mu, Suma Bhat, Pramod Viswanath
Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a {\em very simple}, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a few top dominating directions from the word vectors -- that renders off-the-shelf representations {\em even stronger}. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and { text classification}) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Sentiment Analysis | MR | Accuracy | 78.26 | GRU-RNN-WORD2VEC |
| Sentiment Analysis | SST-5 Fine-grained classification | Accuracy | 45.02 | GRU-RNN-WORD2VEC |
| Subjectivity Analysis | SUBJ | Accuracy | 91.85 | GRU-RNN-GLOVE |
| Text Classification | TREC-6 | Error | 7 | GRU-RNN-GLOVE |
| Classification | TREC-6 | Error | 7 | GRU-RNN-GLOVE |