Dynamic Meta-Embeddings for Improved Sentence Representations

Douwe Kiela, Changhan Wang, Kyunghyun Cho

2018-04-21EMNLP 2018 10Word Embeddings

Abstract

While one of the first steps in many NLP systems is selecting what pre-trained word embeddings to use, we argue that such a step is better left for neural networks to figure out by themselves. To that end, we introduce dynamic meta-embeddings, a simple yet effective method for the supervised learning of embedding ensembles, which leads to state-of-the-art performance within the same model class on a variety of tasks. We subsequently show how the technique can be used to shed new light on the usage of word embeddings in NLP systems.

Results

Task	Dataset	Metric	Value	Model
Natural Language Inference	SNLI	% Test Accuracy	86.7	512D Dynamic Meta-Embeddings
Natural Language Inference	SNLI	% Train Accuracy	91.6	512D Dynamic Meta-Embeddings

Related Papers

Speak2Sign3D: A Multi-modal Pipeline for English Speech to American Sign Language Animation2025-07-09 Computational Detection of Intertextual Parallels in Biblical Hebrew: A Benchmark Study Using Transformer-Based Language Models2025-06-30 Including Semantic Information via Word Embeddings for Skeleton-based Action Recognition2025-06-23 Low-resource keyword spotting using contrastively trained transformer acoustic word embeddings2025-06-21 Characterizing Linguistic Shifts in Croatian News via Diachronic Word Embeddings2025-06-16 Learning Obfuscations Of LLM Embedding Sequences: Stained Glass Transform2025-06-11 Recommender systems, stigmergy, and the tyranny of popularity2025-06-06 Static Word Embeddings for Sentence Semantic Representation2025-06-05