TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Papers/An Analysis of Neural Language Modeling at Multiple Scales

An Analysis of Neural Language Modeling at Multiple Scales

Stephen Merity, Nitish Shirish Keskar, Richard Socher

2018-03-22Language Modelling
PaperPDFCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCodeCode(official)

Abstract

Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.

Results

TaskDatasetMetricValueModel
Language ModellingPenn Treebank (Character Level)Bit per Character (BPC)1.1753-layer AWD-LSTM
Language ModellingPenn Treebank (Character Level)Bit per Character (BPC)1.1876-layer QRNN
Language ModellingWikiText-103Test perplexity334 layer QRNN
Language ModellingWikiText-103Validation perplexity324 layer QRNN
Language ModellingHutter PrizeBit per Character (BPC)1.2323-layer AWD-LSTM
Language Modellingenwik8Bit per Character (BPC)1.232AWD-LSTM (3 layers)

Related Papers

Visual-Language Model Knowledge Distillation Method for Image Quality Assessment2025-07-21Making Language Model a Hierarchical Classifier and Generator2025-07-17VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning2025-07-17The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations2025-07-17Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities2025-07-17Assay2Mol: large language model-based drug design using BioAssay context2025-07-16Describe Anything Model for Visual Question Answering on Text-rich Images2025-07-16InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing2025-07-16