TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Methods/BPE

BPE

Byte Pair Encoding

Natural Language ProcessingIntroduced 200018980 papers
Source Paper

Description

Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations).

Lei Mao has a detailed blog post that explains how this works.

Papers Using This Method

Making Language Model a Hierarchical Classifier and Generator2025-07-17DASViT: Differentiable Architecture Search for Vision Transformer2025-07-17Best Practices for Large-Scale, Pixel-Wise Crop Mapping and Transfer Learning Workflows2025-07-16DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition2025-07-16Langevin Flows for Modeling Neural Latent Dynamics2025-07-15Generative Click-through Rate Prediction with Applications to Search Advertising2025-07-15Biological Processing Units: Leveraging an Insect Connectome to Pioneer Biofidelic Neural Architectures2025-07-15KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding2025-07-15Hashed Watermark as a Filter: Defeating Forging and Overwriting Attacks in Weight-based Neural Network Watermarking2025-07-15Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI2025-07-13Learning from Synthetic Labs: Language Models as Auction Participants2025-07-12Comparative Analysis of Vision Transformers and Traditional Deep Learning Approaches for Automated Pneumonia Detection in Chest X-Rays2025-07-11SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression2025-07-08Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving2025-07-08Geo-Registration of Terrestrial LiDAR Point Clouds with Satellite Images without GNSS2025-07-08Tile-Based ViT Inference with Visual-Cluster Priors for Zero-Shot Multi-Species Plant Identification2025-07-08A Wireless Foundation Model for Multi-Task Prediction2025-07-08Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate2025-07-08SV-DRR: High-Fidelity Novel View X-Ray Synthesis Using Diffusion Model2025-07-07Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations2025-07-07