Token

A unit of text, such as a word, word fragment, or punctuation mark, that a language model processes or generates.

Last updated

AI & Automation

In plain English

A tokenizer breaks text into units the model can work with. One word may become several tokens, and different tokenizers can divide the same text differently. Token counts help measure the amount of text processed and how much can fit within a model's context window.

Example

A startup's chatbot receives a long support question. Its tokenizer may split uncommon product names into several tokens, so the token count can exceed the number of words.

Sources

  1. Google for Developers: Machine Learning Glossary
  2. Hugging Face: Tokenization algorithms

Along the line: Model Basics

View full line