AI Definitions: Tokenization

Tokenization – The first step in natural language processing, this happens when an LLM creates a digital representation (or token) of a real thing—everything gets a number; written words are translated into numbers. Think of a token as the root of a word. “Creat” is the “root” of many words, for instance, including Create, Creative, Creator, Creating, and Creation. “Create” would be an example of a token. This is the first step in natural language processing. Examples

More AI definitions here