AI Definitions: Tokens
/Tokens - Think of a token as the root of a word. “Creat” would be the “root” of words like create, creative, creator, creating, and creation. An LLM looks for correlations — words that go together like giraffe and neck. This group of words are represented by a token. A single word might fall into many tokens since the word might have multiple meanings and the subwords of this word will likely correlate to many other subwords. One token generally corresponds to ~4 characters of text for common English text. Examples