AI Definitions: Tokens
/Tokens – A token can be thought of as the basic unit used by AI to process language, similar to the root of a word. “Creat” would represent words like create, creative, creator, creating, and creation. An LLM looks for correlations between words that go together like giraffe and neck. A single word might fall into many tokens since it might have multiple meanings and its subword might correlate to many other subwords. One token generally corresponds to ~4 characters of text for common English text. Examples: https://platform.openai.com/tokenizer