I had a conversation with an engineer recently that went something like this:
Him: “I explicitly configured a safety limit of 8k tokens on our internal chatbot. I don’t want users pasting entire books and draining our API budget.”
Me: “Smart. How are you enforcing it?”
Him: “Well, I just added a .substring(0, 30000) on the user input.”
Me: “Oh, why 30000?”
Him: “Well, the OpenAI docs say 1 token is about 4 characters. So 30000 chars is roughly 7500 tokens. That fits comfortably inside my 8k budget cap.”
Me: “That might be a really dangerous way of calculating.”
Him: “What? The math is fine.”
Me: “Your math assumes English prose. What if someone pasted in a log file full of UUIDs?”
Him: “I don’t get it.”
Me: “Well, let me explain …”
This interaction perfectly captures the friction of transitioning to AI Engineering. A lot of things might sound intuitive from traditional software engineering, but turn out to be dangerous oversimplifications when your fundamental unit of storage changes from fixed bytes to variable-length tokens.
Today, let’s peel back the abstraction and talk about Tokens — the atomic unit that dictates your costs, your latency, and your system’s stability.
It’s Not Reading, It’s Compressing
We tend to think of LLMs as “reading” our prompts.
Mechanically, they are doing no such thing.
Before your text touches the neural network, it goes through a Tokenizer.
Think of the Tokenizer as a Static Compression Algorithm (specifically BPE - Byte Pair Encoding).
It’s not reading your text. Instead, it’s compressing your text into integer IDs.
The “Happy Path” (High Compression):Common English words are mapped 1-to-1.
“Select” → Token ID 931 (1 Token)
“Cloud” → Token ID 4892 (1 Token)
In this case, indeed, the result is ~4 Characters per Token.
The “Edge Case” (Low Compression):This is where the substring logic fails (or underestimated). If the tokenizer encounters high entropy data (very random), it can’t find a pattern, so it falls back to storing every single byte or small chunk individually.
The Real-World Trap: UUIDs
In a database, a UUID is 36 bytes. In an LLM, it is inefficient noise.
Input: user_id: “a1b2-c3d4-e5f6”
Tokenization: Since “a1b2” isn’t a word in the dictionary, the model shreds it into tiny pieces. Depending on the tokenizer (like cl100k_base), a single UUID often eats up 10–15 tokens.
The Crash:
Remember the engineer’s math? He thought 30000 characters were definitely equal to 7500 tokens.
But if the input is a log dump full of hashes, 30000 characters could easily exceed 15000 tokens.
He just tried to shove 15k tokens into an 8k context window.
Boom — instant 400 Bad Request.
Quick Compression Guide:
Therefore, using characters to determine the number of tokens is not highly reliable.
The Latency Bottleneck: O(1) vs O(N)
Another dangerous assumption from traditional engineering: “If I send 2x the data, the request will take 2x as long.”
That is false. Latency in AI is bimodal.
The Input (Prefill): Parallel Processing.Whether you send 100 words or 10000 words of context, the “Time to First Token” (TTFT) grows sub-linearly. It’s a massive matrix multiplication that GPUs swallow whole.
The Output (Decoding): Serial Processing.The model is “Auto-regressive.” It must generate Token 1 before it can figure out Token 2. It is strictly linear O(N).
Engineering Strategy:
If your AI feature feels laggy, don’t just shorten the prompt. Constrain the response.
Bad: “Analyze this log.” (AI might write a 500-token essay).
Good: “Analyze this log. Output JSON only. Fields: {error, fix}.” (AI writes 50 tokens).
The “Verbose Code” Tax
We often debate “Python vs. Java” for backend performance. In the AI world, this debate takes on a new dimension: Token Efficiency.
BPE Tokenizers are trained on the internet (Common Crawl). Because Python is the dominant language of AI, tokenizers are often hyper-optimized for Python’s whitespace-based syntax.
In my own benchmarks using GPT-4’s tokenizer:
Python: def main(): → Highly compressed.
Java: public static void main(String[] args) → This boilerplate consumes more tokens.
While the exact percentage varies by model, using verbose languages in your Few-Shot Prompting examples is literally more expensive than using Python or TypeScript.
These inefficiencies compound fast. Token efficiency is cost efficiency.
The “9.11 > 9.9” Paradox
This explains the famous “Math Bug” in ChatGPT. Why does it often claim 9.11 is bigger than 9.9?
It’s not because the model is “bad at math.” It’s because the model isn’t performing arithmetic but pattern matching!
To a tokenizer, 9.11 is often split into: [9, ., 11].
And 9.9 is split into: [9, ., 9].
The model sees the token “11” and the token “9”, and in its training data, “11” is usually associated with a higher magnitude than “9”.
The Fix: Don’t let an LLM do mental math on floating-point numbers (at least for now). Use Function Calling to offload math to a deterministic tool.
The Universal Strategy for Counting
All of this leads to the fundamental question:
How do you count tokens safely across different LLM ecosystems?
Google’s Gemini doesn’t always have a straightforward local Python library like OpenAI’s tiktoken. As an AI Engineer, you should use a tiered strategy, prioritized by accuracy and latency.
Strategy A: The “Local Replica” (Best for OpenAI / Llama / Mistral)
If the model provider offers an open-source tokenizer, run it locally. This gives you zero-latency, 100% accurate counting.
OpenAI: Use tiktoken (Python) or JTokkit (Java).
Llama: Use HuggingFace’s tokenizers library.
Strategy B: The “API Check” (Best for Gemini / Vertex AI)
Google’s ecosystem is fragmented. While newer SDKs are adding local support, the most reliable method is often to ask the API.
Method: Use model.count_tokens(text) in the Vertex AI SDK.
Trade-off: It adds a network round-trip. Use this sparingly (e.g., only when input size > 90% of expected limit).
Strategy C: The “Pessimistic Heuristic” (Universal Fallback)
If you are in a lightweight environment and cannot load heavy tokenizer libraries, fall back to Math, but change the formula.
Instead of assuming the “Happy Path” (4 chars = 1 token), consider the “Worst Case”.
The Heuristic: Assume 1 Character = 1 Token.
The Logic: If your context limit is 8,000 tokens, hard-cap your input at 8,000 characters.
Result: You waste some context window, but you guarantee safety without dependencies.