The tokens that carry no signal
Every AI call is billed on tokens in, not information in. A verbose system prompt repeated on every stateless request, a stack trace pasted whole into a debugging session, the same retrieved document chunk arriving in three consecutive RAG turns, a CSV export with four thousand identical trailing rows — none of that carries proportionally more signal than a tenth of its length would, but every token of it is billed at full price, every single time.
Six engines, each targeting a different shape of waste
- normalize — collapses redundant whitespace, normalizes JSON formatting, and removes exact-duplicate blocks. The baseline pass, cheap and safe on almost any content.
- command-output — detects long logs, diffs, and stack traces and truncates them to their informative head and tail rather than dropping them or sending them whole; a 40,000-line log becomes the handful of lines that actually matter for debugging, plus a note that the rest was cut.
- rag-dedup — scans across messages in the same conversation for verbatim duplicate blocks and removes the repeats. This is the specific fix for the RAG symptom above: the same retrieved chunk arriving turn after turn gets sent once, not N times.
- caveman — rule-based compression of verbose English prose down to its core meaning, applied only to user- and system-authored blocks over 300 characters. Assistant output is never touched, because the model chose that phrasing on purpose and rewriting it would change the record of what it actually said.
- tabular — detects large CSV or markdown-table blocks (two or more delimiter characters per line, over a row-count threshold) and keeps the header plus a bounded number of data rows, appending a human-readable notice about how many rows were omitted rather than silently truncating data a downstream process might assume is complete.
- llmlingua2 — a machine-learning compression model running as its own sidecar service, for the cases pattern-matching alone can't catch: dense natural-language prose with no repeated structure to exploit. Verified live at roughly 40–43% token reduction on real long-form text, with internal auto-chunking so there's no hard ceiling on input length.
None of these run unconditionally. The pipeline resolves per request via header, per API key, or as an organization default, and every response reports back exactly what happened: a compression ratio, the raw-versus-compressed token counts, and which specific engines actually fired. If an engine's own inflation guard detects that compression made a block larger — which can genuinely happen on already-terse, structured, or digit-heavy text — it passes the block through unchanged rather than force a "saving" that isn't real.
A concrete before-and-after
Take a realistic support-debugging exchange: a customer pastes a 500-line application log into the first turn, the assistant asks a follow-up, and the customer's next message re-includes the same log block plus three new lines because their client app resent the full context rather than just the delta. Without compression, that second turn re-bills the entire 500-line log a second time for three lines of new information. With rag-dedup enabled, the exact-duplicate block from turn one is recognized and dropped from turn two — only the three genuinely new lines and the surrounding conversation are billed. With command-output also enabled, the surviving 500-line log in turn one itself is trimmed to its informative head and tail rather than sent in full, because a stack trace's middle hundreds of lines are rarely where the actual signal is. The two engines solve different halves of the same real conversation.
Why llmlingua2 runs as a separate service
The other five engines are pure pattern-matching — fast, deterministic, and running in-process. llmlingua2 is different: it's a real ML model (Microsoft's LLMLingua-2, bert-base-multilingual under the hood) that needs its own inference, so it runs as a dedicated sidecar service rather than inline in the gateway. The gateway calls it over an authenticated internal request with a configurable timeout, and if that call fails, times out, or the sidecar is simply unreachable, the engine fails open — the request proceeds uncompressed rather than blocking on a dependency that exists purely to save money, never to gate correctness. Verified against realistic long-form prose, it compresses in the 40–43% range; it deliberately preserves digits, code fragments, and structural characters rather than compressing them, which means synthetic digit- or code-heavy test content can show little to no reduction — that's the model behaving correctly, not a bug, since garbling a digit in a support ticket is a much worse outcome than a smaller savings number.
What the response actually shows you
A compressed request comes back with response headers you can read directly, no dashboard required: a compression ratio between 0 and 1, the raw and compressed token counts, and a comma-separated list of which engines actually fired on that specific request — because not every engine fires on every request, and knowing that "rag-dedup" ran but "tabular" didn't tells you something real about what kind of content you just sent. Combined with the routed-cost and counterfactual-cost figures every response already carries, compression's contribution to the total saving is visible on its own line, not folded invisibly into one aggregate number you have to take on faith.
The default configuration is also just a default, not a ceiling or a floor. A request can override which engines run for that call specifically by naming them in a header; an API key can carry its own standing engine list so every request from that integration behaves consistently without the caller having to repeat itself; and an organization can set what any key without its own override should fall back to. The resolution order is always the same — request header, then per-key setting, then org default, then none — so at any point you can reason about exactly which of the three explains what happened to a given request.
Compression is one lever among several — pull them together
Token compression works alongside, not instead of, the platform's other cost controls: provider-side prompt caching that cuts repeated input cost by roughly 90% on supported models, a semantic cache that serves near-duplicate prompts from a prior response at effectively zero marginal cost, and — for genuinely long-running conversations — an opt-in summarization step that replaces an over-threshold volume of injected history with a compact, cheap-model-generated handoff summary rather than resending the full transcript on every turn.
Compression doesn't cost you your retention posture
A reasonable question at this point is whether running content through an extra pipeline stage before billing creates a second place where your data has to be trusted. It doesn't. Compression happens in-flight, on the request as it's being prepared for the provider — it is not a storage layer, and it doesn't create one. An organization running in zero-retention mode gets the exact same compression behavior as an organization on a retention plan, because the pipeline has nothing to do with what gets persisted afterward; those are two independent decisions enforced at two different points in the request lifecycle, and neither one weakens the other.
The unifying idea is the same one that shows up everywhere else on this blog: don't ask a human to remember to optimize a request. Classify what the content actually is, apply the specific compression that shape of content responds to, and show the receipt for exactly what was saved — automatically, on every request, without anyone having to think about it after the pipeline is configured once.