Qolca Team · 2026-07-16 · 11 min read
Most AI products overpay because they stuff everything into one enormous prompt and send it to the most expensive model on every request. The fix is not a cheaper vendor. It is treating tokens like a scarce resource: decompose the flow, retrieve instead of context-stuffing, cache the static parts, and reserve the expensive model for the one step that needs it.
There is a failure mode in almost every AI product that is quietly bleeding money: one giant system prompt, stuffed with every rule, every few-shot example, every edge case, and every scrap of company knowledge, sent to the most expensive model available on every single request. It works beautifully in the demo. Then real traffic arrives, the bill arrives with it, and latency creeps up because the model is re-reading a small novel before it answers 'what are your hours?'. The instinct is to go shopping for a cheaper vendor. That is the wrong move. The problem is not the price of the model — it is that you are spending tokens like they are free. They are not. This is a piece about treating them like the scarce, metered resource they actually are.
Every token in your prompt is paid for on every request. A frontier-tier model costs on the order of five to fifteen dollars per million output tokens and a few dollars per million input tokens; a small, fast model in the Haiku class costs roughly a third to a fifth of that. Those numbers sound tiny until you do the multiplication. Suppose your system prompt is 4,000 tokens — a modest size once you have pasted in your product catalog, your tone guidelines, and a dozen examples. In a ten-turn conversation, the model re-reads that prompt on every turn, so you have paid for 40,000 input tokens just to restate instructions the model already absorbed on turn one. Now multiply by every conversation, every day. The waste is not dramatic per request; it is dramatic in aggregate, and it scales linearly with your success.
Cost is only half the tax. The other half is latency, and latency is a product feature. A model cannot emit its first token until it has read your entire prompt, and a bigger prompt means more to read and more to reason over before anything appears on screen. A 200-token routing decision and a 6,000-token everything-prompt feel completely different to a user waiting for a reply. When teams complain that their chatbot feels sluggish, the culprit is very often not the model — it is the wall of context they force it to wade through on every turn. You are paying twice for the same mistake: once in dollars, once in seconds.
The mental shift is simple to state and surprisingly hard to adopt: stop asking "what could the model possibly need to know?" and start asking "what does this specific step actually require?" Once you hold tokens to that standard, five patterns fall out naturally. None of them are exotic. They are just discipline applied to a resource most teams treat as infinite.
A single request to "handle this customer message" is really several smaller jobs wearing a trench coat: figure out what language they are speaking, work out what they want, gather any relevant history, and then write a good reply. Bundled into one prompt, every one of those sub-jobs pays for the full context and the full model. Split apart, each becomes a small, cheap, focused call. Language detection needs the message and nothing else. Intent classification needs the message and maybe a line of history. Only the final reply needs the persona, the tone, and the product knowledge. Decomposition is the master pattern — the other four are things you can only do once the flow is broken into steps.
The most common form of token waste is pasting an entire knowledge base into the prompt on the theory that the model "might need it." It almost never needs all of it. If a customer asks about your refund window, the model needs your refund policy, not your shipping zones, your warranty terms, and your company history. Retrieval — fetching the two or three relevant chunks and injecting only those — turns a 6,000-token knowledge dump into a 400-token, on-topic context. It is also more accurate, because a focused context is easier for the model to reason over than a haystack where the relevant needle is buried on line 340.
Some of your prompt genuinely never changes: the persona, the hard rules, the format instructions. Modern model APIs let you cache that static prefix so it is not reprocessed from scratch every time. Cached input tokens bill at roughly a tenth of the normal input price, and they are faster to process. The catch is that caching is a prefix match — put the stable content first and the volatile content (the user message, the timestamp, the per-request data) last, or you invalidate the cache on every request and pay full price anyway. This one architectural habit, keeping the frozen parts frozen and up front, can cut the cost of a repetitive prompt dramatically for the price of thinking about ordering.
This is not a thought experiment for us. Our own production AI sales assistant — the one that talks to visitors on this site — is built on exactly these principles, and it is a clean illustration of the payoff. A single incoming message runs through four steps, not one. First, two cheap classifiers run in parallel on a small fast model: one detects the language, the other classifies the intent into one of a fixed set of categories. These calls are tiny — a handful of tokens in, a word or two out — so they are almost free and they finish in the time of a single call because they run at the same time. Second, a context summary step runs on the same cheap model, but only conditionally: it fires only when the conversation is long enough to actually need summarizing. Short conversations skip it entirely and pay nothing.
Only the final step — generating the actual conversational reply — uses the expensive, more capable model. And it does not receive a giant everything-prompt; it receives the detected language, the classified intent, a compact context, and the message. The most expensive resource in the system is spent exactly once, on the one job that genuinely needs frontier-level fluency. Then a fifth step decides the call-to-action — book a call, open WhatsApp, show a contact form — with no model at all, just plain deterministic code. The result is a system where the overwhelming majority of the token spend goes to the cheap model, the frontier model is invoked once with a lean context, and the static parts are cached. Same user experience, a fraction of the cost and latency of the naive one-prompt version.
The lesson generalizes well beyond a sales chatbot. Whenever we build custom AI agents for a business, the architecture starts from the same question: which steps are cheap classification and routing, and which one step actually needs the expensive reasoning? Getting that split right is most of what separates an AI product with healthy margins from one whose bill grows faster than its revenue.
If you already have a monster prompt in production, you do not need a rewrite. You need a sequence of small, safe cuts, each of which lowers cost without changing behavior. Do them in this order and measure after each.
Two of those steps deserve their own discussion, and we have written them up separately: which model to use for which step is the subject of our piece on model routing, and knowing which steps should not touch a model at all is covered in our piece on when not to use an LLM. Together with this one, they are the three habits that keep an AI product fast and affordable as it scales.
The teams that win on unit economics are not the ones who found a cheaper model. They are the ones who stopped paying frontier prices to answer questions a router could have handled — and reserved their most expensive tokens for the moments that actually earn them.
If you are building something on top of an LLM and the bill or the latency is starting to worry you, that is usually a sign the flow is one big call that wants to be several small ones. That is exactly the kind of architecture we design and build. You can book a free initial consultation at https://calendly.com/qolca-info/consultoria-inicial-gratuita, or message us on WhatsApp at https://wa.me/51991376769, and we will take an honest look at where your tokens are going and what a leaner pipeline would look like for your product.