LinkedIn
Enterprises are on track to spend $10M+ annually on LLM inference alone. If you own AI budgets or are a part of the team implementing AI projects, you need a concrete framework with a set of methodologies and blueprints to assess & optimize AI usage costs. This 3-part series explains a proven architecture approach and provides practical techniques to optimize your dollar spend while maximizing the business outcomes.
Introduction
Enterprise Large Language Model (LLM) spending is rising rapidly because AI adoption is growing much faster than the rate at which LLM token prices are falling. Depending on the AI usage & workload characteristics, an enterprise with 20,000 employees can spend $10M+ annually on LLM inference alone, particularly when AI agents, copilots, and AI-powered workflows operate at scale. The good news is that through careful engineering, tuning, and architecture choices this cost can be brought down significantly.
This three-part series of articles is aimed at helping you, the enterprise customer, maximize your business outcomes per dollar spent on AI. The tenets & recommendations in this series are distilled from my Forward Deployed Engineer (FDE) team's first-hand experiences of building and tuning enterprise AI systems across multiple industries, with and beyond the US Frontier Lab models from Anthropic, Google, and OpenAI.
In this first part, I have outlined the “Top 10 Practical Techniques for ValueMaxxing your AI and LLM Usage”. I explain the shift from TokenMaxxing to ValueMaxxing and focus on practical AI architecture approaches & techniques to consistently reduce your LLM spending. In Parts II and III, we will look at cost estimation and other techniques for reducing LLM costs further while improving your business returns per dollar of AI spend.
The Shift from TokenMaxxing to ValueMaxxing
As LLMs have grown in capabilities, LLM usage has grown exponentially over the last 4-6 quarters. But along with usage, the cost of using AI has also grown substantially, often catching AI and IT executives unprepared for the large bills.
Earlier this year, in March and April of 2026, we saw some companies focusing on “TokenMaxxing”: More AI consumption implies higher productivity. Companies like Meta and Amazon reportedly had leaderboards celebrating people who burned most tokens. These TokenMaxxing metrics measured the wrong number and rewarded the wrong behavior. They have now predictably fallen out of favor.
“ValueMaxxing”: Extracting the highest business value from AI consumption. Now, a quarter later, LLM usage has skyrocketed and companies have shifted to measuring business outcomes per dollar spent on AI. In the Q4'26 earnings statement (July 2026), Microsoft CEO Satya Nadella stated, “We are advancing the frontier on the cost-to-outcome curve, ensuring every customer can turn tokens into business results.”
The question around LLM use has shifted from, “Can the LLM do this?”, to “Can the LLM do this at a much more cost-effective price point?”
LLM Tokens Are Costing Large Enterprises $10M+ A Year!
In our AI projects with AI leaders in large enterprises, I often see LLM token spend of over $1M per month. Consider this case of LLM usage at a large Fortune 500 enterprise customer with 20,000 employees, each having about 20 sessions per day with their AI assistants. Assuming 5 turns per session and 2000 tokens per turn, this enterprise is burning through 4 billion tokens a day!
In fact, it is not uncommon for a single developer to burn through tens of millions of tokens in a productive day.
If we take the blended cost of input & output tokens for a Frontier model at $10 /MTok, we arrive at the cost of LLM tokens of $40,000 /day, which is over $10M a year!
Beyond LLM inference, end-to-end platform costs also include pipelines, document parsing & extraction, MCP servers, embeddings, video transcription, underlying cloud infrastructure, operations, and observability etc. The spend numbers are big enough to warrant close scrutiny & optimization effort.
We are seeing a sudden interest in processes to measure usage, optimization of token spend, optimized architectures & blueprints, and use of FinOps principles to track ROI for AI.
Top 10 Practical Techniques for ValueMaxxing AI & LLM Usage
Based on my team's work in large enterprises across 7 vertical industries, I have identified the Top 10 most effective techniques for LLM cost optimization. These techniques can be structured as a framework with 3 logical categories:
Cut the Effective Token Costs - Pay less for tokens used by offloading, caching, reusing, or batching the information sent.
Shrink the LLM Calls - Reduce the information sent or received per request to the language model.
Match the Model to Tasks - Create architectural optionality and use the LLMs that can deliver the required results at the lowest total cost.
See the Three-Lever Framework in the diagram below, followed by details of how to deploy each of these methods.
Use server-side aggregate queries
Try to shift processing of large volumes of data away from your LLMs towards backend systems; e.g., compute the aggregates using SQL functions in databases or cypher queries in Graph DBs vs. computing and analyzing the data at the MCP Host (AI agent) end. Databases are tuned for query processing, so this approach is typically faster, better, and much cheaper.
Employ Prompt Caching
This is the highest impact and most often neglected method. To illustrate the impact - a query my team ran for classifying 600,000 tickets for a Telco customer eliminated 73% of the cost by caching the system prompt in Claude vs. sending it with every query. I examine & quantify this technique in depth in Part II of this series.
Use Batch Processing for bulk non-interactive calls
Many model providers will offer about 40-60% discount for asynchronous batches vs. real-time responses. Anthropic offers a flat 50% discount on Batch APIs - you get the same model, same quality, but at half the price!
Optimize Your Prompts for Formatted Outputs
- Review and rewrite prompts to consume fewer tokens.
- Text outputs tend to be verbose. Enforce terse and structured outputs for your agents.
- Ensure that you are removing stale tool outputs and inputs from context after tool use by your Agent harnesses.
In our experience, a well-designed prompt using the techniques mentioned here can save 30-40% tokens on average.
Manage Context Tightly
- Summarize and compact context regularly. This also yields better response quality in many interactive agents.
- Output tokens are usually more expensive than input tokens, often up to 5X. Carefully set the value of the maximum number of tokens that the LLM can generate in response through the max_tokens parameter.
- Selectively enable “thinking” mode and “thinking budgets” wherever possible. This sets a cap on how much analysis the model does. Use Interleaved Thinking only when necessary.
- Set the “effort” parameter to the lowest tier that meets your needs vs. defaulting to the largest effort. For example, “high” level may be sufficient (vs. using “max”).
- For LLM responses that are bounded and do not require thinking, tune max_tokens lower. When thinking is enabled, set max_tokens high enough to leave room for reasoning & final response.
Prune the LLM context & retries
Tune the level of context derived from Retrieval Augmented Generation (RAG), semantic layers, and other sources. For example, tune N for your “Top N” chunk choices for RAG context. Reduce duplicates, dedupe retries, and return error details when agents fail to avoid useless retry loops. Also cap the retries on tool failures to avoid runaway costs.
Explore use of Cost-efficient Open-weight Models
Some open-weight models such as Qwen 3.x, GLM 5.2, and Kimi K3, many of them from Chinese companies like Moonshot and Alibaba, are now very close in capability to US Frontier lab models. They can help optimize your budgets, but you need to consider security, hosting, and other aspects.
Choose, Distill, and Tier Models
Choose the lowest-cost model that can handle your task. Consider using a smaller fine-tuned model instead of large expensive LLMs if you have narrow high-volume tasks such as categorization of tickets. For example, use Claude Opus or GPT-5.6 Sol for your Supervisor Agent that creates the plan, and use Claude Haiku or GPT-5.6 Luna to do simpler tasks such as summarization.
Deploy Telemetry & Intelligent Model Routing
- Enable infrastructure to choose the optimal LLM family and the specific models based on the quality, modality, and response time requirements of your use case.
- Deploy AI Telemetry for cost measurement and optimization. Track & optimize calls regularly.
- Remember that bigger models can be cheaper! Boris Cherny, the lead creator of Claude Code, has stated that even though Claude Opus is bigger and slower than Sonnet, for autonomous, complex multi-step coding agents, Opus is almost always faster than Sonnet in the end due to Opus's better tool use and lower steering needs. Our teams have found that for complex agentic solutions, using Opus or Gemini Pro can mean lower total costs too.
Stack Your Savings to create compounding value
The architecture decisions compound; for example,
- Model tiering and routing requests to the appropriate model go together. For example, if you are on Google Cloud, route extraction and classification to Gemini Flash-Lite, but let the bigger, more expensive Gemini Pro model handle complex reasoning tasks.
- Claude's Message Batches API supports prompt caching (include identical cache_control block with every request in your batch), which gives approximately 90% lower cost on cached inputs and 50% discount on batched outputs on top. While actual cost savings vary with workload types and call frequencies, it is often realistic to achieve 75-85% savings for async workloads using batched calls & well-designed cached prompts!
I will cover the details of batch processing in Part II of this article series. I will also do deep dive coverage of open-weight models, model tiering, & intelligent routing in Part III of this series.
If your company is spending any sizable amounts on LLM API calls, you should carefully consider all these techniques for your architecture blueprints. Also ensure that you optimize for the outcomes at a holistic level, not just for LLM costs.
Consider your control plane, guardrails, and security. Products & services like Google Cloud Model Armor and AWS Bedrock Guardrails act as an inspection layer in the path of LLM requests & responses. They help prevent leakage of PII, block prompt injection, and prevent jailbreaking and exfiltration of confidential data. While the inline scans by the central AI Gateway increase the latency & also add some cost, this layer pays back in terms of avoiding reputational and compliance costs.
It is critical to balance the token economics optimization with broader quality, compliance, and security aspects of AI, and to run numbers for your specific use cases.
At Tredence, we have developed AI cost calculators and optimizers that can help you run experiments and get reliable estimates of the costs. Usually this results in the ability to support over twice the number of users with your current LLM budgets. Reach out to us for details on how you can access these tools and make the savings real.
Key Takeaways
As AI usage grows exponentially, companies have shifted from TokenMaxxing to ValueMaxxing. As the LLM token budgets trend towards $10M+ annually, large enterprises have shifted their attention to LLM pricing, AI architecture tuning, and optimization of cost for business outcomes.
In this article, we saw 10 real-world tried-and-tested methods for optimizing your LLM and AI agent costs and boosting the value you get out of AI investments. These 10 techniques, covering areas from server-side aggregation to model tiering & routing, should be a good checklist for every AI architecture & implementation team.
Organizations should ensure that the right architectural fundamentals & blueprints are in place before focusing on token price negotiations. In Part II, we will look at cost assessments and quantify the cost savings through real-world examples. Finally, in Part III, we will look at broader options of high-quality models outside Frontier Labs, along with FinOps and model routing architectures for ValueMaxxing your AI usage.
