Understanding the True Cost of AI Systems: Lessons from MIT Professional Education

Recently, I started the course “Applied Agentic AI for Organizational Transformation” with MIT Professional Education to learn more about Agentic AI and all the buzzwords surrounding AI these days. To my surprise, the first module is not about Agentic AI itself, but about the cost of AI Systems. My first reaction was, “Wait, what?! Why start with costs?!”

As it turns out, it was quite eye-opening to learn how complex the cost structure of Large Language Models (LLMs) truly is. There are numerous components you must consider to keep your budget under control. This article is inspired by the first assignment of this course, featuring a hypothetical AI assistant called DIYgpt. I hope this helps shed light on what you need to consider for your own applications.

The Use Case: DIYgpt

To evaluate costs, I envisioned DIYgpt, an AI assistant designed to guide DIY enthusiasts through home projects like carpentry and repairs. Users describe a task, and the AI generates a step-by-step tutorial with a parts list.

To get more realistic data, I tested prompts across different models (like ChatGPT and Google Gemini) to find the average token counts for inputs and outputs.

The Data Points

The prompts below were utilized for testing and establishing the average token consumption. It should be noted that a real-world use case would typically involve a greater number of prompts, more iterations, and a wider variety of models.

1) “I want to build a simple wooden bookshelf for my home office. It needs to be 4 feet tall and 3 feet wide with three shelves. What materials do I need to buy at the hardware store, and what are the step-by-step assembly instructions?”

2) “My kitchen faucet is leaking from the base of the handle whenever I turn the water on. I have a standard hex wrench set and a screwdriver. Can you walk me through how to diagnose and fix this?”

3) “I just bought my first power drill. I see a bunch of numbers on a dial near the tip (1 through 20). What those numbers mean, and which one should I use if I’m just driving a screw into a piece of pine?”

The result is shown below.

Model	Question		Prompt (tokens)		Response (tokens)
ChatGPT	#1		52		834
			52		828
			52		711
Google Gemini Fast			52		620
			52		643
			52		704
ChatGPT	#2		41		630
			41		541
			41		516
Google Gemini Fast			41		622
			41		675
			41		628
ChatGPT	#3		50		402
			50		288
			50		272
Google Gemini Fast			50		381
			50		465
			50		434
Average			47		566

Therefore, the following average token counts was used in this execise:

• Input: 47 tokens (average of all inputs).
• Output: 566 tokens (average of all outputs).

Projecting the “Burn”: Daily and Monthly Costs

To estimate usage of the DIYgpt, the following assumptions were made:

• Daily Active Users (DAU): 5,000
• Sessions per User: 2
• Queries per Session: 3

Total: 5,000 users * 2 sessions * 3 queries = 30,000 API calls per day

Additionally, an average of 10% of inputs were assumed to be cached.

1. Daily Token Volume

Total volume of data being processed by the API:

• Total Input Tokens: 30,000 calls * 47 tokens = 1,410,000 tokens/day
• Standard Input Tokens (90%): 1,269,000 tokens
• Cached Input Tokens (10%): 141,000 tokens
• Total Output Tokens: 30,000 calls * 566 tokens = 16,980,000 tokens/day

2. Daily Cost Breakdown

Applying the GPT-4.1 pricing model, as an example:

• Standard Input Cost: 1.269M tokens * $3.00 = $3.81
• Cached Input Cost: 0.141M tokens* 0.75 = $0.11
• Output Cost: 16.98M tokens * $12.00 = $203.76
• Total Daily Cost: $207.68

3. Final Monthly Totals

Assuming a standard 30-day billing cycle:

Metric		Calculation		Result
Daily Burn		Sum of all token costs		$207.68
Monthly Burn		$207.68 * 30 days		$6,230.40
Cost Per Query		$207.68 / 30,000 calls		$0.0069

4. Cost of the models:

Following the same line of reasoning for the remaining models, I compared several models. The difference in “daily burn” is staggering:

Model		Daily Cost		Monthly Cost (30 Days)		Cost Per Query
o4-mini		$276.90		$8,306.91		$0.0092
GPT-4.1		$207.67		$6,230.18		$0.0069
Gemini 3 Pro Preview		$206.33		$6,189.79		$0.0069
GPT-4.1 mini		$55.38		$1,661.38		$0.0018
Gemini 3 Flash Preview		$51.58		$1,547.45		$0.0017
GPT-4.1 nano		$13.84		$415.35		$0.0005

The table and chart above illustrate the monthly “burn”. As you can see:

• o4-mini sits at the top, representing the premium price for reasoning capabilities (approx. $8,306 per month). This is sort of expected, as o4-mini is a reasoning model (the “o” series) which usually charges a premium for its “hidden” reasoning tokens - you are paying for the quality of the thought process, not just the final words.
• GPT-4.1 nano occupies the lowest bracket, demonstrating how architectural choices can reduce costs by over 95% compared to the most expensive model.The mid-range models (GPT-4.1 and Gemini 3 Pro) are nearly identical in cost, making their selection a matter of latency and accuracy rather than budget.
• The mid-range models (GPT-4.1 and Gemini 3 Pro) are nearly identical in cost, making their selection a matter of latency and accuracy rather than budget.

The staggering difference in cost between these models is a great segue to what I want to discuss next.

Beyond the Numbers: A Comprehensive Decision

While cost is a primary driver, choosing a model based solely on the “monthly burn” can be a mistake. To make a truly informed decision for an organization, you must consider additional factors, such as:

1. Accuracy and Domain Expertise

When evaluating AI systems, accuracy is paramount; a model with unacceptably low accuracy is essentially useless, regardless of its low cost. Therefore, you must define a strict minimum accuracy threshold and discard any model that fails to meet it. On the other hand, many applications do not require a costly, super-powerful reasoning model. The challenge lies in balancing performance, tailored to your specific domain, with cost. This is an ever-evolving field - recent developments include “LLM-as-a-judge” concepts, but you can start with a simpler approach: use a “golden set” of questions and employ keyword detection in the responses to gauge if they align with your desired outcome.

2. Latency

In a “DIY” scenario, users don’t want to wait 1 minute for a response while holding a power drill. Higher-performing models are often slower, whereas “Nano” or “Flash” models are optimized for speed, but may sacrifice some accuracy. Therefore, when selecting a model, you must prioritize response time for the end-user while balancing overall cost, efficiency, and performance.

3. Context Length and Memory

As users engage in longer sessions, the “memory” of the conversation grows. Models have different limits on how much information they can “remember” (context length). If your project requires analyzing long manuals or maintaining a multi-day conversation, you may need a model with a larger context window, which often comes at a higher price point, as well.

4. Caching Efficiency

Modern APIs allow you to save money by “caching” frequent inputs. In our DIYgpt exercise, I assumed 10% of inputs were cached, which significantly reduced the cost for repetitive instructions. The actual caching percentage is highly dependent on the domain and can fluctuate. Continuous monitoring is essential to establish a more accurate, realistic number over time.

Conclusion

Starting with costs wasn’t just an academic exercise - it was a lesson in viability. Whether you are building a simple assistant or a complex agentic system, understanding your “cost per query” is the first step in ensuring your AI transformation is sustainable in the long run. The DIYgpt case study demonstrated that the cost of AI is not merely a background detail but a decisive factor in a project’s survival. As our financial analysis revealed, the choice of model can result in a “staggering” disparity in daily burn rates - ranging from as low as $13.84 for GPT−4.1 nano to $276.90 for o4-mini, daily cost.

However, the lowest price tag does not guarantee the best outcome. True optimization requires looking “beyond the numbers” to balance cost efficiency with user experience. In a real-world scenario - such as a user waiting for instructions while holding a heavy power drill - metrics like latency and accuracy become just as valuable as the raw cost per token.

Ultimately, this exercise proves that sustainable AI transformation is not just about code; it is about architecture and economics. By leveraging technical efficiencies like caching and selecting models that offer the right accuracy for the specific domain, developers can bridge the gap between a cool prototype and a scalable business solution. As we move toward more complex Agentic systems, where autonomous loops increase token consumption, this ability to forecast and manage the “burn” will be the defining characteristic of successful organizational transformation.

References:

The following price list is reported on the OpenAI pricing website as of the time of writing this document.

Model	Input ($/1M tokens)	Cached Input ($/1M tokens)	Output ($/1M tokens)
GPT-4.1	3.00	0.75	12.00
GPT-4.1 mini	0.80	0.20	3.20
GPT-4.1 nano	0.20	0.05	0.80
o4-mini	4.00	1.00	16.00

This is the pricing reported for Gemini models through Vertex AI platform. It was assumed that the query input context is less than 200K tokens.

Model	Price (/1M tokens) <= 200K input tokens	Price (/1M tokens) <= 200K cached input tokens	Output ($/1M tokens)
Gemini 3 Pro Preview	2	0.2	12
Gemini 3 Flash Preview	0.5	0.05	3