Published 2026-05-20 · Updated 2026-07-23

An Email AI Agent Isn't RAG Over Past Emails: What I Learned Building an Email Agent, From Cost Optimization Through Different Approaches to the Best Solution

What I learned building an email AI agent about model calls, RAG, prompt caching, evaluations, dead ends, and reliable operation.

Building an email AI agent that writes a convincing reply from a few prepared examples isn’t especially hard. What’s much harder is getting such a system into a state where it reliably handles real emails, conversation histories of varying length, changing information, and situations that don’t fit a pre-prepared script.

The final result therefore didn’t come from a single prompt or from uploading documents into a vector database. It required repeated evaluations against actual emails, manually reading drafts, tracing individual steps, adjusting the knowledge base, checks in the code, and reverting changes that looked clever on paper but made the replies worse in practice.

The hardest part wasn’t getting the model to write a decent email. The hardest part was figuring out:

• when it shouldn’t write at all;
• where the current truth should come from;
• which checks another model should do and which plain code should do;
• how to tell that an “improvement” actually made the result worse;
• and what to do when two correct-looking parts of the system combine into a bad reply.

I didn’t want autonomous support

The agent works over a real customer email inbox. It receives product questions, access problems, requests for lecture recordings, responses to campaigns, automated invoices, and messages containing health-related or personal context.

In such an environment it’s not enough for a reply to sound convincing. A convincingly written falsehood is precisely one of the worst possible outputs.

From the start I therefore set a simple boundary: the agent never sends an email automatically. It creates a draft in Gmail, sends a notification via Telegram, and leaves the final check and sending to a human.

This fundamentally shaped the whole architecture. I wasn’t building a replacement for a human, but a system meant to prepare the best possible first version while also clearly signaling when it’s missing information.

One email can mean seven model calls

The entire processing is assembled as a graph in LangGraph. Simplified, it looks like this:

Gmail push notification
→ triage and filtering out automated messages
→ intent classification
→ generating search queries
→ searching the knowledge base and historical emails
→ reranking the retrieved material
→ writing the reply
→ checks and possible correction
→ draft in Gmail
→ Telegram notification
→ human review and sending

A typical full pass can therefore contain roughly seven to eight model calls. But not all of them do the same thing.

The main writer gets the strongest model and is the only one allowed to freely formulate a reply. Smaller and cheaper models handle narrow tasks: they return respond or ignore, assign the query type, generate a search query, select relevant material, or flag that a reply doesn’t address the original question.

Over time I found that this division matters more than the number of calls itself. A useful “micromodel” has a small, predictable output. A dangerous checking model gets the option to rewrite the entire email, fill in missing facts, and become a second writer.

Intercom now publicly describes a similar specialization for its Fin agent: a separate model for retrieving candidates, a reranker, context summarization, human-handoff detection, and understanding the customer’s response. Intercom: Fin AI Agent explained

The difference is in the goal and the scale. Fin is meant to resolve a large share of conversations on its own. My agent prepares a draft that a human always sees. Copying the entire architecture of an autonomous system would therefore mean paying for additional checking layers whose role is already filled in my case by human review.

A historical email isn’t a source of truth

The agent uses two separate collections in the RAG database.

The first contains selected historical emails. These help with style, salutation, length, and the way similar situations were explained before.

The second contains a curated knowledge base: products, access, links, rules, processes, and other facts.

This split arose from a practical problem. A historical reply can sound exactly as it should while also containing an old price, an invalid link, or a process that has since changed. Semantic similarity is not the same as authority.

Historical emails therefore enter the writer only as a stylistic precedent. They have no right to override the current knowledge base.

The opposite boundary is just as important: the state of a specific payment, order, or user account doesn’t belong in RAG at all. If the agent doesn’t have access to a live system, it must not infer this state from text. It has to use the connected API, or mark the reply for human review.

A RAG audit revealed a bug in text chunking

A later audit of the live collection revealed an interesting problem. By that point the database matched what was on disk. The bug was in the algorithm that split text into chunks.

Out of 81 checked chunks:

• 41 started in the middle of a sentence or word;
• 53 ended in the middle of a sentence;
• 48 mixed at least three different headings and topics;
• 9 contained a URL physically cut by a chunk boundary.

So the model sometimes didn’t receive a poorly retrieved link. It received the correct document, but in it saw only https://mar… or the second half of a URL in the next chunk.

No sophisticated reasoning reliably fixes this problem. The model can’t use characters that never made it into its context at all.

The new ingestion design therefore uses headings as semantic boundaries, keeps a single section on a single topic, and includes a hard check that every URL was preserved character by character. The new version is ingested into a separate collection so the old one can be restored with a single switch.

Intercom now recommends the same thing: use H1–H3 as semantic boundaries and write sections so they make sense even when pulled out of the original article. Intercom: Optimizing content for Fin

For me this is one of the main lessons of the whole project: production RAG is not an “upload documents” feature. It’s a data system with versioning, source identity, updates, chunk validation, retrieval tests, and rollback.

The prompt gradually grew into its own programming language

At one stage the writer prompt was roughly 459 lines, 23 sections, and around a dozen blocks marked as a hard prohibition.

It contained the author’s voice, the reply structure, facts, URL rules, sales logic, health-related restrictions, the signature, length by email type, and a list of things the model must never promise.

Each individual instruction made sense. But together they didn’t.

The model sometimes followed a new rule but stopped respecting an older one.

The problem was the number of instructions: a rule exists in the prompt, but among all the other rules it no longer carries enough weight.

The natural idea was to split replies into fixed templates by query type. But the experiment ran into the real nature of the problem. The most common complaint about the drafts wasn’t that they had the wrong structure, but that they felt too generic. Fixed templates only amplified that trait.

So I didn’t use templates.

The newer direction is more restrained: a short shared core with the voice and safety rules, one block for the specific email type, and a few conditional blocks, for example for health warnings or a consultation. Assembling the prompt is deterministic.

Prompt caching isn’t just a switch

The long writer prompt was also the most expensive part of the whole run. The writer’s total input ranged, depending on the email, around 16–20 thousand tokens.

At first glance, turning on prompt caching seemed obvious. For supported OpenAI models via OpenRouter it works automatically. But automatic caching doesn’t mean an automatic hit.

The original system prompt contained dynamic values scattered inside static instructions: retrieved facts, links, similar emails, query type, and other data were inserted through a series of text substitutions. This made the start of the prompt change between two emails, and the cache didn’t have a long enough identical prefix.

The solution wasn’t a new library. I moved the static rules into an unchanging system prompt and all the content of the specific email into a following user message.

One recorded run then showed:

• first writer call: 17,041 input tokens, of which 11,520 loaded from cache;
• repeated writer call: 17,085 tokens, of which 16,640 from cache.

That’s roughly 68% and 97% cached input. It’s not the same as a 68% or 97% saving on the total price — that depends on the specific model and provider. But it’s clear proof that a stable prefix works.

In the current version, the model calls needed to produce a single draft reply cost on average roughly $0.009, i.e. less than one cent. An email filtered out already in triage costs only about $0.0001. The calculation doesn’t include running the server and the vector database.

OpenRouter recommends the same: place unchanging content at the start, variable context after it, and verify actual hits via prompt_tokens_details.cached_tokens. OpenRouter: Prompt Caching

So the biggest lesson wasn’t “I turned on caching” but “I had to change the prompt architecture so that something stable even existed that could be cached.”

A graveyard of clever improvements

The most valuable part of the project’s history is the list of things that sounded right, made it through implementation, and were ultimately turned off or reverted.

These tests aren’t proof that planning or LLM critics never work. They were small and relate to a specific system.

But they showed that in an agent with mandatory human review, their benefit didn’t outweigh the cost and the new failure modes. The checking model wasn’t just a safeguard. It was another model capable of ruining an already correct reply.

Narrow model calls yes, a second author no

The result isn’t a system without model checks. The difference is in their authority.

A smaller model is useful when it’s meant to return a narrow result:

• whether the email should be answered;
• what the main intent is;
• which three pieces of material are relevant;
• whether the draft even answers the question;
• whether the case should be handed off to a human.

Much riskier is the question: “Rewrite this reply so it’s more correct.” The model can then add something that wasn’t in the input, remove an important part, or change a correct link.

I therefore gradually moved firm guarantees into the code:

• automated and infrastructure senders are ignored by a fixed set of rules;
• a closed conversation isn’t reopened;
• disallowed URLs are removed or flagged for review;
• claims about a performed or future action are checked separately;
• Czech diacritics, Slovak leakage, and the signature have their own validation;
• situations requiring action in another system get needs_review.

The model may propose. The code must guarantee. The human decides.

The most serious bug had nothing to do with the prompt

In March, after a restart, the processing of a Gmail notification would occasionally be lost or duplicated. The startup inbox check and a new webhook could process the same thread at the same time and create two drafts.

The cause wasn’t the AI. The Gmail history_id was stored only in process memory, concurrent jobs accessed it without a lock, and there was no final idempotent check before creating a draft.

The fix therefore looked like ordinary distributed systems:

• a persistently stored monotonic cursor;
• an asynchronous lock when it changes;
• claiming a specific thread during processing;
• a label check right before creating a draft;
• Gmail API pagination;
• safe handling of delayed notifications.

An agent can understand an email perfectly and still be an unreliable product if it creates the same draft twice. Idempotence is as important for an email agent as the prompt.

Measure first, then fix

Another problem looked like the model truncating replies. Some drafts ended in the middle of a sentence or without the required signature. The first hypothesis was too low an output token limit.

I therefore added logging of finish_reason, prompt sizes, and reply length to the writer, and prepared a small diagnostic dataset.

In a sample of 24 real pairs, all 24 generations ended with finish_reason=stop. Not a single case confirmed termination due to a length limit. But twenty-two drafts failed the strict signature check: in eleven cases due to spacing, in another eleven due to missing diacritics or another text variant.

So the original hypothesis wasn’t confirmed in this sample. A problem that looked like a model limit was largely a validation and formatting problem.

Similarly, a supposed context limit of 13.7 thousand tokens turned out in the end to be a limit on a specific account and credit at the provider OpenRouter.

Both situations taught me the same thing: before an architectural fix I need an observable signal that confirms the real cause.

An eval score isn’t reality

For testing I assembled a dataset of real queries paired with historical replies. The main set contained 57 pairs, smaller twenty-item subsets served for faster iteration, and other separate cases as a holdout set.

Each run stores not only the final draft, but also:

• the outputs of individual nodes;
• the retrieved RAG chunks and their scores;
• the selected links and facts;
• the prompt and model versions;
• the active feature flags;
• consumption and cached tokens;
• the reason for a retry or handoff for review.

An LLM judge helps quickly find suspicious cases, but doesn’t have the final word. An aggregate score can look good even when a reply omits a decisive link, promises an action that wasn’t performed, or elegantly repeats outdated information.

When debugging I therefore work backwards from the bad sentence:

1. Was the correct information retrieved?
2. Did the correct reply type receive it?
3. Did the writer ignore it?
4. Did the checking layer rewrite it or let it through?

I try to assign each failure a single primary cause: retrieval, routing, writer, or checking. Only then do I change the code.

The feature flags that remain turned off today I therefore don’t see merely as clutter. They’re a record of experiments and let me compare, or instantly revert, every change.

The architecture that ultimately remained

After all the experiments, the system settled on a fairly restrained division of responsibility:

• facts belong in the curated knowledge RAG database;
• historical emails serve mainly as an example of style;
• narrow model steps classify, search, and flag;
• one main model writes;
• hard rules and certain checks belong in the code;
• an unknown state is flagged for review, not invented;
• the final decision stays with the human.

The visible part of the product is the email text. But most of the work lies beneath it: knowledge updates, chunk integrity, webhook idempotence, token measurement, regression tests, rollbacks, and precisely delimiting the authority of each step.

That’s why, in my view, it isn’t enough to pay for a service, upload a few pages into RAG, and say we have an email agent. That produces a good demo. A reliable assistant emerges only through repeated measurement on your own data and a willingness to remove even features that look architecturally clever.

The most important lesson of this project, in the end, isn’t how to add another AI node to the graph.

It’s the ability to prove that it shouldn’t be there.