Published 2026-04-20

When Complex Retrieval Isn't Worth It: A Recommendation Chatbot Over a Video Archive

I built embeddings, BM25, and reranking — then shipped the whole catalog in the prompt. What the measurements said, and where the approach stops working.

I built a recommendation system over an archive of YouTube videos. The user doesn’t have to guess the right keyword — they describe their problem in their own words and the chatbot picks videos that make sense for it.

It’s not a free-roaming advisory chatbot, and that was a constraint from the start. The system isn’t meant to give advice beyond the sources it knows. Its job is to find the best content in the given catalog, explain the choice, and say plainly when it has nothing suitable.

What’s more interesting than the result, though, is the path to it. I built a full retrieval pipeline, then tried a much dumber solution — and in the end the measurements told me something different than I expected.

First, the videos needed to be put in order

The chatbot is only the last layer. First I had to turn the video archive into data: a list of videos, metadata from YouTube, transcripts, structured summaries, and a compact catalog for selecting recommendations. Everything is held together by video_id, so a video, transcript, summary, and later a link to a specific timestamp can all be safely connected.

Transcripts were produced locally with the large-v3 model on a graphics card. I didn’t need a transcript accurate down to the last word — I needed to reliably recognize the topic, the problem, the body part, and the recommendation. For this task, the quality of the local transcript was sufficient, and I didn’t need to reach for the external Whisper API.

For each video I then produced a structured summary: topics, typical user problems, body parts, recommendations, and a flag for whether the video is even suitable to recommend. The video title often isn’t enough — a video can have a generic title and address a very specific situation, or conversely look relevant and not be suitable for the given query.

First I built the complex retrieval

The first version of the search was textbook: embeddings over summaries and transcript segments in Weaviate, plus BM25, merging both result sets via reciprocal rank fusion, and finally reranking the candidates with a model.

It worked. On a gold set of forty queries, this path hit the correct behavior in every case and returned the correct video among the top three in 96.7% of evaluated queries.

Then I tried the dumb solution

The catalog has around 290 videos. It occurred to me that maybe the whole thing could fit in the prompt.

Transcripts aren’t sent — that wouldn’t make sense even at this size. Each video is one line:

[video_id] Title || problems: ... || body parts: ... || topics: ...

The whole catalog in this form comes out to about 23,000 tokens. The model gets the catalog, the current question, and a short conversation history, and picks exclusively from the fixed list of video_ids that literally appear in the catalog. Besides the selection, it has to determine the match quality — strong, partial, none, or needs a follow-up question — and explain in Czech why it chose those particular videos.

The measurement said something different than I expected

Here’s the point of the whole project, and it isn’t “the simpler solution won.”

On the same gold set, the full catalog came out worse. It hit the correct behavior in 93% of cases, but the correct video among the top three in only 64%. Against the semantic pipeline’s 96.7%, that looked like a clear loss.

But when I read through the four failed cases one by one, none of them was a bad recommendation. For a query about back pain while sitting, the model returned videos about sitting and back pain. For a query about the neck, it returned videos about the trapezius. These were relevant videos — I just hadn’t included them in my hand-written list of correct answers.

The archive contains many closely related videos on any given topic. A hand-picked gold set therefore systematically undervalues any approach that picks a different correct video than the one I happened to think of. The weak point wasn’t the system — it was my measurement tool.

That was a more important conclusion than the score itself. I stopped tuning against a gold set that couldn’t decide, and turned on logging of real queries, so the next version of the gold set would come from production, not from my own imagination.

Why I shipped the simpler one after all

What mattered was that behavior accuracy was high, the failures were explained, and the path is significantly simpler to maintain: no vector database to sync, no weights to tune between BM25 and embeddings, no separate reranker.

I didn’t delete the semantic pipeline. It stayed in the code behind a flag, as a fallback — so reverting is a change of an environment variable, not a rewrite.

Where it stops working

This is the question everyone asks about “the whole catalog in the prompt”: what happens once there are a thousand videos?

I know the answer, because I had to calculate it.

Catalog size is measured at application startup — for Czech, it works out to roughly 3.5 characters per token. Above fifty thousand tokens, a warning drops into the log. Around sixty thousand, the reliability of selection from a long list starts to fall in the middle of the context — the well-known “lost in the middle” effect. Above that threshold, an embedding prefilter kicks in, keeping only a few dozen of the nearest candidates and never sending the rest to the prompt at all. That path is built and switched off, because at 290 videos it isn’t needed.

A more practical limit showed up earlier, and from a different direction. The provider account has a cap of 30,000 tokens per minute, so two concurrent calls at 23,000 tokens each exceed it, and the second one fails with a 429. During an evaluation run that fired fifteen queries in two minutes, this looked like a disaster: response times of 48 to 67 seconds and one hard crash.

But that wasn’t a picture of real-world traffic. A single unloaded call takes 4.2 seconds. With dozens of users a day spread out over time, that’s fine. So I handled the 429 with a clear message instead of an error, and made a note that once concurrent queries become common, the fix is raising the account limit or turning on the prefilter.

What I take away from this

The most useful thing wasn’t the choice between the simple and the complex solution. It was the moment the measurement showed a worse number, and it was worth finding out why instead of discarding a working approach on the spot.

A gold-set score is a tool, not the truth. When it disagrees with what you see in individual answers, it’s just as likely that the gold set is wrong as that the system is.

This chatbot is one component of a larger AI layer I run for the same client — I describe it in the article How I Turned an Email Agent Into an AI Layer for an Entire Online Business.