An online clothing retailer came to us with a problem they could see on their own site. Their recommendations were basic, and some of them were simply wrong. If you shop online you know what that does. You stop looking at the row.
Sharing lessons from building an AI-powered recommendation engine for a clothing retailer.
Two ways in
A shopper could type what they wanted, or say it out loud. Both ended up in the same place.
Typed requests went to a search agent. Spoken ones went to a voice agent. Either agent called RAG, which retrieved over the retailer's own data, where we had generated the embeddings. So the answer came from their catalog, and not from whatever a model happened to remember about clothes.
OpenAI models were the frontier models in the system. Gemini models handled classification and decision-making. All of it ran on GCP, and all the infrastructure was Terraform. When something breaks late at night you want to read what's running, not try to remember it.
What was wrong first
The initial version used a frontier model to make every decision.
It made sense on paper. The models are strong, and one model for everything is easy to reason about. But a lot of decisions in a flow like this are small and happen constantly. They don't need a model that can write an essay. Frontier models resulted in higher costs, and higher round trip latency, for decisions that didn't need them.
So we pulled them apart. Gemini models took over classification and decision-making. The frontier models stayed on the language work, which is what they're good at.
The rule I took from it: match the size of the model to the size of the decision.
The hard part: cold start
A new product has no clicks. A new shopper has no history. That's cold start, and it's where basic recommendations fall apart, because they have nothing to go on. Clothing makes it worse, because the catalog keeps turning over.
We didn't lean on one method. For cold start we used three together: vector search over the embeddings, collaborative filtering, and category-based search. Vector search can find products that are close in meaning even when nobody has interacted with them yet. Collaborative filtering brings in what similar shoppers did. Category-based search gives you something sensible to fall back on when there's very little to go on.
What changed
Click-through rate improved by 15 to 20 percent, and add-to-cart improved by 5 to 7 percent. We observed substantial improvement in latency as well. The response time for recommendations improved by 20 to 25 percent. Cost per recommendation was reduced by 60 to 70 percent.
What I'd tell another retailer
Match the model to the decision. Don't make a frontier model do work a classifier can do. Plan for new products and new shoppers on day one, not after launch. Let people type or talk, and send both through the same retrieval. And write your infrastructure down as code.
If your recommendations feel basic or wrong, that's usually a design problem before it's a model problem. A two-week Architecture Sprint is a good way to find out which.
Want help with your AI stack?
If this post matches problems you're seeing, we can map the fastest path from architecture decisions to production outcomes.
Book a call
Manmeet Singh
Founder & CEO, AIshar Labs · Ex-Apple, Ex-Instacart · 15 AI Patents Filed
Built ML systems at Apple (Search: Maps, Safari, Spotlight) and Instacart (Search, Recommendations, Ranking). Writes about production AI tradeoffs and system design.
Follow on LinkedIn →