Contains：我为家庭烘焙者做了一个过敏原检查工具，它宁可说「我不确定」也不瞎猜

This is a submission for the Hacktoberfest Open-Source AI Challenge: Side Hustle.

Here is a cake recipe. Tell me whether it is safe for someone with a soy allergy.

all purpose flour, butter, eggs, milk, dark chocolate, cooking oil

Take your time. The word "soy" does not appear anywhere in it.

Now the real answer: I don't know, and neither do you. Dark chocolate usually contains soy lecithin, and sometimes doesn't. "Cooking oil" in Pakistan is often soybean oil, and sometimes isn't. The honest answer to that recipe is not yes and not no. It's go and look at the packet.

The off-the-shelf model I started with answers questions like that with a confident yes or a confident no. Across 85 hidden allergens, it never once said it wasn’t sure. That’s the thing I set out to fix.

Five confirmed, one to check. The amber card is the whole project: it names the three ingredients that caused the doubt — chocolate, cooking oil, pistachio spread — instead of just hedging.

What I Built

Contains reads a recipe or an ingredient list and produces a US-style Contains: line for the nine major allergens: milk, eggs, fish, crustacean shellfish, tree nuts, peanuts, wheat, soybeans, sesame.

It has three answers per allergen, not two:

That third state is the product. A home baker selling cakes doesn't need a tool that guesses. She needs to know which three ingredients she has to go and check, and which twelve she can stop worrying about.

Every verdict shows its evidence: the exact words that triggered it, and which layer fired. Nothing is a black box the user has to trust.

Who would pay for it: home bakers and small food businesses who sell to customers and get asked "does this have nuts in it?" People with legal exposure and no food-science department. I found one and asked her. She's at the bottom of this post, and her answer was not entirely flattering.

🔗 Live: https://contains.onrender.com
💻 Code: https://github.com/Abeera81/contains

TL;DR

• Fine-tuned Qwen3.5-4B with a LoRA on Tinker, on 5,755 rows built from Open Food Facts.
• The fine-tune lost. On held-out compound recipes it caught 66% against the base model's 78%.
• It did learn something the base model cannot do at all: it abstains, and it produces zero false alarms.
• Shipped a two-layer system, fine-tuned model plus an ingredient-rules layer which catches 86% with zero false alarms on a frozen test set scored exactly once.
• Gemma 3 4B reads photos of labels locally. Deployed free on Render. Whole fine-tune cost about $2.50.
• Every number below has the command that produced it in docs/RESULTS.md.

Demo

It's live, so try it yourself — that's the demo. Start with something nastier than a cake:

tomato, mozzarella, basil, pesto, balsamic glaze, pine nuts

Pesto is the interesting one. It is cheese plus nuts plus oil, and it says none of those things.

One honest note about the demo, and the bug it exposed. It's on Render's free tier, so the instance sleeps. Loading a cold one takes about 75 seconds end to end: Render wakes the container, then the Tinker client takes ~32 s to build. Once it's awake, a new input takes 2–3 s and a repeat comes back in 0.57 s from cache.

I found this the worst possible way, by opening my own live site and seeing the degraded-mode banner. My timeout was 25 s, the warm-up took 34 s, so the very first request after any quiet spell silently fell back to rules-only. The one request most likely to be a judge's, or a customer's, was the one guaranteed to get the worst version of the product.

The fix wasn't a faster timeout, it was the opposite: on a cold start the app now waits, tells you it's waking the model, and returns the real answer. Degrading quietly is worse than being slow out loud. Fixing it also surfaced a second bug. The label endpoint was blocking the event loop during that wait, which would have frozen the health check under any concurrency.

The fallback still exists, but it's now reserved for what it was designed for: an actual Tinker failure.

Photo reading is local-only. Gemma 3 4B needs more RAM than a free instance has, so the deployed app detects that and says so rather than throwing an error at you. The README has the local setup; it's one ollama pull.

The finding that redirected the whole project

I built the baselines before I built anything else, and they nearly killed the idea.

An off-the-shelf Qwen3.5-4B, zero-shot, scored 0.98 recall on ordinary held-out ingredient lists. There was no problem to solve there. If a label says "milk, wheat flour, soy lecithin," a 4B model reads it correctly and I have invented nothing.

So I built a second test set: 62 recipes where the allergen is hidden inside a compound ingredient and never named. Dukkah. Panko. XO sauce. Imitation crab. Béchamel. I hand-wrote and hand-checked every row, froze the file, recorded its hash, and did not look at it again until the very end.

The base model and my system are indistinguishable on ordinary lists. Everything I built only matters on the bottom row.

That reframed the project. The job was never "read labels." It was "know what labels don't say."

How I Built It

The data, and the four bugs in it

Everything starts with Open Food Facts — a crowd-sourced database of several million food products, ODbL licensed. I sampled across 4,703 row groups of their parquet dump to avoid pulling one contiguous slice of similar products.

Then I spent most of a day finding out my data was wrong in four separate ways, each of which would have quietly inflated my results:

1. HTML in the ingredient text. 79% of the rows I pulled had markup embedded in them. A model reading <b>milk</b> is not reading a label.
2. The answer was printed in the question. Many products carry their own Contains: milk, soy declaration inside the ingredient field. A model scoring well on those has learned to copy, not to reason. I flagged every row with a declaration, report results both with and without them, and stripped declarations from the training inputs entirely.
3. My declaration detector was too greedy. When I tightened it, it removed 98 false positives and found 271 real ones I'd been missing.
4. 833 non-English rows had slipped through the language filter. Dropping them meant re-running every baseline I'd already computed, including one of my hand-reviewed gold rows.

None of this is glamorous and all of it is the job. An evaluation you haven't tried to break is a number you made up.

I also hand-reviewed 100 rows myself, one at a time, in a little CLI, to have a gold set I actually trusted. Splits are brand-disjoint, so the model can't memorise "Brand X always contains milk" and score well on Brand X's other products.

Two rules layers, because one would have been cheating

I needed a keyword baseline to compare against. I also needed an ingredient-knowledge layer in the shipped product. My first version used the same dictionary for both — which meant my baseline was written by the same person who wrote the test traps, and was quietly rigged in my favour.

I split it in two:

• rules_naive — the honest baseline. Direct allergen names only. Never touched again after it was written.
• rules_product — the shipped layer, which knows that béchamel implies milk and panko implies wheat.

They are different files with different jobs, and only one of them is ever reported as a baseline.

The fine-tune, and what it actually learned

Tinker gives you a LoRA training API over open-weight models without managing a single GPU. Base model: Qwen/Qwen3.5-4B.

The training mix was 5,755 rows:

The contrast rows exist so the model learns "unnamed → unsure, named → decide" rather than hedging on entire food categories. Without them it would simply learn that chocolate is scary.

The learning-rate probe told me something I didn't want to hear:

Every learning rate made dev loss worse. Partly that's the dev set having no unsure targets while the training mix does, so the model starts hedging and gets penalised for it. But it was also an early warning I chose to read as noise.

Final run: rank 16, LR 3e-5, 1 epoch, 359 steps. Probe plus training plus every evaluation came to about $2.50.

Then it lost

On held-out compound recipes, the fine-tuned model caught fewer hidden allergens than the model I'd started with.

Here's why, and it took me embarrassingly long to see it. My training labels were derived from the ingredient text itself. Every example was effectively: here is a list with the word "milk" in it; the answer is milk. The model was already excellent at that.

What I needed it to learn — that dukkah means sesame, that panko means wheat — appears nowhere in a label, because no product label has ever said "dukkah (contains sesame)." The knowledge I wanted to teach was precisely the knowledge my dataset didn't contain.

It's like teaching someone to recognise faces using photographs that all have name tags in them. They'll get very good at reading name tags.

And here's the proof that the pipeline itself was fine: the one slice that did contain new information — the abstention rows — is the one thing it successfully learned. The base model never says "unsure." Not once in 85 opportunities. Mine does.

So I shipped both layers

The merge rule is safety-first and deliberately boring:

• Contains if the rules fired, or the model said yes.
• Needs checking if the model said unsure and the text contains an unnamed-source term it can point at.
• Absent otherwise.
• A rules hit always beats a model "no."

That last line matters more than it looks. On my first user's three cake recipes, the model never says wheat for "all purpose flour", not once in three recipes. The rules layer carries the safety-critical call, and the UI shows you which layer fired so you can see that happening.

# unsure must be able to point at the ingredient that caused it
if verdict == "unsure" and not trigger_terms_in(text, allergen):
    verdict = "absent"          # counted, logged, never silent

I'll come back to that four-line rule, because it cost me something.

Results

One test set. 62 frozen recipes, 85 hidden allergens, hash verified before scoring, scored once.

A confident "no" tells a baker the allergen is absent. That's the error that sends someone to hospital.

Read that table as a story rather than a scoreboard:

• The keyword layer alone catches barely half.
• The fine-tune alone loses to the model it was fine-tuned from.
• Together they beat every component, and they're the only configuration with no false alarms.

The result wasn't the model. It was the system.

Where it still fails, by name

The 11 confident misses are wheat ×6, milk ×2, egg ×2, fish ×1:

• Bare "flour" (3 rows). rules_product deliberately drops it, because "almond flour" and "chickpea flour" are not wheat, and the model doesn't rescue it.
• Imitation crab — wheat binder, invisible.
• Béchamel — and here is my favourite bug of the whole build: the rule matches bechamel, the trap row says béchamel, and an accent defeated my allergen checker. I found this after the test set was frozen, so it is recorded as a finding and left unpatched. Fixing it would have meant editing the system in response to its own test scores, which makes the score meaningless.
• Caesar dressing with croutons — missed both the wheat and the anchovy.
• Pesto ×2 for milk, tempura batter and frangipane for egg.

The four-line rule I'd now reverse

Remember the downgrade rule above? After scoring, I checked what it cost. Five of the 11 confident misses were cases where the fine-tune had correctly said "unsure" and my rule overrode it into a confident "no" — pesto, tempura batter, frangipane.

I traded warning noise for the exact error class the whole project exists to prevent. With hindsight the trade looks wrong.

I did not change it. The traps were already scored; editing the merge logic afterwards would be tuning on the test set. It ships as measured, and it's the first thing in future work.

Two more disclosures, because a number you can't audit isn't a number

• While testing on my first user's recipes I added named flours (all-purpose, maida, atta) and plural matching to rules_product — before the frozen traps were scored. I checked the overlap: none of the wheat terms appear in any trap row, and the plural fix changes exactly one verdict (trap-021, pine nuts, from missed to caught). With that row excluded the shipped system scores 71/83 — the same ratio.
• The abstention behaviour is not reliable. It's deterministic for a given string — I sampled one input 8 times and got 8 identical outputs — but reordering the same ingredients flipped soy and peanuts from "unsure" to a confident "no". I can't claim it fires dependably, so I don't.

Gemma on the label itself

Typing out an ingredient list is the worst part of using a tool like this. So the local build takes a photo instead: Gemma 3 4B runs through Ollama on your own machine, reads the label, and the extracted text goes through the same pipeline.

On a real packet off my kitchen shelf it read

Milk Solids, Milk Fat, Sugar, Stabilizers (INS 4521, 33911), Emulsifier (INS 407), and Antioxidants (INS 320, 321).

word for word, and the pipeline returned Contains: Milk — with both layers agreeing, and the evidence chip naming the term it matched. On an earlier generated test label it transcribed wheat flour, whey powder, soy lecithin exactly and returned Contains: Milk, Wheat, Soybeans. It takes about 52 seconds on CPU including model load, which is slow and completely acceptable for a thing you do once per product.

The OCR text is editable before you re-run it, because OCR is wrong sometimes and the user is the one holding the packet.

Photograph in, label out. Local, no GPU, nothing leaving the machine — and the evidence chip shows the rules layer and the model reaching the same answer independently.

Code

make setup      # uv venv + deps (Windows: ./make.ps1 setup)
make test       # unit tests
make smoke      # M0 go/no-go smoke tests (needs TINKER_API_KEY + Ollama)

src/contains/
  labels.py      # the 9 allergens, OFF tag mapping, the gluten→wheat rule
  rules.py       # rules_naive (baseline) and rules_product (shipped), kept apart
  pipeline.py    # the merge rule and the Contains line
  sampler.py     # tinker_lora | tinker_base | ollama | rules_only
  ocr.py         # Gemma 3 4B via Ollama
  app.py         # FastAPI: /api/label, /api/photo, /healthz
scripts/         # every data, training and scoring script, one job each
docs/RESULTS.md  # every number, with the command that produced it

SAMPLER is an environment variable, so you can swap the fine-tune for the base model, for Ollama, or for rules-only without touching code. rules_only makes zero API calls, which is how the deployed app stays up if Tinker is unreachable — it falls back, shows a banner saying the model layer is unavailable, and still returns a result. The API never returns an error page.

39 unit tests, plus a headless-Chrome check that actually clicks the interactive elements rather than trusting that they work.

Why Does Open Innovation Matter?

Three reasons, all of them things I ran into this week.

Because the thing I needed to inspect was the thing I needed to fix. My fine-tune failed, and the only reason I know why is that I could read my own training data, my own labels, and my own merge rule. If the model had been an opaque endpoint, "it didn't work" would have been the end of the investigation instead of the middle.

Because the data was open, and so were its flaws. Open Food Facts is crowd-sourced, which means it has HTML in it and inconsistent allergen tags and products whose declarations contradict their own ingredient lists. I could see all of that. A closed dataset with the same problems would have handed me the same inflated numbers and no way to catch them.

Because the people this helps can't pay enterprise prices. My first user runs a cake business from her home in Pakistan and told me, plainly, that she'd pay two or three dollars a month. A 4B open model fine-tuned for $2.50 and hosted on a free tier can serve her. Nothing in the paid-API world can, at that price, and the version that runs entirely on her own laptop costs nothing at all.

My First User

I gave it to a home baker who sells cakes, with three of her own recipes — a chocolate cake, Dubai chocolate cinnamon rolls, and a mango loaf.

(She reviewed the output herself and then wrote her reactions up with an LLM's help, so what follows are her points rather than direct quotes.)

What I expected to learn: whether the tool was right.
What I actually learned: what I was competing with.

Her current method for checking allergens is to Google it, or ask ChatGPT. Not reading labels. Not a reference book. A chatbot that answers confidently, never says "I'm not sure," and can't show you which word in the recipe it keyed on. That's the baseline. That reframed my own project for me more sharply than any benchmark did.

The findings:

• Nothing in the confirmed allergen lists was wrong across her three recipes.
• The chocolate/soy relationship was genuinely new to her, as was checking what's inside pistachio spread and banaspati.
• On the uncertainty flags: useful, but only when targeted. A warning that fired on every chocolate regardless of product would become noise she'd learn to ignore. That independently confirms the over-firing I'd measured — my model flags soybeans as "needs checking" on all three of her cakes.
• Weakest result she found: the peanut flag on unspecified cooking oil. She's right. It's the flimsiest inference the system makes.
• Would she use it? Yes. Would she pay? Not yet — about $2–3 a month, if it became reliable enough to trust while preparing products for customers, and if it remembered her brands.

That last answer is the most useful sentence anyone said to me this week. It's not "this is amazing." It's this is not yet worth money, and here is the specific thing that would make it worth money.

What I'd do next

1. Reverse the downgrade rule. An "unsure" with no named trigger should stay unsure, not become a confident no. I have the measurement; I just can't apply it without invalidating it.
2. Build the compound dataset properly. The fix for the fine-tune is training data that contains knowledge the label text doesn't — an external compound-ingredient reference. With one hard condition: the compound foods in training must be a disjoint set from the ones in the test, or it measures memorisation rather than generalisation.
3. Remember brands. My first user said she'd pay for a tool that knows her chocolate is that chocolate. The "needs checking" state collapses to a definite answer the moment you know the product.
4. Fix the accent bug. After the test set is retired.

Limitations, plainly

• The data is US- and Europe-heavy. A tool for a baker in Pakistan trained largely on Western products is a real mismatch, and banaspati is in the rules because I put it there, not because the data taught it.
• Fish and crustacean classes are thin. 799 positives total, and fish is one of the confident misses.
• The gold set is 100 rows, hand-reviewed by one person — me.
• The "needs checking" state is brand-dependent by nature. It can't be resolved without knowing which product you bought, which is exactly why it says so instead of guessing.
• Abstention is brittle to rewording. Documented above.
• This is not a substitute for a manufacturer's label, and the app says so on every screen.

Prize Categories

Tinker (Thinking Machines). The fine-tune is the centre of the project, not a garnish. A LoRA on Qwen/Qwen3.5-4B: a learning-rate probe across four rates, a 5,755-row mix with purpose-built abstention and contrast slices, 359 steps at rank 16, and a held-out evaluation against three separate baselines with a frozen test set scored exactly once — for about $2.50. It produced a negative result, reported in full, with a diagnosis of why the training data couldn't have taught what I wanted. Tinker made that diagnosis affordable: the cost of learning that my approach was wrong was two and a half dollars, which is the real argument for an API like this.

Gemma (Google). Gemma 3 4B runs locally through Ollama as the vision layer, reading photographs of physical labels and feeding the extracted text into the same pipeline. It was also my local zero-shot baseline. Entirely on-device, no GPU, no data leaving the machine.

Render. Deployed from a render.yaml blueprint on the free tier at https://contains.onrender.com — live, public, with health checks, dashboard-managed secrets, and a graceful degradation path that keeps the site answering even when the model layer is unreachable. Free hosting is the only reason a tool for $2-a-month users can exist at all.

I started out trying to build a model that knows what's hiding in your chocolate.

What I shipped is a model that knows when to tell you to go and look.

Built solo over the challenge window. Every evaluation number in this post has its command in docs/RESULTS.md, including the ones that didn't go my way. Ingredient data from Open Food Facts, ODbL.