I let local AI models help track food without letting them write the numbers
Open Food Facts gives away about 4.4 million food products as one 4.4 GB file. I wanted a tracker that searched all of it as I typed, on my own machine, with no cloud service. I also wanted a local model to save me the typing: describe a meal in chat, or photograph a pantry shelf. The hard part was not getting a model to answer. It was arranging things so its answers could be wrong without the log or the inventory being wrong.
- The 4.4 GB product file is reduced once to an indexed table of the 21 columns the app reads, so a lookup never touches the file again.
- Search ranks product names above brands and brands above categories, because an early version answered "chicken breast" with a fajita kit.
- The coach model proposes a food and a portion; the nutrients always come from the database, and nothing is logged until I confirm a card.
- A shelf photo is read one tile per model call, then duplicates are settled by written rules, and only the pairs the rules cannot settle go back to the model.
- Every scanner change can be scored against hand-labelled photos for precision, recall and F1, using the same duplicate rule the live scan uses.
01Why I built it
Food trackers tend to fail in one of two ways. Some limit you to a curated list, so the thing you actually ate is missing. Others let anyone add products, and the database drifts into duplicates and wrong numbers. Open Food Facts is the open alternative: a volunteer database of millions of products with ingredients, nutrients, Nutri-Score grades and processing levels, published as a single file.
That file is the first problem. It holds 4,380,085 products in 111 columns, product names in many languages, and nutrients packed into nested lists. Reading it directly takes seconds per question, which is far too slow for a search box.
The second problem is effort. Tracking is tedious, and most of the tedium is typing. A local AI model could turn "two eggs and toast" into entries, or list what is on a shelf from a photo. But a model is a poor source of numbers. It guesses calories, invents portions and names one jar three ways. I wanted the convenience without trusting it with the record.
02Two days from spec to tracker
The project started as a written design spec, reviewed and revised the same afternoon, then a 19-task implementation plan. The tracker itself came together over 16 and 17 March 2026 in 34 commits. 60 of the 66 commits in the repository name an AI coding assistant as co-author; the spec and the plan were what kept that work pointed in one direction.
The scanner was a second burst, on 25 May, in 27 commits. Since then the only commits have been dependency upgrades, each checked by comparing the app's output before and after.
- 16 March. Design spec, review fixes and the 19-task plan; then the SQLite layer, the DuckDB product engine, GPU sums, the first search index, the API and a single-page front end.
- 16 to 17 March. Logging made fast, a chat coach, recipes and favourites, and six commits spent getting the pages to scroll.
- 17 March. The coach learns to log food, and the server learns to cope when the model formats its answer differently from what it was asked.
- 25 May. Server-rendered pages replace the single-page app; the shelf scanner, golden labels, benchmark runner, batch queue, scan comparison and optional segmentation models arrive.
- October and November. Dependency upgrades, including newer Python, web framework and machine-learning libraries, each verified by comparing outputs.
03Architecture at a glance
It is one Python web application, built with FastAPI, that serves its pages from the server and updates parts of them in place with HTMX. Product data and personal data live in two separate databases, because they behave in opposite ways: one is large and never changes, the other is small and changes on every tap.
Local AI models are reached through a local server that speaks the OpenAI chat format, so any compatible model can be loaded without changing the app.
The split that matters most is between deciding and proposing. Databases and written rules decide; models propose. Every part above sits on one side of that line, and most of the design follows from keeping it there.
04Making 4.4 million products feel instant
DuckDB can query the Parquet file directly, and the first version did. A single product lookup scanned the file and took seconds. The fix was to stop asking the file anything after the first start.
On first start the app builds a table in a persistent DuckDB file with only the 21 columns it reads. The same barcode can appear more than once in Open Food Facts, so the build keeps one row per barcode, choosing the most complete record, and puts a unique index on the code. Every later start opens that table from disk. Product and nutrient lookups are cached in memory on top of it.
Two rules keep it fast, and both are written into the project's own notes because they are easy to break. The full file is never read on a request path. And DuckDB calls run on the default executor rather than a thread pool of their own, because a DuckDB connection is not safe to share across threads.
Nutrient totals for a portion are a scaling of per-100 g values. The app can do that sum on the graphics card, but only above 500 nutrient values; below that, starting the GPU costs more than it saves, and the CPU does it.
05Search that ranks the right chicken
The first search index was a pure Python BM25 index, pickled to disk. The second ranked in DuckDB SQL with tiers: an exact name match scored highest, then a name that starts with the query, then a name that contains it, then brand, then category. That tiering came from a real failure: "chicken breast" returned a fajita kit, because a category match counted as much as a name match.
The current index is Tantivy, a search library written in Rust, built from the English-named products. Product names are stemmed, so "apples" finds "apple". Results get a small boost for popularity on a log scale, enough to lift a common product over an obscure one without letting popularity drown relevance. The index records how many products it holds and a schema version, and rebuilds itself when either changes.
If the index cannot be built, search falls back to a plain SQL match in which every word must appear. It is slower and blunter, but the app still answers.
Search results are then enriched with calories, macros and grades in one batched query against the lookup table, never one query per result. A barcode, read live by the camera or decoded from a photo, skips search and goes straight to the product.
06A log that keeps every nutrient
The obvious food log has a calories column. This one stores a row per nutrient for every entry, with its value and unit. It costs more rows, but any nutrient the product data carries can be totalled later without a schema change, and nothing is collapsed into a single figure at the moment of logging.
Saved meals and recipes are built from individual items rather than copied entries, so a recipe can be logged as a meal and each ingredient still carries its own product and portion. A recipe whose ingredient has no barcode looks it up by name and remembers the code it found.
The schema has changed many times, so it moves through numbered migrations. Each version is a list of statements applied in order at startup and recorded in a migrations table, which means a database from March and a database from May both arrive at the same shape. The committed code is at version 24, and the database holds 35 tables.
There is also an estimate of daily energy use built from 28 days of logged intake and weight change. It is a small calculation, but it refuses to answer outside a plausible range rather than report a number that cannot be right.
07A model that proposes, a database that decides
The coach is a chat assistant with a context built for it: targets, recent entries, the pantry inventory and a trimmed history, sized to the model's token budget. When it wants something done, it writes a small JSON action, such as logging a food, updating a target or adding to a shopping list.
Models are inconsistent about how they write JSON, so parsing is defensive. Reasoning text is stripped first, fenced blocks are taken if present, and otherwise the text is scanned for balanced braces. Only objects that name an action are kept, duplicates are dropped, and an action that wraps several foods in a list is expanded into one entry per food.
Every number in an action is checked against a range before it is used. A height of 500 cm is simply discarded, and an impossible portion falls back to 100 g.
Food is where the line is drawn hardest. The model supplies a food name and a portion. The app searches the product database, skips candidates that carry no calorie data, and computes nutrients from the product's own per-100 g values. The result becomes a pending action and a card showing the matched product, portion and meal, which I confirm, adjust, swap for another product or reject. When a product has no serving size, the model may suggest one, but only between 5 and 1,000 g, and the card says it is an estimate.
08Reading a shelf, one tile at a time
The scanner turns a photo of a fridge, freezer, pantry or counter into inventory items, each with a name, a category and a numbered pin on the image. It runs in the background while the page polls for progress, and every finished scan is saved as a preview, so nothing is lost if I never commit it.
Dim or washed-out photos get an automatic contrast stretch first. The photo is then cut into a grid that depends on its size: 2 by 2 from 700 pixels on the long side, 3 by 3 from 1,400 and 4 by 4 from 2,400. Each tile is read in its own model call, three at a time, with the tile's dimensions in the prompt so the boxes come back in the right coordinates. One more call reads the whole photo, to catch items that span tile borders. On the largest grid that is 17 calls before anything else runs.
Model boxes are loose, so each pin is moved toward the strongest edges near it, using a plain image edge filter. A verification pass then asks the model, in up to three framings, what it missed: general items, containers, and items at the back. An optional close-up pass crops each item and asks the model to read its label.
There is also an optional front end to all of this. Four segmentation backends, FastSAM, YOLO, YOLO-World and GroundingDINO with SAM, can find objects first, so the model only has to name each crop. They sit behind one registry, load only when chosen, and fall back to the plain pipeline when their libraries are not installed.
09Deciding when two names are one thing
Reading tiles separately means every item near a border is seen twice, often under two names: "Great Value Liquid Honey" in one tile and "honey" in the next. Merging too little leaves duplicates. Merging too much deletes real products, which is worse, because nobody notices something that is not there.
Names are compared by written rules. Each name is reduced to a set of words after removing known brand prefixes, sizes such as 750 ml, container words such as jar or bottle, and plural endings. Identical sets merge. If the words that differ include a distinguishing one, such as skim, almond, strawberry, juice or cheddar, the two stay apart no matter how similar the rest is. Otherwise a strong overlap merges them.
The rules cannot settle everything, so the pairs in a middle band of similarity, at most 25 per scan, go to the model as a judge with a yes or no question. Boxes are compared too: overlapping boxes, and boxes whose centres sit close together with similar sizes, merge only when the names also agree.
Two more checks catch the model's habits. A tight cluster of three or more tiny boxes is cropped and sent back with the question of which are real, because that pattern is usually one item cut into pieces. And each location has a scene: an item that does not belong there, such as yogurt in a pantry, is re-read from a crop, then judged, and if it still does not fit it is flagged for me to review rather than silently kept.
10Measuring the scanner instead of admiring it
A scanner like this improves by changes that each look sensible, and some of them make it worse. Without a measurement there is no way to tell which.
Any saved scan can be marked as golden and given a hand-corrected list of what is really in the photo. The benchmark runner re-scans every golden photo with the current code and scores it for precision, recall and F1. Matching uses the same duplicate rule the live scan uses, so "liquid honey" counts as finding "honey", and longer, more specific names claim a match first. Each run records the commit it measured and is compared with the run before, so a regression shows as a drop rather than a feeling.
Every pass in the pipeline is also measured as it runs: what it added, removed, renamed or recategorised, how far it moved the boxes, how long it took and how many model calls it made. Those combine into a change score. Once a pass has at least 20 samples, it is flagged when its median change is under 5 per cent, its 90th percentile under 15 per cent and its median time over 5 seconds: it costs time and rarely changes the answer.
Two scans of the same photo, before and after a change, can also be laid side by side, showing what was found, lost and renamed.
11Where it stands
The tracker, the coach and the scanner all work on one machine with local models. Photos are stored once by content hash as WebP files with a thumbnail and camera metadata stripped, and full-size copies are kept only for the three latest scans of each location. A batch queue can scan many photos in turn, survive a restart and carry on past a photo that fails.
There is no unit test suite. The benchmark is the check for the scanner, and the dependency upgrades were checked by comparing routes, pages and model outputs before and after. The rest of the app has no automated checks, which is the largest gap.
The structure shows its speed. The page routes and the scan service are each over 6,000 lines in the committed code, and both are due to be split. The single-page front end from March is still in the repository, kept only for its barcode scanner.
What I would build next follows from the lessons below: a duplicate rule that keeps both items when unsure, tests for the rules that decide, and smaller modules around them.
12Ten lessons
Most of these came from the commit history, where each fix explains what went wrong. Several are about models, and several are about the ordinary web plumbing around them, which cost just as much time.
01A category match is not a name match
Early search scored a word found in the category as highly as a word found in the product name. Searching "chicken breast" returned a fajita kit, because its category mentioned chicken.
Now: Name matches outrank brand matches, which outrank category matches, and the current index weights the fields the same way.
where a word appears matters as much as whether it appears.
02Saving an entry read a 4 GB file
Logging a food took several seconds. The save looked up the product again, and the lookup scanned the Parquet file. The first fix sent the nutrients the page had already calculated, which brought a save under 200 milliseconds; the lasting fix was the indexed lookup table.
Now: The source file is read once, at the first start, and never on a request.
a slow click is often a large file being read for a small answer.
03Six commits to make a page scroll
The single-page front end fought its own layout for two days: a bottom bar covering the save button, a fixed-height view that cut content off, a flexbox chain that needed exactly the right minimum heights, and finally inline styles on every element. Six commits over those two days mention scrolling.
Now: The page body scrolls naturally and the navigation bar is fixed at the bottom, with no chain of nested scroll containers.
when a layout needs a fix per screen, the layout is the bug.
04Models answer in the shape they prefer
The coach was asked to write one JSON block per food. It often wrote one block with a list of items, or bare JSON with no fence, and only the first food was logged. A prompt change to force separate blocks did not hold.
Now: The server accepts every shape that has appeared, expands lists into separate entries and drops duplicates.
parse what the model actually does, not what the prompt asked for.
05Many images in one call means some go unseen
Sending every tile of a photo in a single message looked efficient. The model's attention spread across the images, and it routinely skipped whole tiles: the bottom of the shelf simply never appeared in the list.
Now: Each tile is its own call, with one extra call for the whole photo, and the duplicate rules collapse what two tiles both saw.
a model given less to look at looks harder.
06A server that answers one request at a time
The label-reading pass first sent one call per item, all at once. The local model server answers one request at a time, the queue outgrew the 120-second timeout, and 9 of 13 calls failed without an error message.
Now: Those calls were made serial and now run at most two at a time; tile calls run at most three at a time.
concurrency against a single worker is just a longer queue with a timeout at the end.
07Thinking used the whole budget
Models that reason before answering could spend every token thinking and return an empty answer. The scan then reported no items, which looked like an empty shelf.
Now: The client probes the loaded model's context window and reasoning support before each scan, sizes the budget to match, and a truncated answer is reported as truncated, with what to change.
an empty answer and a cut-off answer must never look the same.
08A dictionary key called items
A progress page crashed because the template asked for queue.items, and Jinja resolved that to the dictionary's own items method rather than the key. Switching to bracket syntax did not help either.
Now: The key is called photos.
a name that collides with a built-in is a bug waiting for a template.
09Attributes that travel down the page
Clicking a photo thumbnail inside the inventory asked "Delete selected items?". The confirmation sat on the surrounding form, HTMX passes such attributes down to every element inside it, and so every request in the form inherited the question. A related fix moved a pop-up to the browser's native dialog, because a parent element's styles had been pushing the old one off screen, and then added a small self-installing script, because pages loaded in fragments never re-fetched the updated shell.
Now: Confirmations sit on the button that needs them, the form blocks inheritance, and pop-ups use the native dialog.
in a page assembled from fragments, check what each piece inherits and what each piece assumes is already loaded.
10A fallback that hid a broken feature
After a library upgrade, the label-plausibility scorer raised an exception on every call, because the library now returned an object instead of a tensor. The scorer's safety fallback returned a neutral 1.0, so every label looked plausible and nothing visibly broke.
Now: Upgrades are checked by comparing outputs before and after, and the scorer reads the field the new version returns.
a fallback that turns every error into a neutral answer needs a check that the answers still vary.
13What I would do differently
The tracker came together fast and the scanner measures itself, but a few choices would go the other way next time.
- Write tests for the rules that decide. The duplicate rule, the action parser and the range checks are the parts the whole design trusts, and none of them has a test. The benchmark covers the scanner end to end, not those rules one by one.
- Make doubt keep both. A merge that deletes a real product is invisible. The duplicate rule should have treated unknown words as meaningful from the start.
- Start with server-rendered pages. Two days went into making a single-page app scroll, and the app moved to server-rendered pages three months later anyway.
- Commit in smaller steps. The scanner and the new front end arrived as one commit of 28,220 added lines across 76 files, which hides the order in which things were learned and makes any one change hard to undo.
- Keep design notes next to the code they describe. The scan-flow document, committed alongside the code, describes one call for all tiles while the code makes one call per tile. The code is right; the document misleads.