A distributor rebuilt its product list by hand every morning because the same item was named differently on every supplier site. The fix was less about the crawler than about deciding when two names mean one product.
The client distributed several thousand products sourced from a shifting set of suppliers. Every morning someone opened two supplier websites, found the items they carried, copied prices and specifications into a master workbook, and guessed at the cases where a product appeared under a different name on each site.
It took about eighty minutes for a hundred products, and it was wrong often enough that the sales team had learned to check before quoting.
A scraper had been built two years earlier. It stopped working within a month of a supplier redesigning its site and was never repaired, because repairing it required someone who could read the markup and nobody in the business could.
That is the ordinary fate of automation built as a one-off. The interesting question was not how to rebuild it, but why rebuilding it would not have solved the problem anyway.
Getting the data off the page is the easy half. The hard half is deciding that a Korean product name and an English one refer to the same item, and that the third listing with the same weight and a different manufacturer does not.
This is entity resolution, and it is where catalogue projects quietly fail. A pipeline that extracts perfectly and matches badly produces a price list that is confidently wrong, which is worse than the manual version because nobody checks it.
Korean product names carry a lot of signal: brand, cut, weight, pack count, and a set of descriptors that are near synonyms in ordinary language and meaningfully different in a catalogue. Straight string distance handles the easy cases and then fails in a specific, predictable way. It scores two products with identical descriptors and different weights as near matches, and two spellings of the same brand as unrelated.
Normalising units and pack counts out of the name before comparing recovered most of that. It did not recover the cases where a supplier had simply used a different word for the same thing.
Product photography turns out to be a strong disambiguator in this category, because packaging is the thing the manufacturer controls and the retailer does not rename. Comparing product images gave an independent signal that was wrong in different places than the text signal was.
The ensemble beat either component. Neither alone reached a level where the output could go unchecked. Together, on a 100-product evaluation set, matching accuracy came in around 93 percent, with the residual concentrated in genuinely ambiguous cases a human also had to think about.
The run time for that same hundred products went from roughly eighty minutes to about twenty, and the twenty is mostly review rather than typing.
The client had a template. It had merged cells, conditional formats, embedded images, and formulas other parts of the business depended on. The instinct to replace it with something cleaner would have been a mistake: the workbook was not the problem, and rebuilding it would have broken three downstream processes nobody had mentioned.
So the pipeline writes into the existing template, preserving formats, formulas, merged regions and images. That constraint added real engineering time and it is the reason the tool was adopted in the first week rather than argued about for a quarter.
Blocking, rate limits, retries, session handling and the slow accumulation of per-supplier quirks account for more of this codebase than the matching does. That ratio is normal and it is worth saying out loud, because it is the part left out of the estimate and it decides whether the thing still runs in six months.
The accuracy figure is measured on a hundred-product run drawn from two suppliers. It is a useful number for sizing the work and it is not a guarantee for a different catalogue. The honest framing is that the ensemble moved the task from retyping everything to confirming the fifteen the system is unsure about, and that is the outcome that mattered to the person doing it every morning.
Every operation has documentation that stopped being true at some point. This is a method for finding that point: extracting entities from records, building a graph of what should be, and reading the divergences against what is.
The pilot worked. A year later nothing is in production. The failure is almost never model quality. It is data nobody owns, permissions nobody granted, and a workflow nobody changed.
A seaweed supply operation running from Korean farms and factories to U.S. warehouses kept its truth in four disconnected spreadsheets. Discovery found 25 places the sources disagreed and 34 questions nobody had answered.