For years, the company ran its price checks through a tool an outside developer had built for them. The tool worked. It pulled listings from a rival's site and a big marketplace, matched them against the company's own catalog, and dropped the results into a spreadsheet. It tried to match the same product across sites automatically, but the matches were unreliable enough that someone still had to eyeball the results before the report went anywhere.
The bigger issue was that the developer had built it to be hard to touch. It shipped as a compiled program with no source code included, so any new rule, any new site to watch, meant a call to the vendor and a wait.
It was slow, too. A batch of 100 products took 82 minutes on average, because the tool worked through everything one item at a time instead of running things in parallel. And for the big marketplace listings, it hit pages directly instead of going through any official channel, which made it both slower and more likely to trip a block.
When the vendor raised its maintenance fee again for a change that should have taken an afternoon, the company started looking for someone who'd build them something they could actually own.
"We couldn't even see how the thing worked behind the scenes," said Melissa, E-commerce Operations Manager. "If we wanted anything changed, we called them and waited."
The tool the company had wasn't really broken. It just wasn't built for them to touch. The visible problem was speed. The real one was that the company didn't have the code behind a piece of software that ran a part of their business.
A crawler that acts human. The rival's site and the company's own catalog use completely different page structures, so the crawler needed separate logic for each one just to find a price. Search terms had to be broken into keywords and searched the way a person would, since the same item often carries a different name on every site. For the big marketplace, that also meant switching from scraping pages directly to using its official search interface, which was faster and far less likely to get flagged.
Matching products that don't share a name or a photo. A semantic text comparison model, tuned for the kind of short, cluttered product titles this industry uses, gets a first read on whether two listings are the same item. A second, blended check compares the product photos: a deep-learning visual match carries most of the weight, with a geometric feature check and a basic color comparison filling in the rest.
Designing around bot management tools. A few weeks in, right before the tool was supposed to go live, the rival's site and the wholesale catalog it pulled from both started blocking the crawler outright. Rian rebuilt the crawling logic from the ground up: randomized timing between requests, rotating browser identities, a real browser engine instead of a bare script, and a rotating pool of proxies. Rian looked at using a general-purpose AI model to handle the human-like searching and decided against it, on the theory that hand-built logic would hold up better over time.
Full source code, this time. Unlike the old tool, everything shipped with the source included. Rian trained the team directly, recorded a walkthrough video, and put together a written guide before handing it over.
The company owns the tool now, not just a login to someone else's software. Every rule change or new site to track starts with the team's own staff, not a phone call to a vendor.
Price checks that used to take about 82 minutes now take 20 minutes. Rebuilding the crawler to run in parallel, tuned differently depending on how many products are in a batch, is what did it.
Matching the same item across listings is no longer a judgment call. The combined text-and-image check gets it right about 93 times out of 100, even when a product's name and photo look nothing alike from one site to the next.
Anything the matching model isn't confident about still gets flagged for a quick human look before it lands in the report, rather than being approved automatically. The company's next ask is to fold a third competitor's site into the same tool.
"I ran the first real batch and just watched the numbers fill in, row by row," Melissa said. "That's when I believed it."
A newsroom's international desk used to lose the first hour of every morning just figuring out what was new. Now a tool does that overnight, and flags which stories the desk is likely to run.
Ocean's Halo ran its packaging supply chain across spreadsheets owned by three companies, retyped by hand at every handoff. One dashboard now shows what needs attention, and the plants and the supplier enter their own numbers.
Every operation has documentation that stopped being true at some point. This is a method for finding that point: extracting entities from records, building a graph of what should be, and reading the divergences against what is.
