Search backends break. Not dramatically — they just quietly stop returning results, because a provider changed their markup and the parser no longer matches.

What we learned the expensive way

A configuration that had been correct when it was measured decayed to a single working provider. Runs completed, found nothing, and reported no error — indistinguishable from "there are no results for this query". Silent failure is far worse than a loud one.

The rule now

  • Query several backends concurrently, not one after another.
  • Take the union of results, not the first non-empty answer.
  • Keep the set small enough that a dead backend is not costing a worker slot on every query.
  • Treat the list as a measurement with a date on it, and re-run that measurement rather than trusting it.

Why the union is worth it

Different engines index different corners. Merging four gives materially more unique domains than the best single engine, and it means one provider having a bad day cannot take the whole run down with it.