Scraper Failure Recovery: Retry the Right Stage, Preserve the Right Data

scraper failure recovery

Key Takeaways

  • Classify transport, access, parsing, semantic validation and publication failures before choosing a remedy.
  • Distinguish another source request from replaying stored evidence or reissuing a previously accepted batch.
  • Keep stable job, observation and release identifiers through retries so recovery does not multiply business events.
  • Make partial-source failures visible without stopping unrelated valid source deliveries unless the contract requires all-or-nothing publication.
scraper failure recovery

A scraper failure can mean a timed-out network request, changed page template, incomplete search, corrupted archive or warehouse load that never finished. These problems do not share one remedy. Blindly retrying everything can overload sources, duplicate downstream records and create a new observation that gets mistaken for an earlier missed one.

Scraper failure recovery starts by preserving the original business obligation: what was due, for which source and record, in which window, and what evidence was already collected. Recovery then resumes the smallest safe stage, while the receiving team sees the difference between an on-time result, late result and unresolved state.

First Decide What Can Actually Be Recovered

A request that failed before fetching a source might succeed on a later permitted attempt. A parser that extracted the wrong price may be repaired by reprocessing saved evidence. A failed warehouse load may need only the previously accepted release resent. And a ten-minute market state missed yesterday often cannot be reconstructed from today’s live page. Treating all four as “retry the scraper” discards those differences.

FailurePreferred first responseWhat must not happen
Transient timeoutBounded retry within original deadlineInfinite attempts or source overload
Explicit throttlingObserve source guidance and reduce admissionExtra parallelism to ignore limits
Changed HTML or JSONTest parser against retained captureRecollect everything before checking the parser
Wrong business meaningQuarantine and correct the interpretationPublish wrong product or currency as valid
Interrupted uploadReissue verified objects or manifestTreat partial folder as accepted release
Missing past observationReport late/unresolved statusBackdate a new fetch to the missed window

A useful run ledger contains scheduled job key, due time, source identity, attempt and error history, capture evidence, parser version, acceptance state and publication ID. It makes both operations and audit possible. Datamam’s web scraping monitoring guide addresses what to alert on before recovery begins.

Classify the Incident Before Spending Retry Budget

A timeout, access restriction, malformed content, parser regression, semantic validation error, and failed warehouse load belong to different recovery classes. They should not all trigger the same fetch-again loop. Transient connection failures may justify bounded retries with delay and a deadline. A source template change requires parser repair and tests. A validation failure may require replay of already captured evidence. A warehouse staging failure usually requires safe redelivery, not another request to the source.

The AWS Builders’ Library guidance on timeouts, retries and backoff explains why uncontrolled retries can amplify dependency problems. In a scraping pipeline, the budget must also respect source request limits, work priority, and the remaining usefulness of a late observation. If a price was required before a morning pricing decision, a fresh collection three hours later is not a recovered observation of the earlier market state.

A useful incident ledger contains the original job key, scheduled window, source and business entity, attempt history, capture evidence, parser version, last accepted state, error class, and terminal resolution. Without it, operations staff can count recovered requests while overlooking unresolved business obligations.

Retry, Reparse, Reacquire, and Replay Are Different Actions

A retry repeats a bounded failed operation. Reparse applies revised extraction logic to already retained source content. Reacquire obtains a new source representation at a later real observation time. Replay republishes an accepted earlier result to a consumer. Name these actions separately in operations runbooks; they change the meaning of timestamps and history in different ways.

Retries should consider the source’s backoff or Retry-After instruction, a finite attempt budget, the original business deadline and the continuing value of a late response. The AWS Builders’ Library discussion of timeouts and backoff explains why uncontrolled retries can amplify dependency problems. Correct handling of access restrictions may require stopping the route and reviewing permission rather than trying again.

Datamam’s 200-Website Portfolio: Contain the Broken Site

A documented Datamam engagement operated weekly extraction across approximately 200 websites. One full sweep covered 4.6 million records; four sweeps produced 18.4 million record observations. Each source had its own adapter contract and failure state, while the whole portfolio shared accepted publication rules.

When one website changed its template, it entered a separate recovery state. The 199 healthy sources could continue to an accepted release under the agreed partial-publication policy. Source completeness still showed the affected website as incomplete; it was not erased from the denominator. Immutable S3 release paths and a portfolio manifest let the client replay a missed source batch without reloading unrelated accepted records.

This is a meaningful recovery design choice, not a universal rule that partial publication is always preferable. A tightly coupled dataset might require an all-or-nothing release. The important requirement is that the release boundary and the consumer’s incomplete-data behavior are explicit.

A Recovery Runbook That Protects Healthy Sources

Failure classFirst responseWhat must not happen
Network timeoutBounded retry within source budgetDeclare the item absent
Access or policy changePause, assess permitted route, escalateBlindly increase request volume
Source template driftHold affected parser output, test fixturesPublish wrong fields as valid
Business validation failureRetain evidence, review rule, reparseConvert rejected values into accepted rows
Destination write failureReplay immutable accepted releaseReacquire source without necessity
Deadline expiredMark late or unresolvedBackdate a later result

Datamam’s weekly collection portfolio illustrates containment. A source redesign could trigger a fixture failure and place that site in recovery without converting the other source results into unresolved work. Its exception remained visible in the portfolio release, and recovered source batches could be replayed using stable identities. The key feature was operational independence, not a claim that a partially delivered dataset is always acceptable.

This does not mean every customer should accept partial delivery. The business contract may require all-or-nothing publication for a particular consumer. The point is that source-isolated recovery enables an informed decision: publish accepted partitions with visible exceptions, or hold the aggregate release for a documented reason. The consumer should never have to infer completeness from the presence of a folder.

Recovery Must Preserve the Difference Between Empty and Unknown

A legitimate no-result search can be complete and accepted. A timed-out request, blocked page, missing pagination cursor or broken parser cannot prove that no product, document or offer exists. Record valid_empty only when the required scope was actually checked. Use unresolved for outcomes whose business meaning remains unestablished.

Datamam’s HTML archive project received 25,000 target URLs and accepted 24,250 target pages, with 750 reason-coded exceptions. A challenge notice returned with HTTP 200 did not count as a valid archived target page. That project shows why response status and desired business outcome must remain distinct: the diagnostic response can be retained without fabricating a successful archive entry.

Make Replay Safe for the Receiving System

A durable publication ID should refer to a specific accepted release and object manifest. Reissuing the same release must converge on the same consumer records, even when a previous upload or warehouse job was interrupted. A corrected value needs a separate correction identity linked to the prior observation; it should not quietly overwrite evidence of what was originally accepted.

For large source fleets, recovery boundaries should follow the smallest legitimate partition: one source, category, market, scheduled window or affected transformation. Replaying a complete corpus after every source failure wastes capacity and can rewrite unchanged records. The web scraping reliability benchmark distinguishes completion by the original deadline from later recovery-cutoff completion. Keeping that difference visible prevents an impressive final completion percentage from hiding missed business windows.

Treat Backfill as a New Operation With Historical Obligations

Replaying retained evidence and collecting the source again later are different operations. A parser fix can often reinterpret captured HTML for the original observation time. A new retrieval can only establish what the source presents at the later retrieval time. If a historical snapshot was never captured and the source offers no archive, the earlier price may be permanently unknown. An honest unresolved state is more useful than a fabricated backfill.

Backfill acceptance needs its own deadline, population, and publication identity. A repaired record should point to the original job and evidence when available, and it should not collide with a new scheduled observation. Idempotency at the destination prevents the same released correction from multiplying rows. For a system with mixed freshness tiers, crawl frequency optimization should not be changed automatically because one source had a temporary outage.

Persistent access or source instability can also require a revised operating plan. The buyer and provider should determine whether an alternate source is acceptable, whether the required attributes still exist, and whether the original scope can be delivered under applicable constraints. A mature outsourced scraping operations contract defines the authority to make those decisions. When the root cause is scraper schema changes, fix parsing and acceptance logic rather than hiding the incident behind additional request attempts.

Incident closure should document the affected eligible population, first detection, error category, observed business impact, corrective change, recovered and remaining records, and the consumer notification. Test the fix against retained failures and at least one known-good source. This makes recovery measurable without confusing a successful retry with a complete service restoration.

Close Incidents With Evidence

A useful incident record names the affected expected population, missed periods, failure class, source conditions, accepted history, corrective action, recovered records, unresolved work and consumer impact. Preserve the adapter or parser version that introduced the problem and the test that demonstrates the fix. A release after recovery should state whether it contains new observations, late observations or corrections to old interpretations.

Recovery is complete when the receiving system can explain which records are safe to use and which remain uncertain. A green retried job alone is not sufficient; downstream accepted state must reconcile to the original obligation.

If recurring source failures or missed releases are disrupting your downstream data products, discuss recovery and delivery requirements with Datamam.

Frequently Asked Questions

What causes scraper failures most often?

Causes vary by source and workload: access restrictions, timeouts, changed templates, incomplete pagination, invalid business context, exhausted resources and destination failures. A project-specific failure taxonomy is more useful than one unsupported universal ranking.

When should a scraper retry a failed request?

When the failure appears transient, access remains permitted, the source budget allows another attempt and a result within the defined window is still useful. Apply bounded backoff and deadline rules.

What is the difference between retry and replay?

A retry repeats a failed operation. Replay republishes a previously accepted observation or release. Replay should not require a new source observation when the original evidence and data remain available.

Can a scraper recover yesterday’s missing price today?

It can collect today’s price, but usually cannot establish an unseen historical value unless the source exposes historical events or saved evidence covers the missed window. Do not backdate a new capture.

How do you prevent duplicate rows after recovery?

Use stable job keys, observation identities and publication IDs. Test repeat deliveries against the destination’s idempotent merge or deduplication behavior.

Should one failed source stop every other source?

Not by default. Isolate independent source failures when the consumer contract permits; use an all-or-nothing release only when relationships or business requirements make partial publication unsafe.