Key Takeaways
- Start with the latest acceptable observation age at the receiving decision, not the shortest technically possible polling interval.
- Separate discovery, price or availability refresh, reference attributes and historical snapshots into different work classes.
- Tier records by change behavior, commercial impact, source limits and cost of a missed state.
- Use observed meaningful changes and accepted coverage to revisit schedules; frequent requests are not proof of better data.

“How often should we scrape this website?” sounds like a technical scheduling question. In practice, it is an economic and analytical decision: how quickly does relevant information change, how soon must a consumer act, and what does another observation cost or disrupt? Web scraping refresh frequency should be assigned to business records and source capabilities, not uniformly to every URL.
A high-frequency tier can be justified for live availability or a small set of priority products. A weekly cadence may be more than sufficient for stable product descriptions. Scraping everything every minute wastes capacity when the source has not published new information or the recipient cannot use it that quickly.
Three Clocks Influence Every Refresh Decision
The first clock is when the publisher changes the underlying information. The second is when the collector actually observes that change. The third is when the verified result reaches the consumer. A five-minute polling interval controls scheduled opportunities for observation, not the source’s publication delay or the time needed for parsing and delivery.
For a weekly merchandising report, an accepted observation from yesterday may be useful. For a sportsbook market during play, it may be too old to support an application. Express the requirement as a maximum usable age at the consumer or a publication deadline for an eligible population. The data freshness in web scraping guide discusses freshness measures; this article focuses on choosing the cadence and its workload consequences.
Calculate the True Workload of a Refresh Decision
The unit of scheduling is the required business observation, not a URL. A single competitor page might expose many seller offers; a product may have several accepted source mappings; a travel search may depend on departure date, passenger configuration, and country. Polling one page hourly does not mean every business observation has been refreshed hourly.
For a population of 50,000 accepted product-offer mappings, polling every hour creates 1.2 million scheduled observation jobs per day before retries, discovery, or reconciliation. Polling the same population every six hours creates 200,000 daily jobs. The interval choice changes the required operating capacity sixfold, but it does not automatically change the information value sixfold. Select frequencies using the cost of acting on a stale observation, source change behavior, consumer deadlines, and source access limits.
Separate event-driven monitoring from routine polling when a supported change feed exists. A documented source API or approved notification channel may reduce repeated retrieval. Where polling remains necessary, stagger jobs rather than dispatch every due record on the hour. Keep the next-due ledger independent from source attempts so failed jobs are visible instead of silently skipped.
Different Fields Deserve Different Schedules
Do not give every attribute the same refresh rate merely because it belongs to the same product page. Product identity and technical descriptions may change infrequently, while price and stock can change many times a day. Discovery must find new items; refresh must revisit known accepted ones. The two may follow entirely different queues and source budgets.
| Record class | Typical decision | Schedule design principle |
| Newly added products | Does the eligible assortment contain new records? | Periodic discovery plus source-supported update hints |
| Priority prices and offers | Can a pricing decision rely on the latest accepted quote? | Faster tier when observed changes justify it |
| Regular catalog prices | Has a routine comparison changed? | Daily or other tested interval |
| Stable descriptive attributes | Are identity and specifications still valid? | Slower checks with drift triggers |
| Critical live markets | Is a short-lived state still usable? | Time-window scheduling tied to active events |
These are design patterns, not promises about a particular website. Source restrictions, data rights, actual publication cadence and the cost of execution may impose a slower rate. A purportedly live feed that misses entire partitions is worse than a complete, clearly timestamped slower feed.
The arithmetic of a real tiered schedule
The production retail schedule makes tiering concrete. Priority records were responsible for a disproportionate share of daily work because their four-hour checks occurred several times as often as routine and weekly records. The practical lesson is to budget by scheduled observation jobs, not by the distinct product count in the catalog. A new interval affects queue admission, source pacing, storage, and consumer freshness at once.
Suppose a buyer expands its high-priority tier during a promotion. The planner should calculate additional scheduled observations, inspect source budgets, and test whether downstream validation can publish the larger cohort within the decision window. Otherwise faster theoretical polling can increase late or unresolved observations without making the delivered feed fresher.
Discovery of new products and repair of broken mappings remained distinct from routine refresh. A price cannot be meaningfully checked against the customer’s SKU until its source relationship has been accepted. Use web scraping change detection after establishing stable identities so the consumer receives meaningful differences rather than page-level noise.
Datamam’s Retail Tiers Show Why Uniform Polling Is Expensive
In Datamam’s retail price intelligence operation, 1.2 million client catalog SKUs were connected to 4.8 million accepted product-to-retailer mappings across 12 integrations. The team divided accepted relationships into three disjoint groups: 240,000 priority mappings scheduled every four hours, 1.2 million regular mappings daily, and 3.36 million stable mappings spread across a weekly cycle.
That design generated approximately 3.12 million scheduled observations each day, or 93.6 million over a 30-day month. The priority group represented only 5% of active mappings but accounted for approximately 46.2% of daily scheduled work. It was deliberately expensive in observation capacity because those records mattered more to the buyer’s decisions.
A uniform four-hour schedule across all 4.8 million mappings would require 28.8 million scheduled observations per day, versus 3.12 million under the tiered plan, over nine times the scheduled work, before considering retries or changes in source cost. That calculation illustrates the cost leverage of tiering; it does not prove a universal optimal interval for retail data.
Event Windows Need a Different Sampling Contract
The football odds case illustrates why refresh cadence can change with an event window. Event discovery, odds polling, and result collection needed separate schedules. The accepted market identity included period, line, outcome, and the source event reference. A new handicap line could not be compared to the old line as though the commercial condition were unchanged.
Even when a collector runs frequently, it does not know which source changes occurred between accepted polls. Network latency, queue delay, publication time, and source availability all affect freshness. The consumer should use the age of the latest accepted observation and a visible gap state, not infer that every intermediate market movement was captured.
This distinction explains why near-real-time collection is a feasibility and freshness problem, not an instruction to turn every scheduled feed into ten-second polling. Use tighter event windows for time-sensitive records and explicit off-peak rules for the rest, where justified by the decision being made.
Time Windows Matter More Than One Global Interval
A Datamam football-odds monitoring engagement applied a different schedule for a different decision. It monitored 480 active event-source combinations with ten-second snapshots during a six-hour active window and sixty-second snapshots during 18 off-peak hours. That represented 1,036,800 scheduled active-window jobs plus 518,400 off-peak jobs, or 1,555,200 daily scheduled snapshots before exceptional recovery.
The frequency reflected match-time urgency, and the delivered API exposed the age of the latest accepted market state. It did not imply that all intervening odds changes were captured. When a market’s handicap line changed, that was a different market identity, not just a revised price. The source event and market semantics still governed comparability.
This contrasts with a weekly physician-directory population or a monthly agency publication sweep. The question is not which interval is technologically impressive; it is which observation age materially changes the receiving decision.
Calculate the Tradeoff Before Tightening a Tier
For a fixed eligible population, moving from daily checks to hourly checks multiplies scheduled observations by 24. Moving from daily to a four-hour interval multiplies them by six. The added work may improve information age, but only the source change distribution and decision deadline can establish whether the added captures are valuable.
During a measured pilot, record how often accepted fields change, how long an observed state remains useful, how many changes can be acted on, how often source requests are limited, and whether consumer publication keeps pace. Segment the data by source and business priority. An unchanged check still proves an observation happened; it is not the same as discovering a new market event.
Do not optimize around one percentile calculated only from successful requests. Track scheduled versus accepted jobs, age of last valid observation, missed windows, source budgets and cost per accepted business observation. Datamam’s web scraping monitoring guide provides the operational signals needed to confirm whether the schedule is actually being met.
Treat Retry Policy and Freshness Policy as Separate Controls
If today’s high-priority check fails, the schedule should preserve its due-time obligation, mark it late or unresolved, and decide whether recovery is still useful. Blindly accelerating retries across all records can increase pressure on a source without improving the current feed. The web scraping retry strategies guidance belongs to incident handling, not to routine decisions about how frequently records should be observed.
A useful scheduling review compares actual source change rates, proportion of checks leading to accepted changes, average and tail observation age, overdue jobs, incident rates, and the importance of the affected records. Avoid calculating a universal “optimal” cadence from one average change frequency. Promotions, stock events, filings, and match-day prices have different timing distributions and decision costs.
For organizations evaluating managed web data services, write freshness into the delivery contract as a consumer-visible age or deadline requirement. A provider can then design queue budgets, refresh tiers, and exception handling to meet an agreed outcome. A global “hourly scraping” label hides too many of those decisions to serve as a reliable acceptance standard.
Revise Tiers When the Decision Changes
Set an initial policy, measure meaningful changes, then change cadence deliberately. A promotion period may justify moving a known product to a faster tier for a limited time; after the campaign ends, it can return to routine monitoring. A temporarily inaccessible source might need a different retry policy rather than more aggressive polling. A newly launched category may require broader discovery before any refresh optimization is meaningful.
Keep history when tiers change. A shift from daily to hourly monitoring must not change the meaning of the underlying product or observation identity. Publish the schedule version and due population so the consumer can interpret changing observation counts honestly.
For a recurring feed that needs tiered collection without unnecessary requests or misleading freshness claims, discuss your refresh requirements with Datamam.
Frequently Asked Questions
How often should a business scrape competitor prices?
It depends on the business decision, source changes, permitted access and processing cost. High-impact items may need multiple daily checks, while stable products may be monitored less often. Measure change behavior before choosing uniform intervals.
Is hourly scraping always better than daily scraping?
No. Hourly collection offers more chances to observe changes but adds scheduled work and may not improve decisions if the source rarely changes or the consumer acts daily.
Does a ten-second polling interval guarantee ten-second freshness?
No. Polling is a schedule. Queueing, source publication, retrieval, validation and delivery all add delay, and intermediate states can be missed between checks.
Should discovery and refresh have the same frequency?
Usually not. Discovery finds new eligible records and repairs identities; refresh revisits accepted records. Their coverage and time requirements can differ substantially.
How can we choose refresh tiers scientifically?
Observe representative sources, measure meaningful accepted changes by record group, compare the results with decision deadlines and the cost of missing a change, then validate the schedule under source limits.
How do we report freshness when the latest check fails?
Preserve the last accepted observation and its actual timestamp, and expose a stale or unresolved status according to the consumer contract. Do not create a fresh timestamp for an old value.



