Is Web Scraping Legal? What US, EU and UK Law Actually Allows

Is Web Scraping Legal

Key takeaways

  • Public visibility may lower certain unauthorized-access risks, but it does not eliminate contractual, privacy or intellectual-property obligations.
  • The US hiQ v. LinkedIn decisions do not establish a blanket right to scrape: the 2022 appellate decision concerned a preliminary injunction, and later proceedings reached a very different contractual result.
  • The GDPR can apply to publicly visible personal information; lawful basis, transparency, minimization and individuals’ rights still matter.
  • A sound enterprise approach evaluates each source, records access and reuse conditions, and separates permitted collection from permitted downstream use.


Web scraping is not inherently illegal, but it is not automatically lawful simply because a page is public. Whether a project is permissible depends on how the data is accessed, what it contains, the rights attached to it, the purpose of reuse and the jurisdictions involved. A public product price, a password-protected account, a professional profile and a copyrighted article do not raise the same legal questions.

For businesses, the useful question is not whether scraping software is legal in the abstract. It is whether a defined collection and reuse workflow can be justified under the applicable laws and source conditions. That assessment should happen before collection begins and should be revisited when the source, purpose or operating method changes.

This guide explains the main rules as of November 2026, including US court decisions, EU and UK privacy obligations, intellectual property, website terms and recent European regulatory guidance. It provides general information, not legal advice for a particular project.

A web scraping project is more defensible when it collects lawfully accessible information, respects applicable contractual and technical restrictions, avoids infringing protected material, and has a lawful basis for handling any personal data. None of those factors is sufficient alone. They have to be considered together.

For instance, a government data portal such as Data.gov may offer openly available datasets, but a specific dataset’s license, attribution requirements and any personal-data restrictions still need to be checked. Likewise, extracting public supplier information to analyze procurement may be useful commercially without automatically establishing a right to copy every attachment or redistribute the source material.

Five questions determine the initial legal assessment:

1.  Access: Is the information visible without an account, or does collection require credentials, a subscription, permission or circumvention of a control?

2.  Agreement: What terms, licenses, notices or contractual commitments apply to the collector and intended customer?

3.  Content: Does the material include personal data, copyrighted expression, protected database content, confidential information or regulated records?

4.  Reuse: Is the intended activity an internal comparison, research, commercial redistribution, public display, automated decision or AI training? Different uses can raise different issues.

5.  Jurisdiction: Where are the collector, source operator and affected individuals or customers located, and which legal regimes actually apply?

The distinction between access and reuse is essential. Being able to view a webpage does not necessarily grant permission to republish its photographs, sell the source’s database or combine an individual’s profile with other records for a new purpose.

What does US law say about web scraping?

The United States has no single federal statute that declares all web scraping lawful or unlawful. Disputes can involve the Computer Fraud and Abuse Act (CFAA), contracts, copyright, state laws and the consequences of collection methods. The most-cited court decisions answer narrower questions than many online summaries suggest.

The CFAA, public pages and the hiQ-LinkedIn litigation

The CFAA addresses specified forms of unauthorized computer access. In Van Buren v. United States (2021), the Supreme Court interpreted the statute’s “exceeds authorized access” language narrowly. The dispute involved access to a law-enforcement database, not ordinary website scraping, but the reasoning influenced later web-access cases.

In hiQ Labs v. LinkedIn (Ninth Circuit, April 18, 2022), the appeals court affirmed a preliminary injunction in a dispute over access to publicly available LinkedIn profiles. It found serious questions about applying the CFAA’s “without authorization” provision to material that anyone could view without authentication. This was not a final ruling that all public-profile scraping is legal, nor a decision granting a general copyright or privacy exemption. The case overview is useful background, but the court opinions control the analysis.

The subsequent history matters just as much. In November 2022, the district court found that hiQ had breached LinkedIn’s user agreement through scraping and other conduct, although additional issues remained. A December 8, 2022 consent judgment entered a $500,000 judgment against hiQ and imposed a permanent injunction on specified conduct. The restrictions bind the parties covered by that order; they are not a universal prohibition on other organizations. Together, the rulings demonstrate why a favorable CFAA argument does not resolve contractual risk.

Website terms can create a separate contract question

A site’s terms of service are relevant, but a prohibition written on a webpage does not automatically decide every lawsuit. Contract enforceability and scope depend on factors such as notice, assent, the wording of the agreement and the relationship between the parties.

The distinction appears in Meta Platforms v. Bright Data (January 2024). On the record before it, a California federal court held that Meta had not shown a breach of its particular Facebook and Instagram terms by Bright Data’s logged-out collection of public data. That decision turned on those terms and facts. It did not declare logged-out scraping immune from all contracts or privacy laws.

Older disputes such as Ticketmaster Corp. v. Tickets.com also illustrate how different claims may be raised around automated collection, copying, linking and website agreements. They should be read as case-specific litigation, not as a single rule that governs every website.

For a production program, obtain legal review of relevant terms, record the version reviewed and identify which person or entity has actually agreed to them. A credentialed collection route deserves distinct scrutiny even if some of the same data is visible to logged-out visitors.

In the US, facts such as a publicly displayed price or a historical date are generally not protected by copyright. Original website text, photographs, illustrations and sufficiently creative selection or arrangement may be. The US Copyright Office distinguishes unprotected facts from protected expression. Copying material for internal analysis and reproducing large portions in a competing product therefore require different assessments.

Fair use can permit some unlicensed uses of copyrighted material, but it is a fact-specific defense assessed under four statutory factors, not a blanket exception for research, commercial analytics or AI training. See the Copyright Office’s fair-use explanation.

The Digital Millennium Copyright Act (DMCA) raises a separate issue: Section 1201 generally restricts circumvention of technological measures controlling access to protected works, subject to legal exceptions and exemptions. The Copyright Office’s DMCA guidance explains that prohibition. A project that merely reads an open page should not be conflated with one that attempts to defeat protected access controls.

Public visibility does not remove personal information from the scope of the EU General Data Protection Regulation. Names, identifiable professional profiles, contact details, user posts and location histories may still be personal data, depending on context. An employee’s business email address can identify a person even when it appears on a public company page.

Under the GDPR, a controller processing personal data needs an applicable Article 6 lawful basis and must comply with the Article 5 principles, including purpose limitation, data minimization, accuracy and storage limitation. Consent is not the only possible lawful basis. Legitimate interests may be relevant in some commercial contexts, but require a genuine interest, necessity and a balancing assessment against individuals’ rights. Special-category data can trigger additional Article 9 restrictions.

Collection from third-party webpages also raises transparency duties under Article 14. In some circumstances an exception to individual notice may apply, but that does not automatically remove the need for other transparency measures. Controllers may also need to handle requests for access, erasure and objection, evaluate security safeguards and conduct a data protection impact assessment where the processing is likely to create high risk.

The UK GDPR and Data Protection Act 2018 establish a related but distinct UK regime. The UK Information Commissioner’s Office notes that data obtained through web scraping can still trigger Article 14 transparency obligations. A business collecting data across Europe should establish which regimes apply rather than assuming that compliance with one jurisdiction settles all others.

What changed in European guidance during 2026?

On January 5, 2026, France’s data protection authority, the CNIL, published specific guidance on legitimate interests and web scraping. It states that scraping is not prohibited in itself, but emphasizes predefined collection criteria, limiting unnecessary data, deleting irrelevant material and observing meaningful website opposition measures in the AI-training context. Its examples and safeguards should not be misread as an automatic legal authorization for every scraper.

The European Data Protection Board then issued Guidelines 03/2026 on web scraping for generative AI, adopted in July 2026 for public consultation. The feedback period closed on October 30, 2026. As of this review, these are consultation-stage guidelines, not a confirmed final version. They focus on private-sector scraping for generative-AI development and discuss lawful basis, transparency, minimization, data quality and safeguards. They do not purport to be a blanket rule for every form of commercial web scraping.

How does California privacy law differ?

The California Consumer Privacy Act (CCPA), as amended, has its own definitions, business-applicability thresholds and exemptions. Importantly, some lawfully made public information falls outside the CCPA definition of personal information. That carveout is more specific than an assertion that every public business contact or profile is unregulated. California’s Attorney General guidance, updated August 2026, explains the scope and consumer rights.

US organizations must evaluate the applicable state law and actual handling of data, especially when combining public data with nonpublic identifiers, deriving profiles or sharing information with customers. A CCPA exemption does not cancel contractual, copyright, consumer-protection or other applicable obligations.

Are robots.txt, website terms and CAPTCHAs legally binding?

These controls serve different purposes and should not be treated as interchangeable.

  • robots.txt communicates crawler preferences under the IETF Robots Exclusion Protocol, RFC 9309. The specification explicitly says these rules are not access authorization. A permitted path does not grant a content license, and a disallowed path should not be interpreted as a criminal-law judgment by itself. Regulatory expectations may still require particular actors to honor opposition signals, especially for scraping personal data for AI training.
  • Terms of service can affect contractual obligations when an enforceable agreement applies. Reading the terms is necessary; their legal effect depends on the actual agreement and facts.
  • Login barriers, CAPTCHAs and access controls may indicate that the operator limits automated access. A CAPTCHA is not the same thing as a copyright license or a court order, but bypassing technical controls can introduce additional authorization, contract or anti-circumvention questions.
  • Rate limits and blocking notices signal operational boundaries. Excessive traffic can impair availability or contribute to a denial-of-service problem. Operators may use content delivery networks and other protections to control traffic. Responsible collectors should avoid disruption, implement conservative request budgets and escalate unresolved access restrictions rather than treating them as engineering obstacles alone.

A practical rule is to document the source’s technical and contractual restrictions together. Where the legal position is unclear, pause the affected collection route and seek source permission or legal review. There is no universal “robots.txt makes scraping illegal” rule, nor a universal “robots.txt allows it” safe harbor.

EU law may protect not only creative text and images but also certain databases. Under the Database Directive, a qualifying database maker can have a sui generis right related to substantial investment in obtaining, verifying or presenting its contents. Extracting or reusing substantial parts can raise rights questions even when individual records are factual. Repeated extraction of insubstantial parts can also matter in some circumstances.

The EU’s 2019 Copyright in the Digital Single Market Directive includes text-and-data-mining exceptions with conditions. Article 3 addresses certain research organizations and cultural heritage institutions with lawful access. Article 4 covers broader text and data mining of lawfully accessible material, but can be unavailable when rights have been expressly reserved in an appropriate manner, including machine-readable means for online content. Member-state implementation, the exact activity and other rights still require examination.

AI training is not synonymous with ordinary price monitoring. Collecting article text, retaining a training corpus, making copies, processing personal information and distributing model outputs can involve several different legal acts. An organization must therefore assess the proposed training or reuse independently of whether a crawler can retrieve the source page.

The table below is a screening aid rather than a legal ruling. Risk changes with source conditions, processing purpose, collection method and jurisdiction.

Collection scenarioMain legal questionsPractical starting point
Public retail prices and product attributesTerms, access method, product images, any personal informationSeparate factual price fields from protected images or text; check site restrictions.
Official government notices and open datasetsPublication license, attribution, source scope, redaction, personal dataPrefer official feeds or exports when available; preserve source and license metadata.
Public professional profiles and named contactsPrivacy lawful basis, expected use, terms, profiling and onward sharingTreat public visibility as one factor, not permission for unlimited reuse.
Articles, photographs and commercial databasesCopyright, database rights, licensing, fair use or statutory exceptionsSpecify exactly what is copied, retained, displayed or redistributed.
Subscription or authenticated platformsContractual permission, authorization, security barriers and permitted API scopeObtain permission or a suitable license; review the specific access route.
Automated ticket purchasing or bulk resaleSpecialized anti-bot restrictions in addition to ordinary access rulesDistinguish inventory monitoring from automated purchasing or limit circumvention.

The final row deserves care. The US Better Online Ticket Sales (BOTS) Act targets certain circumvention of ticket-purchase limits and related resale, not routine observation of every event listing. The FTC’s April 2025 explanation describes the prohibited conduct. In the UK, the Digital Economy Act 2017 enabled rules on breaching ticket-purchase limits; the 2018 Regulations provide the operative offense in the circumstances they define. These laws should not be summarized as bans on all ticket-price scraping.

For context on legitimate business applications, Datamam also discusses industry uses of web scraping. The commercial use case may explain why collection is valuable, but it does not establish the collection’s legal basis.

How to assess and operate a compliant web scraping project

The most useful safeguard is a repeatable review process attached to the source inventory and the delivered dataset, not a single legal opinion that is disconnected from future source changes.

1.  Define a specific purpose. Document the required fields, intended users, refresh schedule, geography and forms of reuse. Exclude fields not needed for the decision.

2.  Classify every source. Record its canonical URL, operator, country, public or authenticated access route, API or download alternatives, terms and licensing notices.

3.  Record rights and permissions. Distinguish permission to access, permission to extract, permission to store and permission to redistribute. Keep the relevant terms or license version and review date.

4.  Assess personal data separately. Identify whether records concern identifiable people, select an applicable lawful basis where required, document transparency, retention, rights handling and special-category exclusions.

5.  Set technical boundaries. Use reasonable rate limits, source-specific scheduling and stop or escalation rules when a site blocks access or its conditions change. Do not build the operating plan around circumventing disputed restrictions.

6.  Preserve provenance and corrections. Keep source references, collection timestamps, acceptance states and changes so the client can distinguish a missing observation from a genuinely absent record. Protect retained evidence and delete data when no longer justified.

7.  Control downstream use. Make data licenses, deletion requests, retention periods and limits on redistribution visible to the team or customer using the feed.

8.  Review material changes. Reassess when a new source, territory, login method, data category or downstream purpose is introduced. Escalate uncertain cases to qualified counsel.

Datamam’s first-party delivery records show why source traceability matters operationally. In one public-agency research program, the team maintained 1,460 accepted agency sources across 200 cities, collecting 219,000 relevant pages and documents per monthly full pass. The output linked each result to its publisher, jurisdiction, supporting passage and publication context. In a separate procurement program, 64 sources fed a history of 1.2 million notices, with amendments and document relationships preserved rather than flattened into untraceable rows. Those figures describe data engineering and evidence practices; they do not, by themselves, certify that every potential reuse of every source is legally authorized.

This discipline is part of making data-driven business operations reliable. An accepted record should state what was observed and where it came from, while the project governance file should separately state what the organization is allowed to do with it. The two must not be confused.

Conclusion: lawful web scraping is a source-by-source decision

Web scraping can support research, competition analysis, pricing, procurement and many other legitimate activities. But neither a public URL nor a successful automated request resolves the legality of acquiring, storing and reusing the result. US access law, enforceable contracts, copyright, EU database rights, privacy obligations and specialized anti-bot laws can each apply to different parts of the same workflow.

The practical standard is to define the data and its purpose, evaluate each source and access method, record applicable rights, minimize personal-data exposure, and retain enough provenance for ongoing review. A managed web scraping works and uses service can perform the engineering and documentation work, but responsibility for lawful use still depends on the actual parties, roles and agreements.

If your organization is planning a multi-source data feed and needs to define source access, provenance, restrictions and delivery responsibilities, discuss the project requirements with Datamam.

Frequently asked questions

It depends on the data, route of access, applicable terms and intended reuse. Collecting factual product prices from publicly visible pages raises a different set of issues from copying product photographs, using a restricted account or republishing a proprietary catalog. Review the specific source and the destination use; do not assume every retailer has identical conditions.

No blanket answer follows from the hiQ appellate decision. The 2022 ruling addressed preliminary relief and a CFAA issue involving public profiles. Later proceedings found contractual breaches, followed by a consent judgment restricting hiQ. Privacy law and the purpose of profiling or redistribution may create additional obligations.

Does GDPR prohibit scraping public personal data?

No. The GDPR does not ban web scraping as a technique. However, when a collection processes personal data within its scope, the controller must meet the applicable requirements, including a lawful basis and appropriate safeguards. Public accessibility does not automatically remove those requirements.

No. Under RFC 9309 it is a crawler instruction mechanism, not an access-authorization or copyright-licensing system. Nevertheless, it can be an important signal in a source’s restrictions, and some regulatory guidance expects particular collection activities to respect such opposition.

Not necessarily. A buyer still needs to understand permitted uses, licensing, confidentiality, data protection roles, retention, deletion rights and restrictions on onward sharing. Contractual assurances and source evidence reduce uncertainty but do not override laws that apply to the buyer’s own processing.

Not always. Consent is one possible lawful basis under the GDPR, but other bases may apply when their conditions are satisfied. A public work email address may still be personal data. In the US, applicability and exceptions differ by state and context. Organizations should assess the actual purpose, legal basis and outreach rules rather than assuming that public contact details are free to use.