Web Scraping Entity Matching: Resolve Records Without False Merges

web scraping entity matching

Key Takeaways

  • Define the identity grain before matching: vendor, product, offer, property, unit and observation are not synonyms.
  • Keep source identifiers beside canonical IDs; preserve aliases and evidence for moved or renamed records.
  • Separate exact matches, approved equivalents, candidates and rejected relationships.
  • Version match decisions so correcting a relationship does not erase the earlier accepted observation.
web scraping entity matching

A source URL identifies where something was observed, not necessarily which business entity it represents. The same hotel can appear under different channel names, a supplier can list several products with similar titles, and a residential community may contain many buildings, plans and units. Web scraping entity matching creates durable relationships across source records while controlling the risk of merging things that are not actually the same.

The objective is not the largest possible number of matches. It is an accepted relationship that downstream analysis can explain and, when evidence changes, correct without losing the original source history.

Decide Whether You Are Matching Entities or Relationships

A product can have several seller offers. A property can have several active or historical listings. A doctor can be affiliated with more than one hospital. Those are relationships between entities, not duplicated entities. Flattening them into a single row forces the pipeline to either discard useful observations or overcount the underlying business objects.

Start with a vocabulary: source profile, canonical entity, relationship, dated observation and supporting evidence. Each should have its own identity when the consuming system needs to distinguish it. Then define the cost of a false positive match versus a withheld candidate; in automated repricing, a false equivalence can corrupt a decision more seriously than leaving one price unranked.

Design an Evidence Ladder for Cross-Source Identity

A matching workflow should first generate possible relationships and then test whether each one satisfies the business definition. Exact source identifiers, GS1 Global Trade Item Numbers, government IDs, or explicitly linked official references can be strong evidence within their proper scope. Names, addresses, domains, and text similarity are often useful candidate features, but ambiguous records still need attribute checks or review.

Different match types should have different acceptance gates. Same-entity matching asks whether two source records refer to one underlying entity. Approved equivalence matching asks whether two distinct products are comparable under a declared rule. Parent-child matching links a product to a vendor, a unit to a floor plan, or a physician to an affiliation without making them the same object. A system that collapses all three into a universal “duplicate” flag cannot preserve meaningful relationships.

Define match evidence as a persistent artifact: source references, candidate features, rule version, reviewer decision, decision timestamp, and whether the link is exact, equivalent, provisional, or rejected. That record enables later correction when a source URL moves or a newly discovered attribute contradicts an earlier assumption.

Use Candidate Generation and Acceptance as Separate Steps

Candidate generation can use manufacturer part numbers, stable domains, normalized names, location, dimensions, coordinates or source links. Those signals reduce the search space but do not prove equivalence by themselves. Acceptance applies the business-specific rule: identical products may require matching identifier and pack quantity; substitute products may be allowed only after category review.

Match stateWhat it meansSuitable downstream use
Exact acceptedStrong direct identifier and compatible contextApproved joins and comparisons
Equivalent acceptedExplicit category or business rule supports equivalenceDefined comparative analysis
CandidateSimilarity evidence exists but is unresolvedReview queue, not automatic ranking
RejectedReviewed evidence establishes non-equivalenceExclude from accepted matching
SupersededPrevious accepted relation correctedHistorical replay and reconciliation

Match confidence scores can help triage, but they should not replace evidence or the acceptance policy. A score may be computed from sparse fields and still favor the wrong unit or source. The web scraping data quality resource provides a broader explanation of matching as a quality gate.

The property hierarchy in production

Datamam’s rental-data project required a multi-level identity map: community, building, floor plan, available unit, source listing, and image asset. A plan label and an asset filename refer to different things, even when the page displays them nearby. The project used a specific regression fixture to prevent a machine-generated image token from replacing a user-visible plan name. Evidence from a community page could not automatically establish the identity of a particular apartment.

That distinction is not cosmetic. A floor plan named A4a can describe several available units with different rents and move-in dates. A community profile can connect many buildings and plans but cannot establish the identity of a particular apartment. Source IDs, address components, coordinates, unit designators, and explicit links contributed different kinds of evidence. A changed listing URL updated the alias record rather than creating a new canonical community.

The entity layer must therefore retain cardinality: one community to many buildings, one plan to many units, and one canonical entity to many source aliases. Otherwise a deduplication pass can reduce a legitimate 1.9 million unit population into a much smaller, incorrect count.

A Rental Property Has More Than One Identity Layer

In one Datamam PropTech engagement, the monthly output covered 186,000 rental communities, 742,000 floor plans and 1.9 million unit listing records. The challenge was to connect community, building, plan, unit and source listing correctly. Address components, unit designators, source IDs and coordinates informed matching. A building-level page could support a community match without proving that a particular apartment was identical to another unit.

A specific source-data error had placed a machine-generated image asset name in a user-facing floor-plan field. The fix kept the published plan label separate from the image path, and regression tests checked the relationship. A plan called A4a retained that identity even when a file path contained a different token; multiple available units could refer to the same plan without becoming one rentable unit or one asking price.

This example shows why identity resolution is inseparable from relationship design. A near-perfect text match between two records is not enough if they refer to different grains of the property hierarchy.

Do Not Declare Cross-Directory Gaps Before Resolving Identity

The software-catalog project poses another identity problem: a suite can be represented as one product on one website and several modules on another. A common vendor name or domain was insufficient to prove direct product equivalence. The matching ledger therefore distinguished confirmed product pairs, unmatched profiles, and ambiguous portfolio relationships. A corrected match could recalculate a catalog coverage view without rewriting the original profiles.

A vendor name appearing on two sites is not enough to prove that a specific application is listed twice. A software suite appearing as one profile on one directory and several product modules on another may require parent-child or portfolio relationships instead of direct one-to-one matches. An uncertain match should remain in a review population until enough evidence exists, because a false merge removes a real gap from the enrichment opportunity set.

This illustrates why normalizing scraped data and matching are different steps. Standardized categories, vendor names, and domains make candidate search easier, but they do not settle product granularity. A reliable match ledger documents how evidence was accepted and lets an updated decision recalculate the coverage difference without rewriting source profiles.

Product and Software Catalogs Need Different Rules

In retail, manufacturer identifiers may justify exact candidates, while private-label equivalents require category attributes such as dimensions, material or pack quantity. Datamam’s retail pricing project maintained 4.8 million accepted SKU-to-retailer relationships across 12 integrations rather than assuming every possible catalog pair was valid.

Software directories introduce another mismatch: vendor and product are separate entities. In a Datamam directory project, 92,000 profiles from one source and 118,000 from another were compared after resolution. The project established 71,000 matched profile pairs, leaving 21,000 and 47,000 source-exclusive profiles. Suites and individual modules did not automatically match just because they belonged to the same vendor or shared a domain.

These are not interchangeable matching algorithms. Product-pack equivalence, software ownership and residential-unit identity rely on different evidence. A universal fuzzy-title threshold may be convenient but has no business reason to produce equally reliable decisions across all three.

Preserve Matches Through Source Changes

Store the source ID, canonical entity ID, match type, original evidence, reviewer or rule, rule version and effective interval. When a URL moves or a vendor rebrands, add an alias if continuity is supported. When a source reuses an old URL for a different product, invalidate the affected mapping while preserving older observations under their previously accepted identities.

Corrections should trigger review of downstream records derived from the changed relationship. A source observation can remain authentic even if its mapping was wrong. Do not replace its capture time or rewrite its source content. This distinction protects historical analytics and makes model retraining or retrospective comparisons explainable.

The source-discovery step and the acceptance step should have different metrics: candidates found, candidates reviewed, accepted exact matches, accepted equivalents, unresolved cases and rejected pairs. A high raw match count can hide both dangerous false merges and unexplained omissions.

Review False Merges and Missed Matches Separately

A precision-focused review asks how many accepted pairs are actually the same entity or approved equivalent. A recall-focused review asks how many real relationships the process missed. Both are needed, because conservative rules can overlook useful matches while aggressive rules create damaging false merges. Choose a review sample stratified by category, source, confidence band, and relationship type; don’t publish one pooled accuracy percentage that hides difficult categories.

Test representative edge cases: units without apartment numbers, companies sharing similar names, product bundles versus individual components, discontinued models, reused URLs, and software rebrands. Keep rejected candidates and their reasons so a rule update does not repeatedly rediscover the same false match. Corrections should revise the relationship record and any dependent comparative view while preserving earlier source observations.

When planning scraping output contracts, specify whether consumers receive source IDs, canonical entities, accepted relationship types, and confidence or review states as separate objects. An identity result that cannot be audited is unsuitable for consequential analytics, even if it makes the row count look pleasantly clean. For fully managed web scraping, decide whose business definition controls match acceptance and who approves exceptions.

Make Matching Quality Reviewable

Build review samples around ambiguous names, repeated identifiers, variant changes, suites versus modules, multi-unit properties and recently corrected mappings. Inspect underlying source evidence, not only similarity scores. The sampling unit should match the business assertion being tested: a product equivalence, a property-unit relationship or a vendor-product ownership link.

Use stable output keys so re-running the matching stage does not duplicate business entities. When a decision changes, issue a versioned mapping update and reconcile the affected observations. The web scraping proof-of-concept guide can help define acceptance tests on the hardest cases before scaling the operation.

For multi-source datasets that depend on accurate product, business or property relationships, discuss your matching rules with Datamam.

Frequently Asked Questions

What is entity matching in web scraping?

It is the process of determining when source records refer to the same underlying business entity or an approved relationship, using identifiers, contextual evidence and documented acceptance rules.

Can two matching names be treated as the same product?

Not automatically. Variants, pack quantities, manufacturers, seller conditions and product families can differ despite identical or similar names. Review the attributes relevant to the intended use.

How should uncertain matches be stored?

Keep them as reviewable candidates with their supporting evidence and reason for uncertainty. Exclude them from high-confidence business comparisons until accepted under the relevant rule.

Should the source URL become the canonical entity ID?

Usually not. URLs can redirect, be reused or change during rebranding. Preserve URLs as source references while maintaining a stable entity identity and alias history where evidence supports continuity.

How do you prevent matching one apartment with an entire building?

Define separate community, building, floor-plan and unit identities and the relationships between them. Require unit-level evidence before accepting a unit match; address similarity alone is insufficient.

What happens when an earlier match is found to be wrong?

Version the correction, retain the source observations and earlier decision, and recompute only affected derived views or comparisons under the new accepted mapping rule.