Deduplication

How deduplication works on contact lists

Deduplication starts with a match key. Exact keys include work email, CRM id, or LinkedIn URL. Fuzzy keys include name plus company domain when emails differ. The job groups rows that appear to be the same entity, picks a survivor, and merges or deletes the rest.

Wikipedia describes data cleansing as identifying and correcting or removing corrupt, inaccurate, or irrelevant records from a dataset 1. Duplicate contacts are one of the inaccurate shapes that cleansing programs target. Record linkage, also called entity resolution or data matching, is the task of finding records that refer to the same entity across different sources when a shared identifier may be missing 2.

In outbound stacks, deduplication sits next to data enrichment. If you enrich before you dedupe, you can pay twice to fill the same person. RichAPI Profile enrichment and Company enrichment each cost 1 credit per call and bill on any 2xx response 3. Duplicate rows turn that rule into duplicate spend.

Survivorship rules matter as much as match rules. Decide which source wins for title, email, and owner. Store the merge time and the losing record ids so support can undo a bad merge.

Deduplication also protects deliverability. Two sequences to the same mailbox look like harassment to the buyer and like noisy engagement to filters. Pair list dedupe with verification so the surviving email is still valid.

Measure match rate and fill rate only after duplicates are gone. Otherwise vendor coverage looks worse than it is because the same person appears as two blanks.

Blocking strategies from record linkage keep large files tractable: compare only rows that share a domain or postal region instead of every pair 2. That engineering detail is what makes nightly CRM dedupe finish before morning sequences start.

Normalize before you match. Lowercase emails, strip plus-tags when your ESP treats them as one mailbox, and canonicalize company domains. Skipping normalization creates false uniques that look like growth and feel like spam to the buyer.

Cross-object duplicates matter too. A person record and a lead record can represent the same human. If your CRM keeps both objects, dedupe rules must span them or enrichment will still double-write.

Audit merges monthly. A small sample of merged pairs reviewed by ops catches rules that are too aggressive on common names inside large companies 2.

Internationalization adds another trap. Accented names, translated company strings, and local-part email quirks create duplicates that ASCII-only rules miss. Add locale-aware normalization before you claim the CRM is clean 1.

After a major CRM migration, assume duplicates returned. Re-run deterministic keys before you turn enrichment workers back on 3.

Tooling can help, but rules still need owners. A fuzzy match library without a steward becomes a black box that merges the wrong CFOs. Prefer explainable deterministic keys for automatic merges and reserve fuzzy matches for a review queue 2.

When two surviving records both have activity history, merging is a product decision. Pick the record with the richer timeline, move tasks, and leave a breadcrumb note so AEs understand why the other id vanished.

Why deduplication belongs before enrichment credits

Credits follow rows. Duplicate rows multiply credits without multiplying pipeline. On RichAPI, a 2xx enrichment response can bill even when the body is thin 3. Two thin responses for one human are pure waste.

Import paths create most duplicates. Marketing uploads a conference list. Sales uploads the same badge CSV a week later with slightly different emails. Without a merge key, the CRM grows two living records.

Ownership fights follow. Two AEs get the same account because dedupe never ran. Enrichment then writes conflicting titles onto both rows and reporting breaks.

Treat dedupe as a gate in the same runbook as waterfall enrichment. Find, merge, then enrich the survivor. Verify the surviving email before sequence push.

When identifiers are weak, lean on record linkage ideas: deterministic matches on email first, then probabilistic matches on name and company with a human review queue for borderline pairs 2.

Keep a reject report. Rows that look like duplicates but fail confidence thresholds should not disappear silently. Ops needs a weekly list of near-matches to train the rules.

Legal holds and suppression lists must survive merges. If a losing row was on a do-not-contact list, that flag has to win on the survivor or you will create a compliance incident while celebrating a cleaner CRM.

Enrichment providers cannot save you from bad dedupe. They will happily fill both copies. Your enrichment credit budget is the hostage.

Example of deduping a merged CRM after two conference imports

After two events, a CRM holds thousands of new contacts. Hundreds share a normalized work email with an existing contact. More share LinkedIn profile URLs. Ops runs deterministic dedupe on those keys first 2.

Survivorship keeps the older owner and the newer title when the title source date is fresher. Losing rows are archived with a pointer to the survivor.

Only then does the team enrich survivors that still lack industry, using Company enrichment at 1 credit per call 3. Duplicate spend disappears from the receipt. Sequences see one mailbox per buyer.

A weekly near-match report catches same-person rows that only share name and domain. Those wait for a human merge. That is deduplication as hygiene, not as a yearly panic cleanse 1.

Leadership later asks for a bounce dashboard. Because duplicates are gone, bounce rate finally reflects list quality instead of double sends to the same mailbox. That feedback loop is why dedupe belongs in the same weekly ops review as deliverability.

The scorecard from that run becomes the template for the next cohort, so arguments about coverage start from the same denominator every time.

Deduplication vs data enrichment

Deduplication collapses duplicate records. Data enrichment adds fields to the records that remain.