Decide what counts as a duplicate before deleting rows
In this article
Removing duplicate rows sounds simple until two records share a name but represent different people, or share an account code but contain different details. The difficult part is deciding what sameness means for the current task. A duplicate remover can apply that decision consistently; it cannot supply the missing business definition. Before deleting anything, identify the entity represented by each row, choose a defensible key, and decide how conflicting records should be reviewed. The examples below are invented illustrations, not reports about customer data.
Name the entity represented by a row
Write one sentence describing a record: one customer, one order, one order line, or one event. This is the table's grain. A customer identifier may be unique in a customer list and repeated legitimately in an order list. Selecting it as a deletion key in both tables would confuse repeated business activity with duplicated data.
Inspect a few complete records rather than only the suspected key column. Include rows that repeat the candidate identifier but differ elsewhere. Ask whether those differences indicate a newer version, a separate transaction, or an error. When the answer is uncertain, preserve the records for review. Removing uncertainty from the preview is not the same as resolving it.
Separate exact repetition from entity matching
An exact repeated record has the same values across every field under the comparison rule. An entity duplicate may share only selected identifying fields. Rowlane's Data quality check reports exact repeated records; Duplicate remover lets you choose key columns. Those answer different questions, so their counts do not have to agree for either result to be useful.
Consider an invented table where code C102 occurs twice with different email values. Matching every column finds no exact repeated row. Matching customer_id groups the two records together, but still leaves a conflict about email. A decision to keep one row should explain why that occurrence is preferred, rather than suggesting that the matching key proves all other values are interchangeable.
Assess the candidate key
Look for a documented source identifier first. Where no single field identifies a record, evaluate a combination such as order_id plus line_number. W3C's tabular metadata model provides for primary keys composed from columns. This describes a constraint; it does not prove that an arbitrary combination chosen from your export actually satisfies that constraint.
Check the candidate for blank values, unexpected repeats, and changes between exports. A name, email address, or row position may be unsuitable if it changes or is shared. Rowlane retains records with a blank selected key for manual review instead of merging them. Investigate those rows separately, because a missing identifier supplies less evidence of identity, not more.
Decide whether normalization is justified
By default, Rowlane's duplicate comparison preserves case and surrounding spaces. Its optional matching settings can ignore case or trim key edges while retaining original exported text. Before using either option, determine whether those characters are significant in the identifier system. A normalization rule can reveal likely duplicates, but it can also collapse intentionally different keys.
RFC 4180 treats spaces as part of a field rather than automatically discarding them. That makes trimming a deliberate transformation decision. Try the exact rule first, then compare the groups produced by a justified normalized rule. Review newly merged groups, especially combinations where other fields disagree. Record the reason for the rule so the next run does not silently change it.
Define what keeping first actually means
Keeping the first occurrence refers to input order. It does not mean earliest creation date, latest update, most complete record, or verified source. Keeping the last has the same limitation. If source order is incidental, neither setting supplies a business priority. Establish an approved ordering or resolve the conflicts outside the duplicate-removal step before choosing a survivor.
For an invented example, suppose C102 appears once with a blank phone and once with a populated phone. Keeping last may happen to preserve the populated value, but it does not merge the records or validate that phone. If field-by-field consolidation is required, document that separate operation. Do not describe a retained row as a combined or verified record when it is neither.
Review both retained and removed records
Run the proposed rule and inspect its report before downloading. Reconcile the input count with retained and removed counts. Check several groups with differing non-key fields, not only groups where every value agrees. Keep the removed-record output alongside the chosen rule so the decision can be inspected without reconstructing the original input from memory.
Use a small synthetic fixture to exercise a single occurrence, an exact repeat, a conflicting repeat, a blank key, and a case difference. Predict what each should do before running the tool. If the result differs, revise the rule or expectation first. A repeatable fixture makes future changes easier to evaluate than a visual inspection of a large production export.
Stop when identity remains uncertain
Set aside unresolved groups with a clear review question and the relevant source context. They may need an authoritative identifier, a documented priority, or input from the data owner. A useful cleanup process can end with fewer automatic deletions and a better exception list. The goal is a defensible retained dataset, with every removed record explained, rather than the smallest possible row count.