Product Data Consolidation: Building One Golden Record

Monday morning, 09:12. Three new supplier files have landed in the shared inbox. The first calls the column Kleur, the second Colour, the third simply kl. The same dining chair appears in all three files, but with three different EANs, one of them empty. And these are just 3 of the 40 suppliers you have to process this week.

At supplier A the colour is called antraciet, at B donkergrijs, at C DK-GREY. The dimensions come through as DIM_MM 620x820x450 in one file and as three separate columns in another. You can stitch this together by hand and lose a good chunk of your week to cut and paste, much like you probably already do in the growing Excel chaos of your product list, or you let a system deduplicate the files into a single record per product. But only on one condition: that for every value you can still see which supplier delivered it.

Why supplier files never line up on their own

Every wholesaler delivers in whatever way suits their own system. The column names differ, the spellings differ, the units differ. Some fields are empty, EANs are missing or show up in scientific notation because Excel decided to reformat them. There is no shared standard, and there never will be: you are not going to force 40 suppliers to use your template.

This is not an incident but an ongoing situation. Product data ages continuously: new products arrive, suppliers change and specifications shift, so what is correct today is already partly out of date a quarter from now. Anyone who wants to merge dozens of supplier files into one golden record with per-field provenance needs a process, not a one-off project.

Step 1: deduplicate the duplicate products

Merging starts with recognition. The same product often sits in multiple files, and rarely with exactly the same key. In the technical literature this is called entity resolution or record linkage: records are linked to one another on product codes, names and attributes into a single source of truth, with probabilistic matching assigning weights to the comparison per field.

In practice that means: hard keys like SKU, EAN and barcode first, and where those are missing or contradictory a match on title, brand and specifications. That empty EAN from supplier C is then not simply left out, but linked on the basis of what does match.

Step 2: normalise to YOUR structure

This is where it often goes wrong. The temptation is to adopt the supplier’s structure, because it is already there. But then you end up with a catalogue with 47 ways to spell a colour. Purple, lilac and violet are the same filter box to your customer. So are DK-GREY, donkergrijs and antraciet.

Normalising means you bring those 47 spellings back to your own 12 standard colours, and that 620x820x450 in millimetres lands neatly in your own height, width and depth fields. Your existing shop structure is the starting point, and new sources plug into it. So you do not start from scratch: the system reads how your catalogue is put together and adapts the supplier data to it.

Step 3: product data consolidation into a golden record

Now that the duplicates are matched and the fields normalised, you have to choose. If three suppliers deliver a price, a lead time and a description, which one wins? In Master Data Management this is called survivorship: from the matched records the most reliable value per field is consolidated into one golden record. Informatica and Data Ladder describe exactly these three steps: matching, merging and survivorship.

Survivorship is a set of rules. The primary supplier wins on price, the manufacturer wins on specifications, the longest description wins on text. Fine. But the moment that record sits in your shop, the real question arises.

Take a single pendant lamp that turns up in three files. Different names, three spellings for the colour, three dimension notations and an empty EAN on the third. Click and watch the three raw rows collapse into one golden record:

Per-field provenance: why it makes the difference

Suppose a wrong price is live. With a black-box import you only know that the price is 149 euros. Not which of the 40 suppliers delivered it, not whether an automation or an AI suggestion changed it, not whether someone overwrote it by hand. You are stuck.

Data provenance answers exactly that question: where does this value come from? Data catalogue vendors such as Atlan draw the distinction sharply: provenance is about origin, trust and evidence, while lineage is about how data flows and changes. At field level you see the source for every value. And that is no side issue: without provenance you cannot trace an error back to who or what caused it, and every correction stays guesswork.

With per-field provenance you see for that 149 euro price whether it came from supplier B, whether an automation rounded it, whether AI suggested it, or whether a colleague set it by hand. And you can steer: a manual value wins, and you can lock a field so the next import does not overwrite it again.

Quality control and the last step to your shop

Merging is only done once the deviations are out before they reach your shop. A price that is suddenly ten times too high, a negative stock level, a dimension that does not add up: that belongs stopped at the gate, not discovered through a return. And it pays off, because bad product data costs you a customer who bails out and a return you could have prevented. Every deviation you stop before the shop pays for itself.

The last step is the way out. You connect sources via supplier feeds, SFTP, email, ERP and REST API, and at the other end the clean records go live in your webshop. Along the way every supplier photo is converted to WebP at the right size per location, up to around 98 percent lighter, and duplicate photos are stored only once. Faster pages count towards Google: a good LCP, according to web.dev, is 2.5 seconds or less.