AlloyMDM
How it works

One core, two shells.

The rules of matching and merging live in a small core library with no database, web server or Spark dependency. The Job and the App are thin shells around it, and Unity Catalog tables are the contract between them.

The picture

sfdc_accountyour gold table sfmc_accountyour gold table eb_accountyour gold table AlloyMDM Jobmatch, merge, survivea task in any job Separate UC schema goldencrosswalkreview_queuelineage · rejectsruns · rules_preview · meta Written by peopledecisions (confirm / reject a pair)rules (threshold and survivorship overrides) AlloyMDM Appsteward workbench, planned

Dashed line: the Job replays people's decisions and rules on every run.

What happens in a run

  1. Read every configured source table from Unity Catalog.
  2. Clean values: names, phones, addresses and identifiers are standardized so they can be compared.
  3. Find candidates using blocking keys (tax id, DUNS, website, name plus postal code) so it does not compare every record to every other.
  4. Score each candidate pair. Above the auto-merge threshold the records merge; between the review and auto-merge thresholds they go to the review queue; below it they stay separate. Defaults are 0.92 and 0.55 and both are configurable per domain.
  5. Apply decisions: a person's confirm or reject always beats the engine.
  6. Survive: pick the best value for each field. By default the most recently updated source wins, using the source's own timestamp, and you can set priorities per field.
  7. Write golden, crosswalk, review queue, lineage, rejects and run history, keeping mdm_ids stable by overlap with the previous run.

The tables

TableWritten byHolds
<domain>_goldenJobOne row per real-world entity
_crosswalkJobEvery source row mapped to its mdm_id
_review_queueJobPairs the engine would not decide alone
_lineageJobWhich source supplied each golden value, and why
_rejectsJobSource rows too incomplete to use
_runs, _meta, _rules_previewJobRun history, contract version, and the effect of a proposed rule change
_decisionsPeopleConfirm or reject, per pair
_rulesPeopleRule overrides

Every table has exactly one writer, which is why the Job and the App can share them safely. Tables carry a contract version, and a run refuses to write over tables made under a different version.

Configurable by your team

Matching and survivorship rules are configuration, not code. They layer: product defaults, then the YAML next to your job, then a rules table in Unity Catalog. Later layers win, and the merged result is validated before anything is written. Before changing a rule, run a preview to see how many groups would merge, split or change a golden value.

# mdm_step.yaml (excerpt)
domain: account
output:
  catalog: main
  schema: mdm_account
sources:
  - name: sfdc
    table: main.gold.sfdc_account
    id_column: account_id
    updated_at_column: last_modified_date
    columns:            # canonical field: this table's column
      account_name: name
      website: website
  - name: sfmc
    table: main.gold.sfmc_account
    id_column: subscriber_account_id
    updated_at_column: modified_at
    columns:
      account_name: company_name
rules:
  auto_merge_threshold: 0.94