In production · Public Health – Seattle & King County
Open-source codeMachine learning record linkage
Efficient and accurate way to link identities across datasets
Published October 5, 2026 Updated October 15, 2026
- Skills needed to set it up
- Analyst or data scientist
- Readiness
- Needs customization
- Data it touches
- Personal information (PII)
- Who sees the output
- Internal staff
- Cost to stand up
- No new spend
- License
- GPL / AGPL
- Portable to other platforms
- Yes — platform-agnostic
- Reviews it went through
- None required
- Public health area
- Data modernization and informatics
What do these mean?
- Analyst or data scientist
- Someone who works in Python, R or SQL.
- Needs customization
- Your team will need to adapt it before use.
- Personal information (PII)
- Names, addresses, IDs or other personal identifiers.
- Internal staff
- Used only inside the organization.
- No new spend
- Built with licences, staff and infrastructure the organization already had.
- GPL / AGPL
- Copyleft — derivatives must stay open.
- Yes — platform-agnostic
- Runs on any comparable stack with configuration changes only.
- Data modernization and informatics
- Shared data systems, interoperability and data governance that serve more than one program.
Problem
We need to link records across systems (and deduplicate within systems) all the time, so have built up various methodologies. However, a research grant needed to link data across 12 administrative data systems with over 6m unique identifiers, which was more than existing methods could handle efficiently. Therefore, we needed a new way to link records.
Approach
We built on some work being done at the state department of health to implement a machine learning-based linkage model. Key enhancements included adding a screener regression model that quickly classifies obvious matches and non-matches, flexibility in the variables used to match, and a network/cluster analysis to catch over-linked individuals.
Time and resources
The only cost for the work was staff time, which was paid for by the grant. The model can run on local laptops with no issues up to several million records.
Initial development took approximately 3 months of a data scientist working at full time, and we have spent an additional ~250 hours adding enhancements and improving performance.
Results
The linkage performs well both in terms of accuracy and computational need.
We compared match results with output from Splink (Python package) and our existing integrated data hub, which uses Informatica. In both cases, the ML linkage was generally slightly more accurate across a range of metrics.
The process is also very efficient, able run from end-to-end (including refreshing and cleaning data) in about 5 hours.
Lessons learned
With dedicate time and focus, it is possible to build linkage systems that compete or outperform commercial offerings.
How to reuse this
Other organizations could fork the GitHub repo and start applying the process relatively quickly. Most time will be spent preparing datasets for linkage and building a testing/training dataset with your specific data.
About
- Type of solution
- AI/ML model
- AI use category
- Administrative & task automation
- Review status
- Reviewed & approved
How it's built
- Role of AI
- AI is part of the solution
- Tools and technologies
- R
- Where it runs
- Desktop or local
Sharing & licensing
- Portability notes
- Runs on a desktop or laptop; could be deployed into any VM or adapted for platforms such as Databricks.
What it took
- Cost to keep running
- No ongoing cost
- How it was bought
- Grant funded
Data & access
- No PII/PHI in the shared material
- Yes