Entity resolution is the process of determining when different data records refer to the same real-world entity, and when they don’t. Senzing does this in real time as each record arrives, by resolving it against the entities it already knows and applying principles, with no training, fine-tuning, or ER experts required.
How Senzing Entity Resolution Works
What happens when a record is added
In industry terms, entity resolution is often described as blocking, matching and clustering. Senzing handles blocking with candidate keys built from each record’s features, which find a short list of candidate entities. It handles matching by comparing the record’s features with each candidate entity’s, scoring them and applying its principles. Instead of clustering matched records afterward, Senzing resolves each new record into an entity as it arrives, using everything already known about that entity, while building relationships between entities and revisiting past decisions as new data comes in.
Senzing processes each new record in four stages.
1. Receive the record
You map your data to the Senzing JSON format and ingest it. Each record is identified by a DATA_SOURCE code and a RECORD_ID that is unique within that data source, and adding a record with an existing RECORD_ID replaces it. Senzing stores the record and groups its data into features, such as NAME, ADDRESS and PHONE. Some values are standardized, and Senzing also builds combinations, such as name plus date of birth.
2. Identify candidate entities
Senzing derives candidate keys from the new record’s features. Candidate keys are how Senzing finds potential matches. Values shared by too many entities, such as a bank’s toll-free phone number already associated with hundreds of people, go generic and are not used to find candidates. The remaining keys are used to retrieve candidate entities, because entities must find each other through candidate keys before they can be scored.
3. Compare features and apply principles
Senzing compares the features of the new record with those of each candidate entity and scores each comparison. Principles then decide whether the record is the same as, possibly the same as, or possibly related to the candidate. If entities score below the resolution threshold, they may still be related. Each decision has a match key that lists the features that contributed to it, both positively and negatively, and the principle that was satisfied is the actual reason for the match.
4. Update and respond
Senzing updates its resolved entities and relationships with the outcome and, if a response is requested, returns the entities that were created, updated or are no longer valid. Searching for, deleting or updating records follows similar steps.
Principles, not rules
Senzing invented principle-based entity resolution. A principle is a special form of generalized knowledge that draws on common truths or assumptions about the expected behaviors of entity attributes, such as names, addresses and identifiers. For example, social security numbers typically point to only one person, but dates of birth are shared by many people.
Senzing software comes preconfigured with the common attributes and expected behaviors of people and organizations, so you can start loading and resolving entities without configuration, training or tuning. Expected behaviors always have exceptions, and the software learns about them in real time as new data is received. When multiple people use the same SSN, it labels that SSN as generic and reevaluates all prior records with that number.
Entity Centric Learning
Senzing uses Entity Centric Learning to resolve new records with existing entities. Most other entity resolution systems perform record-to-record matching.
Senzing AI learns locally at the entity level. Over time, it learns about common typographical errors, nicknames, alternate email addresses, and even intentionally falsified information. Entity Centric Learning algorithms connect new records to comprehensive entity profiles containing every name, address and date of birth seen across all records.
Real time and sequence neutral
Senzing can ingest and resolve, query, delete and self-correct as data is received in real time. At the same time, it can ingest batch data, such as historical or third-party data. Deleting a record also performs entity resolution in real time.
Senzing also tracks attribute statistics in real time. This lets it detect unexpected patterns, such as widely reused Social Security numbers or garbage values.
Senzing is sequence neutral: it does not matter whether Senzing encounters record A before record B or the reverse. As new information arrives, Senzing re-evaluates earlier assertions and makes adjustments automatically. This prevents the accuracy drift that plagues systems requiring periodic batch reloads to catch up.
Explainability: why and how
Senzing delivers consistent and explainable results. You can see and explain to others the specific details of why a match was made or not made, and Senzing does not use large language models.
Two commands show the reasoning. The why command explains why two entities did not resolve to the same entity, and a why result helps explain why two records are related but not resolved to the same entity. The how command shows the resolution path for an entity. Each step of a how result shows the scores of all compared features, along with the match key and the principle that was satisfied.
Relationships
Resolution also tracks relationships between entities. Disclosed relationships are based on provided data, such as a guarantor on a credit application. Discovered relationships are determined by records that share an address, phone number, email or other attribute. Possibly-same entities stay separate: they scored below the resolution threshold, though they may still be related. An ambiguous match occurs when a record could legitimately belong to more than one entity.