Senzing Entity Specification

The Senzing engine performs Entity Resolution to determine when entities are the same or related within and across data sources.

This specification focuses on entities that are persons or organizations, such as customers, prospects, vendors, employees, and watch lists. It contains a dictionary of pre-configured attributes that are used to resolve and relate persons or companies and outlines the process of creating data sets with them so that they are readily consumable by the Senzing engine. The dictionary also serves to identify what information is desirable to perform Entity Resolution.

Mapping source data

Data must be presented to the engine in either a JSON or CSV file format using the dictionary of registered attributes contained in this specification. The advantage of JSON is that its hierarchical structure allows for multiple names, addresses, phones, etc to be presented in a single structure as one record may have only one address and another may have five. CSV files are flat and multiple values must be presented as additional columns, so if the maximum number of addresses is five, then all rows in the csv file will include space for five addresses, whether needed for that record, or not.

Types of data sources

Senzing entity resolution finds matches within and across data sources. There are two types of data sources you may want to resolve:

1. A master list of entities such as a customer list, employee list, etc. While this is usually a curated list of unique entities, duplicates may be expected and are often found. As well, you may be loading them in order to see if they have any connections to other data sources such as watchlists, or to supplement or verify them with reference data from external data providers.

2. An event or transaction that contains the identifying information about the entity rather than a link to a master record as defined above. For instance, if your registered customer sends or receives money to an unknown external party, those external parties can be resolved into unique entities so their activities can be accumulated and connections can be made to known entities and watch lists.

When processing events or transactions with external parties, it is best to use a hash of the identifying information for a party as its key and only send parties to Senzing for resolution. The event or transaction itself should be stored outside of Senzing with a linking table that connects them to external party keys sent to Senzing for resolution. The transactions can then be joined to Senzing’s resolved entity IDs for aggregation and decision making.

Updating vs replacing records

When information about a master record changes, that record should be re-sent to Senzing for resolution. But was the change an update or a correction? Was the prior address wrong, or did the entity move. Are you even allowed to keep the prior data - likely not on a watch list! These are issues that are normally addressed by the source systems themselves. For this reason, when given the same key (known as a record_id in Senzing), the record is replaced rather than updated.

Therefore, all the attributes of an entity must be presented in a single json document including any historical values kept by the source system. If an entity is comprised of data in several different tables, those tables must be joined together so the entire entity can be presented at once. We use a json format so that lists of child records such as additional or prior names, addresses, and identifiers can be presented as json lists within the single document for the entity.

Desired attributes for a person

These are the likely fields you will run into when mapping persons to the generic entity format. In fact, you should try to map as many of these fields as possible.

• Primary name
• Date of birth and gender
• Passport, driver’s license, social security number, national insurance number
• Home and mailing addresses
• Home and cell phone numbers
• Email and social media handles
• Groups that they are associated with such as their employer name
• Any key dates, statuses or amounts that can help you find meaning in the matches. For instance, a vendor related to an employee who has influence over purchases is more important than the same vendor related to an employee that doesn’t.

Desired attributes for an Organization

These are the likely fields you will run into when mapping organizations to the generic entity format. In fact, you should try to map as many of these fields as possible.

• Primary name
• National Registry numbers
• Tax ID numbers
• Other ID numbers assigned by agencies or data providers
• Primary and mailing addresses
• Primary and other phone numbers
• Website and social media handles
• Any key dates, statuses or amounts that can help you find meaning in the matches. For instance, a current company you do business with who is on a watch list for bad reasons is more important than the same match to a company you did business with several years ago.

JSON files

Senzing uses the JSON Lines (JSONL) format to load records. Each line of a JSONL file contains a single JSON message.

Below is the basic structure of a JSONL record the engine can consume. Note that most attributes are at the root level. However, lists must be used when there are multiple values for the same attributes.

{"DATA_SOURCE":"TEST","RECORD_ID":"1","RECORD_TYPE":"PERSON","PRIMARY_NAME_LAST":"Fletcher","PRIMARY_NAME_FIRST":"Irwin","PRIMARY_NAME_MIDDLE":"Maurice","DATE_OF_BIRTH":"02/15/1937","ADDRESS_LIST":[{"ADDR_TYPE":"HOME","ADDR_FULL":"123 Main Street, Las Vegas NV 89132"},{"ADDR_TYPE":"APARTMENT","ADDR_LINE1":"3 Underhill Way","ADDR_LINE2":"# 7","ADDR_CITY":"Las Vegas","ADDR_STATE":"NV","ADDR_POSTAL_CODE":"89101"}],"PHONE_TYPE":"HOME","PHONE_NUMBER":"702-919-1300","EMAIL_ADDRESS":"babar@work.com"}

{
  "DATA_SOURCE": "TEST",
  "RECORD_ID": "1",
  "RECORD_TYPE": "PERSON",
  "PRIMARY_NAME_LAST": "Fletcher",
  "PRIMARY_NAME_FIRST": "Irwin",
  "PRIMARY_NAME_MIDDLE": "Maurice",
  "DATE_OF_BIRTH": "02/15/1937",
  "ADDRESS_LIST": [
    {
      "ADDR_TYPE": "HOME",
      "ADDR_LINE1": "123 Main Street",
      "ADDR_CITY": "Las Vegas",
      "ADDR_STATE": "NV",
      "ADDR_POSTAL_CODE": "89132"
    },
    {
      "ADDR_TYPE": "MAILING",
      "ADDR_LINE1": "3 Underhill Way",
      "ADDR_LINE2": "#7",
      "ADDR_CITY": "Las Vegas",
      "ADDR_STATE": "NV",
      "ADDR_POSTAL_CODE": "89101"
    }
  ],
  "PHONE_TYPE": "HOME",
  "PHONE_NUMBER": "702-919-1300",
  "EMAIL_ADDRESS": "babar@work.com"
}

CSV files

Mapping csv files is normally accomplished by replacing the column header names with the registered attributes names contained in this specification. Column names in a CSV must be unique, so multiple values for attributes such as multiple names, multiple addresses, etc must be prefixed with a term denoting the type of name, address, etc. Here are the list of column headers that could be used to flatten out the JSON example above …

DATA_SOURCE,RECORD_ID,RECORD_TYPE,PRIMARY_NAME_LAST,PRIMARY_NAME_FIRST,PRIMARY_NAME_MIDDLE,AKA_NAME_LAST,AKA_NAME_FIRST,AKA_NAME_MIDDLE,DATE_OF_BIRTH,SSN_NUMBER,HOME_ADDR_LINE1,HOME_ADDR_LINE2,HOME_ADDR_CITY,HOME_ADDR_STATE,HOME_ADDR_POSTAL_CODE,MAILING_ADDR_LINE1,MAILING_ADDR_LINE2,MAILING_ADDR_CITY,MAILING_ADDR_STATE,MAILING_ADDR_POSTAL_CODE,HOME_PHONE_NUMBER,EMAIL_ADDRESS
TEST,1,PERSON,Fletcher,Irwin,Maurice,,,,02/15/1937,,123 Main Street,,Las Vegas,NV,89132,3 Underhill Way,#7,Las Vegas,NV,89101,702-919-1300,babar@work.com

Dictionary of registered attributes

Attributes for the record key

Senzing is an entity repository that helps locate records for the same entity across data sources. Think of it as a pointer system to where an entity’s records can be found. These are the fields required to tie the records in Senzing back to the contributing sources.

Important notes:

• The DATA_SOURCE default length limit is 25 characters. Please email support@senzing.com if you need to increase the length limit.
• Caution: If you do not supply a record_id, one will be generated based on a hash of the identifying attributes effectively rendering updates impossible. If you do not supply a unique record_id, you should not load the same set of records more than once.

Names of individuals or organizations

A name is a highly desirable feature to map. Most resolution rules will require a matching name.

Important notes:

• The “PRIMARY” NAME_TYPE helps select the best name to display for an entity. See Special attribute types and labels for when to use this. It is best to always specify name type!
• The NAME_FULL attribute is provided if the parsed name fields are unavailable. You would not map both a NAME_FULL and any other name fields in the same name segment.
• If there is a common or nick name field, it represents a “second” name the individual is known by. In this case, map a second set of name columns duplicating the last name with the common name.
• If using NAME_ORG then this record should be about an organization, not an individual i.e., do not map any of the individual name fields. You would not map both a NAME_ORG and any other name fields in the same name segment.
• Sometimes there is both an organization name and a person name on a record, such as a contact list where you have the person and who they work for. In this case, you would map the person’s name as a name and the company as their employer. See Attributes for group associations for more information on this important distinction.

Addresses

Addresses are important, especially when identifiers are not available. One of the more common resolutions will be made on name and address.

Important notes:

• The ADDR_FULL attribute is provided if the parsed address fields are unavailable. You would not map both an ADDR_FULL and any other address fields in the same address segment.
• The “BUSINESS” ADDR_TYPE adds weight to physical business addresses. See Special attribute types and labels for when to use this.

Phone numbers

Like addresses, phone numbers can be important, especially when identifiers are not available. A common resolution will be based on name, phone, and date of birth.

Important notes:

• The “MOBILE” phone type adds weight to mobile phones. See Special attribute types and labels for when to use this.

Physical and other attributes

Physical attributes can like DATE_OF_BIRTH help reduce over matching (false positives). Usually gender and date of birth are available and should be mapped if possible.

Government issued identifiers

Government issued IDs help to confirm or deny matches. The following identifiers should be mapped if available.

Important notes:

• A TRUSTED_ID is a very special identifier that will resolve records together even if they have different names, dobs, or other identifiers. For example, if the SSN of a data source is so trusted it should resolve records despite other differences, it can also be mapped as a

TRUSTED_ID_NUMBER with the TRUSTED_ID_TYPE of “SSN” to resolve within and across data sources that are so trusted.

• A TRUSTED_ID can also be used to manually force records together or apart as described here… https://senzing.zendesk.com/hc/en-us/articles/360023523354-How-to-force-records-togetheror-apart
• Use * OTHER_ID sparingly! It is just a catch all for identifiers you know nothing about but still want to use to help match. Therefore if you know anything about an identifier not listed above, you should add it as its own identifier as described here … How to add a new identifier_._

Identifiers issued by organizations

The following identifiers have been added over time and can also be mapped if available.

Websites, email addresses, and other social handles

The following social media attributes are available.

Group associations

Groups a person belongs to can also be useful for resolving entities. Consider two contact lists that only have name and who they work for as useful attributes.

Important Notes:

• Group associations should not be confused with disclosed relationships described later in this document. Group associations help resolve entities whereas disclosed relationships help relate them.
• If all you have in common between two data sources are name and who they work for, a group association can help resolve the Joe Smiths that work at ABC company together.
• Group associations are subject to generic thresholds to help reduce false positives and keep the system fast. Therefore they will not help resolve all the employees of a large company across data sources. But they could help to resolve the smaller groups of executives, primary contacts, or owners of large companies across data sources.

Disclosed relationships

Some data sources keep track of known relationships between entities, such as familial relationships and company hierarchies. This structure allows you to tell G2 about such relationships. Look for a table within the source system that defines such relationships and include them here.

For instance, if the relationship says that customer 1001 is the SPOUSE of customer 1002, then customer 1001 should be given an anchor domain and key of 1001 and customer 1002 should be given a rel_pointer_domain and key that “points” to back to customer 1001 as its spouse. This is all it takes to create a disclosed relationship.

However, some source systems point each record to the other with a more descriptive role. For instance, if customer 1001 is pointed to customer 1002 as the father and customer 1002 is pointed to customer 1001 as the son then both customers should have a rel_anchor_domain and key for itself as well as a rel_pointer_domain, key and role that relates it to the other.

Example company hierarchy …

Example bi-directional familial relationship …

Values not used for entity resolution

Sometimes it is desirable to include additional attributes that can help determine the importance of a resolution or relationship. These attributes are not used for entity resolution because they are not configured in Senzing. These attributes may include values such as additional dates, statuses, types, flags, or aggregated amounts at the entity level.

For example:

• The LIFETIME_VALUE of a customer can help determine what kind of discount should be applied to their order or if a new customer is related to a high value customer.
• The TERMINATION_REASON of an employee can help determine if a new job applicant should be hired or not.
• The BUSINESS_RISK or GEOGRAPHICAL_RISK of a customer may help determine if a high dollar transaction should be reviewed before it is executed.
• A vendor related to an employee who has influence over purchases is more important than the same vendor related to an employee that doesn’t.
• A current company you do business with who is on a watch list for bad reasons is more important than the same match to a company you did business with several years ago.

On smaller Senzing systems, you may want to include and store additional non-Senzing attributes. On larger Senzing systems, it is best practice to load only the configured attributes used for entity resolution, and use a data warehouse or other external system to access additional non-Senzing attributes.

Special attribute types and labels

Some features have special labels that add weight to them. For instance, you might find a whole family at a “home” address, but only one company (or company facility) at its physical “business” address. The following special labels can be used to augment a feature’s weight …

How to use:

Labels are either used as an attribute prefix such as:

…

“BUSINESS_ADDR_LINE1”: “111 First St “,

“BUSINESS_ADDR_CITY”: “Anytown”,

…

Or by its “type”attribute in a json list such as:

“ADDRESS_LIST”: [{

“ADDR_TYPE”: “BUSINESS”, 

“ADDR_LINE1”: “111 First St”,

“ADDR_CITY”: “Anytown”,

…

Additional configuration

Senzing comes pre-configured with all the features, attributes, and settings you will likely need to begin resolving persons and organizations immediately. The only configuration that really needs to be added is what you named your data sources.

The way you configure Senzing is through the G2ConfigTool.py script located on the python folder in the directory tree for your project. To use it, go to the python folder and type …

G2ConfigTool.py

Then type “help” at the prompt. There is a lot you can do in there, but most of it you should not use unless directed to do so by Senzing support. For instance, adding new rules or adjusting thresholds should not be attempted without first contacting support for guidance.

However, you will often use this tool to add new data sources and sometimes to add new identifiers that are not in our default for configuration.

How to add a data source

Adding a new data source is a simple as registering the code you want to use for it. Most of the reporting you will want to do is based on matches within or across data sources. For instance …

• If you want to know when a customer record matches a watchlist record, you should have a data source named “CUSTOMER” and another one named “WATCHLIST”.
• Or if you are matching two customer data sources to find the overlap, you should have one data source named CUSTOMER1 and another named CUSTOMER2 or to be more descriptive you might name them based on the line of business such as “BANKING-CUSTOMER” and “MORTGAGECUSTOMER”.

To add a new data source named “CUSTOMER”, at the the G2ConfogTool prompt type … addDataSource CUSTOMER