Senzing v4 SDK Docker Quickstart Guide

Use Senzing v4 with the example Docker images quickly.

Depending on the speed of your internet connection, this may only take you a few minutes.

Senzing provides a free license for 500 source records when downloading the Senzing® SDK. If you require additional records for an evaluation, or any assistance when following this guide, contact Senzing Support . Support is 100% FREE!

Senzing® Smarter Entity Resolution™ SDK with Entity Centric Learning™ technology.
Connect Data. Power Intelligence.™ entity resolution.

Prerequisites

  • A Docker environment for running linux/amd64 containers.

    To run the docker commands without sudo your user should be a member of the docker group

  • If your userid is not a member of the docker group and you use sudo to run the docker command, you will need to add --preserve-env after sudo
sudo --preserve-env docker run ...

Getting it Done

Setting up - SENZING_ENGINE_CONFIGURATION_JSON

The Senzing engine configuration is set via the SENZING_ENGINE_CONFIGURATION_JSON environment variable. This is further described here .

To set the environment variable, run the following in your environment, replacing the CONNECTION details with those of your database:

export SENZING_ENGINE_CONFIGURATION_JSON='{"PIPELINE":{"CONFIGPATH":"/etc/opt/senzing","RESOURCEPATH":"/opt/senzing/er/resources","SUPPORTPATH":"/opt/senzing/data"},"SQL":{"CONNECTION":"postgresql://username:[email protected]:5432:database"}}'

Formatted for readability:

export SENZING_ENGINE_CONFIGURATION_JSON='{
 "PIPELINE" : {
   "CONFIGPATH" : "/etc/opt/senzing",
   "RESOURCEPATH" : "/opt/senzing/er/resources",
   "SUPPORTPATH" : "/opt/senzing/data",
 },
 "SQL" : {
   "CONNECTION" : "postgresql://username:[email protected]:5432:database"}
 }'

If you have PostgreSQL installed on localhost (127.0.0.1), you must use either:

Initialize the Senzing repository database

It takes only a few seconds to initialize the Senzing repository database using Docker. Once SENZING_ENGINE_CONFIGURATION_JSON is set, the setup is a single Docker command that uses the senzing/init-database image.

Initialization first adds the Senzing schema and then the Senzing Entity Resolution Configuration to the database.

This initialization step only needs to be run once!
docker run --rm --env SENZING_ENGINE_CONFIGURATION_JSON senzing/init-database --install-senzing-er-configuration

DONE - Yes, REALLY

At this point, the Senzing database is initialized. You can run any of the Senzing v4 Docker images, as well as any senzingsdk-runtime based custom Docker images you create yourself by following this pattern. Yes, REALLY.

Run the senzingsdk-tools container

Use the senzing/senzingsdk-tools Docker image to run the senzingsdk-tools container:

docker run --rm -it -e SENZING_ENGINE_CONFIGURATION_JSON senzing/senzingsdk-tools
You can now run the sz_command, sz_configtool, or sz_explorer commands inside the Docker container.

Load the Truth Set Data

To get started with some data, load the Senzing example truth set by:

Understanding the Truth Set Files

The truth set demo includes three main types of files, each serving a distinct purpose in entity resolution:

Download the files

Inside the senzingsdk-tools Docker container, run:

wget https://raw.githubusercontent.com/Senzing/truth-sets/main/truthsets/demo/customers.jsonl

wget https://raw.githubusercontent.com/Senzing/truth-sets/main/truthsets/demo/reference.jsonl

wget https://raw.githubusercontent.com/Senzing/truth-sets/main/truthsets/demo/watchlist.jsonl

Add the data source

docker run --rm -it -e SENZING_ENGINE_CONFIGURATION_JSON senzing/senzingsdk-tools

sz_configtool

Type help or ? for help

addDataSource CUSTOMERS

Data source successfully added!

addDataSource REFERENCE

Data source successfully added!

addDataSource WATCHLIST

Data source successfully added!

save

WARNING: This will immediately update the current configuration in the Senzing repository with the current configuration!

Are you certain you wish to proceed and save changes? (y/n)

y

Configuration changes saved

quit

Load the files

Keep in mind that the file path is from the perspective of the Docker container and this example script requires the location of the files ${PWD} in this case) to be mapped into /data inside the container.

docker run -it --rm -u $UID -v ${PWD}:/data --env SENZING_ENGINE_CONFIGURATION_JSON senzing/sz-file-loader -f /data/customers.jsonl

docker run -it --rm -u $UID -v ${PWD}:/data --env SENZING_ENGINE_CONFIGURATION_JSON senzing/sz-file-loader -f /data/reference.jsonl

docker run -it --rm -u $UID -v ${PWD}:/data --env SENZING_ENGINE_CONFIGURATION_JSON senzing/sz-file-loader -f /data/watchlist.jsonl

Explore the results

See EDA Tools: Basic Exploration 

docker run --rm -it -e SENZING_ENGINE_CONFIGURATION_JSON senzing/senzingsdk-tools

sz_explorer


  ____|  __ \     \
  __|    |   |   _ \   Senzing
  |      |   |  ___ \  Exploratory Data Analysis
 _____| ____/ _/    _\


Type help or ? to list commands.

(szeda) get CUSTOMERS 1070

Entity summary for entity 98: Jie Wang
┼───────────┼────────────────────────────────────────┼─────────────────┼
│ Sources   │ Features                               │ Additional Data │
┼───────────┼────────────────────────────────────────┼─────────────────┼
│ CUSTOMERS │ NAME: Jie Wang (PRIMARY)               │ AMOUNT: 100     │
│ 1069      │ NAME: 王杰 (NATIVE)                    │ AMOUNT: 200     │
│ 1070      │ DOB: 9/14/93                           │ DATE: 1/26/18   │
│           │ GENDER: Male                           │ DATE: 1/27/18   │
│           │ GENDER: M                              │ STATUS: Active  │
│           │ ADDRESS: 12 Constitution Street (HOME) │                 │
│           │ NATIONAL_ID: 832721 Hong Kong          │                 │
│           │ NATIONAL_ID: 832721                    │                 │
│           │ RECORD_TYPE: PERSON                    │                 │
┼───────────┼────────────────────────────────────────┼─────────────────┼
│ REFERENCE │ NAME: Wang Jie (PRIMARY)               │ CATEGORY: Owner │
│ 2013      │ DOB: 1993-09-14                        │ STATUS: Current │
│           │ RECORD_TYPE: PERSON                    │                 │
│           │ REL_POINTER: 2011 (OWNS 60%)           │                 │
┼───────────┼────────────────────────────────────────┼─────────────────┼
└── Disclosed relation (1)
    └── --> OWNS 60% (1)
        └── 182: Hajah Mamunah Jln Pisang CUSTOMERS (1) | REFERENCE (1) +REL_POINTER(OWNS 60%:)

Mapping Your Own Data

At this point you are ready to map and load your own data. Mapping is the process of converting your source data into a structure Senzing understands ready to load.

To learn more about mapping, the dictionary of terms and samples to help prepare your own data sources for loading and entity resolving review the Senzing Entity Specification .

Consider these examples, in your data an attribute describing a personal full name is in a database table with the column name fullname. In Senzing a full name is represented by the term NAME_FULL. Similarly for address line 1, your database column is named addressline1, in Senzing this is represented by the term ADDR_LINE1.

Your task in mapping is to determine which attributes in your data source(s) are appropriate for use in entity resolution, extract those attributes and construct the structure describing those attributes to send to Senzing. The following is an example of a Senzing mapped JSON structure for an entry from a data source.

{
  "DATA_SOURCE": "CUSTOMERS",
  "RECORD_ID": "1001",
  "RECORD_TYPE": "PERSON",
  "PRIMARY_NAME_LAST": "Smith",
  "PRIMARY_NAME_FIRST": "Robert",
  "DATE_OF_BIRTH": "12/11/1978",
  "ADDR_TYPE": "MAILING",
  "ADDR_LINE1": "123 Main Street, Las Vegas NV 89132",
  "PHONE_TYPE": "HOME",
  "PHONE_NUMBER": "702-919-1300",
  "EMAIL_ADDRESS": "[email protected]"
}

Start developing

Members of our team have created GitHub projects that show more of what you can do quickly:

If you have any questions, contact Senzing Support . Support is 100% FREE!