When GraphRAG Goes Bad:

We now start these containers with the commands:

Once they have started, you can navigate your browser to Portainer at localhost:9443 and set up a login and password. This is a useful little UI to see all of your running containers, and we will use this information for when we fire up Senzing. In particular, you should navigate into the container list, which will show you all of the running containers as well their internal IP addresses.

(You could also get this information from the command line through docker inspect , but I just happen to find Portainer handy.)

We will specifically need to pay attention to the IP address of the PostgreSQL container, called pg_container above with an address of 172.20.0.5. (Be sure to record whatever it is for your computer, noting that it will change if you restart the container.) Using this and following along with the Docker Quickstart Guide, we will need to set the following environment variable:

Here you see we have used the IP address for PostgreSQL provided by Portainer along with its password established in the docker-compose.yml. Essentially what we are doing is telling Senzing where to look for PostgreSQL, which it will use through various phases in its ER process.

We are now ready to fire up the Senzing container. To do this, we will initialize the Senzing-PostgreSQL configuration:

Next, we need to configure the Senzing API to do the ER:

This will take you into the running container. There are several Python scripts within this container, but to get things started you will run G2ConfigTool.py (it is located in /opt/senzing/g2/python but is linked such that you don’t have to go into that directory if you don’t want to). Once in the configuration tool, you will add some data sources and adjust some templates. We will keep this consistent with the original article, although we are running it manually. To do so, enter the following commands (each on a separate line):

Excellent! It is now time to get to business and run the ER! We can exit out of that API Docker container for now. We are now going to use a different Docker container to load the 3 files we have downloaded.

I really like this file loader because it handles all of the parallelization for you without having to do anything fancy. On my very modest Linux machine it took under 3 minutes to run. And then you are done!

Now let’s see what we have by hopping back into the API container:

The next tool we will use is G2Explorer.py. This gives you direct access to the ER results. It is pretty elegant because Senzing is just a JSON in and out engine and G2Explorer.py is a handy tool for allowing you to interact with that JSON via the command line. (As you get more proficient with using Senzing for ER you will likely wind up writing your own tooling around working with that JSON, but G2Explorer.pyis an easy way to get some quick views into the results.) I would encourage you to read the docs by typing ? or help at the G2 prompt because there are a ton of things this tool can do. Let’s start with a simple search for an entity called “Desert Springs Landscaping.” Here is a screen shot (I am doing this because the colors of the text, which are hard to maintain on this blogging platform, having meaning):

Search results for Desert Springs Landscaping (Image by author)

It is great that it found a single entity, which is the whole point! Senzing assigns resolved entities an Entity ID but there can be other entities linked to it. For example, let’s explore what those relationships are that it found:

Senzing-identified relationships for Desert Springs Landscaping (Image by author)

We can see in the above that Senzing was able to resolve two records from the PPP Loans and SafeGraph into this single entity at the top. This is great because you can see that there are some slight differences in the names addresses of the two records. You also see a series of entities that are also possibly related to this entity, which can be very helpful. Let’s look at that first one, whose name is similar, but it sounds like maybe the same company has two different arms: one that does landscaping and the other that does pools and spas. We can get information on that entity using its entity ID as shown:

Senzing results for Desert Springs Pools and Spas, Inc.

We can see here another relationship that is potentially interesting: “Disclosed relationships.” Here was have a parent that has been identified for this entity. Let’s dig into that one:

Senzing results for Blue Diamond Crossing (Image by author)

Just looking at the list of disclosed relationships, I see that there are many unrelated businesses. However, if you do a Google search you will find that Blue Diamond Crossing is a very large strip mall where all of these businesses are located. So far, this is looking pretty promising to me! I would encourage you to poke around in G2Explorer.py because there are a lot of great insights to be had already!

What I am going to show now is how we can bring the ER results outside of the container and back to our local machine. In general, this is just for demonstration purposes. If you were running a proper production ER system you would have too much data to do this and would then interact with it strictly from Python. So we are just going to do this for the sake of this article.

We need to export the ER results to a file and then copy that file to our local system. There are a zillion data features (a full list can be found here) we could chose to export and for demonstration purposes I am going to get most of them. Again, this is not something we would do in production, but here is how you do it:

Now we can copy the file entities.json to our local machine:

(Note that e0678f39601f is the container ID of my senzing/senzingapi-tools container that I obtained via docker ps -a . You will need to use your actual container value.)

A RAG based on the tabular data

Before we look at whether the resolved data helps the LLM, we should start by asking questions of the non-resolved data. Remember that this data is tabular. You could try to do things to establish relationships among this data to create some sort of graph, but these relationships will be artificial at best and likely miss many things. Instead, I opted to use a quickie SQLite database that I made from Pandas with the following code to ingest it:

Note that for brevity I elected not to get every single different violation type from the DOL_WHISARD data set and just am using the total number of violations.

Now it is time to create an LLM and start asking some questions. For the sake of this post I am not going to get into prompt engineering too much and have just elected to do something very basic. If you were truly creating a production LLM system you would spend a lot of time on the prompt and do serious testing of it such as with LangSmith. For demonstration purposes, I want to make sure that we are not worrying whether the prompt itself is what is generating good or bad results.

Next we need to establish the SQL agent in LangChain and connect it to the database:

Now that we have the connection, we can create an SQL agent for asking questions of the database:

Now we are cooking! I may never write SQL again!

Alright, so let’s start asking actual questions of the data. Given this particular data set, I might care that PPP loans are not given to companies with a large number of DOL violations. How might we get at that information? Let’s start by picking Union Cabs (I am going to start truncating the LangChain verbose output a bit, but the full output is in the notebooks in the GitHub repo):

Uh oh…Houston, we have a problem! According to these basic SQL queries, Union Cabs (an entry in the SafeGraph data) doesn’t have any violations. But if we look at the DOL data there is an entry called “Union Cab” (singular) that actually DOES have violations. Further, if you go into G2Explorer.py you will see that there are several other related cab companies with the same address like LV Cabs and Vegas Western Cabs who do have additional violations. (Once we actually assemble the graph of this data below I will show you a visualization of this and it will make more sense.) So if you were wanting to get an accurate accounting of those violations (or many other questions you could think of to ask this data), you will miss it because you have not resolved the relationships between each of these (seemingly unrelated) cab companies!

Clearly it is going to help us get better answers by having entity-resolved data. This is where the entities.json file is going to come into play. Let’s explore what it looks like before we use it to create a graph and ask some questions. There are a ton of things you actually can have Senzing write out in JSON like this and they are described in these docs. Here is a sample entry from the file I created via exportJSONEntityReport above:

Here we have two keys for each entry: RESOLVED_ENTITY and RELATED_ENTITY. The former is our starting point. Basically, these are “anchors” in our data showing what Senzing has identified as being the true entity for other entities. It is the main result of the ER process and can be used for merging entities. Related entities, on the other hand, are entities that have a potentially meaningful connection to other entities but may not merge entities. For example, two businesses with same address (think a Starbucks within a Target store) would be related even though they might not be the same entity.

If you explore this JSON file you will see that every entry in the JSON has a resolved entity. There may be multiple records that resolve to a single entity. There are also potentially several related entities associated with each resolved entity. Our job is to turn this into a graph that we can use to ask questions of.

To create the graph, I used both the original tabular JSON data as well as the Senzing data, the later being used to establish relationships between the tabular data. I chose this model because it would contain the most amount of text that the LLM can use for the RAG. I used this data to populate a Neo4j database that is running from the original Docker container. You can access it in your web browser at locahost:7474 with the login neo4j and the password graphs_@re_c00l .

For the sake of brevity I will not include the code to populate the graph here. However, you can see how I populated the graph from the original tabular data here and the Senzing data here.

In this code I have used the Neo4j Python driver to establish the connection to the database and populate the graph in batches from Pandas dataframes like I have done in previous blog posts.

As for the graph model, I have created 3 node types based off of the tabular data: SGEntity, DOLEntity , PPPEntity . Essentially, each row in each data set corresponds to a node. I have also created a node called Entity , which correspond to the resolved entities by Senzing. Then there are the relationships RESOLVES and RELATED_TO , which correspond to those keys within entities.json.

Once you run the code in both of these notebooks, you will have a graph of approximately 86,000 nodes and 8600 relationships. The graph model (provided by CALL db.schema.visualization ) should look like this:

Knowledge Graphs Model

At this point, it is important to stop and talk about this a bit. If you have read other posts about Senzing, including Paco Nathan’s original article, you will note that this graph model looks a bit different. When we talk about ER, we think of entities as groups of records. Therefore, we would expect a more hub-and-spoke model with entities in the center and record nodes pointing at them. While this might be beneficial for a variety of reasons, one key reason this is great is that you are creating an entity and not altering any of the records used to create it. So you have the ability to go back and forth easily between unaltered records and resolved entities.

What we have above is more complicated by that. I decided to make the graph model be a bit different for the sake of working with LLMs. I was hoping (although you will see later that this was perhaps a proverbial pipe dream!) that having more relationships in the graph would allow the LLM to move about the graph easier or more accurately. In reality, there is not necessarily a reason to do it this way or use the traditional hub-and-spoke model. I would encourage you to try both models and see which performs better for you.

At any rate, let’s see what it looks like for Union Cabs:

Graph Around Union Cabs, obtained with the query

If you compare this to what we see in G2Explorer.py by searching for Union Cabs, it makes sense. If you explore this subgraph, you will see that the related nodes are all related based on addresses. This is likely due to a corporate office having the same address managing multiple different cab companies.

Similar to how we created a SQL agent with LangChain, we are going to create a Cypher QA Chain for asking questions of our ERKG. We first establish a connection from LangChain to Neo4j like this:

Now, we create the chain:

Great. Time to start asking questions!

Alright, there are some really interesting things to talk about here!

First, check out that Cypher query that the LLM generated. In all of my experience tinkering with this I observed that GraphCypherQAChain really wanted to be searching Entity nodes. It was incredibly rare for it to search anything else. It was also very rare for it to do any sort of string similarity akin to WHERE BUSINESS_NAME_ORG LIKE ‘%Union Cabs%’ in SQL. I found this to be a frustrating limitation of GraphCypherQAChain, but one that could ideally be handled if I did some prompt engineering or tinkered with some of the hyperparameters associated with the function (both beyond the scope of this blog post). After experimenting with this a bit, I knew that in order to get the result above I needed to be sure to show this result for an Entity node with the exact name (see graph visualization above). The performance of assembling the Cypher queries will likely get better over time and should not stop us from realizing the full importance of using ER in a knowledge graph.

Next, let’s look at the answer. Here I was a little heartened. It was great that the LLM was able to pick up on Vegas Western Cab and A NLV Cab Co. as being related to Union Cab Co. I also appreciated how much information the LLM brought into the output.

Let’s now take this a step further and ask questions specifically designed to get at nodes that are related to the starting Entity node:

Well, that is exciting! According to the graph, there is a single DOL entry called “Union Cab” and it has a single violation associated with it. However, because we were able to use ER and resolve the entire subgraph, the LLM was able to see that there were 4 entries within DOL_WHISARD that had a total of 6 violations associated with this grouping of cab companies!

I do not want to get you too excited though. I can tell you that I ran this particular block of code several times and received a variety of different answers, including the sum total of all violations within the database. Oh, the joys of working with LLMs!

As I was exploring around in the tabular data, I asked a question that I got an interesting answer to:

So there is this company called HSG, LLC with big loans and a fair few violations. But who are they? As I started probing, I found some interesting things. They are present in both the PPP_LOANS and DOL_WHISARD data with slight variations on the address: 4001 S. Decatur Blvd. #37–376, Las Vegas, NV, 89103 and 4001 South Decatur Blvd, Las Vegas, NV, 89103 respectively. Pulling the thread on that a bit, I found that 4001 S. Decatur Blvd is actually a strip mall and Suite #37 is a UPS Store. So this is likely a PO Box in the UPS Store that is used as the registered address for this business.

However, things got somewhat more interesting. I noted that these two entities have totally different NAICS codes. NAICS is the North American Industry Classification System, which provides an integer value to represent the type of business. In one instance HSG, LLC has an NAICS code of 541620, which corresponds to “Environmental Consulting Services.” In the other case it has a code of 621330, which corresponds to “Mental Health Practitioners.” Curiouser and curiouser.

So I took this to the knowledge graph. Below is the subgraph I identified for all entities with “HSG” in the name:

Resolved subgraph for HSG

Interesting! We have a new node here that I wasn’t expecting, which is a company called Integrated Mission Support Services, LLC. Senzing identified this business as “possibly related” to HSG, LLC based on the address. Maybe that is correct, maybe it isn’t. However, I noted that it also has an NAICS code of 541620, “Environmental Consulting Services” and has the same address of the environmental consultant HSG, LLC. Perhaps they are the same entity. But then what is the mental health HSG doing here? Maybe someone has a typo in their entry in the data? Something else? Please add comments to this article if you have theories we might test!

As I was exploring around in the tabular data, I asked a question that I got an interesting answer to:

I hope after this post you have gotten a feel for the importance of entity resolution when working with knowledge graphs, particularly as they are using in RAG applications. I have shown an introduction on how to work with Senzing to resolve some real-world data and ask questions of it in a basic LLM+RAG setting. It is important to realize that many of the shortcomings of the results here were due to the GraphCypherQAChain package making some simplistic assumptions about the graph like assigning all query entities the node type :Entity (perhaps fixable with hyperparameters) and only looking out 2 hops from the query node. As work progresses on these packages these results will likely improve. It is also likely that this would be improved by spending time doing prompt engineering rather than just asking these basic questions. That can be the subject for a future post.

What I hope you have come away with is the importance of ER on data science, whether it is applied to LLMs with RAG or just in general data science. To anyone working in the field it is clear that if you have better data you will get better answers. What I mean by this is that if I know I have a table of many rows and am able to know that some of those rows are really the same entity, it is obvious that we will get better data science answers in the long run, whether they involve LLMs or are based on more traditional data science needs.

I was really impressed by how easy it was to do ER with Senzing as well as the quality of the results. It is worth noting though that this really is a commercial-grade ER engine. We are dealing with an incredibly small data set for this post — less than 100k records. Really Senzing is designed and optimized to work with much larger data than this like several millions of records and, no joke, billions and billions too. Plus, it is designed to handle all of these records in real time. If you were using it in production you would not interact with it like I have shown here. This was just for demonstration and experimentation purposes. However, please reach out if you wind up creating more sophisticated data sets and applications with it, particularly with LLMs!

As I was exploring around in the tabular data, I asked a question that I got an interesting answer to:

If you’d like to know more about Senzing entity resolution:

Consult with an Expert – Schedule a call with a Senzing entity resolution expert to discuss your requirements.

Try it Yourself – There are three easy ways to take Senzing entity resolution for a test drive: a simple desktop evaluation tool (for Windows or Mac) and QuickStarts for Linux and Docker. You can install the software, load data and evaluate results in as little as 15 minutes.

When you see how Senzing Entity Resolution AI works, you’ll understand why we provide the most accurate and up-to-date results (that are continually updated in real-time).