Pythia: A Chatbot for your Organisation in 5 Steps!

Here at ThetaPhi we tend to deal with multiple technologies/platforms spanning both IT and OT supplied by various companies with reams and reams of user manuals and operational instructions and then there is all the documentation for our own projects. So if an engineer wants to find a specific piece of information it can often be quite tedious to dig through various documents to find one line of text that contains the required information. So we decided to leverage the power of ChatGPT to build a company specific chatbot. The requirements for the project were:

1. Should be able to ingest pdf documents (for now)

2. Ability create separate document categories to avoid information contamination from overlapping piece of information .e.g. if you want information related to a particular project you should be able to filter for information only from a specified repository of documents

3. All the information should be stored locally including any vectorized databases but the application should be available via the internet over a secure and protected connection.

4. Should be easy to deploy / update

With these requirements in mind we decided to leverage streamlit to deploy a lightweight chatbot application using ChromaDB as the vector store. The result; a homespun chatbot we call Pythia (the oracle of Delphi). See it action here.

If you like what you see and would like to use it for your organisation (or personal use), you are in luck because we have made the code publicly accessible through our github repo.

Keeping with our motto of sharing knowledge, in this post I will go over how to setup it up using docker in 5 steps. If you do not have Docker follow the steps here to install docker for your OS. You will also need to install Docker Compose plugin.

Step 1: Let’s begin by cloning the repository. SSH into the system that you want to run the application on and go to the folder where you would like to download the repository (I have a folder for all my docker deployments) and run the following command:

git clone https://github.com/Theta-Phi/Pythia.git

cd into the cloned repo directory. Let’s take a look at the files in the repository root and src folder:

Step 2: In the repository root you’ve got and example docker compose file that defines and creates the services to run. We will need to modify this to suit our application:

You can see that the application needs access to a ChromaDB (connected over a docker network call chroma_net) server. The application creates a Python client to connect to the ChromaDB server over Port 8000. chroma_net is a bridged network so if you have other services running on the machine that use that port you will have to change the port for the ChromaDB server. Alternatively you can choose to not bridge the port for the ChromaDB server. You can also update the port to be used by the application. It is accessible over 8091 by default. I will update the host port for ChromDB server as I have another service running on that port. I will rename the docker compose file to docker-compose.yml. My updated file looks like this (don't forget to add your OpenAI API Key):

Step 3: The application uses streamlit authenticator for access to the application. The user details and passwords are saved in the ‘credentials.yml’ file in the /src/static folder. As part of the docker image build the passwords are hashed so readable passwords are not available in the deployed application.

Create a copy of 'credentials_example.yml' file and rename it to: "credentials.yml" (the name is import). Update the ''credentials.yml" file to include the user and password details to use:

Keep the file safe and do not share for obvious reasons.

Step 4: With that done we are ready to deploy. Navigate to the root of the repository (where docker-compose.yml file is located) and run the following command:

sudo docker-compose up --build -d

Quick Note: You cannot spin up a docker container over ssh so you will need access to the machine. I will just remote desktop in and run the command:

Once the containers are deployed you can check the status:

docker ps -a

Step 5: With the containers running let see see if we can access the application. Open the browser and go to the <ipaddress/hostname>:<port> of the server. We are greeted by the login screen:

After logging in, you can create a new collection, upload the documents to the collection and start chatting with you documents. Yup it is that easy!

The documents that are uploaded are saved to the local repository (collection) folder. We find this to be a good way to create a central repository of documents as well, so rather than having to save the documents we can just upload the documents to the chatbot.

Now to make the application securely accessible over the internet we added it to our Cloudflare tunnel with zero trust access model. We will be sharing a step by step guide on how to do it on the ThetaPhi blog next week, so stay tuned.

So how does this application really work? There is a lot of information available on the internet about chatbots for documents so I will only briefly outline the process here.

When a document is uploaded the text is broken up into chunks of text and encoded using OpenAI embeddings. The encoding (embeddings) is basically a way of converting each word into a semantic representation using the ChatGPT large language model (i.e. how is the word related to other words in the English language). These embeddings are stored in the Chroma database. When a user asks a question, the question is also converted into embeddings as well and these embeddings are used to retrieve relevant pieces of texts from the ChromaDB embeddings (which came from the documents). Now that we have the question and the relevant piece of information (for context) we can send the question along with the context to ChatGPT to give us a comprehensive answer. Every question and response is also stored in a chat history buffer and this chat history is also embedded and sent as part of the context so follow up questions can be asked.

This is the same as posting the following to ChatGPT:

Given the information provided in the pieces of text and the chat history in the below context tell me <Question here>: 

Context: 
<chunk of text 1> 
<chunk of text 2> 
<chunk of text 3>
.
.

Chat history:
<Question 1>
<Answer 1>
<Question 2>
<Answer 2>
.
.

I know I have not done the topic justice with such a brief explanation but hopefully it gives you the general idea.

I hope you found this post to be informative and helpful. Let us know if you decide to deploy Pythia for yourself and how you like it.

Word of caution: This is an inhouse tool with little development effort put in for aesthetics / user friendliness as it is not meant to be consumer facing. We intend to keep developing it as needs emerge. Feel free to contribute to the repo if you like.