Model Testing - 30 minutes

Parasol’s developers are always expected to thoroughly test their code. In this exercise, you will explore the unique challenges and techniques involved in testing AI models and applications, which differ significantly from traditional enterprise applications. Traditional apps often have deterministic outputs, whereas AI models can produce varied and context-dependent results, requiring specialized testing approaches. A number of tools are being developed to help enable model testing for example promptfoo, deepeval, and deepchecks. In this module we will use one of these tools "promptfoo" to test the consistency of model responses.

In this lab we’re going to use a trained model which has been fine tuned with Parasol Insurance specific data. We’re going to write some tests which will use predefined prompts which should return accurate responses demonstrating the trained model.

To test the responses to ensure they are accurate we’re going to use another model (an embedding model) to test for similarity between two sentences.

Red Hat Developer Hub (RHDH) is an enterprise-grade, self-managed, and customizable developer portal built on top of Backstage.io. It’s designed to streamline the development process for organizations, specifically those using Red Hat products and services. Here’s a breakdown of its key benefits:

Red Hat Developer Hub has already been installed and configured in the workshop environment.

You have just created the Parasol application with Red Hat Developer Hub. This application is used by Parasol Customer Service Representatives to enter, organize, and update existing insurance claims for its customers. An AI-powered chatbot is included for reps to use to answer questions about the claims. This chatbot is driven by an LLM that has been fine-tuned with Parasol Insurance private data, including corporate policies around insurance claims.

Go back to the Parasol component in Red Hat Developer Hub. From the OVERVIEW tab click on OpenShift Dev Spaces (VS Code) to make the necessary source code changes.

Click on Log into with OpenShift.

Log into with the following OpenShift credential on the Red Hat Single Sign-On (RH-SSO) page.

Authorize devspaces to use your account by clicking on the Authorize button.

Wait for your Red Hat OpenShift Dev Spaces workspace to be ready. This can take a few minutes.

You will see a loading screen while the workspace is being provisioned, where Dev Spaces is creating a workspace based on a devfile stored in the source code repository, which can be customized to include your tools and configuration.

NOTE: In case your workspace fails to start, you can click on close running workspace and restart to try again.

Confirm the access by clicking "Yes, I trust the authors".

Then you should see:

As we mentioned in the goals section, we’re going to use two models to perform these tests. The configuration for these models can be found in .vscode/settings.json. Open this file and examine it’s contents, it should look like this:

There are a three things to take note of in this JSON file.

We have included an initial set of tests which should pass when run against the Parasol Chat model.

For each of these tests we’re provided an expected response, so for example, when we ask the question: "In one sentence, who founded Parasol Insurance?" we should get a response similar to: "Parasol Insurance was founded in 1936 by James Falkner and James Labocki."

This test is described using:

If you take a look at the code above, you can see we are calling the model with the Asynchronous function, callModel.

We are then testing the response from the model to ensure it matches semantic similarity a given sample response using toMatchSemanticSimilarity.

The semantic similarity between the response and the example response needs to be above 0.9 (with a similiarity of 1 being an exact match)

Before we get started running these tests, we need to install the required NPM modules, to do this, follow these instructions.

Open a Terminal window in your OpenShift Dev Spaces workspace.

Enter the following commands:

The output should be similar to the following.

We can now run the tests to ensure they pass.

Open the prompt-testing/index.test.js file in the VSCode Explorer.

Then, click on the Testing icon On the left hand panel.

Click on the Run Tests icon at the top as shown below.

When the test run is complete you should see.

Both of the tests should have passed successfully.

Let’s try a scenario where we’re using a model which has not been trained with Parasol Insurance specific data.

As we discussed above, one of our tests is testing responses to the question: "Who founded Parasol Insurance?".

To answer this question our model has been trained with knowledge about the history of Parasol Insurance.

Let’s change the model our tests are performed against to an untrained model (merlinite-7b-lab). To do this, open the .vscode/settings.json file to replace the contents with:

Go back to the Testing panel. Then, click on the Run Tests to run the tests again.

You should now see one of the tests failing.

We’re now going to add a new test to test some new functionality we’ve added to our model.

Our model has recently been trained to understand the Apex plus policy option which will cover drivers for loss of income even if they are at fault.

To test this we’re going to ask our model:

We expect the response from this question to be similar to the following, mentioning the Apex plus policy.

To create this test, open the index.test.js file. Then, add the following code below this comment, // ADD NEW TEST HERE.

Let’s test the model again by clicking on the Run Tests button.

When the test run is complete you should see.

We hope you have enjoyed this module!

Here is a quick summary of what we have learned: