Test and Monitor AI Conversations

Before a change to an AI profile goes live you can test it against simulated customers. Once it is live you can send part of your traffic to it, score real conversations for quality, compare the results in a report, and be alerted when a number slips. If the change makes things worse, restore the version before it.

Menu	Artificial Intelligence > AI Profiles, Design > Workflows, and Reports
Permission	Manage AI profiles, Manage workflows, and View AI chat analytics for the reports
Feature	AI Services and Workflows. The reports need Reports, and the conversation metrics need AI Chat Session Analytics.

Can't find it?

What you see depends on your role. If the menu, a button or a setting on this page is missing, ask your administrator to give you the permission listed above or to turn on the feature. Only administrators can turn features on and change site settings.

Profile versions

Every time an AI profile is saved, the profile as it was saved is kept as a new version, numbered from 1. Each conversation records the version of the profile that answered it.

To see them, open Artificial Intelligence > AI Profiles, open the profile's actions menu and click Versions.

• Compare shows a version beside the profile as it is now, or beside another version: pick it in Compare with. Every field and setting is listed, grouped like the tabs of the profile editor: General, Instructions (with prompt templates and speaking styles), Parameters, Capabilities (tools, tool instances, agents, knowledge), Prompt Security (with guardrails) and Data Processing & Metrics (extraction, analysis, contact memory, privacy). Templates are shown by their titles. Changed settings are highlighted, and Only differences hides the rest.
• Restore gives the profile that version's settings back. The restore is saved as a new version, so the version you replaced is kept too.

The profile's name, its technical name and its owner are not changed by a restore.

Test a change with simulated customers

The Simulate AI Conversation task (under AI Conversations) plays an AI profile against a customer played by a model, then scores the conversation.

1. In Design > Workflows, create a workflow. Start it with whatever should run the tests: a Timer, an HTTP Request Event your release process calls, or nothing at all if you start it by hand.
2. Add Simulate AI Conversation and fill in:

Field	What to write
Profile	The AI profile to test.
Customer	Who the customer is, what they want and how they behave. For example: You are Dana. You want to move tomorrow's 3 pm cleaning to a morning next week. You are in a hurry.
Criteria	One condition the conversation must meet per line, such as The assistant offered a morning slot next week.
Variables	The conversation variables the test starts with, if the profile's instructions use any.
Most customer turns	Where the test stops if the customer has not finished.
Passing score	The lowest score out of 100 that passes.
Label	A name for the test run, such as Release tests, so you can find the results in a report.
Customer model	The deployment that plays the customer. Empty uses the utility deployment.
Judge model	The deployment that scores the conversation. Empty uses the utility deployment.

3. Connect Failed and Error to a Notify task or an email.

The task runs the conversation in text, with the profile's instructions, knowledge and tools. The profile answers with its own deployment, so up to three models take part: the profile's, the customer's and the judge's. A customer and a judge that are not the profile's own model give a fairer test. Tools that run a workflow get Workflow.Input.Simulation set to true, so the workflow can answer with test data instead of booking, charging or texting for real. Add an If/Else on {{ Workflow.Input.Simulation }} at the start of each tool workflow that acts on the outside world.

The transcript is in Workflow.Output.Transcript and the score in Workflow.Output.Evaluation.

Improve the instructions automatically

To have the AI rewrite a profile's instructions until simulated customers pass, use Simulate in the Profile Builder.

Run a batch of tests

Put the test cases in a For Each loop: a list of customers, each with their own criteria, and use the loop's current item, {{ Workflow.Output.Current }} unless you renamed it, in Customer and Criteria. Every result is kept, so the Conversation evaluations report shows the pass rate of the whole run.

Keep a catalog of tests

With the AI Quality feature on, Artificial Intelligence > Tests keeps your simulation tests, so you can run them again after every change instead of building a workflow for each.

A test case has:

• the AI profile to test, and optionally a Version of it;
• the Simulated customer: who they are, what they want and how they behave;
• Success criteria, one per line, and the Passing score;
• Conversation variables, one name: value per line, which the AI sees as it would see real ones;
• Mocked tools, one tool name: result per line: the named tool returns that result instead of running, so a test books, charges and texts nothing for real;
• the Most customer turns, and the models that play the customer and judge.

To run tests, click Run on a test, or select several and choose Run or Run 5 times under the list's actions. Runs are queued and start within a minute; the Runs page shows each one as it finishes, with the pass rate of the batch. Open a run to read the conversation and each criterion, met or not, with the reason.

To run the tests from a build pipeline, turn on AI Chat WebAPI too and call the API with a token from the site's OpenID server: GET /api/ai/v1/tests lists the tests, POST /api/ai/v1/tests/batches with { "profileId": "...", "repeats": 3 } (or testCaseIds) queues a batch, and GET /api/ai/v1/tests/batches/{batchId} returns each run and the pass rate, with completed set once every run has finished. The client's role needs the Manage AI tests, QA cohorts and alert rules permission.

To make a test from a real conversation, open Artificial Intelligence > AI Chat sessions, and choose Create test in a conversation's menu. The test plays that conversation's customer again, with its variables, so you can change the profile until the conversation goes the way it should, and keep the test to check it stays that way.

Compare two profiles on live traffic

The Split Traffic task sends each run of a workflow down one of several outcomes, in proportion to their weights. Combined with Use AI Profile in an AI Conversation Starting workflow, it answers part of your conversations with a different profile.

1. Copy the AI profile and make your change on the copy.
2. Create a workflow that starts with AI Conversation Starting, filtered to the original profile.
3. Add Split Traffic with the outcomes Current: 50 and New: 50. In Keep the same outcome for, write {{ Workflow.Input.CustomerAddress }} so a customer who comes back gets the same profile.
4. Connect New to Use AI Profile with the copy. Connect both outcomes to Set Conversation Variables with variant: {{ Workflow.Properties.Variant }}, so the variant is kept on the conversation.

Compare the two in the reports by profile, or by the variant variable.

Compare two versions of one profile

Every save of an AI profile is a version. To compare the version you just saved with an earlier one, without copying the profile yourself, set Version on Use AI Profile:

1. Open the profile's Versions page and note the number of the version to compare against, for example 3.
2. In the starting workflow, connect one Split Traffic outcome to Use AI Profile with the same profile and Version 3, and leave the other outcome alone, so it answers with the profile as it is now.

The first time version 3 is used, it is copied to a profile named after it, such as Support (version 3), with a Version 3 copy badge in the AI Profiles list. Later conversations reuse the copy. Conversations on the copy report as version 3 of the original profile, so the Profile version column of the AI chat sessions and Conversation evaluations reports compares the two directly. Don't edit the copy: delete it, and the next conversation that needs version 3 makes it again.

Score conversations for quality

The Evaluate AI Conversation task scores a finished conversation with the utility model. It reads the transcript and the profile's instructions and gives:

• a score out of 100;
• each of your criteria, met or not, with the reason;
• whether the customer's need was resolved, and their sentiment at the end;
• the issues found, each with the line of the transcript that shows it: unresolved, an answer with no source, a failed tool, broken instructions, tone, repetition, a transfer that went wrong.

The conversation passes when the score reaches the passing score and every criterion is met.

To score a sample of your conversations:

1. Create a workflow that starts with AI Chat Session Closed.
2. Add Split Traffic with Evaluate: 20 and Skip: 80 to score one conversation in five.
3. Connect Evaluate to Evaluate AI Conversation. Write your criteria, and a Label such as Weekly QA.
4. Connect Failed to whatever should happen: a Notify to a supervisor, or a task in your own system.

Every evaluation, real or simulated, also starts the workflows that begin with AI Conversation Evaluated. Tick Only conversations that failed to react to the failures only. The evaluation is in Workflow.Input.Evaluation.

Score a sample of real conversations

With the AI Quality feature on, a QA cohort scores a share of the real conversations of some AI profiles after they close, without a workflow. Open Artificial Intelligence > QA, click QA cohorts, then Add cohort:

• AI profiles: whose conversations are sampled.
• Sample: the share scored, such as 20%. A conversation is in or out of the sample for good, so it is never scored twice.
• Success criteria and Passing score, as for the Evaluate AI Conversation task.
• Check answers against the knowledge base: the profile's own knowledge base and data sources are searched for what the customer asked, and any fact the AI stated that they don't support is reported as a Hallucination, with the line that shows it.

A cohort starts with the conversations that close after it is saved, and scores them every ten minutes. Each score is an evaluation labeled QA: <cohort name>, so it shows in the Conversation evaluations report and raises the AI Conversation Evaluated event like any other.

The QA dashboard

The QA dashboard is the AI QA Dashboard report, under Reports > Artificial Intelligence Reports, so it needs the Reports feature. Open it from Artificial Intelligence > QA with QA dashboard, or with Dashboard on a cohort to see only that cohort's last 7 days. It shows, for one cohort or for every scored conversation, over the date range you choose (the last 7 days at first; Last 24 Hours, Last 30 Days and Last 90 Days are one click away in the Date range, beside the other choices):

• how many conversations were scored, their average score, and the share that passed and that were resolved;
• the scores day by day;
• the issues found, by kind, and the customers' sentiment at the end;
• the most frequent opening questions, with the share of them resolved;
• the lowest-scored conversations, with the AI profile of each and its session, which opens the conversation.

Like every report, it exports as CSV, and as Excel when Reports (OpenXml) is on. Artificial Intelligence > QA now opens the cohorts list.

Reports

In Reports, the AI Chat data source has a Conversation evaluations data set: one row per evaluation, with its profile, profile version, label, score, whether it passed, and a count of each kind of issue. The Chat sessions data set has the Profile version of each conversation.

For example, a table grouped by Profile version with the average Score and the count of Passed compares version 3 with version 4.

Get alerted when something slips

Alert rules

With the AI Quality feature on, Artificial Intelligence > Alert rules watches a metric without a workflow. Click Add alert rule and choose the Metric, the AI profile (or every profile), when to alert and the Threshold, the Window measured, the Minimum samples a window needs, and how often to Check it. The metrics are the ones the task below measures.

When the threshold is crossed, the rule opens an incident; when a measured window is back within it, the incident resolves. Each change is told once:

• by email to the addresses in Email, when the Email feature is set up;
• by the alert.triggered and alert.resolved events of Webhooks, when the AI Webhooks feature is on;
• by the AI Alert Triggered and AI Alert Resolved workflow events, whose input holds the rule, the metric, its value, the threshold and the window.

The list shows each rule as OK or Alerting, with its latest value and when it last triggered; a rule keeps its last 20 incidents.

Alerts in a workflow

The Check AI Conversation Metric task measures one number over the last window, compares it with a threshold, and remembers whether an alert is already open. Start it with a Timer that runs every few minutes.

Field	What it means
Metric	What to measure: conversations started, resolution rate, conversation length, response time, rated down, the average QA score, QA failures, tool failures, unsupported answers, AI calls, AI calls handed to a person, AI calls that failed, or the time to the AI's first words on a call. Rates are percentages.
Profile	One profile, or all of them.
Alert when the value	Is above or below the threshold, or rises or falls by more than the threshold, in percent of the window before.
Window (minutes)	How far back to measure.
Fewest samples	The fewest conversations, evaluations or calls the number must be measured over before it can alert, so two calls in a quiet hour do not raise a 50% failure rate.

The task goes on by one of four outcomes:

Outcome	When
Alert started	The threshold is crossed and no alert was open.
Still alerting	The threshold is still crossed.
Resolved	An alert was open and the number is back.
Normal	No alert, nothing crossed.

Connect Alert started and Resolved to Notify, Send Email, or an HTTP Request to Slack or Teams. The measurement is in Workflow.Output.Alert, for example {{ Workflow.Output.Alert.Value }}.

For example, to be emailed when more than 5% of a profile's evaluated conversations had a failed tool in the last hour: Metric Tool failures (%), Alert when the value Is above the threshold, Threshold 5, Window 60.

Analyze a conversation again

The Rerun AI Conversation Analysis task runs a finished conversation's post-session tasks and conversion goals again with the profile's current settings, for example after you added a post-session task. The new results are in Workflow.Output.Results. To rerun a set of conversations, put it in a For Each over their session IDs.