AI Chat and Usage Analytics
The AI reports answer the questions a manager asks about an assistant: How many people use it? Does it solve their problem? Do they like the answers? What details did it collect? And what does it cost? Use them to decide where an assistant needs better instructions or knowledge, and to plan your AI budget.
| Menu | Reports > Artificial Intelligence Reports |
| Permission | View AI Chat Analytics, to see the reports; Export AI Chat Analytics as well, to export Session Analytics, Extracted Data and Conversion Goals |
| Feature | AI Chat Session Analytics, AI Usage Analytics |
What you see depends on your role. If the menu, a button or a setting on this page is missing, ask your administrator to give you the permission listed above or to turn on the feature. Only administrators can turn features on and change site settings.
The video shows the settings that collect the data and tours the Session Analytics, Extracted Data and Conversion Goals reports.
| Report | What it shows |
|---|---|
| AI Chat Session Analytics | How chats with your assistants go: volume, resolution, timing, ratings and token use. |
| AI Chat Extracted Data | The details an assistant collected in each chat, one row per chat. |
| AI Chat Conversion Goals | How each chat scored against the assistant's goals. |
| AI Usage Analytics | Every request sent to an AI service, per model and per feature, and the minutes of automated AI phone calls. |
Collect the data
The reports only contain what was collected, and only from the moment collection was turned on.
- For the chat reports, open each chat profile under Artificial Intelligence > Profiles, go to the Data Processing & Metrics tab and tick Enable session metrics. Tick Enable AI resolution detection to have the AI judge whether each chat solved the problem, and Enable conversion metrics to score chats against goals. Data extraction and conversion goals are set on the same tab; see AI profiles.
- For the usage report, an administrator enables the AI Usage Analytics feature under Tools > Features. Usage is recorded from that moment on, and the report appears under Reports > Billing & Usage.
When session metrics are on, every answer in the chat gets Thumbs up and Thumbs down buttons, and people's ratings feed the User Feedback section.
A chat counts as finished when it ends naturally or when nobody writes for the profile's Session inactivity timeout. Resolution, conversion and extracted data appear only after a chat has finished and been analyzed, which can take a few minutes.
AI Chat Session Analytics
- Open Reports > Artificial Intelligence Reports > AI Chat Session Analytics.
- Choose a Date range, such as Last 30 Days or dates of your own, and optionally one profile (or Any profile). The report starts on All Time, every chat; On or after and On or before leave the other end open.
- Click Show.
Point at the i beside a figure to read what it measures. A section with no data says how to start collecting it, for example to turn on resolution detection.
To download a row for each chat, click Export (CSV, or Excel when Reports (OpenXml) is on). It needs the Export AI Chat Analytics permission; without it the button is hidden. The file is one table with the columns SessionId, ProfileId, VisitorId, UserId, IsAuthenticated, SessionStartedUtc, SessionEndedUtc, MessageCount, HandleTimeSeconds and IsResolved, the same as the old Export as CSV, so scripts that read it keep working.
Conversation & Usage
| Figure | What it means |
|---|---|
| Total Sessions | Chats started in the period. |
| Unique Visitors | Different people who chatted, by user account or visitor. |
| Containment Rate | Share of chats the assistant handled without a person stepping in. Higher is better. |
| Abandonment Rate | Share of chats where the person left without a resolution. A high rate can mean unhelpful answers. |
| Avg Session Duration | Average time from the first to the last message. |
| Avg Messages/Session | Average number of messages, from both sides, per chat. |
| Returning User Rate | Share of people who came back for another chat. |
| Avg Steps to Resolve | Average number of messages needed in resolved chats. Fewer is faster. |
| Resolved Sessions / Abandoned Sessions / Active Sessions | Chats that ended resolved, ended by inactivity without a resolution, or are still going. |
Usage by Time of Day and Usage by Day of Week
When people chat, by hour and by weekday. Use them to see peak times, for example to have staff ready when a live hand-off is likely.
User Segmentation
How many chats came from signed-in users (Authenticated Sessions, Unique Logged-in Users) and from anonymous visitors (Anonymous Sessions, Unique Anonymous Visitors).
Model & System Performance
| Figure | What it means |
|---|---|
| Avg Response Latency | How long the AI takes to write an answer, in milliseconds. |
| Total Tokens Used | All tokens used: Input Tokens (what was sent to the AI, including instructions and history) plus Output Tokens (what the AI wrote). Tokens are what AI providers charge for. |
| Avg Tokens/Session, Avg Input Tokens/Session, Avg Output Tokens/Session | Tokens per chat. Useful to estimate the cost of one conversation. High input tokens can mean long instructions or long histories. |
Some AI services do not report tokens for every answer, so these figures can be lower than your bill.
Resolution & Conversion
| Figure | What it means |
|---|---|
| Resolution Rate | Share of finished chats that the AI judged as solved, from the conversation itself. Needs Enable AI resolution detection. |
| Resolved / Unresolved | The number of chats judged solved or not. Read some unresolved chats to find gaps in the assistant's knowledge. |
| Avg Conversion Score | The average score against your conversion goals, as a share of the best possible score. Needs Enable conversion metrics. |
| Evaluated Sessions | Chats that were scored. |
| High Performing / Low Performing | Share of scored chats at 70% or more, and below 30%. |
User Feedback
| Figure | What it means |
|---|---|
| Positive Ratings / Negative Ratings | Chats rated thumbs up or thumbs down. A person rates the whole chat; their latest rating counts. |
| Satisfaction Rate | Share of rated chats with a thumbs up. |
| Feedback Rate | Share of chats that were rated at all. |
AI Chat Extracted Data
Lists the details an assistant collected with data extraction, such as names, emails and products of interest, one row per chat.
- Open Reports > Artificial Intelligence Reports > AI Chat Extracted Data.
- Pick the AI Profile (required), and optionally a Date range.
- Click Show. The table has Session started, Session and one column per extracted detail. The report starts on All Time.
- Click Export to download it, for example to import leads into another system. It needs the Export AI Chat Analytics permission. The file has the columns
SessionStartedUtc,SessionIdand one per extracted detail.
AI Chat Conversion Goals
Shows how each finished chat scored against the profile's conversion goals.
- Open Reports > Artificial Intelligence Reports > AI Chat Conversion Goals.
- Pick the AI Profile (required), and optionally a Date range.
- Click Show. The table has Session started, Session, Total points and one column per goal. The report starts on All Time.
- Click Export to download it. It needs the Export AI Chat Analytics permission. The file has the columns
SessionStartedUtc,SessionId,TotalPointsand one per goal.
AI Usage Analytics
The video turns the feature on, runs the report, groups the figures, explains the voice figures and builds a usage report in the Report Builder.
Shows what AI used across the whole site: every request sent to an AI service, the completions from chats and chat interactions, and the automated AI phone calls. Use it to see which models and which features your AI budget goes to.
- Open Reports > Billing & Usage > AI Usage Analytics. It needs the AI Usage Analytics feature, not AI Chat Session Analytics.
- Choose a Date range, and optionally an AI profile. The report starts on All Time.
- Choose how to group the figures:
- Group metered usage by: Model, Model and category (which feature made the request: chat, SMS, voice calls, the API, workflows) or Model and purpose (why: answering, searching, indexing documents, transcribing, and so on).
- Group audio usage by: Model, Model and voice or Model and language.
- Group completions by: User and model, Model, Deployment, AI profile or Connection.
- Group voice calls by: Deployment (model), AI profile, Campaign, Channel, Engine or Day.
- Click Show.
The Metered usage table counts every request sent to an AI service, of every kind: chat, embeddings for search, images, speech-to-text, text-to-speech and realtime voice. One answer can take several requests, such as searching your documents, using a tool and writing the reply, and each is counted. Next to the tokens it shows the other things AI services charge for: cached input, reasoning, audio tokens, audio minutes, characters spoken and images. These are the figures to compare with your AI provider's bill.
The Audio usage table covers only speech-to-text and text-to-speech: requests, hours of audio, hours of speech, characters, audio size, segments and tokens. An asterisk next to the hours means some requests did not report an audio length, so the figure is a minimum.
Under Chat completions, the top tiles show Provider calls, Chat sessions, Chat interactions and Total tokens. The table shows, per group, the Calls, Sessions, Interactions, Input, Output and total Tokens, and the Avg latency.
The Voice section covers automated AI phone calls:
| Figure | What it means |
|---|---|
| Calls | Automated calls in the period. |
| AI minutes | Minutes the caller heard the assistant speak. |
| Caller minutes | Minutes the caller spoke, on calls where this can be measured. |
| Silence minutes | Minutes nobody spoke, on the same calls. |
| Average call | The average length of the assistant's part of a call. |
| Handoff rate | Share of calls handed to a live agent. |
| Barge-ins per call | How often callers talked over the assistant and it stopped to listen. |
Some figures cannot be measured on every kind of call and are left out rather than counted as zero. How each figure is measured is explained in the Technical Manual. For the automated calls themselves, see Automated AI SMS and Voice.