Skip to main content
Version: Latest
User ManualNeed configuration, code or deployment details? In the Technical Manual: AI Chat Session AnalyticsAI Usage Analytics

AI Chat and Usage Analytics

The AI reports answer the questions a manager asks about an assistant: How many people use it? Does it solve their problem? Do they like the answers? What details did it collect? And what does it cost? Use them to decide where an assistant needs better instructions or knowledge, and to plan your AI budget.

MenuReports > Artificial Intelligence Reports
PermissionView AI Chat Analytics, to see the reports; Export AI Chat Analytics as well, to export Session Analytics, Extracted Data and Conversion Goals
FeatureAI Chat Session Analytics, AI Usage Analytics
Can't find it?

What you see depends on your role. If the menu, a button or a setting on this page is missing, ask your administrator to give you the permission listed above or to turn on the feature. Only administrators can turn features on and change site settings.

The video shows the settings that collect the data and tours the Session Analytics, Extracted Data and Conversion Goals reports.

ReportWhat it shows
AI Chat Session AnalyticsHow chats with your assistants go: volume, resolution, timing, ratings and token use.
AI Chat Extracted DataThe details an assistant collected in each chat, one row per chat.
AI Chat Conversion GoalsHow each chat scored against the assistant's goals.
AI Usage AnalyticsEvery request sent to an AI service, per model and per feature, and the minutes of automated AI phone calls.

Collect the data​

The reports only contain what was collected, and only from the moment collection was turned on.

  • For the chat reports, open each chat profile under Artificial Intelligence > Profiles, go to the Data Processing & Metrics tab and tick Enable session metrics. Tick Enable AI resolution detection to have the AI judge whether each chat solved the problem, and Enable conversion metrics to score chats against goals. Data extraction and conversion goals are set on the same tab; see AI profiles.
  • For the usage report, an administrator enables the AI Usage Analytics feature under Tools > Features. Usage is recorded from that moment on, and the report appears under Reports > Billing & Usage.

When session metrics are on, every answer in the chat gets Thumbs up and Thumbs down buttons, and people's ratings feed the User Feedback section.

A chat counts as finished when it ends naturally or when nobody writes for the profile's Session inactivity timeout. Resolution, conversion and extracted data appear only after a chat has finished and been analyzed, which can take a few minutes.

AI Chat Session Analytics​

  1. Open Reports > Artificial Intelligence Reports > AI Chat Session Analytics.
  2. Choose a Date range, such as Last 30 Days or dates of your own, and optionally one profile (or Any profile). The report starts on All Time, every chat; On or after and On or before leave the other end open.
  3. Click Show.

Point at the i beside a figure to read what it measures. A section with no data says how to start collecting it, for example to turn on resolution detection.

To download a row for each chat, click Export (CSV, or Excel when Reports (OpenXml) is on). It needs the Export AI Chat Analytics permission; without it the button is hidden. The file is one table with the columns SessionId, ProfileId, VisitorId, UserId, IsAuthenticated, SessionStartedUtc, SessionEndedUtc, MessageCount, HandleTimeSeconds and IsResolved, the same as the old Export as CSV, so scripts that read it keep working.

Conversation & Usage​

FigureWhat it means
Total SessionsChats started in the period.
Unique VisitorsDifferent people who chatted, by user account or visitor.
Containment RateShare of chats the assistant handled without a person stepping in. Higher is better.
Abandonment RateShare of chats where the person left without a resolution. A high rate can mean unhelpful answers.
Avg Session DurationAverage time from the first to the last message.
Avg Messages/SessionAverage number of messages, from both sides, per chat.
Returning User RateShare of people who came back for another chat.
Avg Steps to ResolveAverage number of messages needed in resolved chats. Fewer is faster.
Resolved Sessions / Abandoned Sessions / Active SessionsChats that ended resolved, ended by inactivity without a resolution, or are still going.

Usage by Time of Day and Usage by Day of Week​

When people chat, by hour and by weekday. Use them to see peak times, for example to have staff ready when a live hand-off is likely.

User Segmentation​

How many chats came from signed-in users (Authenticated Sessions, Unique Logged-in Users) and from anonymous visitors (Anonymous Sessions, Unique Anonymous Visitors).

Model & System Performance​

FigureWhat it means
Avg Response LatencyHow long the AI takes to write an answer, in milliseconds.
Total Tokens UsedAll tokens used: Input Tokens (what was sent to the AI, including instructions and history) plus Output Tokens (what the AI wrote). Tokens are what AI providers charge for.
Avg Tokens/Session, Avg Input Tokens/Session, Avg Output Tokens/SessionTokens per chat. Useful to estimate the cost of one conversation. High input tokens can mean long instructions or long histories.

Some AI services do not report tokens for every answer, so these figures can be lower than your bill.

Resolution & Conversion​

FigureWhat it means
Resolution RateShare of finished chats that the AI judged as solved, from the conversation itself. Needs Enable AI resolution detection.
Resolved / UnresolvedThe number of chats judged solved or not. Read some unresolved chats to find gaps in the assistant's knowledge.
Avg Conversion ScoreThe average score against your conversion goals, as a share of the best possible score. Needs Enable conversion metrics.
Evaluated SessionsChats that were scored.
High Performing / Low PerformingShare of scored chats at 70% or more, and below 30%.

User Feedback​

FigureWhat it means
Positive Ratings / Negative RatingsChats rated thumbs up or thumbs down. A person rates the whole chat; their latest rating counts.
Satisfaction RateShare of rated chats with a thumbs up.
Feedback RateShare of chats that were rated at all.

AI Chat Extracted Data​

Lists the details an assistant collected with data extraction, such as names, emails and products of interest, one row per chat.

  1. Open Reports > Artificial Intelligence Reports > AI Chat Extracted Data.
  2. Pick the AI Profile (required), and optionally a Date range.
  3. Click Show. The table has Session started, Session and one column per extracted detail. The report starts on All Time.
  4. Click Export to download it, for example to import leads into another system. It needs the Export AI Chat Analytics permission. The file has the columns SessionStartedUtc, SessionId and one per extracted detail.

AI Chat Conversion Goals​

Shows how each finished chat scored against the profile's conversion goals.

  1. Open Reports > Artificial Intelligence Reports > AI Chat Conversion Goals.
  2. Pick the AI Profile (required), and optionally a Date range.
  3. Click Show. The table has Session started, Session, Total points and one column per goal. The report starts on All Time.
  4. Click Export to download it. It needs the Export AI Chat Analytics permission. The file has the columns SessionStartedUtc, SessionId, TotalPoints and one per goal.

AI Usage Analytics​

The video turns the feature on, runs the report, groups the figures, explains the voice figures and builds a usage report in the Report Builder.

Shows what AI used across the whole site: every request sent to an AI service, the completions from chats and chat interactions, and the automated AI phone calls. Use it to see which models and which features your AI budget goes to.

  1. Open Reports > Billing & Usage > AI Usage Analytics. It needs the AI Usage Analytics feature, not AI Chat Session Analytics.
  2. Choose a Date range, and optionally an AI profile. The report starts on All Time.
  3. Choose how to group the figures:
    • Group metered usage by: Model, Model and category (which feature made the request: chat, SMS, voice calls, the API, workflows) or Model and purpose (why: answering, searching, indexing documents, transcribing, and so on).
    • Group audio usage by: Model, Model and voice or Model and language.
    • Group completions by: User and model, Model, Deployment, AI profile or Connection.
    • Group voice calls by: Deployment (model), AI profile, Campaign, Channel, Engine or Day.
  4. Click Show.

The Metered usage table counts every request sent to an AI service, of every kind: chat, embeddings for search, images, speech-to-text, text-to-speech and realtime voice. One answer can take several requests, such as searching your documents, using a tool and writing the reply, and each is counted. Next to the tokens it shows the other things AI services charge for: cached input, reasoning, audio tokens, audio minutes, characters spoken and images. These are the figures to compare with your AI provider's bill.

The Audio usage table covers only speech-to-text and text-to-speech: requests, hours of audio, hours of speech, characters, audio size, segments and tokens. An asterisk next to the hours means some requests did not report an audio length, so the figure is a minimum.

Under Chat completions, the top tiles show Provider calls, Chat sessions, Chat interactions and Total tokens. The table shows, per group, the Calls, Sessions, Interactions, Input, Output and total Tokens, and the Avg latency.

The Voice section covers automated AI phone calls:

FigureWhat it means
CallsAutomated calls in the period.
AI minutesMinutes the caller heard the assistant speak.
Caller minutesMinutes the caller spoke, on calls where this can be measured.
Silence minutesMinutes nobody spoke, on the same calls.
Average callThe average length of the assistant's part of a call.
Handoff rateShare of calls handed to a live agent.
Barge-ins per callHow often callers talked over the assistant and it stopped to listen.

Some figures cannot be measured on every kind of call and are left out rather than counted as zero. How each figure is measured is explained in the Technical Manual. For the automated calls themselves, see Automated AI SMS and Voice.