AI analytics

EARLY ACCESS

Create and run tasks to test the effectiveness of your AI assistant and analyze past user interactions. Use the results as a guide to identify issues with the AI assistant and improve it.

Task types

You can run the following tasks:

• Evaluate: Test the assistant with a set of questions and check whether it responds as expected.
• Analyze traffic: Analyze end user messages for the selected period to gain insights into their interests and the assistant's performance.

Process overview

1. Create and run the task.
2. Check the status of the task.
3. When the task is complete, download and view the results.
4. Update the AI assistant as required.

Evaluate the AI assistant

Prerequisites

Prepare a .csv file that contains a set of sample questions, which end users are likely to ask your assistant.

Example: For an AI assistant that answers FAQ about cars, you can add the following questions:

What car models do you have available?

Do you sell new and used cars?

Can I trade in my old car?

What financing options do you provide?

You can also download a sample of the .csv file and use it as the base to create your file.

Run the task

1. Go to AI hub > AI analytics.

2. In the Select AI assistant field, select the assistant that you want to test.

3. In the Select tasks type section > Evaluate tab, upload the .csv file that contains the set of questions.

   

4. In the Task name field, enter a name for the task so that you can identify it.

   Example: Test FAQ about cars.

5. Select Run task.

You can check the status of the task. When the task is complete, you can download and view the results, and update the AI assistant as required.

Analyze traffic

Analyze goes through all the messages sent by end users to your AI assistant within the selected period, and provides insights into the assistant's performance.

Run the task

1. Go to AI hub > AI analytics.

2. In the Select AI assistant field, select the assistant that you want to analyze.

3. Select the period for which you want to analyze the traffic.

4. In the Task name field, enter a name for the task so that you can identify it.

   Example: Traffic for FAQ assistant.

5. Select Run task.

You can check the status of the task. When the task is complete, you can download and view the results, and update the AI assistant as required.

View task status

You can view all the tasks in the Tasks section.

This section shows a maximum of the 20 most recent tasks of each type.

Status	Description
Completed	The task is complete, and the results are ready to download.
Failed	The task failed. Hover over the Information icon next to the status to identify why the task failed. Example: For the Analyze traffic task, there was no traffic data to analyze.
In progress	The task is in progress.

View the task results

Download and view the task results.

Download the task results

The task results are generated in a .csv file. To view the file, go to the Tasks section and download the file for your task.

Content of the results file

The file contains the following information:

Field in the file	Description
event_timestamp	The timestamp of the end user's message. This field is present only when evaluating questions asked by end users.
session_id	The identifier of the chatbot session. This field is present only when evaluating questions asked by end users.
question	Evaluate task: The question in the text file. Analyze task: The question asked by the end user.
answer	The response generated by the AI assistant.
original_contexts	These are the relevant parts (chunks) of the documentation that were retrieved based on user's question. Context also contains the name of the document from which the chunk was retrieved. Use the context to understand why the assistant responded in a certain way.
reranked_contexts	If you have enabled reranker, only reranked_context is sent to LLM to generate the response. For very short messages, reranking is not required. For such messages, reranked_contexts is not available.
topic	The AI assistant determines the topic based on the question. This attribute gives insight into how many end users are interested in a particular topic or product.
is_question_answered	Shows whether the AI assistant answered a question. TRUE - Chatbot answered the question. FALSE - Chatbot was unable to answer the question. The reason could be one of the following: - The source documentation does not contain the answer. - The AI Assistant concluded that the question is not relevant for the assistant. Example: The Assistant is for answering questions about cars, but the question is about a mobile phone. In this case, the answer field is I'm sorry, I cannot help you with that question. I'm a chatbot for cars. N/A - The message sent to the assistant does not contain a question. Example: The message is Hi.
is_answer_in_context	Indicates that the answer is based on the retrieved context.

Update the assistant based on the task results

The task results are a guide to help you get started and indicate issues. It is important to investigate and identify the reasons behind the issues.

You can then update the assistant to see if the issue is resolved.

Issue	Solution
Assistant answered the question, but the answer is not in the context.	It is possible that the Assistant answered correctly based on the information present in the prompt. If the answer is wrong, check the documentation.
Assistant uses an inappropriate tone of voice.	Check the prompt and update it as required.
Assistant does not answer in the correct language.	Check the prompt and update it as required.
Assistant's responses are cut off.	Check the number of output tokens.
Assistant hallucinates.	It is likely that the Assistant is missing the relevant context. Check the documentation.
Assistant responds to a question that is not part of its scope.	Check the prompt and update it as required.