Tuning, testing, and troubleshooting unexpected answers

Tuning answers using the content designer

By default, QnABot on AWS attempts to match an end user’s question to the list of questions and answers stored in Amazon OpenSearch Service. QnABot on AWS uses full text search to find the item that is the best match for the question asked. Words that are used infrequently score higher than words that are used often, so sentence constructs such as prepositions have lower weighting than unique keywords. The closer the alignment between a question associated with an item and a question asked by the user, the greater the probability that QnABot on AWS will choose that item as the most relevant answer.

The solution tries to find the best answer to questions by applying the keyword filters, and by matching the words used in the end user’s question to the words used in the question fields of the stored answers—giving preference when the same words are used in the same order.

You might find that end users ask questions in ways that you haven’t anticipated, resulting in unexpected answers being returned by QnABot on AWS. When this happens, you can use the content designer to troubleshoot and fix the problem.

For more information, refer to the Tuning Recognition Accuracy Guide in the GitHub repository.

Testing all your questions

Use the following procedure to test your questions.

1. Log in to the content designer, and choose TEST ALL.

2. Use the default filename, or enter your own.

3. Optionally, if you want to test only a subset of questions, you can filter by the qid prefix. Leave this field blank to test all the questions.

4. Select TEST ALL, and wait for the tests to complete.

5. Select the view results icon on bottom right to view the test results. Any incorrect matches are highlighted in red. Test results can be viewed in the browser, or downloaded to your computer as a CSV file.

Tuning the chatbot’s Automatic Speech Recognition

When you ask QnABot on AWS a question, it is processed and transcribed by either Amazon Lex or Amazon Alexa using an Automatic Speech Recognition (ASR) engine. QnABot on AWS initially trains the ASR to match a wide variety of possible questions and statements, so that the Amazon Lex chatbot and Alexa skill will accept almost any question a user asks.

This solution supports AMAZON.FallbackIntent in both Amazon Lex and Alexa, which allows it to process anything end users say without needing to retrain and rebuild the Amazon Lex chatbot or Alexa skill.

Occasionally, the transcription shown in the web client or the Alexa app isn’t accurate. This can happen with unusual words that are confused for other more common words or phrases. Use one of the following approaches to troubleshoot this error:

• Use content designer to add additional question variants that match the actual transcription shown in the web client or in the Alexa app; this allows QnABot on AWS to anticipate the transcription accuracy problem, and respond anyway.

• Retrain the Amazon Lex and Alexa ASR with examples to influence the transcription to more closely match what you want – refer to Retrain Amazon Lex for more information.

Retrain Amazon Lex

QnABot on AWS can automatically generate additional ASR training data for Amazon Lex using questions from the data you have added.

1. Log in to the content designer and choose LEX REBUILD from the top right edit menu ( ⋮ ).

2. Wait for the rebuild to complete.

Retrain Alexa

QnABot on AWS can generate additional ASR training data for Alexa using questions from the data you have added.

1. Log in to the content designer and choose ALEXA UPDATE from the top right edit menu ( ⋮ ).

2. Select COPY SCHEMA to copy the updated Alexa skill schema.

3. Log in to the Alexa Developer Console, open your QnABot on AWS skill, and then use the JSON Editor to paste the new schema, replacing the existing one.

4. Save and then build the updated model.

Monitoring QnABot on AWS usage and user feedback

The solution logs everything that end users say to the chatbot. Amazon Data Firehose stores logged utterances to a new index in Amazon OpenSearch Service.

You can also allow your end users to provide feedback about the chatbot’s answers. Use the following procedure to set up the feedback mechanism.

1. From content designer’s top left tools menu ( ☰ ), select Import.

2. Open Examples/Extensions, and select the LOAD for the QnAUtility demo.

3. From the top left tools menu ( ☰ ), select EDIT and examine the newly imported items, Feedback.001 and Feedback.002; observe the list of default expressions that the end user can input to invoke feedback. (The example QnAUtility demo package also loads items Help, CustomNoMatches, CustomNoVerifiedIdentity.)

4. Test the feedback mechanism.

   • Use the web UI to ask a question, such as: “What happens if I ask an unanticipated question?”. Because we have not entered a suitable answer for this question, QnABot on AWS responds with the newly imported  CustomNoMatches response—indicating that it doesn’t know the answer.

   • From the web UI, speak or type “Thumbs down”, or select the thumbs down icon beside the answer.

The solution publishes Thumbs down feedback messages to the Amazon Simple Notification Service (SNS) topic identified by the FeedbackSNSTopic on the Outputs tab of the CloudFormation stack. To learn how to subscribe to the SNS topic, and receive a message from the each time a user provides feedback, refer to Subscribing to an Amazon SNS topic in the Amazon Simple Service Notification Developer Guide.

If you have selected an non-English language for your QnABot deployment and have multi-language enabled then you should do the following steps:

1. Use the AWS translate API to translate Thumbs up and Thumbs down to your deployment language if it is not English.

2. Add the translation of the Thumbs up and down as a question in your QnABot deployment.

3. Go to the content designer and on the top left, then select Settings.

4. Find the PROTECTED_UTTERANCES variable and insert that phrase in by adding a comma, then enter the translation.

Use the following process to visualize the usage logs and feedback using the Amazon OpenSearch Service Kibana dashboard. Note that it can take up to 5 minutes for new utterances and end user feedback to become visible in the dashboard.

1. From content developer, select the top left tools menu ( ☰ ), and choose Kibana Dashboard. Kibana opens in a new browser tab.

2. Choose the QnABot on AWS Dashboard to visualize usage history and sentiment, all logged utterances, no hits utterances, positive user feedback, and negative user feedback.

   

   Sample Kibana dashboard

3. Edit Kibana to change the time span, customize and build your own visualizations, or to run your own queries.

Using Amazon CloudWatch to monitor and troubleshoot

The solution’s metrics and logs are available in an Amazon CloudWatch dashboard. Use the following procedure to launch the dashboard and visualize the solution’s AWS resources.

Sample CloudWatch dashboard

1. From the CloudFormation stack’s Outputs tab, select the DashboardUrl link.

2. When troubleshooting the chatbots responses to your questions, trace the request and response using the logs created by the Fulfillment Lambda function.

3. Choose the Menu tool in the upper right of the >Fulfillment Lambda wdget, select View logs, then and choose the AWS Lambda function.

   

   FulfillmentLambda function

4. Inspect the log messages. Each interaction with the solution is delimited by START and END messages. Between these messages are insights into how the solution processes the question.