Bot Exam checks your bot the way a customer would, before one does. It writes questions from your own documents, asks your bot each one, and grades the answers. When something fails, it tells you where the answer was lost and what to do about it. It runs again by itself after your knowledge changes, so a re-synced website or an edited document can't quietly break an answer that used to work.
Run your first exam
Add some knowledge first
The exam tests what your bot has learned, so add a document, a website or a sheet on the Knowledge tab and wait until it says Ready.
Open the Exam tab and press Run the exam
It first writes test questions from your documents, then grades your bot on each one. You can watch the progress. It usually takes about a minute.
You should see: a score, how many questions passed and failed, and a list of what needs your attention.
Where the questions come from
Each exam has three kinds of question:
- Written from your documents. Every one comes with the facts a correct answer must state, such as a number, a duration or a name. We only keep a question if those facts appear word for word in your document, so the exam never grades your bot against something your documents don't say.
- Questions your bot should decline. A few near-misses on your topics, and a couple that have nothing to do with your business. A good bot says it doesn't know instead of guessing.
- Your own. Add the questions that matter most to you, with the facts the answer must state. These are never replaced.
You can switch any question off, for example one that your documents don't really cover. Questions written for you are switched off rather than deleted, so they don't come back.
Reading the results
The score is the share of graded questions your bot got right. Questions marked Needs a look are not counted either way: the bot answered a question that looked outside your documents, and only you know whether that's right. Questions marked Not checked hit a problem on our side and don't count against you.
For every failed question, the exam says where the answer was lost:
- Search never found the passage. Reword the document so it uses the words customers use, or teach the answer.
- Search found it, but the bot didn't read it. Teaching the answer makes it certain.
- The bot read the passage and still declined. Check how the passage is worded.
- The answer left out a fact. It lists the missing facts. Fix the document if it's unclear, or teach the answer.
Fixing a failed question
Press Teach the answer
Write the answer your bot should give. It's saved as an approved answer for that question, the same as answering a question on the Knowledge Gaps screen.
Run the exam again
Wait for the new answer to finish indexing, then run the exam. A question that failed and now passes is listed under what you fixed.
Automatic re-runs
Once you've run the exam yourself, it re-runs after your knowledge changes: a document edited, added or deleted, a website refreshed, or a sheet that changed. It waits about ten minutes after the last change, so a refresh that touches many pages is graded once. If a question passed before and fails now, the Exam tab says so at the top. Turn this off with the switch on the score card.