Skip to content
VeraFrame Docs

Chatbot Audit mode

Chatbot Audit is for verifying AI output you did not generate in VeraFrame. You paste in a response that a chatbot (your own or a vendor’s) has already produced, provide the source material it was supposed to use, and VeraFrame tells you which claims in the response are actually supported.

Use it when:

  • You are evaluating a third-party chatbot or AI assistant and want to know how often it is correct.
  • You are auditing AI-generated content that was published elsewhere.
  • A customer has received a response from your AI system and you need to verify before standing behind it.

Chatbot Audit is available to tenants with the tool_audit feature enabled (default).

What you give it

  • The chatbot response — the text that was produced, pasted into the input field.
  • Source material — the sources the chatbot was supposed to be drawing from. Select existing source groups or upload files for this one audit.
  • (Optional) User corrections — if a human has already corrected parts of the response, you can include those corrections so the audit reflects them.

What you get back

A trust report in exactly the same format as Generate mode:

  • Verified claims — backed by a specific passage in the source material.
  • Needs review claims — the chatbot asserted something that is not in the sources you provided.
  • Mismatch / Fabricated claims — the chatbot said something that contradicts the source material, or something the sources do not support at all.
  • Missing — the response omitted information the sources contain that is relevant to the question.

The report shows the original chatbot response with every claim highlighted by status. You can export the audit for evidence.

Step by step

  1. Open the main tool and select the Audit button (orange).
  2. Paste the chatbot response into the response field.
  3. Select source material. For audit mode it is common to upload a single source document — the knowledge base article, policy, or contract the chatbot should have used.
  4. Click Run.
  5. Review the report. Each claim is color-coded; click any claim to see the matching or conflicting source passage.
  6. Export the audit if you need it for a ticket, incident review, or vendor comparison.

Typical use patterns

Continuous vendor evaluation

Audit a sample of your customer service chatbot’s daily responses against the knowledge base. The mismatch rate over time tells you whether the vendor’s AI is getting better or drifting.

Incident response

When a customer complaint mentions a specific chatbot answer, paste it in, paste the source material the chatbot should have used, and produce an audit trail in seconds.

Cross-check before republication

If your marketing or legal team asks an AI assistant to summarize a policy, audit the result before publishing to confirm nothing was subtly paraphrased or fabricated.