SERVICE — AUDIT
AI agent & chatbot audit: does yours answer correctly, leak nothing and comply?
A fixed-scope, vendor-independent review of a chatbot or AI agent you already run — whoever built it. You get a scored report across quality, security (red-teamed), compliance and operations, and a fix list ordered by risk — the evidence an AI governance file needs.
INyour live chatbot or agent
→
OUTscored report + prioritized fix list
The problem
Most chatbots go live after a demo, not a test. Months later nobody knows how often it answers wrong, whether a crafted message can pull customer data out of it, or whether it tells users they are talking to an AI — a duty under Article 50 of the EU AI Act since 2 August 2026. The vendor says it is fine. The vendor built it.
What the report looks like
A score per area, every finding with severity and fix, and the test set to re-run after changes. Demo data shown.
AUDIT REPORT · SUPPORT-BOT V3DEMO DATA
Quality72/100
Security41/100
Compliance55/100
Operations80/100
- IDSEVERITYAREAFINDING
- F-01HIGHSecurityIndirect prompt injection via uploaded PDF reveals the system prompt and another customer's order
- F-02HIGHComplianceNo AI disclosure at first interaction (EU AI Act Art. 50)
- F-03MEDQualityTopic “return policy”: 4 of 20 test answers contradict the current policy
- F-04MEDOperationsTool calls are not traced — yesterday's conversations cannot be reconstructed
- F-05LOWComplianceConversation logs retained indefinitely; no retention rule
12 findings · 2 high · 6 medium · 4 low · test set: 140 cases
What we check
- Answer quality — we build a test set from your real conversations and measure correctness, groundedness, hallucination rate and escalation behavior — per topic, not as one average.
- Security — prompt injection (direct, and indirect via documents, web pages or mail), system-prompt and data extraction, tool permissions and excessive agency — mapped to the OWASP Top 10 for LLM Applications and the OWASP Top 10 for Agentic Applications (2026).
- GDPR and EU AI Act — data flows, processor agreements, retention of logs and vector stores, risk classification, Art. 50 transparency, and whether your Art. 4 AI-literacy measures are documented.
- Architecture and costs — model choice, token spend per conversation, latency, logging, fallback paths — and whether anybody can see what the bot did yesterday.
- Fix list and test set — findings ranked by risk and effort, concrete fixes, and the test set itself — so you or your vendor can re-run it after every change.
How it works
01
Scoping call
What the bot is for, who built it, what it can access. We agree the scope and a fixed price.
02
Access and sampling
Read access to the bot, its configuration and an export of conversations. We sample by topic, not at random — failures cluster.
03
Testing
Quality runs against the test set, red-team sessions against the live system, document review for GDPR and AI Act. Typically 1–2 weeks.
04
Report and walkthrough
A scored report, a prioritized fix list and a 60-minute walkthrough with the people who own the bot. Fixes are optional — by you, your vendor or us.
Specifications
| Typical stack | Custom test set, LLM-as-judge plus manual grading, red-team playbook (OWASP LLM Top 10, OWASP Agentic Top 10), document review |
|---|---|
| Hosting | Your system stays where it is — we need read access and a conversation export |
| Integrates with | Any chatbot or agent: vendor platforms, agency builds, in-house, n8n / LangChain / LangGraph |
| Prototype timeline | 1–2 weeks, fixed price |
Eight questions before you call us
Answer honestly. Nothing is sent anywhere — the check runs in your browser. Open areas are where an audit starts.
Frequently asked questions
Our chatbot came from a vendor. Can you still audit it?
Yes. We treat it as a black box where we must and read configuration where we can. Most findings — wrong answers, missing escalation, leaked data, missing AI disclosure — show up from the outside.
Which legal duties apply to a customer-facing chatbot?
Under the EU AI Act, Article 50 requires that people are told they are interacting with an AI — in force since 2 August 2026, with fines of up to €15 million or 3 % of worldwide turnover. GDPR applies to every conversation that contains personal data. Most customer chatbots are not high-risk, but you should know rather than assume.
What is prompt injection, and why should an SMB care?
A message, document or web page that makes the model ignore its instructions — for example to reveal other customers' data or to trigger a tool it should not use. It is the top-ranked risk in the OWASP Top 10 for LLM Applications, and it is tested in every audit.
Do you then sell us the fix?
The report is complete without us and names fixes any competent team can implement. If you want us to implement them, that is a separate, separately priced step.
CONTACT — HELLO@OLTEN.AI
Get a free 30-minute AI audit
We look at your processes, name the use case with the fastest payback and tell you honestly if AI is not the right tool. No slides, no obligation.
Or write directly: hello@olten.ai