A checklist is only useful if you can actually run it against a project and get a verdict. This one is built to be used: each item is concrete enough to mark pass or fail, and each carries a one-line justification so you know why it matters and can skip it deliberately when it does not apply. Run it before launch and again on a cadence afterward, because chatbot quality drifts.
The list is organized by phase, scoping, building, grounding, testing, launching, and operating, because the right questions change as a project moves. An item that is critical at grounding is irrelevant at scoping, and trying to check everything at once produces a list nobody runs. Work it in order, and treat any unchecked item as a known risk you have chosen to accept rather than one you forgot.
Scoping Checks
Get these wrong and nothing downstream can save the project.
The Job Is One Sentence
- The bot has a single, written, one-sentence job. A bot that needs a paragraph to describe its job is too broad to be good at any part of it.
- It is explicit about what it will refuse or escalate. Boundaries are a feature; an undefined boundary becomes a confident wrong answer.
- Your error tolerance is stated. Knowing whether occasional wrong answers are acceptable tells you how much to invest in grounding and testing.
Success Is Defined Before Building
- You have a measurable definition of a successful conversation. Without it, you cannot tell whether the bot works or just runs. The scoping logic is detailed in Inside One Support Team's Move to a Conversational Assistant.
Platform Checks
The tool should fit the job and not trap you.
Capability and Ceiling
- The platform can do the job at the simplest tier you need. Paying for power you will not use is waste; lacking power you will need is worse.
- You know the platform's ceiling. Knowing where it stops prevents the silent outgrowing that strands successful bots.
Portability
- Your instructions, knowledge, and conversation data can be exported. Portability is your only clean exit if the tool becomes a cage, a trade-off weighed in Hard-Won Defaults for Teams Building Conversational Software.
Grounding Checks
This is where accuracy is won or lost.
Knowledge Is Connected and Verified
- The bot is grounded in your real, current documents. Fluency without grounding is confident fiction.
- You have verified retrieval returns the right document for known questions. Grounding that is configured but unverified is grounding you cannot trust.
- Documents are structured for retrieval, not just human reading. Poorly structured docs cause retrieval to miss answers that technically exist.
Honest Fallbacks Exist
- The bot says it does not know rather than guessing. An admitted gap is more useful and less damaging than a confident error.
- Knowledge has a clear owner and a defined refresh trigger. Grounding decays the moment your business changes, so someone must own keeping it current.
Shaping Checks
The instructions are where behavior is born.
Role, Tone, and Boundaries
- The bot's role and tone are written down and consistent. An undefined personality drifts and confuses users across conversations.
- Boundaries and refusals are encoded in the instructions, not just hoped for. A boundary that lives only in your intentions is a boundary the bot will cross.
- The instructions are readable and version-controlled. Prompts drift over months of edits, and an unversioned prompt becomes something nobody dares change.
The Failure Path Is Designed
- The bot has a defined way to signal uncertainty and escalate. Failure handling is part of quality, since every bot is sometimes wrong.
Testing Checks
Friendly testing is not testing.
Real and Hostile Inputs
- You have an evaluation set of real questions with defined good answers. A standing set is the only way to catch silent regressions.
- You have probed adversarially: off-topic, rude, instruction-leak, and false-promise attempts. Hostile inputs are guaranteed in production, so rehearse them. The patterns are in Walking Through Bots That Shipped and the Ones That Stalled.
- Escalation rules are tested by triggering them several ways. Untested escalation is the classic source of embarrassing failures.
Launch Checks
Going live should be staged and observable.
Staged Rollout and Logging
- The bot launches to a small audience first. Real users ask what your test set never covered; learn it cheaply.
- Conversation logging is on from the first message. You cannot improve what you cannot see, and the test-to-real gap appears immediately.
A Path for Bad Answers
- Users can flag a wrong answer easily. A flagging path turns failures into improvement signal instead of silent attrition.
Operating Checks
The list you run forever, not once.
Ongoing Review and Refresh
- Logs are reviewed on a set cadence. Unreviewed logs mean drift goes unnoticed until users complain.
- Knowledge is refreshed when the business changes. Stale knowledge turns a good bot confidently wrong, as warned in Quiet Ways a Conversational Bot Project Goes Wrong.
- Cost and latency are tracked as volume grows. Both stay invisible until success makes them painful.
Integration and Action Checks
A bot that cannot do anything is often a bot not worth building.
Actions Work and Fail Safely
- The bot can take the actions its job requires, and you have tested each one. A bot that only talks frequently fails to beat a well-organized help page.
- Actions fail safely when something goes wrong. A booking that half-completes or a lookup that returns stale data should degrade gracefully, not silently mislead.
- The handoff to a human carries conversation context. A handoff that drops the thread punishes the user for trusting the bot.
Connections Are Verified End to End
- Every system the bot touches, CRM, calendar, database, has been tested with real, not mock, data. Integrations that work in a sandbox often break against production quirks, the kind of gap that stalls otherwise solid projects.
Security and Privacy Checks
The moment real data is involved, these stop being optional.
Data and Access
- You know where user data goes, how long it is retained, and who can see it. For regulated contexts this is a gating requirement, not a footnote.
- The bot cannot retrieve or reveal data outside its scope. Access boundaries are part of the build, and the consequences of skipping them appear in Quiet Ways a Conversational Bot Project Goes Wrong.
- Instruction-leak and manipulation attempts have been tested and blocked. A bot coaxed into revealing its rules or exceeding its access is a liability waiting to surface.
Frequently Asked Questions
How should I actually use this checklist?
Run it against your project before launch and then on a cadence afterward, marking each item pass or fail. Work it in phase order, since the right questions change as the project moves. Treat any unchecked item as a risk you are knowingly accepting, not one you forgot, which keeps the list honest.
Which checklist items matter most?
The scoping items, because everything downstream depends on them. A bot with one clear job and a defined notion of success can pass the rest; a bot without those cannot be grounded, tested, or measured meaningfully no matter how diligent the later phases are. Get scoping right first.
Why is verifying retrieval a separate checklist item?
Because grounding that is configured but unverified is grounding you cannot trust. Teams routinely connect documents and assume it works, then ship a bot that cannot find its own knowledge. Explicitly confirming the right document is retrieved for known questions catches a large share of accuracy problems before launch.
Should I run the whole checklist at once?
No. It is organized by phase on purpose, because an item critical at grounding is irrelevant at scoping. Trying to check everything simultaneously produces a list nobody runs. Work it in order as the project progresses, and revisit the operating section repeatedly after launch.
What is the most overlooked checklist item?
The operating checks, especially knowledge refresh and log review. Teams treat launch as completion, so the items that keep a bot good over time get skipped. A great bot that is never operated decays into a wrong one, which is why these items belong on a list you run forever.
Key Takeaways
- A checklist is only useful if each item is concrete enough to mark pass or fail; this one is built that way.
- Work the list in phase order, scoping, platform, grounding, testing, launch, operating, since the right questions change.
- Scoping items matter most; a clear one-sentence job and a defined success metric enable everything else.
- Verify retrieval explicitly, because configured-but-unverified grounding is grounding you cannot trust.
- Probe adversarially and test escalation rules; hostile inputs and untested escalations cause real failures.
- The operating checks run forever, not once; skipping them is how good bots quietly decay.