Government AI Chatbot Procurement: 44 Questions Before You Sign

Explain your grounding architecture. How do you prevent answers that are not sourced from our content?

The technically correct answer involves Retrieval-Augmented Generation (RAG): the AI retrieves relevant passages from your indexed content before generating an answer, and the generated answer is constrained to stay within what was retrieved. Ask specifically whether the system can generate answers about topics not present in your content, and what happens when it cannot find a relevant answer.

What happens when a resident asks a question your content doesn't answer?

The correct behaviour is a clear, honest 'I don't have that information, here is where you can find help' response. The wrong behaviour is generating a plausible-sounding answer from the LLM's training data. Get the vendor to demonstrate this failure mode specifically with a question that is off-topic for your content.

Can the AI answer questions about topics outside your indexed content, for example general legal advice?

A well-built government AI chatbot should refuse to answer questions outside its knowledge scope. The refusal should be graceful and helpful, routing residents to appropriate resources. If the vendor's system answers general legal questions or provides advice beyond the agency's published content, that is a liability.

Provide examples of questions your system gets wrong, and what the failure mode looks like

Every AI system has failure modes. A vendor who claims their system never hallucinates has not tested it properly or is not being honest. Ask for specific examples of answer failures, how they were discovered, and what was done. The response tells you about their testing culture, not just their technology.

How do you test for hallucination risk before a customer goes live?

Reputable vendors run adversarial testing against every customer's content before deployment. This includes asking the system questions the content doesn't answer, asking boundary questions that could prompt hallucination, and checking answer accuracy against authoritative sources. Ask for the specific testing methodology and who performs it.

What citation mechanism does your system use? Does every AI answer include a source link?

Government AI answers should cite the specific page, document, or section that the answer is drawn from. Citations let residents verify the answer, let agency staff audit accuracy, and let you identify when outdated content is being cited. A citation-free AI answer is an unverifiable claim.

How do you handle conflicting information in our content?

Government websites frequently contain outdated pages that contradict current policy. Ask specifically what the AI does when it retrieves two pages with conflicting information: does it pick one, synthesise both, flag the conflict, or defer to the most recent publication date? The answer reveals the maturity of the grounding design.

Can we flag specific questions or topics as requiring human escalation?

Some questions should always go to a human: complex benefits disputes, legal threats, safety concerns, complaints. The system should allow you to configure topic-based escalation rules that override the AI and route directly to a human agent or a specific contact mechanism.

What is your answer accuracy monitoring process post-deployment?

Accuracy doesn't stay constant. Content changes, the AI's performance on edge cases evolves, and new questions emerge that stress-test the grounding. Ask how accuracy is monitored after go-live: are there automated red-teaming processes, human review queues, analytics that surface low-confidence answers?

Can residents flag an answer as incorrect or unhelpful?

Resident feedback is your best real-world accuracy monitoring. The system should have a visible and frictionless 'this answer is wrong or unhelpful' mechanism. Ask how those flags are processed: do they go into a review queue, do they trigger human review, and what is the response time for investigating flagged answers?

Where are conversation logs stored, for how long, and who can access them?

Conversation logs contain resident queries and AI responses. Residents asking about benefits eligibility, immigration status, or housing often disclose sensitive personal information. Know exactly where those logs go, whether they are subject to public records laws in your state, and who at the vendor organisation has access.

Are conversation logs used to train or fine-tune AI models?

Many AI vendors use customer conversation data to improve their models. For a government agency handling resident queries, this must be contractually prohibited. Require an explicit contractual provision: resident conversations are not used for model training, fine-tuning, or any purpose other than delivering the service.

Is your infrastructure US-hosted? Is government-specific infrastructure available?

Government and education organisations have data residency requirements. US-hosted infrastructure is the minimum for most SLED deployments. FedRAMP-authorised or GovCloud-hosted infrastructure is required for higher-sensitivity environments. Ask specifically about the underlying LLM API calls, not only the application layer.

How is PII detected and handled in conversation logs?

Residents frequently volunteer PII in chat conversations: name, address, case numbers, social security numbers. Ask specifically how the system detects and handles this: is PII masked in logs, is it flagged for deletion, is it separated from analytics data? 'We advise residents not to share PII' is not an acceptable answer.

What happens to conversation data when the contract ends?

Require a contractual data deletion provision with a specific timeline (30 days from contract end is standard), written confirmation of deletion, and an export option before termination. Verify that deletion covers all environments including analytics platforms, backup systems, and training pipelines.

Is the system compliant with your state's data privacy statute?

California, Virginia, Colorado, Connecticut, and a growing number of states have enacted detailed privacy laws that impose specific obligations on technology vendors contracting with state and local governments. Name your state's statutes and require written confirmation of compliance.

What access controls govern vendor staff access to our conversation data?

Vendor engineers and support staff should have no routine access to resident conversation content. Ask what technical controls prevent vendor staff from accessing live conversations, what audit logs exist for any access events, and whether access is policy-restricted or technically restricted.

For K-12 deployments: how does the system comply with COPPA?

If the chatbot is accessible to students under 13, COPPA requires parental consent before collecting personal information. Ask specifically how the platform prevents data collection from minors and what configuration controls are available to restrict chatbot access by user type.

How many languages does your system genuinely support, and what does 'support' mean?

Some vendors claim 100+ language support but deliver only UI translation while the AI still retrieves English content and translates responses. True multilingual support means the system retrieves relevant content in response to queries submitted in any language, synthesises that content accurately, and responds in the query language. Ask for a live demonstration in at least three languages relevant to your community.

How does the system handle queries submitted in a language your content is not written in?

Most government content is in English. When a resident asks a question in Vietnamese, does the system retrieve the relevant English content and translate the answer into Vietnamese accurately? Or does it return nothing because there is no Vietnamese content to retrieve? The correct answer is the former.

What is your process for identifying translation or language quality issues post-deployment?

AI translation quality varies significantly across languages. Spanish translation from well-trained models is generally strong. Translation for less common languages may be significantly weaker. Ask how the vendor monitors translation quality, whether they have native-speaker review processes, and how language quality issues are escalated and resolved.

How does the language detection work, and can residents select their language manually?

Automatic language detection is not perfect. Bilingual residents, residents who code-switch between languages, and residents using shared devices may be misclassified. The system should have an explicit language selector that residents can use to override detection. Ask how language selection state is preserved across a conversation.

Does your system support right-to-left languages?

Arabic, Hebrew, Persian, and Urdu are read right-to-left. UI components must render correctly for RTL languages, including chat bubbles, input fields, and navigation. Test this if your community includes significant Arabic or other RTL-language populations.

Can we configure specific languages to escalate to human agents?

For languages where AI translation quality is lower, or for culturally sensitive conversations, you may want to route directly to a human interpreter service. The system should allow language-based routing rules that escalate specified languages to phone or in-person interpretation.

Provide your WCAG 2.1 AA conformance documentation for the chat widget

Require third-party accessibility audit documentation, not a vendor assertion. The audit should cover keyboard navigation, screen reader compatibility (NVDA, JAWS, VoiceOver), focus management, ARIA roles, colour contrast, and timing-related interactions. A vendor who cannot provide audit documentation has not had the widget independently tested.

Demonstrate full keyboard operation of the chat interface

Request a live demonstration: open the chat with keyboard only, type a question, navigate to and read the response, follow a citation link, and close the chat, all without touching the mouse. If any step requires mouse interaction, the widget fails WCAG 2.1.1.

How are AI-generated responses announced to screen readers?

AI responses load dynamically. Screen readers need ARIA live regions to announce new content as it appears. Without this, a screen reader user submits a question and receives no indication that an answer has been provided. Ask specifically how response loading is handled for assistive technology users.

What is your process for resolving accessibility bugs in the widget?

Require a committed response time for accessibility bug resolution. WCAG failures that prevent screen reader users from using the chatbot are Severity 1 issues. The vendor should commit to resolution within 5 business days for S1 accessibility bugs, not the standard 30-day release cycle.

Does the chat widget meet colour contrast requirements across all states?

Interactive states, hover, focus, disabled, error, all must meet 4.5:1 contrast ratio for normal text (WCAG 1.4.3) and 3:1 for UI components (WCAG 1.4.11). Request the specific contrast values for each interactive state and ask for design documentation confirming they were validated.

Is the widget compatible with browser zoom at 200% and 400%?

WCAG 1.4.4 requires content to be readable when resized to 200% without loss of function. WCAG 1.4.10 (reflow) requires content to function at 320px wide without horizontal scrolling. Test the chat widget at browser zoom levels and on mobile viewports. Chat widgets that break the layout at zoom are a common WCAG failure.

How does the chatbot access our content: crawl, API, manual upload?

Crawl-based indexing (the chatbot automatically discovers and indexes your web content) is the lowest-maintenance approach. Manual upload requires ongoing effort to keep the knowledge base current. API-based integration is more precise but requires development work. Understand exactly how the AI accesses your content and how it stays current when you update it.

What is the indexing latency: how quickly does a content update appear in AI answers?

If you update a policy page at 9am, when can residents expect the AI to reflect that update? The answer should be minutes for critical content. An AI that reflects 24-48 hour old content is a liability for agencies that publish time-sensitive information.

Can we control which content the AI has access to?

You need the ability to exclude content from the AI's knowledge base: draft pages, staff-only documents, content under legal review, historical pages that have been superseded. Ask specifically how exclusion rules work: robots.txt, URL patterns, content tags, or manual exclusion lists.

How is the chatbot deployed: JavaScript snippet, CMS plugin, or API?

A JavaScript snippet deployment can go live on any website in minutes without IT involvement. CMS plugins are more maintainable for teams that manage content in a specific platform. API integration gives maximum flexibility for custom implementations. Understand the deployment method and the internal resources required.

Can the chatbot be embedded on specific pages or restricted to specific domains?

You may want the chatbot on your benefits portal but not your procurement website. Confirm that deployment scope can be controlled at the page or domain level without separate contracts or configurations.

What is the implementation timeline from contract to live?

For a crawl-based deployment on a standard government website, the honest answer is 3-10 business days. If a vendor says 6-12 weeks, ask what complexity drives that timeline. Either their architecture is genuinely complex (a red flag for maintainability) or the timeline is padded for professional services billing.

What analytics does the platform provide for the chatbot?

At minimum: conversation volume over time, top questions asked, questions escalated to humans, questions with no useful answer, resident satisfaction ratings per conversation. Ask whether analytics data is exportable and whether it feeds into your own data warehouse or BI tools.

What is the pricing model: per conversation, per user, per message, or flat licence?

Per-conversation pricing can be unpredictable for public-facing government deployments where chat volume spikes with news cycles or seasonal service demand. Flat-licence or tiered-volume pricing is more budget-friendly for government annual budget planning.

Are there usage caps and what happens if we exceed them?

A chatbot that stops working or degrades in a high-demand period is a liability. Understand exactly what happens at the pricing tier ceiling: does the system degrade gracefully, stop responding, charge overages, or continue at no extra cost? Get the answer in writing in the contract.

Is the platform available on any state or federal contract vehicle?

GSA MAS, NASA SEWP V, NASPO ValuePoint, and OMNIA Partners are the most common vehicles for state and local government technology procurement. A vendor on these vehicles eliminates most of the standalone RFP burden. Confirm exact contract numbers and authorised reseller names.

What are the contract exit provisions?

Understand the minimum contract term, the notice period for non-renewal, and the penalty for early exit. Also confirm data portability: if you exit, can you export your conversation history, configuration, and analytics data in a standard format?

Is there a pilot or proof-of-concept option?

A 30-day pilot on your actual content with real resident traffic is far more informative than any demo. Vendors with genuine confidence in their product welcome pilots. Require at minimum a 30-day no-obligation pilot period before any multi-year commitment.

What is the government escalation path for critical incidents?

Government agencies need a direct escalation path that bypasses standard commercial support queues. Ask specifically: if the chatbot is down during a peak service period, who do you call, what is the committed response time, and what remedies apply if the SLA is missed?

What training and documentation is provided for content administrators?

The person managing the chatbot post-launch, updating exclusions, reviewing flagged answers, monitoring analytics, needs complete and current documentation. Ask to see the actual admin documentation before signing. Evaluate its quality. Incomplete documentation creates operational dependency on vendor support.