Beeck Center urges government website fixes to improve AI answers

A resident asking an AI assistant about benefits or flood help may never open the government website behind the answer. Georgetown University’s Beeck Center released a guide on October 14 urging agencies to make that information easier for AI to reach and interpret, while keeping it current—a task that involves both website security and content design.

An answer can take three routes

Making Government a Trusted Source in AI Systems begins with a distinction: an assistant does not necessarily read the live web when answering. It may rely on information collected for model training, a search index, or a webpage fetched in response to a user’s request. Each route has a different schedule.

A crawler is an automated program that requests a webpage and reads what the server sends back. Training crawlers gather material that may influence a future model, not today’s answer. Search crawlers help assistants find information, but website changes can take time to appear. On-demand fetchers retrieve a specific page only when the assistant decides a live lookup is needed.

The guide focuses on three things agencies can control across those routes: access, accuracy and structure. Can the crawler reach public content? Is that content correct and current? Can the system extract it? The guide also acknowledges that answers depend on the assistant’s own retrieval, ranking and model behavior—not just the source website.

Two counties, different results

The guide illustrates the stakes with a question about public safety spending in Mendocino County, California. ChatGPT returned a figure 53% higher than the county’s budget tool showed. StateScoop’s coverage describes the guide’s possible explanation: the county’s JavaScript-powered interactive dashboard may have made the underlying figures harder for AI systems to retrieve reliably.

A contrasting example involved special-event permits in Larimer County, Colorado. ChatGPT correctly returned a timeline of 30–60 days, matching the county’s straightforward webpage. The guide recommends checking priority pages for information buried in images, complex tables or other formats that AI systems struggle to extract.

53% higherMendocino County public safety spending

The guide’s example compares ChatGPT’s answer with the figure in the county’s own budget tool.

Open access without opening the floodgates

Security settings create another barrier. A rule intended to stop malicious scraping can also block a legitimate assistant. But allowing more crawlers can strain hosting capacity or budgets, the guide notes. Its recommendation is tiered access, rather than treating every automated request alike.

• Allow crawlers from established providers with broad resident-facing use when their identities are verified.
• Evaluate established crawlers with narrower relevance or compliance concerns individually, considering their usefulness to residents and respect for crawling rules.
• Apply graduated responses—logging, challenging or blocking—to unverified or unlisted bots.

Verification matters because a bot can fake the name it presents in a request. The guide recommends stronger identity checks, such as published network addresses or signatures. Agencies should also check rule order: a geographic restriction can block a legitimate crawler before its verified identity is considered.

The work is divided among named owners. An IT or security contact should document crawler settings and confirm trusted bots get through. A communications or content owner should review high-priority pages for outdated, conflicting, incomplete or ambiguous information. A web or content owner should remove structural barriers, then test whether assistants correctly extract key details.

The recommended checks go beyond whether a page loads. Agencies should ask common questions and compare AI answers with authoritative page content. Corrective steps should be revisited annually. Even the guide’s list of major crawlers is explicitly a mid-2026 snapshot, not a permanent reference, because providers change products and crawling strategies.