A caseworker has an eligibility file open. Sixty pages of history, two prior determinations, and a handful of scanned letters. There's an AI assistant docked in the right-hand pane, so she pastes the case narrative in and asks for a summary of what happened before. The summary comes back fast and reads accurately. She agrees with it, and denies the claim.
Now look at what the system wrote down. Caseworker opened case at 9:14. Caseworker recorded a determination at 9:31. That's the entire trail. Nothing in the record knows a model read the file, nothing captured what it said, and nothing flagged that the summary never mentioned the 2023 appeal that would have changed the answer.
The problem isn't the model. It's where the model sits.
A side panel is a separate system that happens to share a screen. Data gets into it by copy and paste, or by an integration that hands it a document and takes back text. Either way the exchange happens outside the workflow engine, which means the workflow engine has nothing to log.
This is a reasonable way to add AI to a platform that wasn't built for it. If the data model was designed around documents and folders years before anyone was embedding case narratives, a pane is what you can ship without rewriting the substrate. It's genuinely useful for drafting, and for answering questions about a document you're already looking at.
It stops being adequate the moment the output influences a determination that someone can appeal.
If a model shaped a decision and the audit trail can't show it, the agency is in a poor position to defend the decision.
Encryption questions don't cover inference
Most vendor security reviews ask good questions about data at rest and data in transit. Far fewer ask where inference actually happens.
When a caseworker pastes patient history into an assistant, that text is processed somewhere. Maybe inside your tenant. Maybe at a commercial model provider under the vendor's account, governed by the vendor's terms rather than yours. That distinction rarely shows up on a security questionnaire, and it's the one that decides whether the exchange sits inside your compliance boundary or outside it.
For a registry handling protected health information, that's a BAA question with a specific answer, and “the vendor is SOC 2 certified” isn't it. The certification covers the vendor's controls. It says nothing about which model provider saw the record, what that provider retains, or for how long.
Three things worth knowing before the pilot, not after
Where does inference run, and can it run inside our boundary if our data classification requires it? Agencies handling patient, juvenile, or law enforcement records should assume the answer matters to their regulator.
What does the audit log capture when a model is involved? The model's participation should be a logged event carrying the same weight as a user action, including what was sent, what came back, and which version produced it.
Who can turn it on? An assistant pane that any user can paste into is an unsupervised egress path, no matter how good the vendor's security posture is.
Intelligence inside the workflow gets logged like the workflow
The alternative is unglamorous. When classification, extraction, or routing happens as a step in a governed pipeline instead of in a pane beside it, the same machinery that already tracks the work tracks the model's part in it. Every stage is observable, every transform is attributable, and the audit timeline shows the decision path including the automated parts.
That's the posture AIRLIFT Connect takes with document and data movement today. Each import moves through queued, downloading, transforming, delivering, and completed, and every transition is recorded. Adding intelligence to a pipeline built that way extends something already audited. Adding it to a side panel creates a new surface that nothing audits.
Agencies evaluating AI features this budget cycle will see a lot of demos of the second kind. The question that separates them is short enough for a first call: when this makes a recommendation, what does our audit log say about it, and where did our data go to produce it?
If you're working through that question for a regulated workload, we're happy to talk it through.
