How to Evaluate an AI Co-Clinician

Three questions settle most of the evaluation: how was it validated, who reviews the output, and where does the result land.

What should an administrator ask before buying an AI co-clinician?

Every vendor has a demo. The demo is not the evaluation. The evaluation is five questions, asked in order, with answers you can check.

The cost of getting this wrong is not the contract. It is a year of staff time spent on a tool that nurses route around, and a survey or quality review that finds nothing in the record to show for it. The questions below are the ones that would have caught it.

Validation: ask for the pre-registration

Ask whether the outcomes were declared before the test, and whether the test data was sealed. A model tested against data it has seen will look better than it is.

Ask for results per outcome, not per product. Scienza's foundation model for aging was evaluated under a pre-registered sealed test. Five outcomes passed. Falls replicated on a second look. Hospital return is a trained head without a sealed-test result. The manuscript has been submitted to JMIR AI.

An honest answer has a "not yet" in it somewhere. If every outcome passed and nothing is pending, ask again.

Ask what population the test was run on and whether the model had seen it. Ask what happened to outcomes that did not pass. A vendor who can describe a failed outcome is a vendor whose passed outcomes mean something.

Review: who decides

Ask what happens between the model's output and a change in care. The answer has to be a clinician.

Gia returns a ranked worklist with the reason beside each name and Proactive Decision Actions. Options, never orders. The clinician picks one, adds a note and assigns the follow-up. Nothing is ordered by Gia. A check-in is a screening conversation. Gia is clinical decision support, not a diagnosis.

Ask to see the audit trail for one result. It should show who reviewed it, when, what they chose and what follow-up they assigned. If the trail stops at the model's output, the tool is not ready for a surveyor.

Write-back: where the result lands

Ask whether your staff will log into a second system. If the answer is yes, the result will not be read.

Gia writes results, notes and follow-up to PointClickCare, Epic or your EHR. The follow-up record lives where your clinicians already work and where your surveyors already look.

Ask what the write-back looks like in your own chart, on your own screen. Ask whether structured assessments land in the window the MDS coordinator uses. If the vendor shows you a dashboard instead, ask again.

Data: whose project it runs in

Ask where protected health information goes and who holds the agreement. Gia operates under HIPAA with a business associate agreement signed with each customer. Results write to your EHR.

Gia Health Analytics is hosted by Scienza and sold through the AWS Marketplace and Google Cloud Marketplace. Your records never train the model. The two products share one rule: a clinician reviews every result. See governance for the full posture.

Ask who can see identified data and under what control. Families see updates after a clinician releases them, not before. Caregiver check-ins follow the same rule. The clinician is the gate for every output, including the ones that go home.

Workflow: what staff stop doing

Ask what a nurse stops doing on day one. If the answer is "nothing, but they get more insight," the tool adds work.

With Gia, staff stop placing check-in calls and stop chasing the ones that went to voicemail. Gia calls by phone or video, retries missed calls, runs the 40-second speech check-in and administers the BIMS, ADL and IADL in conversation when asked. Results arrive in under two minutes. Staff spend the time on the residents the worklist puts at the top.

Ask for the follow-up record as well. Every attempt, every result and every clinician decision, dated and attributed, in your chart. If the vendor cannot show you one from a running deployment, you are being asked to pilot the record-keeping as well as the tool.

Where to start

Write the five questions down and send them to every vendor on the list, including us. Compare the answers side by side before anyone sees a demo.

See how post-acute and senior living operators deploy Gia, then book a working session and put the questions to us directly.