Why AI scribes still need human oversight

Artificial intelligence (AI) scribes are becoming more common in healthcare settings. These tools can be very useful in recording patient conversations and capturing clinical notes. But as the medical practitioner, your oversight is critical. While the lure of efficiency is strong, your primary responsibility is always to the patient and your professional duties. When it comes to who’s in charge, technology can’t be left to its own devices.

An AI scribe will ‘listen’ to your consultation and use speech-to-text to create a written transcript of the conversation. The AI can then process the transcript to identify clinically relevant information, and draft a clinical note including symptoms, diagnoses, medications and treatment plans.

It can also ‘hallucinate’. AI scribes can fabricate information or stories, mishear medications, or confuse symptoms.

• They can’t replace clinical judgement and shouldn’t be used to make diagnoses.
• They don’t import information from tests, scans or imaging reports, they are only a transcript of a conversation.
• They don’t recognise nuance or body language. AI scribes are not seeing the patient so they will miss speech nuances (hesitancy, reluctance etc), body language cues, the patient’s emotional state or even the subtleties of a long-standing doctor-patient relationship.

1. Get informed consent. You must get informed consent to use an AI scribe. It can take time to discuss the use of AI, so you may want to consider getting consent before the consultation. Patients should have adequate time to consider the information provided and give their consent. And consent can be withdrawn at any point in the consultation.  This should be documented if it occurs.

2.  Make a considered choice. Not all AI scribes are created equal. Choose a product that meets your needs and is fit for purpose. Some scribes are built for general practice, and some are built for hospital or clinic settings. You should also consider if the AI scribe you choose has had clinician oversight or not.

3.  Review the transcript: You must review the transcript for accuracy, completeness and to avoid AI hallucinations. It is your legal and professional responsibility to check this every time. There is no safe point where you can trust the AI to create a record without your review. You, and only you, are responsible for your records.

• Do you have enough information and evidence of the product’s safety, efficacy, and applicability for use in an Australian general practice context?
• Do you know where the data is stored and for how long? How is the data encrypted? Your patients may want to know the answer to this before they provide consent.
• Does this AI model use the data collected from your patient to train their next AI model? You should make sure your agreement prohibits the secondary use of patient data without explicit consent.
• Have you considered how long it will take you to review the AI notes? You might need to adjust your clinical workflows to allow time to check, correct and add other clinically relevant information.
• Have you checked the information in your records is correct? Does this accurately reflect the consultation that occurred?
• How will you record the patient’s consent to the use of the AI scribe? This is different form consent to a procedure and needs to be documented.

• Australian Health Practitioner Regulation Agency - Meeting your professional obligations when using Artificial Intelligence in healthcareopen_in_new
• Artificial Intelligence | Australian Commission on Safety and Quality in Health Careopen_in_new
• For health professionals | Digital scribes | Therapeutic Goods Administration (TGA)open_in_new
• RACGP - Artificial intelligence (AI) scribesopen_in_new
• New Resource: AI Scribes in Australian Healthcare - AIDHopen_in_new
• RACGP - Artificial intelligence in primary careopen_in_new (Position statement)