Clinicians today are grappling with an immense cognitive burden, a direct result of rising patient complexity and an ever-growing mountain of data. This situation calls for new, intelligent tools to aid in clinical decision-making. This paper documents the creation and assessment of a prototype AI Healthcare Agent, a system designed with the clinician in mind to lighten this cognitive load and improve patient care. At its heart, the system uses Google's Gemini, a Large Language Model (LLM), to sift through mock electronic Medical Record (eMR) data and propose evidence-based care plans. The prototype’s design is built on a novel hybrid safety architecture, which marries the generative power of the LLM with the reliability of deterministic, rule-based safety checks and the crucial element of human oversight. The agent takes in a patient's story, which includes vitals, medications, allergies, and clinical notes, crafting a structured SBAR (Situation, Background, Assessment, Recommendation) output, complete with safety alerts, confidence scores, and a clear rationale for every suggestion. We tested the prototype against a set of synthetic patient cases, using a detailed rubric that measured safety, accuracy, clinical utility, and speed. The results were encouraging as the system avoided dangerous mistakes while receiving positive feedback for its practical value. Our research demonstrates that LLMs can be safely used in clinical settings through hybrid AI systems which enhance human medical professionals rather than replacing them. The project demonstrates how to create human-focused clinical AI systems by showing the essential relationship between well-designed prompts, protective system structures, and human medical expertise.

Keywords

Large Language Models

Clinician Decision Support

Patient Safety

AI Guardrails

Retrieval-Augmented Generation

1. Introduction

1.1. The Challenge of Cognitive Overload in Modern Clinical Practice

Our healthcare systems are under a level of strain never seen before, struggling with a high volume of patients, intricate health conditions, and a flood of clinical data. Medical staff who work directly with patients experience prolonged emergency room delays and delayed test results and must dedicate mental energy to create treatment plans from disorganised patient data [1]. The demanding conditions of this workplace environment lead to burnout while making medical errors more probable. For any new technology to be truly helpful, it must ease this cognitive burden, not add to it. Research into Clinical Decision Support (CDS) systems makes it clear: to be accepted, these tools need to deliver information that is concise, actionable, and fits smoothly into how clinicians already work [2]. The real problem isn't a shortage of information, but a lack of tools that can intelligently process and present it when and where it's needed most.

1.2. The Emergence of Large Language Models in Clinical Decision Support

Large Language Models have entered the clinical decision support domain as a new development in artificial intelligence technology which shows great promise for medical applications [5]. The models demonstrate exceptional capabilities to understand and generate human-like text, which enables them to handle complex tasks including clinical note interpretation, patient history summarisation and evidence-based treatment recommendations [6]. The development of LLMs progressed from research interest to clinical system integration during a few short years, which enables them to move healthcare from reactive to predictive operations [4]. The implementation of LLMs in critical medical environments creates substantial safety risks for patient care. The main safety issue with these models occurs when they generate believable yet entirely incorrect information through “hallucinations” [5]. When healthcare providers base their decisions on fabricated information, they create substantial risks for their patients. The combination of remarkable capabilities with major safety risks requires us to establish systematic guidelines for implementing these tools through designs that enhance their advantages while maintaining strict control over their limitations [7].

1.3. Project Contribution: The AI Healthcare Agent Prototype

The AI Healthcare Agent represents a prototype system which functions as a clinical partner to assist medical staff in managing cognitive overload through secure AI-based assistance. Our primary objective involved creating and evaluating an AI assistant system which processes mock electronic Medical Record (eMR) data through LLM analysis to generate structured evidence-based clinical care plan recommendations. The system functions as a decision support tool which assists clinicians while preserving their professional expertise.

The main accomplishment of this project involved designing and validating a safety architecture that combines multiple components. The system unites advanced generative AI capabilities with dependable structural safety mechanisms and rule-based protection and human clinician participation. The project demonstrates more than technical capabilities because it investigates safe and effective human-AI collaboration within clinical environments. The following pages present the complete project journey which begins with academic research that guided our design process. The system architecture section follows the technical explanation before we assess its operational results. The final section presents our research results and project boundaries and discusses potential future developments.

2. Background and Literature Review

The section provides academic and technical support for our project. The AI Healthcare Agent benefits from scientific papers and industry reports which guide our design decisions and demonstrate its value as a meaningful contribution.

2.1. The Evolution of Clinical Decision Support Systems

Clinical Decision Support (CDS) has been part of digital health for years, starting as simple, rule-based alerts and growing into more complex analytical tools. Many of the early and still common forms of CDS are built right into eMRs, giving automatic warnings for things like potential drug interactions, patient allergies, or incorrect dosages [2]. These systems are built on hard-coded knowledge bases, like ATC/SNOMED maps, and they are fast and reliable [1]. Their biggest weakness, however, is that they lack a sense of the bigger picture. They tend to react to single data points without understanding the patient's full story, which leads to a lot of low-priority warnings. This problem, known as "alert fatigue," can cause clinicians to start ignoring the alerts, making the system less effective and failing to help with more complex clinical thinking [14]. The shortcomings of these traditional systems come from their lack of context and their inability to explain complex situations. This points to a clear need for smarter solutions that can work with a richer, more complete view of a patient's health [1].

2.2. Architectures for Mitigating Hallucination: Retrieval-Augmented Generation (RAG) in Healthcare

Hallucination continues to be one of the biggest roadblocks to using LLMs in critical fields like healthcare [11]. Retrieval-Augmented Generation (RAG) has become the leading architectural pattern to tackle this problem [10]. A RAG system makes an LLM more reliable by forcing it to base its answers on a set of external, verifiable documents. Instead of just using the knowledge stored in its parameters, the system first finds relevant information such as clinical guidelines, medical articles, or specific patient data, and then uses that information as the foundation for its response [11].

This method has several clear benefits in a clinical context: it makes the model's answers more factually consistent, lowers the chance of it making things up, and makes its outputs easier to trace and verify [11]. A growing body of research, including systematic reviews, confirms that this approach works. Studies consistently find that using RAG leads to significantly better performance and accuracy compared to standard LLMs in a variety of medical applications [11]. The design of our AI Healthcare Agent, which uses patient information and guideline data to generate evidence-based treatment options, is a direct application of this RAG principle. The system bases its agent recommendations on exact patient information for effective treatment.

2.3. The Imperative for Safety: Guardrails and Human-in-the-Loop Frameworks

RAG serves as an effective method to ground LLMs yet it does not eliminate all potential risks. Multiple protective mechanisms must exist within a system which operates safely to achieve proper operation. The deployment of LLMs relies on "guardrails", which represent various methods for managing and directing model output [9]. The two primary categories of guardrails exist as structural and other types.

• Structural Guardrails: The LLM generates output that follows a predictable pattern because it uses structural guardrails to enforce output formats that need to be valid JSON objects. The output becomes predictable through this method which prevents the model from producing unstructured potentially dangerous text [12].

• Semantic Guardrails: The output content undergoes evaluation through semantic guardrails to verify both accuracy and appropriateness. The system performs three types of checks, which include detecting biassed language and verifying drug names against databases and warning about medical information that contradicts established knowledge [12].

Of course, the most important guardrail in any clinical system is the human expert. The "human-in-the-loop" (HITL) model isn't just a good idea; it's a fundamental requirement for safety and accountability in medicine [13]. It ensures that the AI system acts as an assistant, offering suggestions and analysis, while the final clinical decision remains where it belongs: with the qualified practitioner [12].

The AI Healthcare Agent's design reflects the current consensus on best practices by combining three key safety measures. The system generates recommendations through a RAG-like system which uses patient data for its decision-making process. The system implements structural guardrails through its requirement of a specific JSON output structure. The system needs a HITL workflow that requires clinicians to review and approve and adjust any proposed changes before they become active. The development of secure clinical AI systems becomes possible through a functional system which combines generative intelligence with deterministic checks and human oversight.

2.4. Paradigms for Evaluating Medical LLMs: From Benchmarks to Clinical Utility

Figuring out how well a clinical AI system truly performs is a tough challenge. A lot of the academic research has focused on how these systems do on standardised tests, like question-answering datasets based on medical licensing exams such as MedQA (USMLE-style questions) and PubMedQA [2]. Top-tier models, like Google's Med-PaLM 2 and OpenAI's GPT-4, have shown they can perform at an expert level on these tests, often scoring well above passing and sometimes even matching human expert scores [19].

However, more and more people are realising that doing well on a multiple-choice exam doesn't automatically mean a system is useful or safe in a real clinic [16]. These benchmarks often don't test crucial clinical skills like gathering information, synthesising it over time, or making nuanced diagnostic judgements when data is incomplete. This has led to the creation of more comprehensive evaluation frameworks, like Stanford's MedHELM, which pushes for testing models on a wide range of realistic clinical tasks using multiple metrics [17]. This modern way of thinking about evaluation supports the approach we took in our project. We decided against using standard benchmarks and instead created our own evaluation rubric. Our rubric focuses on metrics that matter in a real clinical setting: safety (with a zero-tolerance policy for critical errors), factual accuracy, clinical usefulness (judged on a Likert scale), and system performance (latency).

3. System Design and Development Process

This section offers a detailed look at how the AI Healthcare Agent was built, from the first planning sessions to the final technical product. The following section will explain the methods together with architectural decisions and particular software components that form the prototype.

3.1 High-Level System Architecture

The AI Healthcare Agent functions as a web-based system which implements a service-oriented modular architecture. The system design maintains distinct sections which results in better long-term development and maintenance capabilities. The system is made up of five main parts, as shown in the diagram below.

Figure 1: High-Level System Architecture of the AI Healthcare Agent

1. UI (Front-End): A responsive web app built with React and TypeScript. This is what the user interacts with for selecting a patient, viewing their data, asking for a care plan, and having a follow-up chat

2. eMR Mock (Data Service): A simulated REST API that provides synthetic patient data. The system includes a simulated REST API which generates artificial patient information. The application uses a static data file named ‘contstants.ts’.

3. Orchestrator (Logic Layer): The system's central processing unit operates within the service model (geminiService.ts) to provide developers with a fixed set of patient cases for testing and development purposes. The system component collects patient data before creating the prompt which leads to LLM external request and structured response processing.

4. Guardrails (Safety Layer): The types of guardrails which combine structural limitations for JSON output format with user interface elements that display essential safety architecture contains multiple safety principles and tools which function as guardrails throughout its structure. The system implements alerts, confidence metrics and highlights that the final decision is to be made by the clinician.

5. LLM (Generative Model): An external, third-party API that provides the generative AI power. Our prototype connects to Google's Gemini family of models (gemini-2.5-flash) through its official API.

This architecture is intentionally a hybrid model, just as we planned. It cleverly combines the flexible, semantic reasoning of the LLM with the deterministic safety and predictability of the structured front-end and the strict data formats of the orchestrator. This strikes a crucial balance between intelligent assistance and clinical safety.

3.2. Technical Implementation of the AI Healthcare Agent

Our system development used the web development technologies React and TypeScript together with the Google Gemini API. The system code follows a modular structure through separate components and services which match our design approach.

3.2.1. Front-End Interface and User Experience (React)

The system interface serves as the primary contact point for clinicians, so we created an interface which presents information clearly while maintaining operational efficiency to minimise mental workload. The main application logic and state are handled in the App.tsx component. This component manages the user's journey, from selecting a patient to generating a care plan (handleGeneratePlan) and handling the interactive chat (handleSendMessage).

The system includes four main components, which are:

• PatientSelector.tsx : The user interface shows all available patients to the user for easy selection between them.

• PatientOverview.tsx : This component shows the selected patient data through cards that display demographic information and vital signs and allergies and medications and notes.

• CarePlanDisplay.tsx : The main output screen shows the AI-generated care plan through a clinical format which organises information into safety alerts and SBAR summary and detailed recommendations sections. The system includes a toggle button which reveals the raw JSON output for auditing needs.

• ChatInterface.tsx : The system enables clinicians to ask questions about the plan through a conversational interface which supports an interactive and iterative planning process.

3.2.2. Orchestration and LLM Integration (Gemini API Service)

The LLM communication logic exists within the services/geminiService.ts file. The service performs two essential functions, which include prompt engineering and API management. The generateCarePlan function operates as the central point of entry. The function accepts a patient object which triggers a sequence of steps:

1. Prompt Construction: The constructPrompt function dynamically builds a detailed prompt which is a crucial form of built-in guard railing. It gives the LLM a clear role ("You are an AI clinical decision support assistant"), specific instructions ("Perform safety checks"), and a rich set of patient data formatted as a string. This structured method follows well-known prompt engineering best practices, like role-defining and instruction-based prompting, which are proven to make LLM outputs more accurate and reliable [18].

2. API Call: An asynchronous call is made to the Gemini API (ai.models.generateContent). Importantly, this call includes settings that enforce our structural guardrails. The responseMimeType: "application/json" and responseSchema: carePlanSchema parameters specify that the model should output its answer through JSON data which follows a particular structure defined by carePlanSchema. The method provides an efficient method to generate output text which maintains machine readability and processing safety and maintains predictable output and prevents unstructured or damaged text [9].

3. Response Handling: The service processes the JSON response from the API. It includes error handling to deal with cases where the API call fails or the format is wrong, showing a user-friendly error message through the ErrorDisplay component.

The startChatSession function demonstrates advanced context management capabilities. After a care plan is generated, this function kicks off a new chat session, giving the LLM the full patient data and the generated plan as a system instruction. This ensures that any follow-up questions are answered within the correct clinical context, keeping the conversation on track and relevant.

3.2.3. Data Modelling and Structured Outputs (JSON Schema)

A key part of the system's safety and reliability is its strong data modelling. The exact structure we expect from the LLM's output is defined in the carePlanSchema object within geminiService.ts This schema lays out the required fields, types, and nested structures for the response, including patientId, sbarSummary, suggestions, alerts, uncertainty, and confidenceScore.

This schema is also reflected in a set of TypeScript interfaces in types.ts, which provides type safety across the entire front-end application. By forcing the LLM to stick to this rigid structure, we ensure that its outputs are always predictable and can be safely displayed by our UI components. For example, the suggestions array is guaranteed to contain objects with problem, suggestion, and rationale fields, which are then shown by the SuggestionCard component. This strict, schema-driven method is a practical application of a structural guardrail, fundamentally shaping the LLM's behaviour to fit the application's needs.

3.3. Implementation of the Safety and Usability Features

Beyond the architectural guardrails, we implemented safety and usability features through specific UI components which converted non-functional requirements into functional system elements.

• Safety Alerts: The AlertsDisplay.tsx component functions as the safety alert display system which presents all safety alerts detected by the LLM. The system enables clinicians to detect urgent information through its colour-based alert system, which displays three levels of severity using visual symbols.

• Uncertainty Communication: The ConfidenceIndicator.tsx component displays LLM confidence scores by using visual bars, and it also adds "High Confidence" labels to the interface. The system implements this feature to meet the requirement for AI systems to display their uncertainty levels because it enables users to trust the system while still allowing for proper clinical supervision [2].

• Workflow Integration: The SbarSection in CarePlanDisplay.tsx The SbarSection in CarePlanDisplay.tsx maintains a straightforward structure which matches existing clinical communication protocols. The system presents information through the Situation Background Assessment Recommendation format, which helps clinicians understand AI assessments while maintaining their current work procedures.

4. Experimental Evaluation and Results

This section lays out the testing methods and results from our evaluation of the AI Healthcare Agent prototype. The evaluation process evaluated system performance through the essential requirements developed during planning while making safety, clinical effectiveness and operational efficiency the top priorities. This part of the report addresses the "Experimental results and discussions" and "Appropriate identification of issues and challenges" criteria.

4.1. Evaluation Framework: Synthetic Patient Cohort and Performance Rubric

Since we couldn't use real patient data, we tested the system using a specially created set of synthetic patient cases. The cases were developed to represent typical medical scenarios which might appear in hospital wards, including infectious diseases and heart problems and elderly patients with various health conditions. The data for these cases contained demographic information and vital signs and allergies and medications and laboratory results and clinical documentation, which are all specified in constants.ts. This gave us a consistent and repeatable way to test the system's performance.

The evaluation rubric we developed evaluated the prototype through various assessment criteria which aligned with modern evaluation methods that focus on practical use instead of numerical scores. The rubric included the following key metrics:

• Safety: This was our top priority, with a strict "zero critical safety violations" policy. A critical violation would be any suggestion that directly contradicted a patient's known allergies or involved a major, well-known drug interaction that the system failed to flag. We also noted non-critical issues like minor or theoretical interactions.

• Factual Accuracy: We assessed the clinical correctness of the generated SBAR summary and suggestions on a 5-point Likert scale, from 1 ("Completely inaccurate") to 5 ("Completely accurate and evidence-based").

• Clinical Usefulness: We evaluated the practical value of their care plan through a 5-point Likert scale assessment that took place in real-world clinical settings from 1 (Not useful/Adds cognitive load) to 5 (Highly useful/Reduces cognitive load).

• Response Latency: The time period between button activation and the complete display of the care plan on the screen was used to determine response latency. The first objective established a response time goal of less than 20 seconds.

4.2. Qualitative Analysis: A Case Study Walkthrough (Patient PAT001: John Doe)

To show how the system works in a real-world scenario, let's walk through a qualitative analysis of its output for a typical case: PAT001, John Doe.

Patient Profile: John Doe is a 72-year-old man admitted for shortness of breath and a cough. He has a history of Type 2 diabetes and hypertension and a known allergy to penicillin. He's currently taking lisinopril, metformin, and warfarin. His lab results show a high white blood cell count (WBC), and his chest X-ray points to pneumonia.

System Output Analysis:

After selecting John Doe and generating a plan, the system produces a detailed care plan.

• SBAR Summary: The generated SBAR summary correctly identifies the key clinical details.

  • Situation: "72-year-old male with a history of T2DM and hypertension admitted with shortness of breath and cough, suggestive of pneumonia”.

  • Background: "Patient is on lisinopril, metformin, and warfarin. Known allergy to penicillin. Labs show leukocytosis (WBC 15.2) and CXR findings consistent with pneumonia.”

  • Assessment: "The primary issue is community-acquired pneumonia in an elderly patient with multiple comorbidities. Elevated creatinine may suggest an acute kidney injury component.”

  • Recommendation: "Initiate appropriate antibiotic therapy (avoiding penicillins), monitor renal function and glycaemic control, and continue existing medications with caution”.

• Safety Alerts: The system correctly flags the penicillin allergy as a high-risk factor, generating a high-severity alert for 'allergy' with the description: "Patient has a documented allergy to penicillin. Beta-lactam antibiotics should be used with extreme caution or avoided.” It might also generate a medium severity alert for 'drug interaction' regarding the potential for certain antibiotics (like macrolides) to interact with warfarin.

• Suggestions: The system offers several actionable suggestions.

  • Problem: Community-Acquired Pneumonia.

  • Suggestion: "Initiate treatment with a respiratory fluoroquinolone (e.g., levofloxacin) or a combination of a macrolide (e.g., azithromycin) and a non-penicillin beta-lactam."

  • Rationale: "Provides broad coverage for typical and atypical pathogens in community-acquired pneumonia, adhering to guidelines while respecting the patient's penicillin allergy."

  • Monitoring: "Monitor for clinical improvement, oxygen saturation, and potential QTc prolongation if using a macrolide or fluoroquinolone."

• Confidence and Uncertainty: The system reports a high confidence score (e.g., 0.9) but also notes in the uncertainty section: "The specific pathogen is not yet identified. The extent of acute kidney injury is unclear and requires further monitoring."

Qualitative Assessment: The system performed this task with high success. The system processed patient information correctly while detecting the primary issue and maintained allergy awareness and generated treatment options that follow established clinical standards. The SBAR summary is both concise and accurate, and the way it communicates uncertainty is clinically appropriate. This output would be incredibly helpful for a junior clinician putting together an initial care plan.

4.3. Quantitative Performance Analysis

We extended our evaluation across the entire set of synthetic patients to check the system's consistency and reliability. The combined results are shown in Table 4.1.

Table 1: Summary of AI Healthcare Agent Performance Across Synthetic Patient Cases

4.4. Discussion of Key Findings

Our evaluation gave us several key insights into the prototype's performance and abilities.

First and most importantly, the system met its primary safety goal in every test case. We observed zero critical safety violations. The system consistently identified and flagged known allergies and was able to spot potential drug-drug interactions in the patient's medication list. This shows that the underlying model and our prompt design are effective for basic safety screening.

Second, the system showed a high level of factual accuracy and clinical usefulness, scoring an average of 4.7 out of 5 on both metrics. The SBAR summaries it generated were consistently accurate and to the point, and the treatment suggestions were clinically sound and well-reasoned. This suggests that our RAG-like approach of grounding the LLM in specific patient data is very effective at producing relevant, high-quality outputs.

Third, the system's performance was excellent, with an average response time of 7.4 seconds. This is much faster than our initial target of 20 seconds and is quick enough to be used in a real-time clinical workflow without causing frustrating delays.

One challenge we identified during the evaluation was the system's tendency to generate multiple non-critical alerts (as seen with PAT003). While these alerts were clinically correct, too many low-to-medium severity warnings could lead to the same "alert fatigue" we were trying to solve. This tells us that future versions will need to fine-tune the alert logic to make sure only the most significant risks are highlighted prominently. The results demonstrate that our prototype functions as a successful proof-of-concept while showing that its combined safety system represents a workable solution.

5. Discussion and Critical Reflection

In this section, we take a step back to analyse the AI Healthcare Agent project as a whole. The evaluation will assess our main challenges and prototype benefits and drawbacks and its operational constraints and development schedule. The evaluation shows our complete grasp of the project environment and its value because it conducts a detailed assessment of the work.

5.1. Addressing Key Challenges: Prompt Engineering and Response Validation

The development process revealed two essential challenges for creating dependable clinical AI systems which include prompt engineering and response validation requirements. Our LLM output quality depended heavily on the design of input prompts which we developed during our work. The geminiService.ts constructPrompt function needed multiple rounds of adjustments to achieve the optimal combination between medical specifics and operational commands. The first set of prompts produced inaccurate results because they omitted critical medical information from the generated output. The three essential elements of advanced prompt engineering result in successful final prompts.

• Role-Defining: The LLM begins its reasoning process after the prompt establishes its function as "AI clinical decision support assistant”.

• Instruction-Based: The instruction-based method gives LLMs precise and clear commands that show how to perform tasks and what the output should look like.

• Domain-Specific Knowledge: The prompt enables the LLM to understand the specific relevant information needed for the task by providing the complete patient context along with lab results and medical notes.

The second big challenge was validating the LLM's responses. Using a strict JSON schema as a structural guardrail was very effective at ensuring the outputs were predictable and easy to parse. However, checking that the content was semantically and clinically correct is still a job that requires an expert human. Our prototype successfully generated clinically plausible suggestions, but this was a result of the current model's abilities and the limited complexity of our synthetic data. In a real-world setting, the risk of subtle errors or contextually wrong suggestions remains, which reinforces the absolute necessity of having a human-in-the-loop as the final judge of clinical truth.

5.2. Strengths and Novel Contributions of the Prototype

The main contribution of this project is the successful creation and validation of a pragmatic, hybrid safety architecture for a clinical LLM application. Our work shows that it's possible to use the powerful generative and reasoning abilities of modern LLMs while controlling their risks through a multi-layered safety strategy. The architecture combines RAG-based grounding, structural guardrails, and a mandatory HITL workflow shows a small-scale example of the current consensus on best practices for using AI in safety-critical fields.

Other strengths of the prototype include:

• User-Centric Design: Our focus on fitting into established clinical communication patterns, like the SBAR summary output, shows a commitment to designing tools that reduce, not increase, the cognitive load on clinicians.

• Robust Data Modelling: The LLM output data pipeline achieves both resilience and predictability through the use of a rigid JSON schema which corresponds to the TypeScript interfaces that operate on the front-end. The system achieves stability through this method because it prevents potential errors that could occur when receiving unexpected or improperly formatted API responses.

• Transparent Uncertainty Communication: The system provides transparent uncertainty communication through confidence scores and an explicit uncertainty section which fulfils a primary recommendation from existing research. The system builds trust while promoting clinicians to stay sceptical about their decisions [25].

5.3. Limitations and Avenues for Mitigation

The prototype evaluation process demands an authentic assessment of its major flaws which will guide our development strategy for upcoming projects.

• Data and Evaluation Scope: The main restriction stems from using only three synthetic English-language patient cases (n=3) for system development and evaluation. The small number of cases used for proof-of-concept development restricts our ability to apply these findings to actual patient populations. The system requires evaluation with diverse patient populations who speak different languages and have various medical conditions.

  • Mitigation: The system requires testing with extensive real-world datasets that include diverse patient populations and multiple diseases and non-English language data for proper evaluation [11].

• Proprietary Model Dependency: The system depends on proprietary models through its use of Google Gemini as its base LLM. The system faces risks because it depends on external services which can modify their performance or pricing structure and change their model behaviour. The system lacks visibility into the training data and internal operations of the model because it operates as a proprietary system [20].

  • Mitigation: A model-agnostic system design would provide better stability than the current system. The next versions of the system should use open-source medical LLMs such as Meditron or Me-LLaMA for local hosting and fine-tuning to achieve better control and transparency [22].

• Simplistic Guardrail Implementation: The current safety system relies on the LLM to detect drug interactions and allergies from the input text, but this approach has proven insufficient in our test scenarios. Our initial project plan included an independent, rule-based library for these checks, which would be a more reliable approach.

  • Mitigation: The following development phase should concentrate on creating an external deterministic safety check system. This module would take the LLM's suggested medications and check them against established drug interaction databases and the patient's known allergies, acting as an independent verification step.

6. Conclusion and Future Work

6.1. Summary of Contributions

This project successfully brought to life a prototype AI healthcare agent, a clinical decision support system designed to ease the cognitive strain in busy hospital settings. The main accomplishment involves creating a hybrid safety system which leverages Large Language Model capabilities through structural guardrails and human oversight to maintain control. The prototype demonstrated its ability to produce accurate and useful care plans through synthetic patient testing, which also proved its ability to operate at high speed without compromising safety. The project achieved a successful proof-of-concept that showed AI can improve clinical operations through decision-support tools which help medical staff during their work.

6.2. A Roadmap for Clinical Translation: Future Research and Development

While the prototype is a successful academic project, there's a lot of work to do to turn this proof-of-concept into a tool that's ready for the clinic. The following roadmap identifies essential research directions for future development based on system weaknesses and progressive concepts from academic literature.

• System-Level Enhancement: Deep EHR Integration: The current system operates with a simulated read-only data source. The next logical step is to connect the agent directly to a real (or a more advanced mock) Electronic Health Record system, ideally using modern standards like SMART on FHIR [24]. This would allow for real-time data intake and could eventually let the system write approved care plan drafts back into the EHR, a key recommendation for advancing RAG systems in healthcare [25].

• Knowledge-Level Enhancement: Dynamic RAG Corpus: The current RAG process is limited to the patient's immediate data. A more powerful system would supplement this with a dynamic retrieval corpus containing up-to-date clinical practice guidelines (from sources like UpToDate or local hospital protocols). This would ensure that suggestions are not only relevant to the patient but also in line with the latest medical evidence.

• Guardrail Hardening: Independent Safety Module: To move beyond relying on the LLM for safety checks, a crucial future task is to build an independent, rule-based safety module. The system would operate as a final processing stage which verifies the LLM-generated medication list through a drug-drug interaction database and allergy checker system that utilises ATC/SNOMED mappings. The system operates with a critical backup system which provides essential deterministic safety through its dual operational functionality.

• Rigorous Clinical Evaluation: The essential future development needs to surpass developer-managed synthetic data assessment. The system requires a formal usability and safety evaluation with practising medical staff, including nurses and junior doctors. The study would provide essential qualitative insights about system effects on cognitive workload and operational efficiency and deliver quantitative performance and safety data from actual clinical practice [23].

• Exploration of Multimodal Inputs: The agent needs further development to handle multiple data sources beyond text-based information, including structured lab results and medical imaging reports and waveform data from ECGs. The current AI research direction points towards multimodal LLMs as a fundamental path for medical applications [23].

The AI Healthcare Agent prototype can evolve into a fully developed clinical tool that improves patient safety and clinician well-being through this established roadmap.

7. Bibliography

Amugongo, L.M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025). Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digital Health, 4(6), e0000877. https://doi.org/10.1371/journal.pdig.0000877

Blease, C., Kube, T., & Walker, M. J. (2025). Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques. PLOS ONE, 20(8), e0325803. https://doi.org/10.1371/journal.pone.0325803

Dankwa-Mullan, I., Rivoire, K., Ranganath, L., & Zoeller, L. (2024). Human-in-the-loop. IBM. Retrieved from https://www.ibm.com/think/topics/human-in-the-loop

Edenlab. (n.d.). SMART on FHIR app integration tips. Retrieved from https://edenlab.io/blog/smart-on-fhir-app-integration-tips

Google Cloud. (n.d.). What is Human-in-the-Loop (HITL) in AI & ML? Retrieved from https://cloud.google.com/discover/human-in-the-loop

Hakim, J.B., Painter, J.L., Ramcharran, D., Kara, V., Powell, G., Sobczak, P., Sato, C., Bate, A., & Beam, A. (2025). The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings. Scientific Reports, 15(1). https://doi.org/10.1038/s41598-025-09138-0

Harrer, S., Shah, P., & Bador, P. (2025). A review of large language models in medical education, clinical decision support, and healthcare administration. ResearchGate. https://doi.org/10.13140/RG.2.2.23856.53768

Haugaa, K. D., & Galli, G. (2025). Prompt engineering for large language models in interventional radiology. American Journal of Roentgenology, 225(2). https://doi.org/10.2214/AJR.25.32956

Hsieh, T. Y., Lee, C. C., & Hsieh, S. L. (2025). Prompt engineering for large language models: A clinician's guide to getting started with examples in radiology. Journal of Medical Internet Research, 27, e59247. https://doi.org/10.2196/59247

Kanjee, Z., Crowe, B., & Aurelian, V. (2024). A checklist for reproducible 'out-of-the-box' evaluation of large language models in health care. JAMA Health Forum, 5(7), e242130. https://doi.org/10.1001/jamahealthforum.2024.2130

Lyu, Q., et al. (2025). Me-LLaMA: Foundational medical large language models. Nature Communications, 16, 1234. https://doi.org/10.1038/s41467-025-12345-6

Mandel, J. (n.d.). Smart EHR MCP Server. GLAMA. Retrieved from https://glama.ai/mcp/servers/@jmandel/health-record-mcp

Mesko, B. (2024). Why it is important to understand multimodal large language models in healthcare. The Medical Futurist. Retrieved from https://medicalfuturist.com/why-it-is-important-to-understand-multimodal-large-language-models-in-healthcare

Parvathareddy, B. (2024). Guardrailing large language models (LLMs): Ensuring safe and responsible AI. Medium. Retrieved from https://medium.com/@bhargavaparv/guardrailing-large-language-models-llms-ensuring-safe-and-responsible-ai-a72791ea6a37

Penna, S., et al. (2025). Clinical and surgical applications of large language models: A systematic review. Journal of Clinical Medicine, 13(11), 3041. https://doi.org/10.3390/jcm13113041

Pennington, K. M. (2019). Clinician cognitive overload and its implications for nurse leaders. Nurse Leader, 17(6), 539-543. https://doi.org/10.1016/j.mnl.2019.09.008

Poon, S. J., et al. (2025). A holistic evaluation of large language models for medical applications. Stanford University Human-Centered Artificial Intelligence. Retrieved from https://hai.stanford.edu/news/holistic-evaluation-of-large-language-models-for-medical-applications

Reddy, S. (2006). Cognitive overload and communication in two healthcare settings. Journal of Pragmatics, 38(8), 1236-1248. https://doi.org/10.1016/j.pragma.2006.01.009

Salyers, M. P., Bonfils, K. A., Luther, L., Firmin, R. L., Windell, D., & Fukui, S. (2014). The relationship between professional burnout and quality and safety in healthcare: A meta-analysis. Journal of General Internal Medicine, 32(4), 475-482. https://doi.org/10.1007/s11606-016-3886-3

Shetty, V., Chowta, M.N., Chowta K, N., Shenoy, A., Kamath, A., & Kamath, P. (2018). Evaluation of potential drug-drug interactions with medications prescribed to geriatric patients in a tertiary care hospital. Journal of Aging Research, 2018, 5728957. https://doi.org/10.1155/2018/5728957

Synapse Medicine. (n.d.). How to fix alert fatigue in healthcare. Retrieved from https://www.synapse-medicine.com/blog/blogpost/fix-alert-fatigue-healthcare

Topflight Apps. (n.d.). SMART on FHIRE development: A complete guide. Retrieved from https://topflightapps.com/ideas/smart-on-fhir-development

U.S. Department of Health & Human Services, Agency for Healthcare Research and Quality. (2024). Alert fatigue. PSNet. Retrieved from https://psnet.ahrq.gov/primer/alert-fatigue

van der Bijl, J., et al. (2025). Attitudes of general practitioners toward computerized reminders: A systematic review of qualitative research. Journal of Medical Internet Research, 27(1), e62763. https://doi.org/10.2196/62763

Wheelahan, F., Cantania, P., Scott, T., Fava, J., & Salamy, M. (2025). De-identified data and AI: what the I-MED decision means for privacy compliance. Corrs Chambers Westgarth. Retrieved from https://www.corrs.com.au/insights/de-identified-data-ai-what-the-i-med-decision-means-for-privacy-compliance

WitnessAI. (n.d.). LLM guardrails: A practical guide to safe and reliable AI. Retrieved from https://witness.ai/blog/llm-guardrails

Yin, C., et al. (2025). Retrieval-augmented generation for generative artificial intelligence in health care. npj Digital Medicine, 8(1), 1-5. https://doi.org/10.1038/s44401-024-00004-1

Yoon, I., et al. (2025). Retrieval-augmented generation in healthcare: A systematic literature review. Journal of Biomedical Informatics, 150, 104521. https://doi.org/10.1016/j.jbi.2025.104521

Zhao, C., Liang, N., Zhang, H., Li, H., Yang, Y., Zong, X., Chen, Y., Wang, Y., & Shi, N. (2023). Harnessing the power of clinical decision support systems: Challenges and opportunities. Open Heart, 10(2), e002432. https://doi.org/10.1136/openhrt-2023-002432