# Lecture 1: From predictive models to clinical systems **Channel:** Gabinet Kraszewskiego **Source:** https://www.youtube.com/watch?v=5mjj78V5xJc **Transcript page:** https://www.withtranscript.ai/video/5mjj78V5xJc ## Chapters - 0:08 — Course Overview and Clinical Utility - 6:41 — Biomedical Data and Model Representations - 13:37 — Time, Prediction, and Decision Pipelines - 21:20 — Metrics, Validation, and Data Leakage - 28:12 — Deep Learning for Images and Signals - 33:35 — Foundation Models and Language Systems - 40:55 — Multimodal AI and Integration Challenges - 44:30 — Explainability, Uncertainty, and Trust - 50:16 — Validation Pathways and Course Outlook ## Transcript **[0:08]:** Good morning everyone and welcome to artificial intelligence two. In this course we will look at artificial intelligence from a slightly different angle than an introductory machine learning course does. Yes, we will talk about algorithms, neural networks and foundation models, but the algorithm itself would not be the center of the story. The central problem is this. What happens when an artificial intelligence model leaves a controlled experiment and enters biomemed research or a clinical environment? A modern mayify an image, predict a probability, generate a report or answer a medical question. And that is only the beginning. We still need to ask whether the data were appropriate, whether the model was validated correctly and whether its uncertainty is understood. We need to ask whether its output can influence a meaningful decision. And what happens when the system is wrong. So today I want to give you a map of this entire landscape. **[1:10]:** We start with data and prediction, move through deep learning, language and multimodal systems and finish with validation, trust and clinical deployment. Let us begin with a question that will come back again and again in this course. When does an artificial intelligence model become clinically useful? It is tempting to answer with a metric 95% accuracy, a high AU, excellent sensitivity, or a very low error. Those numbers can be useful, but none of them on its own demonstrates clinical usefulness. Imagine two systems. The first predicts a disease with impressive discrimination, but it needs information that becomes available only after the final diagnosis has already been established. The second has slightly lower statistical performance, but it uses inexpensive measurements that are available at the first patient visit. Which system is more useful? Or imagine a model that produces a highly accurate probability, but clinicians cannot act on it. **[2:16]:** There is no defined decision threshold, no change in management, and no evidence that using the model improves an outcome. In that situation, we have an interesting predictive model, but not necessarily a useful clinical system. This distinction is fundamental. Model performance concerns the behavior of a statistical or computational model under specified conditions. Clinical utility concerns whether the complete system helps people make better decisions or achieve better outcomes in a real workflow. In between these two concepts lies almost everything we will discuss in artificial intelligence two validation calibration uncertainty explanability workflow integration safety regulation and human oversight. So whenever someone tells you that an AI model is excellent your next question should be excellent for what on which population under which conditions and compared with what alternative? artificial intelligence one was mainly about learning how computational models work. You met methods, algorithms and the basic logic of training and evaluation. **[3:28]:** Artificial intelligence two starts one level higher. We still need the technical foundations but the main question changes. It is no longer can we build the model. It becomes can we trust and use the system. That change may sound small. In fact, it completely changes the type of reasoning we need. Suppose you train a classifier. In a basic machine learning exercise, you might focus on feature selection, hyperparameters, and test set performance. In medical informatics, you must also ask where the data came from and whether repeated observations from the same patient appear in both training and test sets, whether prevalence is representative, whether the test population is independent, whether the prediction is calibrated, whether errors have asymmetric consequences, and whether the model will keep performing when the clinical environment changes. So we move from algorithms towards systems. The aim of this course is not to make you suspicious of artificial intelligence. **[4:36]:** The aim is to make you technically capable of telling a promising experiment from credible evidence and credible evidence from a deployable clinical tool. That distinction matters more and more because contemporary AI systems can produce outputs that look extremely convincing even when the evidence supporting their use is incomplete. This map shows the landscape we are going to explore. In one side we have biomemed data, laboratory measurements, clinical records, medical images, physiological signals, genomic information, molecular structures and data generated continuously by wearable devices and we have different computational approaches. Classical machine learning remains important. Deep learning has become central for images, signals, and large unstructured data sets. Foundation models allow representations learned from very large data sets to be adapted to multiple tasks. Large language models provide a natural language interface to information and reasoning. And multimodal models try to combine several of these information streams. But the map does not stop at the model. **[5:50]:** We also have validation, uncertainty, fairness, privacy, explanability and safety. And finally, the clinical workflow where an output must be interpreted by a human or by another system and translated into an action. Notice that the arrows between these areas matter more than the boxes themselves. A problem with data quality propagates into the model. A problem with validation affects our confidence in the model. A poorly designed interface can turn a statistically correct output into a bad decision and a technically impressive model can fail simply because it was evaluated on the population that does not represent the population where it is deployed. That is why medical AI is fundamentally a systems engineering problem. **[6:41]:** Before we go further, let us put some order into the terminology. Artificial intelligence is the broadest concept. It covers computational systems designed to perform tasks that we associate with intelligent behavior. Machine learning is one major approach within artificial intelligence. Instead of specifying every decision rule by hand, we estimate rules or functions from data. Deep learning is a family of machine learning methods built on multi-layer neural networks. Its importance comes partly from its ability to learn useful representations directly from complex data such as images, waveforms or text. Foundation models are another change in scale and in strategy. Instead of training one model for one narrow task only, we pre-train the model on a broad data set and later adapt it to many downstream tasks. Generative models produce new content, text, images, signals, structures, or combinations of modalities. Now, these categories overlap. A foundation model may be generative, but not every generative model is a foundation model. **[7:51]:** A transformer is an architecture. It is not a synonym for artificial intelligence. And an AI agent is better understood as a system that combines a model with tools, memory, planning, or actions rather than as a completely separate mathematical family. Keeping these terms precise matters otherwise discussions of AI quickly become discussions of labels rather than of technology. One of the biggest changes in biomedical artificial intelligence concerns representation. A computer does not directly understand a patient, a tumor, or an electroc cardiogram. It operates on representations of these objects. In classical statistical modeling, those representations were often selected by hand. We might use age, hemoglobin concentration, blood pressure, or tumor diameter. A more elaborate machine learning pipeline may engineer additional variables, ratios, spectral components, texture measures, morphological descriptors, or aggregated temporal statistics. Deep learning change this process because useful representations can now be learned directly from the input. **[9:04]:** A convolutional network can learn image features instead of relying only on manually designed texture descriptors and a network operating on a physiological signal may learn characteristic temporal patterns. Foundation models take this idea further. A large pre-trained model can map complex input into an embedding, a highdimensional representation that captures information useful across many tasks. However, learned representation does not eliminate the need for domain knowledge. The representation can steal in cold artifacts, confounding factors and biases. A network may learn that a portable radiograph has a particular acquisition pattern linked to a specific patient group. It may learn the hospital rather than the pathology. So moving from hand-crafted variables to learned representations gives us greater flexibility but it also makes careful validation more important not less important. Medical informatics is unusual because its data are extraordinarily heterogeneous. Consider one patient on cedar one patient. We may have radiological imaging such as X-ray, computed tomography or magnetic resonance imaging. **[10:25]:** We may have microscopic or hystopathological images. We may record ECG, EEG or photothismographic signals. We may have hundreds of laboratory variables, medication history, diagnosis, procedures and demographic information. And much of the clinically important information may be hidden in free text notes, not in structured fields. At another level, we may have genomic variance, gene expression profiles, proteomic or metabolomic measurements. At yet another level, continuous measurements from wearable devices. Each source describes a different part in the biological or clinical system. The important point is this. These are not simply more columns in one spreadsheet. The statistical properties differ. The sampling frequencies differ. The measurement errors differ. And missingness has different meanings. An absent laboratory test may itself carry information because clinicians order the test selectively. An image has spatial structure. A physiological signal has temporal structure. Clinical language contains uncertainty, abbreviation, and context. So medical AI begins long before modern selection. **[11:40]:** It begins with understanding how biomedical information was generated. If you misunderstand the measurement process, a sophisticated neural network will not rescue the analysis. It may simply model the misunderstanding more efficiently. This structural diversity helps explain why different AI architectures exist. Tabular data consist of variables arranged around observations. For example, age, sodium, hemoglobin, and C reactive protein for one patient encounter. For many such problems, treebased models and conventional statistical methods remain extremely competitive. Sequential data have order. In an ECG, the position of one waveform relative to another matters. In an electronic health record, the sequence of diagnosis, medications, and laboratory results may matter. Spatial data include images. Here nearby pixels have meaningful relationships and historically that made convolutional networks particularly effective. Graphs are useful when relationships are the object of interest, molecular structures, biological interaction networks or connections between clinical concepts. Text is sequential but it has additional linguistic structure. **[12:55]:** That is why transformer-based language models became so influential and more and more often we meet multimodal problems where a single model receives several types of information. So architecture should follow the structure of the problem. Do not begin a project by saying I want to use a transformer. Begin by asking what information is present, what relationships must be represented, what does the prediction target mean, and which architecture is appropriate for those constraints. It is a small methodological habit, but it prevents a surprising amount of bad machine learning. **[13:37]:** Medicine is fundamentally temporal, and yet many data sets turn the patient into a static row. one patient, one vector of variables, one label. That simplification can be useful, but it discards information. Imagine a C reactive protein value of 20 mg per liter. Its interpretation may be completely different depending on whether yesterday it was 5, 20 or 80. Or take a heart rate of 100 beats per minute. It means something different during exercise than during sleep. A creatinine measurement must be understood relative to previous kidney function and a tumor image becomes much more informative when it is compared with earlier examinations. So we move from a measurement to a trajectory. Once we model trajectories, new tasks appear. We can detect change, forecast future values, estimate time to an event, recognize deterioration or identify response to treatment. And then we reach intervention. This is where prediction becomes operationally important. **[14:41]:** A prediction made after the relevant decision has already been taken is useless. A prediction made early enough to change management may be valuable even if its statistical performance is slightly lower. Time therefore affects feature construction, train test splitting, leakage, evaluation and clinical utility. So whenever you design a predictive model, ask one very simple question. What information would genuinely have been available at the exact moment when this prediction is supposed to be made? Now we can examine the complete predictive pipeline. We start with patient data. These may be laboratory values, images, physiological signals, electronic health records or some combination of them. Before modeling, the data are transformed. We may remove impossible measurements, encode categorical variables, standardize numerical values, handle missing data, select features, or construct representations. This stage is often treated as technical housekeeping, but it is part of the model. If pre-processing is learned from data, it must be learned only from the appropriate training data. **[15:55]:** Next comes the predictive model. Its output may be a class label, but very often the more useful output is a score or an estimated probability. And here we reach an important distinction. A probability is not a clinical decision. Suppose a model estimates a 40% probability of a complication. What should happen next? Nothing in the number itself tells us whether to treat, order another test, admit the patient, repeat the measurement, or simply observe. We need a decision rule and that threshold should depend on the consequences of false positives and false negatives on the available intervention, disease prevalence, resources and the clinical context. For a screening test, we may prefer high sensitivity and accept more false positives. For an invasive intervention with substantial risk, the threshold may be very different. So the final step is not prediction. It is action. It is action and often that action involves a human. **[17:01]:** A clinician may combine the model output with information that the model does not contain. The model may support a decision rather than make the decision. This distinction is crucial when evaluating AI systems. A model can be statistically excellent and operationally irrelevant. Conversely, a moderately performing model may be valuable if it provides information at exactly the right moment in a workflow. So when you see a reported prediction score, mentally extend the pipeline. Ask yourself what happens after that score appears on the screen. There is a persistent assumption in artificial intelligence that a more complex model must represent progress. That assumption is wrong. A logistic regression can be the correct model. So can a random forest. So can a very large transformer. The question is not which architecture is most fashionable, but which approach solves the problem reliably. Simple models have important advantages. They can require less data and they can be easier to inspect, easier to validate, and easier to deploy. **[18:15]:** Complex models on the other hand can represent nonlinear relationships. They can extract information from data such as images, signals or language that simpler approaches cannot use effectively. But complexity has cost. It can increase computational requirements make failure modes less transparent and create more opportunities to overfit subtle properties of a data set. Most importantly, modern complexity is not evidence. evidence comes from the evaluation process. Imagine a neural network achieves slightly better internal performance than a simpler model. That is interesting. But if the improvement disappears in an independent hospital, the practical conclusion changes. Conversely, if a simple model generalizes robustly, is well calibrated, and can be integrated into a clinical workflow, it may well be the stronger engineering solution. In medical AI, sophistication should be demonstrated by evidence, not by the number of parameters. A good engineer should be able to defend why a particular level of complexity is actually necessary. **[19:28]:** Let me give you an example from work in which I was directly involved. Several years ago, we investigated whether routinely collected laboratory data could support prediction in inflammatory bowel disease. The motivation was straightforward. Endoscopy is clinically important, but it is invasive and it cannot simply be replaced by an attractive machine learning result. So, we asked a narrower question. Can patterns present in routinely available blood, urine, and fal measurements provide useful predictive information? The discovery data set contains 702 medical records from 372 patients including patients with ulcerative colitis and Crohn's disease. We evaluated several relatively conventional machine learning classifiers. The strongest models belong to the random forest family. In cross validation, the reported mean average precision reached 97% for Crohn's disease and 91% for ulcerative colitis. Those numbers sound impressive, but this is exactly where the critical thinking from today's lecture becomes important. This was a discovery cohort study. **[20:40]:** It did not demonstrate that a random forest should replace endoscopy. It did not establish a new approved diagnostic procedure. And the paper explicitly stated that prospective validation was required. That is the lesson I want you to take from this example. Now the interesting result is not simply that machine learning produced a high number. It is that multiple routine variables contained combined information. Information that was not captured by looking at a single inflammatory marker alone. This creates a hypothesis for a useful clinical support system. **[21:20]:** But between a promising discovery model and a routine diagnostic tool, there is still a long pathway independent validation, prospective evaluation, workflow testing, and ultimately evidence of clinical benefit. To evaluate biomedical models properly, we need to understand what the metrics actually measure. Let's start with sensitivity. Sensitivity asks among people who truly have the condition, what proportion does the test or model identify? Specificity ask the complimentary question. Among people who do not have the condition, what proportion is correctly identified as negative? Both quantities describe the behavior of the classifier relative to the true condition. Now consider positive predictive value. It asks when the model says positive, how often is that result actually positive? Negative predictive value asks the analogous question for negative predictions. And here's the important difference. Predictive values depend strongly on prevalence. A test can have the same sensitivity and specificity in two populations yet very different positive predictive values. **[22:31]:** If the disease is common in one population and rare in the other, precision and recall which you know from machine learning map closely onto some of these diagnostic concepts. In a binary diagnostic task, recall for the positive class corresponds to sensitivity. Then we have curves rather than single thresholds. The ROC curve shows the trade-off between sensitivity and false positive rate as the decision threshold changes. The AU summarizes discrimination across thresholds. Precision recall codes can be particularly informative when the positive class is relatively uncommon. DEF there is no universal best metric. If missing one dangerous disease is a very high cost, sensitivity may be critical. If unnecessary follow-up procedures are invasive, specificity becomes important and if prevalence is low, accuracy may become actively misleading. So do not ask what is the score of the model. Ask instead which error matters in this test, which population are we considering and which metric actually represents that question. **[23:44]:** AU is useful, but it answers only one question. Can the model rank higher risk observations above lower risk observations? That property is discrimination. A model may discriminate extremely well and still produce poor probabilities. Imagine that patients who eventually develop an outcome generally receive higher scores than patients who do not. The ranking may be excellent, but suppose a predicted risk of 80% corresponds in reality to only 40% of patients developing the outcome. The ranking works, but the probabilities are not calibrated. Calibration matters when probabilities are used for decisions. If a treatment is recommended above a particular estimated risk, systematically biased probabilities can translate directly into inappropriate actions. Then we have external validation. Internal validation estimates how a model behaves under resampling or on held out data originating from the same development environment. External validation asks whether the model still performs when we move to another data set, another time period, another institution or another clinically relevant population. **[24:58]:** And this is where many apparently excellent models become less impressive. Finally, we'll reach critical utility. Even discrimination, calibration, and external validation do not automatically prove that using the model improves patient care. The output must be connected to an actionable decision. We need to understand benefits and harms, thresholds, workflow, consequences, and the alternative to using the model. So this gives us four different levels of reasoning. Can the model separate cases? Are its probabilities meaningful? Does it generalize and does using it improve the decision process? These are not interchangeable questions. Whenever you see a paper whose conclusion is essentially our AU is high, therefore this model is clinically useful, you should immediately recognize that an important part of the argument is missing. High AU is evidence of discrimination. It is not by itself evidence of clinical usefulness. One of the easiest ways to build an apparently excellent medical AI system is to accidentally allow the model to see information that it should not have. **[26:14]:** This is data leakage. A classic example occurs during pre-processing. Suppose we standardize every variable using the mean and standard deviation calculated from the complete data set and only afterwards divide the observations into training and test sets. The test set has already influenced the pre-processing. Feature selection gives a more serious example. If we use the complete data set to identify the features most strongly associated with the outcome and then evaluate the model on a test subset, information from the test labels has indirectly influenced the model. Medical data create additional leakage problems. One patient may have several visits, several images, or many signal segments. If observations from the same patient appear in both training and test sets, the model may partly recognize the individual rather than generalize to a new patient. Temporal leakage is another major problem. A variable recorded after the clinical outcome cannot legitimately be used to predict that outcome at an earlier time. **[27:23]:** The general rule is simple. The stoope define the unit of independence and define the prediction time. split the data appropriately. Then fit every learnable part of the pipeline using training data only. That includes imputation strategies, scalers, feature selection, and many dimensionality reduction procedures. When possible, perform these operations within cross validation rather than once before it. Leakage is dangerous because the final metric can look perfectly reasonable. Nothing in an AU of 0.95 tells you whether the experimental design was valid. That is why reviewing a medical AI study requires understanding the pipeline, not simply reading the result table. **[28:12]:** Medical imaging was one of the areas in which deep learning first demonstrated particularly visible biomemed applications. The input is highdimensional, specially structured, and often difficult to describe completely using manually engineered variables. Different tasks require different outputs. Classification may ask whether an image contains a particular finding. Detection asks where an object or lesion is located. Segmentation assigns labels at the pixel or voxal level. For example, outlining a tumor, an organ, or another anatomical structure. and image to text systems may generate or support radiological descriptions. This distinction matters because the evaluation must correspond to the task. A classification AU tells us nothing about segmentation accuracy and the high dice coefficient for a segmentation algorithm does not prove that the resulting contour improves treatment planning. Imaging models are also vulnerable to data set specific shortcuts. scanner type, acquisition protocol, embedded annotations, image compression, or differences between hospitals can all correlate with the target. **[29:24]:** A network may exploit these correlations even when they have no biological meaning. This is one reason why external validation across devices and institutions can be so informative. Deep learning can extract extremely useful information from medical images, but the image is not produced in a vacuum. The acquisition process, the patient population, and the clinical workflow are all part of the modeling problem. Physiological signals give us a rather different type of biomedical information. An ECG represents the electrical activity of the heart over time. An EEG reflects electrical activity recorded from the brain. And a PPG records optical changes related to peripheral blood volume. What these signals share is structure on multiple time scales. We may be interested in individual events such as a QRS complex, an arhythmia episode or an epileptififor discharge. Or we may analyze rhythm, frequency content, morphology or changes over much longer periods. Traditional signal processing remains highly relevant. **[30:33]:** filtering, spectral analysis, and engineered features are not obsolete simply because neural networks exist. What deep learning adds is the possibility of learning representations directly from raw or minimally processed signals. The task may be classification, event detection, forecasting, anomaly detection, or representation learning. But signals are particularly sensitive to acquisition conditions. Movement artifacts, electrode placement, sampling frequency, sensor quality, and pre-processing can all strongly influence the data. Model trained on clean research recordings may behave very differently on wearable signals collected during everyday activity. Once again, generalization is the challenge. An algorithm may recognize a pattern under controlled conditions. A clinical or consumer system must recognize the relevant pattern despite noise, missing segments, different hardware and differences between people. And this is why signal processing knowledge and machine learning knowledge should complement each other, not compete. Convolutional neural networks and transformers represent two important ways of building structure into deep learning models. **[31:52]:** A convolutional network processes local neighborhoods with filters that are shared across positions. This gives the architecture a useful inductive bias for spatially structured data. Local edges combine at the shapes. Shapes combine it to larger patterns and across the layers increasing the abstract features emerge. Transformers on the other hand use attention mechanisms. Instead of restricting every operation to a local neighborhood, attention lets the representation of one element depend dynamically on other elements. This proved extremely powerful in language and it later became important in imaging signals and multimodel modeling. But we should avoid the simplistic statement that transformers replaced convolutional networks. Architecture selection depends on data scale and task. Convolution provides strong built-in assumptions that can be very efficient, particularly when data are limited. Transformers can capture long range relationships, and they scale effectively with large data sets and pre-training. And modern systems combine ideas, convolutional components, attention mechanisms, and hybrid architectures in the same model. **[33:04]:** For you as medical informatic students, the important thing is not memorizing which architecture currently wins a benchmark. Understand what information each architecture makes easy to represent. Then ask whether the data set is large enough, diverse enough, and appropriate enough to exploit that capacity. Because a powerful architecture cannot compensate for an invalid target or a biased data set. **[33:35]:** Foundation models change the workflow. We move from one model for one problem toward pre-training followed by adaptation. A model is first trained on a broad data set often with supervised objectives that do not require a manually assigned clinical label for every example. In this process, it learns general representations. Those representations can then be adapted to downstream tasks through fine-tuning, prompting, linear probing, or other approaches. Why is this attractive in medicine? Because high quality labeled data are expensive. A hospital may possess millions of images, but far fewer expertly annotated examples of a rare condition. Pre-training can potentially use that broader information before adaptation to the specific task and the concept extends beyond language. Foundation models have been developed for radiology, pathology, of opthalmology, molecular biology, and multimodal data. But the word foundation should not be interpreted as universal. The quality of the representation still reflects the training data. **[34:41]:** If the pre-training population is geographically narrow, demographically unrepresentative or dominated by particular devices and clinical practices, the foundation model may carry those limitations into downstream applications. So large scale can reduce one problem while amplifying another. The paradigm shift is real. Pre-training can make one learned representation useful for many tasks. But the fundamental engineering questions remain unchanged. What information was used? Which population is represented? What is the downstream task? And has performance been validated in the environment where we actually want to use the model? Large language models introduce something unusual into medical artificial intelligence. a general natural language interface. The same model can summarize text, extract information, answer questions, transform documentation, and take part in the conversation. This flexibility makes LLMs attractive for clinical documentation, literature synthesis, patient communication, and decision support interfaces. But language creates an important psychological effect. We are extremely sensitive to fluent communication. **[36:01]:** When a system uses precise terminology and produces a coherent explanation, we tend to attribute competence to it and this can be dangerous. One of the most interesting recent research directions is conversational diagnostic AI. Take the AMIE research system which was designed specifically for diagnostic dialogue. In a 2025 study, it was evaluated in a blinded remote OCE style setting with simulated patients and compared with primary care physicians on multiple aspects of diagnostic conversation. The results showed that sophisticated language model systems can perform remarkably well in this type of controlled conversational evaluation. But the limitations matter just as much as the results. The interaction took place to text chat with simulated patients and the study itself explicitly cautioned that this setup should not be treated as representative of usual medical practice. This is exactly the distinction we need in AI too. The correct conclusion is not that the language model has replaced a physician. **[37:08]:** The correct conclusion is that conversational AI has reached a level at which rigorous clinical evaluation becomes both possible and necessary. Once again, strong performance in an experimental environment is the beginning of translational research, not the end. The central limitation of a language model fits into one sentence. Fluency is not correctness. A language model is optimized to generate plausible sequences of language. Modern models learn enormous amounts of factual and structural information during training and they can perform sophisticated forms of reasoning. But the output mechanism still contains no built-in guarantee that every statement is true. A model can generate an incorrect dosage, invent the reference, misunderstand the negation, or confidently combine facts that do not belong together. In medicine, this is especially serious because plausible errors may be difficult for a non-expert user to recognize. So, confidence in the wording should never be confused with confidence in the evidence. **[38:22]:** There is another more subtle issue. Humans also make errors. So the relevant evaluation is not simply whether an AI system ever makes mistakes. Every diagnostic or decision support system makes mistakes. We need to study what kinds of errors occur, how often they occur, whether they can be detected, and what happens when a human interacts with the output. Automation bias can make a user accept an incorrect recommendation simply because it comes from a sophisticated system. The opposite is also possible. Useful recommendations may be ignored because the user does not trust the system. So the problem is not just model accuracy. It is the behavior of the human AI system. Retriever augmented generation or R A tries to address one limitation of language models by providing external information at inference time. Instead of asking the model to rely entirely in information encoded in its parameters, we first retrieve relevant documents. **[39:27]:** These may be clinical guidelines, institutional procedures, scientific papers, or another curated knowledge source. The retrieved information is then supplied as context for generation. And this has several advantages. Information can be updated without retraining the entire language model. The response can be tied more closely to a defined knowledge base. And the system can potentially show the user which sources supported the answer. However, a does not create a truth machine. The retrieval stage can fail. The correct document may not be in the database. A semantically similar but clinically inappropriate passage may be retrieved. Important context may be split across documents and the model may misunderstand a retrieve passage or combine it incorrectly with other information. Source quality also matters. A perfectly functioning retrieval system that searches a poor knowledge base will simply retrieve poor evidence efficiently. So we have at least three components to validate. retrieval quality, generation quality, and the interaction between them. **[40:35]:** In a medical setting, we may also require provenence. Not only what answer did the system produce, but which evidence supported which claim. R A is therefore best understood as an architecture for improving grounding, not as a guarantee of factual correctness. **[40:55]:** A real patient is not unimodo. The radiologist sees an image. The laboratory provides numerical measurements. The cardiologist may analyze a waveform. The medical record contains diagnosis, medications, and text. And genomic information may add another layer. Multimodal artificial intelligence attempts to combine these views. Imagine a model assessing a patient with suspected disease. The image may show an anatomical abnormality. Laboratory values may indicate inflammation. The clinical note may describe symptoms and their duration. A physiological signal may provide functional information and genomic data may modify the prior probability of particular conditions. Individually, each modality is incomplete. In principle, a model that integrates them can construct a richer representation of the bash. So how do we actually do this? There are several strategies. Early fusion combines representations relatively early in the network. Late fusion lets modality specific models make intermediate predictions which are later combined. And more flexible architectures allow information from one modality to attend to information in another. **[42:16]:** But multimodality should not be confused with simply concatenating everything available. The model must learn which information is reliable, how modalities align in time and what to do when information is missing. The engineering challenge is significant. Still, this is an important direction because clinical reasoning itself is multimodal. Clinicians routinely integrate measurements, images, language, history, and context. The research question is whether AI systems can integrate these sources in a robust and clinically meaningful way. Multimodal models introduce additional failure modes. The first is missingness. A research data set may contain imaging, genomics, and laboratory data for every participant because the study protocol required them. Real clinical data are rarely that complete. So what should the model do when one modality is absent? The second problem is sampling rate. An ECG may contain hundreds of measurements per second. Laboratory data may be collected once per day. Imaging may occur once every several months and clinical notes appear irregularly. **[43:28]:** These screens do not naturally align. Temporal misalignment can therefore create subtle errors. A laboratory value collected after treatment should not be combined as if it described the patient before treatment. Then we have conflicting evidence. An image may suggest one diagnosis while laboratory findings suggest another. A useful system should not simply average contradictory inputs without representing uncertainty. Platet bias also becomes more complicated because every modality has its own acquisition process. A missing genomic test may correlate with socioeconomic or institutional factors. Imaging protocols vary by center. Documentation style varies by clinician and by language. Multimodal AI offers richer information, but it also combines multiple sources of bias, missingness, and measurement error. So, the more modalities we add, the more important it becomes to understand how the data were produced. **[44:30]:** Explanability is often proposed as the answer to a familiar problem. How can we trust a complex model? But the term actually covers several very different techniques. Feature importance estimates which variables influence model behavior at a global level. Shape values go further. They attempt to quantify how individual features contribute to one particular prediction relative to a reference. In imaging, salency maps highlight regions whose modification or presence is associated with the model output. An attention visualization shows where certain attention mechanisms allocate their weight. Now, these tools can be useful. They can reveal that the model relies strongly on an unexpected variable. They can help us identify possible shortcuts and they can support debugging and communication. But an explanation of a model is not necessarily an explanation of biology. Suppose a shape analysis shows that C reactive protein contributes strongly to a prediction. This means the model uses information associated with that variable. **[45:35]:** It does not prove that changing C reactive protein would causally change the patient's outcome. The same applies to images. A highlighted region tells us something about the model sensitivity to that region. It does not automatically demonstrate that the highlighted tissue is the biological cause of the prediction. And there's a deeper issue. An explanation can be visually convincing while being unstable or incomplete. So X AI methods must themselves be evaluated a explanability is valuable where it helps us understand model behavior, identify failure modes or support appropriate human interpretation. It becomes dangerous when a colorful explanation is treated as cause and evidence simply because it looks intuitive. So remember the equation on the slide. Explanation is not causality. A useful medical AI system should not only give us an answer, it should also help us understand when that answer is uncertain and uncertainty has different sources. Oh, althuric uncertainty comes from variability or noise inherent in the observations. **[46:50]:** Two patients can have very similar measured characteristics and still experience different outcomes. Epistemic uncertainty is about what the model does not know because the training data do not adequately cover a region of the problem. The distinction is useful even if a practical system does not estimate each component perfectly. Consider a model trained on adults that suddenly receives data from a pediatric population. The system may still produce a numerical probability because a neuronet network always produces some output. But what is the important question here? Whether the input lies outside the distribution on which the model was validated, a safe system may need the ability to defer. In other words, I do not know can be a desirable model output. Uncertainty can be represented through probabilistic modeling, ensembles, basian approximations, conformal approaches or other techniques depending on the task. But here is a conceptual warning. A neural network softmax score is not automatically a trustworthy measure of uncertainty. **[47:56]:** Confidence must itself be validated. If a system is maximally confident on unfamiliar input, the user may receive precisely the wrong signal. So uncertainty estimation is not decorative metadata. In highstakes applications, it may determine when AI can act, when it can advise, and when it must hand the problem back to a human. Trustworthy medical AI is not one metric. The diagram shows several properties that must work together. Performance is necessary. Of course, a system that does not solve the task is simply not useful. Robustness asks whether performance remains acceptable when the environment changes. Different devices, different centers, slightly different data quality or unexpected perturbation. Calibration asks whether probabilistic outputs correspond meaningfully to observed risks. Explanability may help users understand model behavior and its limitations. Fairness ask whether errors and benefits are distributed appropriately across relevant populations rather than hidden inside one average performance value. Privacy matters because medical data are among the most sensitive forms of personal information. **[49:11]:** Safety requires us to analyze not only expected behavior but also credible failure modes and human oversight concerns a relationship between automation and responsibility. Now notice something. These properties can conflict. Stronger privacy protection may reduce access to the data needed for training. Later model complexity may improve predictive performance while reducing interpretability and a threshold set to maximize sensitivity may increase unnecessary interventions. So engineering involves trade-offs. Trust is not created by adding an explanation panel to a neural network. It is created by evidence that the complete system behaves appropriately under realistic conditions that its limitations are known that failures can be detected and that responsibility for action is defined. That is why trustworthy should describe a property demonstrated through process and evidence not a marketing adjective attached to a product. **[50:16]:** This pathway may well be the most important slide in today's lecture. Research is where we ask whether an idea is technically possible. We define a task, collect data, develop a model, and demonstrate initial performance. Internal validation estimates how reliably the model performs within the development setting. External validation tests whether it generalizes to data that were genuinely independent of development, another institution, another period, another population, or another relevant environment. Then comes perspective evaluation. Instead of reconstructing the past from an existing database, we examine the system in a workflow that resembles its intended use. These can expose problems that retrospective data sets simply cannot show. Missing information, delayed measurements, user behavior, interface problems, and changes in clinical decisions. Workflow integration is an engineering task in its own right. Who sees the output at what moment? How is uncertainty shown? Can the user disagree? What happens when data are missing? **[51:25]:** And who is responsible for followup? For systems that qualify as medical devices or otherwise fall under relevant regulation, regulatory and quality requirements become part of development, not an afterthought. Then comes deployment. And deployment is not the finish line. The patient population can change. Devices can change. Clinical practice can change. Data distributions can drift. Software can interact with new systems. Performance therefore needs monitoring. The complete life cycle is much longer than model training. And this explains why thousands of medical AI papers do not translate directly into thousands of routinely useful clinical systems. The transition from model to system requires evidence at every stage. A research AU may be the first result in that pathway. It is not the final result. Let us close the state-of-the-art section by looking at where current research is heading. One direction is the AI agent. Instead of producing a single answer, an agent can perform a sequence of actions, retrieve information, use software tools, analyze data, maintain state across interactions, and plan its next steps. **[52:43]:** In medicine, research systems are beginning to examine longer clinical interactions rather than isolated question answering. A 2026 extension of the AMIE research program, for example, studied multi-visit disease management reasoning in a virtual OCE style environment. The results were encouraging, but the authors again emphasized that further research was required before real world translation. That last point matters. The frontier is moving quickly, but an emerging capability should not be described as established clinical practice. Other directions include multimodal clinical assistance, continuous monitoring systems, digital twin concepts, and AI assisted scientific discovery. Digital twins are particularly attractive conceptually, a computational representation of an individual or a biological system updated with data and used to simulate future states or interventions. But the term currently covers systems with very different levels of maturity. And the same is true for autonomous research agents. They may search literature, generate hypotheses, run computational analyses, and coordinate tools. **[53:59]:** But scientific autonomy raises questions about verification, provenence, and reproducibility. So the future is not simply more powerful models. It is more complex AI systems interacting with tools, data, humans, and real world processes. This first lecture was intentionally broad because it gives us the map for the rest of the course. In our next meeting, we will go deeper into modern deep learning and biomedical representations from convolutional networks to transformers. Then we will focus specifically on medical imaging and biomedical signals. After that we will examine clinical prediction, electronic health records, time series and decision support systems. The feast lecture will concentrate on validation, explanability and uncertainty, expanding several concepts we introduce today. Then we will dedicate the whole lecture to generative artificial intelligence, large language models and medical foundation models. The seventh meeting will focus on multimodal systems, how text, images, signals and omics can be combined. **[55:10]:** We will then move toward genomics, biioinformatics and drug discovery. The ninth lecture will concentrate on trustworthy medical AI bias, fairness, privacy, safety, and regulation. And finally, we will connect to research to deployment and examine agents, digital twins, and future AI systems. So today's presentation is not meant to answer every technical question. It is the conceptual skeleton. During the semester, we will return to each region of this map and replace the simple overview with methods, evidence, examples, and limitations. Let us return to the question we started with. When does an AI model become clinically useful? Not simply when it achieves a high score. A useful system begins with a meaningful biomedical or clinical problem. It requires data that genuinely represent the information available at the moment of prediction. The model must be evaluated with appropriate metrics and probabilities should be calibrated whenever probabilities are used for decisions. **[56:21]:** Performance must survive independent validation. Uncertainty and failure modes need to be understood. The output must appear at the right moment in a workflow and be connected to an action that can realistically improve a decision. And finally, the complete system must be evaluated, monitored, and used with appropriate human responsibility. That is why I want you to remember the final statement on the screen. A model is not a system. And a system is not automatically a clinical benefit. Artificial intelligence gives us increasingly powerful tools for extracting information from biomedical data. The role of a medical informatics engineer is to make sure that the technical power of those tools is matched by equally strong evidence, validation and system design. That is the perspective we will use throughout artificial intelligence.