Transcript
Use with ChatGPT / Claude Auto-scroll A
Introduces the input-output contract detailing population, environment, and the mapping constraints before selecting a model architecture.
Speaker A
Foreign.
Welcome back to artificial intelligence 2.This lecture moves from the broad framework of lecture one to the technical foundations of biomedical representations and neural networks.Our first step is to turn the intended use, which we established in Lecture 1, into an input output contract today.That contract is a concrete description of of what a computational model receives and returns.The three blocks on the left connect the population and environment, the mapping from input to output, and the timing of available information.
Together they constrain the implementation before any architecture is selected.The target population determines which observations a contract covers.The clinicians identify the intended users.The technical environment determines how the input arrives and how the output can be delivered.An input output contract must describe meanings, not just five formats.
A model receiving a correctly formatted file can still receive the wrong examination, the wrong measurement scale, or information from the wrong time.
Shows how to specify input examinations, output meanings, units, and the timing of measurement versus availability in prediction problems.
Speaker A
Consider the example written on the slide and MRI scan is the input and tumor volume is the output.The input specification must identify the examination and the relevant image information.Rather than simply saying that the model accepts an image, the output specification must identify what volume is being estimated, its unit, and which examination it describes.A numerical value without those definitions is not a complete result, even if the software produces it successfully.
The clinical role completes the mapping.A volume estimate and a triage score are different outputs.Neither should silently become the other merely because both can be stored as a single number.If the software returns one numerical value for each input, what does that value actually represent?Does it describe a patient, an examination, or a particular structure within that examination?
That question fixes the unit of analysis.One patient may contribute several examinations, and one examination may contain several images.We need to know which of these objects corresponds to one model input and one output.Now look at the vertical timeline on the right.It places a biological event before the prediction decision point, while the laboratory result becomes available afterwards.
These are different moments in the information pathway.Measurement time describes when the observation was made.Availability time describes when its result could actually be accessed.The biological event, the measurement and the upload need not occur simultaneously.Imagine that the sample has already been collected.When the model is asked to produce a prediction, its laboratory result is uploaded later.A retrospective database may eventually place the completed result beside the earlier sample timestamp.The earlier sample timestamp does not make the later result available retrospectively at the decision point shown on the slide.That value cannot legitimately be treated as an observed input to the prediction.The practical contract therefore needs an availability rule.
For each input, ask whether it had become accessible by the specified prediction moment.Information that exists biologically is not necessarily information that exists computationally.The same logic applies to a derived feature.If calculating that feature requires a result uploaded after the decision point, the feature is also unavailable regardless of the timestamp assigned to its final database row.Could the input package be assembled at the actual decision point using only information already accessible?
Or does assembling it require knowledge that appears later in the completed record?This is an implementation check, not a claim about predictive accuracy.We can examine the required fields, their meanings, their associations, and their availability without training a neural network or measuring its performance.Our starting contract specifies the population, users, environment, input, object, output, meaning, and permissible information.These specifications will guide how we encode data and prevent a technically valid array from representing an invalid prediction problem.
Explains tensors as arrays with named axes and describes how tabular, image, signal, and embedding data are arranged for neural network input.
Speaker A
The four cards show how different biomedical inputs become tensors.We now move from the meaning of one observation to the arrangement of many observations in a numerical structure.For our purposes, a tensor is an array with named axes.Its shape gives the size of each axis.The names tell us whether an axis indexes observations, features, spatial positions, time points, or embedding coordinates.
Botch is the first axis in every example on this slide.It groups observations for processing together.It does not mean that those observations are consecutive moments in one patient's history or neighboring positions in one image.The position of a number within an array is part of its interpretation.A tensor shape describes organization, but it does not by itself describe the biological meaning of the stored values.
Start with the tabular card.It's to access a budget.Features.Each row represents an observation, and each column represents a specified feature.The feature order must remain consistent wherever the model is used.
Imagine a batch containing four examinations, each described by six measurements.The array has four rows and six feature positions.Those dimensions tell us how many entries exist, but not which measurement occupies each column.If two columns are exchanged without updating the contract, the array still has exactly the same shape.Yet the model now receives one measurement where it expects another.
A successful shape check would not detect that semantic error.Next, the image card has four batch channels.Height and width Height and width preserve a spatial grid.Channels distinguish different values associated with each location rather than introducing another spatial direction.A single channel image and the three channel image therefore differ along the channel axis.
Within either case, neighboring spatial positions remain explicitly indexed.The array tells subsequent operations where local neighborhoods can be found.Pixel indices are not physical distances.If anatomical scale matters, the meaning of the grid also depends on its spatial spacing and acquisition context.The shape alone cannot tell us how large a represented structure is.
The signal card uses batch time and channels within One observation the time axis contains ordered samples at each sample position.The channel axis distinguishes measurements from different recording channels.For a Hypothetical batch of four recordings, each with hundred samples and two channels, this convention gives 4x100x2.The sample order and channel identities must both remain consistent.Other software may place channels before time.
That alternative can also be correct.What matters is that the data pipeline and the model agree about the convention.Changing axis order requires a corresponding rearrangement of the entries.An ordered simple axis does not automatically encode elapsed time.Interpreting the distance between samples requires timing information such as the sampling interval.
Equal array steps and equal physical time steps are not interchangeable.Assumptions.The final code contains token embeddings.Its axes are batch, sequence length, and embedding dimension.A token occupies one sequence position, while its embedding is a vector containing several numerical coordinates.
Those coordinates are representation components, not automatically named laboratory variables or physical units.The sequence axis distinguishes tokens.The embedding axis distinguishes coordinates within the representation of each token.If an embedding contains many coordinates, does that necessarily mean the model has received more independent observations, or have we simply changed the numerical description of each existing input element?We have changed the description.
Increasing embedding dimension is different from increasing batch size or sequence length.These axes represent different objects, so their sizes should not be interpreted as equivalent quantities.We need each shape as a contract between data and computation.Observations by features, observations by spatial grids, observations by ordered samples and channels, or observations by tokens and embedding coordinates preserve both the axis convention and the meaning attached to it.
Covers numerical standardization, categorical mappings, and representation of missing observations with value-mask pairs.
Speaker A
Having specified the axes, we now specify what their entries mean.
This slide concentrates on three encoding, numerical standardization, categorical mapping, and the representation of missing observations.The equation on the right describes standardization.Take a value, subtract a reference mean, and divide by a reference standard deviation.The transformed value expresses distance from that mean on the reference scale.For an illustrative reference mean of 10 and standard deviation of 2, a value of 14 becomes 2.
It lies two reference standard deviations above the mean.This is a change of numerical coordinates, not a change in the underlying measurement.The phrase zero mean and unit variance refers to the data used to estimate those reference statistics.Subject to the variance convention, a different set of observations need not have exactly zero mean or unit variance after the same transformation.As a brief reminder from Lecture 1, estimate learned pre processing statistics within the training data.
Apply those fixed statistics afterwards re estimating them on incoming data would change the encoding that the model was trained to interpret.The division also requires a non zero reference standard Deviation.A constant feature needs an explicit handling rule rather than an undefined division.The formula is useful only when its numerical assumptions are satisfied.Standardization does not resolve unit mismatches.
If training values use one physical unit and new values use another, subtracting the same mean does not make them compatible.Unit conversion must precede applying the agreed numerical transformation.A standardized value of 0 means the reference mean.It does not generally mean that the original biological measurement was zero.Keep the original units and transformation parameters available when interpreting encoded values.
The second decision concerns categories.Nominal categories have distinct identities without an inherent numerical order assigning them consecutive integers can introduce distances and rankings that were never part of their meaning.Imagine three specimen categories stored using the codes 1, 2, and 3.Those codes identify categories.They do not imply that the third specimen is three times the first or that the second lies biologically halfway between the others.
One hot encoding avoids that particular ordering, but by allocating a separate indicator position to each category.A learned embedding is another possibility, representing category identities with vectors whose numerical components are learned.In either case, the mapping must remain consistent.If a category's code changes between data collection and model use, the same number may refer to different objects.Previously unseen categories also need an explicit encoding policy.
The third decision concerns missingness.Look at the first example.The value is 22.4 and its mask is 1.Under the convention shown, 1 indicates that the value was observed.The second example stores 0 with a mask of 0.
Here the 0 is a placeholder accompanying an unobserved value.It must not be interpreted as evidence that the biological measurement actually equals zero.How would we encode an observed value that genuinely equals zero?Would its numerical entry be enough to distinguish it from the missing example shown on the slide?Its value entry would be zero, but its observation mask would be one.The pair distinguishes observed zero from unobserved information.That distinction remains available even when both numerical value entries are identical.Providing a mask does not automatically make every model handle missingness correctly.The representation in the computation must use it consistently.Otherwise the placeholder can still be treated as an ordinary observed measurement.
Also distinguish missing observations from artificial padding used to align array lengths.Missingness describes an absent measurement at a meaningful position.Padding creates positions for storage convenience.Their indicators answer different questions.Complete encoding specifies feature definitions, units, reference statistics, category mappings, and indicator conventions.
With these details preserved, the array can carry not only numerical values but also the information needed to interpret those values correctly.
Uses clinical trajectories to illustrate how encoding can discard distinctions and create collisions that prevent models from recovering lost information.
Speaker A
This is an explicit reminder of the CRP example from Lecture 1.We will not repeat its clinical introduction.Instead, we will Use the visible trajectories to examine exactly what an encoding can discard.All three histories end at 20 milligrams per liter.
If the input contains only today's value, they receive the same numerical representation.Their different previous values are no longer accessible through that representation.The new technical point is representation collision.Different original observations can be mapped to the same encoded input.Once that happens, the model cannot distinguish them using information that the encoding has removed.
Sink of encoding is a function applied before the neural network.In this example, that function keeps the latest measurement and discards the earlier one.Three different histories therefore become one identical input value.For a deterministic network with fixed parameters, identical inputs produce identical outputs.Adding more layers does not change that fact.
The limitation is already present before any learned transformation is applied.The question on the slide about recovering or worsening cannot be answered from these trajectories alone.Rising, stable and falling measurements describe numerical change.They do not by themselves establish a patient's clinical state.Now consider what would preserve the distinction.
Keeping both measurements gives the model access to the earlier and later values.Keeping their timestamps makes the interval between those values available, as well as the three visible changes are an increase of 15 no change and a decrease of 60 milligrams per liter.These differences follow directly from the values on the slide.Without any fitted model or assumed clinical outcome, they say a difference is not yet a rate divide by the elapsed interval to describe change per unit time.Here, gradients refers to temporal change, not the parameter gradients used in neural network training.
If we retain the latest value and its change from the previous measurement, can we reconstruct the previous value?What additional assumption is needed if we also want to interpret the speed of change?Subtracting the change from the latest value recovers the previous value.Interpreting the speed requires the elapsed interval.This shows how a carefully chosen representation can preserve a distinction by without storing it in only one possible format.
However, that two value construction does not recover an entire longer history.Intermediate measurements may still be missing.Preserving one difference is not equivalent to preserving every fluctuation that occurred between the endpoints.The same distinction appears between changing an array's shape and changing its information content.Reshaping a stored latest value can place it inside a larger array, but it cannot recreate an earlier measurement that was discarded.
Imagine copying the latest value into several positions to satisfy a required input size.The resulting array contains more entries, but every entry comes from the same retained measurement.No earlier observation has been recovered, nor is every transformation necessarily destructive.Standardization with known non Zero scale parameters can be reversed.By contrast, keeping only the latest measurement cannot generally be reversed to recover an unknown earlier history.
This gives us a practical design.Identify two observations that differ in a task relevant way.Then ask whether the proposed encoding keeps them distinguishable.If both become identical inputs, that distinction is unavailable to the model.Any retained history must still satisfy the availability contract from the first slide.
We must preserve legitimate prior information, not add later results, simply because a completed database makes the trajectory look more informative.The lesson is not that every model requires a complete trajectory.It is that information loss must be intentional and compatible with the task.Architecture can transform preserved information.It cannot retrieve patient specific observations that never reach its input.
Introduces affine transformations, biases, nonlinear activations, ReLU, sigmoid, and softmax as building blocks of neural representations.
Speaker A
We now move inside the network.The central equation describes a layer that first forms an affine transformation of its input and then applies a non linear activation to produce a new representation.Read the formula as Multiply the input vector by a weight matrix, add a bias vector and apply the activation function.The result is a new vector labeled H that can become the input to another layer.Each row of the weight matrix specifies one weighted combination of input components.
Different rows can combine the same components differently.The bias adds a learned offset to each resulting component.The weights and biases are parameters.Their values are adjusted during training rather than fixed by the meanings of the original features.We will examine the loss and the update process on the following slides.
Huh.The first card calls this stage linear, but the more precise term for the complete expression is is Affine multiplication by the weight matrix is linear.Adding a bias allows the transformation to shift away from the origin.Why do we need the activation?Imagine two successive affine layers with no non linearity between them.
Substituting the first layer's expression into the second produces a combined matrix multiplication and a combined bias.The second weight matrix multiplies both the first transformed input and its bias.The resulting terms can be grouped into one effective weight matrix and one effective bias vector.The overall input output function remains affine.Stacking affine transformations does not by itself create a nonlinear function.
Depth changes the parameterization.But without intervening non linear operations, the composition can still be represented by one affine transformation.A non linear activation changes this situation.It lets the response to a weighted combination depend on which region of its input range is reached.Subsequent layers can combine those altered responses into more complex functions.
The middle card shows relu with a rectified linear unit.Its rule returns the large error of 0 and the incoming value.Negative inputs become 0 while positive inputs pass through unchanged.For example, an incoming value of -2 produces 0, whereas an incoming value of 3 produces 3.The change between these branches makes the whole function nonlinear, despite each branch being individually simple.
This rectifier can create sparse activations because some components become exactly zero.That is the mechanism behind the card's description of sparsity.It does not mean that all network parameters become zero or that every representation will be sparse.A zero activation here is not the missing value indicator from the earlier slide.It is a computed response inside the network.
The same numerical symbol can have different meanings at different stages of the pipeline.The final COD introduces sigmoid and softmax as output transformations.Before such a transformation, a classification head can produce a logit, a numerical score that that is not itself restricted to the probability range.For one binary target, Sigmoid maps a logit into a value between 0 and 1.A 0 logit maps to 1/2.
Increasing the logit increases that output while decreasing it moves the output towards zero.For mutually exclusive classes, softmax transforms several logits into non negative outputs whose sum is one.It expresses each class regardless relative to the competing classes.Several independent binary targets would require a different interpretation.These transformations produce probability shaped outputs, not a guarantee of reliable clinical probabilities.
As noted in Lecture 1, that reliability requires assessment.We will develop the evaluation methods in Lecture 5.The layered equation therefore supports a useful separation.Internal layers build representations while a task specific output head assigns the required output shape and meaning.The appropriate final transformation depends on the target.
Keep three operations distinct learn defined mixing, nonlinear activation and output mapping.Together they explain how a network can transform the information preserved in its input into a representation and then into a specified prediction.
Describes binary cross-entropy, mean squared error, the forward pass, backpropagation, batch updates, and epoch concepts.
Speaker A
The previous slide explained how a network produces an output.We now need a numerical objective that tells training which outputs to prefer.The two cards show losses for different kinds of target.
A loss compares a prediction with a reference target.The slide calls that target ground truth.Computationally, it is the recorded value against which the prediction is scored.Its definition must match the task.Calling a target ground truth does not make its source error free.
Training follows the supplied targets, including any mistakes they contain.The laws cannot independently decide whether a clinical label is correct.The left car shows binary cross entropy.Each target is either 0 or 1.The prediction is a probability assigned to the positive class rather than a continuous measurement such as pressure or volume.
Read the expression one observation at a time.When the target is one, the first term remains and the second disappears.The observation's loss becomes a negative logarithm of the predicted positive class probability.When the target is zero, the rows reverse.The loss uses the negative logarithm of one minus that probability.
That quantity is the probability assigned to the negative class.Both cases follow 1.Assign a larger probability to the observed target to obtain a smaller loss.The two terms but the same equation handle positive and negative examples.Consider an illustrative positive example.A predicted probability of 8/10 assigns more probability to its target than 2/10 does.Binary cross entropy therefore gives a first prediction a smaller loss.For a negative example, the preference reverses.A smaller positive class probability means a larger negative class probability.The loss therefore rewards the probability of the observed class, not simply large positive scores.
Assigning almost zero probability to an observed class produces a very large loss.The logarithm makes confidently wrong predictions.Costly this mathematical penalty should not be confused with a measured clinical harm.The right card shows mean squared error for a continuous target.First subtract the predicted value from the reference value square that difference.Then average the squared differences across the observations.Squaring removes the sign.An overestimate and an underestimate of the same magnitude receive the same squared penalty.This is a symmetry of the objective, not a claim that their clinical consequences are identical.As a numerical illustration, A pressure error of 5 units contributes 25 squared units.
An error of 10 contributes 100.Doubling the error therefore quadruples its contribution to this loss.That explains the card's emphasis on large A few large residuals can contribute substantially to the average.However, a large residual is not automatically an erroneous measurement or an observation that should be removed.Z squared loss also depends on the target's numerical scale.
Expressing the same target in a different unit changes the numerical loss.Comparisons require consistent target units and the same averaging convention.Both equations include a sum divided by the number of observations.Each observation contributes one term before averaging.This reduction turns many prediction target comparisons into the scalar objective used for an update.
As two reported losses have different target definitions, different units, or different averaging rules, can the smaller number alone establish that one model fits the test best?Better?No.The numbers must describe a comparable objective.A binary cross entropy value and a mean squared error value are not interchangeable scores merely because both accord loss.
Bottom paragraph connects this discrepancy to training.The loss supplies an objective.The training algorithm uses its dependence on parameters to change the network.Choosing the objective determines what those changes are trying to improve.Match the loss to the output and target binary probabilities for binary cross entropy.
Continuous values for mean squared error then distinguish the numerical training penalty from target reliability and from the clinical consequences of an error.This slide follows one training step through four Operations.The central list describes the cycle.The equation on the right specifies the update and, and the curves on the left show illustrative behavior over repeated training.First comes the forward pass.
Using the current parameter values, the network transforms the input and produces predictions.This calculation does not itself change the weights.It establishes the outputs to be scored.Second, calculate the loss against the supply targets.For a mini batch, Compute the individual losses and combine them.
Typically by averaging.The resulting scalar summarizes the objective for that group of examples.A mini batch is a subset processed for one update.Different batches can produce different gradients because they contain different observations.Batch size therefore affects both the computation required and the variability of the training signal.
An epoch usually means one pass through the training examples.It is not one parameter update.But unless the whole training set is processed as one batch, several updates can occur within a single epoch.Third comes backpropagation.It computes how the loss changes with each trainable parameter using the chain rule.Through the operations performed in the forward pass.These sensitivities form the parameter gradient.The chain rule connects changes across successive operations.A weight affects a layer's output.That output affects later computations.
Those computations affect the loss.Backpropagation combines the derivatives along these computational paths.Effect propagation calculates gradients.The optimizer uses those gradients to update parameters.These are different operations, even though they occur together within the training cycle.
In the displayed update, theta represents the trainable parameters.The gradient is taken with respect to those parameters.Eta is the learning rate, the the factor controlling how strongly the calculated gradient changes.Then minus sign matters.A non zero gradient points towards the locally steepest increase of the loss.
Moving against it gives a descent direction under the usual differentiability assumptions.For a simple numerical illustration, take one parameter equal to two, a gradient equal to three and a learning rate of 1/10.The update subtracts 3/10, giving a new parameter value of 1.7.A negative gradient would move that parameter in the opposite direction.The update is therefore not an instruction to reduce every weight.
Its direction depends on the loss's current sensitivity to that parameter.A descent direction is not a guarantee that an arbitrarily large step improves the loss.The gradient describes local behavior.After a large move, the relevant curvature and gradient can be different.This explains the learning rate warning on the right.
Very small steps can make progress slow.In a relatively flat region.Parameter changes may remain small even after many repeated updates.Steps that are too large can overshoot, oscillate or diverge.The appropriate scale depends on the objective and the optimization setup A larger learning rate is not automatically a faster route to a useful solution.
The left hand curves distinguish training loss from development loss.Training examples supplied the gradients development observations are used to examine the current model without using their losses for ordinary training updates.These curves are explicitly schematic, not experimental results.Their shapes illustrate why optimization progress and behavior on held out observations must be considered separately.A single mini batch update need not lower every subsequently measured loss.
After updating the parameters, another forward pass uses the new values.Repeating the cycle gradually changes the mapping during ordinary prediction.With a trained model, that mapping is applied without performing these training updates keep the sequence clear.Predictions loss gradients, then parameter changes.Batch composition determines the sample training signal, while the learning rate determines the scale of the update.The learning curves reveal what repeated steps achieved.The previous slide separated optimization from performance and other observations.Here the illustrative curves develop that distinction, while the four methods on the right describe ways to influence the learning process.
Discusses weight decay, dropout, data augmentation, early stopping, and how to audit training loss without equating it to generalization.
Speaker A
The horizontal axis counts training epochs.The vertical axis shows loss.
The training curve continues downwards.The development curve reaches a minimum and then rises as training continues.Here the words development and validation refer to the held out observations guiding development choices.The displayed pattern suggests that further improvement on training examples is no longer accompanied by improvement on those observations.This pattern suggests overfitting, but it does not prove that the model is fitting noise.Specifically, it may be exploiting other training specific details.Simply seeing different training and development losses is is not enough to establish the same conclusion.Weight decay introduces a tendency to shrink parameter magnitudes during updates.It counteracts the unrestricted growth of affected weights while the task gradient still pushes them according to the predictive objective.The card describes this as simplifying the model.
More precisely, it changes the preference of a parameter values.It does not necessarily remove layers, reduce the number of parameters, or produce a smaller architecture.Its strength must be chosen deliberately.Excessive shrinkage can restrict useful fitting.Insufficient shrinkage may have little effect.
Weight decay is a controllable part of training, not a guarantee that development loss will improve.Dropout changes a different part of the calculation.During training, selected activations are randomly disabled.The same example can therefore pass through different temporarily altered computations on on successive presentations.This discourages reliance on a particular combination of active units always being available.
Together, the parameters still belong to one network.The disabled activations do not represent permanent deletion of those units.In ordinary inference, this training time randomness is generally switched off with the appropriate scaling convention.Training in prediction modes must therefore be configured consistently.When using a network that contains dropout dropout can help, but its usefulness depends on the task and configuration.
Randomly disabling more activations is not automatically stronger evidence of generalization and can make fitting relevant information harder.Data augmentation changes the examples presented during training.Rather than only repeating identical inputs, we generate permitted variations.The purpose is to expose the learner to variation that should not change the target.Imagine an image classification example subjected to a small alteration.
If the relevant finding remains present and the assigned target remains valid, the altered image may be a legitimate additional training view.If that alteration removes the finding or changes a clinically meaningful property, retaining the original target can create an invalid example.Augmentation must preserve task relevant information, not merely increase the number of arrays.Early stopping controls how long optimization proceeds.Monitor held out development loss, retain parameter checkpoints, and use a specified stopping rule.When further improvement is not observed, the selected checkpoint need not be the final update.A checkpoint is a saved parameter state from a particular stage of training.Restoring the best observed checkpoint allows the model to use an earlier state rather than the potentially worse state reached when training stop the slides.Worrying about an optimal point needs a qualification.Early stopping selects the best observed checkpoint under the chosen criterion.It does not guarantee a true optimum or ensure that overfitting has been avoided.
Development loss can fluctuate, so a rule may allow several observations without improvement rather than hold at the first increase.Specify stopping and checkpoint selection consistently rather than adapting them after inspecting a favorable curve.These methods act at different locations, parameter magnitudes, internal activations, training inputs, and training duration.Their settings influence learning, but benefits must be assessed on held out observations not assumed from the method's name.
The two cards now turn the learning process into an audit question.A smaller loss is an observation about fitting.We need to examine what produced it before interpreting it as better generalization.The learning card states the narrow conclusion.Correctly decreasing loss indicates better agreement with the training targets under the selected objective.
It does not identify which features or relationships produce that agreement.In particular, prediction is not causal discovery.A network can fit associations without establishing why an outcome occurs.Nothing in minimizing the loss proves that changing a feature would change the clinical outcome.Imagine two networks trained for the same task.
One reaches a lower training loss.That difference could reflect useful predictive structure, greater capacity to fit individual examples, or an optimization process that reaches a better fitting parameter state.What would we need to know before concluding that the lower loss network is the stronger model for new observations?Were the objectives input information, training examples, and comparison conditions actually equivalent?Start by checking comparability.
Different loss reductions target Scales or included penalty terms can produce different reported values.A numerical ordering is meaningful only after establishing what each reported loss.Measures also distinguish how well a network can fit from how easily it is trained.A model that optimizes more efficiently can reach lower loss within a fixed budget without that difference proving a more useful representation.A training budget therefore needs a clear meaning.
Equal numbers of epochs, equal numbers of updates, and equal computing time are different constraints.Record the constraint used rather than treating them as interchangeable.The memorization card describes fitting details specific to the training examples, including noise and network need not store literal copies of records in a separate table for its parameters to encode and example specific associations.Zhang and colleagues demonstrated this possibility experimentally.Convolutional networks fitted randomly assigned training labels.
The result illustrates why training fit alone cannot establish that a learned relationship will extend to new examples.That research result is not a performance estimate for a medical application, nor does it imply that every low loss network has memorized noise.It demonstrates a possibility that an audit must distinguish from useful learning.Learning useful regularities, and memorizing.Particular examples are not necessarily mutually exclusive.A net worth may do both.Their relative contribution cannot be read directly from the training loss value.Consider a trainee examination with an incorrect label.Adjusting the net worth to agree more confidently with that label lowers its measured loss.The numerical improvement is real relative to the supplied target even though the target itself is wrong.
Next, examine behavior on held out development observations using a consistent prediction procedure.If the training improvement is not accompanied by improvement there, the lower training number does not support the same generalization claim.The reminder below the cards refers explicitly to Lecture one.When evaluating new patients, keep each patient's related records within one partition.Otherwise, patient overlap can make familiar observations look like evidence about unfamiliar patients.
For this training audit, the consequence is to distinguish examples from independent individuals.Many recordings may describe a small number of patients.An update can fit repeated patient specific patterns without providing evidence from additional people.Repeated observations can also influence the average objective disproportionately.A patient contributing many examples contributes many loss terms under ordinary example wise averaging.
Whether that weighting is appropriate must follow the defined test.Is the reported improvement comparable, supported beyond the training examples and evaluated at the intended unit of independence?Or could its apparent strength come from target errors, repeated identities, or easier optimization?A lower training loss establishes fitting under specified conditions.Audit the objective targets, optimization, budget development, behavior, and relationships between examples.
Neither memorization nor generalization can be diagnosed from that number alone.
Outlines inductive bias, local operators, spatial hierarchies, tokenization choices, and computational cost considerations for convolutional networks.
Speaker A
We now enter Part two, convolution and detention.The four points on this slide Organize the architectural questions.We will connect local feature maps, contextual representations, tokenization, and computational cost.As a brief link to lecture 1 convolution uses shared local filters.
While attention can mix information across positions according to the input, this part makes those broad descriptions more concrete rather than repeating their introductory comparison.The first point names inductive bias.Here bias means a built in preference or structural assumption that makes some functions easier to represent or learn than others.It is not automatically a mistake in a prediction architecture is therefore not a neutral container for data.Its connectivity and parameter sharing determine which relationships are available directly and and which require additional computational steps to represent.
A local operator starts from a neighborhood.This is a structural choice made before training.Training determines the operator's numerical parameters, but does not turn a fixed local connection into an unrestricted connection to every input position.The second point concerns spatial hierarchy and local feature maps.A feature map retains positions while describing responses at those positions.It is different from producing one summary number immediately from the entire input.A hierarchy allows later computations to combine information supplied by earlier ones.The important architectural question is how much of the original input can influence a particular later representation, not simply how many layers exist?Does the task depend on a small neighborhood, on the relative arrangement of several structures, or on both?Which connections allow the required information to meet within the network?
Locality should not be interpreted as permanent global blindness.Stacked computations can extend the available context.Equally, access to a larger region does not guarantee that its clinically relevant relationships have been learned.The third point introduces tokenization and global context.Tokenization determines the input units that the contextual operation relates to one another.
The choice of units therefore affects which distinctions are represented explicitly.Grouping a large legion into one unit provides a different description from dividing it into smaller units.This is not merely a change in sequence length it changes the granularity at which relationships can be expressed.Attention combines information from permitted positions using input dependent weights.The contribution of other positions can change between observations and unlike a fixed local neighborhood, access still depends on the architecture and its masks.
Which position should be allowed to interact, and what information identifies that arrangement.Does broader contextual access preserve the distinctions needed by the task or merely expose the model to more input?The word global describes potential access within the represented input, it does not imply access to to every record about a patient, and it does not override the input contract or information availability boundaries established earlier.The fourth point reminds us that contextual access has a computational price.In standard dense self attention, pairwise scores are formed across the token sequence.
The full comparison count grows quadratically with token count.Doubling the number of tokens therefore quadruples the full set of pairwise scores.This relationship concerns that interaction calculation, total runtime, and memory also depend on representation width, network depth, and implementation.Token granularity consequently connects representation design with resource demands.Smaller input units may retain finer distinctions but create a longer sequence.
More context is useful only if the computation remains feasible and and the distinctions matter.Both families still learn through objectives, gradients, and parameter updates.What changes is the computation producing the prediction comparisons should keep the task and available information consistent and specify training budgets.Our focus is the relationship between structural assumptions in the task, which local patterns are shared, how context expands or is mixed, what the input units preserve and and what those choices cost.These questions guide the detailed operators that follow.
Explains convolution kernels, stride, padding, and calculates receptive field growth through multiple layers.
Speaker A
This slide turns architectural questions into a calculation output, spacing, boundary treatment, and the region of the original input that can influence one computed feature.A convolutional kernel combines values within a local neighborhood using learned weights.Moving that kernel across the input produces a feature map.Stride determines how far the kernel moves between successive output positions.With stride one, successive neighborhoods begin one input position apart.With stride two, their starting positions are two apart.The second configuration samples output locations more sparsely it does not simply double the kernel's width.Padding specifies how the computation handles positions beyond the original array boundary.For example, zero padding supplies artificial zeros.Those entries allow additional boundary calculations but are not additional observed tissue measurements.
For an arithmetic illustration, imagine eight input positions, a kernel covering three consecutive positions, stride one and no padding.The kernel can start at positions one through six, producing six output positions.Adding one padded position at each end allows eight starts instead of the output length can then match the input length.The boundary outputs, however, involve the artificial entries as well as original observations.Receptive field asked a different which original input positions can influence one output feature?
It measures the extent of the computational dependency, not the number of positions in the output map.Keep output resolution and receptive field size separate.A map can retain its dimensions, while each feature depends on a progressively larger part of the original input.These quantities describe different architectural properties.Example on the left uses kernels of width 3 and stride 1.
For the displayed sequence, also assume dilation 1.Each kernel accesses consecutive positions rather than positions separated by gaps.The first layer one output depends on three input positions.At the second layer, one output combines three neighboring first layer features.Their overlapping input neighborhoods together cover five positions, not nine.
We can verify that union directly.The firstly first layer features depend on positions 1 to 3, 2 to 4, and 3 to 5 combining them gives access to original positions where 1 through 5.The third layer again combines three neighboring features.Now with overlapping five position dependencies.Their union spans seven original positions.This produces the visible progression from 3 to 5 to 7.The equation on the right generalizes that progression.Start with the previous receptive field size.Then add the current kernel width minus 1 multiplied by the product of the preceding strides.That stripe product describes the spacing of neighboring previous layer features when measured in original input coordinates.The additional span must use those coordinates, not assume that every later feature remains one original sample apart.Suppose an earlier layer uses stride two.Its neighboring outputs are spaced two original positions apart.A subsequent width three kernel therefore adds four positions to the receptive field span rather than the two added in the stride one example.
Initially, each input position has a field of one with all preceding strides equal to one, the stride product is one, and every width three layer adds two.
The displayed formula assumes ordinary undilated kernels along the calculation path dilation changes the kernel's effective span.Padding changes boundary alignment in which real input positions are available, rather than creating new biological context.The organ versus edge example motivates this calculation.A feature based only on a narrow neighborhood cannot directly compare distant structures outside its dependency region.A wider field makes that information computationally accessible.
Accessibility is not successful.Interpretation.A receptive field large enough to include an organ does not establish that the model recognizes that organization or detects a pathology correctly.That depends on what the learned computation actually does.Stride sets output spacing, padding handles boundaries, and receptive field tracks Input dependence.
The 3, 5, 7 progression follows explicit architectural assumptions.Potential context is not the same as learned clinical information.The preceding calculation followed one spatial axis.On this slide, the the same dependency growth appears in image grids where field width, covered area, and feature map resolution need to be distinguished.The conditions are written explicitly 3x3 kernels, stride 1 and dilation 1.
Each layer extends the dependency span by two positions along both spatial axes.The calculation is performed separately for height and width.The first field therefore spans 3x3 positions.After two layers it spans 5 by 5, and after three it spans 7 by 7.These are dimensions in the original input, not dimensions of the convolutional kernel itself.
A 3x3 kernel does not become a 7x7 kernel after three layers.The wider dependency comes from composition.Each layer receives features that already combine information from earlier neighborhoods.A third layer feature accesses neighboring second layer features directly.Information from distant original pixels reaches it indirectly through intervening computations.
Now compare width with area.The three disparate squared fins contain 9, 25, and 49 spatial positions, respectively.Although each side increases by 2, the number of covered positions does not increase by a constant amount.When the slide says that the receptive field grows linearly with depth, which quantity is linear?Is it the width along an axis or the total number of positions inside the two dimensional field?
It is the span along each axis.Under the stated conditions, the square area grows differently because it multiplies the two spans.Reporting a receptive field size should therefore make clear which dimensions are being described.These squares describe an architectural footprint, not a guarantee of equal influence.But from every covered pixel, learned weights and activations determine how information within that footprint contributes to a particular feature.
The model may use some available information strongly and other information weakly.The maximum region of possible dependence and the distribution of actual influence are different ideas.The diagram does not measure that influence.Also distinguish spatial coverage from channel count.Different output channels can represent different learned combinations within the same spatial footprint.
Increasing the number of such channels does not by itself widen that footprint.Output resolution can behave differently.Again with one padded position on each side and stride one 3x3 convolutions can preserve map height and width even as the receptive field expands across layers.For another arithmetic illustration, take a 9 by 9 input and use three successive 3x3 layers without padding.Their maps are 7 by 7, then 5x5, then 3x3.
Each feature in that final 3x3 map has a 7 by 7 receptive field.In the original input, the output map shrank while the input dependency of each individual output became wider Neighboring Final features still have overlapping receptive fields.They are not separate.Non overlapping summaries of the input map size and field size alone do not establish independence between those features.Border needs special attention when padding is present.
A nominal field can extend beyond the observed image.Part of its computational neighborhood then comes from the padding convention, not from additional acquired anatomy.Preserving array dimensions does not imply identical information quality at every position.An interior feature can depend entirely on observed pixels, while a boundary feature may depend partly on artificial entries.The local to global language on the slide describes expanding access to context.
Three layers in this example cover seven positions per axis.Whether that includes an entire structure depends on the input scale and spatial extent.How much relevant context reaches the feature, how densely is it represented, and where does the field meet the image boundary?These questions are more informative than calling a representation deep or global without specifying its geometry.The squares connect successive local operations with a wider input dependency.
Distinguish axis spans, covered area, channel count and output resolution.
Covers residual addition, global pooling, and conditions under which convolutional feature maps preserve or transform spatial shifts.
Speaker A
The linear progression requires the stated kernel.Stride and dilation.Two operations connect feature maps with task outputs.Here, residual addition changes flow through a block.
Global average pooling changes spatial organization.A residual block computes a learned transformation of its input and adds that transformation to a shortcut carrying the input.Read the operation as output input plus the learned residual.The shortcut and transform branch are combined element by element.Each spatial position and channel on one branch must correspond to a compatible position and channel on the other.Branch addition is not concatenation.Adding two compatible feature maps preserves their shape.Concatenating maps along the channel axis would instead place their channels beside one another and generally increase the channel count.Imagine two branches that each produce eight channels on a 16 by 16 grid.Their elementwise sum still has eight channels on that grid.It does not create 16 channels or a larger spatial image.Matching dimensions is necessary, but corresponding positions must also be aligned.Adding features from incompatible spatial grids does not acquire a meaningful interpretation merely because the arrays happen to have equal sizes.
If a block changes resolution or channel count, the shortcut needs an appropriate transformation before addition.An unchanged input cannot be added directly to an output whose spatial dimensions or channel count no longer match.
The residual formulation also permits a simple limiting case.If the learned residual is zero, the block returns its input through the shortcut.The transformed branch does not have to reconstruct that input from scratch during backpropagation.The shortcut provides a direct computational path in addition to the path through the learned branch.This is the mechanism behind the paragraph's description of easier optimization and improved gradient flow.
A shortcut is not a guarantee that gradients remain well behaved in every network.The transform branch, Other operations and training settings still matter.Residual connections can ease optimization without removing all difficulties.Global average pooling act on a different part of the representation.Take one feature map channel and and average its values across all spatial positions.
Repeat that calculation separately for every channel.The result is one number per channel.For one observation with C channels, the output is a vector of length.C height and width disappear as separate axes.The channel count remains.
For A small arithmetic illustration, 12 x 2 channel contains 0, 2, 4, and 6.Its average is 3.If a second channel contains pen at every position, it its average is pen.The pooled representation for those two channels contains the values 3 and 10.The operation did not average the channels together.
It produced one spatial summary for each channel independently.Global average pooling removes explicit spatial positions.Not all feature distinctions.Different channels remain distinct, but the pulled vector no longer directly states, but where Within a channel, each response occurred.Rearranging the values within a channel leaves its mean unchanged.Therefore, different spatial arrangements can produce the same pooled vector.Whether that loss of location is acceptable depends on the information required by the task.A task head can map the pooled vector to the required class scores or continuous prediction.The averaging itself contains no trainable parameters.The transformations before it and the head after it can still be learned.
If an output requires spatial detail, a pooled vector alone does not preserve the original coordinate map.Such a task needs a representation or additional pathway retaining the relevant positions.Pooling cannot substitute for that requirement.Pooling also allows a fixed length vector when map height or width varies but channel count is unchanged.This is a shape property, not evidence that predictions remain valid or across changes in image scale or acquisition.
Residual addition combines aligned shape compatible branches and can ease optimization.Global average pooling supplies one spatial mean per channel to a task head.The two operations preserve different aspects of the representation.The shifted outlines on the right provide the cue for the question of position.We now examine what a spatial shift should do to a convolutional feature map and why.The answer depends on architectural conditions.Spatial equivariance means that transforming the input produces a corresponding transformation of the output for translation.Shifting the input shifts the response map rather than necessarily leaving that map unchanged.Weight sharing makes this possible.The same kernel weights are applied at different positions.If an identical neighborhood is presented at another compatible position, the local calculation uses the same coefficients.There.Equivariance and invariance are different requirements.An equivariant feature map follows the shift and invariant output stays unchanged.Detecting a pattern at a new position does not mean discarding the information about its position.
Imagine shifting a complete input grid 1 pixel to the right while retaining the relevant data under Stride 1 and compatible boundary.Handling the the corresponding convolution responses also move one position to the right.The response has moved it has not become positionless.Its location is still represented by its position within the feature map.This is why shared filters need not automatically remove spatial information.
Stride one matters because the output grid samples every kernel position along the input grid.A one position input shift then as a matching one position output shift wherever the relevant boundary conditions permit it.With stride two output positions lie on a coarser grid.Shifting the input by one pixel changes which neighborhoods align with those sampled positions.The new map need not be a simple integer shift of the previous map.
A line shift can still correspond under additional conditions.For a stride two layer, a two position input shift can correspond to a one Position output shift.The translation scales are then different rather than identical.Thus, larger stride does not abolish every possible translation relationship.It restricts which shifts align with the sampling grid.
The slide stride one condition concerns matching shifts at the same spatial step.Boundary handling matters independently.A finite image has edges.Moving content towards an edge can change which observed pixels and padded entries for fall inside a kernel neighborhood.Even with stride 1 0, padding on a finite cropped array does not make all boundary responses exactly translation equivariant.
The condition must include compatible treatment of the shifted domain and its boundaries.Are we shifting the fully represented input or moving one structure relative to the surrounding anatomy?Are all required neighborhoods still present or has the shift changed?What falls outside the image?Moving only one structure changes its relationships with the background, that is a different input from translating the whole scene Identical local appearances can therefore coexist with different larger contextual arrangements.
The anatomical interpretation of a finding may depend on those relationships.Applying the same filter at different positions and is not a statement that every anatomical position has the same meaning for the task.The expanding context in the title connects back to the receptive fields just calculated.Later features can combine local responses over a wider region, allowing relative arrangements to enter the computation.However, the whole network inherits a symmetry only when its relevant operations preserve that symmetry.
Spatially varying parameters so boundary choices down sampling or a position sensitive head can change the behavior.Previous slide provides another useful connection.If a shift only rearranges a channel's feature values without adding or removing them, global average pooling leaves that channel summary unchanged.Real finite image shifts may alter boundary values or sampling alignment, so the conditional pooling argument is not a blanket invariance guarantee.We must inspect the complete operation rather than assign a property from its name.
Shed filters can preserve corresponding shifts when feature maps retain position.The exact relationship depends on stride boundaries and later operations.Wider context does not make anatomical location irrelevant.
Describes patch tokenization, positional embeddings, scaled dot-product attention, multi-head blocks, normalization, and mask handling.
Speaker A
We now organize image input for attention rather than only for grid based local operations.The three squares symbolize patches.The expression beneath them combines a patch's content representation with information about its position.A patch token begins with a specified region of pixels flatten its values into a vector using a fixed order.That vector can then be projected into the embedding dimension used by the model.Flattening does not necessarily discard the patch's pixel values.A fixed ordering still identifies where entries belong within that patch and a learned projection may compress them depending on its dimensions and parameters.
For an arithmetic illustration, divide an 8x8 single channel image into non overlapping 4x4 patches There are four patches, each initially containing 16 pixel values.Each patch becomes one input token.The numerical values within those vectors can remain continuous.Here, discretization means dividing the input into discrete processing units.It does not necessarily mean rounding every pixel value into an integer category.
Token content and token position answer different questions.One describes the patch's measurements.The other describes where that patch belongs in the represented image.Neither should be assumed to supply the other automatically.The earlier tensor slide distinguished the sequence axis from embedding coordinates.The present slide is explains how those token vectors are constructed and how spatial meaning can be supplied alongside their content.A stored sequence has a row order, but that order is not necessarily visible to the mathematical operation processing standard.Self attention without positional information or other order queues does not identify location from row numbers alone.Its permutation equivariance means that rearranging input token vectors correspondingly rearranges the output vectors.It does not mean that every output remains unchanged after the rearrangement.
If two patches have identical content vectors but occupy different spatial locations, what supplied information allows the model to distinguish those locations?Is their physical arrangement represented or merely known to the programmer?Position information addresses that gap.A positional vector represents the assigned position of a token.It can be learned or defined by a fixed scheme.
Its role is to supply an order or spatial queue absent from content alone.Read the expression on the right as content embedding.Position embedding.In this additive construction, both vectors have the same number of coordinates, and corresponding coordinates are added element by element.The resulting vector therefore retains the embedding dimension.
Addition does not place a new block of position coordinates after the content coordinates as concatenation would.The position queue is mixed into the representation.Imagine the same patch content assigned to two different positions.The content component can be identical while the position components differ.Their combined vectors can then distinguish the two assignments.
Applying position information enables the model to use location.It does not guarantee that the learned computation uses location correctly, nor does it automatically supply anatomical identity or physical distance.In clinical units.The assignment must remain consistent with the image grid.If token content is rearranged but associated with the wrong positional vectors, the representation describes an incorrect spatial arrangement even though its tensor shape remains valid.
Separate rearranging stored rows from changing spatial assignments.Reordering tokens while keeping the correct position information attached preserves those assignments.Moving content to different position codes describes a different image arrangement.Patch size determines the granularity of this representation.Smaller patches provide more separate spatial units.
Larger patches place more measurements within each unit.The appropriate choice depends on which distinctions must remain available.The phrase spatial context should not imply reconstruction of information already discarded.Positional encoding supplies position cues.It cannot recreate missing pixels or restore distinctions eliminated by an inadequate content representation.
Hash tokenization constructs content vectors.Positional information supplies their arrangement without order queues.Attention is permutation equivariant.The combined embedding must preserve the content and location distinctions needed for the task.We have constructed token embeddings that include positional information.
The formula now explains how one token's representation can incorporate information from other tokens through learned matching and weighted mixing.For self attention, the current token representations supply all three inputs, queries, keys, and values.They come from the same sequence, but different learned projections give them different computational rows.A query describes the matching pattern used by an output position.A key describes what another position offers for that matching calculation.
A value supplies the representation that would actually be combined into the output.The slide's questions about looking for information are metaphors for these roles.Queries do not express conscious intentions, and keys do not certify that their content is clinically relevant.Each projection multiplies token representations by its own learned matrix.Queries and keys need matching widths for their dot products, while value vectors can have a different width.
Distinguish learned projection weights from attention weights Projection parameters are fitted during training.Attention weights are recalculated from the current input, determining how values are mixed.For this particular observation, first compare each query with the keys using dot products.A dot product combines corresponding query and key coordinates into one score.Larger scores indicate stronger matching according to these learned projections, the transpose on the key matrix arranges the calculation so that every query can be compared with every key.
Each row corresponds to one query position and each column to one key position.In self attention with n tokens, these comparisons form an n by n score matrix.A token can also compare with itself when that connection is allowed.Self attention does not mean attending only to one's own token.Next, divide the scores by the square root of the key dimension.
The dimension counts coordinates in each key vector.The divisor is its square root, not the dimension itself.For a numerical illustration, keys with 64 coordinates give a divisor of 8.Every dot product score in that calculation is divided by 8 before the normalization that follows.The scaling helps control score variation as vector dimension changes.
Large differences between scores can make softmax concentrate almost entirely on one position with small derivatives for its probability outputs.Scaling helps mitigate the problem.It does not guarantee that gradients never become small or that attention remains well behaved throughout training.The learned vectors and the rest of the computation still matter.Softmax then normalizes each query scores across the allowed keys.
The resulting mixing coefficients are non negative and sum to one for that query.Different queries generally produce different rows of coefficients.Finally, multiply those coefficients by the value matrix.Each output is a weighted sum of value vectors, not a weighted sum of key vectors.Keys determine matching values, provide the content passed onwards.
Imagine three values receiving weights of 2/10, 3/10, and 5/10 for one coordinate with values 10, 20, and 30.The weighted sum is 2, 6, 15, giving 23.The same coefficients combine every coordinate of those value vectors.Repeating the calculation for every query produces one output vector per query position.The head's output width follows the value vector dimension.
The slide also specifies separate learned projections for each head.These are distinct parameter sets trained jointly through the overall objective.Separate projections do not mean that each head is trained as an independent clinical model.The normalized coefficients describe information routing within the model.They are not patient level probabilities and as noted in Lecture 1, attention alone does not establish causal importance or a faithful explanation.
Follow the formula as a sequence.Project tokens into queries, keys and values.Score query key matches, scale and normalize the scores, then mix the values.This is input dependent representation building, not direct clinical reasoning.The right hand stack places attention inside a larger block.
It lists normalization, multi head attention, another normalization, and a feedforward network with a residual label indicating shortcut connections.Multi head attention runs several matching and mixing calculations using different learned projections.Their outputs can capture different relationships because each head operates in its own projected representation space.The head outputs are concatenated and projected back into the block's representation dimension.This combines their contributions while producing a shape suitable for the subsequent computation and residual addition.
Multiple heads do not guarantee distinct clinical specializations.A head is not automatically an organ detector, a symptom interpreter, or an independently validated source of evidence.Its role follows from learned parameters.The feeforward network then transforms each token's representation separately.A standard form uses two learned affbind transformations with a non linear activation between them.
The same transformation parameters are applied at each token position.Here, local refinement means computation within one token's feature coordinates.It does not mean applying a spatial kernel to neighboring image patches.Information from other tokens reaches this stage through the preceding attention calculation.Attention mixes information between token positions.
The feedforward network transforms information within each token representation.These operations complement one another but are not the same kind of mixing layer.Normalization acts across the feature coordinates of an individual token.It calculates a mean and variance for those coordinates and uses them to normalize the representation before applying learned scale and offset parameters.Those statistics come from the current representation, not from a population of pair.
This differs from the input standardization discussed earlier, where reference statistics were estimated within the training data and then held fixed.A numericode stabilizer prevents division by zero when coordinate variance vanishes.Learned scale and offset then adjust the normalized vector.Its final coordinates need not retain zero mean and unit variance.The stack identifies normalization beside both main transformations.
Its position relative to residual addition varies between transformer designs.This simplified drawing does not specify every connection or implementation detail.The residual connections reuse the addition mechanism already explained.For feature maps, a block adds a transformed representation to a compatible shortcut representation.Matching token and feature dimensions are required together.
Normalization and shortcuts support the optimization of stacked drops.They influence the scales and paths through which representations and gradients move.Rather than defining a new clinical target or replacing the training objective, the stability label should not be interpreted as a guarantee of convergence.Architecture parameter values and optimization settings still interact.These components help organize training.
They do not make arbitrary training choices safe.Now consider the padding masks.Different sequence lengths may require artificial positions to make a rectangular batch.Those positions are storage conveniences, not additional observed patches or measurements that should contribute information to exclude a padded key masked score before softmax normalization.The normalization then allocates weight only among allowed keys, assuming at least one valid key exists.
The padded value does not enter the valid query's weighted sum.Setting a padded value vector to zero without masking its score is not equivalent.It can still receive attention weight and take weight away from genuine values.The mask must act on the matching calculation.Also distinguish artificial token padding from genuine image background.
A dark patch may contain an observed part of the image.Its appearance alone is not a reason to mark it as an absent token.Masking padded keys does not necessarily prevent outputs being computed for padded query positions.Those outputs must also be handled appropriately wherever later pooling, output selection or loss calculation could otherwise induce that.A block combines cross token, attention, token wise nonlinear transformation, normalization and residual paths.
Padding masks distinguish real positions from artificial batch entries.Each component has a specific computational role.
Analyzes the quadratic cost of dense attention, resource trade-offs, and compares convolutional and transformer design for task-specific context.
Speaker A
The quadratic relationship was previewed at the start of part two.We can now connect it to the attention calculation itself and distinguish pairwise work from stored intermediate arrays and the resources used by an entire model.Let N denote the number of tokens in one sequence for one dense attention head.Each of n queries is compared with n keys.The score calculation therefore contains n times n greeny key pairs.For an illustration, 256 tokens produce 65,536 pairwise scores per head.512 tokens produce 262,144.Doubling both dimensions of the score matrix gives 4 times as many entries.
That is the precise meaning of the 2 times to 4 times relationship printed on the right.The quadratic notation describes growth with token count while other dimensions are held fixed.It does not claim that token count is the only influence on computing time or that every part of the block has the same scaling.Each score is also a dot product across vector coordinates.Wider queries and keys make each comparison more expensive.
Mixing value vectors adds work whose cost also depends on their width.The projections and feedforward transformations have their own costs.Consequently, changing embedding width, number of heads, or number of blocks can alter total computation without changing the length of the input sequence.Batch size introduces another multiplier.Processing more sequences means repeating the relevant calculations for more observations.
A token limit for one sequence is therefore not a complete description of a training batch's resource requirements.Stored scores also occupy memory.In an illustrative implementation, storing 4 bytes per score, 1 million scores require 4 megabytes just for that array.This excludes value vectors, parameters and other intermediate representations.That array size is not the model's total memory footprint.
Training may require additional saved intermediates and gradients.While the optimizer has its own state prediction and training can therefore have different memory requirements.The mathematical score matrix does not have to be stored in full at once.Exact attention implementations can process it in blocks, combining results without retaining every pairwise score simultaneously.The fresh attention work by Dao and colleagues demonstrates this distinction between the mathematical operation and its memory execution.
Reducing intermediate storage does not mean that dense attention has stopped.Considering all token pairs do not turn the fourfold score count into an unconditional fourfold prediction about runtime or total memory.Actual behavior depends on the implementation and on which computations and stored arrays dominate.Input resolution interacts with this, accounting for a two dimensional image partitioned into fixed size patches.Doubling both height and width multiplies the number of patches by 4.
Assuming the same channels and complete patch coverage, 4 times as many patch tokens then produces 16 times as many pairwise scores.In dense attention, increasing image dimensions and doubling sequence length are therefore different operations with different consequences.Patch granularity offers another tradeoff.Representing the same image with fewer, larger patches reduces token count, but changes which spatial Distinctions receive separate representations.Computational savings must be considered alongside that change.
Restricting attention to selected token relationships can reduce the set of comparisons but changes the allowed context.By contrast, merely masking scores after calculating a full dense matrix does not automatically remove the original computation.Which quantity is increasing?Sequence length, embedding width, headcount, batch size, or depth?Are we counting pairwise scores, stored bytes, or measured runtime?A resource estimate is meaningful or only when these choices are explicit.Count dense token interactions using the quadratic relation, then account separately for vector dimensions, batches, depth and implementation.Seek a feasible representation with sufficient task relevant context, not simply the largest possible sequence winfar.The two cards now support a conditional architectural choice.We have examined the operators and their costs, so the comparison should concern a specified task rather than a general ranking of network families.
The convolutional card emphasizes built in local connectivity.Its reference to stationary patterns means that a shared local operator can be useful across spatial positions.It does not mean that the patient's physiology is constant over time.Its receptive field is statically defined by architecture.Kernel strides and the connections between layers determine potential input dependence.The strength of particular contributions still depends on the learned computation.High resolution inputs can make this locality attractive because the operator need not compare every position with every other position that is an architectural resource advantage under specified conditions, not a universal clinical performance result.The usefulness of shared filtering still depends on what location and context mean for the task.A pattern repeated across positions may need interpretation relative to anatomy rather than being treated as equivalent everywhere.The transformer card emphasizes input dependent contextual interactions.
The permitted connections determine which tokens can interact, while learned matching determines their relative contributions for a particular observation.The global access is therefore not the absence of architectural constraints.Tokenization, positional information masks and representation width remain design choices.A transformer can also restrict attention rather than use unrestricted dense mixing.The text and tabular example refers to multimodal input.
Here the architectural question is whether contextual interactions represent dependencies between supplied elements.Detailed alignment in fusion or will be developed in Lecture 7.Neither global access nor the presence of several input types establishes useful clinical reasoning.We must ask whether the required information is preserved, legitimately available and actually used to improve the specified task.Imagine a task dominated by small local patterns in a large image.A shared local encoder may represent them efficiently if interpretation also requires information from distant regions and a contextual pathway becomes another consideration.These are task dependent arguments, not claims that one architecture always wins.The same input can contain both local and distant relationships, and the appropriate design may need to preserve both.The hybrid example combines the operators a combinational backbone first extracts feature maps, an attention component then combines the resulting representations across permitted positions, and a task head maps that representation to the required output.
The attention component cannot reconstruct distinctions discarded by the bad bull.
Introduces transfer, retraining history, self-supervised objectives like masked reconstruction and contrastive learning for feature reuse.
Speaker A
Downsampling and feature summarization must therefore be compatible with what the later contextual operation needs to distinguish.Apply the comparison rules from our training audit.Use consistent targets, information data partitions, and stated resource budgets.If these differ, the comparison concerns different development pipelines, not only different architectural operators.Training history also matters.
Comparing a broadly pre trained model with another model initialized without that training cannot isolate architecture alone.The source of the parameters is part of the comparison.Which design preserves the required local detail, provides the necessary context and and remains feasible under the specified budget.What evidence would distinguish a useful architectural advantage from an advantage in training history?Choose conditionally among convolution, attention and hybrids.
Connect their opportunities and costs to the specified task and comparison conditions that leads directly to the origin and reuse of trained representations.We now enter part three.3 Retraining, transfer, and data boundaries the title connects three questions about the reusable model where its parameters came from, what is reused, and which observations influence that learning.Lecture one introduced the broad idea of pre training followed by adaptation.We will now examine the training choices and data histories that make reuse meaningful.
Rather than treating a pre trained checkpoint as a self explanatory starting point, architecture establishes how a model can represent information.Training history determines the parameter state within that architecture.Copies of the same network can begin a new task with different learned representations.A reusable parameter file is not a blank model.It carries the consequences of earlier training even when the current project has applied no new clinical labels and performed no new updates.
Imagine inheriting a biomedical encoder and being asked to adapt it to a new target.The file loads successfully and its input dimensions match those facts.Establish technical compatibility but do not yet establish useful transfer.What input meanings did the earlier training assume?Which observations shaped the parameters?
What objective encouraged the representation to retain certain distinctions and ignore others?Retraining concerns the earlier learning stage.Transfer concerns reusing its outcome for another task or setting data boundaries concern the information permitted to influence each stage in its assessment.Architecture and training objective are separate choices.A convolutional encoder and a transformer encoder can each be trained under different objectives.
Naming the architecture, therefore, does not tell us where the training targets came from.The next step is to examine how targets can be supplied externally or constructed from the input itself.That distinction explains how learning can occur before the labels for a particular downstream task are available.Which parts of the earlier representation should remain useful for the new target?Does a new task require distinctions that the earlier objective had little reason to preserve?
Reuse must also connect to a defined output.A representation learned for one purpose is not automatically the complete predictor for another.The target's meaning and the required output still determine what adaptation must accomplish.We will distinguish what is retained from what is changed during that adaptation.The relevant issue is not simply whether new training occurs, but which parameters receive the new training signal and under what conditions.
The data history does not begin when adaptation begins.Earlier training may already have involved observations related to the current project.A boundary check limited to the latest training stage would miss that influence.Could examples reserved for assessment have contributed to the inherited representation?Can the earlier data history be traced well enough to answer?
Or is that history unknown?Unknown history is a limitation to document, not evidence that overlap is absent.Equally, a large pre training dataset is not evidence that every relevant population or distinction was represented appropriately.The input pipeline belongs to this history.Interpret a checkpoint alongside the encoding assumptions and transformations that produced its training inputs.
Loading weights alone does not establish that the new pipeline preserves those assumptions.Hotly connects training objectives, adaptation choices, and data prominence.We will ask what was learned, what is reusable, and what boundaries support a meaningful comparison.The aim is justified transfer, not reuse merely because a file is available.The training targets can now come from the input itself.
Self supervision does not remove the objective or the training cycle.It changes how the prediction problem and its reference targets are constructed.The bar and fragment schematic on the right illustrates masked reconstruction.Select part of an observed input, hide it from the prediction calculation and ask the model to predict that part using the remaining context.Because the original input is available during training, the hidden values provide reference targets.
A loss compares predictions with those targets and gradients.Adjust the representation and and the components producing the reconstruction.A deliberately hidden observation is different from a genuinely unobserved measurement.We can score a reconstruction because its reference exists.An absent clinical measurement does not become a known target merely because we mark it as missing.
The objective must specify what is hidden and which components contribute to the loss.Predicting selected pixels, patches, or signal segments creates different tasks even when all are described as masked prediction.Successful reconstruction does not establish successful diagnosis.The features useful for predicting missing input values may differ from those needed to distinguish the downstream target.Low reconstruction loss answers the reconstruction question.
The contrastive example constructs a different target relationship.Generate two permitted views of the same source instance and train their representations to match a according to the chosen similarity objective.Here, a positive pair means a pair designated to match during training.It does not mean that both observations are positive clinical cases.The terminology describes the learning construction, not a disease label.
In a common contrastive setup, selected negative pairs are also encouraged to remain distinguishable.Those pairs are defined by the training procedure.Their designation does not prove that they share no biologically relevant characteristics.Imagine two recordings from different individuals who share the same condition.Treating their representations as a negative pair may separate person specific information without making the condition easier to recognize.
The choice of pairs therefore determines which relationships the objective rewards.Aligning everything would allow an uninformative constant representation, so the training design must also prevent that kind of collapse.Augmentations determine which differences matching views are encouraged to ignore.The rotated scan example assumes that the selected rotation preserves the relevant meaning.That assumption must be checked for the intended representation if orientation or location is important, forcing rotated views to match can suppress a useful distinction.
An augmentation suitable for one task is not automatically appropriate for every medical input or downstream objective.Similarly, imagine two crops from one image, only one of which contains a task relevant finding.Treating them as interchangeable views can encourage the representation to ignore information that the later task needs.Masked and contrastive objectives thus provide training signals without requiring a clinician to assign a diagnostic label to every example.However, their targets, masks, pairs, and transformations still embody deliberate design choices.
The reusable component is typically the learned feature extractor, not necessarily the reconstruction or contrastive output component.A downstream task may attach a different head and supply its own targets.And what distinctions does this pre training objective preserve and which does it encourage the model to overlook?Do those choices support the downstream question or only make the pre training task easier?Self supervision constructs targets from observed inputs or relationships between views.It can produce useful representations, but usefulness remains task dependent.Objective design must connect the learned distinctions to to what adaptation will later require.
Details linear probing, partial/full tuning, transfer compatibility checks, and provenance requirements to assess reuse benefits.
Speaker A
The three cards distinguish adaptation by which parameters can change.We will follow the same pre trained encoder through a linear probe, partial tuning, and full tuning while keeping their different questions clear.In a linear probe, the encoder is frozen and a new output layer is trained.
Inputs still pass through the encoder, but the downstream training objective does not update its parameters.The learn head provides an affine mapping from the fixed representation to task outputs.Linear refers to that mapping before any required output activation.It does not mean that the original encoder itself is linear.This tests how accessible the target is through the existing representation.
A successful probe supports its usefulness under that restriction.A weak probe does not prove that the encoder could never become useful after adaptation.The small data and similar domain uses on the card are conditional motivations, not guaranteed selection rules.Few labels make extensive adaptation risky, but freezing irrelevant features does not make them suitable for the new task.Partial tuning Update selected encoder layers together with the task.
The card describes one common arrangement.Hold lower layers fixed while allowing upper layers to adapt to the target.The genetic versus task specific distinction is a working hypothesis, not a universal division between layers.Which layers transfer usefully depends on the source objective, the inputs and the target task.The freezing boundary is therefore part of the experiment.
Record which layers remain fixed, which are updated, and whether the representation used by the head actually changes during training.Partial toning is not simply a smaller learning rate applied everywhere it changes which parameters receive updates.Learning rate separately controls the size of updates for the parameters allowed to change.Full tuning allows every trainable encoder parameter and the task head to adapt.The inherited weights still provide the starting point.Updating them does not turn this strategy into training from scratch.More adaptable parameters provide flexibility, not guaranteed improvement with limited target data.That flexibility can fit training specific details.The same training versus development distinction from our earlier audit still applies.The forgetting risk concerns deterioration of capabilities learned during the source training.
New updates can change representations that supported those capabilities, even while improving performance on the new training objective.Forgetting is a problem when the previous capability is still needed.Changing an unsuitable source feature can instead be part of successful adaptation.Preserving source behavior is NOP automatically the correct target task goal across all three strategies.Label availability, task similarity, learning rate, and regularization interact.
A sensible comparison permits a pro appropriate development choices for each strategy rather than assuming one configuration suits them all.Freezing also needs a consistent computational configuration.Hoarding parameter values fixed is not the same as making every output identical.If training time, randomness or other changing internal state remains active.The banner below the card requires a scratch baseline from scratch means no reuse of previously learned weights, not setting every parameter to zero.
Compare using the same downstream inputs, targets, partitions and stated budgets.Is the existing representation sufficient with a new head?Does part of it need modification?Or is broader adaptation justified which compares and supports that decision rather than merely demonstrating lower training loss?A linear probe changes the head.
Partial tuning changes selected encoder parameters and the head full tuning changes the whole trainable model.Judge the adaptation policy against suitable alternatives, including training without inherited weights.The previous slide asked which parameter should change.This slide asks whether the source learning provides an appropriate starting point at all.Transfer helps only when the resulting learning process supports the target task.
Source features may provide useful distinctions, a favorable starting configuration, or both.These possibilities should not be treated as equivalent evidence that the source model already represents the new clinical target.Check compatibility across the source population, acquisition context, input encoding, and training objective.A familiar data set name or a matching tensor shape does not answer those questions.Domain similarity is not task identity.
Two data sets can contain similar looking inputs, but while requiring different distinctions.Conversely, useful features can sometimes transfer between inputs that look quite different.The example contrasts a possible texture focused natural image representation with an x ray task requiring subtle shape and intensity relationships.Treat this as a compatibility question, not a rule about every model or radiograph.Ayer, Hos and colleagues tested images with conflicting shape and texture cues and observed texture based choices in the ImageNet training models they examine.
Their experiment conserves those models and test conditions, not every possible pre trained encoder.They also changed the training data and obtained a different shape texture preference with the same architecture.This supports separating what an architecture permits from what a particular training history encourages.The medical example does not imply that texture is always irrelevant or that natural image pre training must fail.The target may depend on texture, shape, intensity, or their relationships.
Compatibility must be established for the specified task.A source objective can also encourage invariants to a distinction the target requires.As the preceding self supervision slide showed.Matching augmented views can suppress differences deliberately treated as unimportant during pre training.Imagine source training that encourages similar representations despite large amplitude changes.
If the new target depends on amplitude, the inherited representation may make an important distinction difficult for a fixed head to recover.Negative transfer needs an operational comparison.Using the transfer produces worse target test performance than an appropriate alternative without that transfer under comparable conditions, a suspected feature conflict alone does not demonstrate it.The baseline panel therefore matters compared with a suitably developed model trained from scratch.Rather than an intentionally weak alternative, the downstream target inputs, partitions and budget must support a meaningful comparison.
Raghu and colleagues Transfusion study found limited performance benefits from ImageNet transfer in the medical imaging tasks they examined.That result concerns those comparisons.It is not proof that medical transfer never helps.An observed disadvantage also does not identify its mechanism.Input incompatibility, adaptation settings, or the source representation can all contribute.
Do not attribute a poor result to texture preference without investigating that explanation.Likewise, faster optimization and improved final target performance are different outcomes.Outcomes A useful initial parameter scale may accelerate fitting even when the source's high level features contribute little to the target.What benefit is being claimed?Accessible target features, quicker adaptation or better held out performance.
What source information and comparison are needed to distinguish those possibilities?Assess transfer as a relationship between source learning and a target task.Check compatibility and adaptation against a credible no transfer alternative.Neither a pre trained label nor a plausible feature story establishes benefit.Model weights have a history linking them to source data.
The connected nodes at the bottom emphasize that lineage provenance makes it traceable.Rather than leaving the checkpoint as an unexplained parameter file for a source record document where and under what collection conditions it originated, keep a stable state study identifier linking that record to its derived examples.A file name alone is not a complete description of origin.Next, record the transformations that produce the model input.The source record transformation version, and resulting training example need to remain linked even if the example is renamed or stored in a different format.
Imagine that one original recording produces several extracted segments.Each segment receives a new file name without a link to its parent record.The resulting collection can appear to contain unrelated observations despite its shared origin.Paul Vernands describes relationships between artifacts, not just a list of files.Those relationships allow us to ask whether two examples share a source and which transformations produce their apparent differences.
Labels need their own history.Document the reference used, the target definition, and the label version.The same input can be paired with different targets after a correction or a change in the task definition.Suppose a dataset keeps its familiar name while a revised release correct some labels.Two training runs can then use nominally the same dataset but different training targets.
The release identifier is necessary to interpret their comparison.A label's presence in a file does not establish that it represents the intended clinical target.Its source and definition must be known.Documentation makes those assumptions inspectable rather than proving them correct.Extend the lineage to weight initialization.
Record whether the run began without inherited weights or from a particular checkpoint.Identify that checkpoint's version rather than only naming its architecture or model family.For pre training.Retain a data inventory and the objective configuration.Masking rules, pair construction, and augmentations help describe which distinctions the training process was encouraged to preserve or ignore.
For adaptation.Link the target data version, the chosen updating strategy, and the selected checkpoint to the same training record.This connects the reusable representation with the specific parameter state eventually assessed.Can we trace a reported prediction back to an exact checkpoint, then to its initialization.Training examples, transformations, and target definitions.Which link is documented and which remains unknown?Documentation supports reproducibility.Repeating a run also requires code configuration, random choices, and computational setup.Recording source labels and objectives alone does not guarantee identical weights.The Practical aim is to distinguish what was actually used from what was intended or assumed.
A data set mentioned in a paper and the examples that reached a training run need not be identical.Inventories unknown tree training Provenance limits both compatibility assessment and overlap checks.It is not evidence that the data were independent of a downstream assessment set.
Covers provenance, patient partitioning, metadata leakage, cross-stage audit, test set independence, and pipeline integrity before assessment.
Speaker A
The uncertainty must remain explicit.This history also helps interpret failures.
A discrepancy may arise from a changed target definition, transformation, or initial checkpoint rather than the architectural property being investigated.Traceability separates these possibilities.Keep dataset and weight versions attached to comparisons.A performance claim concerns a particular developed mapping under particular conditions.Not every file that later appears under the same dataset or model name trace records to transformations and target definitions into training examples and model checkpoints include inherited weight history.
Provenance does not prove performance it exposes the assumptions behind reuse and assessment.The patient overlap warning comes from Lecture one and our training audit.We now apply it to related examples.The cards illustrate patient A in training and patient B in testing.For an assessment of new patients, assign the patient level group before distributing records.
Related examinations, recordings, and derived examples from that patient inherit the same partition assignment.Imagine that Patient A contributes one source examination from which several crops are created.Because the parent belongs to training, those crops also belong to training.Renaming each crop does not make it a new independent patient.Patient B's test, examination, and derived examples remain on the test side.
The separation concerns their source relationship, not whether the files have different dimensions, formats, or or names.The safe construction order is to identify source groups, assign their partitions, and then create variants within those assignments.Do not generate a large collection of variants and randomly distribute them as though they had unrelated origins.This rule applies to crops, extracted signal segments, and augmented views.A transformation changes the example's representation.
It does not erase the parent record or patient that supplied its information.A source record and its descendants must stay together.Separating the parent from its derived crop is not a protection against leakage.Placing those related inputs on opposite sides can create the very overlap we want to avoid.The provenance links from the previous slide supply the grouping information.
Keep both the parent record relationship and the patient level relationship so that derivatives cannot lose their assignment during later processing.Different file names do not establish independence.Conversely, apparently different records may contain duplicate or overlapping content.Grouping needs identifiers and source relationships, not only a comparison of names.The identity mechanism described on the left does not require the model to receive a patient's name.
Stable individual patterns can be shared across related records, allowing familiar individuals to contribute to apparent predictive success.Imagine several recordings from one person carrying the same diagnostic label.If some are used for training and others for testing, recognizing that individual's repeated pattern can help predict the label without demonstrating performance on a new person.Chai Bug Neto and colleagues investigated this identity confounding in repeated measure digital health datasets.Their work shows why a record wise split can mix subject recognition with diagnostic prediction.
Inflation is a possible consequence, not an automatic numerical result.For every overlap, its extent depends on the task and data.The design problem is that the evaluation no longer cleanly answers the intended new patient question.The grouping rule also applies to development observations.Although these cards show only training and test, patient related examples must not move between the relevant partitions simply because development and testing have different roles.
Audit the assignments after transformations as well as before them.Confirm that each derivative still points to its source group and that no source group has acquired conflicting partition labels.What does one independent evaluation unit represent here?Can every crop segment or augmented view be traced to that unit, and does its partition agree with all the related source records?Patient separation addresses this specific dependence.
It does not by itself establish representativeness or clinical usefulness.Other generalization questions require an evaluation design matched to their own intended claim.Assign source groups before creating variants and let variants inherit the parent's partition.Preserve patient and record lineage through processing so evaluation concerns new patients rather than familiar records stored under different names.The preceding slide group related examples now audit what enters Learning before clinical labels appear.
Label free Pre training does not automatically yield an independent or shortcut free representation.Metadata can influence learning through two routes named in the paragraph.It can be included in model inputs, or it can be used to construct training targets and relationships between examples.A field name or identifier is that automatically visible by to a neural network.We must trace whether the data pipeline encodes it, embeds it, or otherwise uses it in the training calculation.
Its presence on disk alone is not the mechanism.Imagine a file name containing a code derived from the eventual outcome.If that string is encoded alongside the biomedical input, the model may exploit the code.Removing the separate label column would not remove the same information from the string if that outcome derived information will be unavailable at the prediction time, supplying it violates the availability contract established earlier.Automatically encoding a field does not make it a legitimate predictive input.
Target construction needs the same scrutiny.A procedure may automatically derive targets from categories or identifiers that already encode clinical labels.Automatic construction is not evidence that the objective is genuinely independent of those labels.Likewise, constructing matching pairs from patient identity can encourage a representation of individual specific patterns.Whether that is appropriate depends on the declared pre training goal and the downstream question.
Shortcut learning and leakage are related but not identical.A proxy can be legitimately available yet unsuitable for reliable transfer.Leakage concerns information use or dependence that contradicts the stated prediction or evaluation boundary.This distinction prevents an indiscriminate ban on metadata.Some metadata can be legitimate inputs.
The audit asks what a field represents when it becomes available, how it is used, and whether its role matches the intended task.Now extend the boundary across training stages.Self supervised pre training can adjust parameters using an input without ever receiving its clinical label.The resulting checkpoint can therefore have been influenced by an example reserved for later assessment.Holding out clinical labels is not the same as holding out the observations from every learning stage.
The complete training history determines whether an assessed input is genuinely unseen by the developed pipeline.A protocol may deliberately permit learning from unlabeled assessment inputs.Such access must be declared and the interpretation must match it.It cannot simultaneously establish performance on inputs never used during any learning stage.Use the provenance inventory to compare pre training sources with downstream training development and test groups.
Check patient relationships and duplicate records, not just whether the final adaptation split contains different file names.Derived variables deserve attention as well.A feature assembled from a later result or an outcome related category can carry forbidden information even when its own name looks neutral.Trace its inputs, not only its final label.The training only pre processing Rule from Lecture 1 still applies.
Self supervision does not exempt fitted transformations from the declared data boundary.What information reaches the encoder, what information constructs its targets, and which observations shade the inherited weights?Does each route respect the available information rule and the stated independence claim?The audit banner concerns the whole pathway inputs, target construction, derived features, and cross stage exposure.Trace these routes before interpreting downstream performance as evidence of generalization.
Leakage can occur before clinical labels appear because learning already uses inputs and constructed targets.Audit the entire history and distinguish a permitted learning protocol from hidden target information or an overstated claim of independence.The roles established during our training audit now apply to the complete pipeline.The slide separates parameter learning development choices and final performance assessment while keeping clinical utility as a different evidence question.Training data supplies the fitting signal development observations, guide choices such as architecture adaptation, strategy regularization, and checkpoint selection.
The selected pipeline reflects those choices as well as its learned parameters.An unchanged wait file does not make an evaluation independent if test performance influence the surrounding design.Information can enter through human selection decisions even when no gradient update uses the test labels.Imagine evaluating several candidate pipelines on the supposed final test set and choosing the highest scoring one.Its weights may remain unchanged, but that test set has now helped select the reported model.
The selected score then serves two choosing the model and describing its performance.Those roles are not independent.Calling the same observations a test set does not remove their influence on selection.Cooley and Talbot demonstrated that model selection at cells can overfit an evaluation criterion estimated from finite data.The concern is therefore broader than memorization during parameter training.
Repeatedly revising a model after inspecting its test result turns those observations into part of development.Renaming files or freezing the revised model afterwards does not retrospectively restore the original independence before final assessment.Frizzly developed mapping its input transformations, inherited and adapted weights, task head, and any prediction procedure affecting the output.The evaluated object is this whole mapping, not only the neural architecture.Freezing here means fixing those choices for the assessment.
It does not mean that an individual normalization layer stops calculating input dependent quantities or that attention stops producing input dependent mixing weights.The pipeline must still perform its specified computations on new inputs.What remains fixed is the rule used to produce predictions rather than a constant output or a fixed set of intermediate activation values.An untouched test set then estimates performance under stated conditions.Describe the assessed population, observation unit, target definition and and available input information so that the result has an interpretable scope.
A result under those conditions does not automatically apply to another population acquisition process or information pathway.The limitation is not repaired by architectural sophistication or a large pre training dataset for a transfer comparison.Assess suitably developed alternatives under common downstream conditions.Preserve the data boundaries and report differences in pre training exposure or resources rather than attributing every difference to feature reuse.Which pipeline was fixed before assessment?
Which observations influenced its selection, and what conditions does the reported result describe?Can those answers be traced through the training and provenance records?An unexpectedly poor final result is evidence to investigate the not an instruction to tune repeatedly until the same test set looks favorable.Further changes require an assessment plan that acknowledges the information already used as lecture 1 established model performance and clinical usefulness at different planes.A test result about predictions does not demonstrate that using those predictions improves patient care.
Clinical utility needs additional evidence about the intended decision process, including benefits, errors and workflow consequences compared with an appropriate alternative.We will develop systematic validation methods in Lecture 5.Develop with the permitted training and development information, then assess a fixed pipeline on independent held out observations.Report a result scope without automatically extending it into a claim of clinical benefit.
Summarizes key decisions about inputs, architecture, learning, transfer, and audit foundations required to justify a biomedical model.
Speaker A
Lecture 2 has established the deep learning foundations.
Next we apply them to medical imaging and biomedical signals.Systematic validation follows later.Today's architectural and data boundary Choices provide the groundwork.We can now connect the four statements in this final paragraph into one technical A reusable biomedical model is developed through decisions about preserved information, learned transformations, architectural relationships, and training history.The first decision is what reaches the model.
Tensor axes organize observations, features, positions, and embedding coordinates, but their meanings depend on the input contract.Compatible array dimensions do not establish compatible measurements.Units, category mappings, missingness indicators, and availability rules determine how those entries should be interpreted.Preserving a distinction at this stage makes it available for learning.Discarding it prevents the later network from using it.
A deeper architecture cannot recover a patient specific observation that never reaches its input.Greater numerical capacity is not a substitute for preserving the information required by the task.
Successive learned transformations then build representations.A fine mixing and non linear activation contribute different functions, while a task head assigns the required output meaning.Hidden features need not correspond to named biological variables.The loss specifies the fitting objective and gradients connect that objective to parameter updates.Learning rate, batch composition, and regularization influence the learning process.
Better training fit alone does not establish useful generalization.Convolution provides shared local operations whose context expands through composition.Kernels, strides, padding, and network connections define potential spatial dependencies, while learned parameters determine how that context is used.Residual connections provide additional computational paths while spatial pooling changes what location information remains explicit.These are choices about representation and optimization, not guarantees that relevant anatomy has been understood.
Attention constructs input dependent combinations of tokens.Its usefulness depends on their content.Positional information, permitted interactions, and the surrounding block.Padding masks keep artificial entries distinct from real observations.Dense attention as a quadratic token interaction count why total resources depend on other dimensions and implementation.
Choose feasible context that preserves tasks relevant distinctions.Larger context alone does not establish a better representation.Pre training adds a history to those architectural choices.Masked and contrastive objectives encourage particular distinctions and invariances.Their success on a pre training task does not establish that the resulting features match every biomedical target.
Adaptation determines which parts of that history can change.A fixed encoder with a new head selected, layer tuning and full tuning create different opportunities and risks.Their usefulness must be compared rather than assumed from the strategy's name.Transfer can supply relevant features or a useful starting configuration.A credible scratch baseline separates those benefits from development or resource advantages.
Source target compatibility is a hypothesis to test, not a result.Pipeline integrity connects this reasoning.Source records, transformations, labels, and checkpoints form a traceable history.Related examples retain source group assignments rather than becoming independent through new file names that history extends before downstream labels appear.Inputs, metadata, constructed targets, and earlier pre training can all influence the developed representation.
An independence claim must describe those routes, not only the final adaptation split.Can you defend the information preserved at the input, the objective shaping the parameters, the relationships enabled by the architecture, and the evidence supporting reuse?Which part of that chain remains uncertain?The practical outcome of this lecture is the ability to justify that chain, explain why the representation preserves the needed distinctions, why the learning process matches the target, and why the comparison supports the conclusion being made.The model's name is not its justification.
Defensibility comes from legitimate inputs, appropriate learning, documented history, and evidence within a stated scope.These foundations will guide the biomedical applications ahead.