Foundation models for predicting the course of cancer

Every patient with cancer asks the same question in some form: what happens now? Will the tumour come back? Will it spread? Doctors answer with stage, grade, a handful of biomarkers and experience. These are good tools, but they compress a long and detailed history into a few numbers, and they are built one disease and one question at a time.

Over the past two years a different approach has started to work. Instead of building a separate statistical model for every outcome, researchers train one large model on the complete recorded histories of hundreds of thousands to millions of people. The model learns the regularities in how health unfolds over time. Afterwards it can be asked about almost any outcome, at almost any time horizon, for a person it has never seen. Delphi-2M (DKFZ and EMBL), RisQ (TUM and Helmholtz Munich) and Apollo (Harvard) are recent examples; we refer to them below where they illustrate a point. The principles are the same across all of them.

A health record is a sequence of events

Start with what a health record actually contains. It is not a snapshot. It is a long list of dated events: a diagnosis of high blood pressure at 48, a colonoscopy at 55, a cancer diagnosis at 61, surgery two weeks later, chemotherapy, follow-up scans. Each event has a type and a time, and the order matters.

Classical risk scores use only a small part of this. They take five or ten variables measured at one moment and compute one number for one outcome. The sequence itself, and the order of events in it, is mostly discarded.

Language models face the same structure. A sentence is a sequence of tokens whose order matters, and the task of predicting the next token teaches a model a great deal about language. The models on this page apply the same idea to health. The rest of this page walks through the steps: turning a patient into tokens, training the model, producing predictions, adding images and text, and testing whether any of it holds up.

Step 1: Turn the patient into tokens

A language model does not read letters; it reads tokens, small units drawn from a fixed vocabulary. A patient model does the same. Every element of the record is mapped to a token: an encounter starts, a complaint is noted, a diagnosis is coded, a lab is ordered and then reported, a drug is prescribed, a procedure is performed. Time is a token too. A gap of two months between visits becomes its own entry in the sequence, so that the model knows not only what happened but how fast.

The result is one long sequence per patient, in chronological order. Nothing is averaged or summarised; the sequence is the record.

Different models draw the vocabulary differently. Delphi-2M keeps it small: roughly 1,300 diagnosis codes, plus age, sex and a few lifestyle variables. RisQ adds about 3,600 structured measurements, from blood counts to air pollution at the home address, each with its own encoder. Apollo tokenises the full hospital record across 28 kinds of data. The trade-off is simple: a richer vocabulary carries more signal and needs more data to learn from.

Step 2: Train the model to predict what comes next

Training needs no labels. The model is shown the first part of a patient’s sequence and asked to predict the next token. Its guess is compared with what actually happened, and the model is adjusted to make the right answer more likely. Then the window slides one token forward and the process repeats. Across hundreds of thousands of patients and hundreds of millions of tokens, the model is forced to learn which events tend to follow which, how soon, and how this depends on everything that came before.

Two details matter for health data that do not arise in language. First, time is continuous, so the model is usually asked to predict not just the next event but when it occurs; Delphi-2M does this by predicting a rate for each possible event, which also handles the fact that diseases compete with one another and that death ends the sequence. Second, outcomes are rare, and a model that only ever predicts the next token learns rare outcomes slowly. RisQ addresses this with a different objective: rather than the next token, it is trained to answer “will disease X occur within Y years?” for thousands of diseases and horizons at once, which lets rare diseases borrow signal from common ones.

Either way, the output of training is a model that holds a compressed representation of the patient at every point in the sequence. That representation is what everything downstream is built on.

Step 3: Predict the future, by simulating it

A model that predicts the next event can be run forward. Give it a patient’s history up to today, draw the next event from its prediction, append it, and draw again. One run produces one plausible future. Because the draws are random, running it again produces a different one. Run it a few hundred times and you have a distribution of futures for that patient, from which any outcome can be read off: the fraction of runs in which a metastasis appears within three years is an estimate of that risk.

Simulation has two useful side effects. The futures can be inspected, event by event, so a clinician can see which paths lead to a bad outcome rather than only a number. And the generated sequences are synthetic data: Delphi-2M’s authors show that a second model trained only on simulated patients, never on real ones, still learns the structure of disease.

Another way to ask: query the representation directly

Simulation is one way to get a risk out of the model. The other is to ask directly. The patient’s representation is fixed; a query names a disease and a horizon; a small decoder reads the answer off the representation. RisQ builds the query from a language-model embedding of the disease name plus a token for the horizon, which means the question can be composed freely, including for diseases the model was never explicitly trained on.

In a tumour board this is the natural interface. The clinician asks: for this patient, what is the probability of distant metastasis within two years? Within five? The same representation, unchanged, answers both.

Step 4: Add images, text and measurements

Diagnosis codes are a thin description of a cancer. The pathology slide, the radiology series, the molecular profile and the free-text reports carry most of what an oncologist actually uses. These cannot be tokens in the same sense as a diagnosis code, but they can be turned into the same kind of object: each modality gets its own encoder, pretrained on that kind of data, which maps a slide or a scan or a paragraph to a vector. That vector is inserted into the sequence at the time the data were acquired, alongside the ordinary tokens. The central model then attends over everything.

Whether images and text add signal on top of structured data for a given outcome is an empirical question, and for most cancer outcomes it has not yet been answered carefully. For metastasis specifically there is good reason to expect that they do: the morphology of the primary tumour and the fine detail of imaging follow-up are exactly where the early signs live.

From population health to cancer

The models described so far were built mainly to predict the onset of disease in broadly healthy populations. Cancer outcome prediction moves the clock. The sequence of interest starts at diagnosis, is densely packed with treatments and scans, and the outcomes are recurrence, metastasis, response and survival over months to a few years.

This is the specific thing DECIPHER-M needs: a model that treats the oncological course as a sequence, reads images and text natively, and can be asked about metastasis at any point in that sequence. Apollo shows that this is feasible inside one hospital system. OncoTRACE, our group’s temporal model for longitudinal cancer trajectories, is a step in the same direction.

Step 5: Test whether it holds up

A trained model is a hypothesis. Three kinds of test decide whether it is useful, and the published results give a sense of where things stand.

Accuracy falls with distance. Predictions one year ahead are better than predictions ten years ahead, for every model. Delphi-2M’s discrimination drops from about 0.76 in the near term to about 0.70 beyond ten years. That is still useful for conditions with a predictable course and much less so for anything shaped by chance.

Transfer is the hard part. A model trained in one country and applied in another, without retraining, loses performance. When RisQ, Delphi-2M and a tuned gradient-boosted model were all moved from the UK Biobank to the US All of Us cohort, RisQ lost 0.055 points of concordance while the other two lost about 0.155. Multimodal input and training across many diseases appear to buy robustness, but every model lost something.

The model learns the system, not only the biology. A model trained on records learns who gets imaged, how often, and what gets written down. It predicts recorded disease, shaped by access and coding practice, not biological onset. Both the Delphi-2M and the RisQ authors say this explicitly. For cancer it is acute, because these habits differ between hospitals.

Two further gaps are not about accuracy at all. These models estimate what will happen under the care a person would typically receive; they do not, by themselves, say what would happen under a different treatment. And a well-discriminating model is not yet a clinically useful one: the step from AUC to a decision that changes an outcome requires calibration, decision-curve analysis and prospective testing, none of which has been done at scale.

What we take from this for DECIPHER-M

DECIPHER-M is a multi-centre German consortium working on AI to understand and predict metastasis. The principles above set the template for how we approach outcome prediction, and the gaps set the agenda.

• Treat the course as a sequence. Baseline prediction at diagnosis is the weakest version of the problem. The record after first-line treatment is far more informative, and the model should be asked again every time the record grows.
• Read the modalities that carry the signal. For metastasis, pathology, radiology and molecular data are not optional inputs. The open question is how much they add over structured data for this specific outcome, and we intend to measure it.
• Make the question composable. The outcomes of interest, local recurrence, distant metastasis, site-specific spread, survival, should be queries against one representation, not separate models.
• Prove transfer before use. Evaluation across institutions and health systems, with calibration and net benefit, is the bar. A single-centre AUC, however high, is a starting point.
• Keep patients in the loop. Predictions about the future course of a cancer are consequential for the person they describe. Which predictions should be made, how they are shown, and who sees them are questions we answer with the patient organisations in the consortium, not after the fact.

Until now, prognosis in oncology has mostly been a label assigned once, at diagnosis. The models described here turn it into a question that can be asked repeatedly, of the whole record, for any outcome. Making that reliable for metastasis is what DECIPHER-M is working on.

References