Keywords: MultiModal Learning, Representations, MultiModal
Need: The NHS vision of a Single Patient Record brings together the full breadth of data relating to a patient: structured EHR tables, clinical notes, lab results, medications, imaging, and more, into a unified, accessible form. For AI to work with this data holistically, it needs a suitable representation: a structure that captures the richness of patient information while enabling fast, reusable, and interpretable analysis.
Two main paradigms exist for building such representations. Graph-based approaches encode patients, diagnoses, medications, and clinical ontologies (e.g. SNOMED CT, LOINC) as nodes and edges, making relationships explicit and semantically interpretable. Embedding-based approaches use deep learning to fuse heterogeneous modalities (structured data, free text, imaging) into dense patient-level vectors, enabling downstream tasks through learned similarity. Each paradigm has distinct strengths in terms of interpretability, scalability, and task suitability.
Previous NHS projects, such as mm-healthfair, demonstrated that combining modalities can both improve performance and introduce new fairness risks with outcomes depending heavily on how data is fused and represented.
This project will implement and evaluate at least one approach from each paradigm on shared downstream tasks, producing a comparative analysis that informs NHS data architecture and AI strategy.
Current Knowledge/Examples & Possible Techniques/Approaches:
Graph-based approaches:
Embedding-based approaches:
Temporal Sequencers:
The MIMIC-iV predictive tasks could be considered as standard benchmarks for these approaches.
Related Previous Internship Projects: Understanding Bias in Multi-modal models within Healthcare; Txt-Ray Align; NHSSynth — synthetic multimodal patient generation
Enables Future Work: Modular patient representations reusable across prediction, retrieval, and fairness auditing projects; Foundation for cross-modal fairness evaluation and bias mitigation; Input for patient-similarity systems and retrieval-augmented generation with clinical KGs; Architecture guidance for NHS multimodal and linked-record AI
Outcome/Learning Objectives:
Datasets: MIMIC-IV; SNOMED CT and UMLS as ontological resources for graph approaches
Desired skill set: When applying please highlight any experience around multimodal machine learning or graph ML methods, NLP, clinical terminologies, python coding experience (including any coding in the open), medical imaging experience is a bonus, and any other data science experience you feel relevant.
Return to list of all available projects.