Keywords: NLP, Embeddings, MultiModalData
Need: Uses of medical imaging is still only reaching a low amount of its potential opportunity. Furthermore image collections have large amounts of variation in both the images themselves (e.g. different machines having different angles and contrasts), the associated reports (for the most part captured in free text), and the indepth structured information often captured alongside.
This project would aim to explore advances in machine learning and explainability to take advantage of the relationship between these different data modalities - utilising techniques to look at automating text descriptions from images.
Building on the work undertaken as part of the on-going TxtRayAlign project, which looks to use contrastive training techniques to embed images and text data in a shared space, this project could focus on a number of areas including: seeking to investigate other modalities of imaging (such as CT or MRI), integrate further sources of information like demographic or test results to enhance alignment, or the addition of segmentation approaches to further target the association with the text reports.
Current Knowledge/Examples & Possible Techniques/Approaches:
Related Previous Internship Projects: Repository for previous internships on this project can be found here (including the associated reports).
Enables Future Work: Demonstration and deeper understanding of explainability approaches in medical imaging and working with multi-modal datasets.
Outcome/Learning Objectives: Open worked examples and explanation of current state-of-the-art approaches to be built on or used by others, highlighting challenges acutely felt in the medical domain.
Datasets: Suitable multi-modal healthcare datasets such as MIMIC-CXR (with further linkage to MIMIC IV).
Desired skill set: When applying please highlight any experience around work with imaging and/or text data and specifically medical imaging data, tagging, explainability in machine learning, coding experience (including any coding in the open), and any other data science experience you feel relevant.
Return to list of all available projects.