Multimodal Vision Research Laboratory

MVRL

Research Area: Medical and Biological Imaging

How can multimodal representation learning improve diagnostic tools when data are scarce or heterogeneous? We apply visual and multimodal learning methods developed in the lab to medical and biological imaging, with emphasis on robust learning under limited labels. Recent work includes self-supervised learning for chest X-ray classification, learnable weighted pooling for 3D PET brain image classification in Alzheimer's diagnosis, contrastive cross-modal pre-training for small-sample medical imaging, and weakly-supervised approaches for breast cancer localization. We address domain shift, calibration, and integration of imaging with complementary clinical signals—extending core MVRL methodology to healthcare applications.

View All Publications →

All Publications

  1. Xing X, Liang G, Wang C, Jacobs N, Lin A-L. 2023. Self-Supervised Learning Application on COVID-19 Chest X-ray Image Classification Using Masked AutoEncoder. Bioengineering 10. DOI: 10.3390/bioengineering10080901.
    paper doi bibtex Medical Representation Learning
  2. Xing X, Rafique MU, Liang G, Blanton H, Zhang Y, Wang C, Jacobs N, Lin A-L. 2023. Efficient Training on Alzheimer’s Disease Diagnosis with Learnable Weighted Pooling for 3D PET Brain Image Classification. Electronics 12. DOI: 10.3390/electronics12020467.
    doi bibtex Medical
  3. Xing X, Peng C, Zhang Y, Lin A-L, Jacobs N. 2022. AssocFormer: Association Transformer for Multi-label Classification. In: British Machine Vision Conference (BMVC).
    paper bibtex Medical Representation Learning
  4. Khanal S, Brodie B, Xing X, Lin A-L, Jacobs N. 2022. Causality for Inherently Explainable Transformers: CAT-XPLAIN. In: XAI4CV: Explainable Artificial Intelligence for Computer Vision (IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops).
    code paper bibtex Medical Representation Learning
  5. Liang G, Ganesh H, Steffe D, Liu L, Jacobs N, Zhang J. 2022. Development of CNN models for the enteral feeding tube positioning assessment on a small scale data set. BMC Medical Imaging 22. DOI: 10.1186/s12880-022-00766-w.
    paper doi bibtex Medical
  6. Xing X, Liang G, Zhang Y, Khanal S, Lin A-L, Jacobs N. 2022. ADViT: VISION TRANSFORMER ON MULTI-MODALITY PET IMAGES FOR ALZHEIMER DISEASE DIAGNOSIS. In: IEEE International Symposium on Biomedical Imaging (ISBI). DOI: 10.1109/ISBI52829.2022.9761584.
    doi bibtex Medical
  7. Khanal S, Chen J, Jacobs N, Lin A-L. 2021. Alzheimer’s Disease Classification Using Genetic Data. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM).
    code paper bibtex Medical
  8. Thumbnail for Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging
    Liang G, Greenwell C, Zhang Y, Xing X, Wang X, Kavuluru R, Jacobs N. 2021. Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging. IEEE Journal of Biomedical and Health Informatics 26. DOI: 10.1109/JBHI.2021.3110805.
    paper doi bibtex Medical Representation Learning
  9. Liang G, Xing X, Liu L, Zhang Y, Ying Q, Lin A-L, Jacobs N. 2021. Alzheimer’s Disease Classification Using 2D Convolutional Neural Networks. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC46164.2021.9629587.
    doi bibtex Medical
  10. Ying Q, Xing X, Liu L, Lin A-L, Jacobs N, Liang G. 2021. Multi-Modal Data Analysis for Alzheimer’s Disease Diagnosis: An Ensemble Model Using Imagery and Genetic Features. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC46164.2021.9630174.
    doi bibtex Medical
  11. Thumbnail for Improved Trainable Calibration Method for Neural Networks
    Liang G, Zhang Y, Wang X, Jacobs N. 2020. Improved Trainable Calibration Method for Neural Networks. In: British Machine Vision Conference (BMVC).
    website paper bibtex Medical Uncertainty
  12. Xing X, Liang G, Blanton H, Rafique MU, Wang C, Lin A-L, Jacobs N. 2020. Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification. In: ECCV Workshop on BioImage Computing (BIC).
    code paper bibtex Medical
  13. Liang G, Zhang Y, Jacobs N. 2020. Neural Network Calibration for Medical Imaging Classification Using DCA Regularization. In: ICML 2020 workshop on Uncertainty and Robustness in Deep Learning (UDL).
    paper bibtex Medical Uncertainty
  14. Hammond TC, Xing X, Wang C, Ma D, Nho K, Crane PK, Elahi F, Ziegler DA, Liang G, Cheng Q, Yanckello LM, Jacobs N, Lin A-L. 2020. Beta-amyloid and tau drive early Alzheimer’s disease decline while glucose hypometabolism drives late decline. Communications Biology 3:352. DOI: 10.1038/s42003-020-1079-x.
    paper doi bibtex Medical
  15. Liang G, Wang X, Zhang Y, Jacobs N. 2020. Weakly-Supervised Self-Training for Breast Cancer Localization. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC44109.2020.9176617.
    paper doi bibtex Medical
  16. Wang X, Liang G, Zhang Y, Blanton H, Bessinger Z, Jacobs N. 2020. Inconsistent Performance of Deep Learning Models on Mammogram Classification. Journal of the American College of Radiology. DOI: 10.1016/j.jacr.2020.01.006.
    doi bibtex Medical
  17. Zhang Y, Wang X, Blanton H, Liang G, Xing X, Jacobs N. 2019. 2D Convolutional Neural Networks for 3D Digital Breast Tomosynthesis Classification. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM47256.2019.8983097.
    paper doi bibtex Medical
  18. Liang G, Wang X, Zhang Y, Xing X, Blanton H, Salem T, Jacobs N. 2019. Joint 2D-3D Breast Cancer Classification. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM47256.2019.8983048.
    paper doi bibtex Medical
  19. Zhang Y, Liang G, Jacobs N, Wang X. 2019. Unsupervised Domain Adaptation for Mammogram Image Classification: A Promising Tool for Model Generalization. In: Conference on Machine Intelligence in Medical Imaging (CMIMI).
    bibtex Medical
  20. Liang G, Jacobs N, Wang X. 2019. Training Deep Learning Models as Radiologists: Breast Cancer Classification Using Combined whole 2D Mammography and full volume Digital Breast Tomosynthesis. In: Radiological Society of North America (RSNA).
    bibtex Medical
  21. Hammond T, Xing X, Jacobs N, Lin A-L. 2019. Phase-dependent importance of amyloid-beta, phosphorylated-tau, and hypometabolism in determining mild cognitive impairment and Alzheimer’s disease: A machine learning study. In: Alzheimer’s Disease Therapeutics: Alternatives to Amyloid.
    bibtex Medical
  22. Thumbnail for GANai: Standardizing CT Images using Generative Adversarial Network with Alternative Improvement
    Liang G, Fouladvand S, Zhang J, Brooks MA, Jacobs N, Chen J. 2019. GANai: Standardizing CT Images using Generative Adversarial Network with Alternative Improvement. In: IEEE International Conference on Healthcare Informatics (ICHI). DOI: 10.1109/ICHI.2019.8904763.
    doi bibtex Medical
  23. Liang G, Jacobs N, Liu J, Luo K, Owen W, Wang X. 2019. Translational relevance of performance of deep learning models on mammograms. In: SBI/ACR Breast Imaging Symposium.
    bibtex Medical
  24. Mihail RP, Liang G, Jacobs N. 2019. Automatic Hand Skeletal Shape Estimation from Radiographs. IEEE Transactions on NanoBioscience 18:296–305. DOI: 10.1109/TNB.2019.2911026.
    doi bibtex Medical
  25. Thumbnail for Automatic Hand Skeletal Shape Estimation from Radiographs
    Mihail RP, Jacobs N. 2018. Automatic Hand Skeletal Shape Estimation from Radiographs. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM.2018.8621196.
    doi bibtex Medical
  26. Jones D, Jacobs N, Ellingson S. 2018. Learning Deep Feature Representations for Kinase Polypharmacology. In: ACM Richard Tapia Celebration of Diversity in Computing Conference.
    bibtex Medical
  27. Zhang X, Zhang Y, Han E, Jacobs N, Han Q, Wang X, Liu J. 2018. Classification of whole mammogram and tomosynthesis images using deep convolutional neural networks. IEEE Transactions on NanoBioscience. DOI: 10.1109/TNB.2018.2845103.
    doi bibtex Medical
  28. Liang G, Wang X, Jacobs N. 2018. Evaluating the Publicly Available Mammography Datasets for Deep Learning Model Training. In: SBI/ACR Breast Imaging Symposium.
    paper bibtex Medical
  29. Jones D, Bopaiah J, Alghamedy F, Jacobs N, Weiss H, Jong WAD, Ellingson S. 2018. Polypharmacology Within the Full Kinome: a Machine Learning Approach. In: AMIA Informatics Summit.
    bibtex Medical
  30. Zhang X, Zhang Y, Han E, Jacobs N, Han Q, Wang X, Liu J. 2017. Whole Mammogram Image Classification With Convolutional Neural Networks. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM.2017.8217738.
    doi bibtex Medical
  31. Mihail RP, Jacobs N, Goldsmith J, Lohr K. 2015. Using Visual Analytics to Inform Rheumatoid Arthritis Patient Choices. In: Loh CS, Sheng Y, Ifenthaler D eds. Serious Games Analytics. Advances in Game-Based Learning. Springer International Publishing, 211–231. DOI: 10.1007/978-3-319-05834-4_9.
    doi bibtex Medical
  32. Mihail RP, Blomquist G, Jacobs N. 2014. A CRF Approach to Fitting a Generalized Hand Skeleton Model. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 409–416. DOI: 10.1109/WACV.2014.6836070.
    paper doi bibtex Medical
  33. Mihail RP, Jacobs N, Goldsmith J. 2012. Real Time Gesture Recognition With 2 Kinect Sensors. In: International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). 1–7.
    paper bibtex Medical
  34. Dixon M, Jacobs N, Pless R. 2006. Finding Minimal Parameterizations of Cylindrical Image Manifolds. In: IEEE CVPR Workshop on Perceptual Organization in Computer Vision (POCV). 1–8. DOI: 10.1109/CVPRW.2006.82.
    paper doi bibtex Medical