Multimodal Vision Research Laboratory

MVRL

Research Area: Medical and Biological Imaging

We apply advanced visual representation learning to improve diagnostic tools and automated healthcare systems. Our research includes self-supervised learning for COVID-19 chest X-ray classification using masked autoencoders, efficient training methods for Alzheimer's disease diagnosis with learnable weighted pooling for 3D PET brain image classification, and contrastive cross-modal pre-training strategies for small sample medical imaging. We also work on multi-modal data analysis combining imagery and genetic features for Alzheimer's disease diagnosis, breast cancer localization using weakly-supervised self-training, and automatic hand skeletal shape estimation from radiographs. Our work addresses challenges in medical imaging including domain adaptation, model calibration, and handling limited training data.

View All Publications →

Publications

  1. Xing X, Liang G, Wang C, Jacobs N, Lin A-L. 2023. Self-Supervised Learning Application on COVID-19 Chest X-ray Image Classification Using Masked AutoEncoder. Bioengineering 10. DOI: 10.3390/bioengineering10080901.
    paper doi bibtex Medical
  2. Xing X, Rafique MU, Liang G, Blanton H, Zhang Y, Wang C, Jacobs N, Lin A-L. 2023. Efficient Training on Alzheimer’s Disease Diagnosis with Learnable Weighted Pooling for 3D PET Brain Image Classification. Electronics 12. DOI: 10.3390/electronics12020467.
    paper doi bibtex Medical
  3. Liang G, Ganesh H, Steffe D, Liu L, Jacobs N, Zhang J. 2022. Development of CNN models for the enteral feeding tube positioning assessment on a small scale data set. BMC Medical Imaging 22. DOI: 10.1186/s12880-022-00766-w.
    paper doi bibtex Medical
  4. Xing X, Liang G, Zhang Y, Khanal S, Lin A-L, Jacobs N. 2022. ADViT: VISION TRANSFORMER ON MULTI-MODALITY PET IMAGES FOR ALZHEIMER DISEASE DIAGNOSIS. In: IEEE International Symposium on Biomedical Imaging (ISBI). DOI: 10.1109/ISBI52829.2022.9761584.
    doi bibtex Medical
  5. Khanal S, Chen J, Jacobs N, Lin A-L. 2021. Alzheimer’s Disease Classification Using Genetic Data. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM).
    code paper bibtex Medical
  6. Ying Q, Xing X, Liu L, Lin A-L, Jacobs N, Liang G. 2021. Multi-Modal Data Analysis for Alzheimer’s Disease Diagnosis: An Ensemble Model Using Imagery and Genetic Features. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC46164.2021.9630174.
    doi bibtex Medical
  7. Liang G, Xing X, Liu L, Zhang Y, Ying Q, Lin A-L, Jacobs N. 2021. Alzheimer’s Disease Classification Using 2D Convolutional Neural Networks. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC46164.2021.9629587.
    doi bibtex Medical
  8. Thumbnail for Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging
    Liang G, Greenwell C, Zhang Y, Xing X, Wang X, Kavuluru R, Jacobs N. 2021. Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging. IEEE Journal of Biomedical and Health Informatics 26. DOI: 10.1109/JBHI.2021.3110805.
    paper doi bibtex Medical
  9. Thumbnail for Improved Trainable Calibration Method for Neural Networks
    Liang G, Zhang Y, Wang X, Jacobs N. 2020. Improved Trainable Calibration Method for Neural Networks. In: British Machine Vision Conference (BMVC).
    website paper bibtex Medical Uncertainty
  10. Hammond TC, Xing X, Wang C, Ma D, Nho K, Crane PK, Elahi F, Ziegler DA, Liang G, Cheng Q, Yanckello LM, Jacobs N, Lin A-L. 2020. Beta-amyloid and tau drive early Alzheimer’s disease decline while glucose hypometabolism drives late decline. Communications Biology 3:352. DOI: 10.1038/s42003-020-1079-x.
    paper doi bibtex Medical
  11. Wang X, Liang G, Zhang Y, Blanton H, Bessinger Z, Jacobs N. 2020. Inconsistent Performance of Deep Learning Models on Mammogram Classification. Journal of the American College of Radiology. DOI: 10.1016/j.jacr.2020.01.006.
    paper doi bibtex Medical
  12. Xing X, Liang G, Blanton H, Rafique MU, Wang C, Lin A-L, Jacobs N. 2020. Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification. In: ECCV Workshop on BioImage Computing (BIC).
    code paper bibtex Medical
  13. Liang G, Zhang Y, Jacobs N. 2020. Neural Network Calibration for Medical Imaging Classification Using DCA Regularization. In: ICML 2020 workshop on Uncertainty and Robustness in Deep Learning (UDL).
    paper bibtex Medical Uncertainty
  14. Liang G, Wang X, Zhang Y, Jacobs N. 2020. Weakly-Supervised Self-Training for Breast Cancer Localization. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC44109.2020.9176617.
    paper doi bibtex Medical
  15. Mihail RP, Liang G, Jacobs N. 2019. Automatic Hand Skeletal Shape Estimation from Radiographs. IEEE Transactions on NanoBioscience 18:296–305. DOI: 10.1109/TNB.2019.2911026.
    doi bibtex Medical
  16. Liang G, Wang X, Zhang Y, Xing X, Blanton H, Salem T, Jacobs N. 2019. Joint 2D-3D Breast Cancer Classification. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM47256.2019.8983048.
    paper doi bibtex Medical
  17. Hammond T, Xing X, Jacobs N, Lin A-L. 2019. Phase-dependent importance of amyloid-beta, phosphorylated-tau, and hypometabolism in determining mild cognitive impairment and Alzheimer’s disease: A machine learning study. In: Alzheimer’s Disease Therapeutics: Alternatives to Amyloid.
    bibtex Medical
  18. Zhang Y, Wang X, Blanton H, Liang G, Xing X, Jacobs N. 2019. 2D Convolutional Neural Networks for 3D Digital Breast Tomosynthesis Classification. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM47256.2019.8983097.
    paper doi bibtex Medical
  19. Zhang Y, Liang G, Jacobs N, Wang X. 2019. Unsupervised Domain Adaptation for Mammogram Image Classification: A Promising Tool for Model Generalization. In: Conference on Machine Intelligence in Medical Imaging (CMIMI).
    bibtex Medical
  20. Liang G, Jacobs N, Wang X. 2019. Training Deep Learning Models as Radiologists: Breast Cancer Classification Using Combined whole 2D Mammography and full volume Digital Breast Tomosynthesis. In: Radiological Society of North America (RSNA).
    bibtex Medical
  21. Thumbnail for GANai: Standardizing CT Images using Generative Adversarial Network with Alternative Improvement
    Liang G, Fouladvand S, Zhang J, Brooks MA, Jacobs N, Chen J. 2019. GANai: Standardizing CT Images using Generative Adversarial Network with Alternative Improvement. In: IEEE International Conference on Healthcare Informatics (ICHI). DOI: 10.1109/ICHI.2019.8904763.
    paper doi bibtex Medical
  22. Liang G, Jacobs N, Liu J, Luo K, Owen W, Wang X. 2019. Translational relevance of performance of deep learning models on mammograms. In: SBI/ACR Breast Imaging Symposium.
    bibtex Medical
  23. Zhang X, Zhang Y, Han E, Jacobs N, Han Q, Wang X, Liu J. 2018. Classification of whole mammogram and tomosynthesis images using deep convolutional neural networks. IEEE Transactions on NanoBioscience. DOI: 10.1109/TNB.2018.2845103.
    doi bibtex Medical
  24. Liang G, Wang X, Jacobs N. 2018. Evaluating the Publicly Available Mammography Datasets for Deep Learning Model Training. In: SBI/ACR Breast Imaging Symposium.
    bibtex Medical
  25. Thumbnail for Automatic Hand Skeletal Shape Estimation from Radiographs
    Mihail RP, Jacobs N. 2018. Automatic Hand Skeletal Shape Estimation from Radiographs. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM.2018.8621196.
    doi bibtex Medical
  26. Jones D, Jacobs N, Ellingson S. 2018. Learning Deep Feature Representations for Kinase Polypharmacology. In: ACM Richard Tapia Celebration of Diversity in Computing Conference.
    bibtex Medical
  27. Jones D, Bopaiah J, Alghamedy F, Jacobs N, Weiss H, Jong WAD, Ellingson S. 2018. Polypharmacology Within the Full Kinome: a Machine Learning Approach. In: AMIA Informatics Summit.
    bibtex Medical
  28. Zhang X, Zhang Y, Han E, Jacobs N, Han Q, Wang X, Liu J. 2017. Whole Mammogram Image Classification With Convolutional Neural Networks. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM). DOI: 10.1109/BIBM.2017.8217738.
    doi bibtex Medical
  29. Mihail RP, Jacobs N, Goldsmith J, Lohr K. 2015. Using Visual Analytics to Inform Rheumatoid Arthritis Patient Choices. In: Loh CS, Sheng Y, Ifenthaler D eds. Serious Games Analytics. Advances in Game-Based Learning. Springer International Publishing, 211–231. DOI: 10.1007/978-3-319-05834-4_9.
    doi bibtex Medical
  30. Mihail RP, Blomquist G, Jacobs N. 2014. A CRF Approach to Fitting a Generalized Hand Skeleton Model. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 409–416. DOI: 10.1109/WACV.2014.6836070.
    paper doi bibtex Medical
  31. Mihail RP, Jacobs N, Goldsmith J. 2012. Real Time Gesture Recognition With 2 Kinect Sensors. In: International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). 1–7.
    paper bibtex Medical
  32. Dixon M, Jacobs N, Pless R. 2006. Finding Minimal Parameterizations of Cylindrical Image Manifolds. In: IEEE CVPR Workshop on Perceptual Organization in Computer Vision (POCV). 1–8. DOI: 10.1109/CVPRW.2006.82.
    paper doi bibtex Medical