Multimodal Vision Research Laboratory MVRL

Research Area: Multimodal Representation Learning

How do we learn representations that transfer across sensors, modalities, and domains? We develop self-supervised and multimodal embedding methods that align vision, language, and audio for robust learning at scale. Recent work includes Frobenius norm minimization for self-supervised learning (FroSSL), global and local entailment learning for natural-world imagery (RCME), probabilistic masked multimodal embedding models for ecology (ProM3E), and unified embedding spaces linking ground and satellite views (TaxaBind). These representations underpin geospatial search, biodiversity monitoring, and generative earth-data synthesis across the lab.

View All Publications →

All Publications

  1. Thumbnail for SimLBR: Learning to Detect Fake Images by Learning to Detect Real Images
    Dhakal A, Khanal S, Sastry S, Arndt J, Dias PA, Lunga D, Jacobs N. 2026. SimLBR: Learning to Detect Fake Images by Learning to Detect Real Images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
    code paper bibtex Representation Learning GenAI
  2. Thumbnail for Beyond Flat Labels: Level-Restricted Contrastive Learning for Hierarchical Fine-Grained Vision Classification
    Tao Z, Sastry S, Thompson M, Campolongo E, Zhang N, Zhang Z, Lapp H, Su Y, Berger-Wolf T, Jacobs N, Chao W-L, Gu J. 2026. Beyond Flat Labels: Level-Restricted Contrastive Learning for Hierarchical Fine-Grained Vision Classification. In: Fine-Grained Visual Categorization (FGVC) (IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshop).
    paper bibtex Ecology Representation Learning
  3. Thumbnail for ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology
    Sastry S, Khanal S, Dhakal A, Lin J, Cher D, Jarosz P, Jacobs N. 2026. ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
    code website paper press bibtex Ecology Representation Learning
  4. Thumbnail for VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semantics
    Cher D, Wei B, Sastry S, Jacobs N. 2026. VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semantics. In: IEEE Winter Conference on Applications of Computer Vision (WACV).
    paper bibtex GenAI GeoAI Representation Learning
  5. Thumbnail for MIND the Gap: A Geographic Implicit Neural Representation with Adjustable Spatial Scale
    Corley I, Rao A, Rolf E, Klemmer K, Shelhamer E, Lehmann N, Rußwurm M, Mai G, Jacobs N, Kerner H. 2026. MIND the Gap: A Geographic Implicit Neural Representation with Adjustable Spatial Scale.
    bibtex GeoAI Representation Learning
  6. Thumbnail for RANGE: Retrieval Augmented Neural Fields for Multi-Resolution Geo-Embeddings
    Dhakal A, Sastry S, Khanal S, Ahmad A, Xing E, Jacobs N. 2025. RANGE: Retrieval Augmented Neural Fields for Multi-Resolution Geo-Embeddings. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
    code paper bibtex GeoAI Representation Learning
  7. Thumbnail for TaxaBind: A Unified Embedding Space for Ecological Applications
    Sastry S, Khanal S, Dhakal A, Ahmad A, Jacobs N. 2025. TaxaBind: A Unified Embedding Space for Ecological Applications. In: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).
    code website paper press bibtex Ecology GeoAI Representation Learning
  8. Thumbnail for Semisupervised Learning for Detecting Inverse Compton Emission in Galaxy Clusters
    Lin S-C, Su Y, Gastaldello F, Jacobs N. 2024. Semisupervised Learning for Detecting Inverse Compton Emission in Galaxy Clusters. Astrophysical Journal 977:176. DOI: 10.3847/1538-4357/ad8888.
    paper doi bibtex Representation Learning
  9. Thumbnail for FroSSL: Frobenius Norm Minimization for Self-Supervised Learning
    Skean O, Dhakal A, Jacobs N, Giraldo LGS. 2024. FroSSL: Frobenius Norm Minimization for Self-Supervised Learning. In: European Conference on Computer Vision (ECCV).
    paper bibtex Representation Learning
  10. Thumbnail for PSM: Learning Probabilistic Embeddings for Multi-scale Zero-shot Soundscape Mapping
    Khanal S, Xing E, Sastry S, Dhakal A, Xiong Z, Ahmad A, Jacobs N. 2024. PSM: Learning Probabilistic Embeddings for Multi-scale Zero-shot Soundscape Mapping. In: ACM Multimedia. DOI: 10.1145/3664647.3681620.
    code paper doi bibtex GeoAI Representation Learning Urban
  11. Thumbnail for Prompt-guided and multimodal landscape scenicness assessments with vision-language models
    Levering A, Marcos D, Jacobs N, Tuia D. 2024. Prompt-guided and multimodal landscape scenicness assessments with vision-language models. PLOS ONE.
    bibtex GeoAI Vision-Language Representation Learning
  12. Thumbnail for GeoBind: Binding text, image, and audio through satellite images
    Dhakal A, Khanal S, Sastry S, Ahmad A, Jacobs N. 2024. GeoBind: Binding text, image, and audio through satellite images. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    paper bibtex GeoAI Representation Learning
  13. Jain P, Marcos D, Ienco D, Interdonato R, Dhakal A, Jacobs N, Berchoux T. 2024. Aligning Geo-Tagged CLIP Representations and Satellite Imagery for Few-Shot Land Use Classification. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex GeoAI Representation Learning Urban
  14. Thumbnail for Vision-Language Pseudo-Labels for Single-Positive Multi-Label Learning
    Xing X, Xiong Z, Stylianou A, Sastry S, Gong L, Jacobs N. 2024. Vision-Language Pseudo-Labels for Single-Positive Multi-Label Learning. In: Workshop on Representation Learning with Very Limited Images.
    code paper bibtex Vision-Language Representation Learning
  15. Thumbnail for BirdSat: Cross-View Contrastive Masked Autoencoders for Bird Species Classification and Mapping
    Sastry S, Khanal S, Dhakal A, Huang D, Jacobs N. 2024. BirdSat: Cross-View Contrastive Masked Autoencoders for Bird Species Classification and Mapping. In: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).
    code paper press bibtex Ecology Representation Learning
  16. Thumbnail for Unveiling Roadway Hazards: Enhancing Fatal Crash Risk Estimation through Multi-Scale Satellite Imagery and Self-Supervised Cross-Matching
    Liang G, Zulu J, Xing X, Jacobs N. 2023. Unveiling Roadway Hazards: Enhancing Fatal Crash Risk Estimation through Multi-Scale Satellite Imagery and Self-Supervised Cross-Matching. Journal of Selected Topics in Applied Earth Observations and Remote Sensing (JSTARS). DOI: 10.1109/JSTARS.2023.3331438.
    doi bibtex Transportation GeoAI Representation Learning Urban
  17. Thumbnail for Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping
    Khanal S, Sastry S, Dhakal A, Jacobs N. 2023. Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping. In: British Machine Vision Conference (BMVC).
    code paper press bibtex Representation Learning GeoAI Urban
  18. Thumbnail for Self-Supervised Learning Application on COVID-19 Chest X-ray Image Classification Using Masked AutoEncoder
    Xing X, Liang G, Wang C, Jacobs N, Lin A-L. 2023. Self-Supervised Learning Application on COVID-19 Chest X-ray Image Classification Using Masked AutoEncoder. Bioengineering 10. DOI: 10.3390/bioengineering10080901.
    paper doi bibtex Medical Representation Learning
  19. Sastry S, Dhakal A, Brodie B, Khanal S, Jacobs N. 2023. Explorations in Self-supervised Learning for Change Detection. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    paper bibtex GeoAI Representation Learning
  20. Thumbnail for AssocFormer: Association Transformer for Multi-label Classification
    Xing X, Peng C, Zhang Y, Lin A-L, Jacobs N. 2022. AssocFormer: Association Transformer for Multi-label Classification. In: British Machine Vision Conference (BMVC).
    paper bibtex Medical Representation Learning
  21. Thumbnail for Causality for Inherently Explainable Transformers: CAT-XPLAIN
    Khanal S, Brodie B, Xing X, Lin A-L, Jacobs N. 2022. Causality for Inherently Explainable Transformers: CAT-XPLAIN. In: XAI4CV: Explainable Artificial Intelligence for Computer Vision (IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops).
    code paper bibtex Medical Representation Learning
  22. Thumbnail for Dynamic Feature Alignment for Semi-supervised Domain Adaptation
    Zhang Y, Liang G, Jacobs N. 2021. Dynamic Feature Alignment for Semi-supervised Domain Adaptation. In: British Machine Vision Conference (BMVC).
    paper bibtex Representation Learning
  23. Thumbnail for Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging
    Liang G, Greenwell C, Zhang Y, Xing X, Wang X, Kavuluru R, Jacobs N. 2021. Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging. IEEE Journal of Biomedical and Health Informatics 26. DOI: 10.1109/JBHI.2021.3110805.
    paper doi bibtex Medical Representation Learning
  24. Thumbnail for Hierarchical Probabilistic Embeddings for Multi-View Image Classification
    Brodie B, Khanal S, Rafique MU, Greenwell C, Jacobs N. 2021. Hierarchical Probabilistic Embeddings for Multi-View Image Classification. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS47720.2021.9554405.
    paper doi bibtex GeoAI Representation Learning
  25. Thumbnail for Optical Wavelength Guided Self-Supervised Feature Learning For Galaxy Cluster Richness Estimate
    Liang G, Su Y, Lin S-C, Zhang Y, Zhang Y, Jacobs N. 2020. Optical Wavelength Guided Self-Supervised Feature Learning For Galaxy Cluster Richness Estimate. In: Workshop on Machine Learning and the Physical Sciences at the 34th Conference on Neural Information Processing Systems.
    paper bibtex Representation Learning
  26. Thumbnail for Learning Geo-Temporal Image Features
    Zhai M, Salem T, Greenwell C, Workman S, Pless R, Jacobs N. 2018. Learning Geo-Temporal Image Features. In: British Machine Vision Conference (BMVC).
    paper bibtex Localization Representation Learning
  27. Thumbnail for On the Location Dependence of Convolutional Neural Network Features
    Workman S, Jacobs N. 2015. On the Location Dependence of Convolutional Neural Network Features. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION). 1–9. DOI: 10.1109/CVPRW.2015.7301385.
    paper doi bibtex Localization Calibration GeoAI Representation Learning
  28. Thumbnail for MPCA: EM-Based PCA For Mixed-Size Image Datasets
    Shi F, Zhai M, Duncan D, Jacobs N. 2014. MPCA: EM-Based PCA For Mixed-Size Image Datasets. In: IEEE International Conference on Image Processing (ICIP). 1807–1811. DOI: 10.1109/ICIP.2014.7025362.
    doi bibtex Representation Learning
  29. Thumbnail for Covariance-Based PCA for Multi-Size Data
    Zhai M, Shi F, Duncan D, Jacobs N. 2014. Covariance-Based PCA for Multi-Size Data. In: International Conference on Pattern Recognition (ICPR). 1603–1608. DOI: 10.1109/ICPR.2014.284.
    paper doi bibtex Representation Learning
  30. Thumbnail for Consistent Temporal Variations in Many Outdoor Scenes
    Jacobs N, Roman N, Pless R. 2007. Consistent Temporal Variations in Many Outdoor Scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1–6. DOI: 10.1109/CVPR.2007.383258.
    website paper doi bibtex Representation Learning