Multimodal Vision Research Laboratory MVRL

Research Area: Social Media

What can large collections of user-uploaded photos reveal about places, events, and environmental perception? We analyze social media imagery to connect visual content with geography, land use, and human experience of landscapes. Recent work includes probabilistic embeddings for multi-scale zero-shot soundscape mapping from social and satellite data (PSM), mapping fine-grained textual descriptions from satellite images (Sat2Cap), and revisiting image geolocalization in the deep learning era. Social media provides ground-level diversity that complements structured earth observation in our geospatial AI and ecology programs.

View All Publications →

All Publications

  1. Thumbnail for Interactive Event Sifting using Bayesian Graph Neural Networks
    Nascimento J, Jacobs N, Rocha A. 2024. Interactive Event Sifting using Bayesian Graph Neural Networks. In: IEEE International Workshop on Information Forensics and Security (WIFS).
    paper bibtex
  2. Thumbnail for PSM: Learning Probabilistic Embeddings for Multi-scale Zero-shot Soundscape Mapping
    Khanal S, Xing E, Sastry S, Dhakal A, Xiong Z, Ahmad A, Jacobs N. 2024. PSM: Learning Probabilistic Embeddings for Multi-scale Zero-shot Soundscape Mapping. In: ACM Multimedia. DOI: 10.1145/3664647.3681620.
    code paper doi bibtex GeoAI Representation Learning Urban
  3. Thumbnail for Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images
    Dhakal A, Ahmad A, Khanal S, Sastry S, Kerner HR, Jacobs N. 2024. Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION).
    code paper bibtex GeoAI Vision-Language
  4. Thumbnail for Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping
    Khanal S, Sastry S, Dhakal A, Jacobs N. 2023. Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping. In: British Machine Vision Conference (BMVC).
    code paper press bibtex Representation Learning GeoAI Urban
  5. Thumbnail for Geoinformation Harvesting From Social Media Data: A community remote sensing approach
    Zhu XX, Wang Y, Kochupillai M, Werner M, Haberle M, Hoffmann EJ, Taubenbock H, Tuia D, Levering A, Jacobs N, Kruspe A, Abdulahhad K. 2022. Geoinformation Harvesting From Social Media Data: A community remote sensing approach. IEEE Geoscience and Remote Sensing Magazine 10:150–180. DOI: 10.1109/MGRS.2022.3219584.
    paper doi bibtex Calibration GeoAI
  6. Thumbnail for Revisiting Near/Remote Sensing with Geospatial Attention
    Workman S, Rafique MU, Blanton H, Jacobs N. 2022. Revisiting Near/Remote Sensing with Geospatial Attention. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR52688.2022.00182.
    paper doi bibtex GeoAI
  7. Thumbnail for Content-Based Detection of Temporal Metadata Manipulation
    Padilha R, Salem T, Workman S, Andaló FA, Rocha A, Jacobs N. 2022. Content-Based Detection of Temporal Metadata Manipulation. IEEE Transactions on Information Forensics and Security:1316–1327. DOI: 10.1109/TIFS.2022.3159154.
    code website paper doi bibtex Calibration Localization GeoAI
  8. Thumbnail for Towards a Collective Agenda on AI for Earth Science Data Analysis
    Tuia D, Roscher R, Wegner JD, Jacobs N, Zhu XX, Camps-Valls G. 2021. Towards a Collective Agenda on AI for Earth Science Data Analysis. IEEE Geoscience and Remote Sensing Magazine 9:88–104. DOI: 10.1109/MGRS.2020.3043504.
    paper doi bibtex Calibration GeoAI
  9. Thumbnail for Learning a Dynamic Map of Visual Appearance
    Salem T, Workman S, Jacobs N. 2020. Learning a Dynamic Map of Visual Appearance. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01245.
    website paper doi bibtex Localization GeoAI
  10. Thumbnail for Implicit Land Use Mapping Using Social Media Imagery
    Greenwell C, Workman S, Jacobs N. 2019. Implicit Land Use Mapping Using Social Media Imagery. In: IEEE Applied Imagery and Pattern Recognition (AIPR). DOI: 10.1109/AIPR47015.2019.9174570.
    paper doi bibtex GeoAI Urban
  11. Thumbnail for Learning to Map Nearly Anything
    Salem T, Greenwell C, Blanton H, Jacobs N. 2019. Learning to Map Nearly Anything. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900646.
    paper doi bibtex GeoAI Urban
  12. Thumbnail for A Generative Model of Worldwide Facial Appearance
    Bessinger Z, Jacobs N. 2019. A Generative Model of Worldwide Facial Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV.2019.00172.
    website paper doi bibtex
  13. Thumbnail for Learning Geo-Temporal Image Features
    Zhai M, Salem T, Greenwell C, Workman S, Pless R, Jacobs N. 2018. Learning Geo-Temporal Image Features. In: British Machine Vision Conference (BMVC).
    paper bibtex Localization Representation Learning
  14. Thumbnail for What Goes Where: Predicting Object Distributions from Above
    Greenwell C, Workman S, Jacobs N. 2018. What Goes Where: Predicting Object Distributions from Above. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8519251.
    website paper doi bibtex GeoAI Urban
  15. Thumbnail for A Multimodal Approach to Mapping Soundscapes
    Salem T, Zhai M, Workman S, Jacobs N. 2018. A Multimodal Approach to Mapping Soundscapes. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8517977.
    website paper doi bibtex GeoAI Urban
  16. Thumbnail for A Unified Model for Near and Remote Sensing
    Workman S, Zhai M, Crandall D, Jacobs N. 2017. A Unified Model for Near and Remote Sensing. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.293.
    website paper doi bibtex GeoAI Urban
  17. Thumbnail for Revisiting IM2GPS in the Deep Learning Era
    Vo N, Jacobs N, Hays J. 2017. Revisiting IM2GPS in the Deep Learning Era. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.286.
    website paper doi bibtex Localization GeoAI Urban
  18. Thumbnail for Understanding and Mapping Natural Beauty
    Workman S, Souvenir R, Jacobs N. 2017. Understanding and Mapping Natural Beauty. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.596.
    website paper doi bibtex GeoAI Urban
  19. Thumbnail for Who Goes There? Approaches to Mapping Facial Appearance Diversity
    Bessinger Z, Stauffer C, Jacobs N. 2016. Who Goes There? Approaches to Mapping Facial Appearance Diversity. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). DOI: 10.1145/2996913.2996997.
    paper doi bibtex Urban
  20. Thumbnail for Crossview Convolutional Networks
    Jacobs N, Workman S, Zhai M. 2016. Crossview Convolutional Networks. In: IEEE Applied Imagery and Pattern Recognition (AIPR). DOI: 10.1109/AIPR.2016.8010593.
    paper doi bibtex GeoAI Urban
  21. Thumbnail for Quantifying Curb Appeal
    Bessinger Z, Jacobs N. 2016. Quantifying Curb Appeal. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2016.7533189.
    website paper doi bibtex Urban
  22. Thumbnail for Analyzing Human Appearance as a Cue for Dating Images
    Salem T, Workman S, Zhai M, Jacobs N. 2016. Analyzing Human Appearance as a Cue for Dating Images. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–8. DOI: 10.1109/WACV.2016.7477678.
    website paper doi bibtex
  23. Thumbnail for Wide-Area Image Geolocalization with Aerial Reference Imagery
    Workman S, Souvenir R, Jacobs N. 2015. Wide-Area Image Geolocalization with Aerial Reference Imagery. In: IEEE International Conference on Computer Vision (ICCV). 1–9. DOI: 10.1109/ICCV.2015.451.
    code website paper doi bibtex Localization Urban
  24. Thumbnail for DeepFocal: A Method for Direct Focal Length Estimation
    Workman S, Greenwell C, Zhai M, Baltenberger R, Jacobs N. 2015. DeepFocal: A Method for Direct Focal Length Estimation. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351024.
    website paper doi bibtex Calibration Geometry
  25. Thumbnail for Face2GPS: Estimating Geographic Location from Facial Features
    Islam MT, Workman S, Jacobs N. 2015. Face2GPS: Estimating Geographic Location from Facial Features. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351072.
    website paper doi bibtex Localization Urban
  26. Thumbnail for Large-Scale Geo-Facial Image Analysis
    Islam MT, Greenwell C, Souvenir R, Jacobs N. 2015. Large-Scale Geo-Facial Image Analysis. EURASIP Journal on Image and Video Processing (JIVP):1–14. DOI: 10.1186/s13640-015-0070-9.
    website paper doi bibtex
  27. Thumbnail for On the Location Dependence of Convolutional Neural Network Features
    Workman S, Jacobs N. 2015. On the Location Dependence of Convolutional Neural Network Features. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION). 1–9. DOI: 10.1109/CVPRW.2015.7301385.
    paper doi bibtex Localization Calibration GeoAI Representation Learning
  28. Thumbnail for GeoFaceExplorer: Exploring the Geo-Dependence of Facial Attributes
    Greenwell C, Spurlock S, Souvenir R, Jacobs N. 2014. GeoFaceExplorer: Exploring the Geo-Dependence of Facial Attributes. In: ACM SIGSPATIAL International Workshop on Crowdsourced and Volunteered Geographic Information (GEOCROWD). 32–37. DOI: 10.1145/2676440.2676443.
    paper doi bibtex Urban
  29. Thumbnail for Exploring the Geo-Dependence of Human Face Appearance
    Islam MT, Workman S, Wu H, Souvenir R, Jacobs N. 2014. Exploring the Geo-Dependence of Human Face Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1042–1049. DOI: 10.1109/WACV.2014.6835989.
    paper doi bibtex
  30. Milliman T, Hufkins K, Lavine I, Jacobs N, Pless R, Richardson A, Frolking S. 2011. The PhenoCam Website: Adventures in “Crowd-Sourcing” Data Collection, Distribution and Analysis. In: American Geophysical Union Annual Meeting.
    bibtex
  31. Thumbnail for Discovering, Localizing, Calibrating, and Using Thousands of Outdoor Webcams
    Jacobs N. 2010. Discovering, Localizing, Calibrating, and Using Thousands of Outdoor Webcams. Computer Science and Engineering, Washington University in St. Louis, MO, USA.
    paper bibtex Calibration GeoAI
  32. Thumbnail for The Global Network of Outdoor Webcams: Properties and Applications
    Jacobs N, Burgin W, Fridrich N, Abrams A, Miskell K, Braswell BH, Richardson AD, Pless R. 2009. The Global Network of Outdoor Webcams: Properties and Applications. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). 111–120. DOI: 10.1145/1653771.1653789.
    paper doi bibtex Calibration Urban