Multimodal Vision Research Laboratory

MVRL

Research Area: Social Media

What can you learn about the world by looking at pictures uploaded to social networking websites? We explore the relationship between visual content and geographic location through large-scale analysis of images from social media platforms, including work on dynamic scene appearance patterns and geospatial understanding from user-generated content. Recent research includes learning probabilistic embeddings for multi-scale zero-shot soundscape mapping from social media and satellite data (PSM), mapping fine-grained textual descriptions from satellite images (Sat2Cap), and interactive event sifting using Bayesian graph neural networks. We also work on revisiting image geolocalization in the deep learning era, learning geo-temporal image features, and understanding natural beauty through large-scale social media analysis.

View All Publications →

Publications

  1. Nascimento J, Jacobs N, Rocha A. 2024. Interactive Event Sifting using Bayesian Graph Neural Networks. In: IEEE International Workshop on Information Forensics and Security (WIFS).
    paper bibtex Social Media
  2. Thumbnail for PSM: Learning Probabilistic Embeddings for Multi-scale Zero-shot Soundscape Mapping
    Khanal S, Xing E, Sastry S, Dhakal A, Xiong Z, Ahmad A, Jacobs N. 2024. PSM: Learning Probabilistic Embeddings for Multi-scale Zero-shot Soundscape Mapping. In: ACM Multimedia. DOI: 10.1145/3664647.3681620.
    code paper doi bibtex Remote Sensing Social Media GeoAI
  3. Dhakal A, Ahmad A, Khanal S, Sastry S, Kerner HR, Jacobs N. 2024. Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION).
    code paper bibtex Remote Sensing Social Media GeoAI Vision-Language
  4. Khanal S, Sastry S, Dhakal A, Jacobs N. 2023. Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping. In: British Machine Vision Conference (BMVC).
    code paper bibtex Remote Sensing Social Media Representation Learning GeoAI
  5. Zhu XX, Wang Y, Kochupillai M, Werner M, Haberle M, Hoffmann EJ, Taubenbock H, Tuia D, Levering A, Jacobs N, Kruspe A, Abdulahhad K. 2022. Geoinformation Harvesting From Social Media Data: A community remote sensing approach. IEEE Geoscience and Remote Sensing Magazine 10:150–180. DOI: 10.1109/MGRS.2022.3219584.
    paper doi bibtex Remote Sensing Social Media Calibration GeoAI
  6. Thumbnail for Revisiting Near/Remote Sensing with Geospatial Attention
    Workman S, Rafique MU, Blanton H, Jacobs N. 2022. Revisiting Near/Remote Sensing with Geospatial Attention. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR52688.2022.00182.
    paper doi bibtex Remote Sensing Social Media GeoAI
  7. Thumbnail for Content-Based Detection of Temporal Metadata Manipulation
    Padilha R, Salem T, Workman S, Andaló FA, Rocha A, Jacobs N. 2022. Content-Based Detection of Temporal Metadata Manipulation. IEEE Transactions on Information Forensics and Security:1316–1327. DOI: 10.1109/TIFS.2022.3159154.
    code website paper doi bibtex Calibration Localization Social Media Remote Sensing GeoAI
  8. Tuia D, Roscher R, Wegner JD, Jacobs N, Zhu XX, Camps-Valls G. 2021. Towards a Collective Agenda on AI for Earth Science Data Analysis. IEEE Geoscience and Remote Sensing Magazine 9:88–104. DOI: 10.1109/MGRS.2020.3043504.
    paper doi bibtex Remote Sensing Social Media Calibration GeoAI
  9. Thumbnail for Learning a Dynamic Map of Visual Appearance
    Salem T, Workman S, Jacobs N. 2020. Learning a Dynamic Map of Visual Appearance. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01245.
    website paper doi bibtex Localization Remote Sensing Social Media GeoAI
  10. Greenwell C, Workman S, Jacobs N. 2019. Implicit Land Use Mapping Using Social Media Imagery. In: IEEE Applied Imagery and Pattern Recognition (AIPR). DOI: 10.1109/AIPR47015.2019.9174570.
    paper doi bibtex Remote Sensing Social Media GeoAI
  11. Salem T, Greenwell C, Blanton H, Jacobs N. 2019. Learning to Map Nearly Anything. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900646.
    paper doi bibtex Remote Sensing Social Media GeoAI
  12. Bessinger Z, Jacobs N. 2019. A Generative Model of Worldwide Facial Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV.2019.00172.
    website paper doi bibtex Social Media
  13. Thumbnail for Learning Geo-Temporal Image Features
    Zhai M, Salem T, Greenwell C, Workman S, Pless R, Jacobs N. 2018. Learning Geo-Temporal Image Features. In: British Machine Vision Conference (BMVC).
    paper bibtex Localization Social Media Representation Learning
  14. Greenwell C, Workman S, Jacobs N. 2018. What Goes Where: Predicting Object Distributions from Above. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8519251.
    website paper doi bibtex GeoAI Remote Sensing Social Media
  15. Thumbnail for A Multimodal Approach to Mapping Soundscapes
    Salem T, Zhai M, Workman S, Jacobs N. 2018. A Multimodal Approach to Mapping Soundscapes. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8517977.
    website paper doi bibtex Remote Sensing Social Media GeoAI
  16. Thumbnail for A Unified Model for Near and Remote Sensing
    Workman S, Zhai M, Crandall D, Jacobs N. 2017. A Unified Model for Near and Remote Sensing. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.293.
    website paper doi bibtex Remote Sensing Social Media GeoAI
  17. Vo N, Jacobs N, Hays J. 2017. Revisiting IM2GPS in the Deep Learning Era. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.286.
    website paper doi bibtex Localization Social Media GeoAI
  18. Thumbnail for Understanding and Mapping Natural Beauty
    Workman S, Souvenir R, Jacobs N. 2017. Understanding and Mapping Natural Beauty. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.596.
    website paper doi bibtex Remote Sensing Social Media GeoAI
  19. Bessinger Z, Jacobs N. 2016. Quantifying Curb Appeal. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2016.7533189.
    website paper doi bibtex Social Media
  20. Jacobs N, Workman S, Zhai M. 2016. Crossview Convolutional Networks. In: IEEE Applied Imagery and Pattern Recognition (AIPR). DOI: 10.1109/AIPR.2016.8010593.
    paper doi bibtex Remote Sensing Social Media GeoAI
  21. Salem T, Workman S, Zhai M, Jacobs N. 2016. Analyzing Human Appearance as a Cue for Dating Images. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–8. DOI: 10.1109/WACV.2016.7477678.
    website paper doi bibtex Social Media
  22. Bessinger Z, Stauffer C, Jacobs N. 2016. Who Goes There? Approaches to Mapping Facial Appearance Diversity. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). DOI: 10.1145/2996913.2996997.
    paper doi bibtex Social Media
  23. Thumbnail for Large-Scale Geo-Facial Image Analysis
    Islam MT, Greenwell C, Souvenir R, Jacobs N. 2015. Large-Scale Geo-Facial Image Analysis. EURASIP Journal on Image and Video Processing (JIVP):1–14. DOI: 10.1186/s13640-015-0070-9.
    website paper doi bibtex Social Media
  24. Workman S, Greenwell C, Zhai M, Baltenberger R, Jacobs N. 2015. DeepFocal: A Method for Direct Focal Length Estimation. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351024.
    website paper doi bibtex Calibration Social Media Geometry
  25. Thumbnail for Wide-Area Image Geolocalization with Aerial Reference Imagery
    Workman S, Souvenir R, Jacobs N. 2015. Wide-Area Image Geolocalization with Aerial Reference Imagery. In: IEEE International Conference on Computer Vision (ICCV). 1–9. DOI: 10.1109/ICCV.2015.451.
    code website paper doi bibtex Social Media Localization
  26. Islam MT, Workman S, Jacobs N. 2015. Face2GPS: Estimating Geographic Location from Facial Features. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351072.
    website paper doi bibtex Localization Social Media
  27. Workman S, Jacobs N. 2015. On the Location Dependence of Convolutional Neural Network Features. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION). 1–9. DOI: 10.1109/CVPRW.2015.7301385.
    paper doi bibtex Localization Calibration Social Media GeoAI
  28. Greenwell C, Spurlock S, Souvenir R, Jacobs N. 2014. GeoFaceExplorer: Exploring the Geo-Dependence of Facial Attributes. In: ACM SIGSPATIAL International Workshop on Crowdsourced and Volunteered Geographic Information (GEOCROWD). 32–37. DOI: 10.1145/2676440.2676443.
    paper doi bibtex Social Media
  29. Islam MT, Workman S, Wu H, Souvenir R, Jacobs N. 2014. Exploring the Geo-Dependence of Human Face Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1042–1049. DOI: 10.1109/WACV.2014.6835989.
    paper doi bibtex Social Media
  30. Milliman T, Hufkins K, Lavine I, Jacobs N, Pless R, Richardson A, Frolking S. 2011. The PhenoCam Website: Adventures in “Crowd-Sourcing” Data Collection, Distribution and Analysis. In: American Geophysical Union Annual Meeting.
    bibtex Social Media
  31. Jacobs N. 2010. Discovering, Localizing, Calibrating, and Using Thousands of Outdoor Webcams. Computer Science and Engineering, Washington University in St. Louis, MO, USA.
    paper bibtex Remote Sensing Calibration Social Media GeoAI
  32. Thumbnail for The Global Network of Outdoor Webcams: Properties and Applications
    Jacobs N, Burgin W, Fridrich N, Abrams A, Miskell K, Braswell BH, Richardson AD, Pless R. 2009. The Global Network of Outdoor Webcams: Properties and Applications. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). 111–120. DOI: 10.1145/1653771.1653789.
    paper doi bibtex Calibration Social Media