Multimodal Vision Research Laboratory MVRL
← back to funding

Learning and Using Models of Geo-Temporal Appearance

Billions of geotagged and time-stamped images are publicly available via the Internet, providing a rich record of the appearance of people, places, and things across the globe. These images are a largely untapped resource that could be used to improve our understanding of the world and how it changes over time. This project develops automated methods of extracting useful information from this imagery and fusing it into high-resolution global models that capture geo-temporal trends. Once the trends have been captured, these models are used to improve performance on computer vision tasks and make geotagged imagery a usable and navigable resource for education and research in other disciplines. The project includes an education and outreach component that brings real-world problems to computer science (CS) students, mentors students across the educational spectrum, and makes the research accessible to the public.

This project develops computer vision technologies to capture spatial and temporal appearance trends and is organized into four main research thrusts: (1) investigating novel methods for extracting information from Internet imagery using weakly supervised learning, (2) developing techniques that integrate ground-level imagery with aerial and satellite data to model the expected image appearance anywhere in the world at any time, (3) evaluating methods for using such models to improve the performance of computer vision algorithms, and (4) automatically creating visual representations that make it possible for novice users to explore the learned geo-temporal trends via the Internet.

See the NSF Award Announcement for additional details.

Related Publication(s)

  1. Thumbnail for Content-Based Detection of Temporal Metadata Manipulation
    Padilha R, Salem T, Workman S, Andaló FA, Rocha A, Jacobs N. 2022. Content-Based Detection of Temporal Metadata Manipulation. IEEE Transactions on Information Forensics and Security:1316–1327. DOI: 10.1109/TIFS.2022.3159154.
    code website paper doi bibtex Calibration Localization GeoAI
  2. Thumbnail for A Structure-Aware Method for Direct Pose Estimation
    Blanton H, Workman S, Jacobs N. 2022. A Structure-Aware Method for Direct Pose Estimation. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV51458.2022.00028.
    paper doi bibtex Calibration Localization
  3. Thumbnail for Automatic Segmentation of Sinkholes Using a Convolutional Neural Network
    Rafique MU, Zhu J, Jacobs N. 2021. Automatic Segmentation of Sinkholes Using a Convolutional Neural Network. Earth and Space Science:19. DOI: 10.1002/essoar.10509794.1.
    code paper doi bibtex GeoAI Geohazards
  4. Thumbnail for Spatio-Temporal Deep Learning Approach to Map Deforestation in Amazon Rainforest
    Maretto RV, Fonseca LMG, Jacobs N, Körting TS, Bendini HN, Parente LL. 2021. Spatio-Temporal Deep Learning Approach to Map Deforestation in Amazon Rainforest. IEEE Geoscience and Remote Sensing Letters 18:771–775. DOI: 10.1109/LGRS.2020.2986407.
    doi bibtex Forestry GeoAI
  5. Thumbnail for Optical Wavelength Guided Self-Supervised Feature Learning For Galaxy Cluster Richness Estimate
    Liang G, Su Y, Lin S-C, Zhang Y, Zhang Y, Jacobs N. 2020. Optical Wavelength Guided Self-Supervised Feature Learning For Galaxy Cluster Richness Estimate. In: Workshop on Machine Learning and the Physical Sciences at the 34th Conference on Neural Information Processing Systems.
    paper bibtex Representation Learning
  6. Thumbnail for Generative Appearance Flow: A Hybrid Approach for Outdoor View Synthesis
    Rafique MU, Blanton H, Snavely N, Jacobs N. 2020. Generative Appearance Flow: A Hybrid Approach for Outdoor View Synthesis. In: British Machine Vision Conference (BMVC).
    code website paper bibtex GenAI
  7. Thumbnail for Machine Learning in Identifying Karst Sinkholes from LiDAR-Derived Topographic Depressions in the Bluegrass Region of Kentucky
    Zhu J, Nolte A, Jacobs N, Ye M. 2020. Machine Learning in Identifying Karst Sinkholes from LiDAR-Derived Topographic Depressions in the Bluegrass Region of Kentucky. Journal of Hydrology. DOI: 10.1016/j.jhydrol.2020.125049.
    doi bibtex Geohazards
  8. Thumbnail for Improved Trainable Calibration Method for Neural Networks
    Liang G, Zhang Y, Wang X, Jacobs N. 2020. Improved Trainable Calibration Method for Neural Networks. In: British Machine Vision Conference (BMVC).
    website paper bibtex Medical Uncertainty
  9. Thumbnail for Extending Absolute Pose Regression to Multiple Scenes
    Blanton H, Greenwell C, Workman S, Jacobs N. 2020. Extending Absolute Pose Regression to Multiple Scenes. In: Joint Workshop on Long-Term Visual Localization, Visual Odometry and Geometric and Learning-based SLAM (CVPR Workshop). DOI: 10.1109/CVPRW50498.2020.00027.
    paper doi bibtex Localization Calibration
  10. Thumbnail for Surface Modeling for Airborne LiDAR
    Blanton H, Grate S, Jacobs N. 2020. Surface Modeling for Airborne LiDAR. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS39084.2020.9323522.
    paper doi bibtex
  11. Thumbnail for Estimating Displaced Populations from Overhead
    Hadzic A, Christie G, Freeman J, Dismer A, Bullard S, Greiner A, Jacobs N, Mukherjee R. 2020. Estimating Displaced Populations from Overhead. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS39084.2020.9324617.
    paper doi bibtex GeoAI Urban
  12. Thumbnail for Single Image Cloud Detection via Multi-Image Fusion
    Workman S, Rafique MU, Blanton H, Greenwell C, Jacobs N. 2020. Single Image Cloud Detection via Multi-Image Fusion. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS39084.2020.9323759.
    paper doi bibtex GeoAI
  13. Thumbnail for Weakly-Supervised Self-Training for Breast Cancer Localization
    Liang G, Wang X, Zhang Y, Jacobs N. 2020. Weakly-Supervised Self-Training for Breast Cancer Localization. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC44109.2020.9176617.
    paper doi bibtex Medical
  14. Thumbnail for Neural Network Calibration for Medical Imaging Classification Using DCA Regularization
    Liang G, Zhang Y, Jacobs N. 2020. Neural Network Calibration for Medical Imaging Classification Using DCA Regularization. In: ICML 2020 workshop on Uncertainty and Robustness in Deep Learning (UDL).
    paper bibtex Medical Uncertainty
  15. Thumbnail for Beta-amyloid and tau drive early Alzheimer’s disease decline while glucose hypometabolism drives late decline
    Hammond TC, Xing X, Wang C, Ma D, Nho K, Crane PK, Elahi F, Ziegler DA, Liang G, Cheng Q, Yanckello LM, Jacobs N, Lin A-L. 2020. Beta-amyloid and tau drive early Alzheimer’s disease decline while glucose hypometabolism drives late decline. Communications Biology 3:352. DOI: 10.1038/s42003-020-1079-x.
    paper doi bibtex Medical
  16. Thumbnail for Inconsistent Performance of Deep Learning Models on Mammogram Classification
    Wang X, Liang G, Zhang Y, Blanton H, Bessinger Z, Jacobs N. 2020. Inconsistent Performance of Deep Learning Models on Mammogram Classification. Journal of the American College of Radiology. DOI: 10.1016/j.jacr.2020.01.006.
    doi bibtex Medical
  17. Thumbnail for RasterNet: Modeling Free-Flow Speed using LiDAR and Overhead Imagery
    Hadzic A, Blanton H, Song W, Chen M, Workman S, Jacobs N. 2020. RasterNet: Modeling Free-Flow Speed using LiDAR and Overhead Imagery. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION). DOI: 10.1109/CVPRW50498.2020.00112.
    website paper doi bibtex Transportation GeoAI Urban
  18. Thumbnail for Learning a Dynamic Map of Visual Appearance
    Salem T, Workman S, Jacobs N. 2020. Learning a Dynamic Map of Visual Appearance. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01245.
    website paper doi bibtex Localization GeoAI
  19. Thumbnail for Dynamic Traffic Modeling from Overhead Imagery
    Workman S, Jacobs N. 2020. Dynamic Traffic Modeling from Overhead Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01233.
    website paper doi bibtex Transportation GeoAI Urban
  20. Thumbnail for Deep Learning for Conifer/Deciduous Classification of Airborne LiDAR 3D Point Clouds Representing Individual Trees
    Hamraz H, Jacobs NB, Contreras MA, Clark CH. 2019. Deep Learning for Conifer/Deciduous Classification of Airborne LiDAR 3D Point Clouds Representing Individual Trees. ISPRS Journal of Photogrammetry and Remote Sensing 158:219–230. DOI: 10.1016/j.isprsjprs.2019.10.011.
    paper doi bibtex Forestry GeoAI
  21. Thumbnail for Learning to Map Nearly Anything
    Salem T, Greenwell C, Blanton H, Jacobs N. 2019. Learning to Map Nearly Anything. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900646.
    paper doi bibtex GeoAI Urban
  22. Thumbnail for Remote Estimation of Free-Flow Speeds
    Song W, Salem T, Blanton H, Jacobs N. 2019. Remote Estimation of Free-Flow Speeds. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900286.
    paper doi bibtex Transportation GeoAI Urban
  23. Thumbnail for Weakly Supervised Building Segmentation from Aerial Images
    Rafique MU, Jacobs N. 2019. Weakly Supervised Building Segmentation from Aerial Images. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8898812.
    paper doi bibtex GeoAI Urban
  24. Thumbnail for Weakly Supervised Fusion of Multiple Overhead Images
    Rafique MU, Blanton H, Jacobs N. 2019. Weakly Supervised Fusion of Multiple Overhead Images. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION). DOI: 10.1109/CVPRW.2019.00189.
    paper doi bibtex GeoAI Urban
  25. Zhu J, Nolte AM, Jacobs N, Ye M. 2019. Incorporating Machine Learning with LiDAR for Delineating Sinkholes. In: Kentucky Water Resources Annual Symposium.
    paper bibtex GeoAI Geohazards
  26. Thumbnail for A Weakly Supervised Approach for Estimating Spatial Density Functions from High-Resolution Satellite Imagery
    Jacobs N, Kraft A, Rafique MU, Sharma RD. 2018. A Weakly Supervised Approach for Estimating Spatial Density Functions from High-Resolution Satellite Imagery. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). DOI: 10.1145/3274895.3274934.
    code paper doi bibtex GeoAI Urban
  27. Thumbnail for Learning Geo-Temporal Image Features
    Zhai M, Salem T, Greenwell C, Workman S, Pless R, Jacobs N. 2018. Learning Geo-Temporal Image Features. In: British Machine Vision Conference (BMVC).
    paper bibtex Localization Representation Learning
  28. Thumbnail for Learning to Look around Objects for Top-View Representations of Outdoor Scenes
    Schulter S, Zhai M, Jacobs N, Chandraker M. 2018. Learning to Look around Objects for Top-View Representations of Outdoor Scenes. In: European Conference on Computer Vision (ECCV). DOI: 10.1007/978-3-030-01267-0_48.
    paper doi bibtex GenAI Transportation
  29. Thumbnail for What Goes Where: Predicting Object Distributions from Above
    Greenwell C, Workman S, Jacobs N. 2018. What Goes Where: Predicting Object Distributions from Above. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8519251.
    website paper doi bibtex GeoAI Urban
  30. Thumbnail for A Multimodal Approach to Mapping Soundscapes
    Salem T, Zhai M, Workman S, Jacobs N. 2018. A Multimodal Approach to Mapping Soundscapes. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8517977.
    website paper doi bibtex GeoAI Urban
  31. Thumbnail for FARSA: Fully Automated Roadway Safety Assessment
    Song W, Workman S, Hadzic A, Souleyrette R, Green E, Chen M, Zhang X, Jacobs N. 2018. FARSA: Fully Automated Roadway Safety Assessment. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV.2018.00063.
    paper doi bibtex Transportation GeoAI Urban
  32. Thumbnail for Revisiting IM2GPS in the Deep Learning Era
    Vo N, Jacobs N, Hays J. 2017. Revisiting IM2GPS in the Deep Learning Era. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.286.
    website paper doi bibtex Localization GeoAI Urban
  33. Thumbnail for A Unified Model for Near and Remote Sensing
    Workman S, Zhai M, Crandall D, Jacobs N. 2017. A Unified Model for Near and Remote Sensing. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.293.
    website paper doi bibtex GeoAI Urban
  34. Thumbnail for Understanding and Mapping Natural Beauty
    Workman S, Souvenir R, Jacobs N. 2017. Understanding and Mapping Natural Beauty. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.596.
    website paper doi bibtex GeoAI Urban
  35. Thumbnail for Predicting Ground-Level Scene Layout from Aerial Imagery
    Zhai M, Bessinger Z, Workman S, Jacobs N. 2017. Predicting Ground-Level Scene Layout from Aerial Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR.2017.440.
    code paper doi bibtex GenAI Localization GeoAI Urban

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. IIS-1553116. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Grant Details

  • Role: PI
  • Funding Source: NSF
  • Budget: $499,426
  • Duration: 2016 – 2021
More Funding: View All Grants →