ContextualEyes: A Context-Aware Surveillance System

Today, essentially all images come with GPS data and a time-stamp, unfortunately most automated image analysis algorithms ignore this information. We propose to develop algorithms that exploit this metadata and, in addition, we propose algorithms for extracting geospatial information directly from the imagery. Explicitly considering the geospatial context will advance the state of the art in automated imagery analysis and lead to tools for improved situational awareness. This proposal is part of a broad research agenda aimed at building ContextualEyes, a surveillance platform that deeply integrates geospatial awareness. This proposal takes important steps toward this goal. We focus on three main research objectives: (1) using known weather conditions for scene understanding, (2) estimating unknown weather conditions and (3) estimating the shape and motion of clouds. We also dedicate time for continued DoD engagement with pilot projects and support for technology transition.

Related Publication(s)

  1. Thumbnail for A Generative Model of Worldwide Facial Appearance
    Bessinger Z, Jacobs N. 2019. A Generative Model of Worldwide Facial Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV.2019.00172.
  2. Thumbnail for Cloudmaps from Static Ground-View Video
    Jacobs N, Workman S, Souvenir R. 2016. Cloudmaps from Static Ground-View Video. Image and Vision Computing (IVC) 52:154–166. DOI: 10.1016/j.imavis.2016.05.013.
  3. Thumbnail for Detecting Vanishing Points using Global Image Context in a Non-Manhattan World
    Zhai M, Workman S, Jacobs N. 2016. Detecting Vanishing Points using Global Image Context in a Non-Manhattan World. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR.2016.610.
  4. Thumbnail for Analyzing Human Appearance as a Cue for Dating Images
    Salem T, Workman S, Zhai M, Jacobs N. 2016. Analyzing Human Appearance as a Cue for Dating Images. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–8. DOI: 10.1109/WACV.2016.7477678.
  5. Thumbnail for Sky Segmentation in the Wild: An Empirical Study
    Mihail RP, Workman S, Bessinger Z, Jacobs N. 2016. Sky Segmentation in the Wild: An Empirical Study. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–6. DOI: 10.1109/WACV.2016.7477637.
  6. Thumbnail for A Fast Method for Estimating Transient Scene Properties
    Baltenberger R, Zhai M, Greenwell C, Workman S, Jacobs N. 2016. A Fast Method for Estimating Transient Scene Properties. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–8. DOI: 10.1109/WACV.2016.7477713.
  7. Thumbnail for Building Dynamic Cloud Maps from the Ground Up
    Murdock C, Jacobs N, Pless R. 2015. Building Dynamic Cloud Maps from the Ground Up. In: IEEE International Conference on Computer Vision (ICCV). 1–9. DOI: 10.1109/ICCV.2015.85.
  8. Thumbnail for Face2GPS: Estimating Geographic Location from Facial Features
    Islam MT, Workman S, Jacobs N. 2015. Face2GPS: Estimating Geographic Location from Facial Features. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351072.
  9. Thumbnail for DeepFocal: A Method for Direct Focal Length Estimation
    Workman S, Greenwell C, Zhai M, Baltenberger R, Jacobs N. 2015. DeepFocal: A Method for Direct Focal Length Estimation. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351024.
  10. Thumbnail for Large-Scale Geo-Facial Image Analysis
    Islam MT, Greenwell C, Souvenir R, Jacobs N. 2015. Large-Scale Geo-Facial Image Analysis. EURASIP Journal on Image and Video Processing (JIVP):1–14. DOI: 10.1186/s13640-015-0070-9.
  11. Thumbnail for Scene Shape Estimation from Multiple Partly Cloudy Days
    Workman S, Souvenir R, Jacobs N. 2015. Scene Shape Estimation from Multiple Partly Cloudy Days. Computer Vision and Image Understanding (CVIU):116–129. DOI: 10.1016/j.cviu.2014.10.002.
  12. Thumbnail for GeoFaceExplorer: Exploring the Geo-Dependence of Facial Attributes
    Greenwell C, Spurlock S, Souvenir R, Jacobs N. 2014. GeoFaceExplorer: Exploring the Geo-Dependence of Facial Attributes. In: ACM SIGSPATIAL International Workshop on Crowdsourced and Volunteered Geographic Information (GEOCROWD). 32–37. DOI: 10.1145/2676440.2676443.
  13. Thumbnail for A Pot of Gold: Rainbows as a Calibration Cue
    Workman S, Mihail RP, Jacobs N. 2014. A Pot of Gold: Rainbows as a Calibration Cue. In: European Conference on Computer Vision (ECCV). 820–835. DOI: 10.1007/978-3-319-10602-1_53.
  14. Thumbnail for MPCA: EM-Based PCA For Mixed-Size Image Datasets
    Shi F, Zhai M, Duncan D, Jacobs N. 2014. MPCA: EM-Based PCA For Mixed-Size Image Datasets. In: IEEE International Conference on Image Processing (ICIP). 1807–1811. DOI: 10.1109/ICIP.2014.7025362.
  15. Thumbnail for Covariance-Based PCA for Multi-Size Data
    Zhai M, Shi F, Duncan D, Jacobs N. 2014. Covariance-Based PCA for Multi-Size Data. In: International Conference on Pattern Recognition (ICPR). 1603–1608. DOI: 10.1109/ICPR.2014.284.
  16. Thumbnail for Exploring the Geo-Dependence of Human Face Appearance
    Islam MT, Workman S, Wu H, Souvenir R, Jacobs N. 2014. Exploring the Geo-Dependence of Human Face Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1042–1049. DOI: 10.1109/WACV.2014.6835989.
  17. Thumbnail for A CRF Approach to Fitting a Generalized Hand Skeleton Model
    Mihail RP, Blomquist G, Jacobs N. 2014. A CRF Approach to Fitting a Generalized Hand Skeleton Model. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 409–416. DOI: 10.1109/WACV.2014.6836070.
  18. Thumbnail for Estimating Cloud Maps from Outdoor Image Sequences
    Jacobs N, King J, Bowers D, Souvenir R. 2014. Estimating Cloud Maps from Outdoor Image Sequences. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 961–968. DOI: 10.1109/WACV.2014.6836000.
  19. Thumbnail for Scene Geometry from Several Partly Cloudy Days
    Jacobs N, Workman S, Souvenir R. 2013. Scene Geometry from Several Partly Cloudy Days. In: ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC). 1–6. DOI: 10.1109/ICDSC.2013.6778227.
  20. Thumbnail for Two Cloud-Based Cues for Estimating Scene Structure and Camera Calibration
    Jacobs N, Abrams A, Pless R. 2013. Two Cloud-Based Cues for Estimating Scene Structure and Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 35:2526–2538. DOI: 10.1109/TPAMI.2013.55.
  21. Thumbnail for Images+Weather: Collection, Validation, and Refinement
    Islam MT, Jacobs N, Wu H, Souvenir R. 2013. Images+Weather: Collection, Validation, and Refinement. In: IEEE CVPR Workshop on Ground Truth. 1–7.
  22. Thumbnail for Cloud Motion as a Calibration Cue
    Jacobs N, Islam MT, Workman S. 2013. Cloud Motion as a Calibration Cue. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1344–1351. DOI: 10.1109/CVPR.2013.177.
  23. Thumbnail for Webcam2Satellite: Estimating Cloud Maps from Webcam Imagery
    Murdock C, Jacobs N, Pless R. 2013. Webcam2Satellite: Estimating Cloud Maps from Webcam Imagery. In: IEEE Workshop on Applications of Computer Vision (WACV). 214–221. DOI: 10.1109/WACV.2013.6475021.
  24. Thumbnail for Real Time Gesture Recognition With 2 Kinect Sensors
    Mihail RP, Jacobs N, Goldsmith J. 2012. Real Time Gesture Recognition With 2 Kinect Sensors. In: International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). 1–7.
  25. Thumbnail for LOST: Longterm Observation of Scenes (with Tracks)
    Abrams A, Tucek J, Little J, Jacobs N, Pless R. 2012. LOST: Longterm Observation of Scenes (with Tracks). In: IEEE Workshop on Applications of Computer Vision (WACV). 297–304. DOI: 10.1109/WACV.2012.6163032.

Acknowledgements

This work was fully, or partially, supported by the Defense Advanced Research Project Agency (D11AP00255). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor

Grant Details

  • Role: PI
  • Funding Source: DARPA
  • Budget: $743,131
  • Duration: 2011 – 2015