Multimodal Vision Research Laboratory

MVRL

back to funding overview.

ContextualEyes: A Context-Aware Surveillance System

Overview

Today, essentially all images come with GPS data and a time-stamp, unfortunately most automated image analysis algorithms ignore this information. We propose to develop algorithms that exploit this metadata and, in addition, we propose algorithms for extracting geospatial information directly from the imagery. Explicitly considering the geospatial context will advance the state of the art in automated imagery analysis and lead to tools for improved situational awareness. This proposal is part of a broad research agenda aimed at building ContextualEyes, a surveillance platform that deeply integrates geospatial awareness. This proposal takes important steps toward this goal. We focus on three main research objectives: (1) using known weather conditions for scene understanding, (2) estimating unknown weather conditions and (3) estimating the shape and motion of clouds. We also dedicate time for continued DoD engagement with pilot projects and support for technology transition.

Related Publication(s)

  1. PDF Bessinger Z, Jacobs N. 2019. A Generative Model of Worldwide Facial Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV.2019.00172.
    bibtex | paper | website | doi
  2. PDF Jacobs N, Workman S, Souvenir R. 2016. Cloudmaps from Static Ground-View Video. Image and Vision Computing (IVC) 52:154–166. DOI: 10.1016/j.imavis.2016.05.013.
    bibtex | paper | website | doi
  3. PDF Salem T, Workman S, Zhai M, Jacobs N. 2016. Analyzing Human Appearance as a Cue for Dating Images. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–8. DOI: 10.1109/WACV.2016.7477678.
    bibtex | paper | website | doi
  4. PDF Mihail RP, Workman S, Bessinger Z, Jacobs N. 2016. Sky Segmentation in the Wild: An Empirical Study. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–6. DOI: 10.1109/WACV.2016.7477637.
    bibtex | paper | website | doi
  5. PDF Baltenberger R, Zhai M, Greenwell C, Workman S, Jacobs N. 2016. A Fast Method for Estimating Transient Scene Properties. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1–8. DOI: 10.1109/WACV.2016.7477713.
    bibtex | paper | website | doi | code
  6. PDF Zhai M, Workman S, Jacobs N. 2016. Detecting Vanishing Points using Global Image Context in a Non-Manhattan World. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR.2016.610.
    bibtex | paper | doi | code
  7. PDF Islam MT, Greenwell C, Souvenir R, Jacobs N. 2015. Large-Scale Geo-Facial Image Analysis. EURASIP Journal on Image and Video Processing (JIVP) 2015:1–14. DOI: 10.1186/s13640-015-0070-9.
    bibtex | paper | website | doi
  8. Workman S, Souvenir R, Jacobs N. 2015. Scene Shape Estimation from Multiple Partly Cloudy Days. Computer Vision and Image Understanding (CVIU):116–129. DOI: 10.1016/j.cviu.2014.10.002.
    bibtex | website | doi
  9. PDF Murdock C, Jacobs N, Pless R. 2015. Building Dynamic Cloud Maps from the Ground Up. In: IEEE International Conference on Computer Vision (ICCV). 1–9. DOI: 10.1109/ICCV.2015.85.
    bibtex | paper | doi
  10. PDF Islam MT, Workman S, Jacobs N. 2015. Face2GPS: Estimating Geographic Location from Facial Features. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351072.
    bibtex | paper | website | doi
  11. PDF Workman S, Greenwell C, Zhai M, Baltenberger R, Jacobs N. 2015. DeepFocal: A Method for Direct Focal Length Estimation. In: IEEE International Conference on Image Processing (ICIP). DOI: 10.1109/ICIP.2015.7351024.
    bibtex | paper | website | doi
  12. PDF Workman S, Mihail RP, Jacobs N. 2014. A Pot of Gold: Rainbows as a Calibration Cue. In: European Conference on Computer Vision (ECCV). 820–835. DOI: 10.1007/978-3-319-10602-1_53.
    bibtex | paper | website | doi
  13. PDF Greenwell C, Spurlock S, Souvenir R, Jacobs N. 2014. GeoFaceExplorer: Exploring the Geo-Dependence of Facial Attributes. In: ACM SIGSPATIAL International Workshop on Crowdsourced and Volunteered Geographic Information (GEOCROWD). 32–37. DOI: 10.1145/2676440.2676443.
    bibtex | paper | website | doi
  14. PDF Shi F, Zhai M, Duncan D, Jacobs N. 2014. MPCA: EM-Based PCA For Mixed-Size Image Datasets. In: IEEE International Conference on Image Processing (ICIP). 1807–1811. DOI: 10.1109/ICIP.2014.7025362.
    bibtex | paper | doi
  15. PDF Zhai M, Shi F, Duncan D, Jacobs N. 2014. Covariance-Based PCA for Multi-Size Data. In: International Conference on Pattern Recognition (ICPR). 1603–1608. DOI: 10.1109/ICPR.2014.284.
    bibtex | paper | doi
  16. PDF Islam MT, Workman S, Wu H, Souvenir R, Jacobs N. 2014. Exploring the Geo-Dependence of Human Face Appearance. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 1042–1049. DOI: 10.1109/WACV.2014.6835989.
    bibtex | paper | doi
  17. PDF Mihail RP, Blomquist G, Jacobs N. 2014. A CRF Approach to Fitting a Generalized Hand Skeleton Model. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 409–416. DOI: 10.1109/WACV.2014.6836070.
    bibtex | paper | doi
  18. PDF Jacobs N, King J, Bowers D, Souvenir R. 2014. Estimating Cloud Maps from Outdoor Image Sequences. In: IEEE Winter Conference on Applications of Computer Vision (WACV). 961–968. DOI: 10.1109/WACV.2014.6836000.
    bibtex | paper | website | doi
  19. PDF Jacobs N, Abrams A, Pless R. 2013. Two Cloud-Based Cues for Estimating Scene Structure and Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 35:2526–2538. DOI: 10.1109/TPAMI.2013.55.
    bibtex | paper | website | doi
  20. PDF Jacobs N, Workman S, Souvenir R. 2013. Scene Geometry from Several Partly Cloudy Days. In: ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC). 1–6. DOI: 10.1109/ICDSC.2013.6778227.
    bibtex | paper | website | doi
  21. PDF Islam MT, Jacobs N, Wu H, Souvenir R. 2013. Images+Weather: Collection, Validation, and Refinement. In: IEEE CVPR Workshop on Ground Truth. 1–7.
    bibtex | paper
  22. PDF Jacobs N, Islam MT, Workman S. 2013. Cloud Motion as a Calibration Cue. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1344–1351. DOI: 10.1109/CVPR.2013.177.
    bibtex | paper | website | doi
  23. PDF Murdock C, Jacobs N, Pless R. 2013. Webcam2Satellite: Estimating Cloud Maps from Webcam Imagery. In: IEEE Workshop on Applications of Computer Vision (WACV). 214–221. DOI: 10.1109/WACV.2013.6475021.
    bibtex | paper | doi
  24. PDF Mihail RP, Jacobs N, Goldsmith J. 2012. Real Time Gesture Recognition With 2 Kinect Sensors. In: International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). 1–7.
    bibtex | paper
  25. PDF Abrams A, Tucek J, Little J, Jacobs N, Pless R. 2012. LOST: Longterm Observation of Scenes (with Tracks). In: IEEE Workshop on Applications of Computer Vision (WACV). 297–304. DOI: 10.1109/WACV.2012.6163032.
    bibtex | paper | doi

Acknowledgements

This work was fully, or partially, supported by the Defense Advanced Research Project Agency (D11AP00255). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor