Real time Tracking and displaying multiple objects in extreme lighting conditions.

Since 2012, the introduction of neural networks has brought about a paradigm change in computer vision. Convolution Neural Network CNN based computer vision algorithms have achieved unprecedented accuracy in tasks such as image classification, object detection and semantic segmentation. in this project we mainly focus on the computer vision task of scene recognition via multiple object detection and tracking. Although state-of-the-art (SOTA) algorithms have achieved unprecedented accuracy in object detection and tracking, this is a highly challenging task especially when it has to be accomplished under extreme lighting conditions. In such extreme lighting conditions, one object can be in a very dark area (e.g. in a deep shadow) while another may be in a very bright area (e.g. in the bright sunshine). In such a scenario, even SOTA CNNs are unable to perform adequately due to their inability to robustly extract features in over-exposed or underexposed regions of a scene (image).  

in this project we attempt to tackle this problem with the introduction of High Dynamic Range (HDR) imaging techniques which can help mitigate the missing information problem by capturing and processing a large dynamic range of lighting that exists in the real world. However, most SOTA detection datasets and CNNs trained on those datasets are unable to effectively process and extract features from HDR images as the models are typically trained with traditional 8-bits/pixel/channel information.

in this project we follow two distinctive approaches to use the extra information provided by HDR imagery.

  1. First, we design and develop a robust pipeline to use existing HDR to traditional Low Dynamic Range (LDR) mapping techniques to convert HDR images to LDR images which are subsequently fed to CNNs trained on SOTA datasets for detection.
    • This technique provides backward compatibility to use existing HDR to LDR mapping techniques and pre-trained detection algorithms for HDR object detection under extreme lighting conditions
  2. Second, we propose a novel methodology to convert existing SOTA datasets to pseudo-HDR datasets in order to modify and train existing CNN based detectors with HDR data.
    • We also provide a detailed evaluation which establishes that HDR based models are indeed able to perform better than LDR based models under extreme lighting conditions
  3. Finally, we develop a fully annotated mini-dataset of approx. 4000 HDR images which is subsequently used to evaluate the performance and veracity of our approaches.