The Infrared Street View is an award-winning project recognized by the U.S. Department of Energy and subsequently funded by the National Science Foundation. The idea is to create a thermal equivalent of Google's Street View that would serve as the starting point to develop a thermographic information system (i.e., the "Map of Temperature"). This is an ambitious goal that is normally only attainable through big investments from the Wall Street or by big companies like Google. However, as a single developer who doesn't have a lot of resources, I decided to give it a shot on my own. Being a non-Googler, I am counting on the citizen scientists out there to help me build the Infrared Street View. The first step is to create a free app so that they have a way to contribute.
My journey started in the mid of July 2018. In two months I have learned how to develop a powerful app from scratch. At the end, the Infrared Street View is coming into sight! This blog article shows some of the (imperfect but promising) results, as demonstrated in Figures 1 and 2.
Fig. 1: Panoramas in visible light and infrared light generated by SmartIR
This milestone is about developing the functionality in the SmartIR app for creating infrared panoramas so that anyone who has a smartphone with an infrared camera attachment such as FLIR ONE could produce a panoramic image and contribute it to the Infrared Street View, much like what you can do with Google's Street View app. Although this sounds easy at first glance, it has turned out to be quite challenging as we must work under the constraint of a very slow infrared thermal camera that can only take less than ten pictures per second. As our app targets average people who may be interested in science and technology, we must provide an easy-to-do user interface so that the majority of people can do the job without being overwhelmed. Lastly, to create virtual reality in infrared light, we must overcome the challenge of stitching the imperfect thermal images together to produce a seamless panoramic picture. Although they are many image stitchers out there, no one can be sure that they would be applicable to thermal images as those stitchers may have been optimized for only visible light images.
Fig. 2: Panoramas in visible light and infrared light (two coloring styles) generated by SmartIR
To support users to make 360° panoramas, SmartIR guides them to aim at the right angles so that the resulting image set can be used for stitching. These images should be evenly distributed in the azimuthal dimension and they should overlap considerably for the stitcher to have a clue about how to knit them together. SmartIR uses the on-board sensors of the smartphone to detect the orientation of the infrared camera. A number of circles are shown on the screen to indicate the orientations which the user should aim the cursor of the camera at. When the cursor is within the circle, an image is automatically taken and stored in a 360° scroller. By turning at a fixed position and aiming at the circles, the user can capture a series of images for stitching. The following YouTube videos show how this image collector works.
Although this is a very primitive prototype, it nonetheless represents the first concrete step towards the realization of the Infrared Street View. Down the road, stitchers for infrared thermal images still need significant improvements to truly achieve seamless effects similar to those for visible light images. Tremendous challenges for weaving the Map of Temperature still lie ahead. I will keep folks posted as I inch towards the goal and I am quite optimistic that I can get somewhere, even though I am not a Googler.
A goal of the SmartIR app is to provide basic instructional support directly in the app so that students, citizen scientists, professionals, and other users can learn what they can do with the incredible power of thermal vision beyond its conventional applications. This requires a lot of development work, such as inventing the artificial intelligence that can guide them through making their own scientific discoveries and engineering decisions. While that kind of instructional power poses an enormous challenge to us, adding some instructional materials in the app so that users can get started is a smaller step that we can take right now.
As of August 30, 2018, I have added 17 experiments in physical sciences that people can do with thermal vision in SmartIR. These experiments, ranging from heat transfer to physical chemistry, are based on my own work in the field of infrared imaging in the past eight years and are all very easy to do (to the point that I call them "kitchen science"). I now finally have a way to deliver these experiments through a powerful app. Figure 1 shows the list of these experiments in SmartIR. Users can click each card to open the corresponding instructional unit (which is a sequence of steps that guide users through the selected set of experiments).
Fig. 2
To do better than just putting some HTML pages into the app, I have also built critical features that allow users to switch back and forth between the thermal view and the document view (Figure 2). When users jump to the thermal view from a document, a thumbnail view of that document is shown on top of a floating button in the thermal view window (see the left image in Figure 2), allowing users to click it and go back to the document at any time. The thumbnail also serves to remind them which experiment they are supposed to conduct. When they go back to the document, a thumbnail view of the thermal camera is shown on top of a floating button in the document view window (see the right image in Figure 2), allowing users to click it and go back to the thermal view at any time. This thumbnail view also connects to the image stream from the thermal camera so that users can see current picture the camera is displaying without leaving the document.
These instructional features will be further enhanced in the future. For instance, users will be able to insert a thermal image into a container, or even create a slide show, in an HTML page to document a finding. At the end, they will be able to use SmartIR to automatically generate a lab report.
These new features, along with what I have built in the past few weeks, mark the milestone of Version 0.0.2 of SmartIR (i.e., only 2% of the work has been done towards maturity). The following video offers a sneak peek of this humble version.
I am pleased to announce Project Snake Eyes. This ambitious project aims to combine image analysis and infrared imaging to create biomimetic thermal vision -- computer vision that simulates the ability of some animals such as snakes to see in darkness. One of the goals of Project Snake Eyes is to create probably the world's first robotic snake that can hunt through thermal sensing. Project Snake Eyes not only can detect heat, but it can also estimate
the size and proximity of the source, giving the robotic snake the artificial
intelligence to figure out whether or not it should strike
the target. Through two weeks of intense research and development, I have come up with original algorithms that allow robust automatic feature extraction in real time from thermal images through a FLIR ONE camera. This article reports my progress thus far. The algorithms will be published in a journal in the field of image processing and computer vision later.
Figure 1 shows the detection of windows at night from outside a house. Half of the window was open at the time of observation. The lower temperature of the upper pane was partially due to the reflection of infrared light from the cooler environment such as the sky by the glass. The lower pane, which was the open part of the window, was due to the warmer inside of the house. As you can see, Project Snake Eyes approximately identified the shapes and sizes of the two panes, which stood out from the background because of their distinct temperatures. This ability will be used to develop advanced algorithms to automatically analyze the thermal signature of a building, paving the road to large-scale building thermal analyses through automation.
Figure 2 shows a more complex scenario using my computer desk as an example. As you can see, Project Snake Eyes automatically detected most of the hot spots (or cold spots, but I use hot spots to represent points of interest at either high or low temperature). Feature extraction such as blob detection through thermal vision could result in enhanced computer vision for technologies such as unmanned vehicles.
Figure 3 shows object reconstruction based on residue heat using my hand as an example. The residue heat that my hand left on the wall revealed its shape as six separate polygons (the largest one corresponds to the palm and the five smaller ones to the fingertips). Object reconstruction through residue heat could find its applications in certain industry monitoring and control.
Project Snake Eyes aims to identify and track objects.
Figure 4 shows object tracking using a colleague as an example. As you can see, Project Snake Eyes basically captured his body shape. The video clip that follows shows how this works in real time. The algorithms still need optimization to reduce the lag, but you get the basic idea of how object tracking using Project Snake Eyes works. Object tracking will be used to realize animal and human tracking in dark conditions for many science and engineering applications. For instance, we are collaborating with biologists in Texas to develop a system that can track bats at night in order to monitor the health of their colonies.
With all these excitements, I expect to carry on the research and development of Project Snake Eyes in the coming years. As a landmark step, we are working tirelessly towards the unveiling of the first prototype of the robotic snake with thermal vision, collaborating with two robotics teams led by Prof. Yan Gu at the University of Massachusetts Lowell and Prof. Zhenghui Sha at the University of Arkansas, respectively.
