A Small Step towards a Big Dream of Infrared Street View

The Infrared Street View is an award-winning project recognized by the U.S. Department of Energy and subsequently funded by the National Science Foundation. The idea is to create a thermal equivalent of Google's Street View that would serve as the starting point to develop a thermographic information system (i.e., the "Map of Temperature"). This is an ambitious goal that is normally only attainable through big investments from the Wall Street or by big companies like Google. However, as a single developer who doesn't have a lot of resources, I decided to give it a shot on my own. Being a non-Googler, I am counting on the citizen scientists out there to help me build the Infrared Street View. The first step is to create a free app so that they have a way to contribute.

My journey started in the mid of July 2018. In two months I have learned how to develop a powerful app from scratch. At the end, the Infrared Street View is coming into sight! This blog article shows some of the (imperfect but promising) results, as demonstrated in Figures 1 and 2.

Fig. 1: Panoramas in visible light and infrared light generated by SmartIR

This milestone is about developing the functionality in the SmartIR app for creating infrared panoramas so that anyone who has a smartphone with an infrared camera attachment such as FLIR ONE could produce a panoramic image and contribute it to the Infrared Street View, much like what you can do with Google's Street View app. Although this sounds easy at first glance, it has turned out to be quite challenging as we must work under the constraint of a very slow infrared thermal camera that can only take less than ten pictures per second. As our app targets average people who may be interested in science and technology, we must provide an easy-to-do user interface so that the majority of people can do the job without being overwhelmed. Lastly, to create virtual reality in infrared light, we must overcome the challenge of stitching the imperfect thermal images together to produce a seamless panoramic picture. Although they are many image stitchers out there, no one can be sure that they would be applicable to thermal images as those stitchers may have been optimized for only visible light images.

Fig. 2: Panoramas in visible light and infrared light (two coloring styles) generated by SmartIR

To support users to make 360° panoramas, SmartIR guides them to aim at the right angles so that the resulting image set can be used for stitching. These images should be evenly distributed in the azimuthal dimension and they should overlap considerably for the stitcher to have a clue about how to knit them together. SmartIR uses the on-board sensors of the smartphone to detect the orientation of the infrared camera. A number of circles are shown on the screen to indicate the orientations which the user should aim the cursor of the camera at. When the cursor is within the circle, an image is automatically taken and stored in a 360° scroller. By turning at a fixed position and aiming at the circles, the user can capture a series of images for stitching. The following YouTube videos show how this image collector works.

Although this is a very primitive prototype, it nonetheless represents the first concrete step towards the realization of the Infrared Street View. Down the road, stitchers for infrared thermal images still need significant improvements to truly achieve seamless effects similar to those for visible light images. Tremendous challenges for weaving the Map of Temperature still lie ahead. I will keep folks posted as I inch towards the goal and I am quite optimistic that I can get somewhere, even though I am not a Googler.

The 2017 Energy Innovation Forum

We are invited to present at the Energy Innovation Forum on October 18 organized by the University of Massachusetts Lowell and the Massachusetts Clean Energy Center. The event will connect about 30 companies in Massachusetts with funders, investors, university researchers, and industry leaders to stimulate innovations in energy technologies.

For those who cannot attend the event, I am sharing our two posters here. You can also take a look at the PowerPoint slides for the Infrared Street View Project and the Virtual Solar Grid Project (we will do both oral and poster presentations). Both projects focus on developing a unique crowdsourcing model that integrates STEM education and energy research. The projects provide examples of using citizen science to support and engage a large number of students to learn science and engineering and participate in large-scale energy research.

The Infrared Street View Project will support research and education in the field of energy efficiency whereas the Virtual Solar Grid Project will support research and education in the field of renewable energy (primarily solar energy at present). Both projects are based on cutting-edge technologies being developed in my lab.

National Science Foundation funds citizen science project to crowdsource an infrared street view

We are pleased to announce that the National Science Foundation has awarded us a two-year, $500,000 exploratory grant to develop, test, and evaluate a citizen science program that engages youth to investigate energy issues through scientific inquiry with innovative technology. The project will crowd-create the Infrared Street View, a citizen science program that aims to produce a thermal version of Google's Street View using an affordable infrared (IR) camera attached to a smartphone. In collaboration with high schools and out-of-school programs in Massachusetts, we will conduct pilot-tests with approximately 200 students in this exploratory phase. The project will develop SmartIR, a smartphone app that will guide users to collect IR images on both Android and iOS platforms for synthesizing a seamless street view. Figure 1 shows a prototype of the Infrared Street View and Figure 2 shows a little math behind the scenes.

Fig. 1: A hemispherical infrared street view (prototype)

In essence, an IR camera serves as a high-throughput data acquisition instrument that collects thousands of temperature data points each time a picture is taken. With this incredible tool, youth can collect massive geotagged thermal data that have considerable scientific and educational value for visualizing energy usage and improving energy efficiency at all levels. The Infrared Street View program will provide a Web-based platform for youth and anyone interested in energy efficiency to view and analyze the aggregated data to identify possible energy losses. By sharing their scientific findings with stakeholders, youth will make changes to the way energy is being used. 

We are completely aware of possible legal implications and complications of the proposed citizen science program. In the case of Kyllo v. United States in 2001,  the Supreme Court has ruled that the use of a thermal camera from a public vantage point to monitor the radiation of heat from a person's home was a “search” within the meaning of the Fourth Amendment, and thus required a warrant. The ruling seems to be limited to the use of thermal cameras by law enforcement, however. Back then, IR cameras were available to only a handful of professionals, but they are only $200 nowadays and just a few clicks away on Amazon. The widespread use of smartphone-based IR cameras is making thermal images commonplace on the Internet and it is probably an interesting question for law scholars to study how civilian use of IR cameras should be regulated.

Fig. 2: Math behind the scenes.

Regardless, we will take the privacy issue very seriously and will take every precaution that we can think of to avoid potential side effects resulted from this well-intentioned program. Fortunately, we have a lot of public supports to conduct this research on large public buildings and possible commercial buildings, where the concerns of privacy are far less than private residential buildings and the needs to reduce the energy waste of those buildings and save taxpayer dollars are far more pressing. Hence, we will start with school, public, and commercial buildings in selected areas where performing thermal scan of the buildings and publishing their thermal images for educational and research purposes are permitted by school leaders, town officials, and property owners.  

From a broader perspective, the Infrared Street View program could serve as a pilot test that may shed light on increasingly important issues related to citizen privacy in the era of the Internet of Things (IoT), which features the ubiquity of sensor data collection that could be viewed by many as invasive into their physical space (not just cyberspace). While no one can deny the tremendous potential of the technology in transforming the ways people learn, work, and live, careful research must be carried out to address legitimate concerns. This program could be one of those projects that provide a unique approach to meet those challenges from a citizen science point of view, which integrates many interesting scientific, technical, educational, and legal aspects. The lessons we can learn from conducting this work could be very useful to the citizen science community in the IoT era.

A demo of the Infrared Street View

An infrared street view

The award-winning Infrared Street View program is an ambitious project that aims to create something similar to Google's Street View, but in infrared light. The ultimate goal is to develop the world's first thermographic information system (TIS) that allows the positioning of thermal elements and the tracking of thermal processes on a massive scale. The applications include building energy efficiency, real estate inspection, and public security monitoring, to name a few.

An infrared image sphere

The Infrared Street View project is based on infrared cameras that work with now ubiquitous smartphones. It takes advantages of the orientation and location sensors of smartphones to store information necessary to knit an array of infrared thermal images taken at different angles and positions into a 3D image that, when rendered on a dome, creates an illusion of immersive 3D effects for the viewer.

The project was launched in 2016 and later joined by three brilliant computer science undergraduate students, Seth Kahn, Feiyu Lu, and Gabriel Terrell, from Tufts University, who developed a primitive system consisting of 1) an iOS frontend app to collect infrared image spheres, 2) a backend cloud app to process the images, and 3) a Web interface for users to view the stitched infrared images anchored at selected locations on a Google Maps application.

The following YouTube video demonstrates an early concept played out on an iPhone:

Infrared Street View won Department of Energy's JUMP competition

Creating an infrared street view using SmartIR and FLIR ONE

Our Infrared Street View (ISV) program has won the JUMP Competition sponsored jointly by CLEAResult, the largest provider of energy efficiency programs and services in North America, and the National Renewable Energy Laboratory (NREL), a research division of the US Department of Energy (DOE). This JUMP Competition called for innovations in using smartphones' sensing capabilities to improve residential energy efficiency. Finalists were selected from a pool of submitted proposals and invited to make their pitches to the audience at the CLEAResult Energy Forum held in Austin, TX on October 4-6, 2016. There is only one winner among all the good ideas for each competition. This year, we just happened to be one.

IR homework

We envision the Infrared Street View as an infrared (IR) counterpart of Google's Street View (I know, I know, this is probably too big to swallow for an organization that is a few garages small). Unlike Google's Street View in the range of visible light, the Infrared Street View will provide a gigantic database of thermal images in the range of invisible IR light emitted by molecular vibrations related to thermal energy. If you think about these images in a different way, they actually are a massive 3D web of temperature data points. What is the value of this big data? If the data are collected in the right way, they may represent the current state of the energy efficiency of our neighborhoods, towns, cities, and even states. In a sense, what we are talking about is in fact a thermographic information system (TIS).

We are not the only group that realized this possibility (but we are likely the first one that came up with the notion and name of TIS). A few startup companies in Boston area have worked in this frontier earlier this decade. But none of them has tapped into the potential of smartphone technologies. With a handful of drive-by trucks or fly-by drones with a bunch of mounted infrared cameras, it probably would take these companies a century to complete this thermal survey for the entire country. Furthermore, the trucks can only take images from the front of a building and the drones can only take images from above, which mean that their data are incomplete and cannot be used to create the thermal web that we are imagining. In some cases, unsolicited thermal scan of people's houses may even cause legal troubles as thermal signatures may accidentally disclose sensitive information.

Our solution is based on FLIR ONE, a $200-ish thermal camera that can be plugged into a smartphone (iOS or Android). The low cost of FLIR ONE, for the first time in history, makes it possible for the public to participate in this thermal survey. But even with the relatively low price tag, it is simply unrealistic to expect that a lot of people will buy the camera and scan their own houses. So where can we find a lot of users who would volunteer to participate in this effort?

Let's look elsewhere. There are four million children entering the US education system each year. Every single one of them is required to spend a sizable chunk of their education on learning thermal science concepts -- in a way that currently relies on formalism (the book shows you the text and math, you read the text and do the math). IR cameras, capable of visualizing otherwise invisible heat flow and distribution, is no doubt the best tool for teaching and learning thermal energy and heat transfer (except for those visually impaired -- my apology). I think few science teachers would disagree with that. And starting this year, educational technology vendors like Vernier and Pasco are selling IR cameras to schools.

What if we teach students thermal science in the classroom with an IR camera and then ask them to inspect their own homes with the camera as a homework assignment? At the end, we then ask them to acquire their parents' permissions and contribute their IR images to the Infrared Street View project. If millions of students do this, then we will have an ongoing crowdsourcing project that can engage and mobilize many generations of students to come.

Sensor-based artificial intelligence

We can't take students' IR images seriously, I hear you criticizing. True, students are not professionals and they make mistakes. But there is a way to teach them how to act and think like professionals, which is actually a goal of the Next Generation Science Standards that define the next two or three decades of US science education. Aside from a curriculum that teaches students how to use IR cameras (skills) and how to interpret IR images (concepts), we are also developing a powerful smartphone app called SmartIR. This app has many innovations but two of them may lead to true breakthroughs in the field of thermography.

Thermogram sphere

The first one is sensor-based intelligence. Modern smartphones have many built-in sensors, including the visible light cameras. These sensors and cameras are capable of collecting multiple types of data. The increasingly powerful libraries of computer vision only enrich this capability even more. Machine learning can infer what students are trying to do by analyzing these data. Based on the analysis results, SmartIR can then automatically guide students in real time. This kind of artificial intelligence (AI) can help students avoid common mistakes in infrared thermography and accelerate their thermal survey, especially when they are scanning buildings independently (when there is no experienced instructor around to help them). For example, the SmartIR app can check if the inspection is being done at night or during the day. If it is during the day (because the clock says so or the ambient light sensor says so), then SmartIR will suggest that students wait to do their scan until nightfall eliminates the side effect of solar heating and lowers the indoor-outdoor temperature difference to a greater degree. With an intelligent app like this, we may be able to increase the quality and reliability of the IR images that are fed to the Infrared Street View project.

Virtual infrared reality (VIR) viewed with Google Cardboard

The second one is virtual infrared reality, or VIR in short, to accomplish true, immersive thermal vision. VIR is a technology that integrates infrared thermography with virtual reality (VR). Based on the orientation and GPS sensors of the phone, SmartIR can create what we called a thermogram sphere and then knit them together to render a seamless IR view. A VIR can be uploaded to Google Maps so that the public can experience it using a VR viewer, such as Google's Cardboard Viewer. We don't know if VIR is going to do any better than 2D IR images in promoting the energy efficiency business, but it is reasonable to assume that many people would not mind seeing a cool (or hot) view like this while searching their dream houses. For the building science professionals, this may even have some implications because VIR provides a way to naturally organize the thermal images of a building to display a more holistic view of what is going on thermally.

With these innovations, we may eventually be able to realize our vision of inventing a visual 3D web of thermal data, or the thermographic information system, that will provide a massive data set for governments and companies to assess the state of residential energy efficiency on an unprecedented scale and with incredible detail.