Thursday, August 2, 2018

Using SmartIR in Science Experiments

Fig. 1: The paper-on-cup experiment with SmartIR
SmartIR is a smartphone app that I am developing to support infrared (IR) thermal imaging applications, primarily in the field of science and engineering, based on the FLIR ONE SDK. The development officially kicked off in July 2018. By the end of the month, a rudimentary version, to which I assigned V 0.0.1 (representing approximately 1% of the work that needs to be done for a mature release), has been completed.

Fig. 2: Time graphs of temperatures in SmartIR
Although a very early version, SmartIR V0.0.1 can already support some scientific exploration. In this article, I share the results from doing the can't-be-simpler experiment that I did back in 2011 with a FLIR I5. This experiment needs only a cup of water, a piece of paper, and, of course, an IR camera (which is FLIR ONE Pro Generation 3 in my case). When a piece of paper is placed on top of an open cup of tap water that has sit in the room for a few hours, it warms up -- instead of cooling down -- as a result of the adsorption of water molecules onto the underside of the paper and the condensation of more water molecules to form a layer of liquid water, as shown in Figure 1.

While the user can observe this effect with any thermal camera, it is sometimes useful to also record the change of temperatures as time goes by. To do this, SmartIR allows the user to add any number of thermometers to the view (and move or delete them as needed) and show their temperature readings in a time graph on top of the thermal image view (this is sort of like the translucent sensor graph in my Energy2D computational fluid dynamics simulation program). Figure 2 shows the time graph of temperatures. To study the effect, I added three thermometers: one for measuring the ambient temperature (T3), one for measuring the temperature of water (T2), and one for measuring the temperature of the paper (T1). Note that, before the paper was placed, T1 and T2 both measured the temperature of the water in the cup. As today is pretty hot, T3 registered higher than 35 °C. Due to the effect of evaporative cooling, T2 registered about 33 °C. When a piece of paper was put on top of the cup, T1 rose to nearly 37 °C in a few seconds!

SmartIR is currently only available in Android. It hasn't been released in Google Play as intense development is expected to be under way in the next six months. A public release may be available next year.

Sunday, July 1, 2018

Artificial Intelligence Research: The Octopus Algorithm for Generating Goal-Directed Feedback

During the 2010 FIFA World Cup eight years ago, a common octopus named Paul the Octopus gained worldwide attention because it accurately "predicted" all the results of the most important soccer matches in the world (sadly it died by natural courses shortly after that). Perhaps Paul the Octopus just got extraordinarily lucky. Eight years later, as reported by the MIT Technology Review, artificial intelligence has been used in its stead to predict the World Cup (which I doubt would achieve the 100% success rate as the famous octopus did marvelously).

Fig. 1: An illustration of the Octopus Algorithm
While my research on artificial intelligence (AI) has nothing to do with predicting which team would win the World Cup this year, octopuses have become one of my inspirations in the past few days. My work is about developing AI techniques that support learning and teaching through solving vastly open-ended problems such as scientific inquiry and engineering design. One of the greatest challenges in such problem-solving tasks is about how to automatically generate formative feedback based on automatically assessing student work. Typically, the purpose of this kind of feedback is to gradually direct students to some kind of goals, for example, to achieve the most energy-efficient design of a building that meets all the specs. Formative feedback is critically important to ensuring the success of project-based learning. Based on my own experiences, however, many students have great difficulties making progress towards the goal in the short amount of time typically available in the classroom. Given the time constraints, they need help on an ongoing basis. But it is not realistic to expect the teacher to simultaneously monitor a few dozen students while they are working on their projects and provide timely feedback to each and every one of them. This is where AI can help. This is why we are developing new pedagogical principles and instructional strategies, hoping to harness the power of AI to spur students to think more deeply, explore more widely, and even design more creatively.

Fig. 2: Learning from AI through a competition-based strategy
Although this general idea of using AI in education makes sense, developing reliable algorithms that can automatically guide students to solve massively open-ended problems such as engineering design is by no means a small job. Through three months of intense work in this field, I have developed genetic algorithms that can be used to find optimal solutions in complex design environments such as the Energy3D CAD software, which you can find in earlier articles published through my blog. These algorithms were proven to be effective for optimizing certain engineering problems, but to call them AI, we will need to identify what kind of human intelligence they are able to augment or replace. In my current point of view, an apparent class of AI applications is about mimicking certain instructional capacities of peers and teachers. In order to create an artificial peer or even an artificial instructor, we would have to figure out the algorithms that simulate the constructive interactions between a student and a peer or between a student and an instructor.

Hence, an optimization algorithm that yields the best solution in a long single run is not very useful to developers and educators as it doesn't provide sufficient opportunities for engaging students. You can imagine that type of algorithm as someone who does something very fast but doesn't pause to explain to the learner how he or she does the job. To create a developmentally appropriate tool, we will need to slow down the process a bit -- sort of like the creeping of an octopus -- so that the learner can have a chance to observe, reflect, internalize, and catch up when AI is solving the problem step by step. This kind of algorithm is known as local search, a technique for finding an optimal solution in the vicinity of a starting point that represents the learner's current state (as opposed to global search that casts a wide net across the entire solution space, representing equally all possibilities regardless of the learner's current state). Random optimization is one of the local search methods proposed in 1965, which stochastically generates a set of candidate solutions distributed around the initial solution in accordance with the normal distribution. The graphical representation of a normal distribution is a bell curve that somewhat resembles the shape of an octopus (Figure 1). When using a genetic algorithm to implement the local search, the two red edge areas in Figure 1 can be imagined as the "tentacles" for the "octopus" to sense "food" (optima), while the green bulk area in the middle can be imagined as the "body" for it to "digest the catches" (i.e., to concentrate on local search). Once an optimum is "felt" (i.e., one or more solution points close to the optimum is included in the randomly generated population of the genetic algorithm), the "octopus" will move towards it (i.e., the best solution from the population will converge to the optimum) as driven by the genetic algorithm. The length of the "tentacles," characterized by the standard deviation of the normal distribution, dictates the pace in which the algorithm will find an optimum. The smaller the standard deviation, the slower the algorithm will locate an optimum. I call this particular combination of random optimization and genetic algorithm the Octopus Algorithm as it intuitively mimics how an octopus hunts on the sea floor (and, in part, to honor Paul the Octopus and to celebrate the 2018 World Cup Tournament). With a controlled drift speed, the Octopus Algorithm can be applied to incrementally correct the learner's work in a way that goes back and forth between human and computer, making it possible for us to devise a competition-based learning strategy as illustrated in Figure 2.

Although there are still a lot of technical details that need to be ironed out as we make progress, this competition-based strategy essentially represents an idea to turn a design process into some kind of adversarial gaming (e.g., chess or Go), which challenges students to race against a computer towards an agreed goal but with an unpredictable outcome (either the computer wins or the human wins). It is our hope that AI would ultimately serve as a tool to train students to design effectively just like what it has already done for training chess or Go players.

Fig. 3: Evolution of population in the Octopus Algorithm
How does the Octopus Algorithm work, I hear you are curious? I have tested it with some simple test functions such as certain sinusoidal functions (e.g., |sin(nx)|) and found that it worked for those test cases. But since I have the Energy3D platform, I can readily test my algorithms with real-world problems instead of some toy problems. As the first real-world example, let's check how it finds the optimal tilt angle of a single row of solar panels for a selected day at a given location (we can do it for the entire year, but it takes much longer to run the simulation with not much more to add in terms of testing the algorithm). Let's assume that the initial guess for the tilt angle is zero degree (if you have no idea which way and how much the solar panels should be tilted, you may just lay them flat as a reasonable starting point). Figure 3 shows the results of four consecutive runs. The graphs in the left column show the normal distributions around the initial guess and the best emerged after each round (which was used as the initial guess for the next round). The graphs in the right column show the final distribution of the population at the end of each round. The first and second runs show that the "octopus" gradually drifted left. At the end of the third run, it had converged to the final solution. It just stayed there at the end of the fourth run.

Fig. 4: Using four "octopuses" to locate four optimal orientations for the energy efficiency of a house.

Fig. 5: Locating the nearest optimum
When there are multiple optima in the solution space (a problem known as multimodal optimization), it may be appropriate to expect that AI would guide students to the nearest optimum. This may also be a recommendation by learning theories such as the Zone of Proximal Development introduced by Russian psychologist Lev Vygotsky. If a student is working in a certain area of the design space, guiding him or her to find the best option within that niche seems to be the most sensible instructional strategy. With a conventional genetic algorithm that performs global search with uniform initial selection across the solution space, there is simply no guarantee that the suggested solution would take the student's current solution into consideration, even though his/her current solution can be included as part of the first generation (which, by the way, may be quickly discarded if the solution turns out to be a bad one). The Octopus Algorithm, on the other hand, respects the student's current state and tries to walk him/her through the process stepwisely. In theory, it is a better technique to support advanced personalized learning, which is the number one in the 14 grand challenges for engineering in the 21st century posed by the National Academy of Engineering of the United States.

Let's see how the Octopus Algorithm finds multiple optima. Again, I have tested the algorithm with simple sinusoidal functions and found that it worked in those test cases. But I want to use a real-world example from Energy3D to illustrate my points. This example is concerned with determining the optimal orientation of a house, given that everything else has been fixed. By manual search, I found that there are basically four different orientations that could result in comparable energy efficiency, as shown in Figure 4.

Fig. 6: "A fat octopus" vs. "a slim octopus."
Now let's pick four different initial guesses and see which optimum each "octopus" finds. Figure 5 shows the results. The graphs in the left column show the normal distributions around the four initial guesses. The graphs in the right column show the final solutions to which the Octopus Algorithm converged. In this test case, the algorithm succeeded in ensuring nearest guidance within the zone of proximal development. Why is this important? Imagine if the student is experimenting with a southwest orientation but hasn't quite figured out the optimal angle. An algorithm that suggests that he or she should abandon the current line of thinking and consider another orientation (say, southeast) could misguide the student and is unacceptable. Once the student arrives at an optimal solution nearby, it may be desirable to prompt him/her to explore alternative solutions by choosing a different area to focus and repeat this process as needed. The ability for the algorithm to detect the three other optimal solutions simultaneously, known as multi-niche optimization, may not be essential.

Fig. 7. A "fatter octopus" may be problematic.
There is a practical problem, though. When we generate the normal distribution of solution points around the initial guess, we have to specify the standard deviation that represents the reach of the "tentacles" (Figure 6). As illustrated by Figure 7, the larger the standard deviation ("a fatter octopus"), the more likely the algorithm will find more than one optima and may lose the nearest one as a result. In most cases, finding a solution that is close enough may be good enough in terms of guidance. But if this weakness becomes an issue, we can always reduce the standard deviation to search the neighborhood more carefully. The downside is that it will slow down the optimization process, though.

In summary, the Octopus Algorithm that I have invented seems to be able to accurately guide a designer to the nearest optimal solution in an engineering design process. Unlike Paul the Octopus that relied on supernatural forces (or did it?), the Octopus Algorithm is an AI technique that we create, control, and leverage. On a separate note, since some genetic algorithms also employ tournament selection like the World Cup, perhaps Paul the Octopus was thinking like a genetic algorithm (joke)? For the computer scientists who happen to be reading this article, it may also add a new method for multi-niche optimization besides fitness sharing and probabilistic crowding.

Thursday, June 28, 2018

Computer Applications in Engineering Education Published Our Paper on CAD Research

Fig. 1: Integrated design and simulation in Energy3D
In workplaces, engineering design is supported by contemporary computer-aided design (CAD) tools capable of virtual prototyping — a full-cycle process to explore the structure, function, and cost of a complete product on the computer using modeling and simulation techniques before it is actually built. In classrooms, such software tools allow students to take on a design task without regard to the expense, hazard, and scale of the challenge. Whether it is a test that takes too long to run, a process that happens too fast to follow, a structure that no classroom can fit, or a field that no naked eye can see, students can always design a computer model to simulate, explore, and imagine how it may work in the real world. Modeling and simulation can thereby push the envelope of engineering education to cover much broader fields and engage many more students, especially for underserved communities that are not privileged to have access to expensive hardware in advanced engineering laboratories. CAD tools that are equipped with such modeling and simulation capabilities provide viable platforms for teaching and learning engineering design, because a significant part of design thinking is abstract and generic, can be learned through designing computer models that work in cyberspace, and is transferable to real-world situations.

Some researchers, however, cautioned that using CAD tools in engineering practices and education could result in negative side effects, such as circumscribed thinking, premature fixation, and bounded ideation, which undermine design creativity and erode existing culture. To put the issues in a perspective, these downsides probably exist in any type of tools — computer-based or not — to various extents, as all tools inevitably have their own strengths and weaknesses. As a matter of fact, the development history of CAD tools can be viewed as a progress of breaking through their own limitations and engendering new possibilities that could not have been achieved before. To do justice to the innovative community of CAD developers and researchers at large, we believe it is time to revisit these issues and investigate how modern CAD tools can address previously identified weaknesses. This was the reason that motivated us to publish a paper in Computer Applications in Engineering Education to expound our points of view and supporting them with research findings.

Fig. 2: Sample student work presented in the paper
The view that CAD is “great for execution, not for learning” might be true for the kind of CAD tools that were developed primarily for creating 2D/3D computer drawings for manufacturing or construction. That view, however, largely overlooks three advancements of CAD technologies:

1) System integration that facilitates formative feedback: Based on fundamental principles in science, the modeling and simulation capabilities seamlessly integrated within CAD tools can be used to analyze the function of a structure being designed and evaluate the quality of a design choice within a single piece of software (Figure 1). This differs dramatically from the conventional workflow through complicated tool chaining of solid modeling tools, pre-processors, solvers, and post-processors that requires users to master quite a variety of tools for performing different tasks or tackling different problems in order to design a virtual prototype successfully. Although the needs for many tools and even collaborators with different specialties can be addressed in the workplace using sophisticated methodologies such as 4D CAD that incorporate time or schedule-related information into a design process, it is hardly possible to orchestrate such complex operations in schools. In education, cumbersome tool switching ought to be eliminated — whenever and wherever possible and appropriate — to simplify the design process, reduce cognitive load, and shorten the time for getting formative feedback about a design idea. Being able to get rapid feedback about an idea enables students to learn about the meaning of a design parameter and its connections to others quickly by making frequent inquiries about it within the software. The accelerated feedback loop can spur iterative cycles at all levels of engineering design, which are fundamental to design ideation, exploration, and optimization. We have reported strong classroom evidence that this kind of integrated design environment can narrow the so-called “design-science gap,” empowering students to learn science through design and, in turn, apply science to design. 
2) Machine learning that generates designer information: For engineering education research, a major advantage of moving a design project to a CAD platform is that fine-grained process data (e.g., actions and artifacts), can be logged continuously and sorted automatically behind the scenes while students are trying to solve design challenges. This data mining technique can be used to monitor, characterize, or predict an individual student’s behavior and performance and even collaborative behavior in a team. The mined results can then be used to compile adaptive feedback to students, create infographic dashboards for teachers, or develop intelligent agents to assist design. The development of this kind of intelligence for a piece of CAD software to “get to know the user” is not only increasingly feasible, but also increasingly necessary if the software is to become future-proof. It is clear that deep learning from big data is largely responsible for many exciting recent advancements in science and technology and has continued to draw extensive research interest. Science ran a special issue on artificial intelligence (AI) in July 2015 and, only two years later, the magazine found itself in the position of having to catch up with another special issue. For the engineering discipline, CAD tools represent a possible means to gather user data of comparable magnitudes for developing AI of similar significance. In an earlier paper, we have explained why the process data logged by CAD software possess all the 4V characteristic features — volume, velocity, variety, and veracity — of big data as defined by IBM. 
3) Computational design that mitigates design fixation: In trying to solve a new problem, people tend to resort to their existing knowledge and experiences. While prior knowledge and experiences are important to learning according to theories such as constructivism and knowledge integration, they could also blind designers to new possibilities, a phenomenon known as design fixation. In the context of engineering education, design fixation can be caused by the perception or preconception of design subjects, the examples given to illustrate design principles, and students’ own previous designs. As it may adversely affect engineering learning to a similar degree as “cookbook labs” underrepresent science learning, design fixation may pose a central challenge to engineering education (though it has not been thoroughly evaluated among young learners in secondary schools). Emerging computational design capabilities of innovative CAD tools based on algorithmic generation and parametric modeling can suggest design permutations and variations interactively and evolutionarily, equivalent to teaming students up with virtual teammates capable of helping them explore new territories in the solution space.

To read more about this paper, click here to go to the publisher's website.

Friday, June 15, 2018

Maine Teacher Workshop on Artificial Intelligence in Engineering Education

In June 10-12, we hosted a successful teacher professional development workshop in York, Maine for 29 teachers from seven states. The theme was around the application of artificial intelligence (AI) in engineering education to assist teaching and foster learning. The workshop was supported by generous funding from General Motors and the National Science Foundation.

The teachers explored how the AI tools built in Energy3D could help students learn STEM concepts and skills required by the Next Generation Science Standards (NGSS), especially engineering design. Together we brainstormed how AI applications such as generative design might change their teaching. We believed that AI could transform STEM education from the following four aspects: (1) augment students with tools that accelerate problem solving, thereby supporting them to explore more broadly; (2) identify cognitive gaps between students' current knowledge and the learning goals, thereby enabling them to learn more deeply; (3) suggest alternative solutions beyond students' current work, thereby spurring them to think more creatively; and (4) assess students' performance by computing the distances between their solutions and the optimal ones, thereby providing formative feedback during the design process. The activities that the teachers tried were situated in the context of building science and solar engineering, facilitated by our Solarize Your World Curriculum. We presented examples that demonstrated the affordances of AI for supporting learning and teaching along the above four directions, especially in engineering design (which is highly open-ended). Teachers first learned how to design a solar farm in the conventional way and then learned how to accomplish the same task in the AI way, which -- in theory -- can lead to broader exploration, deeper understanding, better solutions, and faster feedback.

View my PowerPoint slides for more information.