Natural learning interfaces

Natural user interfaces (NUIs) are the third generation of user interface for computers, after command line interfaces and graphical user interfaces. A NUI uses natural elements or natural interactions (such as voice or gestures) to control a computer program. Being natural means that the user interface is built upon something that most people are already familiar with. Thus, the learning curve can be significantly shortened. This ease of use allows computer scientists to build more complicated but richer user interfaces that simulate the existing ways people interact with the real world.

Research on NUIs is currently one of the most active areas in computer science and engineering. It is one of the most important directions of Microsoft Research. In line with this future, our NSF-funded Mixed-Reality Labs (MRL) project has proposed a novel concept called the Natural Learning Interfaces (NLIs), which represents our latest ambition to realize the educational promise of cutting-edge technology. In the context of science education, an NLI provides a natural user interface to interact with a scientific simulation on the computer. It maps a natural user action to the change of a variable in the simulation. For example, the user uses a hot or cold source to control a temperature variable in a thermal simulation. The user exerts a force to control the pressure of a gas simulation. NLIs use sensors to acquire real-time data that are then used to drive the simulation in real time. In most cases, it involves a combination of multiple sensors (or multiple types of sensors) to feed more comprehensive data to a simulation and to enrich the user interface.

I have recently invented a technology called the Frame, which may provide a rough idea of what NLIs may look like as an emerging learning technology for science education. The Frame technology is based on the fact that the frame of a computer screen is the natural boundary between the virtual world and the physical world and is, therefore, an intuitive user interface for certain human-computer interactions. Compared with other interfaces such as touch screens or motion trackers, the Frame allows users to interact with the computer from the edges of the screen.

Collaborating with Jennie Chiu's group at the University of Virginia (UVA), we have been working on a few Frame prototypes that will be field tested with several hundred Virginia students in the fall of 2012. These Frame prototypes will be manufactured using UVA's 3D printers. One of the prototypes shown in this blog post is a mixed-reality gas lab, which was designed for eighth graders to learn the particulate nature of temperature and pressure of a gas. With this prototype, students can push or pull a spring to exert a force on a virtual piston, or use a cup of hot water or ice water to adjust the temperature of the virtual molecules. The responsive simulation will immediately show the effect of those natural actions on the state of the virtual system. Besides the conventional gas law behavior, students may discover something interesting. For example, when they exert a large force, the gas molecules can be liquified, simulating gas liquifying under high pressure. When they apply a force rapidly, a high-density layer will be created, simulating the initiation of a sound wave. I can imagine that science centers and museums may be very interested in using this Frame lab as a kiosk for visitors to explore gas molecules in a quick and fun way.

A mixed-reality gas lab (a Frame prototype)

As these actions can happen concurrently, two students can control the simulation using two different mechanisms: changing temperature or changing pressure. This makes it possible for us to design a student competition in which two students use these two different mechanisms to push the piston into each other's side as far as possible. To the best of our knowledge, this is the first collaborative learning of this kind mediated by a scientific simulation.

NLIs are not just the results of some programming fun. NLIs are deeply rooted in cognitive science. Constructivism views learning as a process in which the learner actively constructs or builds new ideas or concepts based upon current and past knowledge or experience. In other words, learning involves constructing one's own knowledge from one's own experiences. NLIs are learning systems built on what learners already know or what they feel natural. The key of a NLI is that it engineers natural interactions that connect prior experiences to what students are supposed to learn, thus building a bridge for stronger mental association and deeper conceptual understanding.

Energy2D to reach thousands of schools

Thermoregulation

Project Lead The Way (PLTW) is the leading provider of rigorous and innovative Science, Technology, Engineering, and Mathematics (STEM) education curricular programs used in middle and high schools across the US. The PLTW Pathway To Engineering (PTE) program includes a foundational course called the Principles of Engineering (POE) designed for 10-11th grade students. PLTW curriculum currently reaches 4,780 schools.

According to Bennett Brown, Associate Director of Curriculum and Instruction of PLTW, our Energy2D software will be adopted in the POE curriculum to support a variety of core engineering concepts including power, energy, heat transfer, controls, and environmental factors.

Solar heating cycles

Since the release of the first alpha version in 2011, Energy2D has already been used by thousands of users worldwide, but the collaboration with PLTW will be a big step forward for Energy2D to reach more students. The timing of this collaboration is particularly important to engineering tools such as Energy2D, as--for the first time--engineering has been officially written into the US K-12 Science Education Standards. Once the Standards roll out, thousands of teachers will be looking for leading-edge tools that can help them teach engineering. This will be a great opportunity for Energy2D.

Why is Energy2D so special that people want to use it? Our website provides many self-explanatory examples. But there is one hidden gem I want to emphasize here: Its computational engine is based on good algorithms I devised specially for this simulator. Its heat solver can be so accurate that a simulation can maintain the total energy of an isolated system at a level as accurate as 99.99% for as long as it runs, regardless of the complexity of the structures in the system! The fact that the sum of energy from all the 10,000 grid cells remains a constant after billions of individual calculation steps reflects the holy grail of science and engineering. If anything, engineering is about accuracy. A good engineering tool should be able to give students a good engineering habit of mind and accuracy should be a paramount part of it.

The first Earth science simulation in Energy2D is here: Mantle convection!

It is my goal to make the Energy2D software a powerful simulation tool for a wide audience. Last week I have added some engineering examples and blogged about them.

Last night I came up with an idea for simulating mantle convection, the slow creeping motion of Earth's rocky mantle caused by convection currents carrying heat from the interior of the Earth to the surface. It turned out that the idea worked out.
 
This blog post demonstrates the first geoscience simulation created using Energy2D. The two screenshots show mantle convection at different times. The streamlines in the second image represent the convective currents. From the simulation, you can see the gradual cooling of the core due to mantle convection--This happens in the time frame of billions of years, but a computer simulation can show it in a few seconds. For simplicity, we don't distinguish the inner core and the outer core in this model. Later, we can build a more complex one that includes these subtle details.

The simulation is available online at: http://energy.concord.org/energy2d/mantle.html. Take a look and stay tuned for more Earth science simulations--brought to you by Energy2D!

Energy2D V1.0 released!

The first stable version of Energy2D, an open-source and free heat transfer simulation tool made possible by funding from the National Science Foundation, is now available for download. The program can be installed as a desktop app, which can be used to create high-quality simulations that can be deployed on the Internet as applets. It comes with about 40 templates to help you get started to design your own simulations. The Energy2D website provides plenty of examples that show how you can integrate your simulations on your websites. The examples cover a wide range of topics in heat transfer, fluid dynamics, and thermal engineering. Thermal engineering is a major feature added recently and will be expanded in the future. The example to the right, "How solar cycles affect the duty cycle of a thermostat," showcases this new feature.

When you click the "Java Webstart Installer" on the website, the software will be automatically downloaded and installed on your desktop. The website's Download page has detailed information for how to publish your Energy2D simulations or integrate them with your web stuff.

If you have used the Energy2D app before, you will need to remove the previous installation in order to enjoy the convenience of full OS integration that this version offers. For Windows users, go to "Control Panel > Java." For Mac users, go to the Java Preference. In either case, you can find the previous installation in "Temporary Internet Files."

If you have just used the online applets on our website but haven't downloaded the app, there is nothing you need to remove. Although it is perfectly fine to use the online applets as they are, we think you should try the app--It will give you the full ability to create, design, and test.