Thursday, December 27, 2012

Engineering engineering research: Understanding the fabric of engineering design

A house designed using our Energy3D CAD software.
Perhaps the most important change in the Next Generation Science Standards to be released in March 2013 is the elevation of engineering design to the same level of importance as of scientific inquiry (which was enshrined as a doctrine of science education in the 1996 science standards). But how much do we know about teaching engineering design in K-12 classrooms?

A house made using our Energy3D CAD software.
Surprisingly, our knowledge about students’ learning and ideation in engineering design is dismal. The Committee on Standards for K-12 Engineering Education assembled by the National Research Council in 2010 found “very little research by cognitive scientists that could inform the development of standards for engineering education in K–12.” Most educational engineering projects lacked data collection and analysis to provide reliable evidence of learning. Many simply replicated the “engineering science” model from higher education, which focuses on learning basic science for engineering rather than learning engineering design. Little was learned from these projects about students’ acquisition of design skills and development of design thinking. In the absence of in-depth knowledge about students’ design learning, it would be difficult to teach and assess engineering design.

In response to these problems, we have proposed a research initiative that will hopefully start to fill the gap. As in any scientific research, our approach is to first establish a theory of cognitive development for engineering design and then invent a variety of experimental techniques to verify research hypotheses based on the theory. This blog post introduces these ideas.

In order to study engineering design on a rigorous basis, we need a system that can automatically monitor student workflows to provide us all the fine-grain data we need to understand how they think and learn when they become expert designers from novice designers. This means we have no choice but to move the entire engineering design process onto the computer -- to be more exact, into computer-aided design (CAD) systems -- so that we can keep track of students’ workflows and extract information for inferring their learning. While some educators may be uncomfortable with the virtualization of engineering design, this actually complies with contemporary engineering practices that ubiquitously rely on CAD tools. If we have a CAD system, we can add some data mining mechanisms to turn it into a powerful experimental system for investigating student learning. Fortunately, we have created our own CAD software, Energy3D, from scratch (see the above images about it). So we can do anything we want with it. Since all the CAD tools are similar, the research results should be generalizable.

A cognitive theory of engineering design.
Next we need a cognitive theory of engineering design. Engineering design is interdisciplinary, dynamic, and complicated. It requires students to apply STEM knowledge to solve open-ended problems with a given set of criteria and constraints. It is such a complex process that I am almost certain that any cognitive theory will not be perfect. But without a cognitive theory our research would be aimless. So we must invent one.

Our cognitive theory assumes that engineering design is a process of “knitting” science and engineering. Inquiry and design are at the hearts of science and engineering practices. In an engineering project, both types of practices are needed. All engineering systems are tested during the development phase. A substantial part of engineering is to find problems through tests in order to build robust products. The diagnosis of a problem is, as a matter of fact, a process of scientific inquiry into an engineered system. The results of this inquiry process provide explanations of the problem, as well as feedback to revise the design and improve the system. The modified system with new designs is then put through further tests. Testing a new design can lead to more questions worth investigating, starting a new cycle of inquiry. This process of interwoven inquiry and design repeats itself until the system is determined to be a mature product. 

These elements in our cognitive theory all sound logical and necessary. Now the question is: If we agree on this theory, how are we going to make it happen in the classroom and how are we going to measure its degree of success? Formative assessment seems to be the key. So the next thing we need to invent is a method of formative assessment. But what should we assess in order not to miss the entire picture of learning? This requires us to develop a deep understanding of the fabric of engineering design.

A time series model of design assessment.
Engineering design is a complex process that involves multiple types of science and engineering tasks and subprocesses that occur iteratively. Along with the properties and attributes of the designed artifacts that can be calculated, the order, frequency, and duration learners handle the tasks provide invaluable insights into the fabric of engineering design. These data can be monitored and collected as time series. Formative assessment can then be viewed as the analysis of a set of time series, each representing an aspect of learning or performance. In other words, each time series logs a “fiber” of engineering design.

At first glance, the time series data may look stochastic, just like the Dow Jones index. But buried under the noisy data are students’ behavioral and cognitive patterns. Time series analysis, which has been widely used in signal processing and pattern recognition, will provide us the analytic power to detect learner behaviors from the seemingly random data and then generate adaptive feedback to steer learning to less arbitrary, more productive paths. For example, spectral or wavelet analysis can be used to calculate the frequency of using a design or test tool. Auto-correlation analysis can be used to find repeating patterns in a subprocess. Cross-correlation analysis can be used to examine if an activity or intervention in one subprocess has resulted in changes in another. Cross-correlation provides a potentially useful tool for tracking a designer’s activity with regard to knowledge integration and system thinking.

In the next six months, we will undertake this ambitious research project and post our findings in this blog as we move forward. Stay tuned!

Wednesday, December 12, 2012

Detecting students' "brain waves" during engineering design using a CAD tool

Design a city block with Energy3D.
We were in a school these two weeks doing a project that aims to understand how students learn engineering design. This has been a difficult research topic as engineering design is an extremely complicated cognitive process that involves the application of science and mathematics -- another two sets of complicated subjects themselves.

Two types of problems are commonly encountered in the classroom. The first type is related to using a "cookbook" approach that confines students to step-by-step procedures to complete a "design" project. I added double quotes because this kind of project often leads to identical or similar products from students, violating the first principle of design that mandates alternatives and varieties. However, if we make the design project completely open-ended, we will run into the second type of problem: The arbitrariness and caprice in student designs often make it difficult for teachers and researchers to assess student thinking and learning reliably. As much as we want students to be creative and open-minded, we also want to ensure that they learn what is intended and we must provide an objective way to evaluate their learning outcomes.

To tackle these issues, we are taking a computer science-based approach. Computer-aided design (CAD) tools offer an opportunity for us to move the entire process of engineering design to the computer (this is what CAD tools are designed for in the first place for industry folks). What we need to do in our research is to add a few more things to support data mining.

A sample design of the city block.
This blog post reports a timeline tool that we have developed to measure student activity levels while engaged in using a CAD tool (our Energy3D CAD software in this case) to solve a design challenge. This timeline tool is basically a logger that records the number of the learner's design actions at a given frequency (say, 2-4 times a minute) during a design session. These design actions are defined to be the "atomic" actions stored in the Undo Manager of the CAD tool we are using. The timeline approximately describes the user's frequency of construction actions with the CAD tool. As the human-computer interaction is ultimately driven by the brain, this kind of timeline data could be regarded as a reflection of the user's "brain wave."

There are four things that characterize such a timeline graph:

A sample timeline graph.
  • The height of a spike measures the action intensity at that moment, i.e., how many actions the user has taken since the last recording;
  • The density of spikes measures the continuity and persistence of actions over a time period;
  • A gap indicates an off-task time window: A short idling window may be an effect of instruction or discussion;
  • The trend of height and density may be related to loss of interest or improvement of proficiency in the CAD tool: If the intensity (the combination of height and density of spikes) drops consistently over time, the student's interest may be fading away; if the intensity increases consistently over time, the student might be improving on using the design tool to explore design options.
Timeline graphs from six students.
Of course, this kind of timeline data is not perfect. It certainly has many limitations in measuring learning. We are still in the process of analyzing these timeline data and juxtaposing them with other artifacts we have gathered from the students to provide a more comprehensive picture of design learning. But the timeline analysis represents a rudimentary step towards a more rigorous methodology for performance assessment of engineering design.

The above six "brain wave" graphs were collected from six students in a 90-minute class period. Hopefully, these data will lead to a way to identify novice designers' behaviors and patterns when they are solving a design challenge.