Earlier this year I took a Summer Science Research job at the University of Mary Washington in exchange for getting a free summer class at the college. My professor wanted to study the benefits of SecondLife for educational purposes and computer aided design (CAD). My task was to create an experiment in which someone was asked to build something with instructions written on a piece of paper versus instructions programmed in SecondLife (SL).
SecondLife participants were asked to take a tutorial before starting the experiment to familiarize themselves with the software. Following this they were asked to build the Lego house using the visual instructions programmed in SecondLife, which graphically placed the blocks one piece at a time so participants could see the house’s construction block by block. The program allowed participants to view the house from various angles and as a 3D representation, a feature flat instructions are lacking.
The control group was asked to build the Lego house using the paper instructions provided in the Lego kit. Both groups were timed to see how long it took them to put the Lego house together and were scored on how many pieces were left unused or out of place.
There is a large difference between the group that used the instruction manual and the group that used SecondLife. The average time to complete the Lego house using the paper manual was 18.2 minutes. The average SecondLife tutorial time was 7.28 minutes. The average time to complete, the Lego house using SecondLife was 52.5 minutes. The average time of the SecondLife house for those who gave up was 49.33 minutes, close to the average of those who actually did complete the house. This leaves an average of 51.14 minutes including those who did and did not complete the Lego house.
Another thing to notice is the accuracy of the completed Lego houses. Using the paper manual an average of 0.33 were missing pieces or placed pieces incorrectly. Using SecondLife all of the participants were missing pieces or placed pieces incorrectly.
In comparing the SecondLife tutorial time to the time it took to complete the house there appears to be a difference in the accuracy of the pieces but not in the time spent building the house.
Surveys were used to compare the usability and satisfaction of the participants after the experiment. Although 50% of the group using SecondLife found learning to use the program was moderately easy and 83.3% thought remember names and commands were easy or moderately easy, 66.6% had difficulty visualizing the next block. Of the participants using the paper manual 66.7% found visualizing the next block to be easy or moderately easy. 100% of the SecondLife participants believed the blocks only sometimes followed a logical sequence. 83.3% of the SecondLife participants were confused by the task at hand whereas 83.3% of the other group was confident about their accomplishments.
When asked about their feelings on the process 50% of the SecondLife participants were frustrated and another 33.3% were confused. That leaves 16.7% who had neutral feelings. In comparison to the paper manual, 33.3% found it satisfying while the remaining 66.7% had neutral feelings.
Excitement over the experiment was split. 50% in each group found it exciting. In SecondLife 33.3% were neutral and 16.7% found the task dull. However the remaining 50% of the group using the paper manual was neutral.
When asked if they would use this process again the paper manual group was 100% positive they would. In SecondLife 66.7% were willing to try the program again while the others would not. Not surprisingly, since frustration and confusion ran high, there was a 50/50 split in the SecondLife participants who were willing to refer SecondLife to a friend.
While watching the participants using the SecondLife version of the Lego house instructions it became apparent that although it took them longer to complete the house, they had no difficulty building the lower level. As the house progressed past the double doors to the first set of blue windows is where problems seemed to arise. This can probably be attributed to the use of the elevator. In order to clearly see the pieces closer to the top of the house the participant needed to use the elevator so they could look down on the house as it became larger.
Another issue that was noted was due to SecondLife error. At one point one of the two controllers stopped responding appropriately. When the participant clicked on the buttons it would make blocks lower down in the house disappear instead of appear. This resulted in one of the participant building a Lego house with missing pieces.
Besides the SecondLife error the Lego house seemed to be going well at lower levels where fewer pieces were involved. Another thing that may have been problematic was due to the similarity of the shapes and sizes of the Lego pieces in the white section of the house. As the house continues to grow the pieces blend in a little bit. Perhaps if the Lego house had fewer pieces, and the pieces varied more wildly in color, shape, and size the results would have been drastically different.
Overall there was very little testing done on this experiment. Further testing is needed to confirm how well the SecondLife instructions are in comparison to the paper manual, but one thing is certain; participants had fewer problems using the paper instructions and completed the houses much faster. It seem that there may be limits to the complexity of simulations done in computer aided design but that does not rule out the alternative as a whole.
One of the reasons SecondLife participants may have had trouble visualizing the blocks and the house is due to the quality of the graphics. SecondLife restricts the resolution based on the computer’s capabilities and due to limited online storage capacity. 83.3% of the SecondLife participants found the images used in the experiment were fuzzy or moderately fuzzy while everyone in the other group thought the images were fine. Here one may conclude the quality of the images play an important part in instructions.
Further testing will be done using SecondLife and computer aided design in the fall of 2007. In the second phase of this experiment the participants will be asked to put together a couch that requires simple tools (hammer, screwdriver) using the paper instructions and another SecondLife simulation. These results will be compared with further testing of the Lego house’s construction.