The Effects of Service-Based Learning on MetaCognitive Strategies During an Engineering Design Task

Service-Based Learning (SBL) has become an emerging pedagogical tool for engineering education. Although there is a large body of literature reporting the benefits of SBL activities, most studies have relied on self-report measures and generalized learning contributions. Our evaluation went beyond self-perceptions by investigating what effects, if any, SBL activities had on the metacognitive processes of students during an engineering design task. Verbal protocols were collected from ten engineering students during an openended, model-building design task. The five SBL students and five non-SBL students also completed post-task interviews and reflection papers. The students in our sample who had participated in SBL activities voiced more metacognitive phrases, demonstrated more accurate task analysis and clearer strategic planning skills, were more skilled at discriminating useful from superfluous information, and had a better understanding of clients’ needs and constraints. Index Terms engineering design, service-based learning, verbal protocol analysis, metacognition, self-regulated learning INTRODUCTION ABET (formerly the Accreditation Board for Engineering and Technology1) states in their list of course outcomes and requirements that all engineering graduates should understand the impact of engineering in a global and societal context. One movement to address engineering stewardship has been the development of service-based learning courses and programs. Service learning has been defined as an activity that integrates academic subject matter with service to a community that helps foster civic responsibility and strengthens communities 2-6. Although some researchers5 consider academic credit a necessary component of service learning, Jacoby defined service learning simply as experiential-based activities designed to promote learning through engagement in real life opportunities that address the needs of a community. For the purpose of this paper, the term service-based learning (SBL) will be used to encompass the broader perspective which includes non-credit bearing and extracurricular learning experiences.


INTRODUCTION
ABET (formerly the Accreditation Board for Engineering and Technology 1 ) states in their list of course outcomes and requirements that all engineering graduates should understand the impact of engineering in a global and societal context.One movement to address engineering stewardship has been the development of service-based learning courses and programs.
Service learning has been defined as an activity that integrates academic subject matter with service to a community that helps foster civic responsibility and strengthens communities [2][3][4][5][6] .Although some researchers 5 consider academic credit a necessary component of service learning, Jacoby 7 defined service learning simply as experiential-based activities designed to promote learning through engagement in real life opportunities that address the needs of a community.For the purpose of this paper, the term service-based learning (SBL) will be used to encompass the broader perspective which includes non-credit bearing and extracurricular learning experiences.

LITERATURE REVIEW
There have been numerous studies reporting how SBL activities address ABET Criterion 3a-k such as (a) an ability to apply knowledge of mathematics, science, and engineering, (b) an ability to critically analyze and interpret data, (c) an ability to design under constraints (d) an ability to function on multi-disciplinary teams (e) an ability to identify, formulate, and solve engineering problems, (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively, and (j) gain knowledge of contemporary issues.These studies have reported that students who participated in SBL activities have:  a lower transfer/dropout rate in engineering disciplines compared to students not so involved. a greater ability to work more cooperatively in teams. advanced leadership skills and communication skills. a more concrete understanding of basic engineering design principles. the ability to be innovative and resourceful when confronted with illstructured problems, unforeseen challenges, and conflicting information. the ability to interact with a broader range of socioeconomic classes. enhanced critical thinking and project planning skills. increased self-efficacy and confidence. a new passion and motivation for vocations within engineering.
 increased awareness of their potential to make an impact. a better understanding of how problems are socially embedded, reducing stereotyping and prejudicial perspectives. a higher percentage of women participants in relation to the percentage of women in engineering overall 2,10,12,[19][20][21][22][23][24][25][26] .
The positive effects of SBL activities have been attributed to processing the experience through reflection, discussions, and interactions with other students and faculty.The opportunities to clarify, elaborate, evaluate, justify, and consider alternatives through dialogue with others were powerful components of SBL activities 8,10,12,24 .

THEORETICAL FRAMEWORK
The benefits gained through SBL have been primarily attributed to the processing of SBL experiences through reflection.Based on this finding, we chose to look at the design process of engineering students through the lens of metacognitive and self-regulated learning theories.The construct of metacognition was recognized by Flavell over thirty years ago and is defined as the knowledge and awareness of one's own thinking [27][28][29] .Metacognition is our ability to reflect on, monitor, strategize, and evaluate our learning and the information known to us.Metacognitive knowledge includes knowing general strategies for learning and problem solving, knowing what knowledge one already has, and knowing the conditions necessary for learning new information.
Self-regulated learning is the process involved in the monitoring and regulation of knowledge building.It involves setting goals, adopting strategies for attaining those goals, monitoring progress, reflecting on and evaluating methods, and restructuring as necessary.
Students with effective metacognitive skills can accurately estimate their knowledge and regulate their thoughts and behaviors in order to set strategies, achieve goals, and develop effective plans for new learning.They are aware of the effectiveness of their strategies and can adapt accordingly.Without effective regulation, students may not use the knowledge they have, or properly apply it to the task at hand 32 .Academically able students have higher metacognitive abilities than those less able 33 .Although conceptual and procedural knowledge are important, effective metacognitive strategies play a crucial role in how well learners learn and are needed to support more advanced conceptual knowledge 34 .
Self-regulatory processes consist of three cyclical phases: forethought, performance, and self-reflection.The forethought phase involves task analysis, strategic planning, and goal setting.Specific and proximal goals enhance outcomes better than general and distal goals 35 .More naïve self-regulators who tend to set nonspecific, distal goals are unsure as to what to do next, must wait longer for corrective feedback, and are more easily discouraged by seemingly slower progress 36,37 .
The performance phase refers to the deployment of specific strategies as well as selfobservation.The self-reflection phase includes self-evaluation, and comparing performance with goals 35,36 .Zimmerman 36 considered self-reflective practice, "perhaps the most important performance control process that distinguishes skillful from naïve self-regulators" (p.8).Selfreflection involves keeping track of one's effectiveness as one performs and using the information to alter performance.Naïve self-regulators fail to monitor their performance or monitor haphazardly.This often leads to a reliance on guessing due to incomplete or fragmentary information or misinterpreting the information they do have.We investigated some of these cognitive processes of engineering students during a hands-on, model building design task.

RESEARCH METHOD
A simple design task was created to explore possible differences in cognitive processes between SBL and non-SBL students during an engineering design task.Verbal Protocol Analysis (VPA) was used to collect data as it is considered the most appropriate method to study the cognitive abilities and processes of designers [38][39][40][41] .VPA has been used extensively since the 1970's to study the cognitive processes of engineering students 38,[42][43][44][45] as well as experienced designers 39,[46][47][48] .Although VPA is the most suitable method for documenting cognitive processes, it is not an assessment tool appropriate for large subject populations because of the copious amount of time required for analysis.
During VPA, our subjects were asked to continually think aloud while performing a design task 49,50 .Subjects were asked to report the contents of their working memory, so that their verbalized thought processes could be captured on videotape and later transcribed.The texts were then coded and analyzed, which provided our main source of data.These verbal reports offered insights into how subjects generated and transformed information about the problem and how they went about developing a solution.
After completing the design task, additional data was collected via individual open-ended interviews in order to expand on the verbal protocol data and to better understand participants' cognitive processes.Interviews are considered a mode of systematic inquiry and one of the most powerful ways of obtaining rich, experiential accounts of events from others 51,52  interview, participants were asked to write a short reflection paper of their design project, focusing on their design experience and how they came up with their prototype design.

Participants
Engineering students attending a private university in the northeastern United States were solicited to participate in an engineering design task.Specific recruitment of SBL students was implemented through the relationship between the authors and campus programs.The second author was director of the STOMP program at this institution and the third author coordinates and supervises EWB projects.The final sample of ten students consisted of six males and four females from different academic years and diverse engineering disciplines.One female and four males had been involved in SBL activities.
Students were coded according to gender (M = male, F = female), engineering discipline (e.g., ME = mechanical engineering, ChE =chemical engineering), undergraduate class level (2 = sophomore, 3 = junior, 4 = senior), and s for those with SBL experiences.Since there were two students identified as M-ME-4-s, the student with five semesters of SBL experience was designated M-ME-4-s-5 (See Table I).
In addition to being at different stages in their educational careers, participants had varying levels of design experience.Design courses for students in any of the engineering programs at this institution are not required until the senior year.However, some students may have taken elective courses, summer internships, or participated in extracurricular activities where they may have been exposed to the design process.While the range of previous design experience within this student sample varied, those with SBL experiences did not, as a group, have more design experience than the non SBL students.

Procedure
The study participants were solicited directly via e-mail.Each student who consented to participate was tested individually in a small conference room on campus.A small audio-video camera was mounted on the ceiling to record speech as well as students' hand movements.At the beginning of the activity, participants were told that the purpose of the study was to investigate problem-solving strategies during a design task.Participants were instructed to constantly verbalize what they were thinking as they proceeded through the design task.A practice thinkaloud project of assembling a 24-piece puzzle was given to the students to familiarize them with the process.When the subjects finished the puzzle, they were given an information sheet that explained the design task: To design and build a jar opener for individuals with the use of only one hand.A more detailed description of the procedure can be found in a previous publication 53 .
These activities were presented to the participants using physical sets of cards laid out on a large table.Each activity had five to eight cards that offered various pieces of information.The instruction sheet explained that participants could choose whichever activities they thought might help them in formulating a solution.All activities contained cards with superfluous information to see if participants could filter out the unnecessary information.
When subjects chose the Build a Prototype activity, they were handed a kit of LEGO ® pieces along with a twist top jar to use as a test jar for their prototype.The participants were instructed to use the LEGO ® pieces to build their prototype and to not be overly concerned with any usability challenges arising from the materials.While the functionality of the pieces did not allow for heavy force to be used in opening a jar, the fact that the pieces could be assembled, taken apart, and reassembled quickly and easily outweighed this disadvantage.In addition, due to their long-standing popularity, most students are at least somewhat familiar with LEGO ® pieces.
While the participants performed the design task, two researchers (the first two authors) took observational notes.If participants became silent, they were prompted by one of the researchers to continue thinking aloud.The individual sessions lasted between 32 to 80 minutes with most of the participants taking approximately an hour to complete the task.When finished with the design task, each student was interviewed for approximately 15 minutes to gain further insights regarding their problem solving process.Students were also asked to write a short reflection paper about their design process during this task.All of the participants wrote one to two short paragraphs.
The recorded sessions and interviews were transcribed and initially coded by the first author.Codes were later checked by the second author and reviewed by two outside coders not directly involved with this research project.Each participant was sent their coded transcript and asked to comment on the accuracy of our translation and codes.This member checking was included as an element of triangulation to strengthen confirmation of our interpretations 54 .While some of the students made a few clarifying comments, most concurred with how their thoughts and words were interpreted.

DATA ANALYSIS
The transcribed texts of the verbal protocols and interviews formed the main data for analysis.Comments from the reflection papers were incorporated and used as supporting data.Content analysis 54,55 of these verbal protocols was employed using HyperResearch TM (by ResearchWare), a qualitative data software package that helps organize and keep track of large amounts of verbal data.This form of analysis involves manually searching through text for recurring words, phrases, or themes.Emergent patterns and themes defined the categories or codes.Some phrases were coded for multiple categories.Once categories were created, constant comparison was employed to ensure internal homogeneity and external heterogeneity of the categories 54 .Internal homogeneity concerns cohesiveness, or the extent to which each entry within a category was similar to every other entry in that category in a meaningful way.External heterogeneity concerns independence, or the extent to which differences between categories are distinct and clear.To ensure objectivity, numerous sessions and lengthy discussions with two outside coders were conducted to achieve 100% inter-rater consensus.Eleven categories derived from our data (See Table II).Categories that were strong enough to create for some individuals but did not fit every participant.For example, some participants had numerous phrases regarding testing the model.But the category of "Testing Model" was not strong enough across participants to warrant keeping it as an independent category.In addition to the "Test Model" category, the other categories were: Global Perspective, Kinesthetics, Opportunistic Behavior, and Personal Experience.

RESULTS
All of the categorized phrases were totaled (Total # Phrase) in order to calculate the percentage of phrases uttered in each particular category (See Table 3).As can be seen in Table III, most of the students had roughly the same number of phrases in Idea Generation, Evaluation, Describing Information, and Clarifying Information.The average time between the groups to complete the task differed by only five minutes and the difference in the average number of phrases between the groups reflected this slight time difference.The categories in which there are larger observed differences between the two groups will be discussed in the subsequent sections: Metacognitive Strategies, Confusion, and Discriminates Information.Percentages were rounded to the nearest tenth.

Meta-Cognition
Phrases that were coded for metacognitive strategies concerned participants' awareness of their knowledge or their knowledge building strategies.Examples of meta-cognitive phrases are, "So this is based on the stuff I want to know about; what his limitations are" and "Technical description of prototypes.Oh, that might be helpful, that might explain some things." As can be seen from Table III, most of the SBL students in this sample had more phrases related to metacognitive strategies (M = 15%) compared to non-SBL students (M = 8.8%).Researchers view self-regulatory processes in terms of three cyclical phases, the first being the forethought phase.In addition to goal setting, the forethought phase involves strategic planning (SP) and task analysis (TA).
Comparing the forethought phase during the design process of two students, one SBL and the other non-SBL, it appears that the SBL sophomore demonstrated more sophisticated task This student proceeded to deploy this last strategy by reading about the clients, Jim and Mary, and stating that he wanted to learn about them to better understand their disabilities.This student researched information which he thought was relevant and reached his interpretation of the problem, or task analysis within a half hour: [24:25] I want something for the jar to be able to, be in . . . to put the jar into something . . .that would hold it.(TA) [29:15] So the thing is about holding it still while turning.(TA) M-GE-4 likewise incorporated strategic planning steps, but not to the extent of the first example.He did not explain why he picked the choices he did, but rapidly read through them.As stated previously, naïve self-regulators fail to monitor their performance or monitor haphazardly, and often rely on fragmentary information.M-GE-4 had an incorrect task analysis, or interpretation of the problem (sealing the jar as opposed to opening it) due to incomplete information.M-ChE-2-s wanted to learn about the clients first in order to plan his prototype.M-GE-4 designed his prototype first, and as an afterthought said: Fifty minutes into the project, this student had yet to formulate a concept for a prototype because she was unable to clearly analyze the task or articulate a proper interpretation of the problem.Poor planning strategies and task analysis adversely affected this student's problem solving abilities for this task Confusion A striking difference between the two groups concerned the category of Confusion.While four out of five of the non-SBL students had phrases in this category, none of the SBL students did.Most of the comments in this category pertained to the clients' disabilities.The non-SBL participants had trouble comprehending, remembering, and addressing client limitations due to various physical challenges.For example, the information card clearly stated that there were upper-limb amputees who had trouble opening jars with one hand.The activity cards for the amputee client Jim also explicitly stated that he does NOT have a prosthetic arm.Yet some students still were not clear on this point and questioned what Jim was physically capable of.Naive self-regulators often do not use or recognize the knowledge they have, improperly apply it to the task at hand, or misinterpret the information entirely.Reading the "Talk to Jim" card, F-CE-4 said: One of the researchers explained that Jim was an amputee and had only one arm.A minute later this student again asks: [14:28] So does Jim have the other arm, or like a mechanical one?
M-ME-3 similarly did not fully understand the information presented in the client cards and interpreted the information as: [12:48] From what I gather from both of those (clients) is that they have weak limbs.
M-GE-4 also seemed to be confused about the definition of an upper-limb amputee: [04:41] The major difficulties that they would probably have doing is coordinating two hands or using two hands if you're an amputee.
While building his model, this student commented: [54:27] I'm making it specifically for them.They were lefties now, right?
The non-SBL students in this sample had difficulty understanding obvious limitations of the clients which were important for this particular design task.While the SBL students in this sample understood these limitations, many of the non-SBL students appeared unable to grasp the significance of these client characteristics.

Discriminates Information
With limited time, limited cognitive resources, and an abundance of information readily available, learners need to employ strategies to recognize extraneous information.Irrelevant information can decrease student performance 56 .Naive self-regulators often cannot tease out important from nonessential information, do not use the knowledge they have, improperly apply it to the task at hand, or misinterpret the information entirely.
The information cards included material that was totally irrelevant to the task as a mechanism to determine if students had the ability to discriminate useful from superfluous information.Examples of phrases discriminating information are, "Couldn't do math well.That doesn't really pertain to this" and "She didn't think she was having a stroke because she was young.Not all that relevant to what I need to design." As can be seen from Table 2, students in this sample who had SBL experiences were better able to discriminate useful from irrelevant information (M = 6.6%) compared to non-SBL students (M = 1.4%).There were many unnecessary pieces of information that some of the non-SBL students did not recognize as such.For example, one of the cards mentioned tension and how it was unwise to tie up a boat in a storm with a steel chain.F-CE-4 commented: [28:25]  guess, in the lever arm, I'd like to make sure that the lever doesn't snap off.This student was trying to utilize an irrelevant piece of information and was unable to recognize that it had no practical use for the design task at hand.

Reflection
There were observed differences between the two groups of students in the categories of Metacognition, Confusion, and Discriminates information.These differences emerged from the verbal protocols during the design task.Differences between the groups also emerged from the students' reflection papers.
The third cyclic phase of the self-regulatory processes is a self-reflection phase, which includes self-evaluation and comparing performance with goals.In general, the non-SBL students tended to focus on the outcome or the end product while the SBL students tended to write about themselves as cognizing agents.For example, M-GE-4 wrote: I originally wanted to have two wrenches . . .when I realized that with the jar locked to the ground . . .any amputee or stroke survivor would be able to open the jar.

F-ME-3 wrote:
I decided to use the rack and pinion technique to translate the rotational motion from the knob of the user interface into a linear motion, clamping the jar in place . . .I basically just built . . .a box that held the gears in place, a bearing to keep the arm from deflecting when the jar is turned, etc. . . .By placing a brick over the "bearing," it does a good job of holding it in place.
The SBL students tended to review their actual design process and commented more on their learning.For example, F-EnE-4-s wrote: I liked having the different cards to reference to give me an idea of what considerations to make in my design and who exactly I was designing for.I think that the only cards I didn't look at were the ones about mechanics and basic physics because I felt that I already had enough understanding of those concepts . . .It was a little weird to talk aloud, just because I'm not used to doing that, however . . . it also probably helped me focus on the task since I was constantly reminding myself of the design goals.
M-EE-3-s similarly reflected on himself as a learner, and his design process: It was a good experience to talk through an engineering design process . . .This forced me to be more introspective about my thoughts.
While the non-SBL students had a more objective perspective in their reflection papers by focusing on the final product, those with SBL experiences tended to be more introspective, regarding themselves as learners who were more aware of their metacognitive strategies.DISCUSSION These results support previous research which found that students who participated in a problembased curriculum used more adaptive self-regulation strategies than students who participated in a more traditional curriculum 57 .Our sample of students who had been involved with SBL activities demonstrated more accurate task analysis/problem interpretation skills, better strategic planning skills, a clearer understanding of client limitations, and was better able to discriminate useful from irrelevant information, than our sample of students who had not had such experiences.
The first observed difference between the two groups was in the category of Metacognition.The SBL students with stronger metacognitive skills were more accurate in task analysis and more definitive in strategic planning.Solving well-defined problems in the classroom does not readily transfer to solving complex, ill-defined problems in the workplace, as the former requires cognitive and metacognitive processes that are insufficient for solving the latter 18 .During SBL field experiences, students are confronted with ill-structured, more complex problems, with multiple solutions and unexpected constraints and challenges 58 .These field experiences encourage students to analyze the task more intensely, to plan with a deeper understanding of the problem, and to dialogue and interact with other students, faculty, and clients.The intense interaction required to solve the real-world challenges they were confronted with enhanced their strategic planning and task analysis skills, which they demonstrated during this model-building engineering design task.
The second salient point that emerged from our data was the confusion of non-SBL students regarding the explicit characteristics of the client disabilities.As mentioned earlier, clients are central to the process and equal participants in a collaborative project.Whether working with inner-city elementary school children or building a water filtration system in a third world country, the SBL students had the experience of closely interacting with clients and building a meaningful relationship with them 59 .Those with international experiences often lived with the people they worked with, so students had an obligation to understand authentic client requirements and limitations.Their SBL experiences helped develop strong interpersonal skills in order to understand client needs, characteristics, and limitations.Students without SBL experiences were not as likely to have had the same in-depth client interactions and were therefore, not as aware of the importance of clients' needs and limitations in the design process.
The third major difference between the groups appeared in the category of Discriminating Information.Overall, the non-SBL students appeared less efficient at monitoring and applying information.Some tried to apply useless information to the task or were unable to recognize significant information.Real-world experiences offer learning challenges that are not readily confronted in the classroom.Traveling to a third world country has additional challenges of travel planning and coordination, fundraising issues, working collaboratively on multidisciplinary and multi-cultural teams, and arranging for material delivery across international borders.These make for unfamiliar situations that students are not confronted with in the classroom.In unfamiliar situations, learners need to reflect on discrepancies between what they think is true and new information that may not support that view 33 .Individuals must learn to filter out extraneous information and focus on the core information needed for the task at hand.This sample of students who had SBL experiences was better able to discriminate useful from irrelevant information.

[03: 28 ]
Review the first sources of physics, I would not.(SP) [03:30] View the aesthetics, No. Plan, draw, sketch.No. (SP) [03:42] Alright, first thing I'd probably do is, I would look at the jars that are currently here.(SP) This student rapidly arrived at his interpretation of the problem: [06:27] I'm not really worried about the bottom of the jar . . .I'm more worried about the top of the jar and getting it off.(TA) [09:00] You have to find a way to seal it down with perhaps like an o-ring or something that would . . .provide a seal 'cause jars are pretty much used for your liquid containment.(TA) . After the

TABLE III PERCENTAGES
OF CATEGORIES BY PARTICIPATION (SBL/NON-SBL) So talk to Jim . . .I can get more detail about the disabilities that these people have and the best ways that they think they can solve the problem.(SP) [01:37] Build a prototype.(That's) going to be a little later on, once I figure out the direction I wanna go in.(SP) [01:48] Talk with Mary . . .I'm gonna have to do at the start.(SP) [02:37] So first I wanna talk with Jim . . .figure out what problems they have with opening jars.(SP) I'd probably want to talk to Jim and Mary . . .just to ensure that, that (using his prototype) is something that they can do.After reading about the clients, this student realized his design would not work, saying, [21:49] The problem with this prototype is that it requires redesigning jars in general . . .you could talk to jar manufacturers of jars, see if you could get them to change their design.This student demonstrated naïve task analysis and strategic planning skills, resulting in two incorrect interpretations of the problem (sealing the jar and then redesigning jars).He relied on fragmentary information, and learned about the clients only after designing his prototype.Without appropriate task analysis and strategic planning, learners are unsure as to what to do next and can be more easily discouraged.This was demonstrated by F-CE-4.Although she had adequate skills in evaluating other prototypes, she had poor task analysis and planning strategies.At several junctures, she was perplexed in deciding her next steps: [35:35] I don't know.I don't know if I'd talk to jar manufacturers.I don't know.I feel like I might talk to a jar manufacturer.I wouldn't know what to get out of it though.[39:26] Or a ratchet?I wouldn't know how to build that though.[48:21] I don't know, I guess like the whole gear thing.I can't just really picture in my head how it would all stick together.[50:06] There's just a lot of wheels.So I'm wondering if I could use wheels somehow.
I don't know.
we won't know if his defects are physical or from his brain . . .Ok.So he has both arms?
Don't use a steel chain!Um, so, hm, hm.I'll figure out what that means . . .I just don't know how I would use tension in a twisting application.So I