A Mixed-Methods Study of Cognitive and Affective Learning During a Sophomore Design Problem-based Service Learning Experience

Authentic, real-world problem solving is an integral part of the engineering profession. Yet, research suggests that engineering education is primarily focused on welldefined and well-structured problems, which do not provide students the real-world relevance, context, or experience in solving the types of problems required as a professional engineer. The addition of problem-based service learning (PBSL) to engineering curricula provides an opportunity to introduce students to a variety of real-world projects in a community-based context. Numerous studies have shown the importance and impacts of integrating service learning and problem-based learning into engineering education. Herein, the results of a mixed-methods, longitudinal study on cognitive and affective learning during a PBSL sophomore design experience are presented and discussed. The goals are to demonstrate how both qualitative and quantitative data can be used to measure student learning during a PBSL experience and to provide a framework for assessment of such


INTRODUCTION
A Bachelor of Science in Engineering is a professional degree, and as such, engineering students, will find themselves working, first as Engineers in Training and eventually as Professional Engineers, in a world of complex decisions where tradeoffs have social and ethical implications.As educators, it is important that we train budding engineers to function professionally and ethically upon entering the workforce.Service learning experiences integrated into engineering curricula can provide a platform for students to begin their transition from a student to a professional.Bringle and Hatcher define service learning as, "a credit-bearing educational experience in which students participate in an organized service activity that meets identified community needs and reflect on the service activity in such a way as to gain further understanding of course content, a broader appreciation of the discipline, and an enhanced sense of civic responsibility." 1 Through a service learning opportunity, students are able to apply engineering knowledge to reinforce learning, gain appreciation for the discipline, develop and improve teamwork and leadership skills, make community connections and learn from community partners, as well as develop a sense of civic responsibility. 1,2ll students in the James Madison University (JMU) Engineering program complete a yearlong, problem-based service learning (PBSL) design project during their sophomore year.JMU is a public comprehensive university located in Harrisonburg, VA, USA and is committed to providing students' a total learning experience with a foundation in the liberal arts.The JMU Engineering Department, which admitted its first class in 2008, was founded on the recognition that engineers are no longer constrained to disciplinary boundaries, and instead, must work across disciplines as members of global communities and multidisciplinary teams. 3,4The program offers a single undergraduate engineering degree that focuses on sustainable design and systems thinking.Much of the instruction in the program is based on a problem-based learning framework. 4,5This problem-based learning (PBL) framework has been integrated with a service learning design project during the sophomore year to create a cornerstone design experience which acts as a launch-pad into the two-year capstone experience.
In this paper, the results of a mixed-methods, longitudinal study on cognitive and affective learning during the PBSL sophomore design experience are presented and discussed.The goals are to demonstrate how both qualitative and quantitative data can be used to measure and understand student learning during a PBSL experience, and to provide a framework for assessment of PBSL experiences.The following two research questions ground this study: 1. What are engineering students' cognitive and affective learning outcomes as a result of participating in a two-semester PBSL sophomore design project?
2. What are engineering students' cognitive and affective learning outcome gains as a result of participating in a two-semester PBSL sophomore design project?
Although assessments of student learning during PBSL experiences have been investigated in previous studies, in answering the two research questions, the additive contributions of this study to the current body of literature are three-fold: (1) to showcase the use of both quantitative and qualitative methods in the evaluation of student learning in the context of PBSL experiences, (2) to highlight the learning growth of students longitudinally across a PBSL course sequence

SERVICE LEARNING IN ENGINEERING EDUCATION
Over the past several years, models of Service Learning (SL) or Learning through Service (LTS) have proliferated through engineering curricula as a means to improve student learning of both technical and non-technical (or professional) skills.LTS activities include both service-learning (usually projects) and community service components of courses or programs.LTS experiences are implemented in individual departments, particularly civil and environmental engineering, 6,7,8,9,10,11,12 or across multiple disciplines in engineering colleges. 13,14,15They are incorporated into required courses, 16,17 elective courses, 18 or a sequence of courses at a variety of levels ranging from first year introductory courses to capstone design courses. 19,20,21LTS is also addressed at some universities through co-curricular or student-led activities like Engineers Without Borders or similar organizations. 18,22There are also unique experiences where students serve as mentors for high school teams completing a design task, 23 voluntarily elect to work on a team with both engineering and non-engineering students to solve community problems through technology, 24 or identify and carry out self-directed projects with community partners to develop an appreciation for civic engagement. 25n addition to SL courses focused on engineering science or elective courses, there are a variety of models for incorporating SL into engineering design courses at all course levels. 26,27he most common models are for introductory engineering courses that involve design projects or for capstone design courses.These models, whether at the first-year or capstone level, are generally intended to give students the experience of working with a client or external stakeholders and to develop or apply both technical engineering knowledge and professional skills like teamwork and communication.In addition, there are at least two examples of sophomore-level design courses focused on a client-based product development project. 28,29espite encountering challenges related to course design, increased workload, and lack of support from their institutions, faculty at several universities continue to persist in integrating LTS and SL experiences because they are passionate about service learning and want to improve the quality of student learning. 30

Breadth and Depth of Student Learning through Service
Engineering faculty both desire to improve the quality of student learning and can demonstrate gains in students' achievement of learning outcomes and confidence in their abilities.Most assessments of student learning in SL activities focus, at least in part, on achievement of learning outcomes.Swan, Paterson, and Bielefeldt classified student outcomes from service learning as knowledge (conceptual and procedural), psychomotor skills like writing and presenting, attitudes toward learning and identity, student recruitment and retention, and long-term knowledge retention and application. 31International SL projects often assess global competence in addition to the aforementioned objectives.Assessment of learning outcomes for SL fall into three methodological categories: (1) quantitative, using surveys mostly focused on student satisfaction or perceptions of achieving learning objectives, (2) qualitative, concerned with reflection on gains in affective skills or assessment of journaling exercises, and (3) mixed-methods, involving both outcome-related surveys and interviews or focus groups.Mixed-methods tend to be the least common of methodological approaches, yet offer a great wealth of insight into student learning.Most SL assessment studies have shown that students achieve equal mastery of technical skills compared to traditional course approaches, 32,33,34 but the real advantage of SL activities is gains in non-technical skill development and confidence in abilities.In particular, Bielefeldt and colleagues reviewed several previous studies and found that project-based service learning is equally as effective or more so on teaching students' knowledge and skills with greater gains due to the social and moral development that comes from working on SL projects. 35s stated above, quantitative studies rely on surveys of student outcomes to show achievement of learning outcomes, both technical and non-technical.For example, in comparing senior structural engineering capstone students who either completed an international SL project or a traditional project, a Likert-based survey of eight technical and eight non-technical objectives was administered to the students; students reported that technical objectives were well satisfied by all students with mean responses above 4.00 for all 8 objectives.But students who completed the SL project reported statistically significantly higher achievement of non-technical objectives than those who did not.Overall, non-technical skills were rated lower than technical skills, but still adequately accomplished.In addition, achievement of non-technical objectives increased year-to-year for both groups. 36In another study, two surveys were used to investigate both motivation and learning outcomes of 214 students in an introductory engineering course.The learning outcomes were defined by the Accreditation Board for Engineering and Technology (ABET), a non-profit organization that accredits over 3,100 programs at colleges and universities in 24 countries. 37Sixty-nine students completed SL projects while 145 students completed traditional projects. 38The survey results showed that SL projects were significantly more effective in positively influencing students' interests, recognition of relevance, and satisfaction in learning and their self-assessed engineering abilities in three out of eleven ABET learning outcomes.In fact, ratings for the SL-cohort were higher for all other outcomes (except ability to apply knowledge of mathematics, science, and engineering to solve engineering problems), though not statistically significant. 39A significant limitation of most outcome related surveys is that they generally rely on student self-assessments that are not directly correlated to course performance through other means.
Qualitative studies allow exploration of learning theories like meta-cognition and selfregulated learning and differences within cohorts.Creswell informs us that "qualitative research is conducted because a problem or issue needs to be explored, especially in the case of a complex situation where detailed understanding can be established by talking directly with people". 40or example, Lemons et al. used Verbal Protocol Analysis (VPA) to collect data from ten engineering students during an open-ended, model-building design task to evaluate what effects SL had on meta-cognitive processes beyond self-perceptions.The VPA data was supplemented with post-task interviews and reflection papers.The study showed that students who had engaged in SL activities expressed more metacognitive phrases and demonstrated better performance of design tasks like understanding clients' needs and constraints. 41However, it is important to note that purely qualitative studies 42,43 typically have small sample sizes; and consequently, results may be transferable, but not necessarily generalizable.
Although less common, mixed-methods approaches have been used to address shortfalls of purely quantitative or purely qualitative studies.Such an approach can produce generalizable results and reinforcement but also allow in-depth exploration of unique benefits of service learning in the metacognitive or affective domains.For example, a study of a first year biological engineering design course that incorporated a SL project showed with quantitative (survey) and International Journal for Service Learning in Engineering Special Edition, pp.1-28, Fall 2013 ISSN 1555-9033 5 qualitative (focus group) data that students demonstrated the ability to design a system, component, or process to meet desired needs and could communicate effectively. 44By using a mixed-methods approach, Nesbit, Sianchuk, Aleksejuniene, and Kindiak were able to identify trends in second-year students beliefs about the work performed by civil engineers and the impact of civil engineering knowledge and explore the themes of "Civil Engineering and People", "Different Perspectives", and "Sharing Expert Knowledge." 29,45ssessment data also reveal that female students seek out service learning experiences 46 and experience more significant gains than male students. 47,48These studies suggest that servicelearning course experiences could be valuable tools for attraction and retention of diverse undergraduate students in engineering.

BACKGROUND AND CONTEXT
The sophomore course sequence, Engineering Design I and II, is the cornerstone of the JMU design sequence curriculum. 28,49,50,51The objective of the course sequence is to not only teach students the design process, but also to drive students toward ownership of the engineering design process as well as provide the base knowledge to begin their capstone projects.To achieve this objective, a year-long, client-based, design project is woven into engineering design theory instruction.Throughout the year-long project, students interact with an actual client to design an actual product.The project for the last four years has been a custom, pedaled vehicle for a client with cerebral palsy. 28,49,50,51Having a client and working on a tangible, real-world problem helps to motivate the students-including those who tend to not be mastery driven-to learn the design process and complete the project with a working prototype.

Sophomore Design Experience Structure
Course instruction focuses on theory and assignments challenge students to apply the theory to their client-based course project.Instruction occurs in a single 100-minute block each week allowing for influential activities, discussions, and demonstrations to take place during class as well as providing team members a dedicated time to meet together, which is very beneficial during prototyping of the project.The sophomore design sequence combines directed and nondirected, group-based and independent, structured and unstructured, problem-based learning experiences that incrementally expose and reiterate the engineering design process.Course content topics covered during the data collection year are presented in Table I.The course is structured across two semesters and coverage of the design process begins during the first semester as a directed learning experience where the students are incrementally walked through the early phases of the engineering design process.In Engineering Design I, students are taught how to apply the tools used during the planning and concept development phases of design.The student teams complete the first semester with a report and in-class presentation detailing their design process, design tool usage and final selected concept(s).
In Engineering Design II, students receive instruction in the later phases of the engineering design process (system-level, detailed design, and testing and refinement) as well as join new teams.Each new team revisits the work from the prior semester, iterates upon the designs, and chooses a final human-powered vehicle design to build.During the second semester, students are taught about prototyping, both analytical and physical, and how they inform the system-level and detailed design phases.With this new knowledge, the student teams create physical prototypes of two sub-systems.Often these prototypes reveal hidden assumptions or lead to further investigation of how the mechanism should work.Additionally students are taught to investigate the vehicle sub-systems integration and to convert their hand sketches into engineering drawings.The engineering drawings as well as physical prototypes help the students to understand the limitations of their designs, such as geometric layout and sub-system interaction, which inform their design iterations.Student teams build an alpha prototype of their final vehicle design, and work with a local field expert for guidance while in the machine shop.Throughout the project, students actively engage with the client to solicit feedback.The project culminates with each team giving an in-class presentation with demonstration of their final product (a beta prototype), and a public beta prototype showcase to the client, University and local community.

About the James Madison University Sophomore Design Project
The sophomore design course sequence project has required students to design and construct a human-powered vehicle for a client with cerebral palsy. 28Each year, however, we have changed the client, which has resulted in different customer needs and target specifications related to each International Journal for Service Learning in Engineering Special Edition, pp.1-28, Fall 2013 ISSN 1555-9033 of the different client's manifestations of cerebral palsy.Overall, this has resulted in very different final concepts each year.The client-based project challenges students to work with an actual client under real-world circumstances.To be successful, students must (at a minimum) interface with faculty from both kinesiology and engineering, members of the local community, the client and the client's family.
The overarching goal of the course sequence and project is to provide a learning experience that integrates skills from the humanities, engineering science, engineering design, and fabrication, but also places student's engineering education in context-provides the students with a framework, or cornerstone, for their education.Furthermore, the project also exposes students to the iterative nature of the design process, not just within a single phase of design, but also within the process as a whole.

About the James Madison University Adaptive PBL Model
The sophomore course and project are grounded in a problem-based learning (PBL) pedagogical model, where a student-centered approach to learning is achieved through a practice-based problem solving approach.Having historical foundations in medical education, 52,53 PBL is a powerful student-centered pedagogy that offers a strong framework upon which to build a curriculum that will allow engineering students to learn essential, real-world, problem solving skills.Moreover, a large body of literature highlights the successes of PBL in many domains and in support of many different student learning outcomes (e.g., problem solving, critical thinking, motivation, knowledge retention), and showcases PBL as a pedagogical vision rooted in practical experiences. 54,55,56,57,58Although PBL assignments can take on a variety of forms, 55,59,60,61,62 prior research indicates that they should be open-ended with a moderate degree of structuredness, contextualized in real-world workplace settings, complex enough to be challenging and engaging to students' interests, adapted to students' cognitive development and prior knowledge, and amenable to problem examination from multiple perspectives. 61he JMU Engineering program, supported by a National Science Foundation award, has developed and adopted an adaptive PBL framework that promotes diverse cognitive experiences by acknowledging that PBL experiences can take many forms and shapes. 45,49More specifically, PBL experiences can be designed with three dimensions in mind: complexity, structuredness, and group structure.Complexity pertains to the cognitive load of the problem and encompasses the intricacy of the solution path, and the breadth and depth of domain knowledge needed to solve the problem. 4,61Structuredness pertains to how well a problem is defined or identified, the transparency of the number of unknowns in the problem, and how well the problem solving process is structured in terms of the methods and analysis used. 4,61Group structure pertains to the level of collaboration required in a PBL experience.Although at a high level, the JMU sophomore PBSL design project can be classified as a fairly structured, complex, and team-based PBSL experience, individual components or tasks of the project vary in complexity, structuredness, and group structure.In some cases, PBL tasks are completed individually, in other cases in a group of varying sizes.Through exposure to a variety of PBL environments, students experience different modes of thinking, learning, and problem solving; resulting in adaptive expertise and increasing cognitive flexibility as depicted in the PBL classification framework in Figure I.

METHODS
This study is a longitudinal study of students' perceived learning outcomes and learning gains as a result of participating in the sophomore design PBSL experience.In this section, the mixedmethods research design, data collection, data analysis, and participant demographics are detailed.

Participants
The sample consisted of engineering students who were longitudinally tracked over two academic years within three time periods in the following sequence:  Time period one (pre-test) -80 freshman engineering students were administered the National Engineering Students' Learning Outcomes Survey (NESLOS) during the start of their freshman year. Time period two (post-test 1) -72 students were administered NESLOS at the end of the fall semester of their sophomore year half-way through the sophomore design course sequence. Time period three (post-test 2) -45 students were administered NESLOS at the end of the spring semester of their sophomore year at the completion of the sophomore design course sequence.
The response rates during the three time periods were respectively 70%, 95%, and 90%.The reduction in sample size across the three time periods was primarily due to attrition, but also due to response rates.For the quantitative section of the study, analyses were conducted over the students who were present during the three time points.Although this sample selection reduces considerably the sample size (N = 34), it fits better with the interest of the quantitative piece of the study: measuring learning gains over time.In this sample of 34 students, the majority were Caucasian (92%) and male (77%).No financial incentive was offered to students for participation.In answering the two research questions, a mixed-methods approach was utilized."A mixed methods study involves the collection or analysis of both quantitative and/or qualitative data in a single study in which the data are collected concurrently or sequentially, are given a priority, and involve the integration at one or more stages in the process of research." 63More specifically, a concurrent (qualitative and quantitative data collection and analysis occurring simultaneously) triangulation design was chosen for this study as shown in Figure 2. In a concurrent triangulation design, separate analysis is performed for both the quantitative and qualitative data with a clear indication of the data integration phase at the end.Although the timing of this design is labeled concurrent, the data were actually analyzed in a sequential manner with quantitative analysis followed by qualitative analysis.Both analyses were completed in immediate succession.

Quantitative Data Collection and Analysis
Data collection involved the use of the NESLOS, which has been used in a variety of engineering learning contexts, including service learning, 64 undergraduate research, 65 and design experiences. 50,49The NESLOS was developed by an interdisciplinary team of engineering educators and assessment professionals and is based on the following three key elements: (1) ABET criteria "a-k" and national reports such as Educating the Engineer of 2020, 66,67 (2) current and emerging research on engineering education and learning outcomes, and (3) extensive social science and engineering education research literature. 68The NESLOS consists of twelve subscales, five of which assess cognitive learning outcomes (Problem Identification, Engineering Skills, Analytical and Evaluation Skills, Experimentation Skills, Project Management) and seven of which assess affective learning outcomes (Big Picture Skills, Ethical and Societal Awareness, Communication Skills, Team Skills, Professional Development, Personal Development, Lifelong Learning).
Cronbach's alpha internal consistencies were calculated for each construct. 69Average internal consistencies as shown in Table III ranged from 0.77 to 0.91.Meeting the desired criteria necessary for research purposes, 70 the majority of the constructs displayed a good (.8 ≤ α < .9)or excellent (α ≥ .9)internal consistency, indicating good reliability.During the pre-test only, the construct Lifelong Learning displayed a poor (.5 < α < .6)internal consistency, and constructs Communication Skills and Personal Development displayed a questionable (.6 < α < .7)internal consistency.During the post-test, Project Management was the only construct that displayed a questionable (.6 < α < .7)internal consistency.Although this result could be interpreted as the construct not functioning adequately, in previous uses of NESLOS this was not the case. 65,67,68With each NESLOS item, students were asked to self-report on their ability to International Journal for Service Learning in Engineering Special Edition, pp.1-28, Fall 2013 ISSN 1555-9033 10 achieve the learning outcome as a result of the sophomore design project.The self-report ratings were based on a five-point scale.Students were asked "Because of this project, how able am I … [to meet the specified learning outcomes]."A rating of five corresponded to a "high ability," a four to "more than adequate ability," a three to "adequate ability," a two to "limited ability," and a one to "no ability or no experience." For purposes of gauging the significance of learning gains over the three time periods, oneway repeated-measures analyses of variance (ANOVA) were computed for each of the eleven NESLOS constructs.The sphericity assumption was assessed using Mauchly's test for evaluating significant differences among the between-conditions variances.In the cases where sphericity was violated, degrees of freedom were corrected using the Greenhouse-Geisser estimates of sphericity, 71 and the Greenhouse-Geisser corrected tests were used for reporting significant values.Within-subjects contrasts were also computed for each construct for evaluating the significant differences among the three time periods.Finally, Cohen's d was computed for the effect size of the mean differences. 72It is important to recognize that using repeated-measures analysis of variance involves conducting multiple analyses, which increases the chances of Type I error.This issue is approached by lowering the alpha level of significance to 0.01.Therefore, only statistically significant differences equal to or lower than 0.01 are reported.Also, when discussing the results, more attention is going to be focused on the magnitude of the effect sizes (mean difference and Cohen's d) than on the statistical significance testing.Another possibility to approach the increment of Type I error would be to conduct MANOVA.This type of analysis would not be appropriate because there is no theoretical basis for combining the different NESLOS constructs in a single analysis. 73The NESLOS constructs should rather be taken as addressing different learning outcomes, therefore, they are conceptually independent.

Project Management
Ability to manage a project (tasks, timeline, budget, etc.). 5 Create and follow a timeline when managing projects Manage planning and organization of project tasks and processes 0.79

Big Picture Skills
Ability to recognize connections between coursework, subject matter, disciplines, etc. 12

Qualitative Data Collection and Analysis
Along with the NESLOS quantitative items, several open-ended questions were also included to gain deeper insight into students' learning outcomes.The three questions were: (1) What did you find most valuable during this project?(2) How do you think this project prepared you for becoming an engineer?(3) What aspects of this project were difficult or challenging for you?Qualitative responses from questions one and two were analyzed for exploring students' major gains as a result of participating in the PBSL experience, and qualitative responses from question three were analyzed for exploring student's major challenges faced during the PBSL experience.
To capture a deeper insight into students' learning all qualitative responses from the dataset were included in the analysis.This corresponded to 72 responses from the fall sophomore data set (N = 72) and 45 responses from the spring sophomore data set (N = 45).Data analysis of the three qualitative survey items began with the iterative development of a coding framework.Thematic network analysis, recommended by Attride-Stirling for interpreting complex qualitative data, was deemed most appropriate because it allows for the systematic extraction of common themes and evaluation of the relative importance of each. 74Two researchers developed a coding framework by noting common thematic threads surfacing in the responses.The final coding framework was evaluated by a third and fourth independent researchers and was used to code the data into thematic groups.Subsequently, these groups were merged into common themes.Data collected with the qualitative items were aggregated and descriptive statistics were tabulated and graphed to facilitate interpretation of the results.For enhancing reliability two researchers coded all the data independently using a "line-by-line procedure". 75The inter-coder level of agreement was calculated for each set of data (fall sophomore and spring sophomore).There was an 85% agreement between the two coders, meaning that the two coders used the same codes for coding the same data 85% of the time.This value represents a good level of inter-coder agreement, 76 increasing our confidence in the reliability of the coding process.In Table IV, the themes that emerged from the qualitative data set are listed as well as a brief description of each theme and sample quotes."Practice in working on a team, meeting deadlines."

Team Work
References to working on a team, gaining interpersonal skills, etc.
"Gave me skills to work effectively in a team setting."

Interaction with Client
Interacting with the client/customer and learning from the client.
"The use of an actual and real client."

Problem Solving
Making decisions in order to solve the design problem "Going through the design process and creating a solution to our problem."

Applying Previous Knowledge
Applying knowledge from previous courses such as calculus, physics, or statics and dynamics "Put into practice the material we have been learning."

Creativity
Using one's imagination/creativity in producing unique and innovative concepts and designs "The Concept Generation Phase forced me to think outside of the box and gave me the confidence to be the creative engineer that I know I can be."

Technical skills/tools Software Tools
Using a variety of software during the design process "Microsoft Excel skills."

Prototyping Tools
Using prototyping tools during the building process of the project "It showed me how important prototyping is… not everything is as it looks on paper."

Real-world Application Real-world Application
Being enabled to work on a realworld project that allows students to have a holistic view of engineering practice "Getting a chance to actually work with a real life scenario."

Self-awareness &Engineering Identity
Developing an engineering identity and learning oneself (awareness of weaknesses and/or strengths as engineers).
"It made you think like an engineer.Now I look at everything from the perspective of an engineer: always thinking how to make improvements." International Journal for Service Learning in Engineering Special Edition, pp.1-28, Fall 2013 ISSN 1555-9033

RESULTS AND DISCUSSION
In following with the concurrent mixed-methods triangulation design followed in the study, this section is organized in three sub-sections: quantitative results, qualitative results, and quantitative and qualitative triangulation and discussion.

Quantitative Results
NESLOS results for the 34-student longitudinal sample of this study are presented in this section.More specifically, Table V shows mean and standard deviation values for each of the eleven NESLOS constructs for the three time periods.Table V also shows results of statistical significance testing in comparing across the three time periods.The three comparison tests were: fall sophomore year and fall freshman year, spring sophomore year and fall sophomore year, and spring sophomore year and fall freshman year.Overall, almost all the NESLOS constructs present a significant increase of their means from fall freshman year to fall sophomore year and fall freshman year to spring sophomore year.These mean differences are considered to be practically significant considering the large effect size (Cohen's d ≥ 0.8). 77The only exception is NESLOS construct Experimentation Skills (in the comparison of fall freshman year and fall sophomore year), which revealed a smaller effect size and did not show a significant mean difference.When comparing the longitudinal sample from fall freshman year to the spring sophomore year, all NESLOS constructs present a significant increase of their means and are all considered practically significant given the large effect sizes.When comparing the fall sophomore data with the spring sophomore data (i.e.learning from fall to spring semester of the sophomore design project), it is observed that Experimentation Skills is the only construct that shows a significant increase and a moderate effect size (0.5 ≤ Cohen's d < 0.8).This finding coincides with the focus of the spring sophomore semester being prototyping.The rest of the NESLOS constructs reveal no significant increase from the fall sophomore semester to the spring sophomore semester.In some cases, there is a decrease in the perceived learning (evidenced by the negative mean difference values), which is indicative of the learning focus each semester.
Although mean NESLOS construct values provide insight into students' learning and growth longitudinally, and allow us to perform significance testing and calculate effect sizes to gauge longitudinal learning gains (Table V), it is also important to assess how deep and broad this learning was for students.In this regard, we also present in Table VI the percentage of participants who rated NESLOS items with a three, four, or five during the pre-test (fall freshman year) and post-tests (fall and spring of sophomore year).This represents the number of students who rated their ability with NESLOS items and constructs as "adequate", "more than adequate", or "high."The last three columns of Table VI represent the percent differences between the pre-test and two post-tests.The higher differences, designated to be above 15%, are bolded to show deeper learning in those areas of NESLOS learning outcomes.All constructs show an at least adequate increase in the percentage of students positively rating learning outcomes as a result of the sophomore design project.This was evident in comparing the two post-tests with the pre-test (control sample).As would be expected, on average, the differences between post-test two and the pre-test are greater than between post-test one and the pre-test.More specifically, in comparing post-test one and pre-test differences, the highest-rated NESLOS construct learning outcomes (in bold) were Big Picture Skills, Communication Skills, Personal Development, Lifelong Learning, and Ethical and Societal Awareness.Interestingly, these high-rated constructs represent affective learning gains.Similarly, in comparing post-test International Journal for Service Learning in Engineering Special Edition, pp.1-28, Fall 2013 ISSN 1555-9033 two and pre-test differences, two additional constructs, Engineering Skills and Experimentation Skills, were identified as leading to deeper learning.

Qualitative Results
Qualitative responses from questions one and two were analyzed for exploring students' project learning gains as a result of participating in the PBSL experience.Table VII combines the results of both question one and two.These results are mapped to each identified emergent theme.Responses are categorized as sophomore fall (post-test one), sophomore spring (post-test two), and total (sum of post-test one and two).To account for the change in number of respondents from the fall semester to the spring semester, Figure III reports the percentage of total students in a particular semester who mention each theme.Results for fall and spring semesters are plotted together.

FIGURE III PERCENTAGE OF STUDENTS, FROM POST-TEST 1 (FALL SOPHOMORE SEMESTER) AND POST-TEST 2 (SPRING SOPHOMORE SEMESTER), WHO MENTIONED EACH THEME WHEN ASKED ABOUT THE MAJOR POSITIVE LEARNING OUTCOMES DURING THE PBSL EXPERIENCE.
Design Tools & Methods, Teamwork, and Real-world Application had the most mentions among the respondents with Design Tools & Methods having the highest percentage in the fall, while Teamwork and Real-world Application had the highest percentage in the spring.These results are not surprising as the fall semester tends to focus more heavily on the theoretical underpinnings of the design process while the spring semester tends to focus more on the lessons of implementation (through design prototyping and prototype testing) and interpersonal skills (working in and managing a large engineering team).Interestingly, many students do not recognize the creativity associated with the design-build-test prototype-intense portion of the spring semester, nor do they note the value in documentation, dissemination, or application of previous knowledge.One may speculate that this is due to their immaturity as engineers, and that they may learn to value these skills as they mature into professional engineers.The most surprising finding, however, is the low rate for mentioning interaction with the client in either semester.It is difficult to speculate why this occurred; perhaps the students take this opportunity as a given since they have not experienced engineering design in any other context prior to this experience.The result for Self-awareness and Engineering Identity provides promise though, as students may be learning to reflect on themselves as individuals as well as beginning to see themselves growing into the role of professional engineers.This self-awareness also speaks to students' ability to reflect on their strengths and weaknesses as engineers and as individuals and this all is a positive outcome.For instance, one student from the fall semester mentioned the following: "[the concept generation phase] forced me to think outside of the box and gave me the confidence to be the creative engineer that I know I can be".
Qualitative responses from question three, tabulated in Table VIII, were analyzed to explore the challenges faced by students as they worked through the PBSL experience.Again, the results are mapped to each identified emergent theme.Responses are categorized as sophomore fall (post-test one), sophomore spring (post-test two), and total (sum of post-test one and two), and to account for the change in number of respondents from the fall semester to the spring semester, Figure IV reports the results as a percentage of the total students in a particular semester who mention each theme.Results for both the fall and spring semesters are again plotted together.
Interestingly, the elements which ranked as the highest positive learning outcomes also tend to rank as the biggest challenges.Not surprisingly, the biggest challenge reported by the students is teamwork, with 50% of the student responses being categorized into this theme.One common reason mentioned by many students is that not all the team members worked hard.One student from the fall semester mentioned "I got along with my team members just fine, but found that there were one or two team members that weren't pulling their weight."Another student from the spring semester mentioned that the most difficult part of the project was "working with a team with different values and goals and willingness."During the sophomore spring semester, students at a much higher frequency rated working on a team to be very challenging.This is probably attributed to the fact that team size from the first semester of the project (fall sophomore), which averaged 4-6 students, nearly doubled during the second semester of the project (spring sophomore), which averaged 9-12 students per team.The added challenge of managing a larger team was certainly expected.One quote from a spring semester student represents very well this challenge: "I found working with such a large team incredibly challenging.Unfortunately, with a team that size (11 people), there are bound to be people who don't want to do anything…" Similarly, project management is highly reported with 25% of respondents stating this challenge.The following quote from a spring semester student represents one common issue reported by students: "Time constraints and establishing times for people to meet."Additionally, in the fall, when design theory is the primary theme of the course, 21% of the students report design as the course challenge, and in the spring, when prototyping is the primary theme of the course, 25% of the students describe that the prototyping presents them with a challenge.Again, these results are not surprising as the challenges are in agreement with the desired learning activities and outcomes of each semester.

Quantitative and Qualitative Triangulation
Review of both the quantitative and qualitative results reveals some interesting trends among this cohort of students who participated in the PBSL experience.Most interesting is in the area of Personal and Professional Development and the beginnings of seeing oneself as an engineer.In the qualitative results, 24% of the students report an increase in self-awareness and beginning to develop their engineering identify.This result is supported in the quantitative results, with a 20% positive difference between the pre-test and post-test two.A 19% difference is also noted between the pre-test and post-test two in the area of Lifelong Learning.Students appreciate the real-world application of the project with 32% of the students reporting this as an important learning gain of the project in the qualitative data.This too is supported by the quantitative data with a 17% difference in Ethical and Societal Awareness between the pre-test and post-test one and a 15% difference between the pre-test and post-test two.Also, review of both the quantitative and qualitative results begins to provide insight into the answers to the two research questions posed at the beginning of this article.When comparing the pre-test to post-test two, gains are identified in all constructs of the NESLOS instrument with the highest gains in Big Picture Skills, Communication Skills, Personal Development, Lifelong Learning, and Ethical and Societal Awareness.However, students' self-reported learning gains to open-ended questions tended to focus on the Design Tools &Methods, Prototyping Tools, Project Management, and Teamwork.Interestingly, neither list is mutually exclusive.Communication skills are a vital component of project management and teamwork.Design Process, design tools, design methods, and prototyping frequently push an engineer to look more broadly, explore more options, and consider the big picture (i.e., understand a design as a system of systems).With ethical and societal awareness, students are learning their role as an individual in a system of systems.In short, these results align with reported benefits of service learning projects. 1,2,78,79

CONCLUSIONS AND LESSONS LEARNED
Overall, our findings show that students not only valued this real-world PBSL experience, even though they were challenged by the complexity of the project, but also gained valuable knowledge and skills along the way.In summarizing some of the key findings, the data revealed that exposing students to a "real" PBSL problem with a "real" customer truly motivated students to: (a) learn and apply new domain-specific domain knowledge, (b) understand and implement a design process and design skills, (c) value and be challenged by working in a team setting, (d) gain much needed project management skills, (e) recognize the need and importance of research, (f) deliver a solution to their physically-disabled client, (g) recognize the relevance and connection of this project to real-world engineering practice, (h) understand the responsibility and importance of providing a service to a client, (i) see themselves as engineers or at least becoming engineers, and (j) value and be challenged by real-world PBSL experiences.
We have learned, however, that as sophomores, many students are not ready to be pushed directly into a PBSL project, and consequently, when faced with the reality of unstructured and complex "real-world" problems in their own community, students react in non-constructive and potentially team-destructive ways.For many students, this project is their first realization of social systems and their future role as members of a social system as practicing engineers.In the JMU Engineering Department, understanding and striving to engender positive impact on individuals, communities, regions, and cultures reflects social sustainability and a key element of our program.Over the past few course iterations, the course has been changed to scaffold students into the project and lessen the dissonance created when experiencing the "real-world."This point has been strengthened through not only readings, lecture, and in-class discussions, but also, individual reflection, team-based bonding activities, non-project-based client-team interactions, and local community-based activities.The students are being pushed to see themselves as a part of a larger social system where they can and likely will have a role in the future.Students begin to understand that their decisions as engineers not only affect the company that they work for (in this PBSL case the University and the Department), but that those decisions also affect the broader community and the global system (in this case the client and the client's family).Consequently, students are given the chance to reflect on their own ethics and values so that they have an opportunity to align their ethics and values at the individual level, because ultimately, it is their decisions as engineers that lead to being a successful engineer.

LIMITATIONS
Although a longitudinal study, some of the limitations of the current study include the small sample size.Further, the main use of self-report data could be strengthened by more objective and direct assessment data, such as performance measures by instructors.Although not something that could be done at JMU, since all sophomore students complete the same PBSL sophomore design course sequence, a valuable study would be to compare students who were in our PBSL course with students in a similar sophomore design course without a PBSL experience.Further, during the freshman year survey data collection, some items of the NESLOS constructs were accidentally not included and thus the freshmen year data sample does not have the complete set of NESLOS items.Lastly, although the internal consistencies of most of the NESLOS constructs had acceptable or good level of internal consistencies, some of the constructs did not which could affect the findings.

FUTURE WORK
Complex problem solving is process-oriented.In our academic and professional lives, we are faced day-to-day with unique settings related to problems we have never before faced.We find our approach to problem-based learning and the complex nature of problem solving effective in a variety of academic, professional, and personal settings.It is the process that is important for students to learn and continue to practice in and out of academic settings.Problem and process based learning is inherently useful to students, since it is self-perpetuating and supports continual growth and usefulness in varied settings.We plan to continue to expand the contexts in which these skills are practiced and assess our students' skills in these varied settings.
A problem solving topic discussed infrequently is the relationship between problem solving processes and those used for idea generation and creativity.We have seen students in our own design program employing the same problem solving processes they use in the design studio to confront unexpected problems (i.e. to generate ideas not generated by a specific problem).Ultimately, problem solving and idea generating processes are quite similar, and differ only in the context and settings in which they occur.It is up to students (in our case) to determine alterations in the process that are necessary for addressing each unique situation.To this end, areas of future work include ways to embed creativity throughout PBSL design experiences.
Most engineering programs (design and other) focus on team-based learning without considering individual growth.While this is certainly a critical skill for students to acquire, it is based on students learning these processes in the absence of enough individual work to successfully underlie group work.We believe that if one cannot develop and demonstrate effective thinking skills useful individually, then she or he will likely be of limited use to a collective effort."Personalizing" problem solving (adapting thinking skills to fit students' intellectual style and daily routines) and the effective individual thinking and personal processes that support group work, are underrepresented in most engineering programs.There is thus a need to helping students "triangulate" thinking skills, employing more than one problem solving method or thinking strategy, and then synthesizing and evaluating findings to meet the needs of a particular problem solving or idea generating situation.We believe developing individual thinking and problem solving processes is central to students acquiring useful and varied cognitive skills that are effective across multiple academic, professional, and personal contexts.To this end, PBSL experiences provide a strong framework to teaching "personalized" problem solving skills to our students.
FIGURE I JMU ADAPTIVE PROBLEM-BASED LEARNING (PBL) CLASSIFICATION FRAMEWORK4 C o m p le x it y S t r u c t u r e d n e s s Group Structure Offer a Variety of Multifaceted Problems Enables Students to Experience Different Modes of Thinking, Learning, and Problem Solving International Journal for Service Learning in Engineering Special Edition, pp.1-28, Fall 2013 ISSN 1555-9033 9 Research Design

TABLE VI PERCENTAGE
OF STUDENTS RATING THE NESLOS OUTCOMES WITH 3, 4 OR 5 ("ADEQUATE", "MORE THAN ADEQUATE", OR "HIGH") HIGHER DIFFERENCES (ABOVE 15%) ARE SHADED Items from the NESLOS construct "Professional Development" were accidentally not included in the survey and thus corresponding results could not be included herein.

TABLE VII "
PROJECT LEARNING GAINS" THEME FREQUENCY DURING THE TWO SEMESTERS

TABLE VIII "
PROJECT CHALLENGES" THEME FREQUENCY DURING THE TWO SEMESTERS International Journal for Service Learning in Engineering Special Edition, pp.1-28, Fall 2013 ISSN 1555-9033 21 FIGURE IV PERCENTAGE OF STUDENTS, FROM POST-TEST 1 (FALL SOPHOMORE SEMESTER) AND POST-TEST 2 (SPRING SOPHOMORE SEMESTER), WHO MENTIONED EACH THEME WHEN ASKED ABOUT THE MAJOR CHALLENGES FACED DURING THE PBSL EXPERIENCE.