Service Learning Versus Traditional Project-Based Learning: A Comparison Study in a First Year Industrial and Systems Engineering Course

Service learning involves solving a real community problem while meeting course learning outcomes. Participation in service learning is hypothesized to improve undergraduate student engagement and retention, but little research has been done to measure its impact specifically on beginning engineering students. This study compares two sections of an introduction to industrial and systems engineering course; one with a service learning term project and one with a traditional project-based term project. The service learning project was designed to be a hands-on approach to the material in the project management, communication, and teamwork modules of the course, in addition to giving students the opportunity to practice industrial and systems engineering functions related to their community partner’s defined problem. Surveys, grade data and interviews provide evidence that service learning projects improved students’ perceptions of their preparation for a successful academic and professional career and also lend limited support to improved engagement and retention in engineering compared to the students in the traditional project section. Index Terms – community partners, first year curriculum, industrial and systems engineering, term project.


INTRODUCTION
Service learning is gaining popularity as an educational tool and is widely believed to help with engagement and retention 1,2 . Bringle and Hatcher 3 define service learning as, "a credit-bearing educational experience in which students participate in an organized service activity that meets identified community needs and reflect on the service activity in such a way as to gain further understanding of course content, a broader appreciation of the discipline, and an enhanced sense of civic responsibility." There is a rich body of literature on service learning, including the extensive literature review by Eyler et al. 1 which shows that service learning enhances personal outcomes, social outcomes, learning outcomes, career development, and student retention. A comprehensive study of more than 22,000 undergraduates found that service learning leads to positive effects in academic performance, as measured by GPA, writing skills, and critical thinking, and also student values and plans to participate in community service in the future 4 . Further research has continued to show that service learning improves student learning, energizes the classroom by applying academic concepts to real-world problems, and increases student interest in the field 3,5 . Today's engineering students are increasingly focused on social consciousness 6 . In surveying first-year engineering undergraduates as well as high school students exposed to service learning, Zarske 2 found positive impacts in identity and attitudes towards community service, especially in underrepresented populations, that may help in recruitment and retention of those groups. In engineering specifically, researchers studying a sophomore service learning engineering experience noted benefits included that students learned and were able to apply engineering knowledge, they valued and were challenged by working in a team setting, they recognized the relevance and connection of the project to real-world engineering practice, and they could "see themselves as engineers or at least becoming engineers 7 ." Many engineering programs have implemented successful service learning courses, at levels ranging from just a component of a single course to a complete course to providing opportunities throughout the curriculum. According to a 1999 survey of United States engineering deans that received 52 responses, the majority of service learning in engineering was occurring in senior level courses, especially capstone courses 8 . For example, a Villanova University structural capstone course that incorporated an international service project was studied and it was found that students participating on the service trip achieved significantly higher non-technical outcomes compared to those students that worked solely on the design project 9 . In another example, the School of Engineering at the University of North Florida partnered undergraduate engineering students with physical therapy students in design projects for rehabilitation technology for children with disabilities 10 . Duke, California Polytechnic State University, Michigan Tech, South Dakota State University, San Jose State University, Southern Illinois University, and the US Military Academy are just a few more colleges that have documented successful service learning in engineering capstone courses 11,12,13,14,15,16 .
A few innovative engineering institutions have threaded service learning throughout the curriculum. Since 1995, Purdue has offered a highly successful service learning program, EPICS, in which multi-disciplinary teams of students of varying grade levels partner with local and global community organizations to address human, community, and environmental needs. Students may participate for up to 7 semesters on the same team 17 . The University of Massachusetts at Lowell has made great strides in fully integrating service learning into the engineering curriculum across their various majors such that in many cases students could take a service learning course that applies toward their degree every semester 18,19 . And the University of Vermont Civil and Environmental Engineering Department has integrated service learning into the curriculum with the objective that students have a required course with service learning at least once a year 20 . Yet engineering lags behind other majors of study in implementing service learning 21 .
Student retention and engagement remain key concerns in engineering education. A recent survey conducted by the American Society for Engineering Education 22 showed that while undergraduate engineering education student retention and graduation have been increasing, after just the first year of their undergraduate studies programs are still losing approximately 20% of students who were intending to major in engineering. That same survey found that six-year graduation rates are only approaching 60% and are below 50% for Hispanic/Latino students and below 40% for African American/black students. There is significant variability on the retention rates of these groups, as shown in ASEE 2017 23 and by university, as found in a study by Ohland et al. 24 A 2009 study of 113 undergraduates who left engineering found three key reasons for the students' decisions to leave the major: poor teaching and advising; the difficulty of the engineering curriculum; and a lack of "belonging" within engineering 25 . These may be especially true for women and minority students. As one way to mitigate some of those key shortfalls, ASEE recommends "offering a socially relevant curriculum that emphasizes service learning" as a best practice for student retention 26 .
When students start their first semester at a higher learning institution, they are exploring their interests, their strengths, and shaping their future. Universities often offer introductory classes within a major to give students a glimpse into what their academic and professional future will be like. Many of these orientation classes require a project to be completed, which is a logical opportunity to implement service learning. There is evidence that integrating service learning projects into the academic experience increases student retention rates, and increases overall student engagement 4,18 . Thus more engineering programs have added service learning components earlier in the degree program, such as to orientation or introduction to design courses. At the University of South Alabama an introductory mechanical engineering design course successfully pairs teams of mechanical engineering students with math and science teachers to design, build and deliver hardware and software to meet the needs of those teacher clients 27 . Similarly, the University of San Diego has an introduction to engineering course that partners first-year engineering students to prepare a hands-on, educational activity for sixth grade science students at an economically disadvantaged middle school 8 . Both papers cited include reflection on the experiences, but there is a limited published research regarding the effects of service learning on first year engineering students. One exception is Zarske's dissertation work 2 in which she studied the impact on first-year engineering students engaged in service learning project-based design at the University of Colorado and found that it may help in recruitment and retention, especially of underrepresented groups in engineering.
This paper aims to further reduce the gap in knowledge of the impact of incorporating service learning into first-year engineering courses. Using a mixed-methods approach with quantitative and qualitative survey data and interviews, this research provides insight into how participating in a service learning project can adjust freshmen students' perception of their different skills that are vital to having a successful engineering career, as well as their likelihood of remaining in an engineering field of study. Specifically, the research questions for this study are:  Research question 1: Does adding a service learning term project to an industrial and systems engineering orientation course improve student engagement and retention when compared to a traditional project?  Research question 2: Does adding a service learning term project to an industrial and systems engineering orientation course improve students' perceptions of their preparation for a successful academic and professional career?

METHODS
Kennesaw State University (KSU) is a comprehensive university in the University System of Georgia. With over 32,000 undergraduate students and almost 3,000 more enrolled in graduate programs, it is the third-largest university in the state. The Systems and Industrial Engineering Department enrolls just over 600 students across four undergraduate and three graduate degree programs. The 3 credit hour orientation course described in this paper is in the ABET-accredited Bachelor of Science in Industrial Engineering degree, and is a required course for all students in that degree program. The primary goals of the course are to (1) introduce students to the university in general, (2) introduce them to the field of industrial and systems engineering, and (3) develop positive skills for success in college. As is the case with much of the degree program, this course is offered in a hybrid format -meeting for one 75-minute live session on campus per week with additional asynchronous work for the students for the other 50% of the course. Seeking to enhance the first-year experience for students, and to improve retention rates, the orientation course underwent a targeted redesign to increase the number of hands-on activities and make the course more engaging for students. Weekly class meetings ranged from guest speakers from career services and from the counseling center to presentations by faculty in the department to hands-on learning exercises related to industrial engineering. For the asynchronous portion of the course, students viewed various videos and read material related to study skills and to the field of industrial and systems engineering. In addition, the course included a significant and redesigned term project, which is the focus of this paper. Due to the known benefits of service learning, the instructor chose to experiment with a service learning project for one of the two sections of the first semester of this redesigned course.
Self-reported demographics of both sections of the course are reported in Table I. The traditional project-based section of the course enrolled 35 students and the service learning section 30 students. Due to concerns about obtaining enough community partners in the first year, only one of the two sections of the orientation course utilized service learning. Students registered unaware of this difference between the sections, which allowed for essentially random assignment of students to the service learning "treatment" versus the traditional project "control". As shown in Table I, demographics of the two sections of the course were fairly similar. One notable difference is that more students in the service learning section had already taken some of their other industrial engineering coursework. In most cases, the students who had taken only one or two previous courses had entered the university in the spring as transfer students when the orientation course had not been offered, while those who had taken more than two courses likely should have registered for the orientation course the previous year. The service learning section also had a higher percentage of female students than the control section. The percentage of female students in the control section is much more typical of other classes in the department and the authors are unaware of what caused the difference besides random variation.
The service learning project was designed to be a hands-on approach to the material in the project management, communication, and teamwork content areas of the course, in addition to giving students the opportunity to practice industrial engineering functions related to their community agency's problem. The project was set up with assistance from the university's Department of Student Leadership and Service using best practices from the National Society for Experiential Education 28 . The project was mapped to several of the course learning outcomes and reflection was incorporated into the learning process. Students ranked their preferred projects and based on those preferences 6 teams of 5 students each were formed by the instructor. Every student was able to be assigned to either their first or second choice project, with the majority assigned to their first choice. No other formal teaming criteria were used, except for breaking preference ties by not isolating female students. Some example projects in the inaugural semester were: creating a process flow document for a local recycling center, developing an industrial engineering module for an outreach program to elementary students, and putting together a communication plan for their own systems and industrial engineering department to better reach students with important announcements.
The expectation, shared with students and community partners, was that over the course of the semester-long project each student would spend approximately 20 hours actively working on the project solution (collecting information from the community partner, putting together a solution) and approximately 10 hours managing the project (administrative team meetings) and writing reports. Shortly after meeting with their community partners the students had a course period focused on project management and completed a project plan for their project in their groups. Halfway through the semester the teams met individually with the instructor to go over their progress, and at the end of the semester they completed a final report and reflection exercise and presented their project to their classmates. Further details of the project implementation and assessments are available in 'Incorporating a Service Learning Term Project in an Industrial Engineering Orientation Course'. 30 The traditional project section used a slight variation on the hands-on term project explained in 'Consumer Reports Inspired Introduction to Engineering Project' 29 . In the traditional project students worked in a team to develop a set of three tests to conduct on three brands of an inexpensive product of their choosing, design an experiment, perform controlled tests, and analyze the results to recommend a superior brand. The term project for the control section of the 23 course had similar time requirements and similar assessments of project planning, final paper, and presentation. The project had similar learning outcomes to the service learning project related to project management, communication, and teamwork, though no community partner or real-world problem to solve. Teams of size 5 were formed randomly by the professor, with minor adjustments made not to isolate female students. All other aspects of the two sections of the course were identical, including the instructor.
Students in both sections of the course completed pre-and post-surveys. Both the pre-and post-survey included questions assessing the students' perceptions of their skills, based on ABET required student outcomes, and questions assessing their likelihood to continue to pursue a career in engineering. The pre-survey also focused on the demographic composition of the class and the post-survey gathered student opinions on the overall class and project structure. The surveys were approved by our institute's IRB, were administered by an undergraduate researcher and were kept sealed until grades were submitted for the semester. (Copies of the surveys are available from the authors upon request.) Grade data for the project and for the course as a whole was also collected for both sections. All data was analyzed in Excel or Minitab. Written reflection was included as a minor part of the service learning students' grades, in line with best practices 28 , and additional qualitative feedback was obtained from these students' response to prompts. Finally, in order to obtain further feedback on the effectiveness of the service learning project, an in-depth interview was conducted of three students from the service learning group, focusing on what the students felt they gained from the project. A recruitment email was sent to all students the semester after the course. Three students from the service learning section and none from the traditional section volunteered. The interviews were individually done, semistructured, lasted about 10 minutes, and were audio-recorded at the permission of the participants.

Grade Data
Summary grade information is presented in Table II. There were not significant differences between the course grades of the two orientation sections (Mann-Whitney p-value 0.09). This was due at least in part to the typically high grades for the orientation course. There was a statistically significant difference between project grades in the two sections (Mann-Whitney pvalue < 0.01), but again both were high. In both sections, no students withdrew or failed the class, and in each section only one grade of C was earned. The remainder of students earned A's or B's, with approximately 80% of students earning an A in each section.

Survey Results
Students took pre-and post-surveys that were matched by name, allowing the change in perception over the course of the semester to be compared using a paired test. Twenty-nine of 35 students in the traditional section completed both surveys with usable result and 21 of 30 in the service learning section. Students were first asked about their likelihood of pursuing an engineering career and then specifically an industrial and systems engineering career. Means and standard deviations for those responses are presented in Table III, along with the nonparametric paired Wilcoxen signed-rank test p-values. In the service learning group, the students had a statistically significant increase in their confidence that they wished to pursue a career in some sort of engineering between the beginning and the end of the semester. There were no other statistically significant differences though there were slight increases in confidence across both questions for both classes. 25 of their mastery of the ABET "a through k" student outcomes (listed in Table IV) were used as a measure of their perceptions of their preparation for a successful academic and professional career. For each section the mean for the pre-survey (Pre), mean for the post-survey (Post), mean of the paired differences (Diff), p-value for the paired Wilcoxon signed-rank test (P-val.), and percentage of students that had in increase in their perception of mastery (Inc.) are reported. The ABET student outcomes found in Criterion 3 of accreditation requirements are a well-respected list of outcomes that programs must demonstrate that their students achieve during their engineering program. (Note that at the time of this study, the outcomes were a list lettered a through k, but they have recently changed to a similar list numbered 1 through 7 31 .)  26 In the traditional project group, the only statistically significant increase was in criterion k. However, there was a statistically significant increase in six criteria for the service learning group, marked with bold font in Table IV. Finally, in the post-survey students were asked to rate on a scale of 1-10 (where 1 represents not at all effective) how effective the various class activities were in contributing to what they learned in the class. They were also asked how interesting they found the class activities, on the same scale. Textbook readings were from Engineering Your Future: A Brief Introduction to Engineering 32 . Other faculty in the department were invited to classes to speak for 10-30 minutes about their interest area in industrial and systems engineering and share some information about the department's courses that they coordinate. The instructor's lectures were mostly about success in college, specifics to the major such as the differences between the concentration choices and minors, and a few remaining areas of industrial engineering not covered by the other faculty. Many hands-on activities were done in the class such as a paper airplane assembly line to demonstrate manufacturing and a newspaper layout exercise to demonstrate optimization. Other homework assignments were related to success in engineering such as completing a resume, getting a 2-year course plan approved by the student's advisor, and attending a meeting or speaker related to engineering. Means, standard deviations, and nonparametric Mann-Whitney p-values for the differences between the sections are reported in Table V. percentage from that section have also been academically dismissed, neither of those differences are statistically significant. These results should also be interpreted with caution because KSU unfortunately does not track data on whether students enroll elsewhere when they disenroll. The instructor is aware anecdotally that a few from the traditional section in the "unknown reason" category in Table VI have now enrolled at a larger local engineering school and are continuing pursuit of an engineering degree. While this negatively impacts our university's retention numbers, it is not a bad result for the student, and if that data were fully available would likely make the two classes much more similar in retention to college in general.

Interview Results
Of the three students that were interviewed by the undergraduate researcher, all had positive experiences to relate. They indicated that they learned valuable skills that are not typically learned in a normal classroom setting, such as project management, client relationship management, and specialized engineering techniques. They also spoke to the benefits the project had on their other classes, with two of the three noting that they used the skills they had learned from their service learning project in a few of their classes, such as Project Management, both the same and the following semesters. All also agreed that the team aspect of the project was beneficial because they got more experience working with a team united under the same engineering goal, versus the grade-focused goal of the traditional project. One student even said that he was able to use the skills learned through the project to enhance his resume by adding some skills, both technical and personal. While the students said that the service learning project didn't necessarily inspire them to change directions within their college career, it encouraged them to continue engineering because it gave them a glimpse into what the real world is like. To quote one student, "being able to participate in such a hands-on project showed me how it was going to be like when I graduate and now I'm sure that I was right when I picked engineering." Another student spoke about the challenges that the team faced during the project, saying that she believed the benefits of the project outweighed the difficulties. All three students said they felt like their projects were more meaningful, overall, and said they would much rather do the service learning project instead of the traditional project. They felt the service learning project offers the kind of real-world 28 experience that is lacking in the engineering education. Each one spoke to the benefits they believed they gained from participating in this project and said they were excited to see evidence of project implementation. As one student said, "engineering is really tough, but it was cool to see how actual clients were using our project and hear plans for them to use it more."

DISCUSSION
Research question 1: The authors hypothesized that adding a service learning project to an industrial and systems engineering orientation course improves student engagement and retention when compared to a traditional project.
Preliminary grade and retention data from these two sections did not support benefit existing in the service learning group, except for a slightly higher project grade in that section. Grades were high in both sections of the course and no students withdrew from or failed either course.
In the surveys, students were asked their likelihood of remaining in engineering, as well as in industrial and systems engineering specifically, at both the beginning and ending of the semester. There was a statistically significant increase in the reported likelihood of remaining in engineering or engineering technology from the service learning group which points to some motivation for increased retention in engineering.
Students were asked how interesting they found the term project, as well as the other class activities. Here the students that were in the service learning group rated the project slightly more interesting than those in the control (7.5 versus 7.3 out of 10), but this difference was not statistically significant. In addition, students did not rate the term project as interesting as other components of the course. We were especially surprised that the service learning section found the instructor and guest speaker lectures more effective and interesting, given that section had a high percentage of non-freshmen students. Many of the instructor lectures were on topics like advising and study skills the authors thought would be more relevant to freshmen.
Finally, preliminary retention data is inconclusive at this point. Taken together, these three measures provide only modest evidence that a service learning project assists with student engagement and retention.

Research question 2: The authors also hypothesized that adding a service learning term project to an industrial and systems engineering orientation course improves students' perceptions of their preparation for a successful academic and professional career.
This was first measured based on students' perceptions of their mastery of ABET outcomes a-k.
Here there was more difference between the two classes. In the control group, students perceived a growth in their ability related to outcome k only -"the ability to use the techniques, skills, and modern engineering tools necessary for engineering practice". In the service learning class, students perceived growth in 6 of the 11 outcomes. This difference suggests that each type of project was successful in its original aim. The traditional project was focused on learning new engineering information and techniques, which was the only perception that was statistically significant. Service learning projects focus on enhancing the learning experience while integrating real-world experience, in addition to the new engineering information and techniques. This allowed students to use their engineering skills in a hands-on environment which served to increase their perceived confidence in a wide range of areas from functioning on multidisciplinary teams to solving real-world problems using an engineering approach. This was reflected, for example, in the improvement in outcome c "the ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability", outcome f -"an understanding of professional and ethical responsibility", and h -"the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context". This is also notable in that these outcomes are sometimes the most difficult ones to map to courses in the engineering curriculum. It should also be noted that freshman engineering students may be poorly calibrated to assess their own skills, so some decreases in their responses could be more a result of their increasing awareness of the difficulty of a skill rather than an actual perceived dip in their skill in an area.
Students were also asked how effective they found the term project to be. In this case, when comparing the means, the students in the traditional group rated the term project to be more effective in contributing to what they learned in the class (8.1 versus 7.4 out of 10). However, this was driven by a few students rating the project especially low. The median and third quartile are the same for both sections. In hindsight, perhaps the phrase "learned in the class" is also a driver of that difference. While both projects related to the course learning outcomes, the traditional project aligned more clearly to the topics covered in the rest of the course, while the service learning project was more broadly related to industrial and systems engineering in general. Again, the projects were rated lower than some of the other components of the course. As noted above, with exception of the term project, all other aspects of the class were identical between the two sections.

Qualitative Insights
Through the interviews and course feedback, student comments regarding the service learning projects were almost exclusively positive. In the interviews, students noted additional benefits of the project -such as exposure to a real-world problem, a project they could include on their resume and discuss in interviews, and skills to apply to other classes. They were appreciative both of the "real world" aspect of the project and also found value in working in service directly with a local community partner. All these things could potentially lead to increased engagement and retention. In students' reflection on the project they were able to draw significant connections between the project and their future career. In response to the prompt "How has this service experience related to the industrial engineering major …" one student noted, "Obviously, Atlanta is a large hub for engineering jobs but seeing this helped me to see that there are all sorts of fitting jobs for engineering students like us. I feel that this was a valuable experience not only because we got to work with a company, but also because it helps us further narrow down what this experience could be like for us in the future, and help us see where we would fit in within a group or a company." In response to the prompt, "What did you learn about yourself from this experience?" another student commented, "I learned more about why I want to pursue a career in engineering and I'm eager for more opportunities to put my problem-solving skills to practice." These same opportunities to prepare themselves for a successful future career weren't available to students in the traditional project section.

Limitations
The small sample size and preliminary offering nature of the course are limitations to the value of this research. In addition, survey results are all based on students' perceptions of their knowledge and their likely future career. Students were assured that their survey responses would not be shared with the instructor until the grades had been entered for the semester, but students still may have been hesitant to be completely transparent. In addition, since students were told that the reflection exercise would only be checked for completion and not read by the instructor until after the grades were entered for the semester, many of the students did not appear to take it seriously or include useful information.

CONCLUSIONS
Engineering is largely hands-on but stems from a wide range of theoretical knowledge in the sciences and advanced math. Engineering professors have the challenging position of trying to simultaneously teach the technical skills that students need to succeed in the engineering field, while making sure that there is a deep knowledge and understanding of the foundational theories. Additional pressure comes from a desire to retain students in the degree and field. Service learning projects can offer an innovative answer to this issue of uniting theory and practice. As seen from the statistical results and the qualitative surveys, the service learning projects offered an integrative and reflective experience for the students that is not very common in the engineering degree programs, especially during the first year of study. Without service learning, most students only get hands-on experience in internships and cooperative learning in the field, and these positions are typically held by upperclassmen. Based on this study, the authors believe that integrating service learning into the orientation to engineering course is an innovative and feasible way to increase engagement in engineering and equip students with tools early in their curriculum that will increase success in the rest of their study.
In future work, the authors plan to follow these students to measure their actual retention and graduation rates, as well as those of future sections of the course. Currently, this study adds to previous efforts to measure the effectiveness of service learning in enhancing the engineering educational experience during the first year of a program. The results suggest that the service learning project improved students' perceptions of their preparation for a successful academic and professional career and also lend limited support to their improved engagement and retention in engineering.