The Challenge of Including Customer Satisfaction Into the Assessment Criteria of Overseas Service-Learning Projects

There has been a dramatic increase of student groups participating in international service learning projects. For engineering students it is not difficult to identify meaningful educational objectives. The students improve their analytical and problem solving skills. They design and build something that fulfils a list of engineering specifications; they execute a solution to some problem. However, these projects have a human dimension. Service-learning involves changes in peoples’ beliefs, attitudes and values; impacting both the students and the recipient community. It is important for the academic community to develop assessment criteria that includes perspectives from all stakeholders engaged in the experience. It is imperative to assess not only the technical success but also the sustainability of the project and its larger effect. In courses involving service-learning, assessment needs to occur on three levels: the traditional evaluation of the student’s knowledge of the technical content, the assessment of the experiences impact on the students’ broader more humanistic “soft skills”, and the customer’s satisfaction. The paper examines the obstacles and opportunities in assessing project success from multiple international service-learning programs, and compiles insights and reflections that could serve to inform future projects. Index Terms – Assessment criteria, service-learning pedagogy.


INTRODUCTION
International service-learning projects take a lot of time and money.It is important that these projects fulfill the objectives of all the people involved.From an academic point of view, student learning objectives and program outcomes must be assessed from a technical as well as humanistic perspective, the so called 'hard' and 'soft' skills.A short summary of assessment strategies/ tools for assessing student learning and program outcomes when using servicelearning as pedagogy is first presented.International projects in the developing world expose students to new cultures and perspectives, as well as poverty and injustice.Students get energized in new ways, which may affect their career strategy or participation in civic engagement.The paper then summarizes available assessment strategies/tools for assessing other desirable objectives such as intercultural competency or evidence of civic responsibility.These two assessment methodologies address the goals of an academic program and originate from within an educational institution.
What about the objectives of other stakeholders?After all it's not all about "us".Will we look back at these projects in 10-15 years and realize they made no difference; abandoned and completely irrelevant?Are they just the latest fad in feel good global education?Engineering projects are not scientifically neutral, but rather they exist in a much less precise social and human world.They could actually do harm.The perspective of the recipient community or organization must be part of the overall assessment methodology.Projects should be "successful": desirable and sustainable for the partner community.We propose that this third major assessment approach must be undertaken by the project leaders even though it is outside the realm of the university/learning institution.Perhaps because it is outside the traditional academic sphere that is usually strictly concerned with student learning, that assessment in this area is virtually missing.In order to gain a better understanding of the community partner's perspective, the authors introduce three simple questions to assess customer satisfaction.
The paper then lists several key elements of success and common obstacles compiled from a review of the literature, phone interviews and personal experiences.Finally, the authors share their personal challenges, successes and frustrations in hopes of improving and informing future international service-learning projects.

Assessment of Service-Learning in Engineering
Driven by the increasingly global nature of engineering and by the requirement under the Accreditation Board for Engineering and Technology's engineering criteria (ABET EC 2000) 1 to demonstrate clearly that graduates of an accredited engineering program have "the broad education necessary to understand the impact of engineering solutions in a global and societal context" (3.h), "a knowledge of contemporary issues" (3.j), "an ability to function on multidisciplinary teams" (3.d) and "an understanding of professional and ethical responsibility" (3.f), design courses using a service-learning pedagogy are emerging within the curricula of many engineering programs. 2ABET EC set forth new criteria reflecting the changing needs of our world, our nation and our students, calling for a more "humanistic" approach to engineering education.These courses promote and measure specific professional skills for solving applied problems in a realistic, start-to-finish design experience from initial planning through field testing and re-design.They provide an expanded view of engineering, where the main point is to bring experience into education.
Service-learning explicitly integrates classroom content into service to a partner community or a community service organization and provides a setting where students can learn about complex social issues and their roles as engineers in addressing those issues.Very important to this pedagogy are metacognitive activities, including reflection, to improve learning.Metacognition, the monitoring of one's thinking as one is acquiring new information, is especially relevant in the practice of the engineering design cycle where all stakeholders are expected to collectively determine the goals and evaluate the success of the project.Critical reflection on the partnering process, as well as the design progression, can be integrated into the course milestones through guided activities such as group discussions, journal writing and reflection worksheets.The requisite "design notebook", common to the practice of engineering design, also provides an opportunity to document the reflective judgment and critical thinking necessary for the design cycle. 3Exemplary practice includes many dimensions, mutual learning between students and community served, and empowerment of communities to meet their own needs. 2In the end, the engineering team must communicate with the end user and blend academic material with implications and issues of social context.
The conceptualization and measurement of student outcomes is critical for program evaluation purposes, i.e. accreditation.From an administrative point of view, outcomes must be defined in terms of the engineering program.Assessment instruments are designed to evaluate whether an engineering design course using service-learning is effective in promoting program outcomes.Self-report instruments are attractive for evaluation of the less technical 'soft skills' stated above and required by ABET.Programs can use Likert scale instruments designed to reflect the course objectives and record student responses electronically. 4Surveys have been developed to get from anecdotal information to tangible, measurable outcomes examining how real-life experiences add value to engineering students' education. 5It is important to note that the survey instrument is a self-report measure that assesses perception of the achievement of an objective, rather than actual achievement.Self-report instruments carry with them the possibility of bias due to factors such as the social desirability of responses.Thus care should be taken in the timing and wording of the instruments.Post-experience surveys and semi-structured facilitated discussions administered during a debriefing process at the end of the semester can complete feedback loops to formulate program improvements. 6echnical content is typically assessed through the traditional student product-"the final report".Skills to demonstrate "an ability to design a system, component, or process to meet desired needs" (3.b) and "an ability to communicate effectively" (3.g) (ABET 2000) are usually easier for the engineering educator to discern.Direct student assessment (the end of semester grade) is issued from a single faculty advisor, a faculty team, or by the assessment coordinator in an administrative division of the university. 7Program assessment can be formalized with the collection of both direct and indirect instruments which supply supplemental insight and information.The assessment process enables programs to identify their strengths, improve their weaknesses and provide ways to measure achievement in soft skills as well as technical knowledge.One cannot disagree that these assessment instruments provide engineering educators with meaningful information for measuring academic performance, evaluating program outcomes and subsequently meeting the ABET criteria.
But that is not what service-learning promotes.Service-learning obligates a look at both the student/faculty /university member's objectives and the outcomes of the community partner.It is precisely in the field of service-learning that solution strategies try to incorporate the world from the perspective of those living it.Service learning is based on the involvement of the community and their determination of their own need, which includes technical, social and economic sustainability.Service-learning projects involving engineering students in design courses often have the tendency to report technical success measured in a narrowly defined block of time, the academic semester or the academic year.Naturally, students need grades and engineering programs must address learning outcomes.However, actual project success must have broader more inclusive assessment criteria.Is the community better able to deal with solving their own problem after the project?Are they able to sustain the benefits?It is not difficult to find a project that was technically successful only to be found abandoned a short time later.These concerns are particularly serious with projects undertaken in the developing world.International service-learning projects have added issues of language, culture and values; idiosyncratic site-specific context driven problems that must be communicated and worked on by the partnership.

Assessment of International Projects
In the past five years international service-learning has blossomed.Student chapters of Engineers without Borders (EWB) and Engineers for a Sustainable World (ESW), and other nonaffiliated student projects have admirably taken on aspects of the UN Millennium Development Goals 8 in an effort to reduce poverty and improve health.More and more engineering students are involved with projects in the developing world.The curricula changes enabling undergraduate engineering education programs to provide field-based opportunities as witnessed in the acceptance of service-learning as an acceptable pedagogy should be applauded.However what is the appropriate assessment of such projects?
Nationally there is a great need in internationalizing undergraduate engineering education.The result is an emergence of different models of engineering study abroad from traditional exchange programs to industrial internships to service-oriented models.Different models mean a variety of different student experiences and hence different student outcomes.In general, educators believe that an international experience remains a valuable component of college education.However, most formal assessment tools in the international education literature focus on student satisfaction, and an attempt to figure out if overseas programs are reliable, costeffective and safely-run. 9or engineering majors, data can be collected that are consistent with student achievement in meeting the ABET EC 2000 criteria for program outcomes related especially to the development of skills mentioned before: the ability to communicate effectively, the ability to function on a team, and an understanding of contemporary issues, ethical responsibilities, and the impact engineering solutions have in a global and societal context.From an institutional perspective, similar self-reporting instruments as used for general service-learning experiences can be used for measuring student perception of achieving program outcomes.
One additional student outcome commonly listed for students participating in an international experience is an increase in intercultural competency. 10An online electronic sampling by the Section on US Students Abroad of the Association of International Education Administrators in conjunction with the Institute of International Education (SECUSSA/IIE) concluded that very few US institutions measure 'intercultural proficiency' as a program outcome. 11Little work has been published on the actual impact to the individual student on ethno-relativism and globalmindedness. 12Small sample size, expensive instruments, qualitative interviews that are time consuming can all be cited as barriers to intercultural assessment.In short, there are few assessment tools available to measure the value of an international experience on student learning.
What about the value of the project to the community partner?After all, the students are agents of social change.One cannot just assess such projects in a pseudo-quantitative method: a survey of student satisfaction or student growth in intercultural awareness.The world cannot be objectified by a mathematical or scientific model, or an elegant engineering solution.Assessment of success must include the perspective of the community recipient.

THE CHALLENGE OF ASSESSING CUSTOMER SATISFACTION
We begin by suggesting three simple questions that any service-learning project must answer truthfully.

Have the customers' needs been met?
If the customer is not satisfied then the project is not successful.
Though this may sound simple, it is not.Human satisfaction is based on an understanding of human dignity.A community partner must be a real partner, an agent of their own development.Customer satisfaction begins by meeting the set of customer requirements that are traditionally translated to a list of engineering specifications.However, these requirements may evolve over time, and the university team needs to revisit the customer requirements with the customer or community partner as the project progresses.The customer must be empowered to decide and define their own satisfaction which may take time.

Is the project sustainable and maintainable by the customer?
If the project is abandoned then the project is not successful.
Follow-up is essential to answer this question.To take ownership of an engineering device or process, the recipient must be involved in the planning, design, implementation, manufacturing and maintenance of it.Ideally a 'project' will gain its own momentum and get disseminated or adopted to neighboring communities.

Does the project respect the environment and make effective use of local renewable resources?
If the project does harm then it is not successful.
It must be affordable and practical; adapted for a particular culture and community.

KEY ELEMENTS FOR AND COMMON OBSTACLES TO SUCCESS
From our own experiences and insights and reflections of multiple international service-learning programs from phone interviews and from published "lessons learned", several themes are discussed as key elements for and common obstacles to success. 13,14,15,16,17

Beginning a relationship
It is important to create a situation in which a relationship can flourish.An important element of success is the cultural preparation and education of the faculty.He or she must show respect for the host community by knowing something about their lives and their country before arrival.Knowledge about the local culture is critical to beginning a fruitful rapport.People live in complex societies with different histories, cultures, social structures and traditions.It is important to spend meaningful time getting to know the people, especially getting to know what is considered respectful and what is considered rude.A reconnaissance or pre-site visit is essential.The most important feature of beginning a relationship is mutual respect.It is important to seek to understand the problems the people face, and be humble that you will not really understand all the complexities of the problem.A potential collaboration must originate from the community, thus it is imperative for western academic groups to listen, really listen first.
An American academic team can easily undermine current social structures if they enter a community from the outside without proper introductions.Neglecting the community leaders or not engaging in participatory discussions with end users may irreversibly strain a relationship and thwart community acceptance.Engineering student groups may not understand collective concepts, complex social roles and completely miss cultural formalities.American pride, exaggerated self-centeredness, presumption, vainglory and hypocrisy must be tempered with humility, the readiness to learn from and listen to the community.Finally, not having good local language support will make communication difficult if not exasperating.

Choosing a project
A project should be worth doing, and the expertise of the academy must be significant.Innovative projects that have a genuine discovery component may be able to solve a real problem.A project begins with a motivated community partner with a good idea.The community partner should include a competent organization with value added expertise (ex. a successful indigenous non-governmental organization (NGO)).At the very least this organization could help address culture and traditional wisdom.
There are numerous obstacles to success in choosing the right project.US groups may not have the expertise we think we have and even our best efforts only create new problems.The target community may not understand their own constraints and have false expectations.The community partner interface may be more complex than local service learning projects, and obstacles may appear from unexpected places.A new device may replace a lucrative existing industry or encroach on an official (governmental) territory, or interfere with the turf of a competing NGO.Academic groups should resist performing experiments that serve their academic interests and not the community's interest.Finally sites of greatest need may have safety issues.

Relationship with on-site partner
There are variations in trust across people and groups.Many societies have a strong insider/outsider mentality.Members develop trust in one another through social interactions in family, village or ethic group encounters.Trust must be earned by outsiders.Relationships must be built on real solidarity between the stakeholders.Partners ought to be treated as equals to walk with and learn from-not tragic objects to be rescued.Western academic groups have to be prepared to learn from their community partners: learn a foreign language, learn cultural aspects of non-western countries and learn to listen.The on-site partner is vital to success.They need to be prepared to take ownership of the project.An important element of success is a vested partner who is motivated, articulate and can affect change.
There is great potential for combining indigenous knowledge with contemporary science to address real problems; however it is sometimes difficult to build self-confidence in partner organizations or communities.In many resource poor locations local experts may want to imitate or copy us instead of believing in their own technologies.North-Americans must be cognizant of their can-do-it attitude which can easily be read as arrogance and their obsession with time which can lead to impatience or anger.Host organizations on the ground may be very busy and may not communicate at our level of frequency.Finally avarice and greed of an on-site recipient can quickly sour a relationship.

Executing the project
Successful projects have affordable, economical solutions that can be manufactured and replicated by local people.An on-site logistic team or contact must be able to locate sources and materials in the host-country.Otherwise the project can not hope to be maintained or sustained.Projects have multiple stakeholders.From the university perspective, teams will need cultural preparation, multiple trips, language support and a logistics team.Students must be carefully selected and be prepared to learn more than just design and the scientific process.Successful projects are interdisciplinary as well as multi-cultural.Every effort should be made to engage with local engineers, local professors, entrepreneurs and vested individuals.We need to realize that our role is technical assistance to facilitate our partners' success.On-site work must be structured and set up with definite goals to make sure all tasks are covered.Extra time should be scheduled to anticipate unexpected delays.It is important to set up realistic goals and realistic time frames.Finally groups of academic faculty and students should be maintenance free to host organizations.
Traveling with students may present new and different issues and obstacles.Students may have false expectations, become impatient, may be immature or not prepared for poverty and difficult conditions.Team members may get sick or desire to talk only to each other, eat familiar food, and disengage from the cultural experience.Academic groups must be aware of our artificial restrictions of time and be prepared to re-direct their efforts and energy when things go differently than planned.Information may not be accurate, drastically changing the on-site focus.The desire to build something too quickly or not having all the appropriate expertise required may undermine a project's safety or cultural appropriateness.North Americans need to be humble and not presume we understand our partner's constraints and be patient and not have an overly aggressive time-scale.
One-of-a-kind designs or solutions that cannot be replicated easily will not contribute to customer satisfaction.Hodge-podge efforts that lack standardization may generate community frustration or disappointment.Not planning for expensive shipping costs of equipment or oversize items or custom fees are additional obstacles to project success.Important items checked in luggage may get lost en route.Students may overlook the need for effective appropriate training manuals in a local language and not budget for appropriate project documentation.Finally, local officials, engineers and academics may not be interested in participating or contributing to the project or require financial incentives.

Long-term involvement
Successful projects have successive phases and academic groups should commit to long-term relationships.In setting up realistic goals, institutional continuity must be thought through.Multi-class teams where upper-class students mentor lower-class students, or setting up students as first a 'support team' and then a 'field team' may be good long term solutions.Consciously having one student who has been there before may provide some type of institutional memory.Academic groups must have the patience for completion.Projects must go through the design cycle, which includes testing and re-design, testing and scale up.Faculty have to be prepared to work together with the community and connect to people.Learning more about adult education techniques would be helpful.Engineering faculty and students ought to try to understand the causes behind economic and power inequities of their host country and invest time into learning about the causes of poverty.Finally follow-up must occur over several years.Challenges to longterm adoption should be honestly addressed.
Long term involvement is expensive and a project needs more expertise than just engineering.Obtaining funds and buy-in from our academic administration may lead to project burn-out.On-site partners may also get impatient with excessive local bureaucracy and lose interest in the collaboration.Project adoption may take a much longer period of time than anticipated due to illiteracy or language barriers.Information may not be passed effectively from team to team or from the team to the partner community.Long term commitment is difficult to sustain.The technology may be more difficult to manufacture and disseminate than originally assumed or projects parts may be too expensive to obtain or reproduce.

Defining success
The project must have the target population view it as a success.Academic groups need to realize that our success is secondary.Success can be defined in stages and critical self-evaluation is necessary.Ultimate success is when everybody is satisfied and the project has a momentum of its own and is being replicated and adopted on its own accord.
To make sustainable change one must understand how to contribute to social change: Engineers don't have this training!One could argue that we don't really know what a sustainable design is and we have unrealistic goals.Many recent publications address the dilemma that despite a history of high levels of monetary assistance for development, the gap between the providing of appropriate technology to alleviate poverty and the adoption of the technologies is still large. 18,19

PERSONAL EXPERIENCES
The University of St. Thomas (UST) has engaged three student groups (French, engineering and communication studies) over four consecutive years in interdisciplinary projects in the developing world; the West Indies in 2004 and West Africa in 2005-2007.Our projects have been fascinating learning experiences for 35 students and 8 faculty.From an academic viewpoint, the students completed their objectives and have been exposed to new cultures and new perspectives, and the professors have been able to show that the outcomes of their respective programs have been achieved.The students have been surveyed, reflected on-site in their daily trip logs, and completed top quality reports, videos and papers. 20It is too early, however, to say that any of the projects were "successful".It is instructional to examine one of these projects honestly, especially it terms of its unique-yet universal challenges.

Beginning a relationship
In the first project, UST was asked to assist a local St.Paul based non-profit organization Compatible Technologies Inc. (CTI) to harvest breadfruit in Haiti.Members of CTI have close ties to the Committee on Development (COD) of the Methodist Church in Haiti.The request to stop breadfruit spoilage and find useful breadfruit recipes came from the Haitian director of COD, thus the project had strong support of an on-the-ground indigenous non-profit organization.Breadfruit grows uncultivated all over Haiti, but it spoils within 48 hours of harvest.It is estimated that up to 60% of the breadfruit in Haiti spoils.Currently, Haitians boil or roast fresh breadfruit.However, dried breadfruit has a shelf life of about one year without any added preservatives.Dried and ground breadfruit has a surprisingly similar texture to flour and it was postulated that it could be used in recipes as a flour substitute.CTI is a non-profit organization whose mission is "to improve the lives of people in developing countries by designing food and water technologies that are sustainable and appropriate to local cultures, and by collaborating with in-country organizations to identify needs and to achieve widespread use of our technologies to relieve hunger and poverty." 21Many of their volunteers have ties to Minnesota's large food industry (Cargill, General Mills, Pillsbury) and several volunteer food scientists had come up with over 20 recipes for dried breadfruit.Both COD and CTI wanted to introduce breadfruit drying to 13 identified female cooperatives.The women in these communities were eager to dry breadfruit but stated that it would be difficult to dry even small quantities (<25 lbs).CTI believed the cooperatives needed solar-assisted dryers that could dry up to 100 lbs.The school of engineering at UST was asked to assist in designing a low-cost solar drier.
Taking the essential reconnaissance trip, three UST faculty determined that language and communication would both be difficult; culturally appropriate training manuals for a largely illiterate end user would be necessary, and that shredding the breadfruit into consistent pieces was an important part of the drying process.Three disciplines partnered to tackle the problem.Two engineering teams, enrolled in their senior design capstone, participated in the project.One team worked on designing a manual shredder, and the other team worked on designing a solar drier.A team of French students, enrolled in an upper level language class which incorporated both Haitian history and the Creole dialect, planned to act as cultural and language liaisons and worked on developing picture based culturally appropriate training manuals.Two independent study communication students planned to document the endeavor.

Choosing a project
Sites of greatest need may have safety issues.Problems arose when six weeks before departing; Haiti was closed to all non-essential Americans because of civil unrest.To continue the project with the necessary on-site prototype testing, the teams traveled to St. Vincent in the West Indies where UST had a personal contact.Here we found a polite population, but one not really needing or wanting a breadfruit harvesting strategy.The French students didn't really have the same raison d'être as before and the solar drying team discovered that the Caribbean sun and breeze was more than adequate to dry the shreds.Thus CTI's request to design a 'solar dryer' for breadfruit actually turned out to be a red herring.As long as the shreds were the appropriate size they dried quite nicely in the local conditions.The key it seems was a manual shredder; neither what CTI nor COD thought they needed.

Relationship with on-site partner
The Vincentians told the students that their shredders were easy to use and they "would certainly use them".After the design was finalized using input from the Vincentians, UST engineering made and delivered five shredders to St. Vincent for what we hoped would be an extended field test.Six months after delivery we discovered that no one had actually used them, but were told that that they would begin to use them soon.After two-years, a follow-up visit confirmed that the five shredders were never used and there was no extended field data.We delivered a product to a location that didn't really need it and never really asked for it.The idea after all came from Haiti.

Executing the project
Getting the shredder design to the COD in Haiti presented an entirely different set of challenges.After seeing our successful Vincentian prototype, CTI volunteers became impatient with the academic calendar.The student team had worked on their design for over eight months and was in the process of finalizing their product.They had worked through many iterations of the design.Two CTI volunteers were traveling to Haiti despite the state department warnings and decided to build their own manual breadfruit shredder based on seeing the student design at a mid-term presentation.They did not ask the students for their drawings or interim reports.The CTI model did not work as well as the student shredder and was consequently not well received in Haiti.

Long-term involvement
Even though the final UST shredder functioned well, the cost was still too high and in the opinion of the engineering faculty, the design needed more fabrication simplification.To be considered successful by Haitian standards, the project would need further iteration, this time focusing on manufacturability.One challenge that faculty face is how to repackage the previous years design challenges both for new students and departmental committees.Most projects do take more than one team to complete.Is it the responsibility of the faculty to keep the project going?One set of students needs to pass on their collective knowledge, but it is not always possible to set up a new team in the same academic structure.In this case, a graduate student majoring in manufacturing tackled the job of making the breadfruit shredder significantly less expensive to manufacture in Haiti.The student waited for 18 months for the travel restrictions to travel to Haiti to be lifted.It is absolutely necessary to test the shredder and the tools needed to build it with the recipient community.The travel restrictions were never lifted and the student never traveled to Haiti to test his re-design.A University cannot sanction travel to location where the State department has a travel warning.
This experience suggests that international service-learning sites cannot be chosen on need alone.The site has to be safe enough to take students.A situation must exist for the possibility of a long term stable relationship.From the University's perspective, the safety and health of the students cannot be underestimated.One injured student could mean the end of a faculty career or department effort.International education offices, who usually manage student travel, were also not able to help with much of the travel details.Much of the developing world does not have traditional service providers.The effort to plan and execute this project has been very time consuming.Who does the logistics work?Who continues the follow through?In this project, all the work was done by the faculty.How can we prevent faculty burn-out?
The original student team would have greatly benefited from taking two trips, one to determine the customer needs, one for prototype testing.The follow-up student was never able to travel.The COD believed it wanted a solar fruit dryer, but in fact it was not necessary.There were additional challenges for the faculty.No one person can act as advisor on every project.What started as a request for a heat transfer specialization (solar dryer based on natural convection) ended up as a mechanical device needing manufacturing expertise.

Defining success
It is our belief that international service-learning projects must look to establish broader collaborations.With NGO's, maintain frequent communications and make sure you are on same page, especially given the length of time of academic efforts and the constraints of an academic calendar.We also believe that a working relationship with the university and students in the host country would be beneficial for all involved.We were unfortunately not able to do this in Haiti, despite efforts by the UST faculty.