A Systemic and Systematic Assessment Methodology for Technology-Based Social Enterprises

Academic programs and ventures that focus on appropriate technologies for socio-economically marginalized populations in developing countries are on the rise. However, standardized methodologies to assess the success of the ventures and their economic, social and environmental impact are almost non-existent. Current assessment strategies focus on tangible outputs such as customers reached and revenues generated, which creates a disconnect between an organization’s mission and the impact created by the products and services provided. An emphasis shift from outputs to outcomes requires a move away from the solely growth-based measures to include mission-based measures. This paper presents a systemic and systematic assessment methodology for comprehensive and efficient assessment of early-stage social enterprises. Originally applied to an academic social venture, this methodology has since been generalized to be applicable to a variety of social enterprises. The methodology employs a hierarchical framework to assess metrics at various levels of the venture while tracing them back to the overarching objectives of the venture. Current assessment practices used by nonprofits, social enterprises and academic ventures are discussed in detail to establish the need for systemic and systematic frameworks. The principles and mechanics of our assessment framework and its operationalization are presented with a culminating case study of the application of this methodology to a telemedicine venture in East Africa over the last three years. Index Terms Assessment, Evaluation, Systems Thinking, Lifecycle


INTRODUCTION
Academic institutions have recently initiated a surge in educational programs centered on humanitarian engineering and social entrepreneurship 1 .Often these programs lead to the creation of technology-based social enterprises designed to address global challenges like healthcare access, food security, and information availability.However, the existing literature fails to provide a methodology to assess the social impact of academic ventures.To fill this gap, researchers at the Pennsylvania State University developed a structured assessment methodology that serves to identify and aggregate relevant data to systemically and systematically assess the value created by social ventures throughout their lifecycle.Using a hierarchical framework, the methodology seeks to assess metrics at various levels of the venture while tracing these metrics back to the overarching objectives of the venture.The goal of the methodology is to produce rigorous metrics of success for the venture.
Throughout the course of this discussion, we will define the criteria used by this methodology to assess impact.It first and foremost assesses the value generation and sustainability of the venture as opposed to student learning.To provide a base knowledge of current practices, this paper first explores the assessment processes, including practices used by nonprofits, for-profit organizations, social enterprises, and other academic ventures that influenced the development of this assessment tool.Key differences, best practices, and the pros and cons of these assessment mechanisms are highlighted.Through reviewing the strengths and weaknesses of alternative assessment practices, we will demonstrate the benefits of the new approach.This will be followed by an explanation of the conceptual framework on which the assessment methodology is based.A case study of the operationalization of the methodology for a technology-based social enterprise is then presented.The final section provides a brief review of the strengths and limitations of our approach and plans for future research.

REVIEW OF CURRENT ASSESSMENT PRACTICES
In the United States alone, there are over 1.5 million registered nonprofit organizations 2 .Nonprofits play a significant role in the economy, while simultaneously providing essential social services.In 2009, nonprofit organizations accounted for 9% of all wages and salaries, and more than a quarter of Americans volunteered for a nonprofit in 2010 2 .In addition to seeing the number of charitable organizations triple over the last two decades, the United States has also experienced a significant rise in social enterprises 3 .Social enterprises can take many organizational and legal forms, including nonprofit, for profit, non-governmental organizations, and community organizations.All social enterprises, however, share the common purpose of finding sustainable solutions to social problems 4,5 .Despite the rapid growth of nonprofits and social enterprises, little has been done to understand and measure their social impact 6,7 .
In a for-profit setting, it is typical to define success through profits, income, and investments.Social ventures need to assess not just economic metrics but also metrics that ensure fulfillment of their social mission 8. Assessment information can guide internal processes, describe outcomes, translate impacts to financial figures, improve programs, identify new challenges and opportunities, and communicate the value of a program 9 .However, evaluation is rare across the spectrum of organizations focused on creating social value.Only a few hundred of the over 60,000 foundations across the United States regularly conduct formal evaluations 10,11 .Challenges and obstacles include organizational capacity, monetary limitations, and resource constraints.Assessment can be a difficult and imprecise process, and the lack of standardization in measuring value creation serves as a major barrier 9 .Entrepreneurs also face assessment challenges, as profit margins and economic gains alone do not provide a holistic view of their success.The Organization for Economic Cooperation and Development cites a serious lack of measures of the value of entrepreneurship in existing research, which inhibits the understanding of where and why entrepreneurship is thriving 12,13 .For socially-focused entrepreneurs and nonprofit organizations, finding ways to measure success is a particularly difficult task.
While philanthropic assessment is scarce, it is becoming more common as funding opportunities grow increasingly competitive, making the measurement of program success critical.General constraints to the development and implementation of assessment frameworks include limited funding, decentralized organization and operation, working in a resourceconstrained environment which is often accompanied by cultural constraints, and multi-year venture lifecycles.In addition to these shared constraints, social ventures stemming from academia are further challenged due to variable student dedication, adherence to Institutional Review Board (IRB) protocols, which is mandatory for data collection involving human subjects, and long-term institutional support.
Techniques and rigor vary greatly among evaluations of nonprofit, social, and academic ventures.Many common assessment methods focus on outputs rather than outcomes.Outputs are specific, often tangible deliverables such as number of people reached, treadle pumps sold, or workshops conducted.On the other hand, outcomes refer to the value created for the outputs' beneficiaries, like improved access to healthcare, financial literacy, and food security 14 .Thorough assessment methodologies that associate performance measures with the missions and goals of individual institutions have emerged, but they are open-ended and lack standardization.A closer look at the assessment practices of nonprofits, social enterprises, and academic ventures offers key insights into their strengths and weaknesses.

Nonprofit organizations
A nonprofit organization is a charity or service organization motivated by social impact as opposed to financial gain.Nonprofits use donated funds and generate revenues to replace, complement, or supplement social services provided by governmental agencies.A study completed in 2008 showed that in the past decade, both funders and beneficiaries have increasingly relied on nonprofits to provide an array of services, from child care and job training to community development and health services 15 .Governmental funders and large foundations that provide funding to nonprofits often require them to report performance measures; however, a standardized method for collecting this information does not exist 16 .A review of the assessment practices employed by these groups shows a range of techniques for data collection including agency record information, client surveys, knowledge tests and trained observer ratings 17 .In addition to tangible outputs, there is evidence that organizations should seek out strategies for assessing long-term social impact through outcome assessment 18 .This strong focus on outcome measurement is emphasized in a report from the Urban Institute that conducted a survey of 36 nonprofit organizations, 83% of which collected some form of outcome-related results 17 .Despite a majority of the nonprofits conducting some form of outcome assessment, this study suggested a lack of training in analyzing and using results across the nonprofits, thus undermining the impact of the assessment.This can be attributed, in part to the resource constrained environment and limited funding available for performing assessment.Some recommendations for improvements included more sophisticated data collection techniques and more informative data analysis in order to pinpoint specific areas of improvement.In addition, this type of analysis provides little insight into the success of achieving the organization's mission.
An organization's mission matters.Further, mission-organization alignment, mission satisfaction, and commitment have a positive impact on the performance of the organization 19 .Consequently, a high value is placed on a well thought-out mission statement.Interviews with the leaders of 30 nonprofits, such as the American Red Cross, YMCA, CARE USA, and Teach for America, reveal a growing emphasis on assessing mission impact.However, while many nonprofits are working to develop and implement effective measurements of mission impact measures, few have been able to achieve such a task 20 .To develop an overall schema for assessment, many organizations draw on published lessons from the private sector, including areas such as strategic planning, marketing and finance, and structural organization, yet these tools are not always ideal for nonprofit organizations 21 .A common downfall in the evaluation process of many organizations is collecting too little or too much data.Just as only measuring donation dollars lacks detailed data on mission success, too many metrics of evaluation can bog down analysis, making it difficult to specifically define the important indicators for success and failure.This also over-utilizes an organization's already limited resources.It is important for an organization to narrow down the number of critical metrics by clearly defining the links between its mission, vision, goals, strategies, programs, and measurements 11 .
Another form of assessment called the "Family Measures Model" measures impact, activity, and capacity.Impact considers the overall mission success.Activity measurements focus on meeting shorter-term goals, and implementing more specific strategies.Whereas capacity measurements assess how well the organization is utilizing its resources to fulfill their larger objectives.This framework encourages organizations to focus on the highest priorities, which can be measured across an organization's operational units, and also on regional and national levels.Metrics must quantify progress towards achieving a mission while staying within mandates imposed on the organization whether by government or funders 22 .This assessment method, however, lacks a clear methodology that can be easily utilized, as each organization is expected to use the basic framework and come up with short-term measurable goals and assessment strategies independently.
Potter and Brough argue for addressing systematic capacity building through the use of a "capacity pyramid" as a means of assessment.This approach suggests that capacity building can be achieved through the use of a four-tiered hierarchy of capacity building needs: structures, systems and roles, staff and facilities, skills, and tools.At each of these levels, nine components for systemic capacity building are evaluated.This complex, yet thorough, method makes it easier to detect sectoral shortcomings, improve project monitoring, and ensure effective use of available resources.The systemic nature of the assessment adds rigor to the evaluation process and can therefore provide better problem diagnosis and more effective program design and evaluation.This assessment tool has the ability to show organizations that focusing on less tangible activities, such as staff training, is as important as tangible items like infrastructure and tools 23 .A downside to this method is its complex nature.This system entails assessing multiple organizational levels and components, resulting in a complex web of evaluation criteria, which is time and resource-intensive.
In review, when it comes to assessment, nonprofits are doing certain things well, but there are several areas in need of improvement.Strengths of assessment in the nonprofit sector include (1) emphasis on assessing mission impact and (2) drawing on lessons from the private sector.Several weaknesses include (1) emphasis on tangible outputs during assessment over long-term outcomes, (2) lack of alignment between measured data and mission objectives, (3) collection of too little or too much data, (4) incorporation of complex and resource-intensive methodologies, (5) utilization of assessment models that lack a structured methodology, and (6) lack of standardization in assessment methodologies and metrics that can be used across the nonprofit sector.

Social Enterprises
Social enterprises vary in size and organization and can be run by sole proprietors, for-profit organizations, non-governmental organizations, youth groups, community-based organizations or other entities 24 .Their focus on innovation and rapid growth differentiates them from small businesses 25 , and their primary objective is to provide a social service (without reliance on charity) that solves problems.These ventures may generate a profit, but their mission focuses on social sustainability 24 .Social enterprises pose unique assessment challenges; they are often a cross between nonprofit and for-profit businesses, with both social impact and financial growth being equally important 26 .
Several qualitative tools have been developed for assessing social impact including the Measuring Effectiveness program 27 , the Balanced Scorecard 28 , the Triple Bottom Line 29 , and Social Reporting 30 .Although these qualitative metrics have been shown to be successful in evaluating mission impact, quantitative figures are needed to attract donors and serve as benchmarks.Quantitative tools like Social Return on Investment 31 and Blended Value 32 provide tangible data for assessment but lack rich qualitative narratives that can indicate success.A framework that addresses both qualitative and quantitative metrics is critical to provide a more comprehensive picture of a venture's mission impact and overarching success.A deeper look into two research studies provides insight into the effectiveness and shortcomings of current practices.
The Foundation Strategy Group for The Skoll Foundation interviewed two dozen investors, social entrepreneurs, and scholars to gather information on evaluation in the field of entrepreneurship 26 .Notable interviewees included Susan M. Davis (Chair of Grameen Foundation USA), Sushmita Ghosh (President of Ashoka) and Jacqueline Novogratz (Founder & CEO, Acumen Fund).The study suggested that intense scrutiny, planning, and reviews conducted 6-12 months before project funding is distributed to venture leaders facilitates the quick and efficient assessment of short-term goals while providing the impetus for tracking longterm goals, growth, and scale of impact.Post-grant evaluation is then more focused on specific goals set by the funders and the organization that will maximize value creation.Evaluation in both stages (pre-and post-grant) is tailored to individual ventures 26 .
The Ashoka Fellowship seeks to invest in, and support, leading social entrepreneurs.In order to measure and better understand the progress that these social entrepreneurs are making towards systemic social change, Ashoka developed the Measuring Effectiveness program.With a diverse range of fields represented in the work of these fellows, they together developed indicators that can be compared across the divergent fields of work.These indicators include looking at how often a fellow's work has been replicated or level of influence that their work has over public policy.This evaluation tool tracks progress in both the short and long term, providing one solution to the challenge of measuring the success of a diverse group of social entrepreneurs working towards social change over time.However, some noted challenges with this type of assessment include translation discrepancies, self-reported results, and irregularities in implementation over time due to the open-ended nature of the assessment 27 .
As with nonprofits, strengths exist in the realm of social entrepreneurship assessment, yet there are several weaknesses to note.Strengths include (1) proven method to evaluate mission impact using qualitative metrics, (2) competitive funding process that necessitates assessment of short-term goals to track early venture success, and (3) identified set of indicators that can be compared across divergent fields of work.Despite these strengths many weaknesses remain, such as: (1) prioritization of economic gain and profit margins over social impact, (2) lack of a standardized method to measure social impact, (3) inability to assess both quantitative and qualitative metrics, (4) post-funding assessment goals that are often solely determined by funders, (5) individually-tailored assessment resulting in lost time and the inability to compare projects across divergent fields, (6) difficulty in tracking long-term goals, (7) assessment across cultures and languages, which results in translation discrepancies, (8) self-reported and often open-ended results, and (9) focus on obtaining funding rather than measuring social impact.

Considerations in assessing an Academic Social Venture
At institutions across the United States, there are growing opportunities for learning through service in both the academic and co-curricular settings.Many of these programs strive to engage students through appropriate technology ventures.Courses and immersive experiences related to social enterprise offer both a compelling educational experience and an opportunity to address problems in marginalized communities.For example, the Engineers without Borders student club has grown from just eight engineering students attempting to develop solutions in the developing world to over 12,000 volunteers and 250 active student chapters across the United States 33 .Similarly, Engineers for a Sustainable World (ESW), another student club that develops innovative solutions to address global challenges sustainably, has over 24 chapters at institutions across the country 34 .Despite the growth that these programs are seeing, the resulting sustainable impact does not necessarily match the intended vision.EWB notes in their strategic plan there is a "lack of long term implications and ability to measure program effectiveness 35 ".In addition, EWB notes that there is a disconnect regarding universities' knowledge of the meaning and implementation of international community development, causing a limited understanding of the need for assessment 35 .Currently, the learning assessment models solely examine how students benefitted from such immersive experiences.The assessment methodology proposed by the authors of this paper seeks to measure venture impact as opposed to student-learning outcomes.
As impact assessment is currently not prioritized by leaders of academic social ventures, it may be surprising to include them amongst entities such as nonprofits, for which impact assessment is becoming commonplace.However, most social ventures (academic-based, nonprofits or for profits) often operate in the informal economy prior to becoming an official entity.This is particularly true for ventures operating in developing countries, where as much as 75% of non-agricultural employment operates in the informal economy 36 .Thus, assessment can be of great value to academic social ventures.It can guide the design, business strategy and implementation strategy for the venture.It is the intent of this paper to help fill the gap in literature addressing venture assessment in the academic realm.By highlighting the social value and impact on stakeholders, students increase their chances of obtaining funding for venture growth.Publishing assessment outcomes increases visibility of the venture and allows other ventures to learn from mistakes, giving opportunities for greater success in the academic arena.
When developing an assessment methodology for an academic social venture, there are several constraints to take into consideration including the unique challenges of resourceconstrained environments, dynamics of working in a developing country, limited funding, large number of students working on the venture, decentralized teams, a multi-year implementation timeline, and adherence to guidelines set by regulatory organizations like the University's Institutional Review Board (IRB).The IRB operates independently within academic institutions to review, approve and monitor all biomedical and behavioral research involving human subjects.The research strictures imposed by the IRB are more conducive to experiment-based research carried out in controlled settings rather than the chaotic, dynamic environment inherent to social enterprises 37 .Further complications arise when venture teams engage in entrepreneurship and research at the same time 38 .

SYNTHESIS OF ASSESSMENT CHALLENGES
The assessment framework presented in this paper seeks to address the weaknesses in assessment methodologies used by both nonprofits and social enterprises, along with the noted challenges in assessment of academic social ventures.The framework will be supported by examples from current social and academic ventures along with a thorough case study of how this methodology was implemented by the Mashavu telemedicine venture.To start, the following eleven weaknesses emerge as significant limiting factors to successful assessment: 1. Lack of standard assessment practices 2. Lack of continuous engagement for student-centered ventures 3. Inability to prioritize metrics and determine how many, and which, metrics to track 4. Disconnect between measured metrics and mission objectives 5. Assessment practices draining already-limited resources (time, people, money) 6. Working across cultures 7. Lack of integration of quantitative and qualitative metrics 8. Lack of coherency between short and long-term goals 9. Focus on outputs rather than outcomes 10.Challenges in data collection and information reporting practices 11.Difficulties in adherence to IRB protocols It is important to address these weaknesses to develop a complete and thorough assessment methodology.The assessment methodology presented in this paper strives to address these weaknesses in a systemic and systematic way.Systemic assessment aligns mission with outputs and outcomes, focuses less on end goals, is dynamic, and incorporates feedback throughout venture development.To ensure harmonious operation, a systemic assessment incorporates feedback from the environment in which the system exists, such as the culture and context.
Weaknesses 1-6 suggest that current assessment methodologies are not systemic because they are not effectively considering the relationships between the system and its components.Systematic assessment refers to a comprehensive, structured and methodical process that displays a deeper understanding of the entire system.A systematic approach gives us an in-depth assessment of each component of the system and provides an opportunity to refine the details, uncover a deep understanding of how the system works and reveal modifications that could significantly impact the overall system 39 .Weaknesses 7-11 indicate a need for systematic assessment.
For example, consider an academic venture focused on developing low-cost LED lanterns to sell in communities with limited access to electricity.Since a standardized method for assessment does not exist, the venture develops its own metrics for assessment.The team is comprised of students from several disciplines, many of whom are new to the project.The new team develops an assessment from scratch with little insight on successes and failures experienced by the previous team.One metric that they choose to assess is the number of lanterns sold.While the number of lanterns sold is an important metric in the short-term, it provides little insight into the overall impact of the venture on people's lives and inefficiently utilizes valuable resources.
The team also chooses to assess its customers' opinions of the LED lanterns and the impact of the lanterns on their lives.However, language barriers prevent the team from accurately capturing qualitative feedback.Furthermore, due to culture and custom, customers make an effort to shield the team from insult and give only positive feedback of the product, thus skewing the data.Language and culture are not the only challenges in data collection.Adherence to IRB protocols is essential to the academic venture to ensure that their work and lessons learned can be shared with others.The time-consuming process of getting the IRB protocols approved leads the team to simplify the protocol, resulting in essential metrics not being identified.If this team had utilized both systemic and systematic assessment, they could have avoided many of these challenges and missteps.
A systemic assessment helps a venture focus on its mission.It ensures an understanding of how the end goal of selling lanterns relates back to the mission of providing resourceconstrained communities with access to light.It would have been beneficial for the group to focus on the social impact created by the lanterns rather than the quantity of lanterns sold.Systemic assessment would allow the team to identify the canonical data set, the simplest and most significant set of data, which is necessary to assess mission impact while ensuring that the team utilizes limited resources most efficiently.By surveying qualitative metrics the team could have discovered important relationships between access to light and improved social and economic conditions in the community.For example, access to lanterns could have improved student performance because it enabled them to study after dark.Student performance might have emerged as the proxy outcome to be measured.
After looking at the big picture through systemic assessment, it is important to look at the assessment systematically.Systematic assessment ensures a deep understanding of local culture so that culturally appropriate metrics and probes are employed and the studies are conducted in a methodical manner.For instance, to understand how the LED lantern has impacted quality of life, students from a local school could be asked to volunteer their report books from both before and after the LED lanterns were introduced to that child's home.At times, it may be necessary to employ translators or community members who will elicit an honest response from the customer.
After understanding the culture and how to obtain critical data points, the systematic assessment will help the team to identify short and long-term goals.In essence, the convergence of systemic and systematic assessment allows the team to comprehensively assess their venture by identifying and measuring the canonical set of data while optimizing the use of scarce resources.Towards this end, our team has developed and field-tested an assessment framework along with a step-by-step procedure, or lifecycle, for implementing this assessment geared for use by earlystage social enterprises.

ASSESSMENT FRAMEWORK
Our framework integrates existing evaluation techniques with our own insights to evaluate a venture with multiple stakeholders and diverse contextual challenges, while simultaneously considering the economic, social and environmental impacts.We posit that a systemic and systematic assessment framework should have the following characteristics: 1. Method for identifying the critical canonical metrics.2. Quantitative and qualitative feedback mechanisms.3. Hierarchical structure that allows for the assessment of both short and long term goals for diverse stakeholders.4. Application across divergent fields of work. 5. Alignment of venture mission to the assessment metrics and methods.
Combining these five characteristics into an assessment methodology can help overcome the challenges experienced by academic ventures.Ventures with limited funding and resources must identify the critical metrics that need evaluation in order to avoid collecting frivolous data.Utilizing both quantitative and qualitative feedback leads to comprehensive program assessment.Working in the dynamic environment of a developing country may necessitate last minute changes to protocols and implementation strategies, thus requiring multiple methods of data collection.In an academic venture, students come from a variety of backgrounds and may work on a venture anywhere from one semester to several years; an assessment structure that measures both short and long term goals gives students a (short term) semester goal to work towards while overarching (long term) goals enables the venture to achieve its mission.Academic ventures may span various sectors, from healthcare to agriculture to education and hence a framework that can be applied to divergent fields of work is needed.Finally, in order to ensure continued assessment throughout venture growth, the framework should be adaptable, scalable and closed-loop.

A Systemic and Systematic Process
For the systemic assessment, a framework is utilized for identifying the components and their hierarchy within the venture.These hierarchical levels include: macro-value created by the enterprise, users, network, product or service as a system, and product or service sub-systems (Figure 1).Using this hierarchical framework, a venture can be assessed holistically, examining the interactions between existing parts and identifying metrics at each level of the venture.For example, an abstracted assessment matrix illustrates interactions between the various levels of a venture (Figure 2).

FIGURE 1 HIERARCHICAL FRAMEWORK
The Hierarchical Framework: 1. Value: What is the overall social impact of the venture?2. Users: Who benefits from the system and what do they gain from various parts of the venture?3. Network: What is the effect of enterprises/products/services when the venture is scaled across a region?4. Product or Service as a system: How does the product/service operate as a system, and how does it interact with the sub-systems and the contextual systems of interest? 5. Sub-systems: How well do the sub-systems of the product/service operate?What are their interconnections?

FIGURE 2 ASSESSMENT MATRIX TO UNCOVER INTERACTIONS BETWEEN EACH LEVEL OF THE VENTURE, THE AXES ARE A TEMPLATE WITH HYPOTHETICAL CONTENT WITHIN THE MATRIX
This model can be used as a framework for assessing many different types of ventures, from technical products to telemedicine networks to microfinance organizations.In order to better understand how this framework can be applied to ventures from different sectors with varying objectives and implementation strategies, this framework is applied to a technical venture (KickStart), a financial organization (Grameen Bank), a health care organization (Aravind Eye Clinic), and an energy venture centered on LED lanterns that emerged from an academic ecosystem 40 (Table 1).This table should be viewed as a source of examples; it conveys how the hierarchal framework can be applied to different forms of social ventures.The value, users, network, systems, and sub-systems are identified for each of the four types of organizations.Each organization is unique, and therefore their framework details will also be unique.Not all organizations will have a network, or sub-systems for instance.Others may have several objectives that add to the organizations value, while others may have only one or two.The ultimate purpose of this table is to show how any organization can fit into this hierarchical framework and can use the systematic assessment lifecycle detailed in the next section.Once the hierarchical framework for the venture is determined, entrepreneurs can progress through the systemic assessment lifecycle, detailed in Figure 3.

FIGURE 3 LIFECYCLE OF THE SYSTEMATIC ASSESSMENT PROCESS
The assessment lifecycle: Step 1: Define Overarching Objectives: Determine the 3-4 overarching objectives of the venture.
Step 2: Identify Levels of Abstraction based on the Hierarchical Framework: Determine the various levels of abstraction for the venture.The levels of abstraction should follow a hierarchical order, with increasing specificity as you move down the levels.Depending on the complexity of the venture, there might be one or more sub-levels under the "Product/Service as a System".
Step 3: Organize Teams: Identify the sub-teams responsible for operationalizing and assessing the various aspects and levels of the venture.
Step 4: Develop Exhaustive Metrics List: Using their expertise on a specific aspect of the venture, each team should utilize the design matrix (Figure 2) to conduct a thorough investigation of potential quantitative and qualitative metrics for each level of abstraction and the level's interactions with other levels.
Step 5: Determine Critical Canonical Metrics: Teams should identify the critical metrics needed for assessing the success of the venture's objectives.
Step 6: Develop IRB Protocols: Develop data collection instruments.For academic social venture, the IRB will have to be consulted for protocol approval.
Step 7: Perform Data Collection & Analysis: Collect and analyze data for metric success.
Step 8: Report Critical Metrics: Report critical metrics that are well-aligned with the overarching venture goals.
The next section presents a case study on how this assessment methodology has been applied to a telemedicine venture in Kenya over the last three years.

A CASE STUDY: MASHAVU TELEMEDICINE SYSTEM
Mashavu: Networked Health Solutions is an academic entrepreneurial venture that is striving to improve access to pre-primary healthcare for rural Kenyans.By connecting the residents of rural communities with healthcare professionals in urban centers through a telemedicine system, Mashavu closes the physical distance between the two groups 41 .Kiosk operators utilize computer/smartphone technology, along with ruggedized biomedical devices, to gather and report patient information.The kiosk operator completes a thorough past medical history of the patient, gathers information on the patient's basic hygiene and nutrition habits, and takes photographs, if needed.Mashavu kiosks transmit this information through a web-based portal.Medical professionals (specifically nurses) view the medical information and respond to the kiosk operator with personalized recommendations for the patient within 20 minutes 42 .The kiosk operator relays the nurse recommendations to the patient in-person, by phone call, or via a text message.The systemic assessment framework presented in this paper was employed to validate Mashavu and rigorously amplify its operations.The following sections detail the way in which the framework was applied to Mashavu and how it led to the identification of critical metrics.The lifecycle of the assessment process is shown in Figure 4 and is discussed in this case study.
Step 1: Overarching Objectives The overarching objectives of Mashavu are 1.Improved access to pre-primary healthcare 2. Active community health education 3. Socio-economic development through micro-enterprise The first objective of Mashavu addresses the limited access to primary healthcare in Kenya.Mashavu kiosks are designed for use in remote areas, providing community members with the opportunity to track their health over time and receive input from a health care professional as to whether or not they need to spend the time and money seeking medical attention.This saves patients both time and money and empowers users with the ability to take an active interest in their health.Mashavu's second objective is to improve and expand community health education.Mashavu kiosk operators are responsible for educating a community about basic health information.With the rise of chronic diseases in developing countries like Kenya, the country and its healthcare system are suffering from a double burden of chronic and infectious disease 43 .
Many chronic diseases such as diabetes and heart disease are both preventable and manageable.A trusted kiosk operator who is well-respected in the community has the power to disseminate this knowledge and encourage community members to take an active interest in their own health.

FIGURE 4 LIFECYCLE OF THE ASSESSMENT PROCESS AS USED BY THE MASHAVU VENTURE
The unemployment rate exceeds 40% in Kenya, leading to Mashavu's third objective: socioeconomic development through micro-enterprise 44 .A Mashavu kiosk benefits all stakeholders by returning money to the community, creating jobs for nurses, and supporting local clinics and chemists.Despite the limited number of doctors, a surplus of nurses provides the needed human capital for sustained operations.The kiosk operator charges patients for the service and receives a portion of the money generated per consultation.The nurse who gives patient feedback also receives payment per consultation, providing a source of income for the individual.If a patient receives a recommendation from the healthcare professional to seek medical attention, patients travel to the closest clinics for their services, effectively creating more business for often underutilized clinics.
Step 2: Levels of Abstraction using the Hierarchical Framework The Mashavu system utilizes five levels of abstraction and specifically defines them as: value, stakeholders, network, entity (kiosk), and sub-entity (devices).Value explores how macro-value is created as the various stakeholders benefit from the Mashavu system.The stakeholders level encompasses all of the Mashavu stakeholders including the community members, the kiosk operator, and the partnering healthcare professionals.This level is designed to assess the user experience, tangible benefits and value created, and includes stakeholder education.The network level assesses a network of Mashavu kiosks across a region and includes a standardized user interface, data management and processing for informing public policy decisions, and analyzing interactions between kiosks.The kiosk level includes the structure of the kiosk and the way by which it integrates the biomedical devices, the daily concept of operations, the clinical encounter and the clinical (medical) appropriateness of the Mashavu experience.The device level considers all aspects of the biomedical devices that are a part of the Mashavu kiosk.Utilizing these levels provides a structured simplification of the system allowing for easier identification of critical metrics to assess.
Step 3: Teams The Mashavu venture integrates a number of courses working on various aspects of the venture 45 .Mashavu students were divided into teams tasked with championing critical aspects of the core venture.The nine teams worked on the following areas: biomedical devices, kiosk design and device integration, website development, clinical encounter, concept of operations, systemic assessment, stakeholder education, user experience, and marketing and business planning 46 .These teams were responsible for understanding what groundwork had been laid in the past, what strategies were successful, and what needed to be improved, developed or reinvented.Each team was assigned a short-term goal or task to work on for the semester and was required to identify metrics for assessment for the implementation of that goal or task.
Step 4: Large Question Set Each of the nine teams was responsible for identifying metrics that would assess their specific areas of focus.Interactions between different teams were considered for assessment.The assessment matrix was used to uncover the relationships between the different levels of the system (Figure 5).Each block of the matrix represents either an individual abstraction level or a connection between levels.For each connection, a list of metrics was compiled, resulting in a thorough exploration of the interactions between the levels of abstraction and the identification of hundreds of metrics that could possibly be assessed.
For Mashavu, the sub-entity level revolves around the system's biomedical devices.Students enrolled in a junior-level bioengineering class created new versions of common biomedical devices specifically designed and ruggedized for the Kenyan context.The devices included a thermometer, pulse oximeter, sphygmomanometer, weighing scale, stethoscope and spirometer.In terms of assessment, the first step is to determine what metrics are relevant.For example, it is important to benchmark the developed devices against commercial-off-the-shelf (COTS) devices.Specific metrics analyzed were accuracy, precision, durability, functionality, and cost.The second step is to determine how the devices interact with the other levels of the system.For example, in examining the relationship between devices and the kiosk, metrics to consider include integrating the devices with the netbook that runs the kiosk and measuring the reliability of data transferred from the device to the kiosk's netbook.Relationships between the devices and stakeholders include how the stakeholders interact with the devices and whether or not the devices are contextually and ethically appropriate for use in Kenya.

FIGURE 5 ASSESSMENT MATRIX FOR THE MASHAVU TELEMEDICINE SYSTEM
The System level focuses on the Mashavu kiosks.To assess the value created from the kiosk, one important area to examine is the efficacy of Mashavu branding and what people associate with the kiosk's logo.Determining whether or not customers enjoy the Mashavu service helps assess the user experience.The connection between the network and kiosk can be assessed through metrics such as the amount of time that internet was available, the speed at which consultations were sent, and how long it took for patients to receive recommendations from the nurse.Finally, the efficiency of customers' visits is measured to better understand potential areas of kiosk improvement.This design matrix may lead to hundreds of metrics to assess.For an academic venture with limited time and money, assessing every metric is not practical.The next step is to identify the critical metrics needed for assessment from this exhaustive list of metrics.
Step 5: Canonical Question Set The next step was to consolidate and simplify the list of questions generated from each block in the design matrix into a more manageable set of approximately fifty critical metrics.The responsibility of choosing which metrics would be assessed was assigned to the sub-teams, as they knew what was most important in their respective areas of focus.In addition, each team was tasked with determining the feasibility of assessing these metrics on the ground in Kenya and with developing a detailed plan for how the data collection would be performed during the three weeks the team spent in Kenya.For the quantitative data needed, spreadsheets, surveys, and data collection techniques were created as part of the implementation plan.For the qualitative data, the method for collecting data, whether by survey, interview, or focus group, was decided and questions were written.The conceptualization of questions was an arduous process since qualitative questions needed to be structured in such a way that value could later be derived from the data.
Step 6: IRB Protocols for Data Collection In order to obtain both the desired quantitative and qualitative data, extensive surveying of Kenyan community members, doctors, and nurses had to be conducted.To manage risk and keep the protocols tractable, the study was split into four separate IRB protocols.Doing so ensured that if one part of the study was not approved, the rest of the research could carry on.These studies were approved by Penn State's Institutional Review Board.An Institutional Review Board reviews the ethics of research conducted on humans with the aim of protecting the rights and welfare of the research subjects 47 .Despite the complicated and tedious process of gaining study approval, the IRB protocols helped in structuring data collection on the ground 37 .These strict protocols helped us evaluate and refine our proposed process for collecting necessary data to assess the critical metrics.
Step 7: Data Analysis The individuals conducting the surveys, interviews, and focus groups were responsible for adhering to IRB protocols, compiling responses and analyzing the collected data.Determining the critical metrics for success early in the assessment process helped simplify and streamline the analysis process and avoid extraneous data analysis.Results were quickly obtained from the data and could be used to evaluate the critical metrics identified for the venture.Achievement of short-term goals and tasks by the venture were documented.
Step 8: Reporting Critical Metrics The objective of reporting critical metrics was to determine what worked and what did not work in the Mashavu system.The clear and concise list of assessment metrics made it easier to evaluate the analyzed data and determine what the results indicated.The collected data supported the potential for success of the Mashavu venture, and also highlighted areas in need of improvement and redesign.Attaining swift results was a key contributor to our success on-theground.We were able to provide valuable insights to our partners in Kenya, helping us to create new partnerships and strengthen existing ties.
Step 9: Traceability of Overarching Objectives This assessment process is designed to be a holistic examination, starting with the venture's objectives, using systems thinking to determine metrics needed for success, evaluating these metrics, and then analyzing how well they support the overarching objectives of the system.Does the data support the objectives of the system?How can the subsystems be modified, redesigned, or improved to better address these overarching objectives?The assessment process provides insight into these questions.By continuing the cyclical assessment process and utilizing systems thinking the simplest pieces of the system can be evaluated and improved in order to support the success of Mashavu.Every assessed metric was traceable to the three overarching objectives.
Although the use of this assessment methodology with the Mashavu venture was quite successful, it was not without complications.Utilizing the design matrix to simplify and organize the assessment of the system was crucial to ensuring a complete assessment.It allowed us to systematically look at every relationship in the system and evaluate it individually.The difficulty came in verifying that every relevant metric was considered.There was no protocol for ensuring that any of the system components were not forgotten.The team faced several challenges related to collecting accurate data.Problems arose as the data collectors were students, and they did not always operate the devices to the same standards.This problem was later fixed by training community health workers to operate the devices.User feedback was collected through focus groups and post-experience interviews.During these, we experienced some problems with translation, medical terminology, and inexperienced researchers.Occasionally, respondents would provide a 2-minute response in Kiswahili, and the translator would say, "She said she liked it."It was hard to convey to the translators, who were also students, that we wanted the details of their answers.Eventually, we determined that focus groups with proficient English speakers were far more useful for our needs.They allowed for conversations to occur and the ability to ask follow-up questions that led to the rich responses that we were seeking.In subsequent assessment activities, we referred to the significant body of knowledge with practical guidance on collecting accurate data.Nevertheless, this systemic and systematic approach to assessment helped the entire venture team immensely in assessing performance efficiently and effectively.

CONCLUSION
This paper discusses a systemic and systematic approach for assessing academic social ventures.This approach has been employed successfully for complex multi-year ventures in East Africa.Further work includes improving the process that discerns the essential canonical question set from the larger list of questions that emerge from the assessment matrix.Currently the canonical questions are chosen by the team's qualitative judgment and priorities.Using a quantitative approach like the Analytical Hierarchy Process (AHP) to weigh the priority of metrics would help optimize the canonical question set resulting in simplified IRB protocols and more efficient data collection and analysis processes.Over the next two years, we intend to apply this framework to other social enterprises that emerge from our program.We are also seeking collaborators to provide feedback to explain the framework and its operationalization more clearly as well as apply it to other social enterprises.

TABLE 1 EXAMPLE
OF LEVELS OF ABSTRACTION FOR SYSTEMIC ASSESSMENT