Agile Corps - A Public Service-Learning Program Part II: Pilot

- Annually, the U.S. government invests $94 billion on IT products and services. The majority of these projects fail--they are late, over budget, canceled outright, or, if delivered, are outdated or not user-friendly. Due to barriers in hiring and training, the government tends to outsource IT talent at a premium through contractors, but outsourcing talent has not changed the results. Inside the government, the small amount of talent that exists tends to be senior, and there currently are very few, viable options for high-quality, junior and mid-level technologists to find a job in government and professionally develop and progress. Agile Corps is a program designed to identify, recruit, train, and retain junior and mid-level technology talent in the government. Agile Corps exemplifies public service-learning, a learning approach and strategy that combines learning objectives, instruction, and reflection with government service for the public. After completing a discovery research process followed by prototyping and testing the program design, we piloted the Agile Corps program at the US Department of Labor. This paper presents the Department of Labor pilot of Agile Corps and the concept of public service-learning, and measures the impact of the Agile Corps pilot at the Department of Labor.

days a week, for a mixed cohort of students 11 . The mixed cohort was intended to be 50% internal government participants and 50% external participants hired into government. Certain features were flexible. An agency could choose any number of tracks, an in-person experience or a fully remote program, and 100% internal government employees or 100% new hires.
The 18F design team then continued to market the revised syllabus, one-pager, two-pager, and pitch deck to various agencies to see if any would purchase a pilot. The Environmental Protection Agency (EPA) began talks with the 18F design team to purchase the software engineering track. The Department of Treasury Office of Financial Research (OFR) began contract negotiations with 18F for the design track for a cohort of their employees. However, the first agency to purchase and host a pilot was the Department of Labor (DOL). The Wage and Hour Division (WHD) of DOL purchased a 3-month, mixed program for $256,000. They were the first and only agency to request a mixed track where participants learn to apply HCD, Lean Startup, and Agile to their job regardless of the function, engineering or non-engineering (i.e. finance, HR, legal, communications, design, etc.) in an effort to transform the organizational culture and improve outcomes for the public related to employment and fair labor practices.
This paper focuses on initial qualitative outcomes of the first Agile Corps pilot at the DOL. Quantitative measures of learning will be discussed but will not be the focus as we used a masterybased, competency-based learning approach [17][18][19][20][21][22][23][24][25] . Learnings that are transferable from the DOL WHD pilot will be applied to future, bigger pilots at other agencies, such as EPA and OFR, and, more broadly, to service-learning programs with students.

Lɪᴛᴇʀᴀᴛᴜʀᴇ Rᴇᴠɪᴇᴡ
In part I of this paper, Agile Corps had not yet been designed. Even though we found no examples of PSL, we reviewed the literature for research-validated, modern educational methods and practices, as well as research on service learning. That literature review and subsequent research, synthesis, and ideation, led to the design of Agile Corps, the prototype 11 .
Before piloting, our literature review focused specifically on government service-learning programs (PSL). We found no such examples in the literature. However, the book Peak Academy chronicles the deployment of a Lean, continuous improvement educational program with workers in the city government of Denver, Colorado 26 . As opposed to an ongoing, multi-month service learning program, Peak Academy offers a few one-off courses. Each learner is eligible for two of the trainings: an introductory level training (Green Belt) and an advanced training (Black Belt or Superhero) [26][27] .
• Green Belt Training: a 4-hour course on process improvement • Green Belt for Leaders: a course on process improvement for supervisors through executive directors, designed to help learners support employee innovations initiatives and to build high-performing, innovative, data-driven teams • Black Belt Training: a 5-day course on process improvement leadership; there are two versions for this, one for Denver city and county government employees and one for other government and nonprofit employees • Public Service Superhero Training: a two-day virtual course in place of the Black Belt due to the global pandemic COVID-19, in an effort to reduce time away from government work during the crisis. The Peak Academy model diverges from the PSL model due to the separation between the classroom instruction and government public service work. In the Peak Academy model, the instructors (8 full-time staff) hope that the learnings continue and impact the government work of the learners, but there is no continued learning and coaching or post-service work reflection that continues to influence the government service work. However, the similarities and impact were significant enough to impact our work and to be cited as work upon which Agile Corps is built.
The similarities suggest the possibility of transferable insights or designs as we pilot Agile Corps at DOL. First, Peak Academy and Agile Corps are both government educational programs for adult civil servants. Both use lean principles with a focus on continuous improvement and the reduction of waste. Both intend to transform the culture, a definitive part of PSL programs. Both presume that such increases in efficiency and effectiveness through continuous improvement will result in cost savings to the government. By 2016, Peak Academy had trained over 5,000 workers and saved over $15 million in government spending; by 2019 total savings had increased to over $37 million with over $4 million in 2019 alone [26][27] .
The Agile Corps pilot builds on the Peak Academy model in three ways. First, Agile Corps more tightly integrates classroom training with government service work and reflection, constantly iterating between through the cycle and naturally embedding continuous improvement. This is a strong service learning approach. Second, Agile Corps not only infuses Lean Principles in the culture and work of government employees but it infuses Lean Startup methodologies which focuses on experimental testing launching new products and services [28][29][30][31][32][33][34] . Agile Corps goes beyond continuous improvement to include social entrepreneurship and social intrepreneurship. Third, Agile Corps tries to improve engineering through Human-centered Design (HCD) methodologies and Agile mindsets as well as apply engineering methodologies (like HCD and Agile) to other functions in the government [35][36][37][38][39] . This is an effort to improve government products and services for the public, or humanitarian engineering. Agile Corps uniquely combines humanitarian engineering and social entrepreneurship through service learning resulting in cultural transformation.

Mᴇᴛʜᴏᴅᴏʟᴏɢʏ
The work to research, design, prototype, and pilot the Agile Corps program involved five research phases. First, we utilized an exploratory research and design process to create the concept. Next, we used an experimental research process to test and refine a prototype of Agile corps. Both phases were explained in Part I of this paper.
Here, in Part II, we describe three research or implementation phases. First, we performed research to customize the Agile Corps model for the needs of the DOL Wage and Hour Division (WHD) and learners. Second, we implemented the pilot at the DOL. Third, we immediately evaluated the initial impressions and impact of the program and learning. Each of these three steps involved different methodologies. All three steps are still embedded within an overall question: What are the learning needs and skills gaps of public servants who work in technology in the government? How might we design a program that fits the needs of those students as well as the government agencies' need for more high-quality technologists at all levels including junior levels and mid-levels?
The most important factors in determining which research methodology to use are the research questions 12 . Because our research questions are open-ended, lack a specific hypothesis, and seek to uncover the mindsets, fears, and difficulties of public servants in government institutions, a qualitative research methodology is best.
However, the general answers to the questions across the federal government were used to create the initial Agile Corps design 11 . Since the answers were used to design an unvalidated, educational program intervention to improve government products and services for the public, we also need a design experiment or design-based research methodology for validation. Design-based research (DBR) is characterized by pragmatism, theory, iterations, integration, and contextualization 41-42 .
Because DBR often is initiated by the educational researchers as opposed to practitioners and may never be implemented in an actual learning environment, we employ a more narrow version of DBR called Design-based Implementation Research. Design-based Implementation Research (DBIR) has four guiding principles. These include jointly negotiated work, iterative and collaborative design processes, development of theories, and design for viability [43][44][45][46][47][48][49] . Following those principles, this work was initiated and requested by different government agencies, 18F, and technologists who find it difficult to be hired by the government. Our design team includes government workers as well as the potential students. Theories guide and emerge from our design research and work. Finally, our goal is sustainability via career development and growth potential for junior and mid-level technologists brought into government service.
Design-based Research, in general, does not have to use a mixed methodology, though sometimes it does. In this case, we combined DBIR with an exploratory qualitative methodology (Part I); therefore we chose the mixed methodology called Human-centered design (HCD, usercentered design, or design thinking) as our overarching framework because it combines a qualitative methodology with DBIR [50][51][52][53] . In education, HCD is often called student-centered design or learner-centered design (LCD). Human-centered design is only a methodology, not a solution. It can be used to design a building, a project, software, a process, a service, a product, an environment, a program, an organization, or even a curriculum. Human-centered design mixes skills from engineering design such as defining the problem, setting the constraints, and rapid prototyping, with practices from the social sciences such as ethnographic studies, interviews, extracting insights from qualitative data, and empathic immersion [54][55][56][57] . In short, HCD keeps the design work centered on the users, the beneficiaries, or the community. In this work, HCD creates two pillars. First, instead of the traditional model where the designers go away, design a solution, and then bring it back to the community, in HCD, we practice participatory co-design where the community is involved in every stage of the design process [58][59][60] . Second, instead of the design team only including members from outside the community, the design team also includes federal government workers who lack the ability to hire the talent.
In this work, the design team includes a core team of four government workers and a larger team of 20 government workers. Second, we involved not only local design team members but also technologists outside of government.
Human-centered design is normally broken into three stages, Discover, Design, and Deliver. The first stage (Discover) is a research stage where you listen and learn from the community for whom and with whom you are designing, developing empathy and connecting yourself to the needs and desires of that community. The second stage (Design) is a stage of data analysis and synthesis of the previous qualitative and quantitative work, followed by ideation, prototyping, feedback, refinement, and iteration. The Discover and Design stages were completed in Part I of this paper 11  success, partnerships, and developing business models for financial viability and sustainability. This paper, Part II, focuses on the Deliver stage (Table I). In the first phase of this work, we utilized an exploratory research and design process to customize the Agile Corps design for the DOL. The Agile Corps model was designed to allow flexibility for customization to the specific needs of the federal government agency who purchased the service. The general Agile Corps model had not been customized for the DOL WHD group.
To answer questions about the DOL WHD learner needs, motivations, and availability, we chose a qualitative methodology which helped to determine the learning topics, frequency, time, duration, focus, tracks, synchronicity, and more. Specifically, we employed interviews, observations, and focus groups.
Second, we implemented and ran the pilot at the DOL. We needed to determine feedback cadence, success criteria, staffing, and evaluation. Rather than a research methodology, we chose two implementation methodologies used by Agile teams for developing software-Scrum and Kanban (Table I) [61][62][63][64][65] . This choice allowed us to model the type of continuous improvement the students were learning, as we shared our process for constructing the course. Secondly, we invited them into our course construction process (participatory design) as they participated in Kanban and Scrum ceremonies such as Agile retrospectives providing feedback to continuously iterate and improve the program while it was being constructed.
Third, we immediately evaluated the initial impressions and impact of the program and learning. Because we wanted to understand the percentage of students that achieve all competencies as well as the satisfaction and experience of the learners, we chose a mixed methods approach (Table I). We used quantitative research to evaluate the learning and qualitative research to evaluate the experience and satisfaction.
The remainder of the paper is broken into two sections. First we provide a brief overview of the procedures for the customization research and program implementation. Then we provide the results of the initial evaluation research. This work will inform the iterative process and bigger, future pilots at other agencies to be presented in future papers.

Agile Corps Pilot: DOL Customization Research
In Part I of this paper, 18F sold a three-month pilot of Agile Corps to the DOL WHD, starting with a discovery sprint in August, two months ahead of an October start of the pilot program. The Agile Corps prototype was already designed and tested, so the purpose of this discovery research process was to determine how to adapt the program prototype to the specific needs of the DOL WHD learners. The 18F team also used the discovery sprint to help the DOL WHD leadership determine how many and which WHD employees should participate in the Agile Corps PSL program. Not knowing who would be participating presented a problem for the research team; learning designers need access to learners in order to uncover learning needs and skills gaps through observation, assessments, and interviews. However, at the time of discovery, the WHD leadership did not yet know who they would place in the program for a majority of the program openings. The few employees that WHD leadership already wanted in the program were future employees who would start working at the DOL in October and were currently inaccessible.
The second limitation of the customization research was stakeholder-driven design. The customized design of the DOL WHD Agile Corps pilot was not solely based on the analysis and synthesis of the customization research but subject to the approval of WHD leadership including two Assistant Administrators. The customized pilot design was influenced by this collaboration rather than being solely data-driven.
With only two weeks to conduct a customization research discovery sprint, the 18F team used a fast ethnography methodology in order to recruit, interview, analyze and synthesize without fully coding complete transcriptions [66][67] . Next, the findings were presented to WHD leadership and in multiple meetings and consultations with leadership, an agreement was made for the pilot design and which DOL employees should participate. The pilot would diverge from the prototype, making program execution and instructional design harder: a larger group of participants for a shorter service-learning program with a mixed-discipline cohort (Table II). Instead of 6-20 participants, there would be 35. Instead of 3-6 months, the PSL program would last 2.5 months or 10 weeks. Originally, the Agile Corps prototype embeds HCD, Agile, and Lean Startup learning into a single track like design or multiple, parallel tracks like design and data science. The DOL WHD pilot design, instead, would use a single, mixed-discipline track where the general learning was HCD, Agile, and Lean Startup. This change was due to the WHD leadership's desire to apply these principles and methodologies outside of the narrowly defined group of technologists, and include employees that work on technical products and services. The pilot participants would include digital acquisition specialists, digital program managers, project managers, finance, etc. in addition to engineers, product managers, and designers. Training a larger, more heterogeneous group of learners for less time was going to be more difficult than the original prototype design. The 18F team decided to continue with the pilot because it was still a chance to test some version of the prototype. Also, the process of adapting the prototype to the DOL WHD constraints could demonstrate to the participants the type of Agile mindset the students would be encouraged to develop.
The pilot matched the prototype in two ways. The pilot would include mentorship and office hours as planned. Mentors would be provided from 18F's staff of product managers, designers, data scientists, acquisition specialists, and strategists. Based on the customization research and WHD leadership agreement, the pilot design included five hours of weekly instructions, split between Tuesday and Thursday mornings with one hour of mentorship on Mondays and two hours of optional office hours on Friday ( Figure I).

FIGURE I Aɢɪʟᴇ Cᴏʀᴘs Pɪʟᴏᴛ Sᴄʜᴇᴅᴜʟᴇ
Lastly, the Agile Corps prototype could be synchronous or asynchronous. Since the DOL WHD employees participating in the Agile Corps PSL were in different cities, the 18F team chose a semi-synchronous design including asynchronous learning online and project-based assignments as well as synchronous learning for live classroom sessions. Participants in the same city would gather together in person and connect via video to those in other cities while the instructors taught from Washington, DC, which was the city in which 60% of participants lived.
The 18F team designed a two-day, in-person program overview and introduction to HCD, Lean Startup, and Agile to occur during week 1. The regular schedule then started in week 2 after those working outside of Washington, DC, had traveled back home ( Figure I). Each week, participants would receive recap emails to help those who missed sessions and to aid students in explaining to their managers what they were learning. Weekly retrospectives would be held with the participants so that they could practice the Agile principle of team reflection and watch, in realtime, how the 18F team changed the class based on the retrospective synthesis. Finally, each student would work on a public service, group project with other students. Students were asked to make the project part of their public service job.

Agile Corps Pilot: DOL Implementation
The 18F team emailed the selected participants who were chosen by the DOL WHD leadership. Unfortunately, most of them had not signed up or volunteered for the program so they did not enter the program excited. The program kicked off with a soft launch in late October while the participants were still located in different offices around the country. Then in November, the 18F team conducted the official kick-off with all participants in-person in Washington, DC. This event provided a general overview of HCD, Lean Startup, and Agile.
Progress through the topics was tracked through a publicly accessible Trello board. As planned, recap emails and retrospectives occurred weekly, and the 18F team adjusted the delivery mode and content as they learned what worked for the learners and what program or class components did not work well. In December, the 18F team conducted a one-day training for WHD leadership to explain what the participants were learning and create a shared vision and increased buy-in so that employees had more flexibility to try innovative methods on their public service projects. Through the program implementation, the 18F team faced four challenges.
First, the program started in October and November and crossed the winter holidays including Thanksgiving and Christmas. In the USA federal government, many federal workers have "use-itor-lose-it leave." If they do not use it, they will lose it. So, the use of vacation or leave is much higher in the months of November and December than the rest of the year. This meant people missed class or were busy trying to make up for job-related work to either prepare for leave or to catch up after returning from leave. To minimize this, the 18F team canceled and postponed some live instruction weeks to accommodate the smaller attendance that would have resulted.
Second, many students had difficulty using the technological tools. The Department of Labor blocked access to Google Hangouts and Zoom, which were the videomeeting tools 18F used in 18F daily work. It was difficult for students to join not only live class sessions but also mentor meetings and office hours. As a temporary fix, participants outside of Washington, DC were asked to join on a personal device or without signing into the DOL virtual private network (VPN). Participants were also asked to call in through a phone, but that meant they could not see anything.
As a medium-term solution, the 18F team ran training sessions for participants. In the end, the 18F team thought it would be better to use the DOL videoconference tool which was Adobe Connect. The 18F team had avoided this as Connect is meeting software not educational software. Through research and working with the CIO office, the 18F team discovered that the DOL had a license for the educational version of Adobe Connect, Adobe Learn. Once this was uncovered, the program switched to this option in November and ran training sessions for the students.
Third, the pilot design included public service-based, group projects for each student. These projects were supposed to be a natural part of their public service work and completed as group projects with other students who also had the same project as part of their public service work or job. Based on the WHD leadership's assignment of participants, there were participants who entered the program without any type of project. Others entered the program with a project but none of their teammates were assigned to the Agile Corps PSL so they were not all being trained. Others had a project and teammates who were also in Agile Corps to work on the project while being trained together. Due to the variety of project statuses, participants' experiences of the project-based learning were very diverse. Without shared projects and prioritization of projects by managers, students could not effectively work across cohorts. The 18F team assigned people to groups as a temporary fix, but it meant there were participants working on a public service project not directly related to their public service work in their job.
Fourth, the process of recruiting, assigning, and assisting mentors was slower than expected. The ideal design had mentors ready on day 1, but student assignment did not happen until October. The 18F team shared a survey with students on day 1, the last week of October. The survey asked people to describe their job and the skills they were looking to acquire. In November, the 18F team had made initial matches between mentors and mentees. Realizing the need for additional mentors, the 18F instructional designers hosted an "Ask Me Anything" (AMA) event inside 18F to recruit more mentors. All mentors received a baseline level of training and were asked to introduce themselves via email. Mentors provided mentorship from mid-November through mid-January. In late December, 18F mentors gathered for a retrospective to identify problems, suggest solutions, Lastly, the 18F team created a list of HCD mindsets and Agile values students should embody by the end of the course. The team also created a list of Lean Startup and Agile principles in which students should demonstrate minimum competency through their project work (Table III). Principle 4: Business people and developers must work together daily throughout the project.
Principle 6: The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Principle 8: Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Rᴇsᴜʟᴛs
In the Kirkpatrick learning evaluation model, there are four levels of evaluation [68][69][70] (What is the impact?) -What is the effect on the organization? What organizational outcomes or key results occurred due to changed behavior of students?
Because this is a public service-learning program, ultimately the goal is to affect not just the organization, but the public. Thus, we add a fifth level according to the Kirkpatrick Plus model 71 .
• What social good is produced for citizens, refugees, immigrants, or the environment?
For the DOL WHD Agile Corps Pilot, Level 4 results include the following.
• Shorten the process of producing communications, policy, working software, usable content, etc. • Improve the acquisition of software • Improve product management abilities within WHD • Remove roadblocks of burgeoning documentation, long contract negotiations, and tirelessly produced memo after memo before making decision • Faster decisions made at the team level Level 5 results relate to wage and salary violations and remunerations affecting the public. Ultimately, this Agile Corps will help the DOL WHD to better identify, find, notify, and hold in compliance employers who have violated the law by not paying workers their wages. Agile Corps should also help WHD better collect those back wages and deliver them to workers thereby recovering months of rent, child care, food, etc. The Kirkpatrick Plus model still suffers from a large jump between levels 2 and 3. A skill learned in one context is actually a different ability than the same skill in a different context [72][73][74][75] . Someone who learns a skill in a classroom (level 2) may actually not have that skill in a work context which prevents their behavior from changing (level 3). Based on the contextualized understanding of learning from social learning theory, here, we create a modified version of the Kirkpatrick Plus evaluation levels which we call Social Kirkpatrick Plus levels of evaluation.
• The Social Kirkpatrick Plus model has three benefits. It highlights the artificial nature of the traditional classroom context and encourages instructional designers to create a training environment as close as possible to the environment students will experience while working for the public in their jobs. As the classroom environment approaches the conditions of the work environment, it is more likely the acquired skills in the classroom will be transferable and that level 2.5 will be reached. Secondly, it reminds instructional designers that further on-the-job training is often needed after classroom training due to differences in the contextualized learning experience. Third, the model focuses Level 3 (Behavior) on environmental and structural inhibitors and enablers of behavior change and recognizes the effect of context on non-cognitive skills such as motivation, perseverance, relationship, confidence, commitment, etc. These are included in Level 3 because of the context shift from Level 2 which affects non-cognitive skills as well as cognitive skills (knowledge and abilities).
The Agile Corps pilot sits somewhere between classroom training and on-the-job training since it involves both. In this analysis using the Social Kirckpatrick Plus model, we consider the Agile Corps PSL program to be Level 1 and 2. We consider evaluation level 2.5, which measures student learning in the job context, to be an assessment of participant learning in the job context after the program has ended. In other words, did the students acquire the knowledge and skills and demonstrate it in the context of the job without the support of mentors, project coaches, office hours, and intermittent classroom training?
Because this evaluation was done immediately at the conclusion of the 10 weeks of the program, the only levels that are assessed and presented here are reaction and classroom learninglevels 1 and 2. However, because the hybrid Agile Corps model does involve assisted application to job tasks with coaches and mentors, level 2 will include measures of assisted demonstration of skills in job contexts. We will continue to follow and measure on-the-job learning, behavior change, organizational results, and social impact (levels 2.5 -5) of the program in future work.
The initial Level 1 and 2 evaluation of the Agile Corps pilot was completed by an independent evaluator from 18F who did not take part in the program, in order to reduce bias. The evaluator spent two weeks conducting evaluation research upon the conclusion of the program. The evaluator reviewed Agile Corps content, end-of-class surveys, and retrospectives, and interviewed 20 people including 18F business development workers who sold Agile Corps to the DOL, program facilitators, mentors, students, and the DOL WHD leadership.

Emails & Technology
Agile Corps facilitators sent weekly recap emails to the cohort and made space for frequent retrospectives. Recap emails were generally well received. Two interviewees said they found the emails especially useful for catching up on classes they missed and for forwarding along to their colleagues who were not in the cohort but had questions about what they were learning. Recap emails followed a predictable pattern, including sections for summary, assignment deadlines, reflection questions, and additional resources.
There were students who had a different experience with technology. In an email, one participant said: "We can't install software on our laptop. [18F] didn't seem to put any forethought into our needs and assumed 'things would just work.' … How are they going to teach us to put customers first when they didn't plan for our customer needs?" Another added "During the group exercises, the remote users spent almost all of their time fighting technology." The next day remote users asked facilitators to postpone all training until 18F and the DOL devised a workable solution for remote participants.
In response, the 18F team changed the synchronous learning to be remote for all participants. The fully-remote classroom created an equivalent experience for all learners no matter if they were located in DC or located in other locations. The fully-remote classroom also forced the 18F facilitators and instructional designers to address technology issues for all students, not just a minority. The 18F team also ran training sessions as many participants were not accustomed to live, remote educational experiences nor the Adobe technology being used.
There were still students who had not fully mastered the technology. One exit survey noted "I hated the fact that people never learned how to bind their calls to WebEx. Our technology challenges created a significant number of challenges that we still need to address."

Transition to Service Projects
The Agile Corps vision involved a smooth transition and interplay between project-agnostic classes to project-based service-learning. At least one participant didn't experience it smoothly due to misunderstanding 18F's role: "I thought we were going to get an agile PM to support our project, to help us move through a live project. I thought we were going to get a hands on PM to help push us through!" Another participant's exit survey response (to the question "What advice would you give to another student who is considering taking this course?") suggests that Agile Corps' optional mentorship and office hours opportunities were much more important to this transition than they might have otherwise seemed. "Mentoring, office hours and homework … is how I really sorted through what I thought I knew versus what I did know. I learned more in those environments than I did in the classroom." Another factor inhibiting the smooth transition to project-based learning was the nature and number of projects. One participant said, "When we first talked about it, my understanding was that we wanted to identify key [service] projects that could benefit from an agile experience … I didn't think we'd have 30 people and 20 projects. I thought we'd have maybe… three projects? You know, with self-organized teams -or at least teams mandated by managers for the duration of the course." There were participants who joined Agile Corps without a project (they were asked to find one), while other participants were the only people assigned to their project (which made concepts 74 like standups unnecessary). There were participants already working on projects being run in an agile way, such as one team which was already working with TTS Acquisitions, a sibling organization to 18F inside of GSA. In general, participants expressed difficulty working on service projects with other participants due to a lack of shared projects and work project prioritization by their managers.
Project prioritization was also a recurring theme across interviews and exit surveys. There were participants who felt as though they did not have the authority to apply the skills they were learning to any project due to their manager's expectations of their role. One participant said, "The [18F] team could have done a more in-depth briefing with leadership beforehand. [The team] conducted a [DOL WHD] leadership training in December, but could have done a more in-depth briefing before then. The [18F] team ended up being more reactive than proactive." Another exitsurvey response read: "Given the fact that managers have volunteered staff to participate is not bad in itself, but … managers [should have also gone] the extra step of identifying the highest priority service projects with highest potential value for which the entire project team can participate in the Agile Corps."

Course Instruction
Lectures were consistently lauded as one of the program's best parts. In their own words, participants said:

"[The trainers] were very good, very knowledgeable."
"Duane was the teacher for the majority of classes I attended. He's so dynamic and engaging. He really connected well with everyone in the group and I'd be surprised if anyone gave you any negative feedback on his teaching style.
"The teaching style was great, though I kinda like the in-person teaching more. Like the first two days we were there we were all able to interact, and when we broke into groups Duane played music -we all picked a genre -and it kind of made it a better environment. It's different when everyone's on camera and people are entering and leaving and the mic comes and goes." End of class surveys further corroborated these sentiments ( Figure II).

FIGURE II Pᴀʀᴛɪᴄɪᴘᴀɴᴛ Rᴀᴛɪɴɢs ᴏғ Iɴsᴛʀᴜᴄᴛᴏʀ Cʟᴀʀɪᴛʏ
The success of Agile Corps classes seems also related to the facilitators' work in facilitating a conversational environment that lowered the boundary between teachers and students. In addition to the in-person kickoff meeting, weekly recaps, and retrospectives, the pair also instituted a "virtual-first workflow" (for example, assigning homework that shared concepts ahead of time) and used in-class polling to see how well lessons were received in real time. Many participants appreciated the degree to which these factors helped them learn.

Pacing and Duration
Some interviewees expressed frustration that Agile Corps continued throughout the month of December when they had planned vacations and/or absent teammates. December was also the time the pilot transitioned to more of a project-based learning environment -or, more accurately, to an environment in which participants began applying what they were learning to their existing public service projects. A few participants struggled with this. One said "I get the project-based decision but it got confusing to me to apply agile to my project, then human-centered design to my project, then to apply lean to my project." An exit survey question also highlighted this struggle. When asked what they might add to the program's courses, participants responded: "Not as much theory," "More real agile project exercises," "The application of methodologies," and "[To] see how some of the topics can be applied." Some of the interviewed participants didn't really understand the ways in which Agile development, Human-centered Design, and Lean Startup overlap or support each other. They were frustrated to note how, in their opinion, graphics, explaining the relationships between the 76 methodologies, appeared later in the pilot. Participants described classes as both "pre-packaged" and "in the weeds;" and responses to one exit-survey question reinforced this ( Figure III).

FIGURE III Pᴀʀᴛɪᴄɪᴘᴀɴᴛ Rᴀᴛɪɴɢs ᴏɴ Cᴏᴜʀsᴇ Oʀɢᴀɴɪᴢᴀᴛɪᴏɴ
The last group of critical comments related to the duration of the program. Many participants asked for a shorter or more concentrated program in an attempt to facilitate continuity and cohesion: However, participant responses did not entirely trend in one direction. Some participants asked for a longer program, and someone actually suggested extending it to six months (the initial vision for the program). It is unclear whether the program and class length were too short or too long.

Relevance
The final recurring, instructional theme was the relevance of the material presented. Many participants expressed concern with the degree to which design thinking, Human-centered Design, Agile development, and Lean Startup could ever apply to government work. Other participants said it felt as though 18F never really came to understand, empathize, or appreciate how the DOL WHD actually worked before Agile Corps. These participants felt as though 18F just assumed WHD was not working as well as it can.
Other participants were concerned that many of the examples used in the program came from the private sector "world of software development," a world where requirements are sourced in a bottom-up fashion and solutions are iterative. (A majority of participants did enjoy thinking about their work iteratively.) Participants also expressed concern with the degree to which the program celebrated the private sector as a model for innovation. When discussing the Business Model and Mission Model Canvases, for example, some participants did not understand why the Business Model Canvas was presented first [76][77] .

Service Projects
Within the first month of the program, participants began to connect and apply their classroom learning to a specific service project [78][79][80][81][82][83] . All projects have a digital component that undergirds the public service work.

• 14(c) Procedures Manual -A manual for Section 14(c) of the Fair Labor Standards Act
which allows employers to receive permission from the DOL WHD to pay special subminimum wages to persons with disabilities • Ag Safety Video -A safety video for agricultural workers • Back Wage Working Group -Back wages are wages legally owed to workers who were unpaid or underpaid • Customer Service Workgroup • Davis-Bacon Survey Process -A process in which a survey is sent out to collect wage and fringe benefit information in order to make and publish fair wage determinations for labor in construction, repair, and alteration. Participants were asked to what degree did the classroom training affect their service projects.
"I worked on the FMLA resource tool, which is presently just a spreadsheet that links admins to regulations, laws, etc. So our group did a few things. One participant described the Ag safety video project, specifically how the team began to measure its own success more incrementally.
"I'm seeing [improvement] in our communications work -even internal comms. One of our guys in Agile Corps is creating a one-page storyboard of a video so he can get people's reaction much earlier. Previously he might write a script for the film, spend some production time, and then receive feedback later in the process. I think there's a lot of benefit to allowing teams and projects to work in this way." One interviewee spoke to the program's impact more broadly.

"I am amazed when I look back and see how much we have accomplished in the last 3 months.
The agile approach has revolutionized our project." At the beginning of the program, there were participants responsible for multiple projects, while others had a single project or no projects. After assigning projects to all participants, there were multiple groups of work teammates from the same work project participating in Agile Corps together. In other cases, students participated in Agile Corps alone or as a representative for a larger team (in which case they were also actively engaged in the work of culture change).
The project selection and assignment process drew criticism from a participant. "Once they went down the path of 'what's your project,' they increased the [program's] scope dramatically. They also helped highlight something that WHD struggles with, [which is] prioritization. I would argue that my project is one of the highest priority projects -it's just not the sexiest. … There were also a few wishlist projects that came up early on. People would say 'Oh I think [person 1] and [person 2] should work together because this is [a specific kind of project]!' And I was thinking 'No no no, this is wrong!'" The participant felt the flashiest projects were chosen instead of the service projects of highest priority.

Mentorship
Satisfaction responses were solicited from participating mentors and mentees. This paper focuses on the satisfaction of the mentees, or student participants. Mentor-mentee relationships forged during the pilot fell into three categories. There were mentors and mentees who met regularly, those that met a few times, and those who never met. Some mentees felt exceptionally well matched with their mentor.
When the mentoring meetings were regular, mentees frequently commented how conversations with their mentor helped reinforce what they had heard in classes, build connections between theory and practice, and reduce anxieties about how cumbersome Agile, Human-centered Design, and Lean might be. One mentee said, "This is what [my mentor] really helped with: following every step isn't key, nor is having a worksheet. Empowerment is the goal. (That said, the clearance hurdles around our existing infrastructure are still too great.)" In many ways, mentorship helped mentees see a direct path between what they were learning in the program and the avenues available for their own future growth as well as the growth of the organization. One mentor said, "We often discussed organizational-wide challenges to implementing the materials that were being taught." Another participant added "It's particularly 80 challenging in government where we have other requirements we need to manage like records management requirements or the Paperwork Reduction Act. Human-centered Design is difficult to do with people external to the agency because of the sensitivities around information collection. I know we're not alone. 18F, OMB, and others have tried to streamline that process. But every team needs to work its way through that. That's where I see mentorship's greatest value." Most mentees were project-oriented and did not see their mentorship as a tool for personal professional development. One mentee, "It would have been really helpful if each project had a mentor. [I wish I] had someone who could consult on my project vs. everyone having a mentor." This response is supported by participant responses to the exit survey question asking if mentorship was a helpful addition to in-class instruction. Reactions were mixed ( Figure V).

Satisfaction with Student Outcomes
Agile Corps participants described the program and its impact in altogether positive terms. Most participants described Agile Corps as empowering. Participants said the program helped them see their impact in days and weeks rather than months. Many interviewees intimated a desire to more deliberately and uniformly collaborate on future service projects at WHD.
Participants also felt that Agile Corps helped them identify avenues for future growth. Many participants spoke about the benefits that could come from broader adoption of the principles and practices of Agile, Lean, and Human-centered Design. Specifically, they expressed a desire for WHD to think more critically about project prioritization, work-in-progress (WIP) limits, and perproject success metrics.

Level 2: Learning
A competency-based assessment model was used in which students needed to demonstrate a minimum level of competency in all the mindsets and principles through various activities and artifacts. In workshops and through service projects, 100% of participants demonstrated competency with the pilot's core concepts and principle methods, such as: starting with user needs, iterative development, usability testing, Kanban, project prioritization, and retrospectives. One hundred percent of participants demonstrated all principles and mindsets for Agile, Humancentered Design, and Lean found in Table III. Lastly, during interviews, 100% of participants were conversant with words and phrases often used by technologists, such as "discovery," "market fit," "usability testing," "MVP" (minimum viable product), "blockers," etc 84 .

Rᴇᴄᴏᴍᴍᴇɴᴅᴀᴛɪᴏɴs ᴀɴᴅ Fᴜᴛᴜʀᴇ Wᴏʀᴋ
The next steps of the Agile Corps initiative follow two perspectives-the internal, organizational impact and external social impact for the DOL WHD as well as the external government-wide impact of the Agile Corps for 18F. At the DOL, the next steps are monitoring and evaluation of the impact of the pilot. Specifically, we want to measure contextual learning after the classroom sessions and mentorship has ended (Level 2.5), sustained behavior change in their public service jobs (Level 3), organizational impact (Level 4), and ultimately social impact on laborers, labor International Journal for Service Learning in Engineering, Humanitarian Engineering and Social Entrepreneurship Vol. 16, No. 1, pp. 58-89, Spring 2021ISSN 1555 82 rights, and employers (Level 5). Secondly, across government, we want to continue to test which Agile Corps model works to resolve the lack of junior and mid-level technologists in government with career growth paths. This paper describes a 100% internal Agile Corps model in which all participants are already government workers; no one is hired into government through the program. The original Agile Corps model uses a mixed cohort of new recruits with government workers. The original model requires 18F to use its faster hiring mechanism to hire people directly into 18F into government, and run Agile Corps while the longer government agency hiring process continues in the background, preparing to hire the 18F recruits when the Agile Corps PSL ends. One possible pathway is for the 18F Talent team to add training (Agile Corps) to its Recruitmentas-a-Service (RaaS) offering to other agencies; then 18F would train people they recruit for other agencies. However, it is possible that the 100% internal version of Agile Corps, tested at the DOL, can still address the problem using normal government hiring timelines to grow the community of junior and mid-level technologists. Measuring the impact of this pilot will help determine the effect of the 100% internal model on the lack of junior and mid-level technologists.
Additionally, there are a number of recommendations that we can make for future, bigger pilots of the Agile Corps model at other agencies and in other divisions of the DOL. Some of the recommendations are for the government agency hosting the pilot; while others are for the 18F facilitators. They are labeled.
• In the introduction, we explained how the PSL model can be used in primary, secondary, and higher education as a "service-learning as pedagogical training" framework to transform the organizational culture of the education institution. Public Service-Learning can be used to improve the agility and design literacy of an organizational culture as in the example of Agile Corps. More interestingly, PSL can be used to train faculty in schools and universities to better implement service-learning with their students. In this case, service-learning becomes the means to teach how to implement service-learning, transforming the culture across an educational institution. The second implication, mentioned in the introduction, for schools and higher-education institutions is to lengthen the duration of service-learning programs beyond the length of a quarter, trimester, or semester course. A longer duration, as demonstrated in Agile Corps, allows more learning and impact.
The Agile Corps pilot evinces two other ways to apply PSL to schools and higher-education institutions. First, transform the service-learning approach from an approach about service goals and individual learning goals to an approach about service goals and communal, organizational or class learning goals. Traditionally, in schools and universities, the service-learning approach tries to balance the service goals with the individual learning goals by focusing on what service can be done and what learning can be achieved during the length of a particular service-learning course or class. In Agile Corps and in PSL, while individual learning is important, there is greater priority on the organizational or class learning, learning about what causes the most impact in the service work. For example, if a participant leaves the government, they are replaced by someone else to continue achieving the service goals and impact. If an individual switches to a different project in a different part of the government, they are replaced by someone else to continue pursuing the service goals. Thus, in PSL, the service-learning outlasts the individual learning because the service and, thereby, the service-learning is ongoing. This suggests that schools and universities might employ a model that uses on-going projects or partnerships with community organizations that outlast a particular student or student group or class session, and simply allow the changing of students at the start of each semester or trimester or quarter. In this PSL model, the documentation of class or organizational learning is key for new participants beginning the service work, highlighting the priority of the class or organizational learning over individual learning. Over time, as individual learners and participants change, the organization served by students or the class is improving its learning and understanding about how best to implement the service and how best to achieve and surpass social impact goals.
Second, the Social Kirkpatrick Plus evaluation model for PSL can be applied to servicelearning in schools and universities. It can be applied directly for PSL of school and university staff and in the "service as pedagogical training" framework for faculty. Also faculty can use the Social Kirkpatrick Plus evaluation model to evaluate the service-learning in their courses. The only change is the transformation of the phrase "on-the-job learning" in PSL to "in-the-field learning" in traditional service-learning. Ultimately the use of the Social Kirkpatrick Plus evaluation model for PSL implies that service-learning in school and universities needs to improve its ability to measure the true social impact of the service projects completed by students. Truly measuring social impact will equitably elevate the social impact goals to the learning goals.

Cᴏɴᴄʟᴜsɪᴏɴ
International Journal for Service Learning in Engineering, Humanitarian Engineering and Social Entrepreneurship Vol. 16 Public service-learning (PSL) is a learning approach and strategy that combines learning objectives, instruction, and reflection with, specifically, public or government service on behalf of the public. Agile Corps is a PSL program designed to address the problem of a lack of junior and mid-level technologists with career pathways in the federal government. This work documents and evaluates the first pilot of Agile Corps at the Department of Labor's Wage and Hour Division (DOL WHD).
The pilot reinforced the advantages PSL has over traditional service learning programs. First, because PSL participants are adult government workers, service projects do not need to be found; they are already inherently part of their work. Second, even though a PSL program is designed to focus on educational benefits of the learner (public servant), PSL naturally creates a balance between benefits to the public servant (learner) and benefits to the public community. We hope to measure the benefits to the community in continued, future work.
The pilot also highlighted the applicability of PSL to schools, colleges, and universities. First, PSL can be used for training of school or university staff or pedagogical training for faculty to achieve organizational culture change. The school or university culture change can be related to any change, or, more specifically, the change can be the catalyst of service-learning through most school and university courses. In this "service-learning as pedagogical training" framework, service-learning is the faculty training and means to increase student service-learning in the courses. Second, schools and universities can apply the PSL model to increase the duration of service-learning programs beyond the duration of courses. Third, schools and universities can move from balancing service goals with individual learning to balancing service goals with class or organizational learning. In this model, schools and universities can enter into long-term service partnerships with community organizations, switching and substituting students with the start of each new quarter, trimester, or semester. As students change, the documentation of organizational learning is important for incoming students, elevating the class or organizational learning over the individual learning. The organizational learning goal is to learn how to implement the service project or work (often of a partner organization) in such a way as to improve, surpass, or achieve intended impact more effectively or efficiently. Fourth, schools and universities can apply the Social Kirkpatrick Plus evaluation model in order to equitably elevate the social impact goals of the service with the student learning goals through the true measurement of social impact.
In this work, we presented the use of a human-centered design (HCD) methodology to guide the customization research, design, and implementation of the Agile Corps pilot at the DOL WHD in Washington, DC. Through the customization research, the pilot was designed to include two 1.5-hour class sessions each week, weekly office hours (two hours), and an hour of weekly mentoring. The pilot included a one-day training for executive leaders as well as service project assignments for participants. Recap emails were sent after each class, and retrospectives were held weekly. After experiencing technical difficulties, the facilitators switched to a fully online model to make the experience equivalent for DOL participants in Washington, DC and those in other locations in the US. The facilitators also worked to find service projects for participants who joined without one. Lastly the facilitators found mentors for each person and worked to facilitate mentoring relationship issues when any mentor was not meeting regularly with their mentee.
An independent auditor evaluated the pilot after its conclusion. Qualitative learner satisfaction results were presented. Overall, participants described a positive impact of the Agile Corps program which affected a large change in the perspectives of participants. Participants demonstrated competency with the pilot's core concepts and principle methods. Participants became conversant with words and phrases often used by technologists. Most participants described Agile Corps as empowering. All participants found the instructors and teaching to be good or excellent and were satisfied with outcomes. Participants had mixed feelings about project selection, project prioritization, the duration of class, the organization of topics, and the helpfulness of mentorship.
Quantitative learning achievement results were also presented. A competency-based, mastery-based model was utilized during this pilot. All participants demonstrated at least the minimum competency in all learning objectives, mindsets, and principles.
The next steps are to measure the behavior change, organizational impact, and external social impact of the PSL program at the DOL. To sustain behavior change after Agile Corps ends and ensure ultimate social impact upon labor rights and wage equity, the DOL WHD leadership asked 18F what additional measures should be implemented. Nine suggestions were made.
• Have goal-setting workshops every six months for each project.
• Assign a Product Owner to each project and empower them.
• Set up a Kanban Board at the National Office.
• Have monthly demo days.
• Have Agile Corps participants meet together quarterly to reflect and brainstorm.
• For project teams, encourage daily standups and retrospectives.
• Focus on training teams to use the right collaboration tools.
• Encourage that all meetings be video chat by default.
Lastly, we will run larger Agile Corps pilots using the learnings from the DOL WHD pilot. We will continue to test the model and measure the impact of Agile Corps on the governmentwide problem of a lack of junior and mid-level technologists in the government with career pathways.