Did the agency have a senior staff member(s) with the authority, staff, and budget to evaluate its major programs and inform policy decisions affecting them in FY16?

• CNCS’s Office of Research and Evaluation Director (R&E) oversees the development of social science research designed to measure the impact of CNCS programs and shape policy decisions; encourage a culture of performance and accountability in national and community service programs; provide information on volunteering, civic engagement, and volunteer management in nonprofit organizations; and assist in the development and assessment of new initiatives and demonstration projects. The R&E Director, who is overseeing R&E’s $4 million budget and a staff of 9 in FY16, is a member of CNCS’s Leadership Team and Policy Council. The R&E Director also meets regularly with CNCS Program Directors to identify areas where evidence can be generated and used for various decisions.
• The R&E Director meets annually with all CNCS program offices to identify priorities and negotiate which pools of funds are need to support the year’s priorities. The FY16 plan was developed through a series of formal and informal conversations.

Did the agency have an evaluation policy, evaluation plan, and research/ learning agenda(s) and did it publicly release the findings of all completed evaluations in FY16?

• CNCS has an evaluation policy that presents 5 key principles that govern the agency’s planning, conduct, and use of program evaluations: rigor, relevance, transparency, independence, and ethics.
• CNCS has an evaluation plan/learning agenda that is updated annually based on input from agency leadership as well as from emerging evidence from completed studies. This agenda was shared with the CNCS Board in 2015 and is reflected in the CNCS Congressional Budget Justification for Fiscal Year 2016 (pp. 55-56) and Fiscal Year 2017 (pp. 5-6, 55-56). CNCS’s R&E Office is currently developing scopes of work and will meet with program officers in April 2016 to discuss them.
• CNCS creates four types of reports for public release: research reports produced directly by research and evaluation staff, research conducted by third party research firms and overseen by research and evaluation staff, reports produced by CNCS-funded research grantees, and evaluation reports submitted by CNCS-funded program grantees. All reports completed and cleared internally are posted to the Evidence Exchange. CNCS expects to release 34 additional reports in FY16, and all evaluations are expected to be cleared.
• In FY16 CNCS developed Evaluation Core Curriculum Courses, which are presented to its grantees through a webinar series and are available on the CNCS website along with other evaluation resources. The courses are designed to help grantees and other stakeholders easily access materials to aid in conducting or managing program evaluations.

Did the agency invest at least 1% of program funds in evaluations in FY16? (Note: Meeting this criteria requires both Agency and Congressional action.)

• CNCS plans to spend a total of $5.1 million (representing .46% of CNCS’s $1.1 billion budget in FY16) in evaluation and evaluation capacity building activities (R&E evaluation and program funds combined), including:
  • $1.2 million of Senior Corps funding for the supplemental award, program funding used for evaluation and evidence purposes versus funding given to sponsor organizations;
  • $400,000 in Senior Corps funds for the longitudinal survey in FY16, an evaluation of the volunteers who participate in Senior Corps programs (examining their health and well-being outcomes over time);
  • $500,000 in supporting reviews of grantee evaluation plans and reports, including for research & evaluation expertise to review studies submitted by grantees applying for funding; and
  • $3 million in evaluation funds and SIF funds to support program evaluations and technical assistance for grantees to conduct evaluations.

Did the agency implement a performance management system with clear and prioritized outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY16?

• CNCS’s performance management framework is described in the Congressional Budget Justification for Fiscal Year 2016 (p.3) and Fiscal Year 2017 (p.6).
• CNCS has a focused set of Agency-Wide Priority Measures derived from the 2011-2015 Strategic Plan. Every CNCS Program contributes to the Agency-Wide Priority Measures. There are also specific grantee/sponsor measures that roll up into the Agency-Wide Priority Measures, which can be found in the Agency-Wide Priority Measures chart. Grantees are required to select at least one national performance measure, and they are required to report performance measures data. CNCS encourages grantees to use these measure for continuous program improvement. CNCS uses the agency-wide priority measures to assess its own progress toward attaining the goals and objectives of its strategic plan.
• Additionally, CNCS produces state profile reports, which provide a picture of agency resources in each state at a given point. These reports contain a number of priority indicators, including the number of participants engaged in national service activities as well as the amount of non-CNCS resources generated by the agency’s programs. Along with its stakeholders, CNCS uses this information to understand the capacity of service available in different geographic regions and discuss related implications with key service partners.
• CNCS’s Chief Operating Officer (COO) is currently piloting a proof of concept performance framework that aligns with GPRA. The COO is finalizing their objectives, measures, and targets, and they will be conducting quarterly performance reviews starting in the fourth quarter of FY16. The goal is to establish an effective performance framework within the COO, work agency-wide to implement a similar process, and have an enhanced performance management framework in place as CNCS begins its new Strategic Planning process in 2017.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) federal, state, and local programs in FY16?

• As the nation’s largest grant maker for service and volunteering, CNCS collects data about service program members, volunteers, and the organizations in which members and volunteers are serving. Member/volunteer demographic, service experience, and outcome data are collected in a variety of ways – both through administrative processes and through surveys:
  • In FY16 data collected from a revised member exit survey allowed CNCS to generate more accurate reports on key experiences and anticipated college, career, and civic engagement outcomes, which were shared internally. Survey results are being shared with program and agency leadership in FY16 for program improvement. In FY16 R&E will also begin generating state-level reports for its State Commissions. The longer-term goal is to finalize response rate standards across the AmeriCorps programs so that data sets can be made available for public use in FY17.
  • A report summarizing cross-sectional survey findings on Senior Corps Foster Grandparents and Senior Companion Program volunteers will be released in FY16. The paper compares health, mobility disability, and life satisfaction between participants in both programs; and examines how their health status differs from similar adult volunteers and non-volunteers in the general population (a matched sample of volunteers and non-volunteers from the Health and Retirement Study (HRS). A longitudinal survey of volunteers in these 2 Senior Corps programs was implemented in FY15, and preliminary findings are expected in FY16.
  • For the first time, results from the redesigned AmeriCorps member exit survey were merged with administrative data sets on member demographics, program characteristics, and service locations to produce a new unified data set that currently has almost 70,000 observations. Analysis began in FY16, and preliminary findings are expected by the end of FY16.
  • Findings from an alumni outcome survey pilot were published in FY16.
• In FY16, CNCS’s R&E Office is executing a new administrative data match between a sample of AmeriCorps alumni records and postsecondary outcome data from the National Student Clearinghouse. R&E also plans to execute a second administrative data match between alumni records and the Census’ LEHD dataset to obtain employment and employment sector outcomes for AmeriCorps alumni. Although R&E currently relies on surveys, CNCS would prefer to reduce its reliance on this method so that key college and career outcomes can be obtained from more objective sources and for less cost.
• CNCS’s Office of Research and Evaluation (R&E) makes publicly available (1) state profiles that depict national service resources (grant funds, members, volunteers, grantees) and program performance metrics across the country and (2) volunteering statistics at the local, state, and national levels collected for CNCS by the U.S. Census Bureau through an interagency agreement. (https://www.volunteeringinamerica.gov/)

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding decisions and did it disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY16?

• CNCS’s Office of Research and Evaluation (R&E) Office is actively involved with 3 other federal agencies in the interagency Common Evidence Framework working group in order to ensure consistency in definitions and use of evidence standards in grant-making. CNCS uses the Cross-agency  Federal Evidence Framework for evaluation planning and dissemination.
• CNCS also adapted the evidence framework used by its Social Innovation Fund and the Investing in Innovation Fund at ED and included it as part of the AmeriCorps State and National program’s FY16 grant competition. The evidence framework used in the FY16 AmeriCorps competition was revised from FY15 to make it more consistent with what is used in other federal agencies.
• In March 2015, CNCS released Phase I of the CNCS Evidence Exchange, a virtual repository of reports intended to help CNCS grantees and other interested stakeholders find information about evidence- and research- based national service and social innovation programs. Phase 1 includes a database of single study reports with some additional descriptive information about the study, as well as a systematic review of the national service evidence base. Phase 2 in FY16 added studies as grantees completed their independent evaluations and submitted reports to CNCS.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY16?

• CNCS remains a partner in the Performance Partnership Pilot (P3) program and has contributed to the national evaluation of this program. R&E estimates that one pilot site is leveraging CNCS funding through P3.
• CNCS awarded 10 grants that launched in FY16 as part of a new grantmaking initiative called Operation AmeriCorps. This initiative was designed to encourage tribal and local leaders to identify a high-priority local challenge that AmeriCorps State and National, AmeriCorps NCCC, and/or AmeriCorps VISTA members can holistically address in a relatively short period of time (no more than 2 years). This grantmaking initiative is innovative for CNCS as it is the first grant program that requires the blending of resources from different AmeriCorps programs – which usually operate separately – to create a new transformative service solution. In addition to requiring a blended service model, the grant program streamlined the application process and facilitated an internal examination of ways the agency’s business processes can be improved. CNCS’s R&E Office is conducting a two-year process evaluation of Operation AmeriCorps, which is designed to provide more formal findings on the extent to which the goals of the initiative were achieved.

Did the agency use evidence of effectiveness when allocating funds from its 5 largest competitive grant programs in FY16?

• CNCS is operating three competitive grant programs in FY16: 1) AmeriCorps State and National program (excluding State formula grant funds) ($386 million in FY16); 2) Senior Corps RSVP program ($49 million in FY16); and 3) the Social Innovation Fund (SIF) ($50 million in FY16).
• SIF provides competitive grants to non-profit grant-making organizations to help them grow promising, evidence-based solutions that address pressing economic opportunity, healthy futures, and youth development issues in low- income communities. The FY14-16 Omnibus Appropriations Acts have allowed CNCS to invest up to 20% of SIF funds each year in Pay for Success initiatives. There are 2 Pay for Success competitions planned for FY16, which will invest both the FY15 and 16 appropriations (approximately $11.6 million at minimum).
• CNCS’s AmeriCorps State and National Grants Program (excluding State formula grant funds), application (see 10-14) allocated up to 27 points out of 100 to organizations that submit applications supported by performance and evaluation data in FY16. Specifically, up to 15 points can be assigned to applications with theories of change supported by relevant research literature, program performance data, or program evaluation data; and up to 12 points (a 4 point increase from FY15) can be assigned for an applicant’s incoming level of evidence with the highest number of points awarded to strong levels of evidence. These categories of evidence are modeled closely on the levels of evidence defined in the Social Innovation Fund.
• In FY16, Senior Corps RSVP grantees seeking funding (see p. 1) through the administrative renewal process are encouraged to fulfill the National Performance Measures requirement by committing a certain number of volunteers to serve in an evidence-based health education program. A total of $500,000 (just above 1% of program funds) is allocated to support organizations in implementing evidence-based interventions or to evaluate programs.

Did the agency use evidence of effectiveness when allocating funds from its 5 largest non-competitive grant programs in FY16? (Note: Meeting this criteria requires both Agency and Congressional action.)

• CNCS operates one formula grant program, the AmeriCorps State formula grants program ($130 million in FY16). CNCS also operates four direct grant programs in FY16: 1) National Civilian Community Corps (NCCC) ($30 million in FY16), 2) VISTA ($92 million in FY16), 3) Senior Corps Foster Grandparents ($108 million in FY16), and 4) Senior Corps Senior Companion Program ($46 million in FY16).
• In FY16, for the first time, the Senior Corps Foster Grandparents and Senior Companion programs embedded evidence into their grant renewal processes by offering supplemental funding to grantees interested in deploying volunteers to serve in evidence-based programs (see 2-4) and providing evaluation data on implementation fidelity, including outcomes. A total of $400,000.00 is allocated for the Foster Grandparents program in FY16, or .4% of program funds. A total of $300,000.00 is allocated for the Senior Companion program in FY16, or .65% of program funds.
• VISTA is currently developing a theory of change that will make explicit the link between the work that the volunteers perform, the design of a sponsor’s project to address community needs, and the evidence to support this activity. This effort will impact several management aspects including project approval, volunteer assignment descriptions, member activity, data collection, and the role of evidence in the design and implementation of projects.

In FY16, did the agency shift funds away from any practice, policy, or program which consistently failed to achieve desired outcomes? (Note: Meeting this criteria requires both Agency and Congressional action.)

• In FY13-FY14, Mile High United Way, a grantee of the Social Innovation Fund (SIF), ended funding relationships with 3 of its sub-grantees who were not able to conduct rigorous evaluations of their In FY15, United Way for Southeastern Michigan, also a SIF grantee, ended its funding relationship with one of its sub-grantees for the same reason. These actions are consistent with the SIF National Assessment findings, which recognize the role SIF has played in fostering evidence-based grant making among its grantees.