2016 Federal Index
Evaluation & Research
Did the agency have a senior staff member(s) with the authority, staff, and budget to evaluate its major programs and inform policy decisions affecting them in FY16?
Score
9
9
Administration for Children and Families (HHS)
- ACF’s evaluation policy, established in 2012, addresses the principles of rigor, relevance, transparency, independence, and ethics and requires ACF program, evaluation, and research staff to For example, the policy states, “ACF program offices will consult with OPRE in developing evaluation activities.” And, “There must be strong partnerships among evaluation staff, program staff, policy-makers and service providers.”
- ACF’s Office of Planning, Research, and Evaluation (OPRE) proposes an evaluation plan to the Assistant Secretary each year in areas in which Congress has provided authority and funding to conduct research and evaluation.
- ACF’s annual portfolio reviews describe recent work and ongoing learning agendas in the areas of family self-sufficiency, child and family development, and family strengthening, including work related to child welfare, child care, Head Start, Early Head Start, strengthening families, teen pregnancy prevention and youth development, home visiting, self-sufficiency, welfare and employment. Examples include findings from Head Start CARES; the BIAS project; multiple reports from the first nationally representative study of early care and education in over 20 years; early findings on the Maternal, Infant and Early Childhood Home Visiting program; and a report on challenges and opportunities in using administrative data for evaluation.
- ACF’s evaluation policy requires that “ACF will release evaluation results regardless of findings… Evaluation reports will present comprehensive findings, including favorable, unfavorable, and null ACF will release evaluation results timely – usually within two months of a report’s completion.” ACF has publicly released the findings of all completed evaluations to date. In 2015, OPRE released nearly 120 publications.
Score
8
8
Corporation for National and Community Service
- CNCS has an evaluation policy that presents 5 key principles that govern the agency’s planning, conduct, and use of program evaluations: rigor, relevance, transparency, independence, and ethics.
- CNCS has an evaluation plan/learning agenda that is updated annually based on input from agency leadership as well as from emerging evidence from completed studies. This agenda was shared with the CNCS Board in 2015 and is reflected in the CNCS Congressional Budget Justification for Fiscal Year 2016 (pp. 55-56) and Fiscal Year 2017 (pp. 5-6, 55-56). CNCS’s R&E Office is currently developing scopes of work and will meet with program officers in April 2016 to discuss them.
- CNCS creates four types of reports for public release: research reports produced directly by research and evaluation staff, research conducted by third party research firms and overseen by research and evaluation staff, reports produced by CNCS-funded research grantees, and evaluation reports submitted by CNCS-funded program grantees. All reports completed and cleared internally are posted to the Evidence Exchange. CNCS expects to release 34 additional reports in FY16, and all evaluations are expected to be cleared.
- In FY16 CNCS developed Evaluation Core Curriculum Courses, which are presented to its grantees through a webinar series and are available on the CNCS website along with other evaluation resources. The courses are designed to help grantees and other stakeholders easily access materials to aid in conducting or managing program evaluations.
Score
9
9
Millennium Challenge Corporation
- MCC has developed a Policy for Monitoring and Evaluation of Compacts and Threshold Programs in order to ensure that all programs develop and follow comprehensive Monitoring & Evaluation (M&E) plans that adhere to MCC’s The monitoring component of the M&E Plan lays out the methodology and process for assessing progress towards Compact (i.e., grant) objectives. It identifies indicators, establishes performance targets, and details the data collection and reporting plan to track progress against targets on a regular basis. The evaluation component identifies and describes the evaluations that will be conducted, the key evaluation questions and methodologies, and the data collection strategies that will be employed. Pursuant to MCC’s M&E policy, every project must undergo an independent evaluation and analysis to assess MCC’s impact. Once evaluation reports are finalized, they are published on the MCC Evaluation Catalog. To date, fifty-three interim and final reports have been publicly released, with several additional evaluations expected to be completed and released in the coming months. MCC also produces periodic reports for internal and external consumption on results and learning, and holds agency-wide sessions that help to translate evaluation results into lessons learned for future compact development. Finally, in February 2016, MCC launched “NEXT: A Strategy for MCC’s Future” which outlines new strategic directions on how it will invest more in strengthening feedback systems to harness this learning for ongoing adaptation of design and implementation, both for its own effectiveness and for the benefit of country partners and others in the development community. NEXT is designed to be a five-year strategic plan for MCC, but also includes MCC’s learning agenda by incorporating agency-wide learning and knowledge goals to be pursued within that timeframe
Score
8
8
U.S. Agency for International Development
- USAID has an agency-wide Evaluation Policy. The agency just released a report to mark the five-year anniversary of the policy.
- USAID field missions are required to have an evaluation plan, and all USAID missions and offices provide an internal report on an annual basis on completed, ongoing and planned evaluations, including evaluations planned to start anytime in the next three fiscal USAID provides a Performance Management Plan (PMP) Toolkit to assist missions worldwide.
- Given USAID’s decentralized structure, individual programs, offices, bureaus and missions may develop learning agendas, which several have done, including the USAID’s Bureau for Food Security for the US government’s Feed the Future initiative and USAID’s Democracy, Human Rights, and Governance (DRG) Center. All Washington Bureaus have annual evaluation action plans that look at quality and use and identify challenges and the priorities for the year ahead.
- All final USAID evaluation reports are available on the Development Experience Clearinghouse except for approximately five percent of evaluations completed each year that are not public due to principled exceptions to the presumption in favor of openness guided by OMB Bulletin 12-01 Guidance on Collection of U.S. Foreign Assistance Data.
- USAID is currently updating its operational policy for planning and implementing country programs. A key change in the policy is that missions will include a learning plan as part of their five-year strategic plan, also known as the CDCS. The plan will outline how missions will incorporate learning into their programming, including activities like regular portfolio reviews, evaluation tracking and dissemination plans, and other analytic processes to better understand the dynamics of their programs and their country contexts.
Score
8
8
U.S. Department of Education
- ED’s Institute of Education Sciences (IES) supports research and conducts evaluations of ED’s major IES’ evaluation policies are set by the IES Standards and Review Office, addressing issues of scientific quality, integrity, and timely release of reports. Related, the National Board for Education Sciences, IES’s advisory board, has approved policies for Peer Review, which are implemented by the Standards and Review Office.
- EPG works with program offices and ED leadership on the development of ED’s annual evaluation This happens through the Department’s annual spending plan process and through identification of high priority evaluations for use of the pooled evaluation authority. IES and PPSS work with programs to design and share results from relevant evaluations that help with program improvement.
- ED’s current evaluations constitute its learning agenda.
- ED’s evaluations are posted on the IES website and the PPSS See FY15 Annual Performance Report and FY17 Annual Performance Plan for a list of ED’s current evaluations. IES publicly releases findings from all of its completed, peer-reviewed evaluations on the IES website and also in the Education Resources Information Clearinghouse (ERIC).
- ED’s supports research through IES’s National Center for Education Research (NCER), which makes grants for prekindergarten through postsecondary research and IES’ National Center for Special Education Research (NCSER), which sponsors a comprehensive program of special education research designed to expand the knowledge and understanding of infants, toddlers, children, and young adults with IES also manages the Regional Educational Laboratory (REL) program, which supports districts, states, and boards of education throughout the United States to use research in decision making.
Score
8
8
U.S. Dept. of Housing & Urban Development
- HUD’s evaluation policy (see 1–6, 21, 23), which guides HUD’s Research Roadmap described below, includes reaching out to internal and external stakeholders through a participatory approach; making research planning systematic, iterative, and transparent; focusing on research questions that are timely, forward-looking, policy-relevant, and leverage HUD’s comparative advantages and partnership opportunities; aligning research with HUD’s strategic goals; and using rigorous research methods including program demonstrations with randomized controlled trials as appropriate.
- HUD’s Office of Policy Development and Research (PD&R) has developed the Research Roadmap FY14-FY18, (see pp. 6-20) a strategic, five-year plan for priority program evaluations and research to be pursued given a sufficiently robust level of PD&R also integrated its evaluation plan into HUD’s FY14-FY18 Strategic Plan (see pp. 57-63) to strengthen the alignment between evaluation and performance management. During FY16, PD&R is using similar principles and methods to refresh the Roadmap to address emerging research topics.
- HUD also employs its role as convener to help establish frameworks for evidence, metrics, and future research.
- According to the Research Roadmap FY14-FY18, (see 28), as part of HUD’s annual performance report required by GPRA, “agencies should describe findings from agency-funded evaluations or other research completed during the prior fiscal year.” Further, “Agencies are expected to have a web page on the agency’s evaluations or links to other evaluations relevant to the agency’s work with summaries of the findings and specific citations.” PD&R publishes and disseminates evaluations in a timely fashion through these and other means, and also follows a policy of including language in research and evaluation contracts that allows researchers to independently publish results, even without HUD approval, after not more than 6 months.
Score
9
9
U.S. Department of Labor
- DOL has a formal Evaluation Policy Statement that formalizes the principles that govern all program evaluations in the Department, including methodological rigor, independence, transparency, ethics, and relevance. In addition, the Chief Evaluation Office publicly communicates the standards and methods expected in DOL evaluations in formal procurement statements of work.
- DOL also develops, implements, and publicly releases an annual Evaluation Plan (i.e., Department-level learning agenda), as do each of DOL’s 17 operating agencies. The agency learning agendas form the basis for the DOL’s Evaluation Plan. The 2016 Evaluation Plan was released for public comment in the Federal Register and is posted on the CEO website.
- All DOL reports and findings are publicly released and posted on the CEO website. DOL agencies also post and release their reports.