2016 Federal Index
Administration for Children and Families (HHS)
8
Leadership
Did the agency have a senior staff member(s) with the authority, staff, and budget to evaluate its major programs and inform policy decisions affecting them in FY16?
- In late FY15, ACF established the new career position of Deputy Assistant Secretary for Planning, Research, and Evaluation to oversee its Office of Planning, Research, and Evaluation (OPRE) and support evaluation and other learning activities across the agency. ACF’s budget for research and evaluation in FY16 is approximately $142 million. ACF’s evaluation policy gives the OPRE Director (now Deputy Assistant Secretary) “authority to approve the design of evaluation projects and analysis plans; and…authority to approve, release and disseminate evaluation reports.” OPRE’s staff of 42 includes experts in research and evaluation methods as well as ACF programs and policies and the populations they serve. OPRE engages in on-going collaboration with program office staff and leadership to interpret research and evaluation findings and to identify their implications for programmatic and policy decisions. OPRE also provides written summaries of emerging findings and holds monthly meetings with agency and program leadership to discuss their implications.
- While OPRE oversees most of ACF’s evaluation activity and provides overall coordination, some ACF program offices also sponsor evaluations. ACF’s evaluation policy states, “In order to promote quality, coordination and usefulness in ACF’s evaluation activities, ACF program offices will consult with OPRE in developing evaluation activities. Program offices will discuss evaluation projects with OPRE in early stages to clarify evaluation questions and methodological options for addressing them, and as activities progress OPRE will review designs, plans, and reports. Program offices may also ask OPRE to design and oversee evaluation projects on their behalf or in collaboration with program office staff.”
9
Evaluation & Research
Did the agency have an evaluation policy, evaluation plan, and research/ learning agenda(s) and did it publicly release the findings of all completed evaluations in FY16?
- ACF’s evaluation policy, established in 2012, addresses the principles of rigor, relevance, transparency, independence, and ethics and requires ACF program, evaluation, and research staff to For example, the policy states, “ACF program offices will consult with OPRE in developing evaluation activities.” And, “There must be strong partnerships among evaluation staff, program staff, policy-makers and service providers.”
- ACF’s Office of Planning, Research, and Evaluation (OPRE) proposes an evaluation plan to the Assistant Secretary each year in areas in which Congress has provided authority and funding to conduct research and evaluation.
- ACF’s annual portfolio reviews describe recent work and ongoing learning agendas in the areas of family self-sufficiency, child and family development, and family strengthening, including work related to child welfare, child care, Head Start, Early Head Start, strengthening families, teen pregnancy prevention and youth development, home visiting, self-sufficiency, welfare and employment. Examples include findings from Head Start CARES; the BIAS project; multiple reports from the first nationally representative study of early care and education in over 20 years; early findings on the Maternal, Infant and Early Childhood Home Visiting program; and a report on challenges and opportunities in using administrative data for evaluation.
- ACF’s evaluation policy requires that “ACF will release evaluation results regardless of findings… Evaluation reports will present comprehensive findings, including favorable, unfavorable, and null ACF will release evaluation results timely – usually within two months of a report’s completion.” ACF has publicly released the findings of all completed evaluations to date. In 2015, OPRE released nearly 120 publications.
7
Resources
Did the agency invest at least 1% of program funds in evaluations in FY16? (Note: Meeting this criteria requires both Agency and Congressional action.)
- In FY16, ACF plans to spend $138 million on evaluations, representing 3% of ACF’s $53 billion budget in FY16 (in addition to investments in evaluations by ACF grantees).
- The Administration’s FY17 budget request seeks authority for numerous new investments in learning, including set-asides of up to 5% of the Social Services Block Grant program ($18.5m in FY17, including $10 million for a demonstration and evaluation on supplying diapers to low-income families and $8.5 million for research and evaluation in FY17) and 1% of the Community Services Block Grant program ($3.5 million in FY17) for evaluations.
8
Performance Management / Continuous Improvement
Did the agency implement a performance management system with clear and prioritized outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY16?
- ACF’s performance management framework focuses on outcomes and aims for coordinated and results-oriented management and operations across all ACF ACF’s Strategic Plan establishes five priorities, which align with the HHS Strategic Plan. ACF formally reviews progress toward its Strategic Plan goals every quarter. ACF’s Strategic Plan establishes five priorities in support of the agency’s mission of fostering health and well-being by providing federal leadership, partnership, and resources for the compassionate and effective delivery of human services. The five priorities are: 1) Promote economic, health, and social well-being for individuals, families, and communities; 2) Promote healthy development and school readiness for children, especially those in low-income families; 3) Promote safety and well-being of children, youth, and families; 4) Support underserved and underrepresented populations; and 5) Upgrade the capacity of ACF to make a difference for families and communities.
- ACF aims to develop performance measures that are meaningful and can be used by program managers, leadership, outside stakeholders, and Congress to assess and communicate progress. Results for these metrics are reported annually in the ACF Congressional Budget Justification. ACF reports on a total of 156 performance measures (94 outcome measures and 62 output measures) in the FY17 Congressional Budget A selection of ACF performance measures is also highlighted as part of the FY 2017 HHS Annual Performance Plan and Report, which describes HHS’ progress toward achieving the goals and objectives described in the FY 2014-2018 HHS Strategic Plan. This report includes the most recent results available at the end of FY15 for HHS, including ACF.
- As part of the FY17 President’s Budget request, HHS announced the FY16-17 HHS Priority Goals. ACF is the lead agency for the goal to “Improve the quality of early childhood programs for low-income children” in collaboration with the Substance Abuse and Mental Health Services Administration (SAMHSA) and the Health Resources and Services Administration (HRSA). On a quarterly basis, ACF provides updates on this Priority Goal on Performance.gov. ACF also participates in the GPRAMA-required Strategic Objective Annual Review process. HHS maintains an internal performance dashboard where ACF provides regular performance updates on the 156 performance measures included in the annual ACF Budget Request.
9
Data
Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) federal, state, and local programs in FY16?
- ACF has made numerous administrative and survey datasets publicly available for secondary use, such as data from the National Survey of Early Care and Education, Child Care and Development Fund, National Survey of Child and Adolescent Well-Being, and Adoption and Foster Care Analysis and Reporting System, among many other examples.
- ACF’s Interoperability Initiative supports data sharing through policies and guidelines to accelerate adoption; standards and tools that are reusable across the country; field-based pilots; and addressing common privacy and security requirements to mitigate risks.
- Several ACF divisions have also been instrumental in supporting cross-governmental efforts, such as the National Information Exchange Model (NIEM) that will enable human services agencies to collaborate with health, education, justice, and many other constituencies that play a role in the well-being of children and families.
- ACF’s National Directory of New Hires has entered into data sharing agreements with numerous For example, DOL’s CEO and ETA have interagency agreements with HHS-ACF for sharing and matching earnings data on 9 different formal net impact evaluations. The NDNH Guide for Data Submission describes an agreement with the Social Security Administration to use its network for data transmission. Also, ACF Administers the Public Assistance Reporting Information System, a platform for exchange of data on benefits receipt across ACF, Department of Defense, and Veterans Affairs programs. This platform entails data sharing agreements between these three federal agencies and between ACF and state agencies.
- The Administration’s FY17 budget request includes $261 million over five years for human services data interoperability, including grants for Statewide Human Services Data Systems and a Systems Innovation Center.
9
Common Evidence Standards / What Works Designations
Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding decisions and did it disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY16?
- ACF has established a common evidence framework adapted for the human services context from the framework for education research developed by the Department of Education and the National Science Foundation. The ACF framework, which includes the six types of studies delineated in the ED/NSF framework, aims to (1) inform ACF’s investments in research and evaluation, and (2) clarify for potential grantees and others the expectations for different types of studies.
- ACF maintains an online clearinghouse of evidence reviews of human services These reviews rate the quality of evaluation studies using objective standards vetted by technical experts and applied by trained, independent reviewers, and similar to those used by other agencies such as the U.S. Department of Education’s What Works Clearinghouse and the U.S. Department of Labor’s CLEAR. The clearinghouse includes results of the reviews in a searchable format as well as comprehensive details about the review standards and process. Reviews to date have covered teen pregnancy prevention; home visiting; relationship education and responsible fatherhood; and employment and training; and include both ACF- sponsored and other studies.
8
Innovation
Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY16?
- ACF’s Behavioral Innovations to Advance Self-Sufficiency (BIAS) project is the first major effort to apply a behavioral economics lens to programs that serve poor families in the U.S. Since its inception in 2010, the project has conducted 15 rapid-cycle randomized tests of behavioral innovations in seven states with nearly 100,000 sample members.
- ACF’s Behavioral Interventions for Child Support Services (BICS) demonstration project is applying behavioral insights to child support contexts, developing promising behavioral interventions, and building a culture of regular, rapid-cycle evaluation and critical inquiry within the child support community.
- The Administration’s FY17 budget request (p. 347) proposes to repurpose the Temporary Assistance Contingency Fund for a targeted set of approaches to reducing poverty and promoting family economic security. These include demonstration projects to improve parental employment outcomes concurrently with child and family wellbeing outcomes; subsidized employment programs; and program improvement initiatives, such as monitoring and oversight, technical assistance, and research and the proposed demonstration programs would set aside funds for evaluation.
- ACF has actively participated in the HHS IDEA Lab, an entity within HHS launched in 2013, to invest in internal innovation, leverage external innovation, and build collaborative communities to tackle cross-cutting issues of strategic importance. Current projects include the ACF Administration for Native Americans’ Application Toolkit and DataQuest: Making ACF Native Data Visible and Useful, the ACF Office of Family Assistance’s Understanding Temporary Assistance for Needy Families Through Data Vizualization, and the ACF Office of Head Start’s Partnership Alignment Information Response System.
- ACF is participating in the Performance Partnership Pilots for Disconnected Youth initiative by providing flexibility for grantees to join partnerships at the state level. In addition ACF staff have served as technical and evaluation reviewers for selecting the Round 1 pilot sites, participated in the flexibility review process, and contributed evaluation expertise to planning for local and national evaluations of pilot sites.
7
Use of Evidence in 5 Largest Competitive Grant Programs
Did the agency use evidence of effectiveness when allocating funds from its 5 largest competitive grant programs in FY16?
- In FY16 the 5 largest competitive grant programs are: 1) Head Start ($9,168,095,000); 2) Unaccompanied Children Services ($948,000,000); 3) Early Head Start-Child Care Partnerships ($915,799,422); 4) Transitional and Medical Services ($490,000,000); and 5) Preschool Development Grants ($250,000,000).
- ACF’s template (see 14 in Attachment C) for grant announcements includes two options, requiring grantees to either 1) collect performance management data that contributes to continuous quality improvement and is tied to the project’s logic model, or 2) conduct a rigorous evaluation for which applicants must propose an appropriate design specifying research questions, measurement and analysis.
- In FY12, ACF established the Head Start Designation Renewal System requiring Head Start ($9.2 billion in FY16) grantees to compete for grants moving forward if they failed to meet criteria related to service quality, licensing and operations, and fiscal and internal control.
- ACF’s Personal Responsibility Education Program ($75 million in FY16) includes three individual discretionary grant programs that support evidence-based competitive grants that teach youth about abstinence and contraception to prevent pregnancy and sexually transmitted infections.
- To receive funds through ACF’s Community Based Child Abuse Prevention (CBCAP) program, states must “demonstrate an emphasis on promoting the increased use and high quality implementation of evidence-based and evidence-informed programs and practices.” CBCAP defines evidence-based and evidence-informed programs and practices along a continuum with four categories: Emerging and Evidence-Informed; Promising; Supported; and Well Supported. Programs determined to fall within specific program parameters will be considered to be “evidence informed” or “evidence-based” practices (EBP), as opposed to programs that have not been evaluated using any set criteria. ACF monitors progress on the percentage of program funds (most recently 89.4% in FY14) directed towards evidence-based and evidence-informed practices.
7
Use of Evidence in 5 Largest Non-Competitive Grant Programs
Did the agency use evidence of effectiveness when allocating funds from its 5 largest non-competitive grant programs in FY16? (Note: Meeting this criteria requires both Agency and Congressional action.)
- In FY16, ACF’s 5 largest non-competitive grant programs are: 1) Temporary Assistance for Needy Families ($17,345,407,000); 2) Child Care and Development Fund (Block Grant and Entitlement to States combined) ($5,678,000,000); 3) Foster Care ($4,799,573,280); 4) Child Support Enforcement Payments to States ($4,303,998,000); and 5) Low Income Home Energy Assistance ($3,390,304,000).
- ACF’s Foster Care program ($4.8 billion in FY16) has approved over 30 jurisdictions to develop and implement child welfare waiver demonstration projects to improve outcomes for children in foster care or at risk for entry or re-entry into foster care. Through these demonstrations, ACF waives provisions of the law to allow flexible use of funding normally limited to foster care for other child welfare services. Many participating jurisdictions are implementing evidence-based or evidence-informed interventions and all demonstration projects are required to have a rigorous evaluation conducted by a third-party evaluator. Although ACF does not currently have statutory authority to grant new waivers, current projects are expected to continue through September 30, 2019. General information on this program, including a fact sheet and summary of relevant legislation/policy is available at the online Children’s Bureau portal.
8
Repurpose for Results
In FY16, did the agency shift funds away from any practice, policy, or program which consistently failed to achieve desired outcomes? (Note: Meeting this criteria requires both Agency and Congressional action.)
- In FY12, ACF established the Head Start Designation Renewal System, requiring Head Start ($9.2 billion in FY16) grantees to compete for grants moving forward if they failed to meet criteria related to service quality, licensing and operations, and fiscal and internal controls. The 2007 Head Start Reauthorization Act made all Head Start grants renewable, five-year grants. At the end of each five-year term, grantees that are running high-quality programs will have their grants renewed. But grantees that fall short of standards are now required to compete to renew grants. Grantees whose ratings on any of the three domains of the Classroom Assessment Scoring System, an assessment of adult:child interactions linked to improved outcomes, fall below a certain threshold, or in the lowest 10 percent of grantees, must also compete.
- ACF’s FY17 budget request (p. 418) proposes to eliminate funding for Abstinence Education grants because the program is not focused on funding evidence-based models.