Visit NAP.edu/10766 to get more information about this book, to buy it in print, or to download it as a free PDF.
Suggested Citation:"Executive Summary." National Research Council. 1999. Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/9672.
The Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996 fundamentally changed the nation's social welfare system, replacing a federal entitlement program for low-income families, called Aid to Families with Dependent Children (AFDC), with state-administered block grants, the Temporary Assistance for Needy Families (TANF) program. PRWORA furthered a trend started earlier in the decade under so called "waiver" programs—state experiments with different types of AFDC rules—toward devolution of design and control of social welfare programs from the federal government to the states. The legislation imposed several new, major requirements on state use of federal welfare funds but otherwise freed states to reconfigure their programs as they want. The underlying goal of the legislation is to decrease dependence on welfare and increase the self-sufficiency of poor families in the United States.
In summer 1998, the Office of the Assistant Secretary for Planning and Evaluation (ASPE) of the Department of Health and Human Services (DHHS) asked the Committee on National Statistics of the National Research Council to convene a Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs. The panel's overall charge is to study and make recommendations on the best strategies for evaluating the effects of PRWORA and other welfare reforms and to make recommendations on data needs for conducting useful evaluations.
Under the broad charge, the panel is considering many evaluation issues, such as the proper mix of national-level and state-level evaluations; the appropriate roles of experimental and nonexperimental evaluation methodologies; and the importance of monitoring versus evaluation. The panel is also considering the
Suggested Citation:"Executive Summary." National Research Council. 1999. Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/9672.
needs for data from surveys and administrative records at both the federal and state levels, as well as general issues of data quality for studies of the low-income and welfare populations. The panel began its work in fall 1998 and will issue a final report in late 2000.
This interim report presents the panel's initial conclusions and recommendations. Given the short length of time the panel has been in existence, this report necessarily treats many issues in much less depth than they will be treated in the final report. The report has an immediate short-run goal of providing DHHS-ASPE with recommendations regarding some of its current projects, particularly those recently funded to study ''welfare leavers''—former welfare recipients who have left the welfare rolls as part of the recent decline in welfare caseloads.
Many of the conclusions reached by the panel in its initial examination of welfare reform evaluations under way around the country concern the data bases used for evaluation. The devolution of program responsibility begun under the pre-PRWORA waivers and institutionalized by PRWORA has led to wide variation in programs across states and even within states. While this proliferation of programs has some advantages from an evaluation standpoint—it makes use of federalism as a laboratory for testing (as it has been so often used in American history)—it imposes the need for a significant data infrastructure. Such an infrastructure should include national-level data sets that can capture state variations in policies and outcomes; state-level data sets that are sufficiently detailed and standardized to permit comparisons of welfare policies as implemented and family and individual outcomes (employment, income, etc); and state-level data sets for evaluating the effects of welfare reform in each state, even if not intended for comparison with other states. The panel has concluded that such a data infrastructure is not in place at the current time.
Gaps in the data infrastructure for determining the effects of welfare reform are numerous. The new welfare reform environment imposes great strain on traditional national-level survey data sets, both those supported by the Census Bureau as well as those supported by other federal agencies. These data sets usually have relatively small sample sizes for the welfare-relevant population, and they often do not have all the major variables needed to comprehensively assess the effects of welfare reform. The Survey of Program Dynamics, a national-level panel data set intended to remedy some of these defects, has so far not been as successful as expected. A further difficulty straining the usefulness of national-level surveys is the relative lack of information on the policies that states have actually adopted since 1996. The 1996 legislation not only devolved operational and design responsibility to the states, it also removed many requirements for state reporting to the federal government on their policies. Federal data-gathering activities on the details of state policies have been slow and hap-
Suggested Citation:"Executive Summary." National Research Council. 1999. Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/9672.
hazard, and it is only since early 1999 that an institutional structure has begun to be put in place for collecting such data on a permanent and long-run basis.
Program devolution has also constrained the ability of the federal government to mandate data collection or research to track the effects of the changes in social welfare programs required by the new legislation. Yet it is clear that the federal government and Congress will need a national-level assessment of how the new programs are working, especially to make informed decisions about the renewal of PRWORA. In addition, a national-level assessment containing results from all the states should be of interest to state policy makers, who are interested in how other state programs are working in comparison with their own and who are interested in what other states' efforts have been successful.
The new welfare reform environment also imposes burdens on state-level data sets. Most state-level data sets are obtained from administrative records that have historically been used for management, not evaluation, purposes. These data sets are being put under strain as states seek to develop and use them in more sophisticated ways in order to meet the legal requirements—for example, to keep counts of time limits as well as to satisfy federal reporting requirements on counts of recipients in different categories—and to conduct evaluations of their own state's policies. Considerable effort is required to convert these databases for evaluation and research uses, for which they were not originally intended. The use of these state-level databases for cross-state comparison research or, more ambitiously, as a substitute for national-level data sets, is further hindered by noncomparable variables and data constructs across states and by the same lack of comparable policy information that is hindering national-level surveys. State-level household surveys have been only rarely conducted in the past, but are now in great demand, and states are hurrying to build the capacity to conduct them.
The panel also reached some initial conclusions about evaluation methodology. Both national-level and state-level evaluations have roles to play in welfare reform evaluation, and neither should be pursued to the exclusion of the other. Moreover, cross-state comparability in state-level data sets and the development of comparable measures of state policies are important for both national-level and state-level evaluations. Although nonexperimental methods have become the dominant evaluation method for PRWORA, the panel believes that experimental methods should be kept on the table and still have a role to play in the future.
The panel also concludes that monitoring and evaluation both have great value, but that the distinction between the two has not always been sufficiently delineated and that evaluation should be the ultimate goal. Relatedly, the panel concludes that many existing studies do not have credible groups for which policy comparisons can be made and hence are of limited value as evaluation studies. Other studies use comparison groups that will require great care in reaching sound conclusions. Specifically, the panel concludes that welfare leaver studies have great value but represent only one group potentially affected by
Suggested Citation:"Executive Summary." National Research Council. 1999. Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/9672.
welfare reform; they miss those who were diverted or discouraged and even those families remaining on the welfare rolls.
Although the panel will be issuing a broad set of recommendations to address the many issues of methods and data for welfare reform evaluation in its final report, there are several immediate, short-run recommendations it wishes to make to DHHS-ASPE concerning some of its current sponsored activities. In particular, for the purposes of cross-state comparisons and national-level assessments, there are benefits, including economies of scale, for DHHS to take a leading role in obtaining data and guiding research. Several key areas in which guidance in data collection and research is needed, for which the department already has some initiatives under way, include: (1) identifying key policy issues over the next 3-5 years that should guide priorities for data collection and research; (2) taking steps to ensure that research addresses key populations of interest for social welfare policy analysis; (3) fostering improved capabilities for social welfare program-related data collection and research at the federal and state levels; (4) encouraging efforts to make data and research results comparable across jurisdictions; and (5) documenting state policies.
Because resources for data collection and research on social welfare programs are limited across all levels of government, it is important to set priorities so that key policy questions can be answered. While not all states will have the same specific policy concerns, there is likely a core set of issues that will be the focus of federal and state policy making over the next 3–5 years and on which it makes sense to concentrate scarce resources for data collection and research.
The department can take steps toward identifying critically important issues by maintaining an information-gathering operation that obtains input from a variety of sources, such as the Congress and other relevant federal departments (Agriculture, Housing and Urban Development, and Labor), state welfare agency staff, and state legislators and their staffs. Maintaining close relationships with interstate coordinating mechanisms that the states have already established would also be useful. The department already has many of these kinds of ties.
(1) The panel recommends that the Department of Health and Human Services be proactive in identifying important current and emerging issues for welfare policies at both federal and state levels that, in turn, can guide priorities for investment in data and research.
Suggested Citation:"Executive Summary." National Research Council. 1999. Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/9672.
