Performance
Accountability in the Workforce Investment Act: An Application with
Division of Vocational Rehabilitation Data Part Two
by:
Tony Glover, Senior Analyst
"The problem with employment evaluation research lies in isolating the economic factors that influence the population, control group and the performance of clients in a training program."
P
art One of this series addressed the
performance management principles of the "Performance Accountability
System" specified by the Workforce Investment Act (WIA)1 using Division of Vocational Rehabilitation
(DVR) program data. Where performance management techniques focus on
the measurement of some value (i.e., earnings) before and after training
for a group of clients, they fail to
account for external factors that influence program outcomes. By
contrast, when the design of the analysis shifts to performance
evaluation, it means we also measure the earnings for representative
groups of individuals over the same period and within the same conditions
as the client group to assess the impact of a training program. These
control groups (theoretically) represent all of the contextual changes
affecting the client group as well as such things as maturation. Part Two examines
evaluation methods, bringing into focus the shortcomings of exclusive
reliance on management techniques and showing the value of a combined
approach.
This article finds that two of the three core indicators for workforce
investment activities, from Part One--entered
employment rate and earnings gained in
employment--are sensitive to influences from surrounding economic
conditions. Economic conditions directly affect the opportunity
structure of the labor market, impacting prospects of finding and
retaining employment and increasing earnings. Thus, interpreting the
performance outcomes of a workforce training program requires an
awareness of local economic conditions.
This study contrasts DVR clients with two groups, a matched control group and a population group.
Simply defined, a matched control group is a representative subset of
the population similar to DVR clients
on a number of characteristics (i.e., age, sex, prior income) that
operates in the same environment as DVR clients. The population group
represents the activity of the Wyoming labor force in the context of
surrounding conditions.
A true population criterion design would designate a control group from
the population that does not differ significantly from the client group
analyzed. However, we lack an indicator to select a control group
mirroring the DVR population on the variable of disability. Therefore,
this method does not lend itself to a true impact analysis of whether and
to what extent the DVR program affects its clients. However, building a
longitudinal baseline of the differences between the DVR and control
groups permits us to assess future changes in the DVR program itself.
Differences in labor market outcome between DVR clients, the control
group, and the population discussed in the next section, are consistent
across time. The hypothetical examples shown introduce the idea that
success of a program is discovered through changes in the differences
between cohorts (common groups of
individuals) of the clients, control and population groups.
Consider a hypothetical situation from Table 1,
where for the past seven years, DVR clients’ entered employment rate on
average fell 10 percent below (plus or minus 2 percent) that of the
control group. If in the eighth year the program’s performance declines
to 20 percent below the earnings of the control group, consequently
falling outside the eight to 12 percent range (explained by DVR clients’
disabilities), the performance of the program should be investigated to
assess changes in program administration or other internal factors.
In the three hypothetical examples (Figures 1,
2 and 3), each point on a
line corresponds to a cohort’s performance on some indicator for that
respective year; for example, the entered employment rate discussed in
Part One. These examples contrast DVR clients’ program performance with
a proportionally matched control group, displaying the relationship
between performance and economic factors affecting the entire labor
market.
Figure 1: DVR clients perform consistent with the economy in
this example (as represented by the control group). Knowing the
performance of DVR clients, without knowing how the environment and
similarly situated individuals are changing, does not allow us to
determine whether the performance decreases occurring after 1994
stemmed from changes in the program or economic conditions.
Figure 2: The difference between DVR and the control group’s
performance decreases over time. This positive result shows continuous
improvement. Although DVR clients show a performance level below the
control group, the example illustrates DVR performance consistent with
the economy.
Figure 3: This example represents a worst case scenario. The
program displays a continuous decline in DVR performance relative to
the control group, while the control group improves. In this case,
some aspect of the program (i.e., training provided, participant
selection) creates a situation where the program fails to meet
expectations.
As these examples reveal, tracking DVR program performance over time
without the benefit of a control group, would lead to incorrect
conclusions based entirely on management techniques. Interpreting a
performance management system demands a frame of reference to compare
the effects of external factors such as the economy on performance.
Population and Control Group Selection
We selected the population group by compiling the Wyoming Department of
Employment’s (DOE) administrative databases and determining if
individuals resided in Wyoming in the
reference years (1994, 1995 or 1996). The years correspond
to available DVR client data. To determine residency, we relied on
the following databases: Unemployment Insurance Claims (UI), Wyoming’s
Driver’s License (DL), Employment Services (ES) and wage records (WR). A determination
of Wyoming residency required a valid Wyoming zip code in the
reference year from the first three databases listed above. Residency
was further determined by whether a person had wages in WR during the
reference year.
The largest possible control group for the reference year was selected
from the available population group of the corresponding reference year.
To accomplish this, the three cohorts of DVR clients were
proportionally matched to a subset of the population on sex, age
and total wages of the four quarters prior to the reference year.
Table 2 shows the number of DVR clients along
with the persons in the population and control groups for each of the
three cohorts of analysis.
In Table 2, only those DVR clients with no wages in WR the quarter
prior to program participation are used to calculate the entered
employment rate. Similarly, the entered employment rate of the
control group reflects those with no wages in the quarter prior
to the reference year. Only those DVR clients or control group
members who entered employment or who had wages in the quarter
prior to the reference period were used to calculate retention in employment and
earnings gained. Due to the large number of those with wages
in the quarter prior to the reference period among the population
group, the number of people increases for core indicators 2 and 3.
As mentioned earlier, the matching criteria did not use disability
status. Despite the omission, differences between DVR clients and
the control group are consistent over time: DVR program eligibility
requires a disability, and the number of disabled persons in the
control group and the population remains unknown. The performance
of the DVR program is not assessed by whether its clients achieve
better results than the control group. Instead, assessment of the
DVR program considers program performance relative to the economy
as represented by control group performance.
Analysis of DVR’s database revealed that the duration of services
for clients corresponded on average to a four-quarter period. Where
the reference year for DVR clients represents training periods of
various duration, the four quarters of a given year define the
reference year for the population and control groups. The progress
of DVR clients or members of the control or population groups are
measured using quarterly data from wage records, prior to and
following program participation or the equivalent reference year.
The entered employment rate for DVR clients represents the number
of clients with no wages the quarter prior to application
(Q A-1) and wages the quarter following closure
(Q C+1) divided by all clients with no wages the quarter
prior to application (Q A-1 - see Formula 1). The control
group and population consist of individuals with no wages in the
quarter prior to the reference year (Q RY-1) and wages in
the quarter following the reference year (Q RY+1) divided
by all participants with no wages the quarter prior (Q RY-1 -
see Formula 2). The retention in employment rate and earnings
gained in employment discussed in Part One are based on different
formulas using wage records.
Formula 1: Entered Employment Rate of DVR Clients
Formula 2: Entered Employment Rate for Population and
Control Groups
Results
Table 3 shows the three cohorts’
year-to-year entered employment rate, retention in employment
rate and earnings gained in employment for DVR clients, the
control group and the population. As mentioned in Part One,
the DVR program showed a continuous improvement in the entered
employment rate (Figure 4) from 35 percent
for the 1994 cohort to 37 percent for the 1995 cohort. The
control group’s entered employment performance decreased about
four percent from 35 percent for the 1994 cohort to 31 percent
for the 1995 cohort, and the population’s entered employment rate
also decreased (-5%). The decrease in the performance of these
two groups suggests economic change that had no apparent impact
on the DVR clients’ performance.
Wyoming experienced a slowdown in economic growth
2 in 1995 that corresponds to the decrease
in the control group and population’s performance. Figure 4
clearly illustrates the manner in which the control and population
groups move in tandem with the economy. Figure
5 graphically represents the economic slowdown’s impact on the
growth in the total number of jobs based on the
ES-202 database. The 1994
cohort responded to the creation of 9,584 jobs in the economy
from 1993 to 1995, while the 1995 and 1996 cohorts experienced
a situation whereby only 4,250 and 4,809 jobs were created in
those years, respectively. While the total number of jobs
continued to increase from 1993 to 1997, the slowdown in jobs
created for the 1995 cohort decreased the opportunity to find
employment, thus impacting the entered employment rate.
The type of industry employing individuals following the reference
year contributed to the difference in results among the three
groups. For the most part, DVR clients and the other groups,
across all three cohorts, were placed in the Retail Trade and
Services industries. However, the population and control groups
of the 1994 and 1996 cohorts showed more even distribution among
higher paying industries.
The combination of the slowdown in job growth and the shift in
available jobs to industries generally paying lower wages
contributed to the decline in performance for the control and
population groups. Perhaps, when faced with economic conditions
such as these, individuals in the control group and population
left the state. Although DVR clients participated in a program
designed to place them in employment (most likely in Wyoming),
the analysis cannot infer motivation to work in Wyoming to the
population and control groups. Currently, Research & Planning
has not yet gained the means--interstate agreements for sharing
wage records--to track the labor movement beyond state borders.
Figure 6, based on data from Table 3,
compares the year-to-year six month earnings gains for DVR
clients, the control group and the population. The earnings
gained by all three groups decreased for the 1995 and 1996
cohorts. To demonstrate the impact of the economic slowdown on
earnings gained, Figure 7 represents the
economic conditions the cohorts faced, measured by the
average annual wage (AAW)
per job from the ES-202 database. The AAW increased $1,495
for the 1994 cohort, $24 for the 1995 cohort and $1,243 for
the 1996 cohort.
We expected to see the decrease in DVR performance for earnings
gained in employment for the 1995 and 1996 cohorts, considering
the control group and population both experienced a corresponding
decrease in performance for these cohorts. Further evidence
generated from the ES-202 database supports the conclusion
that the decrease in earnings gained stemmed from deteriorating
economic conditions, not DVR program changes. This conclusion
illustrates the benefits realized by the combined application of
performance management and performance evaluation techniques.
If, as mentioned in Part One, only DVR performance across the
three cohorts is known, the program would be at a loss to justify
the decrease.
Conclusions
The two indicators discussed, entered employment rate and earnings
gained in employment, show the descriptive concepts surrounding
population criterion performance evaluation. The analysis
presented economic factors influencing the performance of DVR
clients, the control group and the population. These factors
offer a limited explanation of economic influences on performance.
The hypothetical examples in the introduction compensate for a lack
of historical data for the DVR program. With more data available,
researchers can better determine the relationship between the factors
discussed (data generated from the ES-202 databases) and program
performance. Access to a longer time series of data would have
informed and enriched the processes and results addressed in this
article. Further, the process must be applied across other workforce
investment activities. Currently, we do not know if clients of the
Job Training Partnership Act (JTPA) programs behaved similarly to the
DVR clients under the same economic conditions.
The research presented in this series is exploratory in nature.
While the content is difficult, we will continue to familiarize
readers with the concepts of performance management and performance
evaluation. The problem with employment training evaluation research
lies in isolating the economic factors that influence the population,
control group and the performance of clients of a training program.
After isolating the factors involved, evaluation research attempts
to understand the relationships between the factors and the subsequent
impact the interrelationships have on program performance.
1
The Workforce Investment Act of 1998, Pub. L. No. 105-220 (1998).
Sec. 136.
No Wages Q(A-1) with Wages
Q(C+1) Entered
Employment
=
Rate
All Individuals with No
Wages Q(A-1)
No Wages Q(RY-1) with Wages
Q(RY+1) Entered
Employment
=
Rate
All individuals with No
Wages Q(RY-1)
2David Bullard, "Total
Payroll as a Tool for Identifying Business Cycles in Wyoming,"
Wyoming Labor Force Trends,
May 1999.
Table of Contents | Labor Market Information | Wyoming Job Network | Send Us Mail
These pages
designed by Gayle C. Edlin.
Last modified on
by
Valerie A. Davis.