Workforce Development and Community College Outcomes
by: F. E. “Skip” Gillum, Vice
President, Casper College; Rosanne Crossen, Director of Institutional Research, Casper College;
and Sara Saulcy, Economist, Research & Planning
"This partnership...has resulted in information being developed for the first time that documents the way that Casper College interacts with Wyoming’s job market."
S
ince
February 2000, Casper College and the Wyoming Department of Employment, Research
& Planning (R&P) have worked together to design a process for describing the
interaction between Casper College graduates and the Wyoming labor market. A primary
goal of our work is meeting the training provider requirements for performance
measurement in the Federal Workforce Investment Act of 1998
(WIA).1 Our partnership also meets the needs
of Casper College: supporting educational improvement and the accreditation process.
These needs meet the criteria for data sharing between Casper College and R&P as
established by the Family Educational Rights and Privacy Act (FERPA).2 To meet the statutory requirements of WIA, we
developed a pilot employer questionnaire to obtain information not available through
other sources. Research & Planning’s administrative databases contain information
about employers and workers, which permits us to identify the work histories of college
completers but that in other ways is incomplete. To fill these gaps, we collected
additional information directly from employers by mail questionnaire. In this article
we present background information on performance measurement, and discuss users who
benefit from this information. The research design and data collection techniques,
as well as limitations of the data, are also explained. In a complementary article
(see "Implementing the Workforce Investment Act..."), summary
statistics are provided.
Purpose of Performance Measurement
Little direct information about the outcomes of training programs is available. Educated
guesses about the impact of education by measuring earnings before and after training are
possible. We also know, for example, that on average a physician earns more than a mechanic.
It is difficult to measure how training programs compare for occupations having less
clear-cut outcomes. The incomplete, inequitable, or untimely provision of career
information may lead to outcomes for employers and employees that are undesirable or
unacceptable.
The performance measurement process as described in WIA is an attempt to provide
consumers of training programs useful and objective comparisons. Students are the
direct consumers of training programs through coursework and hands-on instruction.
Employers are also consumers in that they apply the skills, knowledge, and experience
that labor acquire through training programs to produce goods and services. The
benefits of the consumption of educational services accrue to society at large as well.
Neil Bruce points out that, “It is difficult to imagine how an advanced industrial
economy could function if the population were illiterate and ignorant….”3
Society has recognized the value of a skilled and knowledgeable workforce, and publicly
funds education from pre-school through post-graduate studies. However, in return for
expending funds for postsecondary training programs, accountability has been introduced
through WIA to provide for objective measurement of education outcomes. Ideally, labor
market performance measures lead to
• program offerings more consistent with positive labor market
outcomes;
• an increased ability by training providers to learn about and
adapt to the changing needs of both employers and students;
and
• support for the accreditation process through better
outcomes-based planning.
Community colleges are often viewed as homogenous training organizations. In fact they
differ, sometimes greatly, in the quality and type of training they provide, and in the
role they play in the labor market. As part of their mission, Wyoming’s community
colleges function as dynamic institutions serving the sometimes-competing demands of
student aspirations, local employer needs, and the changing economic and industrial
conditions of the local, state, and regional labor market. For many students, they also
serve as stepping stones to the University of Wyoming (UW) or other institutions of
higher education outside of Wyoming.
Students from colleges have typically been viewed as new labor supply. In reality, many
are returning adult learners who already have workforce experience, (see "Implementing the
Workforce Act...", Figure 4). Furthermore, students often work
while attending school, sometimes by choice, but often because their personal
circumstances dictate their doing so.
New and incumbent workers use the training provider system for a variety of purposes.
Some individuals may truly be new labor supply. Others represent a part of the workforce
seeking skills upgrades for their existing jobs. Students may also be looking to improve
their skills or acquire new skills so they can change jobs, or are acquiring education in
order to move on to another institution. Still others may have been absent from the labor
force for some time and are seeking skills to improve their chances of acquiring
meaningful employment upon returning to the labor force.
WIA Requirements and the Role of R&P
Training providers, whose Wyoming clients receive services under WIA, are required to
furnish labor market outcomes reports to the Wyoming Department of Employment (DOE).
Under Section 122d of WIA, training providers must make available to the public
information about
• program completion rates;
• the percentage of individuals participating in a training
program who obtain unsubsidized employment, including
those who obtain employment in an occupation related to their
chosen training program;
• the percentage of individuals who completed the applicable
program;
• the retention rate six months after hire of program
participants who completed the applicable program;
• rates of licensure or certification and other credentials, or
attainment of other skills;
• wages earned at initial placement in employment by all
program participants; and
• wages earned by training program participants who
completed the program six months after hire.
The goal for R&P was to develop a template, using one college, to demonstrate how to
meet the requirements of one provider of training services. Ideally the pilot process
becomes the vehicle for answering practical questions about producing the required
information, and identifies the resources needed to produce consumer reports and
performance measures for training providers in general. The initial information derived
from the questionnaire responses gives us a baseline for how training program participants,
and therefore training providers, are performing. Once we have this information, training
providers and other participants in the workforce development system can experiment with
different labor market interventions (i.e., adjusting career development services,
expanding internship offerings, identifying appropriate training programs). Ultimately
the goal of labor market interventions is to achieve desirable outcomes in a timely manner.
Such outcomes include obtaining employment in a preferred career, attaining employment with
a desired compensation package, or meeting career or job goals within a certain time frame.
Beneficiaries of Performance Measurement Information
The information derived from the pilot survey of employers of Casper College graduates
is potentially beneficial to a variety of groups. Among them are Casper College itself,
industry, job seekers and job changers (traditional and nontraditional students), and
policy makers.
Casper College benefits by utilizing the information that is developed over time to
improve existing programs and to develop new program offerings, which will continue
to meet the changing needs of the Wyoming labor market. The data also document the
earning power of program completers who attended Casper College.
Employers who require post high school education of their workers also benefit from
access to local labor supply information. For example, an employer might want to know
if industries employing welders are satisfied with the skills individuals bring to
their jobs once they have completed or participated in a related technical program at
Casper College. Positive feedback from local peers could encourage that employer to
send his/her employees to Casper College for additional training. As another example,
an employer could review the wage results of the survey to determine if the compensation
package he/she is offering is competitive, and if not, adjust the compensation package
offered accordingly.
Job seekers and job changers benefit by having access to information that identifies the
average salary for employment fields in the state, and assists in identifying the most
suitable program of study. Among those who benefit are students, parents of students,
career counselors, dislocated workers, and other individuals currently outside of the
workforce (such as stay-at-home parents or spouses). All have access to the objective
information that they need to make decisions consistent with their goals and interests.
Policy makers benefit by having documented evidence of the long-term return on investment
in higher education in the State of Wyoming. The information also helps to describe how
Casper College interacts with the state job market. Among the policy makers who can
benefit from this project are legislators, state and local workforce investment boards,
and others involved in workforce policy.
Advantages and Disadvantages of the Casper College/R&P Partnership
This partnership has resulted in a win-win undertaking for all concerned. It has resulted
in information being developed for the first time that documents the way that Casper
College interacts with Wyoming’s job market. It provides reliable information concerning
the earning power of graduates and allows for conclusions to be drawn concerning the
impact that the college has on the earning power of those individuals. Through subsequent
employer surveys conducted by DOE, the college will have reliable data concerning the
quality of their programs of study. Performance measurement also provides information
to the college about areas where employers feel their employees need additional training,
thus allowing the college to respond to those training needs.
The resulting data, however, are not without limitations. Wage Records data only
account for those individuals working in Wyoming and are not all-inclusive. Not found
in these records are individuals working for the Federal Government, persons in the
military, agricultural workers, and persons who are self-employed. Despite the
limitations of the survey conducted jointly between Casper College and R&P, the
information is better than the self-reported data traditionally gained by Casper
College through the use of graduate surveys.
Pilot Survey Process
The pilot survey process was driven by a number of factors:
• the availability of student data;
• the timing of downloads from the Unemployment Insurance
database; and
• a goal of publishing draft consumer reports by summer 2001
to provide information to Casper College prior to the next
academic year, enabling administrators to make decisions
based on the information.
Availability of student data, and the timing of the download of Unemployment Insurance
(UI) Wage Records4 were the primary determinants
guiding the employer follow-up survey. The Wage Records database consists of all
employers submitting UI tax records to the DOE, and contains detailed work behavior
information on individuals working for employers who are required to pay Unemployment
taxes.5 Student data are generally available shortly after
the end of a semester. In contrast, the time delay for Wage Records from the end of a
reporting quarter to its availability for downloading is two quarters. For example,
data for the first quarter of 2000 (2000Q1) was available at the beginning of the
third quarter of 2000 (2000Q3). In order to meet the requirements of Section 122 of
the WIA that we obtain information about graduates at the time of hire and six months
later, student data were matched with employment data from the Wage Records database
two quarters after the date of graduation. The number of employers of graduates who
were included in the survey are shown on Table 1 (see Implementing
the Workforce Investment Act...). Because the employer survey was a pilot, mailing
questionnaires six months after graduation (as opposed to six months after hire) served
as a proxy time frame for considering graduates six months after hire. By doing so we were
able to consider how using administrative databases in concert with student data works in
practice, even though we do not precisely conform to the six-month time frame outlined
by WIA.
Graduates of the May 2000 class represent the population under consideration. The class
had a total of 303 graduates. In the fourth quarter of 2000 (the time that Wage Records
are available from the second quarter of 2000), employer, student, and Wage Records were
matched to form the subset of the population we were evaluating (employed graduates).
Student records were matched with employer Wage Records using Social Security Numbers
(SSNs) as the matching criterion. The student SSNs and program enrollment data were
obtained under a Memorandum of Understanding between DOE and Casper College. The data
sharing agreement is consistent with the Family Educational Rights and Privacy Act
regulations describing the sharing of information for purposes of improving educational
programs, and support of the accreditation process.6
Based on the match, 145 out of 303 graduates (47.9%) were found to be working in Wyoming.
Although only 47.9 percent were found to be working in Wyoming, this does not necessarily
mean that the remaining 52.1 percent are unemployed. Some may be working in non-UI
covered employment, such as Federal Government, railroads, or Agriculture, or are
self-employed. Others may have left Wyoming and are not tracked by Wyoming’s UI
Wage Records. The possibility also exists that they have moved on to the University
of Wyoming or another institution of higher education. Individuals attending UW or
other schools may appear in Wage Records as either full-time or part-time employees,
or may not appear at all because they are attending school only and not working.
Because individuals occasionally work for multiple employers, there were a handful
of instances of multiple employers for a single graduate. Similarly, many individuals
may work for a single employer. Overall, 183 matches between student and employer were
identified in Wage Records.
Research & Planning distributed the survey to Wyoming employers of completers of
Casper College programs; employers of program non-completers were not sent a
questionnaire. We collected information from firms who employed one of two different
types of Casper College graduates (see Table 1):
• graduates employed in April, May, and June of 2000 (the
quarter of graduation, denoted as 2000Q2) and the quarter
following graduation (third quarter of 2000, denoted as
2000Q3); and
• graduates not employed at the quarter of graduation, but
employed the quarter following graduation (2000Q3).
The questionnaire was mailed in late January 2001. A follow-up letter with the
questionnaire was mailed in mid-February to employers who had not responded to the
first questionnaire by early February. Employer questionnaires that were returned
because of inadequate addressing were re-mailed to an updated address for the employer.
Depending on the employment status of the graduate at the time of graduation, an employer
was sent one of two questionnaires.7 For graduates employed
only in 2000Q3 as identified in wage records, we asked employers what they paid when the
graduates were first hired. For graduates employed in both 2000Q2 and 2000Q3 (as
identified in wage records), we asked the employer what the highest rate of pay was
in April, May, and June of 2000. The questionnaire consisted of ten questions on
the following topics:
• whether or not the person was still employed as of January
12, 2001 (six months after graduation);
• wage rates at first hire, or in April, May and June of 2000,
and wage rates as of January 12, 2001;
• number of hours worked per week;
• benefits or other non-monetary compensation received by the
employee;
• whether or not the employer received a public subsidy for the
position;
• the occupation and primary activities of the person;
• training or education required for the job;
• satisfaction with the employee’s work and work habits; and
• whether the employer considered the supply of labor for the
occupation to be sufficient and skilled.
January 12, 2001 was the date we requested information from because it is the UI
reporting date for wages paid by employers at that point in time in the first quarter.
The goal was to ease the reporting burden on employers by permitting them to draw
information on employees from reports they were already required to submit.
Questionnaire Responses
Responses to questions by employers were a reflection of whether or not graduates
worked for the same employers as of January 12, 2001. If an individual was still
employed, we asked the employer to complete the entire questionnaire. If the individual
was not working for the same employer as of January 12, 2001, the employer was asked to
complete questions regarding wage rate at first hire, or wage rate for April, May and
June of 2000, employer satisfaction with the employee’s work, and to comment on the
available supply and skill of labor for the occupation.
A total of 181 questionnaires were mailed to employers. Two employers of graduates
were not sent questionnaires because of incomplete UI Wage Records for the employers.
Of the 181 questionnaires, 158 were returned. Five of the 158 survey responses were
invalid.8 The bulk of responses were tallied as reported. Three
exceptions were wages, childcare (as a benefit), and employer opinion of the labor supply.
Employers were given the option of reporting wages as hourly, weekly, biweekly or
monthly. Since the majority of wages were reported as hourly, R&P converted all
wages to hourly to aid the comparison of wages. Where employers reported wages as
weekly, biweekly, or monthly, and when hours normally worked were provided, R&P
converted those wages directly to hourly wages. In cases where the employer reported
the wage but not the number of hours, R&P calculated hourly wages assuming a 40-hour
workweek.9 When a conversion resulted in a value
inconsistent with other information provided, the value was entered as missing.
Employer opinion of labor supply was evaluated using content analysis. The analysis
yielded four groups of responses:
• employees were not willing or committed to work;
• the labor supply is not sufficiently skilled;
• there is a lack of labor supply in general; and
• the labor supply is sufficient and skilled.
Data Limitations and Remedies
Although we received 158 of 181 questionnaires sent (an 87.3% return rate), our
success is somewhat muted. One problem involved the wording of the benefits question.
We asked if employees received (as opposed to simply being offered) any of the
benefits10 listed on the survey. Several employers
stated that the employee was offered the particular benefit, but did not necessarily
receive it. Consequently employer-provided benefits may appear lower than is
actually the case. For example, the initial response for child care benefits was
that none of the employees received them. Based on other survey programs conducted
within R&P, we knew that certain employers offered subsidized child care. As
confirmation, we contacted certain employers for further information. From the
telephone contact with the employers, we revised the number of employees receiving
child care benefits. Another error was in omitting asking employers about
retirement benefits employees received. Employers had the option of writing in
other benefits that employees received, but none elected to write in retirement.
Another problem involved employer non-response. Despite assurances that information
would remain confidential under state and federal law, some employers declined to
answer certain questions, citing company policy. This occurred most frequently for
questions regarding salary and satisfaction. Non-responses also occurred because
the employer would only report whether or not the company still employed the
graduate, and would not answer any of the other questions as requested. Some
employers responded that they did not have knowledge of the particular question
being asked. Larger employers more frequently had this problem, where the individual
filling out the survey may have had little direct contact with the employee. As a
result, some employers were unable to provide answers in such areas as satisfaction
with the employee’s work or work habits.
Contributing to the problem of non-response were employers who did not answer certain
questions (even though they were requested to do so), and did not report their reason.
In some cases, employees were temporary, seasonal, or volunteer, and therefore do not
ordinarily receive standard salaries and benefits. These yielded a small number of
non-responses for certain questions.
Basic statistical problems are also part of the data set. First, we are unable to
generalize to other populations (e.g., graduates of Casper College in general, graduates
of other community colleges) using the data from the questionnaire regarding May 2000
graduates. The small number of responses to various questions also means that
R&P is unable to do much in the way of statistical modeling (i.e., wage or
occupational predictions). Because of the small number of observations, we will
need more than one year’s worth of information to establish any trends in the data.
In addition, the small number of responses precludes us from providing more detailed
occupational and industry data.
Future Plans
The most immediate plan for R&P is to revise the survey. One revision is a change
in the benefits question to read, “Is the employee offered [as opposed to receiving]
any of the following job benefits?” Another possible revision is to the question on
the employer’s opinion of labor supply. The question was open-ended to allow employers
to express their opinion of the current status of labor supply and skills. While
R&P made attempts to code responses using content analysis, we are somewhat
uncertain about whether or not we have accurately depicted employers’ sense of labor
supply and skills. We expect to format the question in a way that will elicit a more
precise response. We also will evaluate other wording possibilities to encourage
employers to respond.
The most important development is that Northwest College (NWC) in Powell and Laramie
County Community College (LCCC) in Cheyenne have entered into agreements with R&P
to assist in their completion of the WIA requirements. Our objective is to build on
the pilot that R&P and Casper College have developed.
The outgrowth of the performance measurement process is consumer reports. Consumer
reports will integrate information from performance measurement and administrative
databases. Through the provision of information, we hope to reduce the level of
uncertainty for those in the market for educational or training providers. Consumer
reports will aid in achieving the goals of labor market participants.
Summary
As our article shows, measuring the performance of graduates of training programs in
the labor market yields many benefits beyond the requirements of the Workforce
Investment Act. The partnership between Casper College and Research & Planning is
a mutually beneficial relationship, allowing our institutions to draw on the resources
with which we are familiar, and to access alternative resources where we have either
less knowledge or do not have the tools, such as Wage Records. The American
Association of Community Colleges advocates the use of UI wage data for performance
measurement tasks.11 While there are limitations to
the data, we are encouraged by the initial results of the Casper College pilot
survey. Now that we have conducted the pilot survey, we look forward to improving
our methods and our results. For further information, see the related article on
the summary statistics in this issue, “Implementing the Workforce Investment Act:
Results from an Employer Survey Follow-up of Casper College Graduates.”
1 105th Congress, “Workforce
Investment Act of 1998,” n.d..
2 “Family
Educational Rights and Privacy Act (FERPA),” n.d..
3 Neil Bruce, Public Finance and the American Economy, 1998, p. 354.
4 The UI Wage Records database consists of all employers submitting UI tax
records to the Wyoming Department of Employment (DOE), and contains detailed work
behavior information on individuals working for employers that are required to pay
Unemployment taxes. See Tony Glover, “Enhancing the Quality of Wage Records for
Analysis Through Imputation: Part One,” Wyoming Labor Force Trends, April 2001.
5 Tony Glover, “Enhancing the Quality of Wage
Records for Analysis Through Imputation: Part One,” Wyoming
Labor Force Trends, April 2001.
6 See Endnote 2.
7 The survey instruments are available on Research & Planning’s
Education Links website.
8 Four employers reported they did not have the person on record
as an employee.
Potential sources of this problem include incomplete or incorrect records on the part of
the employer, lack of knowledge of the employee by the person filling out the
questionnaire, unwillingness of the employer to participate in the survey, or
incomplete or incorrect UI Wage Records. A fifth response was invalid because the
employer reported once that the employee had worked for them, but then responded a
second time that they had no record of the employee. The first questionnaire sent
to the employer was received after response deadlines had passed. A second survey
was sent to the employer to encourage a response. A comparison of the two surveys
for the same employer and graduate yielded the conflicting responses. Consequently
responses for the employer for the particular graduate were entered as missing.
9 The Alien Labor Certification program uses a 40-hour work week to convert
wages reported in any units other than hourly. See
General
Administration Letter No. 4-95, n.d., for further discussion.
10 The benefits question was asked in a manner similar to the collection of
information for our employee benefits survey (see Wyoming Department of Employment,
Research & Planning, Employee Benefits in
Wyoming 2000).
11 Robert T. Mundhenk, “Institutional Effectiveness and
Unemployment Insurance
Data,” Fall 2000, American Association of Community
Colleges White Paper.
Table of Contents | Labor Market Information | Wyoming Job Network | Send Us Mail
These pages designed by
Gayle C. Edlin.
Last
modified on
by
Valerie A. Davis.