Creating a Data Tracking System
Colleges are being asked to demonstrate student success while dealing with budget reductions and significant increases in enrollment. The need to do more with less is confounded by the institutions’ efforts to provide supports and services to prepare low-income and non-traditional students for a career that will lead them to economic stability. As colleges look for ways to improve practices while providing a holistic approach to preparing our future workforce, they frequently are asked to provide evidence of the effectiveness of their education and training services. Coupled with an increasing number of state legislatures moving towards performance-based budgeting—which applies a formula to student outcomes (instead of enrollment) in order to allocate funding — institutions need to be able to document student progress towards meeting outcomes. To do so, colleges must develop measures and indicators for monitoring student progress towards the attainment of a degree or occupational credential.
The design and implementation of a data collection process can be cumbersome for colleges, but data collection is necessary to understand the impact the strategy has on the students and the institution. In addition, data can be the tool used to communicate to external stakeholders, partners and funders on why they should invest additional resources in the institution.
Data collection should be thought of as a two-tiered process:
- Participant-level data are necessary to establish a baseline for students and track their progress towards goals.
- Program-level data is collected to identify specific interventions and monitor and evaluate institutional and community impacts to identify challenges and improve practices.
Establishing appropriate outcomes for individuals and for colleges
The approach used to implement the core strategies of the network vary by institution and thus, the collection and analysis of data may vary by institution. Participant-level and program-level outcomes are important in designing interventions and determining the practices that work best at the college. The Working Families Success Network has identified a list of outcomes that are important to assessing the impact of the WFS strategy. These are a broad set of outcomes focused on the results a student achieves as they connect to the various pillars of the strategy. The outcomes established by AECF capture results on the individuals who connect to WFS services and data on implementation of the program. The list below shows the outcomes AECF uses to monitor the WFS strategy at community colleges.
Common Outcomes for students receiving Working Families Success services:
- Completion of (or persistence towards) a degree or occupational credential
- Increased Income
- Debt reduction
Common Outcomes for colleges using the Working Families Success strategy:
- Increased retention rate for students receiving WFS services
- Increased completion rate for students receiving WFS services
- Improved employment outcomes for students receiving WFS services
- Efficient and effective service delivery (bundling)
The outcomes identified above are just a starting point. There likely will be other outcomes that are better measures of the impact on WFS participants. As services evolve to fit student needs, colleges may need to revisit the outcomes the institution has chosen to determine if they still fit the goals of the participants and provide enough information to assess the strategy.
While examining the effects of the WFS strategy at the institution, consider additional outcome measures up front and begin to set indicators to monitor progress.
Additional outcomes for consideration include:
|Participant Outcomes||College Outcomes|
|Increase in savings.||Reduction in the lifetime cost of educating a student.|
|Increase in net worth.||Increased investment in WFS strategy.|
|Retained employment for at least six months.||Institutionalization of services.|
Tracking Progress towards Outcomes
Tracking data for individual begins with the intake process. During intake staff should collect enough information to set a baseline for tracking progress towards meeting the individual outcome measures while providing the staff with enough information to determine the type, and sequence, of services the student will require in the short-term.
The indicators a college collects should be based on the outcomes it decides are most appropriate for the services provided to students and the information needed to monitor and evaluate the WFS strategy. A college needs to decide whether to track every student or a particular cohort of student to assess the effectiveness of the strategy. The table below displays a sample list of indicators and related services that can be used to measure progress towards achieving the chosen outcomes.
|Outcomes||Indicators to track||Services|
|Completion of degree or credential.||Enrolls in a degree or occupational credential program; Receives an educational award; Enrolls in a transfer program to obtain a 4-year degree.||Education and training instruction.|
|Increased Income.||Receives tax refund; receives public benefits; placed in unsubsidized employment; increase in wages.||Benefits screening (SNAP, Emergency cash assistance, Unemployment Insurance); financial aid; EITC/VITA Outreach; tax assistance; job placement services.|
|Debt reduction.||Improves credit score; lowers debt-to-income ratio.||Review credit report; budget and loan counseling.|
|Increase in savings.||Opens and contributes to savings account; purchases a Certificate of Deposit; Opens and contributes an IDA, retirement, or educational savings account on a regular basis.||Development of savings plan; provide access to financial products.|
|Increase in net worth.||Increase in assets (buys a house, adds to a financial account).||Financial education and coaching (i.e. Goal setting, budget workshops).|
|Retained employment.||Maintains employment on the same job for at least six months.||Career counseling; job readiness training.|
|Institutionalization of services.||Number of students served.||N/A|
|Increase in student retention.||Fall to spring retention; increase in fall to fall retention.||N/A|
|Increase in completion rate.||Number of students that received a degree, credential, or transfer to a four-year degree program.||N/A|
|Improved employment outcomes for students.||Number of students placed into unsubsidized employment.||N/A|
|Efficient and effective service delivery.||Number of students that receive 2 out of 3 core services; Number of students that receive 3 out of 3 core services.||N/A|
|Reduction in the lifetime cost of educating a student.||Average expense of educating a student.||N/A|
|Increased investment in WFS strategy.||Amount of leveraged resources increases over time; Grants/funds received for implementation of services.||N/A|
Self-reporting of data
In many cases, participant data will be self-reported, meaning that students will need to be educated or at least given clear instruction as to what they are expected to report back to the college. Measures of decreasing debt or increasing income require student’s honest and clear reporting of where they stood before and after a WFS intervention. In certain cases it may be possible to track the impact of WFS services after a student has left the institution. Some colleges, for instance, are able to track aggregate wage data for students once they graduate. However, it is unlikely that the data can be ascribed to particular students, meaning this may not be a fruitful exercise for colleges that only serve a small cohort of students through their WFS. Of course, not all data is self-reported; student completion and retention can be tracked by the college
Strategies for data collection
Data collection should be integrated into the service delivery model. Since the WFS strategy can take many forms within a college, the first place to start when designing a data collection process is deciding when a student is considered a participant in your institution’s WFS. Taking the time up front to understand when to count a student as a WFS participant will reduce confusion about who is in the “denominator” when assessing the strategy’s impact on participant- and college-level outcomes
Deciding how to track data on WFS service delivery is often a function of resources. Some colleges use case management software products, such as Efforts-To-Outcomes (ETO) by Social Solutions. Case management software allows staff to enter in every interaction with a student and results towards meeting intermediate goals and longer-term outcomes. A case management tool allows colleges to provide timely and more accurate reports on the percentage of students receiving the different types of services.
Many software products, like ETO, require a significant investment and may be difficult for colleges to justify when just beginning to implement the WFS strategy. Some colleges use paper and pen to track student services and create either a spreadsheet or database, which uses the same unique identifier to identify each student in a school’s data system, to store data on services and analyze progress toward outcomes. Involving the college’s IT staff in the design of the data collection process up front will help determine which tool will best fit the college’s needs.
Finally, tracking data on services received and outcomes is time-intensive and is not likely to be the role of one person. Design a workflow process that is flexible but still allows for the timely delivery of information. For example, a financial coach may use a paper-based form to record information from their coaching sessions. Then the form is later given to an office assistant to input into ETO or a Microsoft Excel spreadsheet or ACCESS database.
One additional item to consider is how external partnerships impact data collection. If some of the service provider staff are not employees of the college, or if data is shared with a community based organization, then the college will need to develop a data-sharing agreement.
Adapting existing data systems
Regardless of the tool used to input, aggregate and analyze the data, the college must put this information into existing student data systems such as Banner, Datatel or Peoplesoft to see how WFS services affect education, employment and economic outcomes for participants. While data from ETO, MS Excel and MS ACCESS can be uploaded to college data systems, some colleges have chosen to integrate tracking of WFS services into their existing data systems. See the case study on Des Moines Area Community College in the Center for Working Families at Community Colleges: Clearing the Financial Barriers to Student Success report.
Determining the audience for data collection
When determining the data to be collected, colleges should carefully consider their audiences and their strategic interests and ensure that the right data points are included. The numbers that resonate most strongly with a student may be different for a funder or community partner. The table below offers some suggestions on the different types of data that should be presented for different parties.
|Audience||Examples of Data to Collect|
|Students||Dollar Amount of financial aid accessed; Amount of public benefits accessed; Qualitative stories of student success; retention; participant level financial outcomes such as debt reduction, achieving personal financial goals.|
|College staff and leadership||Term-to-term Retention rates; degree or credential completion rates; leverage of additional funds.|
|Community partners and funders||Retention rates; completion rates; measures on participant level financial outcomes including income, savings, credit score, employment, job retention and advancement; qualitative stories of student success; change in student financial behavior.|