Blog entry by Outreach Evaluation Hub
Within Lincolnshire, the Uni Connect programme is delivered by LiNCHigher based at Bishop Grossteste University in Lincoln. The project team ordinarily delivers outreach activities as part of the Uni Connect programme to over 40 schools and colleges reaching approximately 4,200 target learners. The local evaluation for Phase 2 of the programme is managed within the Lincoln Higher Education Research Institute (LHERI) at the University of Lincoln.
During the current Covid-19 crisis, Uni Connect partnerships are exploring alternative ways of engaging with learners and evaluation methodologies are evolving to reflect the different modes of delivery. Like other evaluators, LHERI is currently embedding evaluation in the online delivery of the LiNCHigher programme. Within the LHERI evaluation team, we also manage a LiNCHigher funded project called ‘Explaining the Gaps’. This project has been running since 2018 and informs the local evaluation using analysis of the Lincolnshire-specific data collected through the national survey managed by CFE. The survey addresses various aspects of higher education including learner awareness of HE, their level of confidence and the practical aspects of HE. ‘Gaps’ in these aspects are identified based upon responses to the survey. One of the aims of the project is to enable the delivery of targeted outreach activities and to identify hidden sub-groups of learners within larger cohorts using quantitative methods. In light of the current situation, the ‘gaps’ identified will be used as a forward planning tool for when schools reopen.
Explaining the Gaps: Methodology and analysis
Gaps are identified at a school and year group level, and where possible by further sub-groups. The sub-groups include male and female students, Uni Connect target learners and students with a self-reported disability. This targeted approach is intended to enable LiNCHigher to deliver effective outreach activities within schools, making the biggest impact whilst maximising resources.
Although the survey is focussed on Uni Connect target learners, responses are not restricted to these students. Similar to the experience of Uni Connect partnerships in other parts of the country, many schools in Lincolnshire prefer not to single out Uni Connect learners, and collecting responses from a wider population makes it possible to have a comparison group. In 2019, LiNCHigher collected 10,875 responses to the Wave 2 follow-up survey, 9,800 for Wave 1 (2018) and 2,400 at baseline (2017). Wave 1 and 2 were completed online, whilst the baseline data was collected via a paper survey, which is the reason for the reduced response rate.
The LiNCHigher version of the survey contained 29 questions about various aspects of higher education. Whilst it is valuable to look at the responses to each of the individual questions on a school by school basis, we have found that it is more beneficial for planning targeted interventions to group questions that address similar themes together.
The national learner survey is arranged so that distinct blocks of questions address different topics. In order to confirm that questions may be aggregated, we used a technique called principal components analysis (PCA) to reduce the number of variables to produce a smaller set of so-called hidden or latent variables, or principal components, in order to aid description and analysis. We used standard criteria (e.g. scree plot, eigenvalues greater than one) to determine that five was the optimal number. Once established, a simple score determined from the average of each survey item can then be applied to each. Finally, the components are summarised by the underlying theme of the grouped variables.
The five LiNCHigher survey themes included:
- Application knowledge
- Participation knowledge
- Confidence and resilience
- Study skills
- Personal benefits of Higher Education
LiNCHigher outreach activities have been mapped to both the Gatsby Benchmarks and the Network for Evaluating and Researching Participation Interventions (NERUPI) Framework in order to categorise each activity by the intended learning objective. The five learner survey themes generated using PCA broadly align with the NERUPI Framework.
In order to use the data for targeted outreach, firstly the scores within a year group for each of the five survey themes were ranked by centile and reported by quartile. Secondly, an average score for each of the themes was calculated for each year group within individual schools. These scores were then compared to the overall Lincolnshire year group quartiles. Where there was enough data, average scores were also compared for sub-groups. Scores that fell within the lowest or the highest quartiles were highlighted, and all scores were summarised using a red, amber and green traffic light scale presented in table format. Scores for each of the themes were calculated using all the survey responses within each year group and not just the Uni Connect learner responses. In this way gaps identified through the survey data (i.e. scores that are in the lowest quartile) can be tackled by appropriately mapped activities. For example, Uni Connect target learners in a specific year group within a given school might have a group average score for confidence and resilience within the lowest quartile of scores (shown as red in the summary table). In this case these students might benefit from activities that specifically address this issue.
As part of our evaluation, we hope to track learners that participated in both Wave 1 and Wave 2 follow-up surveys longitudinally. At the beginning of Phase 2 we identified six schools from different parts of the county as case study schools. When we can visit schools again, we will run a series of focus groups with a number of students from each location but specifically a sample of those that we have longitudinal data for. We originally planned to hold these focus groups at the beginning of the summer term to capture students’ views of the outreach activities they had participated in throughout the school year and explore the impact. As the focus groups have been postponed, we plan to additionally use them as an opportunity to help understand the impact the Covid-19 situation has had on these learners. If the Wave 3 follow-up survey can be carried out in the next academic year we will follow up with the same students again in 2021.
There are some limitations to the overall analysis. Data collection took place in the Autumn term and could be seen to be out of date as the school year progresses. However, there are observable patterns in the data between Wave 1 and Wave 2, which suggests that the findings can be used as a template – or a starting point – for a school which helps LiNCHigher to plan outreach activities to take place in the next academic year. In addition, the depth of data relies on the number of responses from a school to enable reporting for sub-groups. However, providing feedback on students’ responses is popular with schools and supports collaboration. In practice, it has meant that schools have been more inclined to participate in the survey and this has translated to an increased response rate.
Lucy Mallinson, Lincoln Higher Education Research Institute, University of Lincoln