Blog entry by Outreach Evaluation Hub
Our recent survey to assess the
implications of school closure for Uni Connect evaluations confirmed that most
people feel that the current restrictions and school closures as a result of
the Covid-19 pandemic will have consequences for the short and long term
evaluation of the programme. In the short term, evaluators were most concerned
about the interruption to their data collection methodology, particularly in
terms of collection of post activity data (for pre/post activity level and/or
programme level analysis). Some mentioned problems for their evaluation
activities in relation to being able to take forward planned qualitative data
collection methods (notably inability to undertake focus groups and interviews
due to social distancing regulations). A suspension/change in activities
delivered has implications for the quality of evaluation data in terms of the
ability to draw conclusions about any particular new or existing activities
that had been scheduled but not taken place. Plus, it undermines assessment of sustained/progressive
programmes because the learners will not be receiving the programme as
envisaged or in a tailored or optimal way. Some respondents anticipate problems
due to data sharing arrangements being disrupted. Plus the current situation
will have an effect on skewing outcome measurements.
Respondents tended to feel that online delivery will be more difficult to evaluate due to issues of not being able to know the students taking part/get quality data on them and their circumstances, or be able to follow-up/track the people who are participating. There is also the issue of the sample who take part in online delivery being skewed to the more engaged learners. Generally the opportunities for date collection appear to have been limited by the current situations, although there also appear to have been some emerging possibilities. For example, putting effort into collecting reflective logs from teachers who may have more time to complete them. In some areas time has been freed up to support professional development on evaluation for practitioners.
Long term problems anticipated included difficulties associated with being able to unpick the effects of the Covid-19 situation on the participant outcomes being measured as part of Uni Connect evaluations. Most people felt it will be harder to prove the impact of Uni Connect. Respondents mentioned a range of potential issues such as ‘false negatives’, issues due to confounding factors caused by anxiety or mental health problems due to the pandemic, or students needing to work to support their families, with increases in inequalities between student groups (as well as a gap in the amount of guidance/support being delivered and received during this time). Changed priorities affecting school engagement in evaluation and data collection/sharing activities were also mentioned as a potential long-term issue for Uni Connect evaluators. There were calls for improved access to pupil level administrative and guidance on how to approach GDPR under new working condition (including specifically, messaging around use of public task and reassurances to schools/colleges that this was a requirement). We should all become advocates for better data at this time because systematic data analysis will be needed to establish how the pandemic is affecting different target populations in the short and long term.
It is clear that work will need to be done to adapt evaluation plans and designs now. In this new phase, evaluation becomes more developmental and will need to adapt to unknowns. Ideally, Uni Connect evaluators will be proactive. This will probably involve talking to the stakeholders you're working with to implement your evaluations (even though evaluation might be the last things on the minds of practitioners who aren't evaluators). However, adjustments need to be made now, to update your evaluation and data collection. For example, outcome measured may have changed, which means the evaluation design and criteria for judging effectiveness will need to change. Furthermore, as an evaluator, you are in a position to help practitioners by showing that thinking on evaluation and real-time data now will be highly relevant because the findings will be really useful to helping to understand the current situation.
Unfortunately, data collected in a crisis may not be a rigourous as under ‘standard’ conditions, and its important to acknowledge where there are gaps and uncertainty about data quality. However, usually it is possible to formulate at least tentative conclusions, even from a small amount of data. For example, interviews with a small sample may be done quite quickly, and will provide results where a full-scale survey is not possible. Systematic and valid evaluation data and information will be at a premium right now, and there is much to learn.
The Evaluation Capability Building team is seeking to find ways in which we can support each other as an evaluation community. Please, let us know what you are doing to update your plans and how we can support this. If you have advice and tips for others these can be shared on the live Q and A Forum part of the Website.
Read the write-up of the implications of school closures for Uni Connect evaluation here
Join our Webinar at 10.00 on 22nd May which will discuss the findings of our surveys: pre-registration is not required, just join at the time using link here https://us02web.zoom.us/j/81108606474
You might also be interested in the results of the responses from the survey of parents and teachers which are going out via the University of Exeter Twitter feed at https://twitter.com/UniofExeterNews/status/1257602401210695690
Online links to materials to support evaluation during a pandemic:
Inspiring impact - https://www.inspiringimpact.org/impact-support-during-covid-19/