Blog entry by Outreach Evaluation Hub
On 23rd March, as schools across the country closed their doors and a stream of rolling news outlined the unquantifiable but certain impacts on learning and attainment for young people, colleagues across the Widening Participation (WP) and outreach community swiftly turned their attention to alternative, socially-distanced delivery models and ways to quickly address the inevitable widening of the ‘disadvantage gap’.[i] Although the effects of school closures have doubtless been far-reaching, the cohort of young people uncomfortably labelled ‘disadvantaged children’ have reportedly felt the immediate impacts most keenly and look set to experience the repercussions for years to come. Becky Francis, Chief Executive of the Education Endowment Foundation, predicts ‘at least a reversal’ of the progress made over the last ten years in closing the disadvantage gap for GCSE students, while Robert Halfon, Chair of the Commons Education Select Committee, warns of a ‘potential cascade of mounting social injustice’ that could itself last a decade.[ii]
The uncertain context in which schools and learners found themselves inevitably manifested unprecedented support needs for teachers, parents, and learners, while simultaneously unprecedented challenges and limitations to some aspects of WP outreach as we knew it. Many experienced practitioners well-versed in face-to-face outreach and evaluation were suddenly traversing uncharted terrain in the new WP world order where online approaches necessarily feature as a primary mode of engagement (in spite of pertinent, lingering concerns about the ‘digital divide’[iii] and accessibility of online content). What quickly became evident as delivery staff attuned themselves to this brave new world was an aporia of evidence on the subject of ‘what works’ in online delivery, and an urgent scrambling for practical, accessible guidance which might enable practitioners to continue to engage young people in effective ways from a distance.
Although similarly beset by challenges, evaluators and researchers quickly reached consensus that we had also been presented with an organic opportunity to implement neoteric research approaches through which we might better understand the impacts of the remarkable and unique circumstances school-age young people now find themselves in. Implementing robust approaches to evaluation of online and distanced outreach provision affords us the prospect of starting to lay some data-driven foundations on which delivery staff might begin to build a new frontier for WP: one which is quickly reactive to the specific needs and requirements of our learners; is innovative and evidence-based; is applicable in a technology-driven context; and which can potentially bridge barriers and divides where face-to-face modes of WP might not be able to.
In reaction to the immediate needs and questions being posed by Uni Connect delivery staff, the Study Higher Evaluation Team compiled a toolkit to practically support and inform the design, monitoring processes, and evaluation of online activity. The toolkit is intended to be an empowering resource for anyone involved in the creation, monitoring and evaluation of online activity and resources, having been created to be accessible and easy to implement regardless of the level of evaluation-specific knowledge or experience on the part of those designing and delivering new online interventions.
Although elements of the toolkit are bespoke to Study Higher’s specific context and operation, some of the guidance and suggestions may be of use for other consortia and institutions seeking practical cornerstones to creating, monitoring and evaluating their own activity. Five helpful highlights from the toolkit are included below.
1. Continue to set intended aims and outcomes at the activity/resource design stage
The context we are operating in may be radically different at the moment, but it’s business as usual in terms of how to go about activity and evaluation design. Although this may pose some practical or logistical challenges in an online setting, the principles of robust activity and evaluation design remain the same: all activities and resources should have intended aims and outcomes outlined from the start, as well as clear indicators of what engagement and impact will ‘look like’. For Study Higher it has been important for us to be able to support our Higher Education Liaison Officers (HELOs) to retain an outcomes-focussed approach as they develop new activities and resources or work to adapt existing activity and content for alternative modes of delivery; in our case, these outcomes continue to sit within Study Higher’s broader progression framework which is founded upon the praxis-based NERUPI model.[iv]
2. Capture small steps towards change
Guidance from the Office for Students outlines that a sustained and progressive programme of outreach with multiple activities is likely to have a more positive impact on learners’ knowledge and attitudes toward higher education than single or ad-hoc outreach activity.[v] However, engaging learners at all is currently an outright challenge, and many Uni Connect consortia are re-evaluating and adapting progression frameworks to reflect the unusual circumstances learners find themselves in, and the new challenges and barriers they face.
To effectively outline how activity contributes to the ‘bigger picture’, it is worth creating a very simple theory of change[vi] model up-front to demonstrate exactly how your activity/resource will impact your target learner(s). Theory of change models range from the very simple type through to incredibly complex, programme-level models, but at their heart, are simply a visual means of mapping the assumptions which inform an intervention at each stage, and of capturing the network of factors which can influence project or programme outcomes.
Where previously we may not have logged conversations or ad-hoc interactions with learners, it might be helpful to establish a method for recording more ad-hoc interactions and conversations you might have with individual learners given the current context and added complexities of engaging with young people directly. Given the lack of certainty and direction many young people are experiencing in knowing the best options for them to take in this climate of such colossal uncertainty, even minor, less ‘formal’ interactions may help shape learners’ decisions and contribute to their progression in the medium and longer term.
3. Reflective practice is now even more essential to evaluation and outcome
For activities and interventions delivered ‘at a distance’ where capturing quantifiable outcome data may be challenging (for example, in a virtual mentoring setting, an informal Q&A, or discursive workshop) the practitioner’s experience and perceptions of the intervention and outcomes for participant(s) become all the more pertinent.
As soon as the ‘live’ activity ends, note down anything - positive or negative – you perceive in relation to the new, distanced approach to delivery, and anything which evidences a perceived impact(s) of this (or not, as the case may be). Include any comparative insights you may have in relation to face-to-face modes of delivery for similar activity, being sure to document notable changes to dynamics or elements that worked well / didn’t work as well as anticipated. This could be especially insightful if you have delivered the activity in a face-to-face setting before, as you will be well-placed to take note of how the activity or interaction is enhanced or impacted due to being facilitated online. While is not possible to entirely mitigate the issue of subjectivity in this practice, it is helpful to contextualise and corroborate practitioners’ insights with supporting qualitative and/or quantitative data from the participant(s) themselves.
Although this sounds like a straightforward undertaking, it is important for the person undertaking reflective practice to feel comfortable to be as honest and open as possible. This potentially means including any impacts or outcomes which they perceive to be negative or unintended, and highlighting if an activity fell ‘flat’ or failed to engage participant(s) for some reason. Acknowledging shortcomings in this way may feel uncomfortable, but is an important feature of self-reflection and evolution, as well as for facilitating the following suggestion.
4. “Move fast and break things”…
Adopted as a popular mantra by entrepreneurs after being quoted by Facebook’s founder and CEO Mark Zuckerberg, the heart of this statement reminds us that sometimes we need to break things to improve them and that making mistakes is, to some extent, an intrinsic consequence of innovation. With ten years of lost ground in terms of closing the disadvantage gap, and the potential for detrimental impacts on some young people to last a further decade, now is the time for affirmative and (reason-informed) radical action.
Zuckerberg’s mantra doesn’t hold up entirely under scrutiny – not least for its brash application and lack of forethought about possible consequences or room for reflection[vii] – but it remains true that, in these unprecedented times, we have an opportunity as a sector to break away from some of the ways WP operated prior to lockdown, to try new things, and to learn from the outcomes of our shared experiences. Now is an apt time to take attenuated risks, to try new things, to ask for honest feedback, and to be prepared to adapt and improve quickly as a result of new insights. To do this responsibly and in such a way as to create a strong infrastructure in the WP digital frontier, it’s vitally important that practitioners keep a record of the new approaches we are trying, along with our rationale and risk-mitigation measures. As more evidence becomes available, it’s possible we will find that these innovations have forged a path to new and exciting modes of engaging and supporting young people.
5. …but use your data to fix them quickly and regularly
Policy and priorities continue to change quickly at both a local and national level at the moment, and insights into impact and what does/doesn’t work in distanced outreach and evaluation practice are bound to emerge at a fast pace. Swift recording, sharing, and analysis of monitoring data and outcomes from process and impact evaluation enables us to continuously improve and develop in-house provision and maximise the impact of interventions. Given the fast pace of change and new research being conducted, a swift turnaround of evaluation and impact data is essential for us to be able to quickly learn more about ‘what works’ in socially-distanced outreach activity and to share this with our colleagues across the sector.
Vitally, we also retain an ethical responsibility to ensure that we evaluate our activities responsibly. The Academy for Social Sciences list among their key principles for ethical research that ‘All social science researchers should acknowledge their social responsibilities’, and also that ‘All social science should aim to maximise benefit and minimise harm’.[viii] Swift review and adaptation is essential to pre-empt the potential perpetuation of any negative impacts on participants and to maximise the benefits for the young people we work with.
Although the current situation poses unprecedented challenges and limitations to some aspects of our work as it was, the shift to online delivery and evaluation is an exciting change laden with possibilities and potential for innovation, great insights, and to contribute meaningfully to an emerging area of research. Over the forthcoming weeks and months we are bound to learn much more – as individual practitioners, as consortia, and as a sector - about the many benefits, challenges, impacts, and possibilities intrinsic to distanced and online outreach approaches.
Emily Scott, Monitoring
and Evaluation Manager, Study Higher
[i] Children’s Commissioner for England, 20th April 2020. Tackling the disadvantage gap during the Covid-19 crisis.https://www.childrenscommissioner.gov.uk/publication/tackling-the-disadvantage-gap-during-the-covid-19-crisis/
[iii] H. Holmes & G. Burgess. (n.d.) “Pay the wi-fi or feed the children”: Coronavirus has intensified the UK’s digital divide. University of Cambridge, https://www.cam.ac.uk/stories/digitaldivide
[iv] Further information about the NERUPI framework is available on their website: http://www.nerupi.co.uk/about/nerupi-framework-overview
[v] C. Millward, 30th October 2019. Sustained outreach makes the difference. Office for Students, https://www.officeforstudents.org.uk/news-blog-and-events/blog/sustained-outreach-makes-the-difference/
[vi] Information about theory of change models and their rationale can be found at: https://www.theoryofchange.org/what-is-theory-of-change/
[vii] An explanation of the adoption of Zuckerberg’s controversial mantra and how this interesting article on this subject can be found at https://www.businessinsider.com/facebook-hack-shows-again-the-downside-of-move-fast-and-break-things-2018-10?r=US&IR=T
[viii] R. Dingwall et al (2014). Towards Common Principles for Social Science Research Ethics: A Discussion Document for the Academy of Social Sciences (pp.7-8), https://www.acss.org.uk/wp-content/uploads/2014/06/Ethics-Final-Principles_16_06_2014.pdf