Acknowledging Challenge and Avoiding Survivorship Bias in EdTech
Updated: Oct 8
In the world of EdTech, we are surrounded by success stories. We hear from passionate educators who are rightly proud of the long hours, research and training that is involved in the rollout of services and platforms. We also hear from EdTech providers who are eager to highlight the transformative impact of digital tools that they have developed.
What we rarely hear about are failed, or flawed implementations. These can be initiatives where impact was limited or stifled by unforeseen factors, difficulties with infrastructure or insufficient training.
While thinking about this issue, I was reminded of Abraham Wald's famous story of aircraft in World War II. When military engineers looked at returning planes, they initially concluded that they needed to patch up the areas of the aircraft riddled with bullet holes and planned to reinforce the wings with metal plates. In fact, planes hit in the wings were the ones that survived. The planes that failed to return were hit in critical, unarmoured areas, namely the engines and cockpits.
If we build national and school procurement policies only on curated success stories, we risk making the same mistake. Making decisions only based on success stories, is like armour to the wings of a plane that has already landed safely.
In other words, without acknowledging difficult, problematic or failed implementations we risk making decisions based on data about tools working in perfect conditions, rather than the friction and challenges that are the reality in classrooms!
Statistical data to support this metaphor
This bias towards success is backed up by research, highlighting the difficulties in evaluating the true impact of educational technology.
According to a 2026 Department for Education report, 'Assessment of the education technology market in England', (link),
"Reliable data on the actual use of EdTech in schools is limited. Existing evidence often relies on self-reported surveys or procurement data, which indicate what schools purchase but provide little insight into how technologies are used in practice."
The Department for Education recognises the difficulties in schools providing data which self-reports on efficacy of tools and does not give sufficient depth into the practice of using these tools.
In relation to making informed decisions, the DfE report states,
"Schools must balance multiple pressures, including curriculum goals and budget limits. These constraints often force trade-offs, where the most affordable option might take precedence over the most pedagogically effective one. Some schools have also described the situation in which high-stakes accountability and curriculum pressures can force schools to expedite decisions, skipping necessary consultation to implement a quick fix."
This acknowledges the limitations that schools face in choosing truly effective tools, against constraints that can include budgets, training time and infrastructure concerns. These are all factors that should be considered, alongside the educational aims of adopting new technology.
In relation to adoption and training, the DfE states,
"Tools must align with the specific sequencing of the curriculum. Technology that is not closely linked to curriculum goals rarely becomes part of regular teaching."
This section recognises the potential for choosing tools without considering how they benefit teaching the curriculum. There is a huge range of powerful and impressive tools available to schools, but ultimately, schools are judged on how successfully they help children to understand required knowledge. Any selection of tools must include this as a key consideration.
In a previous blog post, I wrote about the OECD publication, 'Education in Sweden, A Diagnostic Review with Analysis of PISA 2022 Results' (link).
One of the most significant findings of this report was that despite the scale of the expansion of use of digital technology in schools, limited data is available, stating,
"Despite a growing body of rigorous research on the use of digital technologies in schools, generalisable evidence concerning their impact on students' educational development is limited, given the highly context-dependent and diverse use of digital resources in schools."
UK AI Strategy
The UK Department for Education has announced plans to use technology and AI in education to improve pupil outcomes and reduce staff workload. (link) This plan includes the aim of using AI tools to tackle the attainment gap and support up to 450,000 disadvantaged pupils.
The Department of Education plan includes a roadmap, spanning from 2023 to present (2026), covering infrastructure, testbeds that include teacher workload and learner outcomes, safety, security and training.
Next steps include creating a taskforce that helps establish whether tools are grounded in high-quality, curriculum aligned content and co-designing and trialling AI tutoring tools. It is stated that these tools will be "curriculum aligned, safe by design and tested through in-school trials to ensure they deliver real educational value."
On paper, this is a welcome and ambitious framework that rightly identifies teacher workload and educational equity as top priorities.
However, the true test of this roadmap will not be found in controlled testbeds or design trials. The risk is that these trials end up studying only the 'surviving planes', the best-case scenarios with reliable Wi-Fi, trained staff and low student to device ratios.
To ensure that data about the rollout addresses real needs in education, the DfE must ensure that they gather information about the challenges faced by teachers and students, areas of success and unexpected failures and importantly, use in realistic, uncontrolled settings.
Gemini Study in Sierra Leone
A recent, high profile trial testing Google Gemini-powered learning tools in Sierra Leone made headlines for yielding up to 2.5 years of maths progress in just eight weeks. This was an impressive achievement, but also a good example of the need to recognise the context and detail of a trial.
The trial succeeded partly because it was a heavily supported intervention. On-site field monitors handled technical glitches, tablets were made available and teachers received extensive and dedicated training. Students learned in paired setups.
The technical report and blog are available through the Google blog site here.
The study was useful in many regards, including highlighting the Socratic potential of educational tools, challenging students rather than providing answers. As the blog describes, Guided Learning is designed to be pedagogically grounded and to build understanding over providing answers.
The blog states,
"The data from Sierra Leone suggest this approach is working. An analysis of over 113,000 interactions during our trial revealed that students used the tool to build conceptual understanding in 91.4% of conversations, rather than simply seeking solutions."
The study's technical report reveals that the software was embedded in a dense human support system. The trial relied on structured teacher training, shared 2 to 1 devices, structured peer-pairing roles and on-site field monitors from partner organisations who managed hardware distribution and technical glitches.
When policymakers cite gains without accounting for the human intervention required to achieve them, they are at risk of mistaking controlled and supported research trials with the reality and challenges of real classrooms.
How Policymakers Can Overcome Bias
If UK education leaders want to avoid the pitfalls related to basing rollouts on incomplete data, they must design evaluation frameworks that specifically account for survivorship bias.
This includes the following steps:
1) Track challenges and unsuccessful trials. Evaluation teams must include analysis of why some schools abandoned tools during of following a pilot. Asking why some schools abandoned use of a product reveals data that is just as significant as asking what the benefit to schools that continued usage was.
2) Be clear about objectives. Aims can be academic, but also developmental. While a digital tools could increase test score by 10%, impact on face-to-face contact, communication skills, motors skills and other developmental skills should also be taken into account.
3) Align with future ready skills. One of the purposes of education should be to prepare children to enter the modern world beyond school. This should include developing their project skills, critical inquiry, digital literacy and problem solving.
Acknowledging failed and challenging implementations is not an argument against educational technology, it is the key to unlocking its potential.
By looking beyond success stories and acknowledging the realties of schools and classroom deployment, policymakers and school leaders can move from hopeful procurement to intentional design.
When we evaluate tools according to classroom challenges, we build systems that are resilient, adaptable and effective. We recognise the unique impact that technological tools can have, while retaining the aims of individual education settings, and the ethos that shapes the identity of schools.
By acknowledging failures as well as successes when considering deployment of technology in education, we stop making our decisions based on the 'planes that landed safely.'
Instead, we align our decisions with the reality of classroom use and start building long-lasting school strategies that use EdTech to address problems where it can be most impactful and most beneficial for students.
