Literacy software effectiveness is best judged by one question: did students learn more with the tool than they would have learned without it? Most vendor reports do not answer that question. They show growth scores, usage minutes, or percentile gains that look impressive but cannot separate the software's effect from good teaching, extra practice time, or normal maturation.
The difference matters because reading software is expensive in both money and instructional minutes. A district that buys a tool on the strength of a dashboard alone may spend a year finding out that the growth it measured would have happened anyway. This guide explains how to read the evidence behind a literacy product, what an effect size does and does not tell you, and why correlation in an EdTech study is not proof of cause.
What counts as real evidence that software improved reading?
The strongest evidence is a controlled comparison. In a randomized controlled trial, students or classrooms are assigned by chance to use the software or to continue with business as usual. Chance assignment balances the groups on everything the researchers did not measure, which is what makes the comparison fair. If the software group outgains the control group by the end of the study, the tool is the most likely explanation.
Weaker designs exist, and they are common. A pre-post study measures the same students before and after using the tool and reports the gain. But students grow in reading anyway across a school year, so a gain alone proves little. A comparison-group study measures software users against non-users, but without random assignment the groups may differ from the start in prior achievement, home support, or teacher quality. Those differences, not the software, can drive the result.
When you review a vendor's evidence summary, look for three things: who was compared with whom, how the groups were formed, and whether anyone independent of the vendor ran the study. A study the vendor funded is not worthless, but it deserves more scrutiny than one conducted by an independent research team.
How should district leaders read an effect size?
An effect size expresses a gain in standard deviation units, a statistical measure of how much a group spread out around its average. An effect size of 0.20, sometimes called a small effect, means the treated group moved about a fifth of a standard deviation further than the comparison group. Larger numbers mean larger gaps between groups. The scale is standardized so results from different studies can be compared.
Two cautions apply. First, small effects can matter at scale: a modest gain spread across every student in a district adds up to real learning. Second, tiny effects on narrow tests may not matter at all. Ask what outcome was measured. A gain on the vendor's own embedded assessments is weaker evidence than a gain on an independent, nationally normed reading test the vendor did not write.
Also ask how long the study ran and who the students were. A twelve-week study with fourth graders in one suburban district may say little about a full-year deployment with different students. Evidence travels poorly when the study population looks nothing like your own.
Why does correlation not prove the software caused the gains?
Correlation means two things move together. Students who use a reading app more may score higher. That pattern alone cannot tell you which way the arrow points, or whether a third factor explains both. Students who log more minutes may be the students who were already more motivated, had better attendance, or drew the more experienced teacher. Any of those could raise both usage and scores.
Researchers call these hidden differences confounding variables, factors that influence both the thing being measured and the outcome. A dashboard that shows high-usage students outperforming low-usage students is displaying a correlation, not a causal finding. The honest reading is: these went together in this dataset, for these students, in this year.
Usage data has a second trap. Minutes logged measure exposure, not learning. A student can spend thirty minutes in a program without mastering a single skill, and a well-designed program can produce gains on far less time. Treat usage as a compliance signal, never as an outcome.
What should districts ask vendors before signing?
Practical steps here are questions, not promises. No answer guarantees results, and your district, state, and student population vary. But a vendor's responses reveal how much of its claim rests on evidence and how much on marketing.
- Who ran the studies? Independent researchers or the vendor's own team. Ask for the study reports, not just the summary slide.
- How were comparison groups formed? Random assignment is the gold standard. Matched comparison groups are acceptable if the matching was on prior achievement. Pre-post gains alone are the weakest tier.
- What outcome was measured? An independent, normed reading assessment beats the product's internal metrics.
- Who was in the sample? Grade levels, prior achievement, and settings should resemble your schools.
- Is the evidence current? A study of an older version of the product may not describe what the district would buy today.
- What did the study cost the schools? Instructional time, staffing for rollout, and coaching all carry a price the effect size does not show.
Districts that want a structured approach to the broader purchase decision can pair these questions with guidance on how school districts actually buy EdTech platforms, which covers procurement steps beyond the evidence review.
How can districts run their own check after adoption?
Many districts pilot before a full purchase. A well-built pilot can produce local evidence, but only if it includes a comparison. The cleanest version is a waitlist design: some schools or classrooms start now, comparable ones start next semester, and both take the same independent assessment before and after. If the early group outgains the waitlist group, that is evidence with local relevance.
Without a comparison group, a pilot can still tell you useful things that are not causal claims: whether teachers found the tool usable, whether students could access it reliably, and whether the data pipeline worked. Those implementation findings matter. A program with strong published evidence still fails in practice when rollout is poor, which is why teacher preparation deserves as much attention as the evidence base.
Set the success criteria before the pilot starts, in writing, and decide in advance what result would justify expansion. Deciding the bar after seeing the data invites wishful thinking.
Our analysis: what the evidence pattern actually shows
Across the research on education technology, a consistent pattern emerges: effects are usually modest, they depend heavily on how the tool is used, and they shrink when the study design gets stronger. That is not a reason to avoid literacy software. It is a reason to buy with expectations calibrated to what the data supports. The honest summary of most evidence in this field is that some tools, for some students, under some conditions, produce measurable gains, and that the conditions matter as much as the product.
Districts that treat vendor claims as hypotheses to test locally, rather than conclusions to accept, tend to make better decisions. Pair an evidence review with a comparison-group pilot, an independent assessment, and clear criteria. For a related lens on evaluating adaptive products, see what research says about adaptive learning software, and for the teacher-facing side of a rollout, how districts prepare teachers for a new EdTech platform. More coverage of tools and procurement lives in the site's EdTech section.
The unknowns are real. Independent evidence on many specific products is thin, study results age as products update, and no published study can fully predict what a tool will do in a particular district's classrooms. What the evidence establishes is the method: compare, measure with independent instruments, and be skeptical of gains that arrive without a control group. What remains unknown, district by district, is exactly which tool fits which students, and only a local test can answer that.
