apprun.icuGuides

Guide · Reading research

How to spot weak evidence in a research paper

Weak evidence usually shows up as a gap between what a paper says and what its numbers can carry. Check the wording first, then the p-values, sample sizes and reporting, and write down the questions the text leaves open.

By apprun.icuUpdated Read 9 minIncludes a free browser toolSources 7 cited
On this page
  1. The short answer
  2. Start with the wording, then check the numbers
  3. Read p-values next to effect sizes and intervals
  4. “Not significant” does not mean “no effect”
  5. Check the sample size, the comparisons and what is shown
  6. Hedging can be honest
  7. Questions to ask the authors
  8. What this cannot tell you
  9. Common questions
  10. Sources and further reading
Claim wording compared with evidence strength, with a gap flaggedClaim wordingproves, always, showsEvidence strengthtest, interval, nGap?ask the authors
Compare what the claim says with what the data can carry, and note any gap.

Start with the wording, then check the numbers

Every claim makes a promise about how far the data reach. A verb such as “proves” claims more than “suggests” does. Compare that promise with the numbers before you accept the claim. The table lists common wording and the check that follows from it. If you want the wording sorted for you, paste the paper into the Claim–Evidence Map. It groups words into hedged, moderate, strong and absolute, and shows which words it matched.

Wording in a paper, and what to check next
WordingWhat to checkSource behind the check
“proves”, “establishes”, “confirms”, “shows that” (strong)Is there a test, interval or effect size behind it? Does the claim stay within the sample?The ICMJE advises against unqualified statements and conclusions not adequately supported by the data
“always”, “never”, “all”, “every” (absolute)Does the data cover every case, or only the cases studied?The UNC qualifiers page suggests softer forms such as “many, most, some” for “all”
“may”, “might”, “could”, “suggests” (hedged)Does the hedge match the strength of the evidence? A hedge is not a flaw on its own.UNC warns that excessive qualifiers can make a writer sound unsure of the facts
“not significant”, “no significant difference”Look for an effect size and an interval. Do not read it as “no effect”.The ASA notes that a large p-value does not imply evidence in favor of the null hypothesis
“p < .05” or “p = .03” with nothing beside itIs there an effect size or interval? Is there a sample size nearby?The APA Task Force advises it: “Always provide some effect-size estimate when reporting a p value”
“see Figure 4” or “Table 2”, with the numbers not in the textOpen the figure. Check that its axes are labelled and whether error bars are shown.Keshav’s second pass covers figures and error bars
“[12]” or “consistent with earlier work”Does the cited work test this same claim with its own data?The Claim–Evidence Map scores a citation alone one step above no evidence

Read p-values next to effect sizes and intervals

A p-value is a statement about data in relation to a stated model, not a verdict on the hypothesis itself. The American Statistical Association’s 2016 statement lists six principles. The table gives them in plain words. The wording is paraphrased from the ASA release; the fuller discussion is in the archived statement by Wasserstein and Lazar.

The ASA’s six principles on p-values, in plain words
#Principle in plain wordsSource
1A p-value can indicate how incompatible the data are with a specified statistical model.ASA release
2A p-value is not the probability that the hypothesis is true, nor the probability that chance alone produced the data.ASA release; the archived statement elaborates
3A conclusion should not rest only on whether a p-value passes a threshold.ASA release
4Sound inference needs full reporting and transparency.ASA release
5A p-value does not measure the size of an effect or how important a result is.ASA release
6By itself, a p-value is not a good measure of evidence about a model or hypothesis.ASA release

Principle 5 matters most when a p-value appears on its own. A tiny effect can produce a small p-value when the sample is large enough. The archived ASA statement puts it this way: “Any effect, no matter how tiny, can produce a small p-value if the sample size or measurement precision is high enough”. So a p-value needs an effect size beside it. The APA Task Force recommends the same, and the ICMJE says that p-values alone fail to convey important information about effect size and precision of estimates. Look for an interval as well, such as a confidence interval, which shows the uncertainty around an estimate.

“Not significant” does not mean “no effect”

A result that does not pass a paper’s chosen threshold is not the same as a result showing that nothing happened. The ASA’s archived statement says a large p-value “does not imply evidence in favor of the null hypothesis”. So when a paper says that a difference was not significant (p = .21), the sentence tells you the test did not reach the threshold. It does not tell you the effect is zero.

Check what else the paper gives. Is there an effect size? Is there an interval, which shows the uncertainty around the estimate? If the paper reports only the p-value, the sentence says less than it seems to. Write that down as an open question for the authors rather than as a finding.

Check the sample size, the comparisons and what is shown

A p-value depends on the sample size, so a result with no sample size near it is hard to judge. Because a large sample can make a tiny effect look significant, look for the number of participants or units beside each main result. If it is missing, the result is harder to weigh.

Then check how many comparisons the paper ran. The ASA’s archived statement says “Researchers should disclose the number of hypotheses explored during the study”. The ASA also asks for full reporting and transparency. A paper that reports only its significant results cannot be fully interpreted, because you cannot see what else was tried.

Next, see how results are shown. The ICMJE advises presenting results in “the text, tables, and figures, giving the main or most important findings first”. A key result that appears only in prose, with no table, figure or interval, is harder to check. For figures, Keshav’s second pass asks whether the axes are labelled and whether the results carry error bars. Citations need the same care. A citation shows where a claim came from, not that this study confirmed it. Open the cited work and check that it tests the same thing.

Hedging can be honest

Not every hedge is a warning sign. UNC’s qualifiers page calls hedging an important feature of academic writing, and says writers need to indicate whether they think claims are certain, likely, unlikely, or just false. It also uses the phrase “confident uncertainty”, and its advice is that writers should say clearly how certain they are. A paper that reports a finding, gives its interval and says the data suggest rather than prove something is writing with the confidence its evidence supports.

The reverse problem is more useful to catch. A hedge can sit over a strong result and understate it, but a strong verb over a thin result overstates it. Ken Hyland’s 1998 book on hedging in research articles, as its publisher’s abstract describes it, concludes that hedging is central to scientific argument. This guide has read only that abstract, so treat the book’s detailed findings as unverified here.

Questions to ask the authors

When the text cannot answer a question, write it down. These are the questions the sources above point toward:

  • How many participants or units does each main result come from?
  • What is the effect size, and what interval goes with the p-value?
  • How many comparisons did you run, and which of them are reported?
  • Does the claim apply beyond the sample, and what evidence supports that?
  • Where is the table or figure behind this result, and what do its error bars show?

What this cannot tell you

A weak-evidence check finds gaps in what a paper says. It does not show that the conclusion is false. A paper with a clear table and full intervals can still rest on flawed data, and a paper with a gap can still be right. The ASA’s archived statement ends with a wider caution: “No single index should substitute for scientific reasoning”.

The Claim–Evidence Map is a useful first filter for this checklist, because it flags overclaims and unsupported claims in the text you give it. It cannot judge the methods, whether the sample represents anyone or whether a claim is true. Try it on a paper you already understand and compare its flags with your own reading. For the step-by-step version, read the companion guide on how to read a research paper with a claim–evidence map.

Some common mistakes to avoid:

  • Treating a p-value below .05 as proof. The ASA says such rules “can lead to erroneous beliefs and poor decision making”.
  • Reading a strong verb as strong evidence without looking at the test behind it.
  • Counting every hedge as a flaw.
  • Stopping at the abstract, where a summary can state more than the Results show.

Common questions

Does a p-value below 0.05 mean the finding is true?

No. The ASA states that a p-value does not measure the probability that the studied hypothesis is true. A threshold does not settle a claim by itself, and the ASA cautions against basing conclusions on a threshold alone. Look for an effect size, an interval and full reporting before you accept the claim.

What does “not statistically significant” mean?

It means the result did not pass the threshold the paper chose. It does not show that the effect is zero. The ASA says a large p-value does not imply evidence for the null hypothesis. Check for an effect size and an interval before reading anything into the result.

What is an effect size, and why does it matter next to a p-value?

An effect size measures how large a difference or association is. A p-value does not measure size, according to the ASA. The APA Task Force advises authors to provide an effect-size estimate whenever they report a p-value. Without it, a tiny effect in a large sample can look impressive.

Is a hedged claim a sign of weak evidence?

Not always. UNC’s qualifiers page says academic writers need to indicate how certain they are, and warns that too many qualifiers can make a writer sound unsure. Compare each hedge with the evidence behind it. A hedge over a clear, well-reported result is honest. A strong verb over a single citation is the pattern to question.

How can I tell whether a paper left out other results?

You usually cannot be certain from the text alone. Look for a disclosed count of the hypotheses and comparisons explored, and for full reporting of the results. The ASA’s guidance asks for both. If either is missing, write the question down and ask the authors.

Can a tool tell me whether the evidence is weak?

It can flag wording that runs ahead of the text, such as an absolute claim with no linked number, test or citation. It cannot judge the methods or the truth of a claim. The Claim–Evidence Map reports these results as unsupported in the text you pasted, not as false.

Sources and further reading

Share this guide