Evidence Quality Judgment

Rate each finding on its source, its data and its analysis, check whether independent sources agree and who is missing, then choose words that match the grade.

Mixed research: grading evidence comes from medicine (Guyatt et al., 2008), and critics argue that management questions need a match between question and study type rather than one ladder (Petticrew & Roberts, 2003). Triangulation is a widely accepted validation strategy (Denzin, 1970; Jick, 1979). The survivorship lesson rests on strong statistical ground (Mangel & Samaniego, 1984). The three confidence levels and their wording are a rule of thumb.

Use it when:

• You have themes or results from a first pass and are about to present them as findings.
• A finding will drive a costly or hard-to-reverse decision, and you need to say how sure you are.
• Stakeholders disagree about what the data shows, and you want the disagreement to be about evidence, not tone.

Skip it when:

• You have not sorted the raw data yet: start with 2.8 First-Pass Analysis.
• You are judging a single outside source (an article, a report) rather than your own findings: use 2.15 Source Quality Ladder.

Grading evidence instead of counting it is a habit from medicine, where structured systems rate how trustworthy the evidence is and, as a separate call, how firmly a recommendation should be made (Guyatt et al., 2008). Outside medicine the picture is mixed. Critics point out that the best kind of study depends on the question, so no single ladder fits management work (Petticrew and Roberts, 2003). This page therefore grades along three dimensions rather than one rung.

Two methods sit underneath. Triangulation asks whether independent sources point the same way (Denzin, 1970; Jick, 1979). Agreement earns more confidence, but it is no guarantee, because sources can share a blind spot. The survivorship lesson asks who never made it into your data, since the missing cases are often the most telling (Mangel and Samaniego, 1984). So Step 2 rates source, data and analysis separately, Step 3 asks about independence and absence, and Step 4 ties each grade to the words you will use. Stated limits are not an apology: they let a decision maker use medium evidence safely.

Write each finding as one checkable sentence, and name the decision it would change. Grade the findings that carry the most weight first.

Strong answer: "Cafes that stop ordering mostly throw away bread at the minimum order. Feeds: whether to lower the minimum for small cafes." Not: "Cafes are unhappy."

Illustrative example: the business, the roles and all numbers are made up.

The Gothenburg bakery from 2.1 has three findings from its first pass (2.8).

Step 1: (1) Cafes that stop mostly waste bread at the minimum order. (2) Cafes ordering at the minimum stop more often. (3) Price is not the main reason. All three feed the owner's April decision on the cafe route.

Step 2: Finding 1: source H, data M, analysis M (one coder). Finding 2: source H (the order system), data H, analysis L for cause, since small cafes may also be fragile. Finding 3: source M, data L (price raised once), analysis L.

Step 3: Findings 1 and 2 come from separate sources that agree. Four former cafes declined an interview and are missing.

Step 4: Finding 1 medium: "indicates". Finding 2 medium for the pattern, low for the cause. Finding 3 low: "we found little sign that price drove the decision, but we did not test it directly".

Step 5: The sales lead puts the quality statement on the first page of the brief, and asks the head baker, who believes in price, to challenge Finding 3 before the meeting.