Key Takeaways
- Peer-reviewed research published in Risk Analysis found that qualitative risk assessment systems can produce reversed risk rankings, meaning a genuinely higher risk scores lower than a less serious one, and that the same qualitative label such as "high" can apply to risks that differ by many orders of magnitude in actual expected loss.
- Qualitative risk assessment is the right method when data is scarce, speed matters, or the goal is triage. Getting risks onto a register with named owners is more valuable at that stage than waiting for data precision that does not yet exist, and the team conversation the scoring process generates is often more useful than the scores themselves.
- The most dangerous use of qualitative risk assessment is when a risk matrix score becomes the basis for budget allocation or board-level decisions, because two identical scores on a heat map can represent completely different levels of financial exposure.
- A qualitative risk score is an ordinal judgment rather than a measurement. A score of 12 out of 25 implies precision the underlying process does not have, and teams that treat it as data rather than opinion are making resource decisions on a foundation that cannot support them.
- Qualitative and quantitative methods serve different decisions. When the question is which risks need an owner and a review date, qualitative scoring is sufficient. When the question is how much contingency to set aside or where to allocate risk reduction budget, expected monetary value or Monte Carlo simulation gives you an answer you can actually defend.
What qualitative risk assessment actually is and where it falls short
Qualitative risk assessment is the dominant approach across organisations of all sizes. The format varies, high/medium/low, red/amber/green, a five-by-five matrix with colour-coded cells, but the underlying method is the same. Ask any operations lead, quality manager, or risk officer how they assess risks, and the answer is almost always some version of this.
The appeal is obvious. You do not need a statistician or historical loss data, just two people in a room, a shared understanding of the risk, and a rough sense of how likely it is and how bad it could be. Thirty minutes later, you have a populated risk register.
That accessibility is a genuine strength. But there is a body of serious academic research that should give any practitioner pause, and that the majority of articles on this topic quietly ignore.
A qualitative risk assessment assigns descriptive categories to probability and impact rather than numeric probabilities or financial values. Risks are scored on a scale (usually 1-5 or low/medium/high), the two scores are combined to produce a risk score, and that score places the risk somewhere on a matrix. The method itself is sound. The problems begin when the output is treated as something it was never designed to be.
The specific failure modes
In 2005, research published in Risk Analysis (Cox, Babayev, and Huber) identified several structural limitations of qualitative risk rating systems. These are worth naming directly, because a great deal of risk management content either ignores them or buries them in caveats.
Reversed rankings. A qualitative system can assign a higher risk score to a less serious risk than to a more serious one. This happens because the categories are ordinal rather than cardinal. The difference between 'low' and 'medium' probability is not the same as the difference between 'medium' and 'high', but the matrix treats them as if they were. When you combine two ordinal scales, the resulting score is mathematically unreliable, and two risks with the same red score can have wildly different actual expected losses.
Uninformative ratings. If most of your risks cluster in the amber zone, and they usually do, the rating tells you almost nothing about relative priority. When a register shows forty-seven risks all rated amber with nothing ever moving to red, the rating system has produced a category that is too broad to act on.
False precision. A score of 12 out of 25 sounds precise, but it is a subjective judgment expressed as a number rather than a measurement. The precision of the format implies a rigour the underlying process does not have, and teams that treat the output as data rather than opinion are making decisions on a foundation that cannot support them.
A separate 2008 study in Risk Analysis (Cox) went further, arguing that risk matrices can be 'worse than useless' in specific conditions, actively leading decision-makers toward the wrong prioritisation. That is a strong claim, and it applies most acutely when the matrix is used to allocate resources, a purpose it was never designed to serve.
None of this means qualitative risk assessment should be abandoned. It means understanding which jobs it is suited for and which ones call for something more rigorous.
When qualitative methods are exactly the right choice
The counterintuitive reality is that the academic criticism of qualitative risk assessment is aimed at a specific misuse of it. When used appropriately, qualitative assessment is often the best available option, and in many contexts it is the only honest one.
Early-stage assessment. You are scoping a new project with no historical data and no time for a quantitative model. A qualitative risk register built in an afternoon is far more useful than waiting for data that does not exist yet, and far more honest than a quantitative model built on assumptions nobody can justify.
Limited data environments. For the majority of operational risks in SMEs, there is no actuarial database. You do not know the base rate of a supplier failure or a regulatory change, and forcing quantitative precision onto risks where the underlying data does not exist creates false confidence rather than reducing it. A structured qualitative judgment is the more honest starting point
Triage. At the triage stage, the goal is to identify which risks need attention now, which need an owner, and which can wait rather than to produce a precise ranking of every risk. Qualitative methods are well-suited to that sorting function, and getting risks onto a register with owners and next steps is more valuable at this stage than spending three weeks on a Monte Carlo model.
Facilitating team conversations. When a quality manager runs a risk session with an operational team, the scores are almost secondary to the conversation that produces them. Qualitative methods give a shared language to people who have never formally assessed risk before, and the discussion surfaces things that would otherwise stay unspoken.
We see this regularly with teams using Risk Companion's interactive risk sessions: the scores matter less than the fact that everyone is in the room, using the same categories, and naming risks they had previously only half-acknowledged.
Where qualitative assessment quietly goes wrong in practice
The failure is rarely in the method itself. It tends to happen in the moment the output gets used for something it was never designed to handle.
Picture a construction company with a well-maintained risk register, regularly reviewed, with clear owners, probability scores, and impact scores for every risk. The risk matrix shows a cluster of amber risks and one persistent red. When the board reviews the register quarterly, they focus on the red risk and treat the amber cluster as managed.
One of those amber risks is a dependency on a single subcontractor for a specialist process, with a qualitative score of probability 2, impact 3, giving a risk score of 6. Moderate, by the matrix. But the actual financial exposure if that subcontractor fails is close to €400.000 in project delays and remediation costs, a figure that would make it a top-three risk by any quantitative measure. The qualitative score has effectively hidden it.
The risk was not badly assessed. The probability and impact judgments were reasonable as ordinal estimates. The problem was the board treating a score of 6 as a signal that the risk was under control, rather than as a rough triage indicator that still required deeper analysis.
That is the gap between qualitative risk analysis used as a starting point and qualitative risk analysis mistaken for a conclusion.
If your risk register is the basis for a conversation about risk, qualitative methods are appropriate. If it is the basis for a €500.000 contingency budget decision, you need something more rigorous.
What a more calibrated approach looks like
The practical answer is to be clear about what question you are trying to answer and to match the method to that question rather than applying qualitative scoring to everything by default.
For day-to-day risk management in SMEs and mid-market organisations, a well-maintained qualitative register, with clear owners, reviewed measures, and a regular cadence of updates, is sufficient. The goal at this level is accountability and visibility rather than statistical precision. Risk Companion's risk register is built for exactly this: probability and impact scores that give you a triage view, with owners and due dates that turn the register into a working management tool.
But when the stakes justify it, qualitative scoring is only a starting point. Two specific situations call for something more rigorous.
When you need to defend a contingency budget. "We need €200.000 in contingency because our risk matrix has three red risks" will not survive a CFO's first question about how you arrived at that figure. Monte Carlo simulation runs your risk register through thousands of scenarios to produce a probability distribution of financial outcomes. You can show the board that there is a 90% chance the project stays within €180.000 of budget, or that there is a 15% chance of exposure exceeding €350.000, and either figure is something you can defend with the working behind it.
When two qualitative scores look identical but the financial exposure is very different. A probability 3, impact 3 risk involving a €10.000 operational disruption and a probability 3, impact 3 risk involving a €1.200.000 supplier failure look identical on a matrix, but the resource allocation decision between them is completely different. When you need to prioritise between them, expected monetary value or a simulation gives you a basis for that decision that a colour on a grid cannot.
The current versus target assessment approach in Risk Companion helps bridge this gap even without full quantitative modelling. You can see where a risk sits before measures are applied, where you expect it to be once measures are in place, and what gap remains, giving you a more calibrated view than a static score alone provides.
The practical verdict
Qualitative risk assessment is a practical tool that becomes unreliable when organisations stop treating it as one and start treating the output as a conclusion.
Used for triage, team communication, early-stage assessment, and building a shared risk vocabulary, it is exactly the right tool. Used as the basis for resource allocation decisions involving material financial exposure, it is insufficient, and the academic research we have cited is clear on why.
The question to ask before relying on a qualitative score is whether this is the right level of rigour for this particular decision. For the majority of day-to-day risk management, structured qualitative assessment with clear ownership is what you need. Occasionally, the decision in front of you is consequential enough to warrant something more.
Organisations that distinguish between those two situations manage risk better than those that apply the same method to everything, and the difference rarely comes down to better data or bigger teams. It comes down to knowing what their tools are actually for.
Risk Companion's free 14-day trial builds a demo project from your own organisation's profile, so you can see how qualitative scoring, current versus target assessments, and Monte Carlo simulation work together in practice before you commit to anything.
Ready to improve your risk management?
See how Risk Companion can help you implement these best practices with powerful, easy-to-use tools. Sign up and we'll prepare a demo project tailored to your company.