The Dunning-Kruger effect in accessibility

Published on , in Design and development,

Contents

If you write invalid code, it throws an error. If you get the layout wrong, you can see it when it renders. But, if you build a custom component that doesn’t work with a keyboard, nothing really happens. Your unit tests pass, the page renders, and everything looks fine. So, you ship it!

The people that can’t use it usually won’t file a bug report. They’ll just close your website or delete your app. So you don’t really get any signals telling you that something is wrong. And, a lack of signals, feels a lot like you’re doing everything right!

I’ve been thinking about why this happens so often in accessibility, and it keeps bringing me back to an old study, and a term which gets thrown around a lot in product development. The Dunning-Kruger effect.

What Dunning and Kruger found

In 1999, Justin Kruger and David Dunning published a paper called Unskilled and unaware of it. In their study, they gave people tests in logic, grammar and humour, and then asked them to estimate how well they thought they had done compared to everybody else.

What is really interesting, is that often, the people with the worst scores thought that they performed above average. For example, people who scored around the 12th percentile, estimated that they were around the 62nd. These people did pretty awful on the test, and yet they had absolutely no idea and were confident in their performance.

The argument Kruger and Dunning made, is that the skills we need to do something well, are actually the same skills we’d need in order to be able to assess whether we actually did it well. It’s like that old saying, “you don’t know what you don’t know”, and when the gap between what we know and what we don’t is too big, we just cannot make an accurate assessment.

There was also a second finding, which I’ve noticed gets quoted far less. The people who did well on the tests often underestimated how they compared to everybody else. They were reasonably accurate about their own score, they just assumed that the things they found easy were easy for everybody. And, when the researchers showed them how other people had actually done, only then would they revise their own estimates to be higher.

There are some caveats

Before we get into it, there are a couple of things worth calling out.

First, a lot of articles about the Dunning-Kruger effect show a completely made up chart. If you look up the effect on Google Images, you’ll see a lot of charts with the same labels, like “Mount Stupid” and “valley of despair”. But, this chart does not appear anywhere in the original paper, so be careful when looking up other articles on it.

Second, not everyone agrees the effect is even real. In 2020, Gilles Gignac and Marcin Zajenkowski argued that most of it is just a side effect of the maths. Almost everybody guesses they are somewhere just above average. So, the people who score at the bottom will always look like they have massively overestimated themselves, and the people at the top just look a little pessimistic. In 2023, Dunkel, Nedelec and van der Linden re-ran this analysis, and they concluded that the effect is still real, just much weaker than the original paper claimed.

Despite these caveats, I think the Dunning-Kruger effect is still a useful label for some patterns I’ve observed when people attempt accessibility during product development. I’m not going to suggest we think of it as a scientific principle, or that it’s always going to apply to everybody. But, I do think the model holds up well, if even just to explain my observations.

How the Dunning-Kruger effect often plays out for developers

Broken feedback loops

I think the reason accessibility is such an easy environment for the Dunning-Kruger effect to play out, is because feedback loops are often broken. Kruger and Dunning were just describing people who couldn’t judge their own abilities. But, in accessibility, even if we can judge our own work, most of the time, nothing is actually prompting us to.

Most mistakes in development have a very short feedback loop. Invalid code throws an error. A broken layout looks broken as soon as we open it in the browser. But, with accessibility, the app compiles, the page renders, our tests pass, and none of our tooling is saying it’s inaccessible. Yet, it might still be completely broken for somebody using assistive technology, and we might never know.

Automated checkers can provide a false sense of security

Automated checkers can be both a blessing and a curse, because they’re great at what they do, but what they do is very limited.

GDS created the world’s least accessible webpage which has 143 common accessibility issues on it, and they tested a bunch of automated tools against it. They found that the best tool only found 41% of them, and the worst only found 17%. Which obviously isn’t great.

However, the problem is not really that automated tools only find around 40% of known accessibility issues, it’s that the gap in the test coverage is not clear. They just give us a big green tick, a score, or say there are zero violations. And, that’s really convincing when we don’t have the knowledge to realise this shortcoming. We’re just blissfully unaware of the other 60% of issues the tool couldn’t find.

When this scenario plays out, there is still an immediate feedback loop, but the gaps in the tooling and the gaps in our own knowledge perfectly align. So, we’re confident that it’s accessible, but we ship a broken feature.

The real feedback loop is far slower. We don’t actually find out there is a problem until somebody tells us about it. And, this can be months later, as users often won’t bother to file bug reports, they’ll just leave our website, delete our app and move on.

A common accessibility journey

If we have no signal or knowledge to tell us something is wrong, it’s easy to just assume we’ve got it right. I guess this is one of the main takeaways from the Dunning-Kruger paper, and I’ve seen this process play out in many teams, in a fairly predictable order.

To be clear, these phases are my own personal observations of how people often learn accessibility. They’re not from the paper. Kruger and Dunning compared different people at a single moment in time, but never followed anybody on a learning journey.

1. Little knowledge, and over-confidence in tooling

Phase 1 is the start of most of our accessibility journeys. We’ve been told we need to do accessibility, so we do some Googling, and we install a few automated checkers like axe, Wave or Lighthouse, which all market themselves as the solution to our problem.

The process for phase 1 looks something like the following:

  1. We run an automated checker, like Google Lighthouse
  2. It flags an issue, for example a chart is missing an alt attribute
  3. We do what the tool suggests, we add a basic text description like “chart”
  4. We re-run the tool, the issue looks fixed, our score is 100, and everything is green
  5. We believe it is now accessible, we mark the task as done, and we ship the broken feature

2. Some knowledge, and the discovery of WAI-ARIA

Phase 2 is usually a rude awakening. We suddenly find out that what we’ve delivered does not meet the required standards. It’s usually an external audit, a customer complaint or a tanked usability session. We re-run Google Lighthouse, everything is still green, and the score is still showing as 100.

In phase 2, we start digging deeper, and we often discover WAI-ARIA (Web Accessibility Initiative - Accessible Rich Internet Applications). If you haven’t heard about WAI-ARIA before, welcome to phase 2. And, pay close attention!

WAI-ARIA is a specification, which includes additional roles, states and properties. It gets implemented in browsers under the hood, so that we can add attributes to native HTML elements and augment them with additional cues for accessibility. For example, an HTML <button> element is just a button, but when we add aria-expanded="true", it communicates to a screen reader that it now has a state, which can be either expanded or collapsed. This sets the expectation for the user that it will behave like a disclosure component.

There’s an old saying, that “the first rule of ARIA, is don’t use ARIA”. This probably sounds a bit weird after I’ve just told you how useful it can be. But, when overused, it creates a lot of noise and cognitive load for screen reader users. Native HTML elements are far more consistent and predictable with assistive technologies, so we should use them wherever possible, and only reach for ARIA when we’ve run out of options.

The process for phase 2 often looks something like the following:

  1. We learn of WAI-ARIA, and start making assumptions about how assistive technologies will behave
  2. We add aria-label attributes to almost every element
  3. We add role attributes to <div> and <span> elements
  4. We add aria-live regions to announce anything that changes
  5. We believe it is now accessible, we mark the task as done, and we ship the broken feature

Now, here, the feedback loop starts to get murky. Because it relies less on the kind of knowledge we can read up on. For example, it becomes less about how to use a screen reader for testing, and more about experience and context. We can’t really learn the preferences and behaviours of real people without talking to them, observing them, and understanding them. And, what works on one application might not work on another.

The reason phase 2 can be challenging, is because different browsers and different assistive technologies have their own quirks. It’s like speaking the same language in two different countries. They’re the same, but different. In English, we have stupid phrases like “cat got your tongue?”, which make absolutely no sense without any cultural context. Somebody can speak English perfectly, and still be absolutely dumbfounded the first time they hear it. But, through repeated exposure, we all just come to know what it means, and our brains translate it without thinking.

The same kind of process happens for users of assistive technology. For example, TalkBack on Android devices will announce a button using the disabled attribute as “disabled”. All good! But, VoiceOver on Apple devices will announce it as “dimmed”. And, a VoiceOver user will have learned this nuance, because that’s just always how it is.

It’s fairly common to see this as a bug when we first start using VoiceOver, because we have an expectation, or an assumption, of what we think should be read out. So, we try to hack the component around it. For example, instead of using the disabled attribute, we might try and mimic the behaviours of a disabled button, to trick VoiceOver into announcing it as disabled, and to meet our own expectations:

<style>
  .btn-disabled {
    background-color: grey;
    cursor: not-allowed;
  }
</style>

<button aria-label="Continue: Disabled" class="btn-disabled" id="continue" type="button">
  Continue
</button>

<script>
  const button = document.getElementById("continue");

  button.addEventListener("click", (event) => {
    if (button.classList.contains("btn-disabled")) {
      event.preventDefault();
      return;
    }
    submitForm();
  });
</script>

We could probably get this to work if we put in enough effort. But, for regular users of VoiceOver, it would be extremely jarring! When every website and app they use announces disabled buttons as “dimmed”, but this particular one is announcing it as “disabled”, it adds cognitive load. And, any additional cognitive load very quickly degrades the user experience.

3. Enough knowledge to realise the gaps, but not enough to bridge them

Phase 3 is the most humbling. It’s the phase where we realise that we’ve been doing a lot of things wrong, and our confidence can take a bit of a knock. I remember this phase well. I remember the guilt and the embarrassment, when thinking about all of the inaccessible things I had built and put out into the world.

We now know enough about accessibility to see our own knowledge gaps, but not enough to feel confident we could find or fix the issues in our own code. The amount of available information feels overwhelming, and it’s easy to get stuck in analysis paralysis.

But, this is also the phase where most of the learning is done. It’s where we actually commit properly to learning the craft, the tooling and the needs of the people that use our products. We read articles and case studies, we reverse engineer accessible components and patterns, and we iterate and iterate until our expectations start to align with our test results.

4. Enough knowledge to doubt everything

Phase 4 is an interesting one. And, I see it a lot in very experienced people. By now, we have so much knowledge, and so much experience, that we start to underestimate just how much we actually know.

We’re so well versed in the Web Content Accessibility Guidelines (WCAG) that we start to notice nuances in the wording. We begin to understand that the success criteria are normative, and are the only things we can genuinely pass or fail, and that the techniques and the understanding docs are only informative. The difference between “sufficient” and “advisory” techniques also starts to become clearer. Once we read it this way, we notice the things we’ve been failing, which probably were never actually failures at all.

For example, 1.4.11 Non-text Contrast requires user interface components to have a contrast ratio of at least 3:1 against their adjacent colours. So, for years, a lot of auditors, myself included, failed buttons whose background colour or border didn’t contrast enough with the page. But, if we read the understanding docs properly, the text on a button needs to pass 1.4.3 Contrast (Minimum), and an icon needs to pass 1.4.11 Non-text Contrast at 3:1, but the physical boundary of the button does not actually need to contrast at all. The text or icon is the thing which needs to be perceived, not the actual hit area of the button. So, at times like this, we can suddenly find ourselves wondering how many times we’ve written reports or given advice which was wrong.

But, by now, we’re usually the people everybody goes to for answers. We’re “the accessibility expert”, so it’s easy to start feeling like a bit of a fraud. And, when we look to the community for answers ourselves, everybody else seems equally unclear or non-committal. So, we just end up feeling unsure about our abilities all over again.

5. Enough knowledge and experience to accept it will never be perfect

Phase 5 is where we all need to get to. In this phase, we recognise the shortcomings of WCAG, and we understand that compliant and usable are very different things.

Here, our confidence comes back, but it’s a different kind of confidence. We’re now confident to say something is compliant, not because we’re following a checklist created by somebody else. But, because we’ve got experience, we’ve formed our own interpretations of WCAG, and we can articulate them well when questioned.

We’re also confident in the things we release, because we’ve usability tested them with a diverse group of users, and we have the data to back it up. And, crucially, we accept that things will never be perfect. Everybody is different, and what works for 99% of people, might not work for others.

And, most importantly, we understand the importance of feedback loops, we’re approachable, open to listening, and we’re constantly learning and iterating.

How the Dunning-Kruger effect often plays out for organisations

The same process happens at organisational level, just on a much larger scale. We can usually spot an organisation’s accessibility maturity by the types of things it is doing in the public domain.

The process for organisations often looks like the following:

  1. They install an accessibility overlay
  2. They acknowledge the need for accessibility experts, but don’t commit to a specialist role and attempt to attach the responsibilities to regular developer roles
  3. They attempt to hire accessibility specialists, but don’t commit to a full-time position, and instead advertise impossible job roles on short-term contracts
  4. They attempt to hire permanent accessibility specialists, but pitched at a junior level, or with an insultingly low salary
  5. They commit to accessibility properly, they hire specialists at the correct salary, and they have a properly structured practice with senior leadership

Essentially, like a developer, when an organisation’s accessibility maturity is low, there is no way for it to measure its own capabilities. Because, the people who could measure it aren’t there or aren’t included. For example, complaints often go to customer services and never reach the product team. The legal risks often get assessed by lawyers and passed on to policy makers. So, by the time it reaches product teams as a governance process, they know they need to do something, but they’ve no idea what it is, or how to do it. So, again, the feedback loop is broken.

In my experience, people and organisations often incorrectly rate themselves highly, not because they’re lying, but because they simply have no other data to compare with. I find the people and organisations that admit they’ve got a long way to go, are usually in a better spot than the ones that tell us they’ve got accessibility covered!

A bit of self-reflection

As I mentioned earlier, the second finding was that the people who did well often underestimated how they compared to everybody else. This is the one I keep coming back to. Because, as accessibility specialists, I think we fall into this trap all the time.

My own experience of this is definitely true. I’m a quick learner, so I tend to pick up the basics of most things pretty quickly. I think, “meh, this is easy!” Then, I get obsessed, go down an ADHD rabbit hole, and suddenly, I learn just how much there is to actually learn. Impostor syndrome kicks in, and the more I learn, the less confident I feel. Then, the more I start to think everybody else just gets it, and I don’t.

Learning something complex takes time. And, when progress is so slow, it’s often difficult to notice it. It’s a bit like watching the moon, if we stare at it constantly, it doesn’t appear to move. But hours later, it’s much easier to notice just how far it’s travelled across the sky. So, I think we often forget just how far we’ve come, and it’s easy to assume everybody else has the same understanding.

Because of this slow acquisition of knowledge, I think it’s also why we can unknowingly be less helpful than we intend to be. One of my favourite quotes is by Jeff Patton in his book User Story Mapping: Discover the Whole Story, Build the Right Product:

Shared documents aren’t shared understanding.

Jeff Patton

We often just share something and assume people will read it. And, if they read it, we expect that they’ll understand it. We believe everybody has access to the information, so it’s easy to think we have been clear and helpful, when actually we’ve been neither of these things.

As an example, we often just think everybody knows what an accessible name is. So, we’ll write in our audit report, “fails 4.1.2 Name, Role, Value, you must provide an accessible name for the button”, and we just assume the reader of the report understands what this means, and what to do about it.

Conversely, it’s also easy to feel like we can’t admit to something we don’t know, because we feel like we might be challenged by somebody that knows more than we do. We answer a lot of accessibility questions with “it depends”. Which, is usually true, and an interesting debate for people with the same level of knowledge, but it isn’t very useful to somebody who is inexperienced and just needs to know whether something passes or fails.

There are even some of us who pressure people who are less experienced into implementing their own preferences, by stating that they’re WCAG failures when they are not. I’ve ranted about this before in my post “best practice” is just your opinion.

Final thoughts

I know for certain that there are still things I don’t know, and there are mistakes I’ll still make. Which, is kind of the point. As my wife always says, “you win, or you learn”.

Confidence in accessibility definitely tends to fluctuate the more we learn about it. It can feel easy, and it can feel impossible, but eventually, it usually settles somewhere above average and below absolute certainty. And, the people I tend to trust most, are those who are confident enough to be able to come to a defensible decision, based on sound logic and reasoning, rather than those who will say with absolute certainty that they have all of the answers.

The Dunning-Kruger effect might be overstated, and it might just be a statistical quirk. But, I think there is a definite correlation between what the effect suggests, and what I’ve observed in accessibility.

And, the part of Kruger and Dunning’s paper that nobody quotes is the fourth study. When they gave the bottom quartile training in logic, those people got notably better at judging their own performance in that category. So, the fix for “you don’t know what you don’t know”, unsurprisingly, is just learning the thing. And, for those of us who already know the thing, I think we need to get better at teaching it.

As always, I hope this was useful.

Thanks for making it to the end.
Craig

Back to top