Becoming a Trusted Tester
The good, the "bad", the confusing.
I love learning. If you're new here, you might not know this, but if we've met, this is not new information. I wish there were more hours in the day for everything I would like to learn and experience. I don't understand boredom as a concept.
After earning my CPACC (Certified Professional in Accessibility Core Competencies), I was already looking at what would be next. I've been taking courses through DeQue University, but I had also been curious about the DHS Trusted Tester certification. I spent a lot of time reading about the process on the DHS website and digging through Reddit posts from people who had completed it. Eventually, I decided it sounded like something I might actually enjoy, while also building the practical testing skills used in federal accessibility work and opening the door to roles that specifically look for Trusted Tester certification.
So let's dive in to my experience. And I'm going to be very honest about it.
When I was deciding whether to pursue the certification, I relied heavily on other people's experiences to figure out what I was getting myself into. The problem was that most of what I found described an older version of the program. Now that I've made it through the current training, I want to do the same for the next person: talk about what worked, what didn't, what was confusing, and what I wish I'd known going in.
I'm also looking at this through two lenses. I went through the program as an accessibility professional learning a standardized testing methodology, but I also experienced it as someone with a background in learning design and training development. That means I couldn't help noticing not only what the courses taught, but also how they taught it: the sequencing and scaffolding of the content, opportunities for practice and feedback, alignment between instruction and assessment, cognitive load, and ultimately how well the training prepared me to apply what I'd learned.
One quick note before I dive in: "The Good, the Bad, and the Confusing" makes for a ctachy title, but I don't really love labeling learning experiences as simply "bad." There were things that worked really well and things that I think could use significant improvement. So consider "The Bad" less of a verdict and more of a very enthusiastic list of opportunities for improvement.
Let's dive in.
The Basics
The training is split into 6 "courses". I'm going to touch on each briefly here, along with the approximate amount of time I spent in each course.
What is Section 508 and Why is it Important?
This course covered the basics of Section 508 and the responsibilities of a Trusted Tester. Most of this was not new information to me, but I like to take handwritten notes when I'm learning something new to help solidify the material, so I spent about 90 minutes in this course.
Section 508 Standards for Web
This course went more in depth into Section 508 and how it applies to electronic content. Success Criteria were introduced with examples, organized around the WCAG (Web Content Accessibility Guidelines) POUR principles: Perceivable, Operable, Understandable, and Robust.
I was already familiar with much of this content as an accessibility professional who had earned my CPACC, but I spent about 3 hours reviewing the material and taking notes.
Trusted Tester Testing Tools
This course introduced the Color Contrast Analyzer and ANDI (Accessible Name & Description Inspector), including how to download and configure the tools so they were ready to use throughout the training. I spent less than 30 minutes in this course.
Trusted Tester Training for Web
This is where the main content of the Trusted Tester certification process lives. It took me around 23 hours to gradually work through this course.
Each testing criterion is addressed individually, with explanations and examples of how to apply it. The content is divided into 20 categories with additional subsections, with knowledge checks and practice opportunities throughout. This is where the training shifts from learning about Section 508 to actually learning and applying the Trusted Tester methodology.
Trusted Tester Practice Exam
The practice exam consists of 100 multiple-choice questions. For almost every question, you're given a web page to analyze and asked to answer a specific question about it. You must score at least 85% to pass and gain access to the certification exam.
You can attempt the practice exam six times before a 24-hour waiting period is imposed between additional attempts. You then have three more attempts. If you haven't passed after nine attempts, you must retake the course and pass all of the knowledge checks again.
The practice exam took me about 3.5 hours, and I passed with an 89% on my first attempt. Some of that time, however, was spent reporting issues I encountered with the exam. More on that later.
Trusted Tester Certification Exam
The certification exam follows the same format as the practice exam: 100 multiple-choice questions, most requiring you to analyze a provided web page and apply the Trusted Tester methodology to determine the correct answer.
You must score at least 85% to pass. The retake process is similar to the practice exam, although the waiting period increases to 72 hours after the first six attempts. If you don't pass after nine attempts, you must retake the entire course. You then receive three additional attempts, with a five-day waiting period between each.
The certification exam took me about 5.5 hours, and I passed with a 91% on my first attempt. A significant chunk of that time was also spent reporting issues with the exam, which I'll get into later.
The Good
Hands-On Practice with Realistic Examples
For every Success Criterion, there were examples where the criterion Does Not Apply, examples where it Passes, and examples where it Fails, with sample web pages to explore both manually and using ANDI. (Note to test takers: Make sure you are clear on what Does Not Apply vs. Passes as I hear this is a main point of confusion for many test takers.)
From a learning design perspective, this was one of the strongest parts of the training. Rather than simply explaining each criterion and asking learners to remember the testing process, the course provided worked examples followed by opportunities to apply the process yourself. Seeing passes, failures, and situations where a criterion did not apply was particularly useful because it helped build the ability to distinguish between similar scenarios, not just memorize a testing procedure.
The practice was also well scaffolded. I could learn the steps for one criterion, immediately apply them to an example, check my understanding, and then repeat the process with the next criterion. By the time I reached the practice exam, I had already performed each type of test multiple times in a context similar to what I would encounter on the assessment.
Time to Complete
The training is entirely self-paced, and aside from the exams, you can start and stop as your schedule allows. You have 180 days from the time you begin to complete the certification process. (The training portal shut down for a couple of weeks while I was working through it, and I'm not sure whether that downtime was included in my 180 days.)
There is also a lot of room to try again. Even after the fairly generous number of exam attempts, you can retake the training and work your way back through the process. If you're determined to learn the methodology and willing to put in the practice, there are multiple opportunities to eventually earn the certification.
Cost
It's free. Completely free.
And even if you have no intention of earning the certification or taking the final exam, you can work your way through the first four courses just to learn and practice accessibility testing. That's a substantial amount of hands-on training with a standardized accessibility testing methodology available at no cost.
Real World Application
Not only are the skills learned in the course immediately applicable to testing real world digital content, but the course also provides the SCRT (Section 508 Conformance Reporting Tool) and a user guide so you're ready to start making reports.
The Bad
Browser Pop-Ups/Navigation
Each module has to be opened in a separate pop-up window to work through. Once it's finished and you select the exit button, you're returned to the main browser window, which then reloads. It's clunky. It's not great. And the navigation created even more problems when I tried to submit a ticket about an issue with the course.
When you select "Report a Technical Issue" from the main navigation, it doesn't open in a new window. Instead, it takes over the original browser window. The form then asks you to paste the webpage URL and provide details about what you're reporting... from the page it just navigated you away from.
Once I realized what was happening, I started right-clicking the link and opening the reporting form in a new tab, but this feels like a workflow that wasn't fully thought through when the interface was designed.
More importantly, inconsistent navigation and unexpected changes in context create unnecessary cognitive load for the learner. Instead of focusing on the testing process I'm trying to learn, I'm also having to remember which window contains what, how to preserve the page I need, and how to get back to where I was. That's annoying for anyone. I can't imagine navigating the same workflow using assistive technology. (I might actually go back and try some of it with a screen reader).
Incorrect Content
I had to report a lot of inconsistencies and some straight-up incorrect content. There were even knowledge check questions I had to answer incorrectly in order to move on.
In one particular example, I reported an incorrect question and received a response confirming that I was right and they were working on fixing it. Then the same question appeared on the practice exam. I answered it correctly according to the testing methodology... and got it wrong because the answer hadn't been corrected there. So I reported it again.
Then it appeared on the final exam. (Insert internal screaming here).
At that point, I deliberately answered it incorrectly because I had learned what answer the system expected. From an assessment-design perspective, that's a pretty significant problem. Instead of measuring whether I could correctly apply the Trusted Tester methodology, that question was measuring whether I could recognize and remember an error in the course. That's not the skill the assessment is supposed to measure.
This was... exhausting.
I actually LOVE this kind of analytical detail work when that's what I'm supposed to be doing. (Hey, DHS: pay me to go through this course again and help you improve everything!) But when my focus was supposed to be on learning the content and demonstrating that learning on the assessments, identifying and documenting course defects became a huge hindrance. It added time, stress, and a considerable amount of extraneous cognitive load that had nothing to do with the actual learning objectives.
Unclear Instructions
The instructions weren't always clear and, in some cases, were completely incorrect.
The knowledge checks in the first course did not follow the same format as the others, but the instructions were the same. There were warnings about needing to pass a knowledge check with 100% accuracy. In one place, I was told I could retake it. But when I went to submit it, several pop-ups warned me that once I submitted, I would NOT be able to change any of my answers.
You're also supposed to be able to select "Check" for each answer to find out whether it's correct. Sometimes that button exists. Sometimes it doesn't. Sometimes, when you answer a question incorrectly, you can try another similar question. Sometimes you can't.
Consistency matters in learning design because once learners understand how an interaction works, the interface itself should largely disappear into the background. Here, I couldn't build that expectation. I had to continually stop thinking about the content and start figuring out how the course worked.
At one point, I was stuck trying to determine whether I should submit a knowledge check with one incorrect answer when I had also been told I needed 100% to pass and might not be able to change my answers afterward.
These details may sound small individually, but they add up. They consumed time and working memory that should have been spent learning and applying the testing methodology.
The exam instructions had a similar problem. They didn't specify whether there was a time limit, whether I could pause and return later, or even approximately how much time I should plan to set aside. I still don't know whether my progress would have been saved if I had stopped, so I took the final exam in one sitting. For 5.5 hours. Not ideal.
Not Accessible
For several of the reasons I've already mentioned, this course about accessibility testing was not completely accessible.
And now I need to take this opportunity to vent about the XYZ News page.
If you've taken this version of the course, you know what I'm talking about. I still have nightmares about the TIP TIP TAP of the autoplaying typewriter sounds that occur on well over 100 different web pages throughout the course.
Allow me to explain.
In one instance, my job as the tester was to analyze autoplaying content on the page and determine whether there was a keyboard-accessible way to stop it. Great. The audio served a purpose there because it was part of the accessibility barrier I was supposed to identify and test.
The problem was that this page, or a similar version of it, was reused over and over and over again when the testing criterion had NOTHING TO DO WITH autoplaying content.
That distinction matters. An accessibility barrier intentionally included because it is relevant to the learning objective is part of an authentic practice environment. An unrelated accessibility barrier that repeatedly interferes with the learner's ability to complete an entirely different task is just a distraction.
I turned my volume down, but I couldn't turn it off completely because I couldn't risk missing some other piece of audio that might actually be relevant to what I was supposed to test.
And it gets worse.
Most of these pages used frames. That meant EVERY SINGLE TIME I opened ANDI to investigate the page, even if I had already stopped the audio, selecting the frame caused it to START AGAIN.
I actually felt like I might be losing my mind.
This became, by far, the worst part of the entire course.
After about the first 20 unnecessary encounters with the typewriter from hell, I reported it. I explained that the audio was distracting and creating an accessibility issue unrelated to the criterion being tested, and I included the question and page I was currently working on as an example.
The response asked me to provide all of the places where it happened. At this point, there may have been some maniacal laughter on my end.
Had this been a job I was being paid to do, I honestly would have loved tracking them all down. But as a learner, I couldn't afford to spend even more time going backward through the course to catalog every occurrence.
And it KEPT HAPPENING.
Every section of the course. Every practice quiz. The practice exam. The final exam.
It followed me throughout the entire certification process.
I was haunted. Every time I had to open a new web page sample, I was bracing myself for the worst. I barely made it out alive. (Okay, this statement may be a slight overdramatization).
DHS: PLEASE FIX THIS. You can prevent the pain and suffering of thousands of Trusted Testers yet to come. Wield your power for good, and fix this site, please.
Poorly Written Surveys
Every course ended with a survey. And you HAVE to complete the survey to move on.
One question asks whether you used the message boards. If you select "No," a new response box appears asking you to explain why. Great. That's appropriate conditional logic. Then the very next REQUIRED QUESTION asks you to rate how helpful the message boards were.
I repeat: REQUIRED QUESTION.
I just selected the response that I didn't use them. How am I supposed to meaningfully rate them? And I couldn't continue unless I selected an answer. I couldn't move on to the next course or exam.
So every time, I went back to the previous text box and added a note explaining that the next question shouldn't be required for someone who selected "No." The survey already demonstrated that it could use conditional logic based on a previous response, which made this particularly frustrating.
Beyond being annoying, this creates bad data. If respondents are required to evaluate something they didn't use, they're forced to provide an answer that doesn't reflect their experience. At that point, the survey isn't just frustrating for the learner; the resulting evaluation data is less meaningful for the people trying to use it.
The survey also asks whether you submitted any reports. (And oh, did I submit reports.) It then asks you to rate the response you received. You can indicate that a response was helpful but not timely, timely but not helpful, neither timely nor helpful, and so on.
I completed the fourth course on a Saturday and went straight into the practice exam. So while I had, in fact, submitted a number of requests, it didn't feel fair to rate responses I hadn't received yet because... it was Saturday night.
But I had to complete the survey to continue. And I really wanted to start the final exam.
After the final exam, I had to take the survey again. I actually typed a comment apologizing for the number of reports they were going to receive from me on Monday morning and explained that I was just selecting the highest rating and hoping they would actually look at everything I had submitted because I didn't think it was fair to give a low rating about a response I didn't receive over the weekend.
The Exams
The exams weren't all bad, but this is where the problems with incorrect content and unclear wording became much more consequential.
I felt easily 95% confident with the actual testing methodology, but I didn't feel like that knowledge was consistently translating to the exam. Instead, I sometimes felt like I was playing a guessing game: Had they corrected an inaccuracy I encountered earlier? Was I supposed to answer according to the documented testing process or according to what I had learned the system expected? Did the person who created this question notice the same small detail on the provided webpage that I noticed, or was that detail outside the intended focus of the question?
That happened several times with content I reported throughout the course.
An assessment should allow learners to demonstrate whether they have achieved the intended learning outcomes. When ambiguity or errors in the assessment itself influence the answer, it introduces construct-irrelevant difficulty: I'm being challenged by something other than the skill the assessment is intended to measure.
After the final exam, you receive only a score. There's no indication of which questions you answered correctly or incorrectly. I understand the rationale for limiting feedback on a certification exam. Protecting item security and exam integrity matters, particularly when candidates are allowed multiple attempts.
The problem is that there was also no mechanism for identifying potentially flawed questions after completing the exam. Unless I anticipated that a question might be problematic and reported it while I was still taking the exam, I lost the opportunity to identify it afterward.
So that's what I did.
Any time I encountered a question where there was ambiguity, an apparent error, or a conflict with something I had encountered earlier in the training, I moved the exam browser out of the way, right clicked on "Report a Technical Issue" (which also I don't think is the best name for this link, but in the interest of time let's move on) and submitted a report. Even questions I deliberately answered "incorrectly" because previous experience had taught me which answer the system expected got reported.
That meant repeatedly breaking concentration and interrupting a testing process that ultimately took me 5.5 hours.
I'd be more understanding of the lack of post-exam feedback if the certification exam used a clearly separate and tightly controlled item bank from the practice exam. But in my experience, many of the questions were the same questions from the practice exam.
Even during only the final exam, four different questions appeared twice on the same exam, with the same webpage to analyze and the same answer options. One question appeared three separate times. Exactly the same.
So if I understood that item correctly, great: I essentially received credit for it two or three times. But if I misunderstood it, or if the item itself contained an error, that single issue could count against my score two or three times.
From an assessment perspective, that makes the resulting score harder to interpret. A 100-question exam implies 100 opportunities to demonstrate knowledge and skill, but duplicated items mean some individual skills or questions are being weighted more heavily simply because the system happened to serve them more than once.
Given the overlap with the practice exam, the duplicated questions within the final itself, and the issues I had already encountered with individual items, I struggle to understand why candidates receive no more detailed information about their performance after the exam. I am absolutely open to other viewpoints about this, so please let me know if you have a differing opinion for me to consider.
Wrap-Up
For a process that asks the tester to have meticulous attention to detail, I think there were a lot of details missed in the creation of these courses and the certification process.
I think that's what makes it more frustrating: the course has so much potential to be excellent.
Presenting a concept, clearly defining the testing process, providing examples of Does Not Apply, Pass, and Fail using web pages created specifically for that purpose, and then following those examples with knowledge checks and practice questions that mirror the format of the final assessment is a strong instructional sequence.
The content moves from explanation to demonstration to guided practice to independent application. For training someone to consistently apply a standardized methodology across such a huge breadth of content, that structure makes sense. Unfortunately, a lot of small details were overlooked, and together they added up to a huge headache for the learner.
There are bound to be occasional issues in a course of this size. I encountered broken links, a video that repeatedly froze in the same spot, and other little glitches. Those things happen. (I did also submit requests for these). What concerned me more were the problems that appeared repeatedly across courses, practice activities, and assessments: inconsistent instructions and interactions, incorrect content, accessibility barriers, ambiguous assessment questions, and examples where important details seemed to have been overlooked.
The course became difficult not because of the complexity of the content I was learning, but because I was trying to figure out what the course expected me to do. It was like a carnival escape room where I had to keep guessing, checking, remembering previous inconsistencies, and stopping to report things. All of that created friction and cognitive load that had nothing to do with learning how to perform accessibility testing.
Do I still feel confident with the material presented? Yes. I had the Trusted Tester Conformance Process manual, enough previous accessibility testing experience to recognize when something didn't make sense, and enough opportunities to practice the methodology that I ultimately felt confident applying it. (Also the manual PDF is missing 6.B completely... and 6 only has A and B so testers might not realize 6.B event exists if using the provided PDF).
The course teaches a valuable process, and I genuinely enjoyed learning it. (I always enjoy learning a new process.) But the experience of completing the training was far more frustrating than the content itself needed to be, and for a program devoted to accessibility testing, the training itself was not as accessible as it needs to be.
I never did receive any follow-up about the probably more than 30 reports I submitted during that final weekend, but I did receive my certificate at 10:14am the following Monday morning.
Were they just happy to be rid of me? Probably. Will they take the feedback I provided and use it to improve the course? I really hope so.
My attention to detail can sometimes be seen as a problem by people who care more about completion than quality. But in accessibility testing, attention to detail is the job. It's noticing the small inconsistency, testing the thing that doesn't quite make sense, asking whether something actually works instead of assuming that it does, and documenting what you find.
So perhaps spending my Trusted Tester certification experience finding, testing, documenting, and reporting problems with the Trusted Tester certification experience was strangely appropriate.
I hope DHS takes the reports being submitted and uses them to continue improving the courses in the Trusted Tester certification program. And I sincerely hope there isn't actually just one person responsible for tackling all of these issues. If there is, apologies once again, sir, for the Monday morning when you opened your inbox and found 30+ reports from little old me. I hope you had your coffee before logging in.
Comments