Test Windows vs. On Demand Testing

There are two common strategies used to manage a testing schedule for credentialing. Depending on your program’s philosophy about access and availability, and policies for managing your exam item bank, one of the two strategies will likely fit you better than the other. These strategies are pre-defined “Test Windows” and “On Demand Testing”. I thought I would post a brief discussion of each approach and the potential advantages of each one. Both of these strategies describe calendar availability of exams, not eligibility. Exam eligibility management policies may be employed with either strategy.

Test windows are defined calendar periods where a credential sponsor offers exam scheduling. For example, a candidate might be allowed to sit for their exam within the following time periods:

• · Jan 1 – Feb 28
• · May 1 – June 30
• · Sept 1 – October 31

In this case, the three test windows are the only times when a candidate can take the exam at the test service provider.

In on demand testing, a credential sponsor allows a candidate to sit for their exam on any day of the year (per seat availability at the test service provider). On demand testing has no strict time periods where exams may or may not be scheduled.

There are certain advantages to test windows. First, it provides you and your candidates a defined calendar in which to plan and execute the events surrounding testing. Many people work better with deadlines. Test windows, by their nature, provide a series of deadlines for everyone to strive for. A published calendar for everyone to work towards tends to reduce confusion and provide motivation for everyone to complete by the deadline.

Second, it gives your staff well defined periods to expect peak demand for customer service from your candidate population. With test windows you can coordinate you yearly activities for more efficiency. For example, during the testing windows, you can focus your attention on successful exam delivery. Outside the testing windows you team can focus on other program related tasks like CE processing or new applications. These cycles will make your staff more efficient allowing them to seamlessly move from one focused activity to the next.

Finally, a unique benefit of testing windows is to support ongoing item statistics and analysis. With testing windows you can normalize the exam performance, establish the exam cut score for each window, and generally maintain your item bank. If you choose to use testing windows in this way, the test service provider (TSP) holds the results until all tests in the window have been delivered. Then, the TSP or scoring agent runs the item psychometrics on all the results in the test window to eliminate poor items and establish the passing score for the exam. Given a large enough examinee population, this strategy offers you the opportunity to prove and vet the skill set of the entire “class” of examinees. Unfortunately the downside of this approach is timeliness and cost. Candidate’s don’t walk out of the exam room with a final score and the additional steps add time and cost to the scoring process.

I provided the above advantages based on our experiences with clients. I am certain there are others, especially in terms of the relationship with a test service provider. For example, it may be cheaper to contract with a test service provider for test windows than for on demand testing, if the TSP has a better forecast to reserve seats. Can anyone provide more detail regarding test window vs. on demand pricing or around additional advantages?

Next posting – On Demand Testing advantages

Detailed Relevancy

Continuing the “start over” discussion, this post explores issues to consider when making candidates start over.

The first issue is candidate frustration about losing activities when they are forced to start over. If you have a timeline to get certified, your program most likely requires that all activity must be recompleted if a candidate starts over. Being forced to recomplete requirements is frustrating for the candidate, especially for those activities they have recently completed.

For example, suppose the candidate must pass three exams to get certified and there is a three-year timeline from their initial application in which to get certified. Assume they pass the first exam in year one and the second exam in year three. When they don’t pass the third exam by the end of third year, they must start over. The candidate will be especially frustrated about the now-invalid second exam that they passed within the last year.

Additionally, you may have candidates in this scenario who are incented to not take that second exam in year three. This disincentive stems from the low probability that they will pass both exams in that last year. And, if they don’t pass both, they will lose credit for the one they do pass. Of course, this is a barrier to program revenue that must be avoided.

If the example above is applicable for your program, a good option for dealing with it is to have validity periods for each activity instead of forcing recompletion of all activities when starting over. If you add an expiration period of three years to each exam, passed exams can remain valid for a period of time into that second attempt at the certification. In the above example, their passed second exam will be available for another two years even though they have been forced to start over. The candidate still must prove relevancy in regards to all the requirements (must be completed recently), but each requirement is made invalid per its own timeline.

A second issue to consider when candidates are required to start over is whether certain requirements require a different approval path when they are recompleted. For example, if the candidate must recomplete an application, perhaps it is not necessary to repeat the full manual review of all parts of the application. Instead, you might streamline the application so it includes only those necessary items to recomplete. You could require that the candidate needs to pay the fee, sign the affidavit, and answer ethics questions, however the candidate may not need to resubmit all education and work experience. As in the first discussion point above, this approach delineates relevancy at a more granular level. In this example, work experience was an initial bar that needed to be met, but additional work experience makes no difference when they reapply. The affidavit and ethics responses, however, need to be updated and therefore should be recompleted.

Implementing this strategy would save the candidate significant time when recompleting the application. This approach removes a barrier to the candidate starting over and makes it more likely that you will receive the revenue from the reapplication. In addition, you should see an internal cost savings as your staff won’t spend as much time reviewing and approving reapplications.

Next posting – Test windows vs. on demand testing