Test Windows vs. On Demand Testing

There are two common strategies used to manage a testing schedule for credentialing. Depending on your program’s philosophy about access and availability, and policies for managing your exam item bank, one of the two strategies will likely fit you better than the other. These strategies are pre-defined “Test Windows” and “On Demand Testing”. I thought I would post a brief discussion of each approach and the potential advantages of each one. Both of these strategies describe calendar availability of exams, not eligibility. Exam eligibility management policies may be employed with either strategy.

Test windows are defined calendar periods where a credential sponsor offers exam scheduling. For example, a candidate might be allowed to sit for their exam within the following time periods:

• · Jan 1 – Feb 28
• · May 1 – June 30
• · Sept 1 – October 31

In this case, the three test windows are the only times when a candidate can take the exam at the test service provider.

In on demand testing, a credential sponsor allows a candidate to sit for their exam on any day of the year (per seat availability at the test service provider). On demand testing has no strict time periods where exams may or may not be scheduled.

There are certain advantages to test windows. First, it provides you and your candidates a defined calendar in which to plan and execute the events surrounding testing. Many people work better with deadlines. Test windows, by their nature, provide a series of deadlines for everyone to strive for. A published calendar for everyone to work towards tends to reduce confusion and provide motivation for everyone to complete by the deadline.

Second, it gives your staff well defined periods to expect peak demand for customer service from your candidate population. With test windows you can coordinate you yearly activities for more efficiency. For example, during the testing windows, you can focus your attention on successful exam delivery. Outside the testing windows you team can focus on other program related tasks like CE processing or new applications. These cycles will make your staff more efficient allowing them to seamlessly move from one focused activity to the next.

Finally, a unique benefit of testing windows is to support ongoing item statistics and analysis. With testing windows you can normalize the exam performance, establish the exam cut score for each window, and generally maintain your item bank. If you choose to use testing windows in this way, the test service provider (TSP) holds the results until all tests in the window have been delivered. Then, the TSP or scoring agent runs the item psychometrics on all the results in the test window to eliminate poor items and establish the passing score for the exam. Given a large enough examinee population, this strategy offers you the opportunity to prove and vet the skill set of the entire “class” of examinees. Unfortunately the downside of this approach is timeliness and cost. Candidate’s don’t walk out of the exam room with a final score and the additional steps add time and cost to the scoring process.

I provided the above advantages based on our experiences with clients. I am certain there are others, especially in terms of the relationship with a test service provider. For example, it may be cheaper to contract with a test service provider for test windows than for on demand testing, if the TSP has a better forecast to reserve seats. Can anyone provide more detail regarding test window vs. on demand pricing or around additional advantages?

Next posting – On Demand Testing advantages

Detailed Relevancy

Continuing the “start over” discussion, this post explores issues to consider when making candidates start over.

The first issue is candidate frustration about losing activities when they are forced to start over. If you have a timeline to get certified, your program most likely requires that all activity must be recompleted if a candidate starts over. Being forced to recomplete requirements is frustrating for the candidate, especially for those activities they have recently completed.

For example, suppose the candidate must pass three exams to get certified and there is a three-year timeline from their initial application in which to get certified. Assume they pass the first exam in year one and the second exam in year three. When they don’t pass the third exam by the end of third year, they must start over. The candidate will be especially frustrated about the now-invalid second exam that they passed within the last year.

Additionally, you may have candidates in this scenario who are incented to not take that second exam in year three. This disincentive stems from the low probability that they will pass both exams in that last year. And, if they don’t pass both, they will lose credit for the one they do pass. Of course, this is a barrier to program revenue that must be avoided.

If the example above is applicable for your program, a good option for dealing with it is to have validity periods for each activity instead of forcing recompletion of all activities when starting over. If you add an expiration period of three years to each exam, passed exams can remain valid for a period of time into that second attempt at the certification. In the above example, their passed second exam will be available for another two years even though they have been forced to start over. The candidate still must prove relevancy in regards to all the requirements (must be completed recently), but each requirement is made invalid per its own timeline.

A second issue to consider when candidates are required to start over is whether certain requirements require a different approval path when they are recompleted. For example, if the candidate must recomplete an application, perhaps it is not necessary to repeat the full manual review of all parts of the application. Instead, you might streamline the application so it includes only those necessary items to recomplete. You could require that the candidate needs to pay the fee, sign the affidavit, and answer ethics questions, however the candidate may not need to resubmit all education and work experience. As in the first discussion point above, this approach delineates relevancy at a more granular level. In this example, work experience was an initial bar that needed to be met, but additional work experience makes no difference when they reapply. The affidavit and ethics responses, however, need to be updated and therefore should be recompleted.

Implementing this strategy would save the candidate significant time when recompleting the application. This approach removes a barrier to the candidate starting over and makes it more likely that you will receive the revenue from the reapplication. In addition, you should see an internal cost savings as your staff won’t spend as much time reviewing and approving reapplications.

Next posting – Test windows vs. on demand testing

Start Over Options

This post looks at the considerations when you choose to force a candidate to start over. There are two main areas to think about – how long before the candidate has to start over (allowable in process time) and whether their previously completed requirements must be recompleted.

The allowable in process time should be linked to either the recertification time or to an analysis of your candidates’ certification timeline. The first option harkens to my previous post – the reason for making the candidate start over is to ensure their qualifications at time of certification is relevant. Relevancy over time was already addressed as part of recertification policy so it makes sense to correlate the allowable in process time to that has well.

The second option ties the start over option to an understanding of your candidates’ motivation. Essentially, you want to find the length of time after which an impending forced start over will motivate your candidate to complete their certification.

In either case, we’ve generally seen the start over period in the 3-5 year range.

The relevancy issue is the foremost factor in deciding how to deal with previously completed requirements. Is an aged requirement still valid? If the candidate passed an exam 4 years ago, have they retained competency of the material? What about a document where they have outlined their relevant work experience? Perhaps the document is ok because they merely needed to prove a minimum level of experience in the field (and this job experience hasn’t grown weaker). However, perhaps the candidate needs to reprove that they have the knowledge and pass the exam again. A final item to review is if the program requires an affidavit where the candidate has attested to statements around their criminal background, adherence to program ethics, etc. Most likely the candidate should be required to recomplete this requirement.

Candidate satisfaction is another consideration. Candidates will feel differently about having to recomplete an exam that requires payment vs. merely submitting updated documentation. This will impact whether they are likely to start over.

Next posting – Some unique considerations in the start over process

Forcing Candidates to Complete

The next series of posts focuses on the issue of candidates who pursue their certification for an inordinately long time. These candidates start the certification process and, unlike the disengaged candidates from previous posts, they stay engaged. However, they take a long time to get certified. Or, they stay just engaged enough to stay active but never put in the effort to finish the certification. The first topic I’ll explore is the question – should the candidate be forced to start over from the beginning?

The primary reason that most programs allow the candidate an infinite time to complete the requirements is a desire to keep the candidate happy. If a candidate starts the certification process and then goes a length of time with no further activity, it doesn’t really matter if you make them start over because they have already disengaged. For example, if they fail the exam and you see nothing else from them for 3 years, most likely they are no longer pursuing your certification so nothing you do to their record is of much importance. However, if they are remaining active (continuing to take exams for example), this type of policy can frustrate candidates and lead to complaints to your customer service team. This is why many programs do not have a deadline for a candidate to complete their certification.

In addition to customer satisfaction, as long as a candidate is actively engaged, most likely they are continuing to generate revenue for you. If you make them start over, it is possible that the candidate will disengage in frustration leading to the loss of that revenue.

Finally, usually the activities at the beginning of the certification cycle are most impactful in terms of your staff’s time. If a candidate is merely continuing to take exams, they probably are minimally impacting your customer service team. However, if the candidate starts over, they might need to recomplete an application and resubmit documentation – activities which generally require a good deal of review time from your staff.

In contrast, the main business driver for forcing a candidate to start over is to ensure that their experience is relevant at time of certification. If you have an application process where the candidate provided industry experience or background information, then a candidate who takes a long time to get certified might no longer be one who should get certified based on what has transpired with them since time of application approval. For example, many programs have applications where the candidate completes a personal affidavit where perhaps they attest that they haven’t been charged with a felony. If the candidate takes six years to attain their certification, is this affidavit still valid? Or, if your program requires two exams for certification and the candidates passes them five years apart, have they truly demonstrated mastery of material at the same time? In either case, you increase the chances of certifying a candidate who does not truly meet the requirements for certification. A related example to think about is if your program requires recertification. If your program requires a candidate has to demonstrate proficiency every 4 years to stay certified, does it make sense that a candidate would have longer than those 4 years in which to demonstrate initial competency?

Two supporting reasons for enforcing a start over policy are:

1) Some candidates need a push – its human nature to procrastinate and some people just need a deadline. They want the certification and you want both that revenue and the additional person in your certification pool. The start over policy just gives the candidate a target to aim at.

2) It’s helpful for your program monitoring to know your pool of active candidates that are still pursuing certification and who might respond to marketing. Ones who are not making progress toward certification could be moved to a “Start Over” status.

Next posting – Options for the start over rule.

Keeping Certified Candidates Engaged

Continuing the candidate engagement thread, its time for a discussion on how to keep certified candidates engaged. Again, as with candidates pursuing their first certification, the trick here is to motivate candidates to stay connected. A great method for this is to organize a set of tiered certifications that progressively challenges your candidates’ expertise—as a candidate’s knowledge and experience grows, so does the challenge that your program offers. Having an entry-level certification, a senior certification, and an advanced certification provides a certification path that mirrors a candidate’s career path.

Another option is to have ongoing requirements that force a candidate to stay engage. Recertification is the primary means to do this. Many clients require candidates to submit continuing education (CE) in order to maintain certification. In these cases, its important that you have a site that the candidate can report their CE immediately upon completion rather than forcing them to wait until the end when they have completed all of the CE credits and are about to recertify. This way, they have periodic engagement with you as they login and record their progress rather than the single submission at the end of the recertification cycle.

A final option to consider is engaging your certified candidates as judges of who is qualified for certification. There are a couple of strategies you can use to engage your candidate population this way—as judges or as mentors. As judges: consider introducing a practical requirement to you certification path. These work products must be evaluated and graded to ensure they meet the rubric of your program. Senior certified professionals are your subject matter experts and take pride in vetting the next generation of colleagues—many are enthusiastic to volunteer their time to this effort. As mentors: in many trades and professions, mentorship is integral to maintaining professional standards. In some, the relationship is valued highly enough to formalize in the certification and licensure process. The mentor would ensure young candidates are performing the right tasks in the industry while internalizing and practicing the behaviors that are essential to success, thus incubating a truly professional society. As Judge or Mentor, your candidates take on a material role in defining the tenor and quality of your future professionals and you build a culture of connected and engaged candidates.

What do you think?

Keeping Candidates Engaged

As discussed in my last post, the best method for engaging disconnected candidates is never to let them disengage. We’ve seen a variety of methods for this, however the basic approach is to keep your program top of mind for the candidate. If they are thinking about their certification often, they are likely to interact with you, your website, and your program’s initiatives on a regular basis. For this post, I’ll focus on engagement for candidates who are in process of getting their certification.

The easiest method to keep in-process candidates engaged is to bound the time they have to get certified. If their pursuit of certification is open ended, it’s more likely that candidates will drift through the certification process without actively pursuing the requirements. We have clients where the candidates have a timeframe, 3 years, to earn their certification from the time they started the process by completing the application. If the timeframe elapses without the candidate certifying, the candidate must start over. Usually this means they have to recomplete the application and pay the application fee. A simple deadline is often enough to keep a candidate focused and on task toward completion. The energy and expense associated with starting over is a motivator for candidates to finish their certification in a timely manner. This timeframe has the additional benefit of ensuring that the process you designed and approved for the candidate is the one they actually experience and is still relevant. (I’ll discuss this relevancy issue more in a later post.)

Another option to motivate candidates is to have visual, incremental steps in your program. If the candidate sees that they are making measured progress toward certification, its probable that they feel successful and will continue to pursue your certification. You can use different statuses for each step in the process to give the candidate a visual demonstration of progress. After each step, let them know how far they have come and how little they have left to do.

A final tool is to find candidates who are in danger of disengaging and market to them quickly. These candidates are at risk of leaving your program. If they take this step, they are unlikely to come back. As such, incentives and discounts may make sense as reduced revenue earned from these candidates may be better than the nothing you’ll get from them if they disconnect. However, finding these candidates takes some sophisticated reporting. Adding the incremental statuses as mentioned above can help with this as it provides greater insight into the progression of your candidate population. You’ll need to do some trend analysis to determine the inflection points where candidates are unlikely to continue if their progress stalls. We’ve had some fun investigating this with some of our clients and its been exciting to see analysis results that generate actionable data to drive a marketing project.

I welcome your thoughts.

Next posting – Keeping certified candidates engaged.

Engaging Disconnected Candidates

A hot topic at our annual user conference last week was how to engage candidates who haven’t connected with you recently. For example, this might be a person who got a certification a few years ago and hasn’t had any contact with you since that time. From time to time, you want to connect to your candidate’s for various marketing campaigns or operational initiatives. These “disconnected candidates” are at risk of missing the message because their contact information may be out of date. Here are a few ideas on how best to reengage with candidates in this situation:

1) Direct contact via email or physical letter – this is the easiest but the success rate decreases with the time since last contact with the candidate.

2) General marketing with incentives – candidates may be interacting with your organization and merely not connecting. For example, they might be visiting your main website for information but just not accessing their certification record. A general marketing message on your website that offered an incentive for the candidate to connect should increase your success rate with cases such as these.

3) Seek help – professional marketing people either within your organization or at an outside firm spend a great deal of energy trying to solve this problem of reaching unconnected people and getting them to reengage. Use their expertise.

4) Validate that its important – contact for operational an operational reason may be less important than for specific marketing and communications issues. For example, getting notice to disconnected candidates that you have transitioned their certification record to a new system probably isn’t as critical as marketing to them to come take an updated exam. The risk in the former case is that they may stay disconnected and have customer service issues if they ever resurface. This pales in comparison to the lost revenue opportunity in the latter.

5) Never let them disconnect – Of course, this is the best answer but deserves its own discussion.

Has anyone had experience with any of the above? What options have worked for you?

Next posting – How to keep candidates connected and engaged.