Today I had the pleasure of reading an excellent article by Malcolm Gladwell over at NewYorker.com.
In it the author talks about how scholars are exploring different ways to predict how a candidate will succeed in their careers. Gladwell touches upon three different fields that have almost zero consistent predictability methods, including teaching, drafting college quarterbacks to the pros, and investment banking.
The problem with predicting whether or not a teacher will be a great teacher is the fact that there are no accurate predictors in place. According to a group of researchers cited in the article, test scores, graduate degrees, and certifications have turned out to be useless in ascertaining whether someone is capable of teaching.
Some school districts have taken matters into their own hands and have developed focus groups to evaluate competencies related to student-teacher interaction. A team of researchers from University of Virginia’s Curry School of Education watch tape after tape of teachers interacting with preschool students, stopping to chart and grade the teacher’s moves. Does he/she maintain control of the class while allowing free thought and movement? Does he/she use let each child express their own answers, whether it be right or wrong, without immediate correction?
The consequences for allowing a below-average teacher to lead a classroom are dire. The article cites recent research from Eric Hanushek, an economist at Stanford, who estimates that the students of a very bad teacher will learn, on average, half a year’s worth of material in one school year. The students in the class of a very good teacher will learn a year and a half’s worth of material. Think of the lag the former student will have when trying to catch up to his peers, who might be seated in a room right next door but had the good luck to get paired with an above-average instructor.
Good recruiters, specialized or not, are tasked with these same situations on a daily basis. They engage in lengthy behavioral-based interviewing, distribute and grade assessments, and pit candidates up against hypotheticals over and over. They perform second, third, and sometimes fourth interviews, analyze body movements and handwriting, and dig out any faults that might impede work. They shell out money, time, and a significant amount of sweat trying to predict on-the-job success.
But are any of these tests a true measure for how the candidate will perform? Can questions or assessments predict how a new employee will react to an extremely competitive co-worker or an especially needy client?
What kinds of testing methods or success predictors have you enjoyed success with, and what kinds do you deem failures? In what instances are assessments still a good indicator of performance?
Popularity: 6% [?]
December 11th, 2008 at 11:43 am
This is an area I’ve spent a great deal of my time trying to better understand. First and foremost I feel that many organizations find failure with predicitive indicators, because in order to accurately predict success you have to accurately measure success; and far to many organizations do a poor job of measuring and differentiating success of employees. This is particularly true when you are measuring and defining non-revenue or production generating positions.
I’ve always found that the best (not perfect, that word should never be used) method to follow is to define all of the competencies required to perform a role. Then measure the top 20% against the bottom 20% of performers to see how they compare and where the differences lie. Looking at the bottom 20% against the top twenty is key, by the way, you must be able to differentiate.
When you design a strong interview model around the competencies, I use a behavioral model myself – with a 5 point scoring model, you then have to measure their performance, not just overall, but against how they scored in the interview. 5 points was chosen as the interview points, since that matches with our ratings for employees. You then need to analyze annually, and make adjustments.
It is a lot of work to get this in place and functioning properly, it is not a quick fix, but I’ve always found it produces very storng returns.
December 11th, 2008 at 2:24 pm
Years ago I took a psych profile before I was “hired” to work with IDS Financial Services (American Express Financial Advisors/Ameriprise). I passed. I did not do well. I was miserable working for a company with which my ethical standards were in conflict with company policies. I loved my clients. Management seemed to be in conflict with the espoused values. While in many ways it was a great company, I did not fit in. So, I left. The screening efforts did not work in my case.
As an employer I am constantly looking at why certain employees did not work out. After 20 years I’ve gotten better. But, I’m far from perfect.
December 12th, 2008 at 12:54 am
I’ve worked as a consultant for an architectural firm that looks for recent architectural and engineering grads that will fit their corporate values. We’ve created 10 to 15 different scenarios that pose problems to be resolved. And different interviewers present different scenarios to a candidate. We listen to their response, not for the resolution they come up with. Instead, we want to understand how they think about problems. Can they suggest potential data that should be considered, give a sense of how well they interpret data, and how they come up with conclusions. We also set up scenarios that will show us how their training has taught them to think about architecture and clients. We are looking for recruits who pay attention to clients, and show that client values are truly fundamental. We’ve found that we can weed out the design impresarios and identify people who will put clients first, be able to collaborate with them, and at the same time offer significant value to the client. We’ve used the model for three years and are quite happy with our success. It does take some planning to create the scenarios, and we find ourselves readjusting them after a set of interviews. I also know that the process can be used in any work setting. I’m quite convinced that how a person thinks about their business is the most important attribute–not their technological training. Most companies do well at technological training, but piss-poor at people skills.
June 29th, 2009 at 4:36 pm
Dan-
That is a very interesting concept. I have always felt that work-based scenarios/problems serve as a much better indicator of potential performance than the typical cover letter/resume/interview process. In fact, I have been stunned to learn how little stock people in HR often put into the interview- and perhaps with good reason- http://www.fastcompany.com/magazine/136/made-to-stick-hold-the-interview.html.
I work with a company (www.GrouperEye.com) that wants to move college recruiting along these same lines. We run case competitions and increase the interaction between college students and the companies who recruit them. While it is not a new concept (consulting companies have used case-interview process for years, Google runs specific case competitions, etc.), nobody has taken the time to try to apply this broadly to college recruiting. We have used our own system and it works. Companies get better fits in their hires, and the kids who might get weeded out early on because of that C- in Spanish freshman year have a chance to prove to companies that they are the one for the job.