(not fraud related, but…) How to identify top performers?

It’s the end of the year which is typically the time when many companies go through torturous annual “performance evaluations” process for their employees – perhaps that’s why there seems to be a renewed interest in this subject by media and professional forums. Indeed, from NPR reviewing much-hated ‘classic’ “rank and yank” approach to re-kindled debate about suddenly popped up old presentation covering unorthodox HR methods at Netflix  – it seems the methodology of how we assess employees’ performance is anything but a closed subject.

Of course, on surface any approach would argue that it is about creating great teams by selecting the “top performers” – just like the ‘father’ of forced ranking ex-CEO of GE Jack Welch explains in the recent WSJ article or Patty McCord from Netflix who talks about the importance of having “stunning colleagues” who inspire each other to deliver the best performance. This is all true, of course, but there’s one critical question to answer here – how to single out the top performers in an objective and consistent way across the entire organization which is typically huge (at small companies everything is kind of obvious anyways). The widely adopted approach here is tracking multiple types of behavior – such as “seven aspects of Netflix culture” (the number can vary depending on the company). But that doesn’t address the core issue – how to distinguish the true performer from a careerist who is more preoccupied with making a good impression than actually contributing to the bottom line?

The truth is – no matter how good of a manager you are – at the end of the day the employees with their boots on the ground working with each other on a daily basis – they know the best who is who. You just need to find a way of getting it out from them. As simple as it sounds 🙂

But how? Myself, I have been dealing with this issue for years now – having worked with dozens of peers, bosses and subordinates and going through the whole spectrum of experiences – from horror stories to collaboration made in heaven. In retrospect, while most of people have varying ‘hard’ and ‘soft’ skills which are more or less important in a particular context, at  the end of the day there was one thing which really counted. That something could be best described as an answer to a single question – “would I hire him/her to my own startup“?

…which prompts another question – can we have a single measurement which would assess an employee’s value to the company? Could we for example do the following:

At the end of each cycle we generate the list of, say, up to 20 people each individual worked with the most throughout that period. It is important to adopt a consistent approach of generating the list – and under no circumstances leave it to the employee to cherry-pick it. One way of doing it could be via analyzing the email traffic, meeting invites etc. Alternatively, the manager could come out with the list for each engineer in his/her team (naturally, the list will have to include the manager and all the direct reports by default). The point here – to come out with the full list of colleagues the given employee had interacted with (or “got exposed to”) the most. This is a separate question not really tied to the gist of the proposal, which is coming next.

Each person on the list then asked to anonymously answer a single question: “On a scale of 1 to 10, how likely are you to hire the person X into your own company?” – with 1 meaning “under no circumstance” and 10 meaning “would absolutely hire”.

Naturally, the results will be skewed towards positive (we usually do not like to throw our colleagues under the bus), but that could be taken into account – i.e. consider the answers 1 through 6 as generally negative, 7-8 as neutral, and only 9-10 as positive.

If the above looks a lot like “net promoter score” (NPS) – the similarity is actually intentional. Indeed, NPS – after a lot of wranglings about how to measure the success of a product or a company – ended up as the single point measurement increasingly embraced by various industries (the key – only! – question there is “On a scale of zero to 10, how likely are you to refer to a friend [a product or service of a given company]?”). Naturally, it’s not perfect – as any attempt to jam a complex assessment of business (or a person for that matter) into a single digit is by definition just an approximation – but it comes pretty close to what  companies want to track: market data shows a strong correlation between the NPS and financial success of a company – unlike any other single measurement out there.

My hunch (do not claim of having any real-life data) is that by applying this methodology you’d end up with pretty good understanding of who is the top performer in your organization – those would consistently fall into the 9s and 10s – while others would be more in the middle. If not the only metric, this could be at the very least used as a strong signal “from the bottom” which is better be listened to.

Online Identity services – an emerging new business model?

Every time I visit one of financial institutions’ websites I happen to be client of, I am daunted by the hops I need to go through (neither of which is really unstoppable from the fraudsters standpoint) to login to my account. It’s obvious that serious businesses are trying to counter account takeovers and each is doing that in its own way – possibly spending lots of money on something which is not its core expertise. Countered by fraudsters for whom it actually is the core expertise, these businesses seem to be doomed to continue investing lots of resources on online identity management with only a modest success.

Needless to say, the online identity is becoming a big issue. Little wonder – whole chunks of our daily life – including very personal fields like romance and friendship – is being absorbed by the Net. In all the mess one thing stands out – acute need of a better identification. A need which itself may warrant a separate industry – call it online identity services. I do not mean anything ominous (“I’ve got lightly used identity of Judd Law! Anybody?”) – just satisfying a legitimate need of identifying people online – like an online bank needing to make sure person logging into their website is the actual account holder. Today it’s moving from traditional password-based identification (see my earlier post) to more sophisticated multi-layered mechanisms (some less efficient than others) – pictures, personal questions, 2FA tools etc. It is becoming more costly to develop and maintain, hence it would make a lot of sense to delegate this headache to a company which actually specializes in online identification. In that case the bank just needs to redirect the login to the company’s page (for non-technical user that could be quite seamless, e.g. by putting the bank’s logo to the site it redirects to or do it in a iframe), let it do all the dirty work, and return the user to the bank’s page with full guarantee (covered by the third party) that the user is authenticated. Just like PayPal handles all the payment and gets back to the merchant with guaranteed payment, the ‘identity merchant’ would come back with ‘successful login’. Now, the ‘services’ may charge per login or per month or per user – details will depend on particular business model. Such services may even offer multiple types of support – the spectrum would include periodic user screening (e.g. verifying the phone), sending 2FA tokens, sending SMS-es, in short focus on linking the physical identity with cyber one.

Now, I am not saying this has never occurred to anybody else – the Open ID concept is similar one. Too bad it didn’t really take off. My take is – people who care about this most (online banks, for example) are inherently distrustful to anything free or open source. And that serious identity management needs serious resources – to screen, to support 2FA tokens etc. Microsoft passport probably was ahead of its time. PayPal could use its clout to add “identity management” to its portfolio, or better yet Facebook could do that (the model of your identity being vetted by your friends is quite powerful), too. However, either of these companies have their hands in many jars, and the last thing a bank wants is to divulge its user base to some 3rd party who can turn out to be a competitor. My take is – in order to succeed, these services should be very specific – commercial, stand-alone, not engaged in any other type of business, but solely focused on online identity and committed by binding agreements to not to use the information for any other purposes. Naturally, there needs to be safeguards that each client’s (bank’s) user data is secure and stays its property even if login is supported by the third-party.

Perhaps there are such companies, I admit I didn’t do much research here, but even if there are – it’s anything but a mature industry. I wonder if it will ever become one.

Applying “Google spellchecker” principle in detecting online fraud

One of the ways bad guys manage to penetrate/influence a web site’s functionality – is “poking around” by hitting different pages – often on different geolocations (e.g. instead of XYZ.com – country specific sites XYZ.de, XYZ.ca etc.) – coupled with “playing” with input parameters – thus looking for input validation breaches or other site inconsistencies. If successful, bad guys can do a lot of harm – including manipulation of data (e.g. changing a user’s state by following some quixotic page sequence), stealing information and so on.

Such breaches could be successfully detected in early stages by using a technique I call “google’s spellchecker” approach. Anybody who used google to check the spelling of a word – or the right collocation/phrase – knows the underlying principle. It’s (paraphrasing eBay’s motto) “people are basically educated”. That is – if we have 5 million hits for one spelling and 5 thousand for the “competitor” spelling – then the former is the correct one. (BTW, that is one of the basic principles of linguistics: if enough people say ‘nucelar’ – it automatically becomes a legitimate word).

The way the same principle would work in detecting bad behavior is similar:

  1. assign each page a unique ID (normal practice)
  2. define boundaries of individual user sessions
  3. record the sequence of pages hit during individual sessions – e.g. 23 (login),887 (account setting landing page), 368 (account setting confirmation), 99 (logout); in other words create a “page trail” of each session
  4. record and at the end of each session increment the number of times a particular trail appeared on the radar – e.g. 23,887,368,99 -> 1035 times;

Leave the system to bake for some time. Assuming that most people use the site for legitimate purposes, the numbers eventually will reflect the “normal” usage of the site. Maintaining that information would help in detecting abnormal usage of the site (e.g. jumping to 368 “account setting confirmation” without hitting 887 “account setting landing page”) very soon after the “probe” is done. It is important to detect this early, as – if the hole becomes widely abused, its sequence may approach the “normality” level. We also should have some safeguards/mechanism to avoid false positives – e.g. if a new page is added to the site, we want to know about it (e.g. have page age information) and treat it as an exception.

Naturally, the approach is not bullet proof (hardly any one is). Indeed, if fraudsters are sophisticated enough – they could mask their behavior by mimicking legitimate sequence, or trying to make session tracking more difficult. Nevertheless that would be a serious complication of their lives – or another “bump” on their way – so the goal of slowing them down would be fully achieved.