A story came out recently about super-sophisticated self-driving cars being easily duped by relatively simple tricks used by some hackers. On the surface it seems to be shocking, as the brightest minds of top engineering companies have been working hard on making the promise of self-driving cars a reality – and triggering a true revolution in our day-to-day lives. In fact, this is hardly surprising. The algorithms, and the signals they were relying on, were probably never trained to resist an active sabotage. They were merely trying to replace human beings in routine activities, just like they do in other areas such as language translation or image recognition. In ‘non-adversary’ circumstances, the performance of the algorithm can be steadily improved over time. Once you achieved a certain reasonable threshold (e.g. detecting objects in pictures), you are not going to slip back into not recognizing them even if you stop adding more features or bigger training data sets.
With fraud you are dealing with a different animal – the patterns you are trying to detect are actively trying to hide from you. Your successful detection yesterday doesn’t guarantee the same performance tomorrow. As famous security expert Bruce Schneier once noted “Attacks never get worse, they only ever get better.” And they do evolve, change, adapt and advance in quite unexpected ways.
Does this mean machine learning is ultimately powerless against the human creativity directed against it? Of course not. It is is being successfully used to detect online fraud in top-tier financial and business institutions, some with spectacular results. Not to mention select human-vs-machine clashes, such as games of chess or Jeopardy!, where ML algorithms actually proved to beat the best in kind human experts. However, to achieve consistent results in practice, one should keep the following in mind:
- Continuous learning that relies on fresh data is imperative. You are essentially teaching the algorithm to detect a constantly moving pattern. The models will easily degrade over time if they stay intact.
- Consistent investment into ever-more sophisticated features is also non-negotiable. Throwing more of the same data (going back into history) is not going to help much. Squeezing more juice from the same data has its natural limits, too. The world constantly evolves and so should your features (in the self-driving car hacking example, the ‘feature’ itself was actually compromised)
- Typically, no one solution would suffice to cover the entire (again, constantly evolving) fraud landscape – thus proper investment is necessary into “plumbing” which would enable complex execution plans such as multi-tier decisioning, running in parallel, and applying different modeling techniques.
Back to the self-driving cars. Making them robust in the face of attempted sabotage will prove to be much more costly and complex exercise, but it is nevertheless needed to make them compete with human-driven quality (even while the latter itself is getting more vulnerable). Realizing the differences in ‘classic’ machine learning practices vs. those aimed to fight active fraud/sabotage is going to help along the long road ahead…