Autonomous vehicles are a combination of highly integrated Artificial Intelligence (AI), working with the Internet of Things (IoT) against a backdrop of regulation dictating exactly how far this autonomy will be allowed to go, and what will be released onto the road.
But what does this mean for how we develop and deliver software and applications surrounding autonomous vehicles? Testing will need to be able to adapt not only due to the highly connected nature of autonomous vehicles, but also to accommodate the behavioural differences associated with cars developed for continuous learning.
In testing the autonomous vehicle development experience, AI and analytics will become critical. Simply put, companies cannot get away without applying AI-driven testing rules to AI-driven cars. Humans and machine learning will have to come together to test autonomous vehicles, and the idea of crash test dummy with an AI brain may soon become a very necessary reality.
Whereas traditional testing processes are designed to exercise deterministic code pathways to identify errors, ‘next generation' AI testing for autonomous vehicles will have to move beyond exact circumstances to test a combination of non-deterministic simulated and real test scenarios. Testing will need to be able to adapt and behave as humans do both to tick the regulatory boxes and to ensure delivery of the best possible user experience that users of autonomous vehicles will expect. It's not going to be about “does it work as specified?” – it will be about “will it delight customers with a safe, high quality, performance and usable ‘digital experience'?”
AI & true test automation
There's a lot of talk currently about test automation. However, in reality, we've only automated one key element: test execution. AI and analytics will be the catalysts to deliver true test automation that learns continuously, enabling it to predict business impacts that help dev teams to fix issues before they occur.
Here are three ways AI is enabling this kind of predictive testing in autonomous vehicles:
1. Intelligent automation - The only way to realistically test a digital app is through an intelligent automation engine accessing the application as a user would. This is where the idea of a crash test dummy with an AI brain really comes into play, because it means taking control of a machine, and actually using the app to exercise workflows while collecting intelligent analytics along the way. This involves technology to understand on-screen images and text, such as smart image search and dynamic neural networks (also known as “deep learning”).
2. Intelligent test coverage generation and ‘bug-hunting' - There are a potentially infinite number of paths through a complex app, so which ones should we follow in our automation? We can use AI classification algorithms such as Bayesian networks to select paths and 'bug hunt'. As these paths are explored, the bug-hunting AI algorithm continues to learn from correlations in data to refine the coverage and help developers identify root causes and fix defects.
Through a combination of bug hunting and coverage algorithms, AI and analytics will exponentially increase coverage and productivity. AI algorithms will hunt for defects in applications based on user journeys automatically generated from this bug-hunting model, while coverage algorithms will select the user journey that is the furthest away from others that have already been executed.
Non-instance-based learning algorithms also reduce the amount of 'learning' required giving quick results, essential in Agile and DevOps environments. Ensuring that the algorithms are delivering defects and coverage is a balancing act, and one where AI will need input from a smart tester who knows the system well and can dynamically adjust the trade-off between coverage and bug-hunting.
3. Continuous test, continuous learning, predictive trends - Testing AI driven apps is not just a ‘one and done' exercise. It should be a continuous process so that we are essentially monitoring a user's experience over time, not just to predict the failure of a vehicle, but to predict and fix anything that may interfere with an optimal user experience.
AI algorithms will watch test results over time, learning and looking for trends. These learning algorithms can then build decision trees that enable predictive analytics to identify, for example, the increasing delay on a particular workflow and whether we're heading for a system outage. With AI, we can address this before it becomes critical and causes customer outrage.
So will these smart algorithms replace humans? No. Intelligent test automation will increase the productivity of the human tester when it comes to testing autonomous vehicles. AI and algorithms will evaluate systems to auto-generate previously resource intensive scripts, and to analyse the results to predict bugs and adjust the scripts to improve test coverage.
In summary, to achieve the goal of automation in testing for autonomous vehicles, you need tools that can intelligently navigate applications, predict where the quality issues are most likely to be, and identify the key data correlations that will help developers resolve issues quickly. AI will transform app development and, in turn, have a dramatic impact on autonomous vehicle user adoption, conversion, retention and—most critically—revenue generation.
Contributed Dr John Bates, CEO of TestPlant
*Note: The views expressed in this blog are those of the author and do not necessarily reflect the views of SC Media or Haymarket Media.