From Spoken Thoughts to Automated Driving Commentary: Predicting and Explaining Intelligent Vehicles' Actions

Abstract

Commentary driving is a technique in which drivers verbalise their observations, assessments and intentions. By speaking out their thoughts, both learning and expert drivers are able to create better understanding and awareness of their surroundings. In the intelligent vehicles' context, automated driving commentary can provide intelligible explanations about driving actions, and thereby assist a driver or an end-user during driving operations in challenging and safety-critical scenarios. In this paper, we conducted a field study in which we deployed a research vehicle in an urban environment. While collecting sensor data of the vehicle’s surroundings, we also recorded a driving instructor using the think-aloud methodology to verbalise their thoughts while driving. We analysed the collected data to uncover necessary requirements for effective explainability in intelligent vehicles. We show how intelligible natural language explanations that fulfil some of the key elicited requirements can be automatically generated based on observed driving data using a simple tree-based approach. Finally, we discuss how our approach can be built on in the future to realise more robust and effective explainability for driver assistance as well as partial and conditional automation of driving functions.