I want to recall that Tesla autopilot has failed for more than one time because of the perception module. It shows the unreliable nature of general machine learning (ML) algorithms. We know that 90% precision level is very high for almost very highly trained ML models. It is great for many applications, however, it is dangerous for mission-critical applications such as automated driving.
We can consider self-driving cars are probabilistic robots in which there are many uncertainties. One of them is the unpredictable environment where the robots operating. Thus, the training set size of the perception module is very small compared with the number of all possible scenarios in the environment. In summary, we could say that we are deploying a unreliable module into a safety-critical system.
People can say that we have high definition (HD) maps to support the decision of the automated driving system. However, HD maps cannot deal with dynamic objects such as other vehicles (e.g. the semi-truck). Therefore, it seems that it is not safe when the system’s logic relies on perception outputs only to make decision.
It suggests that we might need to think about other solution instead of relying on ML algorithms-based perception. One of the possible solution is reasoning about the robustness of perception and other modules in the automated driving system with respect to the uncertainties as discussed in this post.
Now, we think a little about a perception (object detection and tracking) whose accuracy is 90%. Let’s assume that it runs with the frequency of 10Hz. We want to know what is the probability that it makes a wrong detection during one hour of operation. In one hour, it performs \(10 \times 60 \times 60 = 36000\) inferences. Thus, the probability is \(1 - 0.9^{36000}\) which is very closed to \(1.0\).