Gradual loss of control

"On 6th May 2010, in an incident later known as the 2010 Flash Crash, leading U.S. stock indices abruptly fell and rebounded in less than half an hour, in the process erasing almost $1 trillion in market value. An investigation by the Security Exchange Commission found that a single order of large amounts of E-mini S&P contracts and subsequent selling orders by high-frequency algorithms triggered the drastic decline of market value [129], [130]. This event demonstrated the problem of algorithmic collision, where an increasing deployment of algorithms interacting with each other can lead to unforeseen and catastrophic consequences [131]. While the crash did not result in a loss of human autonomy or lives, it serves as a historical precedent for gradual loss of control, where it showed that risks do not require a sudden "fast takeoff" in AI capabilities. Instead, a gradual diffusion and deep entrenchment of AI into a system can lead to a sudden and catastrophic event."