In the worst-case scenarios that the Biden administration has quietly simulated over the past year or so, Russian hackers working on behalf of Vladimir V. Putin bring down hospital systems across the United States. In others, China’s military hackers trigger chaos, shutting down water systems and electric grids to distract Americans from an invasion of Taiwan.
As it turned out, none of those grim situations caused Friday’s national digital meltdown. It was, by all appearances, purely human error — a few bad keystrokes that demonstrated the fragility of a vast set of interconnected networks in which one mistake can cause a cascade of unintended consequences. Since no one really understands what is connected to what, it is no surprise that such episodes keep happening, each incident just a few degrees different from the last.
Among Washington’s cyberwarriors, the first reaction on Friday morning was relief that this wasn’t a nation-state attack. For two years now, the White House, the Pentagon and the nation’s cyberdefenders have been trying to come to terms with “Volt Typhoon,” a particularly elusive form of malware that China has put into American critical infrastructure. It is hard to find, even harder to evict from vital computer networks and designed to sow far greater fear and chaos than the country saw on Friday.
Yet as the “blue screen of death” popped up from the operating rooms of Massachusetts General Hospital to the airline management systems that keep planes flying, America got another reminder of the halting progress of “cyber resilience.” It was a particularly bitter discovery then that a flawed update to a trusted tool in that effort — CrowdStrike’s software to find and neutralize cyberattacks — was the cause of the problem, not the savior.
Only in recent years has the United States gotten serious about the problem. Government partnerships with private industry were put together to share lessons. The F.B.I. and the National Security Agency, along with the Cybersecurity and Infrastructure Security Agency at the Homeland Security Department, issue bulletins outlining vulnerabilities or blowing the whistle on hackers.
President Biden even created a Cyber Safety Review Board that looks at major incidents. It is modeled on the National Transportation Safety Board, which reviews airplane and train accidents, among other disasters, and publishes “lessons learned.”
Just three months ago, it released a blistering account of how Microsoft allowed intrusions into its cloud services that permitted Chinese spies to clean out State Department files about Beijing and Commerce Secretary Gina Raimondo’s emails. But by the time the report came out, American officials were focused on a more urgent problem: the spread of ransomware attacks, many from Russia.
It was the Russians, in fact, who woke up America about the vulnerability of the “software supply chain” problem that lets small errors ripple into large consequences.
In the run-up to the 2020 presidential campaign, Moscow’s most skilled intelligence service bored into a component of that supply chain, worming its way into the update systems of software made by Solar Winds. The company’s products are intended to manage large computer networks, and the Russians knew that once they had access to the update system, they could spread a lot of malicious code fast.
It worked. Hackers soon gained access to the Treasury and Commerce Departments, parts of the Pentagon and scores of America’s biggest companies. They did no visible damage. They did not trigger panics like the kinds seen on Friday. But they got the incoming administration’s attention.
“In a globally interconnected economy, we need to ensure that we have the resilience” when an event like this happens, said Anne Neuberger, the deputy national security adviser for cyber and emerging technologies, a job that did not exist until the Biden administration invented it.
Ms. Neuberger was awakened by the White House Situation Room at 4 a.m. on Friday in Aspen, Colo., where she was preparing to speak on a panel titled “Securing Trust in the Global Digital Economy.” She spent the day assessing the risks to U.S. government systems, then calling allies and executives, including the chief executive of CrowdStrike, George Kurtz. She asked, “Is there anything we can do to help?”
Ms. Neuberger, a former senior official at the National Security Agency, knows better than most that for now, there are no magic bullets. By the time an event like this happens, the only response is to mount a painstaking effort, step by step, to patch the error, push it out and try to wrench thousands of systems back online.
Sometimes it works. Sometimes, as the British Museum discovered recently after an enormous ransomware attack that British intelligence officials think may have ties to the Russian government, even the best of efforts to recover can fail.
“This is not something that is new, but it has been accelerated by technology and by the interconnectivity,” Sir Jeremy Fleming, the recently retired leader of GCHQ, Britain’s famed code-making and code-breaking agency that is the equivalent of the N.S.A. And these days, he worries more about criminals than nation-state attacks.
Criminals will certainly be gleaning lessons from the CrowdStrike debacle, learning how to exploit the kinds of vulnerabilities that brought television stations and airports and insurance companies to a halt. So will Mr. Putin and President Xi Jinping of China, who now have, by accident, a more detailed road map for disruption, in an election year when they may well have an interest in interfering.
It is not hopeless.
“We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.
But that will take awhile. And in the meantime, unintended cascades of chaos will keep rippling around the globe — some, like Friday’s, a product of error. The fear is, in an election year, that the next digital meltdown may have a deeper political purpose.
The post What Happened to ‘Digital Resilience’? appeared first on New York Times.