Why Facebook went down: What caused 6-hour outage on Facebook, Messenger, Instagram, WhatsApp
massive 6-hour outage that left billions of users off the world’s largest social network and its other apps.
The outage was “caused not by malicious activity, but an error of our own making” said Santosh Janardhan Facebook’s vice president of infrastructure in a Tuesday blog post.
The incident occurred when an engineer doing routine maintenance work issued a command “which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally,” Janardhan wrote.
“Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command,” Janardhan said. “This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse.”
Frances Haugen tells Senate that Facebook makes ‘disastrous’ choices, prioritizes profitDid the Facebook, Instagram outage have you on edge? 6 signs you need to unplug
Facebook, which also includes Instagram, dominates all social ad spending in the U.S. with an 82% share, according to research firm eMarketer. By comparison, the next closest is LinkedIn with 4.3%, Twitter with 3.8%, Pinterest with 3.4% and Snapchat with 3.1%. Others including the likes of TikTok, are at 2.2%
For example, Facebook’s U.S. digital advertising revenue is estimated to be more than $48 billion this year and $57 billion in 2022, eMarketer said. And, Instagram’s digital ad revenue is projected to be $25 billion this year and more than $32 billion next year, eMarketer reports.
Social media has been the most popular marketing tool among small businesses, said Patrick Gillooly, marketing director at digital marketer Constant Contact. He said a recent Constant Contact survey found 63% of small businesses use social media, predominately Facebook, to promote their businesses.
“A lot of businesses, small businesses, in particular, rely on social media to start, grow and scale and (Monday’s outage) could be make or break moment for some of them,” Gillooly said.
Advertisers and businesses using Facebook are likely breathing a sigh of relief. The tech giant said late Monday that there was “no evidence that user data was compromised as a result” of the outage.
However, one cybersecurity expert thinks that Facebook wouldn’t be as forthcoming if it had faced an attack.
“Would they even admit to being attacked, given the current environment they’re in?” said Tamara Schwartz, an assistant professor of cybersecurity and business administration at York College in Pennsylvania. “Because many people don’t already trust them due to the recent whistleblower’s accusations about their unethical choices and Congress accusing them of being irresponsible to its users.”
Schwartz added, “It’s not unreasonable to think that something else happened.”
When the outage occurred, Facebook experienced a one-two punch as “all of this happened very fast,” Janardhan said.when the outage occurred.
In the race to figure out what was wrong, the engineering team soon learned that Facebook and its litany of apps were apparently removed from the DNS (Domain Name System) servers that basically make up “the address book of the internet,” Janardhan said.
“(The engineers) faced two large obstacles: first, it was not possible to access our data centers through our normal means because their networks were down,” Janardhan said. “And second, the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this.”
Therefore, Janardhan said, the end result was that “our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers.”
As Facebook’s “out-of-band network access was down,” Janardhan said engineers had to go to the data centers onsite to “debug the issue” and restart the systems. But this took time, he said, as those onsite locations have high levels of physical and system security in place.
“Once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers,” Janardhan said. “Only then could we confirm the issue and bring our backbone back online.”
While Facebook is up and running again, Janardhan didn’t rule out the platform and its properties experiencing another outage, albeit one less impactful, from happening again.
“I believe a tradeoff like this is worth it – greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this,” he said. “From here on out, our job is to strengthen our testing, drills, and overall resilience to make sure events like this happen as rarely as possible.”