By Jillian Deutsch, Todd Shields, Jake Bleiberg and Jennifer Jacobs | Bloomberg
AT&T Inc. said a widespread outage that took hours to resolve Thursday was caused by “an incorrect process” while expanding the wireless network.
The software issue interrupted wireless service for hundreds of thousands of subscribers and prompted the FBI and US Department of Homeland Security to investigate the outage.
“Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack,” an AT&T spokesman said in a statement. “We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve.”
RELATED: AT&T seeks to shut down landline service for most of the Bay Area, much of California
AT&T said all wireless service was restored Thursday afternoon, capping a day of frustration that began in the early hours of the morning New York time. AT&T customers filed more than 1.5 million outage reports on service-tracking website Downdetector.
The federal government began investigating whether the network failure was caused by a cyberattack, according to two US officials familiar with the situation, who requested anonymity to discuss sensitive information.
The Federal Communications Commission also has been in touch with AT&T to try and ascertain the cause, White House spokesman John Kirby told reporters earlier. “DHS and the FBI are looking into this as well, working with the tech industry, these network providers, to see what we can do from a federal perspective to enhance their investigative efforts to figure out what happened here,” Kirby said.
Early Thursday, mobile-phone customers from multiple carriers started reporting problems, but it soon became clear that AT&T’s network was the culprit. Outages were reported from cities including New York, Houston, Atlanta, Miami, Chicago and Dallas. The service disruption upended communications with emergency responders, and officials took to social media urging AT&T customers to use landlines to call 911 for emergencies.
With about 87 million subscribers, AT&T is the third-largest US retail wireless carrier, behind Verizon Communications Inc. and T-Mobile US Inc., according to data compiled by Bloomberg. Verizon and T-Mobile both said their services were working normally. AT&T shares fell 2.4%.
The impact was felt far and wide. Emma Smits, an AT&T customer, was on the Metra express train into Chicago. Normally, the public relations account executive spends the hourlong commute prepping for client calls, pitching reporters and catching up on overnight tasks. Not on Thursday.
“I couldn’t cross anything off my work to-do list,” said Smits, who watched her fellow commuters turn their phones off and on to try and catch a cellular signal in a futile attempt to send emails or get work done.
Workarounds abounded, sending some back to simpler times. Vanessa Stowe had to screenshot directions to a morning meeting “like it’s 2009 MapQuest.” Sarah Kittel, a communications strategist, had to connect to a nearby restaurant’s Wi-Fi to pay for parking.
“So much of our work and lives depend, sadly or otherwise, on our ability to be reached or reach others at a touch of a button,” Kittel said. “That simply didn’t happen this morning.” Cate Luzio, traveling from New York to Washington for a big client event, was more blunt: “It was a mess.”
Among those impacted were employees of the FBI. In an email to all staff reviewed by Bloomberg, the agency’s Information & Technology Branch said “a nationwide cell service outage is preventing some Google Pixel devices from connecting to cellular coverage. Some users have been able to reconnect to the network by rebooting their Pixel (press and hold the Power button, then press ‘Restart’).”
The public safety network FirstNet was the only government entity impacted, and service there has been restored, said Kirby, the White House spokesman.
FirstNet provides communications services to about 27,500 public safety agencies in a partnership with AT&T.
Related Articles
Santa Clara tech campus lands leases, offering hope for real estate market
California’s US Senate candidates agree AI should be regulated — but they differ on why
Magid: How to protect your online security
AT&T says the outage to its US cellphone network was not caused by a cyberattack
Opinion: Biden proposal threatens California’s engine of innovation
Ken Corey, a former Chief of Department for the New York City police, said the outage at FirstNet “has impacted 911 response and public-safety communications across the country.” Corey, who spoke via email, leads a group that opposes letting FirstNet manage additional airwaves.
It’s not the first time AT&T has suffered a widespread outage. In 2008, the company dealt with an extensive wireless internet failure in the US Northeast. In that incident, a glitch in the way the company’s network was routing traffic was identified as the culprit and the service failure was relatively short-lived. In 2020, AT&T internet and phone service was knocked out in Nashville by an explosion in the city’s downtown area.
T-Mobile also has been hit with widespread outages, including one last February and another in 2020 that prompted the FCC to launch a probe.
“In my experience, this type of outage can negatively impact financial results in the quarter in which it occurs and cause short-term lost goodwill with customers,” David Heger, an analyst at Edward D. Jones & Co., said in an email. “However, it does not have a longer term impact on the business.”
–With assistance from Brendan Walsh, Antonia Mufarech, Jamie Tarabay, Matthew Boyle, Akayla Gardner, Brendan Case and Miranda Davis.
More stories like this are available on bloomberg.com
©2024 Bloomberg L.P.
