The US government will announce soon the root cause of the router outage that led to the third-ever nationwide ground stop issued by the US Federal Aviation Administration. But questions remain about whether the FAA’s effort to modernize the national airspace system is being done with all the proper risk assessments and adequate agency oversight of the private contractors involved. The nationwide ground stop forced about 3,500 aircraft to immediately land and impacted more than 1 million passengers across the United States in the middle of the afternoon on Friday, November 13. More than 12,000 flights were canceled on that day alone, with the number rising to more than 20,000 over the weekend as stranded travelers scrambled to find accommodations or alternative transportation. Nearly one week later, FAA Administrator Bryan Bedford told news reporters that government investigators are “very close” to figuring out the root cause of the nationwide ground stop, according to CNN . Bedford also said that Transportation Secretary Sean Duffy plans to announce the findings at a news conference as soon as Friday, November 20. Problems first arose when “data began flooding” the FAA Telecommunications Infrastructure (FTI) network with 1,000 connection attempts every 4 to 5 seconds, according to an FAA Q&A post on November 15. The FTI network supports the flow of crucial flight data along with voice and video communications to air traffic control centers across the country. That caused “instability with aircraft communications and radar feeds” at 17 of the 22 Air Route Traffic Control Centers that manage en-route air traffic across the country, according to the FAA. The instability also impacted several Terminal Radar Approach Control facilities that guide aircraft both on approach and when departing from airports. An initial agency investigation identified a radar site in Amarillo, Texas, and a radio communications site in El Dorado, Arkansas, as “the reason for the excessive data flooding the systems.” The FAA’s network service provider, L3Harris, was installing a new router at the El Dorado site when the incident occurred and had recently installed a similar router in Texas. Specifically, the L3Harris contractor was deploying a new Juniper Networks router to replace a 25-year-old Cisco router at the El Dorado site. But the subsequent incident prompted the FAA to pause installation of the Juniper routers and turn off those that had already been installed, falling back on the older Cisco routers, according to the Associated Press . FAA employees typically conduct risk assessments and coordinate with contractors, such as L3Harris technicians, on system maintenance or upgrades, according to The Professional Aviation Safety Specialists (PASS) union that represents FAA technicians. But in a statement , the union said that none of its FAA technicians were involved in the upgrades that led to the ground stop. Similarly, CNN reported that the router replacement at the remote radio communications site in El Dorado was done by an L3Harris engineer without direct FAA oversight. The union and two sources told CNN that the FAA technician merely escorted the contractor into the facility and then turned his attention elsewhere without knowing what work the contractor was doing. To get a former insider perspective, Ars conducted a lengthy interview with Philip Mann, a principal consultant at Vector Strategic Consulting LLC, where he advises on aviation safety and AI governance. Mann previously worked at the FAA in multiple roles for 17 years, starting as a technician at Denver International Airport and eventually becoming a training program manager at the FAA Academy in Oklahoma City. (This interview has been condensed and edited for clarity.) Ars Technica: What do you see as some of the potential contributing factors that led to the nationwide ground stop? Philip Mann: I spent a lot of time as a technician, and then did a bunch of things in between. And later on, I was an instructor at the FAA Academy, and then I did training program management. So I’ve seen a lot of this from all sides about how it’s supposed to work, and the thing that stood out to me the most immediately when I saw this is the time during the day in which the work was being done. As an FAA technician, this is managed at the field person level. This is not like a top-down command-control thing. It is your job to cultivate a little credulity and think, “Okay, this thing I’m working on, what’s the worst thing that could happen? How far can it reach? What’s the blast radius?” This impacted a nationwide network, so by default the blast radius is the largest possible blast radius, so that basically means you take the maximum possible mitigations. To kind of contrast that, when I worked at Denver [International Airport], a lot of times I was working on things that were on a single runway, like one approach lighting system or one electronic system, and the blast radius was just that runway, not even the whole airport, just that specific runway, and sometimes only that one end of that one runway. So you can still use the rest of it just fine. And then maybe we work on a radar site, and a radar site might have a little more of a regional area. Most of the ones I worked on were kind of short-range radar, so it might be like a 50-mile radius or something. In which case you look at what the edges are, and you assume that if something goes wrong, that whole area just goes dark for air traffic controllers. So you plan around that, and the coordination goes on up to regional, national stuff like this. And this is the far end of that, just basic risk management stuff. The bigger the potential impact, the more people and operations potentially affected, the more you emphasize doing things like working at night when there’s less traffic, which is when this should have been done. And even Duffy has kind of acknowledged that [the FAA] confused the physical location of the facility with how far that system and service could reach within the national airspace system, and they had it completely backward. Yeah, on a map this is out in the middle of nowhere, but in a network diagram this router was immediately adjacent to all 22 of the [FAA’s air route traffic control] centers. Of those, 17 were affected, but it was immediately adjacent to all 22 of them. That was the thing that immediately struck me: How clear it is that basic impact analysis and basic risk management seems to have just been completely gone. It even goes a little bit further than that. As a technician, we used to have this rule that you don’t break anything on a Friday. We would actually not schedule certain work past lunch, and it wasn’t to get people off early. It was basically because, if something broke, somebody was going to have to come in on Saturday to fix it. You break it Thursday, people are going to be in on Friday anyway. And then we had the commandment: “Thou shalt not be on the front page of the newspaper,” and it seems like all of that was missing here. L3Harris should know better, because Harris has a history. Back in 2009, they misconfigured something and caused a big network outage . At the time, they had only identified a small handful—I want to say it was like a couple dozen potential areas of impact—and it wound up being like six times that. So they should know better. (Ars note: The 2009 network incident was caused by a Harris technician incorrectly configuring a router that directs air traffic data through the FAA telecommunications network. That error ended up delaying thousands of travelers and grounded hundreds of flights nationwide.) I don’t know how it was done on site, but when I did this work, quite often when contractors were working, I was the person who was there that got them cleared to the fence, that got their badges, that got them scheduled, that did the coordination with the airport authorities, and whatever else we had to do. So there was always an FAA person that was the point of contact, and even on some facilities it might have to be an FAA person that physically pushes the button to turn the system back on. Like in a lot of cases, a contractor cannot do that, and it might be quite literally as simple as just flipping a switch. But the thing is, that action puts that system onto the national airspace system. That has to be done by the government. It can’t be done by a contractor. Now, I don’t know if that’s the case here. I’m just saying it’s typical. Ars Technica: The Professional Aviation Safety Specialists (PASS) union representing FAA technicians put out a statement saying that, “historically contractors have followed a protocol where FAA employees conduct a risk assessment and coordinate the maintenance of the system or equipment.” The union then noted that “PASS-represented technicians” were not involved in the upgrades on the “contractor owned and maintained routers” that led to the ground stop. Mann: In my experience, the union a lot of times is one of the groups that raises their hand and says, “Hey, hold on, we need a qualified person. What does that actually mean? Like, who on this list of people is qualified? Does that mean that they’ve had the class? Does that mean they’ve got a certification? Does that mean they’ve been here for a certain period of time?” But they’ll figure that out, and the answer is not always the same. So, in a different FAA system at a different time, we had to get somebody who had been to a specific set of schools and, in one case, some specific vendor schools. The FAA at that time had only sent like a half-dozen people through those schools because they’re really expensive and really long. They had a couple in the east, a couple in the west, a couple in the middle, and every time they did one of these projects they would grab one of those people and put them on there, because that’s what a qualified person is. Maybe they had somebody for the FAA that was there, but maybe that person was a new hire who was administratively qualified to be on site, meaning they had their background check and their driver’s license and had their access badges and all that. But maybe they didn’t even have a networking background or whatever. They were just the person who had the access and the free time and could be there. But this is certainly a case, especially with the national-size blast radius of a network, that highlights the importance of having that qualified person, whatever that means. That is very much a union push to make sure that we know what the right qualified person means, that we know how to identify those people and that we’re treating those people fairly. It’s important to understand that in the union leadership, they’re all former technicians and engineers, everybody who runs that union. Even the local shop people are all fully qualified. So they all know what’s going on. Their role in all this potentially is to cover that blind spot that management and contractors share. Ars Technica: Based on what you described, it does sound like both the contractor side and the current FAA side were underestimating the potential blast radius for whatever reason. Mann: Well, yeah, and Duffy has even indicated that they mistook the physically remote location for being a network remote location, and those two don’t correspond to each other. Duffy has said that it was the physical smallness and the physical remoteness of the site that somehow let them forget about what they were actually working on. Ars Technica: That’s kind of astounding, isn’t it? It’s like the current FAA leadership acknowledging that somehow they didn’t know about their own network and the importance of different nodes in the network. Mann: In recent years they’ve brought a lot more people into the fold with the FAA, and you can see that overall a lot of it is more Silicon Valley-type people. It’s not aviation people or risk people or even necessarily