Undersea Surgeons – The New York Times

Something was wrong in the vast undersea canyon known as the Bottomless Hole.

One by one, internet cables were failing on a seabed so deep that no human has ever set foot on it.

And as they did, life in the cities far above them ground to a halt.

Life Without Internet

Any number of things can knock an undersea cable out of service.

Landslides can do it. So can a ship dragging its anchor. There may be unintended damage from military skirmishes. And then there is sabotage, a growing concern.

But most components of the physical internet are privately owned, and the companies behind them have very little incentive to explain any failures. That can make it daunting for people who rely on the cables to try to get a handle on why an outage is happening. Especially in real time.

On March 14, the regional chief information officer for the Ecobank Group in Ivory Coast knew only one thing for sure as he stared at signals blipping red in his offices: There was a problem.

Still, it was early in the day. Banks were not due to open for another 30 minutes. That was probably enough time, figured the information officer, Issouf Nikiema, for his I.T. engineers to sort it out.

Those hopes faded when the techs came back to his office in Abidjan. “Even their body language — I realized that something was really wrong,” Mr. Nikiema said.

Ecobank alone serves 28 million people across the continent. But many other businesses, from sprawling bank chains to modest food stands, were hit, especially after the fourth cable went out and the internet went into freefall.

Africa is a continent of 1.4 billion people where economic ambitions are high but the infrastructure often lags. People have learned the art of the workaround, and so when the electricity fails, generators often come to the rescue. If the WiFi goes down, mobile data might still do the trick.

But this time was different. In many places, the shutdown was total.

“Imagine waking up in New York with no WiFi at home, no data on your phone, no internet available at your local Starbucks, at your office, no way to check your bank accounts on your Chase app,” said Sarah Coulibaly, a technology expert at Ivory Coast’s national telecommunications agency.

In Accra, Ghana’s capital, international travelers arriving at the airport could not locate their rental cars.

Internet traffic to Ghana

Four cables broke on March 14, 2024

Note: All times for the four cable breaks are in UTC. The chart shows changes in internet traffic over time, rather than displaying total internet usage levels.

Source: Doug Madory, Kentik

In Abidjan, Ivory Coast’s largest city, restaurants couldn’t use WhatsApp to order local produce.

And more than 500 miles away in Ibadan, Nigeria’s third-largest city, Oke Iyanda couldn’t collect money for the food that she sells to students and university workers. Sales of abula, a popular mix of yam powder, vegetables, pepper stew and goat meat, plummeted and food spoiled.

The failures highlighted a broader problem for African countries: For all their techological progress, they are served by far fewer cables than more developed countries are, and often lack backup systems.

By contrast, when two data cables linking four European countries were cut in quick succession in the Baltic Sea earlier this month, service interruptions were relatively minimal. (American intelligence officials assessed that the cables had not been cut deliberately, but the European authorities have not ruled out sabotage.)

For Africa, some help is on the way. Starlink’s satellite internet technology now operates in at least 15 countries, and a 28,000-mile-long cable being built by a consortium of companies has begun to come online. Still, the continent’s dependence on private — and for the most part Western — internet providers can make true sovereignty elusive.

“We’re at the mercy of these cable operators,” said Kalil Konaté, Ivory Coast’s minister for digital transition.

For an Uber driver in, say, Stockholm or Buenos Aires, an internet outage is a big inconvenience. In Lagos, Nigeria’s largest city, it can mean calamity. With his clients locked out of their bank accounts, one driver there, Segun Oladejoye, said he went without work for three days.

In Lagos, Nigeria, as in much of West Africa, merchants rely on the internet to conduct daily transactions.

Taiwo Aina for The New York Times

The timing could hardly have been worse. Months earlier, Mr. Oladejoye, a 46-year-old father of four, had taken out a loan for his Uber car. With barely any savings, the only way he could pay back the $30 weekly installment and feed his family was through even longer hours of work.

Worried that the lending company might seize his car, Mr. Oladejoye said, he borrowed still more money, this time from a Chinese lending app. “It still hurts me and my family,” he said, “because I now have to pay back both loans.”

A Web of Fiber-Optics

According to Telegeography, an internet data and mapping company, there are hundreds of cables crossing the floors and canyons of the earth’s oceans. Stretched end to end, they would reach approximately a million miles.

Though not dramatically different in appearance from the slender cables a local TV provider would run into an apartment building, at any moment they are conveying a vast number of messages, from WhatsApp flirtations to complex financial transactions.

Note: A length of cable on the seabed was laid for training purposes and photographed by one of Orange Marine’s remotely operated vehicles.

Source: Orange Marine

People have been laying cables underwater since the dawn of the telegraph age in the mid-1800s, but those being put down now bear little resemblance to their forebears.

At the center of modern cables are fiber-optic lines, usually numbering four to 24 fibers. Thinner than a human hair, each is coated with a different color so they don’t get mixed up. The composition of the cables depends in part on the depth of the water, said Verne Steyn, director of subsea networks at WIOCC, a major digital wholesaler in Africa.

In deep-water locations, the cables often have a black outer polyethylene layer. Below is a wrap of metal tape, then another polyethylene layer, a copper sleeve to conduct electricity, and a tangle of stainless steel wires to provide strength. Only then comes a small metal tube holding the fiber-optic lines, which are often coated with glycerine jelly as a last protection against the water.

The result is a remarkably sturdy conduit — but not an invulnerable one. And in a world ever more dependent on the uninterrupted flow of data, that worries people.

Just weeks before the cables went out in the Trou Sans Fond, cables in the Red Sea serving East Africa and Asia were severed by a ship’s anchor. They were a casualty of war: The ship had been hit by a missile fired by militants in Yemen backing Palestinians in Gaza.

And about two months later, two more cables were torn apart in shallow waters off Mozambique by a fishing trawler. Its crew had reportedly switched off its tracking system so it could operate in protected waters.

Some communications experts argue that the way to make internet infrastructure more resilient to the inevitable problems is redundancy — just lay more cables, so there are more alternative pathways for data, and that has happened. Twenty years ago, for example, there were just two major cables strung along the West African coast, according to Mr. Steyn.

But sometimes, that just means more cables are cut at once.

“The seabed is not as peaceful as it once was,” said Doug Madory, director of internet analysis at Kentik, a network monitoring company. “Just adding more cables doesn’t solve all your problems. The fact of today’s internet is that we’ve got to survive multiple cable cuts in a single incident.”

It might be better, he and other experts say, to diversify the location of the cables and set up more on land, though that can be more expensive and pose geopolitical challenges.

A grapnel, used to retrieve cables, on the deck of the Léon Thévenin in April.

Tommy Trenchard for The New York Times

And more cables can do only so much.

Katarzyna Zysk, a professor at the Norwegian Institute for Defense Studies in Oslo, said that there were mounting, credible reports of sabotage around the world. “I believe that the infrastructure is highly vulnerable and presents an attractive target,” Professor Zysk said.

Sabotage did not, however, appear to play a role in the outage in the Trou San Fond, analyses of the crews that eventually repaired the cables and independent experts interviewed by The New York Times said.

.

To try to understand what happened, Mr. Madory, a pathologist of sorts for the undersea communication network, used clues from the internet’s global addressing system, known as BGP, and the network’s attempts to route traffic around the broken connections. He was able to pinpoint the time of the first cable failure at 5:02 a.m. local time. The three others followed at 5:31, 7:45 and 10:33.

“You can see in the routing system a little scramble as the rest of the internet tries to figure out how to reach these networks,” Mr. Madory said.

The cascade of failures offers strong evidence that the culprit was almost certainly one of the underwater mudslides or avalanches— scientists call them turbidity currents — that are fairly common in that region.

The Repair Crew

As the Léon Thévenin steamed northward along the coast, it was outfitted with a curious mix of old and new.

Coiled in its belly were miles of replacement cable and heavy rope. Steel grapnels were fastened to lengths of chain that would be dragged along the sea bottom to snag broken cables and haul them to the surface. The master of the ship, Capt. Benoît Petit, delicately rolled out huge charts — they resembled scrolls — showing the broad topography of the Trou Sans Fond.

Capt. Benoît Petit aboard the Léon Thévenin with his second officer, Sandile Mthembu, in April.

Tommy Trenchard for The New York Times

But there was also high-tech splicing equipment, and needles on dials in the ship’s work areas quivered as amber, red and green lights flashed.

Always on call, with sailors rotating in and out to keep the active crew at about 55, the Léon Thévenin is one of six repair ships operated by Orange Marine, a subsidiary of Orange, the French telecommunications giant. Orange Marine says it carries out 12 to 15 percent of the roughly 200 cable repairs that take place around the world each year.

Crew members sometimes have trouble making their families and friends online understand what they do on long voyages.

“I say it straight: ‘I’m a fiber optics splicer,’” said Shuru Arendse.

“What is that?” comes the reply, so he tries again.

“I repair the data communication cables on the seabed.”

But still no. So Mr. Arendse keeps it simple.

“I keep Africa connected to the rest of the world,” he says.

But before he can, his crew has to find the cable breaks — no easy task.

Tracking and repairing undersea cables

Frédéric Salle, the onboard mission chief, regards each repair as a forensic investigation and each break as a “crime scene,” even if malfeasance is not suspected.

But the evidence in this case would have to be deduced from surveys, charts and hauling up the cable itself rather than imagery of the sea bottom. The waters of the Trou Sans Fond were too deep and the canyon walls too steep to send down a camera-laden remote vehicle.

Didier Dillard, the chief executive of Orange Marine, said the crews operated in a world of the unknown.

“When you go beyond 1,000 meters depth,” he said, “nobody really knows what the seabed is like, because nobody goes there. It can be rocky, sandy, muddy — you can just imagine.”

But there were clues to where the breaks the Léon Thévenin was looking for might be, and what had caused them.

The cables’ depth put them out of reach of passing fishing nets or anchors. And Mr. Salle determined that they had broken in order from closest to the coastline to farthest — strong evidence that there had been an avalanche, since that was the direction one would speed down the slope of the canyon. Another sign: Light signals sent through the fiber optics revealed that the break was squarely within the canyon, where avalanches occur, Mr. Salle said.

“There was no doubt as to the identity of ‘the perpetrator,’” he said.

Note: A lab simulation of turbidity currents, which can damage undersea cables.

Source: Eurotank Laboratory, Utrecht University.

The repair itself, Mr. Salle said, involved cutting the cables on either side of the breaks and fastening them to buoys. Then jointers like Mr. Arendse got to work splicing a length of new cable into place.

First stripping off the colored coating, they carefully melted and joined the strands from two cable pieces — the microsurgery of internet repair — checking to be sure that laser light was flowing freely across the repaired joint.

They boxed it all up into an elaborate splint. Then it was time to drop the cable back into the sea and move on to the others.

When the last cable was patched, about a month after the crew left South Africa, it was time to head home.

With the breaks repaired, internet service returned to normal in West Africa — but “normal” is relative. Outages, though shorter, remain frequent. And some think another cable-snapping avalanche is just a matter of time.

Mr. Konaté, the Ivorian digital transition minister, said that the March outage was a wakeup call and that he had asked cable providers like Google to offer terrestrial backup solutions.

“This cannot happen again,” he said.

In the port of Cape Town, another Orange Marine mission chief, Didier Mainguy, said that for all the lasers and fiber optics, little had changed fundamentally from a century and a half ago. To make his point, Mr. Mainguy held up a mounted piece of old telegraph cable in his quarters.

“It’s still a cable,” he said. “It was a cable a hundred years ago. Voilà.”

Cable spooled aboard the repair ship.

Tommy Trenchard for The New York Times