Riding Out the Hurricanes
Calcasieu Parish Network Team Shelters in NOC to Protect Critical Data
“You just do what you gotta do,” says Yvette Ardoin, Network Operations Coordinator for the Calcasieu Parish School Board.
The Network Operation Center, or the NOC as its employees call it, is the hub for the School Board’s 72 sites, including 60 schools and 12 administrative buildings in the Lake Charles, Louisiana community.
The team looks after more than 40,000 users, 35,000 students and 5 thousand plus employees.
“All that data that’s not in the cloud, is in this little room,” says Ardoin.
That data includes every user account, student file, maintenance record, and security video.
That’s a lot of responsibility when your town becomes the target of a Category 4 storm surge.
“We are the keepers of the data,” says Ardoin.
Being the keeper of information is not a job Ardoin and her staff take lightly.
In order to protect the data, the NOC is running on a generator, keeping their servers powered along with fans and air conditioners to prevent equipment from overheating.
Guarding the information, means ensuring the servers never turn off without being shut down properly, particularly when grid power is unreliable.
The team has faced close calls in the past with Hurricanes Rita and Ike, and acted on their lessons learned to be better prepared for the next one.
After Rita, the NOC requested showers be installed to accommodate the 24 hour babysitting the servers required. After Ike, the NOC invested in a bigger generator, providing more hours of power.
But with Hurricane Laura brewing in the Gulf of Mexico and rolling towards Louisiana, the keepers battened down in the NOC, expecting the worst.
Hurricane Laura, a Category 4 storm with 150 mph winds, and a storm surge of 17.2 feet slammed into Southwest Louisiana in the early morning of August 27, severing power and communications throughout the Lake Charles area.
While the NOC escaped a direct hit, the building sustained roof damage, blowing water into the sacred server room. The keepers held the water at bay by creating a makeshift water chute to divert the flow away from the equipment.
The uncertainty leaves the team awake at all hours, checking every buzz or drip of excess water leaking from the ceiling tiles.
“We might as well be here” says Ardoin, “All of our houses have suffered major damage, who knows when that wait will be over.”
“I’d rather work,” she adds.
Losing Track and Looking for Answers
Walking through the aisle of servers in the Datacenter, Ardoin and her lead technician, Michael Franks check each one.
Squinting their eyes as they note individual green lights, signs their oversight has been successful.
“Did you make your bed?” Ardoin asks Franks. The question is not common practice in office culture, but they aren’t simply co-workers during this event, they are roommates.
Ardoin and Franks, and a handful of other employees have been living at the NOC since the hurricane.
The team has been sharing one shower, one bathroom, an office dog, and a kitchenette since Hurricane Laura took away what was left of normalcy in an already difficult year.
Ardoin takes turns with Franks making water runs as the town is still on a boil notice, while the other team members look for food cooked by disaster relief organizations.
These errands are their only break from work, and the only time they leave the office.
“It’s weird baby sitting servers and never going home, you never leave work” says Franks, “You lose track of the days and something always comes up.”
The team hooked up alarms that sound throughout the NOC if the generator fails.
As a back up to their back up, a team member sleeps within earshot of the generator. If it goes quiet, he’ll be the first to notify.
Ardoin and Franks have converted two rooms closest to the servers into bedrooms, one at the front and one at the back. If the generator fails, they will work towards the middle, shutting off servers on their side of the room.
Together they would have 50 minutes to properly power down the equipment. These living arrangements will continue until Lake Charles is back up with electricity.
“Welcome to my bedroom,” Ardoin says entering the vault next to the server room.
On the side of the shelves hangs her clean clothes, next to the backup landline phones. Ardoin has been sleeping in a room smaller than a cell, for more than 20 days.
“We’ve been here ever since the storm, 24/7” says Ardoin, “My work family keeps me going, I’m glad I’m not having to do anything alone.”
The problems inside the NOC were enough to keep the team busy day and night, but the problems in the district were growing weeks after Hurricane Laura.
Hurricane Delta is the newest threat, with grim predictions for Lake Charles and it’s 80,000 residents.
“Now there are more hurricanes brewing in the gulf. We just keep watching the news thinking ‘not us, please,’ I cannot take another hurricane.
I’ve had enough” Ardoin said.
“The amount of anxiety and stress and long hours, it takes almost all four of us to think through something because we are so tired” says Ardoin, “We can’t do it by ourselves”
Help Found the NOC
After months of challenges adapting processes for COVID, the hurricane couldn’t have come at a worse time.
“We thought COVID set us back but now some of our schools don’t even have roofs. Our problems were so big, ITDRC heard about us,” Adoin says.
During a briefing with the National Guard, ITDRC’s Assessment Team learned of the challenges Ardoin’s team faced, and met with the school board’s leaders and tech team.
“Joe and Noah were here and planning how to help us, we got so excited for progress” says Ardoin “It was such a relief to talk to people that can help and not come up with all the answers by ourselves.”
Although ITDRC is a volunteer-driven organization, its members are subject matter experts in their respective technology fields. The diverse response team included expertise in systems, networking, wireless, voice, infrastructure, and RF communications — most every resource the district needed to be successful.
With service restoration “months out”, ITDRC helped the district come up with a temporary connectivity plan, providing network and microwave hardware, and leveraging the resources of its Tech Task Force partners. “We just didn’t have the equipment, and the equipment we had was down,” says Franks.
“The great thing about the schools ITDRC connected is that they are K-12 so if we can get one of those schools up, it serves the whole community pretty much,” says Franks.
ITDRC also helped the district’s small Technology Team assess each of the schools network infrastructure for damage; repairing what they could, and identifying equipment that was destroyed.
“We are now calling ourselves internet island,” adds Ardoin “Because we are the only ones up.”
“It was nice to have someone come in and say we are going to give you some answers, we are going to give some help,” says Franks.
Following up
Six weeks after Hurricane Laura, category 4 Hurricane Delta added insult to injury for the NOC and Calcasieu Parish School Board.
Ardoin had received a new roof on her family home, just two days before Delta ripped it off for the second time in a matter of months. While inspecting the damage with her husband, Ardoin fell 25 feet off of her 2-story roof, landing on the concrete below.
She suffered a broken pelvis, hand, foot, and blood clots in her lung and brain. Due to the severity of her injuries and hip replacement surgery, Ardoin was in recovery for 4 months.
Once Ardoin was released from the hospital, she was left wheelchair bound while attending physical therapy.
“I feel very blessed that I’m doing as well as I am,” said Ardoin, “I’m just thankful that I’m here to talk about it.”
Unfortunately, school facilities and buildings suffered additional damage as well, but the LA school district remains vigilant with distance and in person learning.
Delta missed the power grid for the NOC, and the network is well protected by the data keepers. They have avoided all leaks through their temporary roof and hope to fashion a permanent one at the NOC in the near future.
“It’s been very, very devastating” Ardoin said, “If you were to drive around and here we are 6 months later from the last hurricane, you would look at it and think this must’ve just happened.”
Ardoin rejoined the data keepers in February, a testament to her dedication to her team and community.
