Just ask officials at the National Finance Center in New Orleans, who saw to it that more than half a million federal workers got paid on time while Hurricane Katrina was bearing down, then made subsequent payrolls—NFC’s largest ever—without missing a beat.

Their continuity-of-operations plans stood up to Katrina in the short term. But no COOP plan is perfect, and NFC’s is being revised to account for unexpected long-term difficulties.

Despite its success, NFC’s mental picture of a COOP event was flawed, NFC director Jerry Lohfink said.

“All our scenarios, all our tests, all our planning were geared around loss of government facilities and resources.” he said. “We never imagined a disaster scenario that left the government facilities basically intact but that wiped out almost half the employee housing and devastated the local community infrastructure to a point that it couldn’t really support us.”

COOP and COG (continuity of government) planning dates to the Cold War, when government began to plan for maintaining operations after a Soviet nuclear strike. COOP planning was refined in light of such subsequent events

"If you can get 80 percent of it down, the other 20 percent becomes very manageable. " — Jerry Lohfink

as the Oklahoma City bombing and the year 2000 IT problem.

COOP planning has been formalized in executive orders and such other source documents as the Federal Emergency Management Agency’s self-assessment tool and Federal Preparedness Circular 65. FEMA is responsible for formulating COOP guidance across the government, coordinating interagency exercises and assessing agency COOP plans. But so far it has fallen short of that task, according to the Government Accountability Office (see story, Page 29).

NFC, a unit of the Agriculture Department, has 1,250 workers and provides payroll services for 565,000 federal employees in 132 federal organizations, including the Commerce, Labor and Homeland Security departments.

As one of the government’s major franchise providers of administrative and financial services, NFC has seen the scope and volume of its work expand rapidly over the last decade.

And just as NFC officials were getting ready to add 60,000 Transportation Security Administration and Coast Guard employees to the payroll system, they had to deal with a worst-case crisis-management scenario—a category 4 hurricane.

Lohfink said it was “inspirational” the way his people pulled together to get the job done despite personal loss and tragedy. He also credits the immediate, personal involvement of secretary of Agriculture Mike Johanns and his staff, who called Lohfink daily during the crisis to ask what they could do.

It’s no accident that NFC proved resilient. The agency had been doing some form of disaster planning and practice since 1985.

When Katrina was still a churning threat at sea, the first thing Lohfink did—and he said he would start exactly the same way the next time—was to reach for his hard copy of NFC’s disaster recovery plan. Then he checked the storm forecasts, assembled his senior managers and started going down the checklists in the plan. The checklists got him through the first week.

His next move was to walk around the building talking to employees. While this may not seem like something that would make it into a formal COOP plan, Lohfink considered it important to show people that the agency cared about them. He asked about their hotel reservations and whether their families and relatives had made it out of town. “Take care of your people during an event, and they’ll deliver for you,” he said.

The night before Katrina hit, NFC sent data backup tapes by truck to an alternate computing facility hosted by SunGard Data Systems Inc. of Wayne, Pa., near Philadelphia. The agency was able to resume data processing in Wayne by that Tuesday, with NFC IT personnel deployed there in advance of the storm.

Voice communications in New Orleans were a shambles after the storm. Land line switching centers were flooded, cell towers were down and what remained of the cell network was overwhelmed with calls. The COOP plan called for establishing temporary teleconferencing capability, and an 800-number had been set up as the storm approached. With two teleconferences a day, top managers were able to maintain command and control even though they had dispersed to different locations—and even if they were stuck in traffic. “That was phenomenal,” NFC CIO Gil Hawk said. “We could continue to coordinate even as we were evacuating.”

Also helping were new BlackBerry wireless devices from Research in Motion Inc. for top managers, an 800-number playing updated recordings for rank-and-file employees and a Web site to update customers, Hawk said. However, as part of an updated plan, he is issuing new disaster recovery cell phones with area codes outside New Orleans that will continue functioning if cell service is disrupted there again.

Hawk said another lesson from the degradation of local phone service is that the agency needs employee contact information outside the city (for example, relatives in another state) so officials can locate all staff members after a future event.

Before the storm hit, NFC also deployed administrative and acquisitions staff to its primary alternate work site, a SunGard facility in Grand Prairie, Texas. Getting the site up and running went smoothly as a result, NFC’s chief administrative officer Debbie Byrne said.

“We realized prior to Katrina how critical [the staff was] to the business recovery of the NFC. Without a contracting person, we wouldn’t have been successful,” said Anita Fincher, NFC’s alternate work site manager. The first acquisitions were mundane office supplies such as pens and paper. With 250 to 500 people arriving, and half of a six-foot table constituting a workstation, it was important to be organized. Paper clips became “very precious commodities, you wouldn’t believe it,” Fincher said.

Additional small-purchase credit cards were issued, because not all previous cardholders had deployed to the site. This freed up Byrne and her team to move on to such larger emergency procurements as additional computers and employee housing. The emergency gave Byrne a much better idea of what each unit needs, she said.

Personal losses. But providing administrative support to employees who had left New Orleans with little more than the clothes on their backs was a “real strain,” Byrne said. Her team had to ad lib to meet employees’ personal needs—coordinate the distribution of donated food, bring a physician on-site to give tetanus shots and write temporary prescriptions, and arrange for counseling employees, some of whom had lost everything.

“That’s really a major lesson learned for us—that we’re going to need more administrative support” in future events, Byrne said. NFC did not adequately plan administrative support, given that the agency eventually was spread out over some eight alternate work sites.

It’s no accident that NFC proved resilient through Katrina and its aftermath. The agency had been doing disaster planning and practice in some form or other since 1985. There is a structured timeline for reviewing the COOP plan every year, beginning with COOP requirements and business impact analysis at the unit level in February.

While NFC consulted all the usual source documents (FPC 65 and others), what made the difference, in Lohfink’s view, was that COOP planning was not simply left to a COOP staff.

“We took a different approach here,” he said. “We involved the day-to-day operations employees in designing, planning, testing, refining and executing the COOP plan. It was basically their plan.” Operational employees often have the best ideas, and you have to be open to them.

“If you don’t have people involved [who] fundamentally understand every aspect of the business model and how it works, you’re not going to design an effective COOP strategy,” Lohfink said. “You’re going to have so many little holes that it may just collapse.”

In Washington, USDA deputy CFO Patricia Healy played a key role, providing management oversight, dispatching associate CFOs to the alternate facilities in Texas and Pennsylvania, and acting as a conduit between NFC officials and the secretary’s office at Agriculture, said USDA CFO Charles Christopherson Jr. Streamlining the chain of command brought important issues to top management’s attention earlier and served to communicate decisions more quickly. Having a narrow, focused set of priorities made management decision-making up and down the chain very simple.

NFC had, over time, winnowed its list of 100 activities down to 22 critical services. Post-Katrina, Lohfink told employees that NFC only had three priorities: Serve the customer, take care of NFC employees and maintain fiscal management. He told his people not to spend time on things that didn’t relate to those three objectives. There has been considerable attention given to the difficulty of defining essential functions throughout the government, and GAO has criticized agencies for having too many (more than 500 at one agency, according to GAO). NFC uses business impact analysis and planning software from Strohl Systems Group Inc. of King of Prussia, Pa., which puts the focus on workflow in and out of the various units.

“It’s very comprehensive analysis, and it provides very good information for building our recovery plan,” said Archie Bertrand, NFC’s chief cybersecurity officer and disaster recovery manager. The software helps COOP planners recognize activities that could break down without proper planning and calculate recovery times.

Risk management at NFC is built on the work of contractors who performed extensive formal risk assessments, developed a list of threats, ranked them in order of probability and computed annual loss expectancies for each. Risk management is incorporated into the budget, Bertrand said.

When others suggest trimming COOP expenses and putting the money toward something else, Bertrand reminds them of the value of contingency planning and practice. He gets his point across, he says, by telling colleagues, “OK, here’s the risk. If you want to take it, here’s what it could cost you.”

NFC has specifically planned and practiced for hurricanes, deemed the most likely threat given the near-misses that have occurred in the past. The agency originally had an alternate work site in Atlanta, but switched to Texas long before Katrina because hurricanes generally turn east after landfall. It didn’t make sense to have a backup facility directly in the path of the same threat that could bring disaster to New Orleans.

Twice-a-year testing “really helped hone our readiness,” Lohfink said. The drills involve activating call trees and deploying advance teams to alternate sites. While there, the teams test communications to the data center and ensure that network applications and tools are available at workstations.

Glitches from software upgrades and other things that go wrong in each drill are resolved on the fly or in the next COOP planning session and then tested in subsequent drills, Byrne said.

Customers are invited to participate in drills, and additional USDA and governmentwide exercises take place, said Bertrand. “We don’t pass up drills anymore—that hurricane was real,” he said after returning to the building. John White, NFC’s deputy director and executive in charge of human resources, gave his agency high marks for its early response to Katrina.

“Our planning was excellent in that it got us up and running, and then the longer term was where we had to be a bit more creative,” he said. It’s hard to imagine now, but the NFC staff expected to return to headquarters in short order. “We all thought we’d be gone for a couple of days,” White said.

But problems arose as the days and weeks stretched into months. Because returning to New Orleans took longer than expected, a command and control center—an option in the COOP plan—was set up for senior managers in Alexandria, La., a four-hour drive from New Orleans. The center was housed in vacant space at a USDA facility connected to the USDA network. More people on administrative leave had to be called to the alternate work sites, so more workstations were needed than planned. The contract with SunGard only guaranteed equivalent computing facilities for six weeks. But NFC officials found they needed six months to fully restore their computing capabilities, posing a risk of mission failure.

On the fly, Hawk’s team successfully set up essential computing capability at a SunGard “cold” site (suitable space, bring your own equipment) to bridge the gap. NFC is currently proceeding with its pre-existing dual data center project, under which a new government-owned primary computing facility will be built away from New Orleans, and the existing data center in New Orleans will serve as a backup, also owned by the government.

A bigger alligator to wrestle was the prolonged effect on employees. “We didn’t realize the magnitude of [the housing problem] associated with keeping people at an alternate work site for extended periods of time,” Fincher said.

NFC initially found hotel rooms for employees and their families but later sought less costly space. It leased some 210 apartments for six months in Grand Prairie and 80 in Georgia, rented furniture and arranged for General Services Administration shuttle vans to bring employees to work.

Because workers were away from home longer than anticipated, those with more than 30 consecutive days of service at alternate sites were given 40 hours off and paid trips home, something not in the COOP plan. That’s a lesson NFC will incorporate into its plans, White said. Also, “we didn’t anticipate the need for an AWOL policy in our original planning,” he said, referring to the handful of employees who were on paid administrative leave but refused to report to an alternate work site when directed.

Meanwhile, back at HQ. “The one lesson that’s become apparent to me is [that] reconstitution is actually more difficult than setting up the alternate work sites,” White said. Whatever the normal flow of business was, it’s not that way when you get back. It takes effort to refocus your employees and re-establish internal controls.

But before that, you first have to see if you have a building to return to. Administrative staff went back in early October 2005 to assess whether the agency’s buildings could be occupied or needed remediation. Some space was lost, but there was enough left to reconstitute the headquarters. However, Byrne could see that employees could not be brought back to the city without some advance planning for their needs.

“We had no idea, [given] the devastation that we experienced here,” Byrne said. It was difficult to get gas and groceries. Staff would also need to know where to get a flat tire fixed, “which is a big thing around here,” Byrne said. “There are a lot of nails in the road.” They also made a list of banks and doctors’ offices that were open. “We’ve identified these things as lessons learned. We will incorporate them in our plans,” Byrne said.

Again, housing was critical. Officials did not anticipate before the storm that they would need to get trailers from FEMA for employee housing, or find a place to park them. White praised the secretary of Agriculture for having the foresight to get a single point-of-contact at FEMA. Trailers were obtained for 200 USDA families, and some of them still sit on employees’ lots while their homes are being rebuilt.

Lingering difficulties. Employee counseling also remains high on the list. Byrne and many of her colleagues who also lost everything found it very therapeutic to dive into work in the early days after the storm. While life was relatively normal at the alternate work sites, the reality of the catastrophe could no longer be denied once people returned to New Orleans. Counseling is continuing for those who still have family members scattered all over the country or who suffered confusion, lack of concentration or a delayed reaction on their return.

Mail service was also a problem (an “agency interdependency” in COOP parlance). Plans to reroute mail to the alternate sites worked well, but bringing it back to New Orleans was not so easy. The post office there was flooded and is still not open, Bertrand said. Part of NFC’s brief is to pay purchase orders that customers’ vendors send through the mail. Some bills were delayed one or two months because mail service was disrupted. In some instances, a bill was sent a second time and arrived before the first, meaning NFC staff had to spend time scrubbing duplicates from the system. While largely resolved now, the problem was bigger than anticipated and is under review for the next COOP plan. Much will depend on what contingency plans the Postal Service adopts.

Lohfink also sees a disconnect between the mandate that federal agencies have “exclusive use rights” to their COOP backup facilities and what is available on the GSA schedule. Even the last organization in the door of a shared subscription service facility has a claim on the space and resources inside.

NFC is remodeling its COOP plan now to incorporate lessons learned. While FEMA has not evaluated NFC’s COOP performance, Bertrand expects that USDA’s Office of Inspector General and other federal officials will visit in due course and that the lessons will eventually circulate throughout government.

Lohfink sums up his view of COOP planning this way: “You have to commit to doing good continuity planning on a continuous basis, and it has to be integrated into the business units. Plan, drill and learn, and you’ll have positioned yourself to live through such an event.”

As NFC’s experience shows, a perfect plan is not required. “If you can get 80 percent of it down, the other 20 percent becomes very manageable,” Lohfink said.