June 3, 2005  


The Payroll Must Go On!

By Ron Barrett

What do you do when you have ~10,000 square feet of computers with no electrical power, lighting or cooling and the payroll is due in one day?

Add to that scenario the fact that the whole site is down with an estimated up time after the deadline has passed. Well, that is exactly the scenario we had at the Lab on Wednesday, May 18.

SLAC electricians convert 480v from diesel generator to 120/208v to be used for power in Building 50 to bring network and servers back temporarily. Shown left to right: Brandon Mai, Fikrit Alisic, Francisco Castillo, Ricardo Delgado, George Quilon and Richard Altieri.
(Photo by Ron Barrett)

The Solution

Decide how many systems you actually need to get the payroll run and sent to the banks. Then, call in electrical planning to figure out how to safely bring in enough temporary generator power to make the systems work.

If you received your paycheck on time following the surprise power outage at SLAC, you can thank the electricians and the planners for their dedicated efforts.

Following an evaluation of safety and electrical needs, 13 electricians and other essential professionals worked many long hours into the night on Wednesday to safely restore electrical power via generators, temporary transformers and distribution cables to payroll dependent services.

Electrician Team:
Fikret Alisic (CEF)
Richard Altieri (CEF)
Richard Boushey (CEF)
Francisco Castillo (CEF)
Brad Catania (CEF)
Raymond Cuadrado (CEF)
Gabriel Diaz (CEF)
Ricardo Delgado (CEF)
John Healy (CEF)
Brandon Mai (CEF)
Ray Manuel (CEF)
Doug Mitchell (CEF)
George Quilon (CEF)
Satwinder Singh (CEF)
Cliff Whitton (CEF)

Dino Amador (CEF)
Perry Anthony (CEF)
Phil Cutino (CEF)
James Kang (CEF)
Liam Robinson (CEF)
Tom Sherry (CEF)
John Weisend (CEF)
Jim Winstead (CEF)

Once the power was restored for these machines additional staff members worked through the night to bring the services on-line and run the payroll system. The payroll system finished running at about 1:30 a.m. and data was sent out at 8:30 a.m. on Thursday morning before the deadline.

The folks working directly with payroll software were Ernest Denys and Jeanne Pierre. A very thorough understanding of the network and everything else by Gary Burhmaster made it possible for the systems to talk together. Commitment to getting it done and creative solutions were supplied by John Weisskopf.

From Friday night through Sunday most computer systems were brought back to life by staff working long hours, carefully documenting and fixing the requisite hardware.

SCS staff included—but was not limited to—the following people:

Computer Services:
Neal Adams (SCS)
Yemi Adesanya (SCS)
Karl Amrhein (SCS)
Ron Barrett (SCS)
John Bartelt (SCS)
Jon Bergman (SCS)
Chuck Boeheim (SCS)
Matt Campbell (SCS)
Bob Cowles (SCS)
Renata Dart (SCS)
Teresa Downey (SCS)
Shirley Gruber (SCS)
Mike Hogaboom (SCS)
Ricardo Kau (SCS)
Randy Melen (SCS)
Len Moss (SCS)
Ian MacGregor (SCS)
Richard Mount (SCS)
George Maclin (SCS)
Lance Nakata (SCS)
Sean Roberts (SCS)
Bill Weeks (SCS)
Lois White (SCS)
Wei Yang (SCS)

Bonus Quiz

Question: Can you guess how many total systems on the second floor in Building 50 were restored following the payroll system?

Answer: 3,211 servers


The Stanford Linear Accelerator Center is managed by Stanford University for the US Department of Energy

Last update Friday June 03, 2005 by Topher White