SLAC Computer Security		Search SLAC
SLAC Home | Computing Home | Computing Outages | Help

Cyber Security Awareness Month

These tips are part of a month long effort to distribute useful computer security information to the SLAC community.

Day 11 - File System Backups

Notes from the SANS Internet Storm Center folks: Backups are one of the staples of the operations teams that is oft overlooked and even more often under rated.  Backup and recovery are essential to the organizations IT health.  That very statement should resound with our audience with a big DUH!  But through the years of consulting, I have been shocked at how little attention and operational practice is put towards proper backup and recovery. Tip #1: Back it up or lose it for ever. We have all been there and done that when it comes to losing files on a system when it dies, because we were not as diligent as we should have been in regards to backups. Tip #2: Test your recovery procedures at minimum 1 time per quarter As a consultant, I have walked into a data center and asked if all the systems are properly backed up.  When the client says yes, I ask if they mind if we test the recovery procedure (as part of the scope of engagement of course).  They often get very squirrelly at that point.  Point is, you have to know that recovery is going to work, because you never know when you are going to need it. Many of us in the industry were pleasantly surprised when more financial data was not lost during 9/11.  Financial organizations are required to have proper offsite backup/replication processes in place, and what do you know... they did! Tip #3: Ensure that your backup software (agent and server) are properly patched Why should the exploit writers go after each individual server when they can go after the backup server and the storage device.  Tip #4: Protect your backup tapes On many occasions while visiting client data centers I encounter this bizarre situation: Biometric cages to get access to systems, armed guards, firewalls, laser beams (well, no laser beams but it sounded cool) all protecting client systems.  Then, on the loading dock of the data center, a box with tapes labeled: For Iron Mountain or similar. Just think of what happened back in 2002: Backup tapes stolen from group digitizing military medical records. Backup tapes stolen from Japanese company van that was creating national ID cards. Backup tapes stolen from a military shipment going through international customs. All of these incidents happened within 2 months of each other.  Were they related... who the heck knows.  Point is, protect your backup tapes as if they were the actual systems they came from. Tip #5: Use backup diffs to find rootkit file installations. Infosec Guru Randy Marchany reminded me of an incident that happened at Virginia tech a couple of years back.  A large number of Solaris systems were compromised and they were not sure how or what files had changed.  Their brilliant network backup expert, Judy Albert ran out of the meeting and came back minutes later with a precise list of all the files that had changed on the system.  Since they were diligent about backups, they had a "pristine" state snapshot from the day before the incident and could diff the results of the current backup to see which files had changed.  Great thinking!