Oxygen IT breathes life into Christchurch backups

It was by sheer coincidence that Stace Hema and his team were out working on other locations at 12:51pm when the quake struck, instead of the CTV building where over 100 people lost their lives.

"We were in the CTV building that morning because a client was looking to move in and we were there scoping it out for them to see what they needed to setup network infrastructure,” recalls Hema.

"Our cellphones had been going off all that morning so I sent our guys out to clients. All but one of our guys were out of the CBD when the earthquake hit."

Hema is the managing director of backup and infrastructure company Oxygen IT, which last February found itself in the middle of a nightmarish situation.

Oxygen's offices, which also housed the datacentre used by its clients, were severely damaged by the 6.3 magnitude quake on 22 February. The Cashel Street location was a short distance away from the ill-fated CTV building.

Fortunately for the company, it had mirrored its client’s data to a facility outside of the CBD to provide redundancy, only weeks before the earthquake. Hema says none of his datacentre clients were affected by major outages, but admits things could have been much worse.

Hema and his team of five staff and six contractors were all unharmed, and once their families had been checked on, Hema says they went about recovering client systems.

He says he knew most of his clients were unaffected because of server monitoring software to which he had access on his phone.

"We have Linux-based server monitoring software installed on all our clients' machines. I could tell that all our clients who backed up to online were fine, but I needed to check on the health of some of the clients who only kept local backups," says Hema.

He says while his clients had mostly moved to online backups, some were still lagging behind on the uptake.

"Even now a few of my customers need a good kick in the ass to get them going," says Stace.

The only way to do check on these backups, which in some cases represented the livelihoods for several businesses, was to retrieve the backup disks at each premises.

One-and-a half hours later he was back in the CBD.

“It was eerie going back into the city, there was smoke and dust still hanging in the air. There were thousands and thousands of people heading away, and only a handful heading towards,” says Hema.

Hema says he gained access to the Red Zone by getting a pass from a contact in the telco industry. Shortly after, he was in the cordoned off area of the CBD retrieving recovery disks for various clients.

Word got out that Hema was in the Red Zone, and it was not long until he was getting requests from IT partners and friends to retrieve their backup devices too.

"There was one company who managed to do a full backup to their datacentre, but the files were corrupt," says Hema.

"I had to run up four storeys on a leaning building to get this USB disk, which I wouldn't have needed to if they had checked to make sure their backups were recoverable."

Hema says the disaster recovery plan Oxygen set in place after the September earth quakes were key to their business continuing operations.

This includes server monitoring solutions, datacentre redundancy plans, and even survival kits in the trunks of each of his staff.

Oxygen now has a second datacentre in Palmerston North. Staff there have been trained to recognise if emergency situations arise from Christchurch and react appropriately. Hema says Oxygen is investing in UHF walkie talkies to communicate with engineers and the Palmerston North datacentre in case communication channels are affected in a future disaster.

His take away advice for IT workers is simple: Stick to your disaster recovery plan and check your backups.

"Backups can retrieve files sure, but you need to make sure the images are bootable," says Hema.

"It's not much use having all that data if the applications aren't working."

To comment on this article, please email the editor.
Sign up to receive CIO newsletters.
Follow CIO on
Twitter @cio_nz
Facebook
LinkedIn

Click here to subscribe to CIO.


Send news tips and comments to divina@cio.co.nz