Business Continuity Management (5): Testing & Exercising

1. Looks at your organisation in a way that makes the recovery requirements and priorities clear.
2. Assesses the threats to the organisation that need protecting against (incidents that are serious enough to completely stop your organisation from functioning, usually referred to as Business Continuity Incidents) and refines them into a set of common response triggers.
3. Provides a structured approach to developing an appropriate business continuity strategy and preparing business continuity plans (BCP’s) that will help the organisation to respond to a Business Continuity Incident.

Embedding BCM in the Organisation's Culture
(click to see enlarged view).

The best plans in the world though are of limited use until they have been thoroughly tested. Most Business Continuity professionals tend to refer to the testing process as exercising so as to emphasize that failure is not a negative result but a step towards a better outcome.  Analysing the results of exercising identifies what parts of the plan work and where further work is required.
 
Most organisations have historically exercised by, once a year, simulating the loss of a building on a weekend, relocating staff to their recovery site and resurrecting their systems from backups stored offsite. It would usually be considered a success if the key systems were restored within a few hours and users, firstly turned up and secondly, could log in. The outcome would invariably be a list of issues for IT to address and a few embarrassments for users who couldn’t remember their password

To their credit most organisations, particularly in the Financial Services sector have moved forwards from the above.  Users are now generally expected to show that they could carry out a day’s work in the recovery location, not just doing data input but checking all their key contact details as well. Increasingly, larger organisations are scheduling to spend at least one or two working days a quarter at their recovery site carrying on with business as usual.

The recovery of the IT systems and phones is now more or less taken for granted. This is partly due to improvements through practice and partly due to the advances in technology, particularly in the area of Storage Area Networks (SAN’s), that enable most types of database to be replicated simply to backup data centres.  This gives the capability of hot backup data centres that can be switched automatically to production and thus remove much of the manually intensive and time consuming IT tasks.

However the more sophisticated the technology and the more that can be realistically achieved in an exercise, the more that planning is key!

It usually makes sense to run the exercise itself as a project, including all of the logistics and preparation work as phases leading up to the exercise itself and completing with a review on its successes and failings.  Expecting staff to just turn up and organize themselves is not usually very useful to anyone.

Example for simple plan for the project
(click for enlarged view)

There are many factors to agree at the start of the project. The first and most important is to set objectives for what you are trying to achieve through the Exercise.  Are you attempting to prove that all aspects of the business continuity plans are workable, or are you testing the plans work for a particular building or business line?

Clear objectives will help you set the scope for what is included and what is to be excluded.  A good check is to ask yourself if your organisation is ready for the level of the planned exercise or if you should be building up to it through a series of less ambitious tests.  An exercise where half the users spend their weekend sitting around waiting for the other half to complete their work first or waiting for a system to be made available, usually backfires and it becomes ever harder to convince users to take it seriously.

Just as the preparation work needs planning, so does the exercise itself.   Plan out the time required, determine the dependencies and when people are required.  Then communicate it to all those involved. Encourage feedback on whether anything is missing and time allowances are realistic.

Always allow a higher level of contingency than you think you will need, particularly as you may need to restore systems back to your production environment at the end of the exercise.

Example of plan for the exercise day
(click to see enlarged view)

In order to manage the process during the exercise, establish a Control and Monitoring team even if it is just yourself. Prepare a briefing note for all the participants so they know what and where to communicate.  Insist they keep the Control and Monitoring team updated with progress problems and issues.  Keep a written log of all this communication for later analysis.

A good way of gathering feedback on how the exercise is running is to appoint an Observer who has no other responsibilities but to watch what is going on and report back on the way users, systems staff and the Control and Monitoring team are working; ideally picking up inefficiencies and miscommunications as well as things that seem to be working well.

On the day of the exercise itself, make sure that all the participants are accounted for as they arrive at your recovery site(s).  Distribute briefing packs so that everyone knows what is expected of them, where to go, how to inform the Control and Monitoring team of progress and issues.  Food is a good motivator, ensure that everyone knows where and when it will be available and if possible tie it to completion of tasks.

When staff have finished their activities try not to let them slip away but interview them to get their initial feedback on what went well, what needs improving and so on. Some prefer a structured form for self completion others a more formal interview approach.  As long as you are gaining useful feedback the mechanism doesn’t matter much.

The post project review is when all of this feedback is analyzed, together with the Observer’s comments.  The post project review should be carried out shortly after the exercise is completed, usually within a week or so. The output is usually a short report for senior management that lists the objectives and achievements of the exercise, together with the participants, the issues faced, lessons learned and the next steps.  The next steps should detail what actions will be taken as a result of the test, whether they require system changes, updates on the business continuity plans or even the underlying strategies. The report itself, should kept available for the next exercise otherwise all those valuable lessons will be lost and the next exercise will just repeat the mistakes of the past.