Resiliency, Not Just Recovery

SMG Information Security
October 28, 2009 | COMMENTS

What happens when there's a glitch in the payments platform? Or an Internet banking website goes down over a weekend? Or hundreds of ATMs are inaccessible for a day? Or there's an outage in a high-traffic branch office? That kind of down time is virtually incalculable—giving new meaning to business continuity.

Today, customers demand uninterrupted access to ATMs and online banking. Employees and partners need immediate availability of core business applications, putting tremendous pressure on institutions to ensure the availability of data and business processes—24/7, 365 days a year.

An institution's viability is at stake, says Tom Wills, a senior security analyst with Javelin Strategy & Research, a California-based provider of independent research and strategic direction for financial services. “The consequences are some combination of reputational damage, lost customers and revenue, and lower share prices.” The stuff institutions can ill afford to ignore today.

What's more, whether in the U.S. , Europe or Asia , regulators are increasingly mandating business continuity requirements. Now, financial institutions everywhere are beholden to a myriad of complex regulations—from those specific to business continuity and emergency preparedness to those stipulating security and privacy thresholds.

“The challenge is that data is growing exponentially, application requirements are increasing, and all the while budgets remain flat,” says Daniel Koopman, business continuity and resilience practice manager, UK Services at Symantec Corporation, a leader in security, storage, and systems management solutions. “Banks typically either over-protect their applications, costing the company unnecessary budget, or worse, they under-protect leaving the business vulnerable.”

But with proper business continuity planning and best practices, institutions can minimize risk, maximize uptime and reduce costs to survive and thrive in these competitive times.

Continuity Planning: Everybody's Doing It

Continuity planning has evolved greatly. It had to. “Operational risk is more diverse today,” says Rolf von Roessing, senior external advisor at KPMG Germany, one of the largest professional services firms in the world. It's not just about credit posture and liquidity; rather institutions have to look at facilities management, communications, IT infrastructure, security and more. That's a lot to keep going, and without effective business continuity planning in place, he adds, institutions can sustain significant damage

And obviously, regulations are forcing institutions to address various aspects of business and IT continuity. For example, the international accord Basel II has direct business continuity requirements. There are also regulations specific to the financial sector, including NASD 3510 and NYSE 446, which call for business continuity planning. And there are even obligations to relocate secondary sites due to specific city requirements as in London or Paris . Not to mention, legislation with indirect requirements around business continuity, such as Sarbanes Oxley in the U.S.

Thus, business continuity and operational resiliency planning has become part of life for most large institutions within the financial services community, and very much on the radar for smaller firms. “There are many regional and international variations,” Roessing notes. In some countries it's still a project-based exercise, while in others it's more of a holistic initiative.

Interestingly, even organizational roles are evolving with the trend. Many institutions are introducing a new title for the head of IT Risk Management. And, in some cases, the classic role of CISO is starting to include business continuity and risk management. Or at the very least, the two functions are working more closely. In fact, according to a recent survey by Symantec, business continuity and operational resiliency is now a CXO discussion, with 70 percent of respondents reporting that their disaster recovery committees include the CIO, CTO, or IT director.

On whole, everyone is pretty much doing something in the financial services community. That “something” varies from institution to institution. In the past, everyone focused on disaster recovery and emergency preparedness, primarily planning for some natural or man-made catastrophic event. Today, business continuity encompasses much more.

Business continuity is essentially all the activity required to ensure that critical business functions are available to employees, customers, supply chains, regulators, and other entities and that recovery occurs in the event of some disruption in operations—whether that's a fire at corporate or brown-out in a branch office. Going one step further, high availability is the system design and implementation protocol that ensures a certain degree of operational continuity to minimize downtime for users and processes. In the end, it all comes down to operational resiliency, or the institution's ability to adapt to the risk that affects its daily operations and move forward with the business at hand.

Operational Resiliency in the Real World

When it comes to business continuity today, resiliency is the key, and institutions of all sizes face similar challenges. Complexities abound, uninterrupted operations are a must, and cost cutting is king. So resiliency is no easy feat. That's certainly the case for one Luxembourg-based financial services firm.

The company offers worldwide services to the tune of 250,000 transactions per day. Any outage could threaten its business, let alone the world's securities markets. Determined to deliver uninterrupted transactions, the firm was challenged with a highly complex heterogeneous environment—borne of mergers and acquisitions—that introduced disparate, scattered applications. Significant manageability and integration issues ran rampant, resulting in gaps in its continuity planning and budget concerns. So the firm standardized its infrastructure with datacenter availability and storage management solutions from Symantec.

Indeed, the firm has been able to achieve the resiliency it sought—operating without disruption. But perhaps even more important, the standardization implemented across platforms and applications allowed the firm to fully optimize performance and high availability; and that resulted in cost savings in the neighborhood of US$700,000.

Technology as the Gating Factor

In today's information age, technology has become a key piece of the puzzle. So business continuity is further refined to encompass information technology and communications services. That means managing the risk associated with all its components—from the business applications and phone systems to networks and storage. Recovery from a branch server crash, an ATM malfunction, a denial of service in online banking, a storm-ravaged datacenter or worse must be achieved in haste with little to no interruption in technology service. There's simply too much at stake.

So when it comes to resiliency, institutions must assess and rank the criticality of systems, processes and applications for risk potential and impacts, warns Symantec's Koopman. “They need to consider the potential impact on the institution's business and overall sustainability, not just the likelihood that something bad could happen.” And despite the typical siloed approach to management of the business and technology disciplines, they need to do it from both perspectives.

IT may not be fully tuned into what's at stake—the financial and reputational risk—but business managers should be thinking of nothing else. Because, now more than ever, acquiring and retaining customers (and their deposits) is their top priority. And technology could be the only thing standing between the institution and its ability to sustain stability and the perception of strength in an increasingly volatile marketplace. That puts continuity planning top of mind.

Technology Hurdles Abound

But sustaining that stability can be a huge challenge today, as Koopman explains that he's encountered a vast range of complex technology issues that can hamper resiliency efforts.

For starters, there is the ever-growing data explosion and all it entails. Essential to business operations and customer relationships, data is growing at incredible speeds. And as it grows, it's becoming increasingly difficult and costly to manage, secure and recover.

Similarly, applications have run amuck. Many institutions are literally consolidating from more than 1000 applications. Some are integrating their channel applications, while others are updating their core banking with standard packaged applications. And environments that were once just UNIX and mainframe have morphed into Internet-server and Windows-hosted scenarios.

Of course, 60 percent of those applications, a recent Symantec survey reveals, are deemed mission-critical. Some of those applications include everything from security and accounting to online banking. And the dependencies that exist between seemingly disparate applications and resources make high availability and recovery operations even more important.

At the same time, infrastructures are sprawling. With online services, branch office expansions, and global reach, manageability is complicated and requires centralization. And fast-moving M&A's are bringing disparate—yet interdependent—infrastructures together to wreck havoc on resiliency.

Worse still, portions of that infrastructure are aging. Yet overhauls of legacy equipment, such as ATM networks, are often cost- and time-prohibitive. And where infrastructure overhauls are actually underway, new complexities and risks are coming to bear, as with server and desktop virtualization initiatives. In fact, Symantec research says over one-third of virtualized environments are not even backed-up at this time.

The resulting environment is a veritable hodge-podge of silos, and the underpinning is a delicate matrix of dependencies that needs to be detailed in order to understand the continuity impact that any one disruption can have on the business.

And all the while, cost pressures and mandates to “do more with less” are a constant in financial services, as with most other industries. So institutions need to spend smart, utilize existing technology and eliminate wasteful expenditures to make sure flat budgets fit their growing requirements.

Maintaining Operations All Day, Every Day

When it's time to take on the continuity challenge, “regulation is the biggest driver for risk management spending in general—which is unfortunate because it engenders a mentality of compliance with the letter of the law, as opposed to basing programs on the results of ongoing risk assessments that account for the fast-changing threat landscape,” says Javelin's Wills.

Risk management should be a business mandate spurred by competitive pressures and revenue goals. “In these times when customers expect instant gratification, a disruption of, say, online banking services for even a few hours can have a very significant negative impact on the institution's bottom line,” reports Wills. “If the website is down or overloaded some customers will stop using online banking or use it less in the future, and some will close their accounts.”

Institutions really need to take a customer-centric view of business continuity. So day-to-day operations and high availability should be the ultimate focus of continuity efforts. On the technical side, that means preparing for system patches, server failures, email outages and online glitches among other disruptive events, planned or unplanned. Consider a datacenter flood caused by a cooling system water leak for a network failure during a branch relocation effort. More damaging, a UK banking giant experienced an ATM failure during which customers were unable to withdraw funds from five hundred machines. The disruption had a huge impact on business, but it turned out to be just a storage hardware issue.

Traumatic events garner a lot of attention, too, for the magnitude of damage that could come in their unlikely wake. Natural disasters like earthquakes, hurricanes, typhoons and tsunamis can be devastating. While most recently, the threat of the H1N1 flu pandemic has global institutions evaluating their readiness.

And specific, large-scale events have to be taken into consideration. One example of historical relevance is Y2K. “Formal business continuity planning was triggered by Y2K,” KPMG's Roessing claims, changing the way institutions look at their operational resiliency from an IT standpoint. More sinister events have emerged as historical milestones, including the 9/11 terrorist attacks in 2001 and a few years later the 7/7 London bombings, both of which crippled the respective banking communities and changed the way the finance world looks at business continuity.

Whatever the motivation behind recovery plans, action is non-negotiable in these technologically and economically turbulent times.

Best Efforts toward Good Practice

“There's certainly a degree of encouragement needed to get banks to take business continuity seriously in some parts of the world,” says Lyndon Bird, FBCI and global technical director for the Business Continuity Institute (BCI), a leading organization in the development of best practice in business continuity management and contributor to relevant legislation and standards based in Caversham, United Kingdom. These regulations aren't always international or prescriptive, “but there are still various routes which banks might choose to follow.”

Several standards initiatives have been introduced for the express purpose of helping organizations (financial institutions included) to develop and implement business and IT continuity plans. Only in the EMEA, the U.S. , and Australia , he says, are those standards fairly mature. “Traditional business continuity planning prepares for things that have a low probability of happening, but that have big impact,” notes Bird. But mature business continuity is a more holistic approach which plans for catastrophes and the resilience of day-to-day operations against more likely disruptions—meaning “what you need to keep the business running, rather than what you need to respond to a major incident.”

In the UK , that's what drives guidance from the British Standards Institution (BSI), he says. BS 25999 is its business continuity standard, providing instruction in regard to processes, principles and, most importantly, requirements for implementing, operating and improving business continuity. Certification is available from organizations accredited by the United Kingdom Accreditation Service (UKAS). Similarly, BS 25777 is BSI's standard for information and communications continuity, outlining specifications for planning and implementing an ICT continuity strategy.

Other countries have their own standards initiatives, too. In Australia , Handbook 229-2006 deals with operational and technical risk management; while ISO 27002 in the U.S. provides specifications for information security management.

Institutions can also tap other resources for best practices. BCI has its Good Practice Guidelines, which promote business continuity “based on the same fundamental principles, but in much more detail in terms of practical implementation by organizations.” The guidelines cover the process of identifying potential risks and impacts and provide the framework for building resiliency into the business with the right policy and program management. Other key elements include understanding the organization, determining continuity strategies, developing and implementing response plans, exercising and monitoring plans and making business continuity part of the organizational culture.

The U.S. even has a voluntary initiative to certify business continuity efforts, called Title IX of the 9/11 Commission Act of 2007, which draws on those existing standards—including BS 25999, HB 229-2006, and ISO 27002, among others—for the development and implementation of a business continuity program. Less-formal guidance comes from the Federal Financial Institutions Examination Council (FFIEC) in the form of an examination handbook for “evaluating financial institution and service provider risk management processes to ensure the availability of critical financial services;” and the Financial Services Technology Consortium (FSTC) has its Resiliency Engineering Framework of good practices in managing operational risk.

Mind the Gaps and Overlaps

Even with all that guidance, financial institutions still have to put their own houses in order to be successful. For example, there tends to be a wide disconnect between IT and business within many institutions. No one will argue that technology disruptions can have a huge impact on business applications and processes. The environment and interdependencies are always changing and institutions don't always understand the full impact of it all. And that's what Koopman says is the weakest link today. “They may know there's a problem, but that doesn't mean they're actually talking and running scenarios together.”

Unfortunately, that disconnect can result in gaps in continuity. Such as an insecure notification mechanism regarding a service outage that's implemented by a business line and inadvertently exposes the institution to risk and unplanned costs for damage control. Institutions may even experience overlaps in planning. For example, duplicate backup and recovery procedures implemented by IT and the business line for the same application—a cost that could have been avoided altogether.

“These gaps and overlaps produce increased risk exposure and unbudgeted costs,” says Koopman. “Institutions don't have the money to just throw at the problem so they simply have to act more strategically.”

That said, being smart about operational resiliency is not just about abiding by the letter of the law. It's an opportunity to save money and cut or avoid costs altogether. According to Symantec research, the average cost for each downtime incident worldwide is US$287,600. In North America alone, the median cost for financial institutions is US$650,000. And since 93 percent of respondent organizations have had to execute on their disaster recovery plans, a little strategic thinking can go a very long way.

With so much that can go wrong and with so much as stake, smart financial institutions are indeed getting very strategic about recovery. At a time when stability and 24/7 availability has become their key competitive differentiator, what other choice is there?

This white paper by SMG Information Security Media Group may be accessed via CUInfoSecurity. Reprinted with permission.