The intial deadline for agreement between the Unions ansd those mounting a rescue plan was set for Thursday (11 September 2008) - This as I expected passed, because I thought the Unions would play the brinkmanship card until the after the deadline . But hope still appeared to linger as talks conntiued into Friday.
But finally Roberto Colaninno's CAI investor group ran out of patience abandoned talks to buy the state-owned carrier because of union opposition.
So where does this leave Alitalia?
The simple answer should be immediate cessation of operations. But that is of course commercial common sense, not something that can ever be assumed when a company is owned by the Italian Government and de facto controlled by nine Italian Unions.
The labour Minister was quoted as saying: "The situation is worrisome and getting worse" and followed this with "Alitalia won't be able to operate much longer if there aren't new developments". Well there is an understatement to say the least.
If this was a daytime Soap Opera, even the most hardened Soap Addict would consider the plot too surreal.
The Government Appointed Administrator it appears will not start firing staff immediately, which was previously his implied position if talks failed. But Alitalia will run out of funds at the end of the month and then at least in theory there is no choice company will be liquidated
But it is obvious the Unions in a total suspension of any understanding of reality still believe that a better deal will magically appear, Well this is Italy and suppose there might be divine intervention, but if there is it won't be based on any commercial logic.
Other Posts on Alitalia in this blog:
13th January 2009:
Alitalia Reborn - But will it Survive?
06th October 2008:
Alitalia - Is the Opera entering its final act?
29th August 2008:
Alitalia Files for Bankruptcy Protection
29th July 2008:
Alitalia - How long before the life support is switched off?
Showing posts with label Business Continuity. Show all posts
Showing posts with label Business Continuity. Show all posts
Saturday, September 13, 2008
XL Leisure Group Failure; - Here we go again
The XL Leisure Group went into Administration today
(12th September 2008) and the joint administrators decided that immediate cessation of all trading was the only option.
The failure of the United Kingdoms third largest travel group has an immediate effect on customers, staff, companies supplying services to the group (which in turn may affect the staff in these companies) and investors, whether shareholders, or banks and because of the range of XLLG activities these effect will extend globally.
Before background to this failure I think it necessary to list the companies which form XL Leisure Group many of which do not have XL as part of there trading Name and at the same time provide links to useful information to XLLG Customers:
XL Leisure Group - Companies and Holdings
First I will cover companies in the group which should provide British customer protection under the CAA run ATOL Scheme:
The XL Leisure Group also own the following companies offering services to British Customers:
The XL Leisure Group also have Airline Subsidairies in France, Germany and Ireland: Neither the French nor German Airlines have been affected by the Parents failure and continue to trade normally (sources CAA and XL Web Site), but the Irish Airline appeared moribund even before the collaspe:
The Commercial Arms of XL Leisure Group:
Impact on those who have to deal with the Immediate Aftermath
Whilst the media obviously concentrate on the impact on Customers they never consider the efforts of the failed groups employees and agencies staff who have to deal with the immediate and ongoing aftermath of the failure.
British Travel Group and AIrline failures always seem to be decided in the small hours of the morning (Court Line, Laker Airways, International Leisure Group, spring to mind). This means the first non-executive staff who are aware of the situation are the Night Shift Operations and Crewing Staff.
Whilst being now aware they probably soon won't have a job in the near future, they have to get on with the task of getting a sitrep out to all their Customer Representatives at destinations around the world and to crews Nightstopping overseas, in most cases this will have to be initiated by a Phone Call which will slow things down.
As this progresses they will have to commence the notification of Ground Handling Agencies around the Route Network to brace themselves for what is to come (bearing in mind these Ground Handling Agencies will also be owed money).
The CAA will also be among the first notified and they will have to start the work of planning and implementing the rescue of ATOL covered passengers, plus building it Web Site Advisory Notices etc, other parties such as ABTA will also have to be advised and by this time the Media will be 'on the case'.
Bad News travels fast and soon the Operations and Crewing Centres will be bolstered by PR and Enquiries Staff. By the time the first Passengers start to arrive at European Destination Check-in, most Sections in the affected Airline and Travel Companies will be manned, whether these are customer contact, or not.
Whilst these sections workload will dramatically increase as the day wears on at least they are spared directly facing the Passengers.
Even when pre-warned the Ground Handling Agencies will not have much time to call-in extra staff and initially at least they and the Company Representatives will be overwhelmed, especially as the rest of the Ground Handling Agencies Operations have to continue normally.
The situation will get worse if there are a lot of affected flights through the Airport and each flight has Passengers some of whom are protected under ATOL and some who aren't. With the assistance of the Airport Authority it may be possible to at least move these passengers to a special area of the terminal building. But it is very unlikely that the Company Representatives will have sufficient local funds to provide even Light Refreshments.
Representative at the Hotels where customers are staying will now also be in the 'firing line' and with even less direct support than their colleagues at the Airports.
Whilst the first day is always the worst day in reality, for the company and agency staffs it is usually the second day that seems the worst because although the rescue of the stranded has started, those not covered by the ATOL Scheme will becoming more desperate, and although the general situation will improved at the individual level, the customers will be even more frustrated and angry and the staff attempting to deal with the situation will now be overtired.
By now writs and impounding of assets by Agencies and Airport Operators probably will have started and even as the stranded passengers , crews, station engineers if expatriates and later as the Hotels clear, the Hotel Representatives are repatriated the work for the Airport Based staff changes, but goes on as they try to keep Head Office informed and deal with local debts and in some cases income.
The fact that almost all such staff stay on the job in some cases without knowing if they will get paid never ceases to amaze me (the two extremes I know of was a representative on when being told his company had failed took the cash box and company car an ran for it, the other was a Guy who spend six months in Lagos after the Airline Failure and having told Head Office that he could do no more, was sent a thank you message, but no Ticket. British Airways very kindly flew him home, to discover for some to the period he hadn't been paid).
However normally after seven days most of the 'front line' action will be over and most overseas representation will either have been repatriated, or made redundant if locally employed.
The callcentres and operations centres including crewing and engineering will linger on with decreasing coverage for about six months.
The Head Quarters and Operational Back Office Sections will be rapidly wound down, but a small number of staff will be retained for an extended period as the company is liquidated some might even be there for a year or two.
But most staff will be out of work within two months. For the Customers a ruined Holiday for the Staff a difficult and in some cases bleak future.
It must be borne in mind staff in supplier organizations may also in some cases be affected and some businesses owed a lot of money may themselves go under.
Regardless, the Administration of the Liquidation such a large and relatively organization will continue and indeed in 5 years time may still be ongoing.
Reasons for Failure
The reasons quoted are the result of volatile fuel prices, the economic downturn, and were unable to obtain further funding.
All of which is no doubt true, just as it was for the failures in the past of Court Line, Laker Airways and ILG (International Leisure Group).
But just as in these cases there is slightly more too it than that. Like the three mentioned above XL Leisure Group has expanded aggressively in a very short time. At lot of this expansion following the Management Buyout of 51% of the equity, with the previous owners Avion Group of Iceland retaining 49%.
Such rapid expansion requires funding and that funding is obtained on the basis of sustained growth and a favourable interest rate coupled with NO major changes in operating costs. Well two out of the three have not gone XLLG's way!
It appears that XLLG's failure has caught the General Media at least by surprise, but like Court Line, Laker and ILG the signs have been noted in the Financial and Airline and Travel Industry Presss for weeks, with early indicators back at the end of 2007.
For Example:
Travel Weekly - November 2007 - XL Leisure Group sheds 500 jobs (although the reference to Libra Holidays is I consider the most important)
TTG - June 2008 - XL Leisure Group records £24m loss
Times Online - 31 August - XL Leisure Group enters crisis funding talks
Telegraph 31 August 2008 - Hope for XL after lenders spend the weekend in talks
From the 1st September the coverage in Business sections continued to rise, but I think you have got the flavour of the concerns being expressed.
As additional funding did not come to fruition and the main revenue earning season coming to an end (which obviously had not been that impressive as far as the financial instutions were concerned), then Administration was going to be likely. When it came about the Administrators were obviously so concerned they shut down operations immediately.
(12th September 2008) and the joint administrators decided that immediate cessation of all trading was the only option.
The failure of the United Kingdoms third largest travel group has an immediate effect on customers, staff, companies supplying services to the group (which in turn may affect the staff in these companies) and investors, whether shareholders, or banks and because of the range of XLLG activities these effect will extend globally.
Before background to this failure I think it necessary to list the companies which form XL Leisure Group many of which do not have XL as part of there trading Name and at the same time provide links to useful information to XLLG Customers:
XL Leisure Group - Companies and Holdings
First I will cover companies in the group which should provide British customer protection under the CAA run ATOL Scheme:
- The Really Great Holiday Company PLC - ATOL 3827 Trading as:
- Cruise City
- Excel Holidays
- The Florida Skytrain
- Transatlantic Vacations
- Travel City Direct
- Travel City International
- Kosmar Villa Holidays PLC - ATOL 1760
- Freedom Flights Limited - ATOL 5296
- Aspire Holidays Limited - ATOL 6536
The XL Leisure Group also own the following companies offering services to British Customers:
- XL Airways - Flight No's XLA - Whilst NOT Covered by the CAA ATOL Scheme see this page: CAA run ATOL Scheme See also the ABTA Advisory
- Medlife -See the ABTA Advisory
- In addition it has a 20% Stake in Libra Holidays
- It also has very close relationship with two French Tour Operating Companies CrystalTO and Heliades.
The XL Leisure Group also have Airline Subsidairies in France, Germany and Ireland: Neither the French nor German Airlines have been affected by the Parents failure and continue to trade normally (sources CAA and XL Web Site), but the Irish Airline appeared moribund even before the collaspe:
- XL Airways France Wikipedia Entry
- XL Airways Germany Wikipedia Entry
- XL Airways Ireland
The Commercial Arms of XL Leisure Group:
- XL Aviation -XL Aviation is the commercial sales arm of XLLG. It sells charter capacity on flights operated by XL Airways and also buys and sell charter capacity on numerous other airlines.
- XL24 - XL24 is a dealer in "relief" aircraft in the airline-to-airline market, providing sub-chartered aircraft from the XL Airways and other fleets to other airlines who require the extra capacity on short notice.
- Xtra Airways - USA 19%
- Skywest - Australian 5%
Impact on those who have to deal with the Immediate Aftermath
Whilst the media obviously concentrate on the impact on Customers they never consider the efforts of the failed groups employees and agencies staff who have to deal with the immediate and ongoing aftermath of the failure.
British Travel Group and AIrline failures always seem to be decided in the small hours of the morning (Court Line, Laker Airways, International Leisure Group, spring to mind). This means the first non-executive staff who are aware of the situation are the Night Shift Operations and Crewing Staff.
Whilst being now aware they probably soon won't have a job in the near future, they have to get on with the task of getting a sitrep out to all their Customer Representatives at destinations around the world and to crews Nightstopping overseas, in most cases this will have to be initiated by a Phone Call which will slow things down.
As this progresses they will have to commence the notification of Ground Handling Agencies around the Route Network to brace themselves for what is to come (bearing in mind these Ground Handling Agencies will also be owed money).
The CAA will also be among the first notified and they will have to start the work of planning and implementing the rescue of ATOL covered passengers, plus building it Web Site Advisory Notices etc, other parties such as ABTA will also have to be advised and by this time the Media will be 'on the case'.
Bad News travels fast and soon the Operations and Crewing Centres will be bolstered by PR and Enquiries Staff. By the time the first Passengers start to arrive at European Destination Check-in, most Sections in the affected Airline and Travel Companies will be manned, whether these are customer contact, or not.
Whilst these sections workload will dramatically increase as the day wears on at least they are spared directly facing the Passengers.
Even when pre-warned the Ground Handling Agencies will not have much time to call-in extra staff and initially at least they and the Company Representatives will be overwhelmed, especially as the rest of the Ground Handling Agencies Operations have to continue normally.
The situation will get worse if there are a lot of affected flights through the Airport and each flight has Passengers some of whom are protected under ATOL and some who aren't. With the assistance of the Airport Authority it may be possible to at least move these passengers to a special area of the terminal building. But it is very unlikely that the Company Representatives will have sufficient local funds to provide even Light Refreshments.
Representative at the Hotels where customers are staying will now also be in the 'firing line' and with even less direct support than their colleagues at the Airports.
Whilst the first day is always the worst day in reality, for the company and agency staffs it is usually the second day that seems the worst because although the rescue of the stranded has started, those not covered by the ATOL Scheme will becoming more desperate, and although the general situation will improved at the individual level, the customers will be even more frustrated and angry and the staff attempting to deal with the situation will now be overtired.
By now writs and impounding of assets by Agencies and Airport Operators probably will have started and even as the stranded passengers , crews, station engineers if expatriates and later as the Hotels clear, the Hotel Representatives are repatriated the work for the Airport Based staff changes, but goes on as they try to keep Head Office informed and deal with local debts and in some cases income.
The fact that almost all such staff stay on the job in some cases without knowing if they will get paid never ceases to amaze me (the two extremes I know of was a representative on when being told his company had failed took the cash box and company car an ran for it, the other was a Guy who spend six months in Lagos after the Airline Failure and having told Head Office that he could do no more, was sent a thank you message, but no Ticket. British Airways very kindly flew him home, to discover for some to the period he hadn't been paid).
However normally after seven days most of the 'front line' action will be over and most overseas representation will either have been repatriated, or made redundant if locally employed.
The callcentres and operations centres including crewing and engineering will linger on with decreasing coverage for about six months.
The Head Quarters and Operational Back Office Sections will be rapidly wound down, but a small number of staff will be retained for an extended period as the company is liquidated some might even be there for a year or two.
But most staff will be out of work within two months. For the Customers a ruined Holiday for the Staff a difficult and in some cases bleak future.
It must be borne in mind staff in supplier organizations may also in some cases be affected and some businesses owed a lot of money may themselves go under.
Regardless, the Administration of the Liquidation such a large and relatively organization will continue and indeed in 5 years time may still be ongoing.
Reasons for Failure
The reasons quoted are the result of volatile fuel prices, the economic downturn, and were unable to obtain further funding.
All of which is no doubt true, just as it was for the failures in the past of Court Line, Laker Airways and ILG (International Leisure Group).
But just as in these cases there is slightly more too it than that. Like the three mentioned above XL Leisure Group has expanded aggressively in a very short time. At lot of this expansion following the Management Buyout of 51% of the equity, with the previous owners Avion Group of Iceland retaining 49%.
Such rapid expansion requires funding and that funding is obtained on the basis of sustained growth and a favourable interest rate coupled with NO major changes in operating costs. Well two out of the three have not gone XLLG's way!
It appears that XLLG's failure has caught the General Media at least by surprise, but like Court Line, Laker and ILG the signs have been noted in the Financial and Airline and Travel Industry Presss for weeks, with early indicators back at the end of 2007.
For Example:
Travel Weekly - November 2007 - XL Leisure Group sheds 500 jobs (although the reference to Libra Holidays is I consider the most important)
TTG - June 2008 - XL Leisure Group records £24m loss
Times Online - 31 August - XL Leisure Group enters crisis funding talks
Telegraph 31 August 2008 - Hope for XL after lenders spend the weekend in talks
From the 1st September the coverage in Business sections continued to rise, but I think you have got the flavour of the concerns being expressed.
As additional funding did not come to fruition and the main revenue earning season coming to an end (which obviously had not been that impressive as far as the financial instutions were concerned), then Administration was going to be likely. When it came about the Administrators were obviously so concerned they shut down operations immediately.
Wednesday, September 03, 2008
Gustav and Civilian Telecommunications
This post is related to my more general comments on the
Handling of the Gustav Hurricane Emergency
Whilst in the US the Emergency Services, Police National Guard and Coastguard, have dedicated Radio Networks and Landlines to maintain their Communications, Civilians are dependent on their normal service providers.
As my interest in telecommunications is both professional and personal, my attention was quickly drawn to an article by:
Rajani Baburajan on TMC Net entitled Service Providers Step up to the Challenges Presented by Hurricane Gustav.
Most of this post is made up of comments on extracts from her article, which starts:
As for Landline Telephony, both in the US and the UK, there is an almost religious belief we will be able to connect a landline to landline call, but actually this is NOT the case. Whilst landline networks normally have a large contingency volume handling capacity they can under certain circumstance reach this. Especially if there has been damage to local, or enroute network infrastructure. Whilst my limited knowledge of the UK Disaster procedures for Landline Networks is at least 15 Years out of date, I assume the Cascading Priorities still apply with at its most extreme only Government and nationally vital telephone numbers will be allowed to connect calls.
Back to the Gustav Emergency: The two dominant operators in the affect areas are the resurgent AT&T and T-Mobile. Again quoting from Rajani's article:
No it wasn't a mistake as the next few paragraphs relating to AT&T revealed:
Apparently not, as Rajani moves onto the practical efforts being made by T-Mobile inc (part of T-Mobile International AG, the mobile communications subsidiary of Deutsche Telekom AG) to deal with the emergency:
T-Mobile deployed an incident command team to Birmingham Command Center and locations throughout the Southeast and in Texas, to mobilize its team of engineers and technicians in case the hurricane affected networks. The company said it fortified its network switch operations that serve the Gulf Coast. Additionally, it planned to beef up its microwave radio equipment and send out Cell-on-Wheels (COWs) teams to facilitate wireless communications affected areas.
Good practical engineering and command control planning and implementation. No instant PR value, but what I would expect a reputable Telecommunication Provider to do in a Disaster Situation. Well done T-Mobile.
This good corporate citizen stuff is followed by some practical (and what I consider commonsense) advice to evacuees which I hope was made available via PBS, Public Service Advertising and handed out on the Evacuation Buses:
Service providers also requested people to adopt wise communication strategies to help them serve better during such crises. For example, instead of making voice calls to their friends or relatives, providers suggested using text messages, which use fewer network resources. Providers also asked customers to use landline phones as a backup since they do not require electricity to operate.
Finally Rajani covers what the Web (providing access is available) can do to help:
With advanced Web communication technologies such as Google Maps, weather, news, and alerts available on mobile phones, customers can utilize these resources to determine the status of weather conditions and handle emergency situations wisely. Examples of such services include AT&T’sMobiTV, AT&T Mobile TV and My-Cast Weather. Wireless data devices such as Apple’s iPhone and BlackBerry smartphones allow sending of messages to large groups of people simultaneously. However, these services are available under special subscriber plans and limited to a few subscribers.
All good practical stuff.
For More Articles by: Rajani Baburajan<
For Related Articles in this Blog See:
Handling of the Gustav Hurricane Emergency
Whilst in the US the Emergency Services, Police National Guard and Coastguard, have dedicated Radio Networks and Landlines to maintain their Communications, Civilians are dependent on their normal service providers.
As my interest in telecommunications is both professional and personal, my attention was quickly drawn to an article by:
Rajani Baburajan on TMC Net entitled Service Providers Step up to the Challenges Presented by Hurricane Gustav.
Most of this post is made up of comments on extracts from her article, which starts:
The strength and weaknesses of communications technologies are revealed during disasters. People depend on these technologies during emergencies, and during such times offering flawless services is of utmost concern to providers.
The impact of Hurricane Gustav is the latest example of such a situation. Maintaining telecom infrastructure during hurricanes is a difficult task, made more difficult by increased volume of calls from people checking on friends, colleagues and loved ones in the hurricane zone — volume that can result in overloaded networks.
Extremely well put, I wish I could write as clearly and concisely as Rajani. In the UK we tend to forget how easily Mobile Networks can be overloaded especially when a very large number of simultaneous calls are attempted from the same cellular cluster. In the US this can be exacerbated by operator coverage limitations, which in the UK are only experienced by a relatively small percentage of users regardless of operator.As for Landline Telephony, both in the US and the UK, there is an almost religious belief we will be able to connect a landline to landline call, but actually this is NOT the case. Whilst landline networks normally have a large contingency volume handling capacity they can under certain circumstance reach this. Especially if there has been damage to local, or enroute network infrastructure. Whilst my limited knowledge of the UK Disaster procedures for Landline Networks is at least 15 Years out of date, I assume the Cascading Priorities still apply with at its most extreme only Government and nationally vital telephone numbers will be allowed to connect calls.
Back to the Gustav Emergency: The two dominant operators in the affect areas are the resurgent AT&T and T-Mobile. Again quoting from Rajani's article:
In anticipation of power outages and network jams that can occur during hurricanes, major service providers including AT&T and T-Mobile this weekend prepared their teams to deal with hurricane Gustav in a competitive manner.
The last two words which I have highlighted caused me to raise an eyebrow! Dealing with a disaster of potentially massive proportions in a competitive manner, hmm, surely this is a typo, obviously this should be cooperative manner.No it wasn't a mistake as the next few paragraphs relating to AT&T revealed:
AT&T on Sunday announced the availability of more than 2,000 GoPhone devices with $15 airtime to people who were ordered to evacuate. The phones were made available on a first-come first-serve basis to residents who live in areas where mandatory evacuation was ordered.
AT&T distributed the phones through its retail locations in Houston and San Antonio, Texas, where a large number of evacuees from Louisiana and southeast Texas were staying. AT&T said it planned to make the phones available as early as Monday evening.
Chris Penrose, vice president and general manager for AT&T's wireless unit in south Texas, said this initiative will help people connect with family and friends or take care of critical needs in times of crisis.
This is what I consider to be 'a nice gesture', but with more than one eye on the PR value than actually being a good corporate citizen and now surely having got that out of the way, AT&T will tell Rajani what they have been doing to maintain service to customers!Apparently not, as Rajani moves onto the practical efforts being made by T-Mobile inc (part of T-Mobile International AG, the mobile communications subsidiary of Deutsche Telekom AG) to deal with the emergency:
T-Mobile deployed an incident command team to Birmingham Command Center and locations throughout the Southeast and in Texas, to mobilize its team of engineers and technicians in case the hurricane affected networks. The company said it fortified its network switch operations that serve the Gulf Coast. Additionally, it planned to beef up its microwave radio equipment and send out Cell-on-Wheels (COWs) teams to facilitate wireless communications affected areas.
Good practical engineering and command control planning and implementation. No instant PR value, but what I would expect a reputable Telecommunication Provider to do in a Disaster Situation. Well done T-Mobile.
This good corporate citizen stuff is followed by some practical (and what I consider commonsense) advice to evacuees which I hope was made available via PBS, Public Service Advertising and handed out on the Evacuation Buses:
Service providers also requested people to adopt wise communication strategies to help them serve better during such crises. For example, instead of making voice calls to their friends or relatives, providers suggested using text messages, which use fewer network resources. Providers also asked customers to use landline phones as a backup since they do not require electricity to operate.
Finally Rajani covers what the Web (providing access is available) can do to help:
With advanced Web communication technologies such as Google Maps, weather, news, and alerts available on mobile phones, customers can utilize these resources to determine the status of weather conditions and handle emergency situations wisely. Examples of such services include AT&T’sMobiTV, AT&T Mobile TV and My-Cast Weather. Wireless data devices such as Apple’s iPhone and BlackBerry smartphones allow sending of messages to large groups of people simultaneously. However, these services are available under special subscriber plans and limited to a few subscribers.
All good practical stuff.
For More Articles by: Rajani Baburajan<
For Related Articles in this Blog See:
Tuesday, June 03, 2008
Data Center Outage and the Contingency Plan Cannot be Implemented!
In the late afternoon of the 31st May (16:55 CDT) The Planet's H1 Data Center in Houston Texas went off line due a power failure.
Not I hasten to add any old power failure, as the The Planet System and Network Status Forum explained:
Quote:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
Unquote:
For any company the above would be a challenge to both Contingency and Business Continuity handling, but 'The Planet' is not just any business and Data Centre H1 is not just any Data Center. 'The Planet' is a major Web Hosting Company and Data Center H1 was 'inherited' when it merged with Everyones Internet (EV1) which was formerly RackShack. and was the original RackShack Data Center, which means its server farms include many legacy systems.
So this situation was not just an internal problem for 'The Planet' but a very serious Business Continuity Problem for the 7,500 Customers with a total of 9000 Servers depending on this Data Centre..
Could things get any worse? - Oh Yes it can!
The Building is evacuated, the Fire Department are in charge and even with the fire out they will not allow implementation of the Back-up Power Plan for safety reasons and finally they won't let staff back into the Building.
The company did what it could in the first few Hours after the explosion:
From then on the company posted updates either to a 1 hour base schedule, or more often whenever there was 'breaking news,.even if the news was bad!
For many updates there was little to report, but the temptation to make rash promises was resisted.
This policy of NOT raising Customer expectations too high was proved the correct course when it was discovered the damage to power cabling was even greater than suspected. By lateral thinking, plus the use of additional generators and 'blood, sweat and tears' step by step recovery moved forward.
You can look at Bulletins to Customers - Current to Oldest here:
The Planet H1 Maintenance Updates
or in Oldest to Current here :
The Planet System & Network Forum
Whilst 56 Hours later recovery was not fully complete the CEO felt able to issue this audio statement: Doug Erwin, chairman and CEO of The Planet
Fifty Six hours without service is for a company dependent on internet sales a disaster and a lot of customers are obviously angry andconcerned, worried sick about their business, but 56 Hours is a lot better than some original estimates of 5 Days.
Whilst The Planet is going to have to deal with the aftermath of this incident in terms of Customer claims and possibly litigation, I believe that Operationally and from a Customer Advisory Viewpoint the Company did everything it should and could.
It is rightly proud of the 24 Hour a day efforts from its staff many brought in from Dallas and the speed and effectiveness of Contractor and Supplier teams also working 24 Hours a day.
Whilst Operationally I admire the company's management of the incident, I am concerned that there maybe some questions over Contingency Planning and Risk Management including Safety Inspection Policies.
Whilst Web Site and Server Hosting is lucrative it is also extremely competitive and therefore the possibility of mass migration of Customers to a fall back site, or dispersing them across other data centres was never part of the equation for handling this incident and I suspect would not be an economic proposition (although offering it as a premium option could possibly make sense).
However I do think that during the 'lessons learned studies' some consideration is given to looking at how all the 'single point of failure' situations (not just electrical) can be mitigated, not only in H1 but across other Company Data Centers .
I also hope that the legal eagles allow The Planet to share the results of the investigations and studies industry wide as this is NOT about a marketing war, it is about safety and customer confidence that its Web Site, which for many is the core of the business is well protected.
Initially some 'The Planet' customers may feel moving to another host is the answer, but would they be able to handle a similar problem any better than 'The Planet' and would they the Customer be willing to pay the premium required for mirroring across Data Centers to provide guaranteed fallback! In most cases probably not
Throughout the industry worldwide today, there is almost certainly the feeling that 'thank God it wasn't us' and then the depressing thought: 'this time'.
There is probably one group of people seeing a 'silver lining; in all this: Insurers, who I suspect are already drafting new improved Business Continuity Insurance for Internet Dependent SME Businesses.
The Idle Man.
.
Not I hasten to add any old power failure, as the The Planet System and Network Status Forum explained:
Quote:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
Unquote:
For any company the above would be a challenge to both Contingency and Business Continuity handling, but 'The Planet' is not just any business and Data Centre H1 is not just any Data Center. 'The Planet' is a major Web Hosting Company and Data Center H1 was 'inherited' when it merged with Everyones Internet (EV1) which was formerly RackShack. and was the original RackShack Data Center, which means its server farms include many legacy systems.
So this situation was not just an internal problem for 'The Planet' but a very serious Business Continuity Problem for the 7,500 Customers with a total of 9000 Servers depending on this Data Centre..
Could things get any worse? - Oh Yes it can!
The Building is evacuated, the Fire Department are in charge and even with the fire out they will not allow implementation of the Back-up Power Plan for safety reasons and finally they won't let staff back into the Building.
The company did what it could in the first few Hours after the explosion:
- It started to alert Internal, Contractor and Supplier teams and began moving them to site
- It start to alert Customers via Forum and Automated Voice Service and through one Customer Portal (the other was hosted in the affected center and was down).
From then on the company posted updates either to a 1 hour base schedule, or more often whenever there was 'breaking news,.even if the news was bad!
For many updates there was little to report, but the temptation to make rash promises was resisted.
This policy of NOT raising Customer expectations too high was proved the correct course when it was discovered the damage to power cabling was even greater than suspected. By lateral thinking, plus the use of additional generators and 'blood, sweat and tears' step by step recovery moved forward.
You can look at Bulletins to Customers - Current to Oldest here:
The Planet H1 Maintenance Updates
or in Oldest to Current here :
The Planet System & Network Forum
Whilst 56 Hours later recovery was not fully complete the CEO felt able to issue this audio statement: Doug Erwin, chairman and CEO of The Planet
Fifty Six hours without service is for a company dependent on internet sales a disaster and a lot of customers are obviously angry and
Whilst The Planet is going to have to deal with the aftermath of this incident in terms of Customer claims and possibly litigation, I believe that Operationally and from a Customer Advisory Viewpoint the Company did everything it should and could.
It is rightly proud of the 24 Hour a day efforts from its staff many brought in from Dallas and the speed and effectiveness of Contractor and Supplier teams also working 24 Hours a day.
Whilst Operationally I admire the company's management of the incident, I am concerned that there maybe some questions over Contingency Planning and Risk Management including Safety Inspection Policies.
Whilst Web Site and Server Hosting is lucrative it is also extremely competitive and therefore the possibility of mass migration of Customers to a fall back site, or dispersing them across other data centres was never part of the equation for handling this incident and I suspect would not be an economic proposition (although offering it as a premium option could possibly make sense).
However I do think that during the 'lessons learned studies' some consideration is given to looking at how all the 'single point of failure' situations (not just electrical) can be mitigated, not only in H1 but across other Company Data Centers .
I also hope that the legal eagles allow The Planet to share the results of the investigations and studies industry wide as this is NOT about a marketing war, it is about safety and customer confidence that its Web Site, which for many is the core of the business is well protected.
Initially some 'The Planet' customers may feel moving to another host is the answer, but would they be able to handle a similar problem any better than 'The Planet' and would they the Customer be willing to pay the premium required for mirroring across Data Centers to provide guaranteed fallback! In most cases probably not
Throughout the industry worldwide today, there is almost certainly the feeling that 'thank God it wasn't us' and then the depressing thought: 'this time'.
There is probably one group of people seeing a 'silver lining; in all this: Insurers, who I suspect are already drafting new improved Business Continuity Insurance for Internet Dependent SME Businesses.
The Idle Man.
.
Subscribe to:
Posts (Atom)