The Sunshine State is the home of oranges, grapefruits, sun, sand and surf.  It is also the home of hurricanes - and our firm's data centers.  Prior to 2004, our firm's long-term IT plans called for server consolidation at our Orlando data center and expansion of a backup data center in our Miami office.  Our focus was on building large-scale fault-tolerant systems with some disaster-recovery capabilities.  Then, in 2004 and 2005, Charlie, Francis, Ivan, Jeanne, Dennis, Katrina, Rita and Wilma came to visit.

Because of hurricanes, our firm experienced 64 cumulative days of office closures in 2004 and 2005.  Although our business remained robust throughout these office closures, it was not without a great deal of work by our IT folks and dedication from our legal professionals and support staff.  Office closures in Orlando were particularly disruptive since the "hub" of our computing network was there.  After three hurricanes crossed central Florida in 2004, we decided to focus more on hardening our facilities infrastructure as well as building centralized fault-tolerant systems.  Our initial infrastructure investigations included:

• Backup diesel power generation
• Backup battery power capacity
• Backup WAN capacity
• Redundant air conditioning capacity
• Dry fire-suppression systems

Because of the high cost of these systems, we were asked to investigate other, less capital-intensive alternatives.  Having used hosted colocation facilities in the heyday of the "dot-com era," I suggested we look at local Florida colocation facilities.  We found the less sophisticated facilities to be more cost-effective.  Most facilities, however, even the top tier ones, were less than ideal because they did not have a significant technical staff available 24x7 in case of an emergency.

Concurrent with our decision to look for out-of-state colocation facilities, we hired a prominent law firm consulting group to help us design and build our data centers based on a colocation model.  During the winter of 2005, we developed a disaster-avoidance data center model that utilized a Citrix server farm as the primary method for delivering applications to the desktop.  In March, 2006, we received approval to move ahead with our dual data center model.  The challenge was that we had less than three months before the start of the next hurricane season!

The Design
The classic dual data center model includes a primary data center (PDC) and a secondary data center (SDC).  By placing the PDC in a hardened colocation facility, we created a disaster-avoidance model rather than a disaster-recovery model.  As part of this model, we elected to upgrade our hub-and-spoke, point-to-point wide area network (WAN) with a more fault-tolerant MPLS "meshed" WAN.  We also decided to buy all new equipment for the PDC and move to blade server, virtual machine, SAN and clustered technologies.  Finally, we took the design one more step by abstracting the PC image from the local office to run it from a bank of Citrix servers in the PDC.  This final step proved to be the most difficult part of the design to implement because of the issues inherent in integrating the the many different law firm applications in a Citrix environment.  The technologies utilized in the PDC include:

• PDC/SDC model
• MPLS WAN
• Blade servers
• VM servers
• MS clusters
• MS Windows 2003
• MS SQL 2005
• Citrix PS 4.0 farm

The Plan
As my old boss back at Vinson & Elkins used to say: "Mike, we have to eat the elephant one bite at a time."  Our "bites" for this project were divided into four phases.  Phase III was truly the most difficult, with a plan my staff referred to as "drinking out of a fire hose" due to the relatively compressed schedule.  We had to have some level of backup ready for the upcoming hurricane season, and we most certainly had to have the bulk of the plan finished by the next hurricane season.  Florida hurricane season runs from June 1 until November 30.  The four phases we elected to follow were:

Phase I, June 2006 - Replicate documents from local servers to colo.  Negotiate colo, WAN, hardware and software contracts; purchase, install and run replication from 13 locations.

Phase II, October 2006 - Access colo documents remotely.  Build remote access environment including integrated suite of document processing and management products.

Phase III, March 2007 - Run most applications live from the PDC.  Build the Citrix environment, move all major systems to the PDC, build the new PC image (removing the last vestiges of Netware), and connect one office at a time to the PDC.

Phase IV, June 2007 - Move the remaining applications to the PDC.  Finish by moving the more difficult, less widely-used applications to the PDC, unless it becomes more appropriate to run them locally.

The Implementation
Although Phase I had a very aggressive time frame, we managed to meet the deadlines, albeit with no time to spare.  Our documents were now offsite and accessible in an emergency.  We also met our Phase II goals and had our Citrix farm and limited integrated DMS image ready to go before the end of the hurricane season.  As luck would have it, the 2006 hurricane season was quite mild for Florida, and we progressed into Phase III with little outside pressure.

However, as March 2007 approached, we realized that Phase III would not be finished in March, and Phase IV really needed to be rolled into Phase III.  Our first major application planned for the PDC was an upgrade to our Elite financial system.  We moved Elite from version 3.3 to 3.6, upgraded the Windows servers to 2003 and clustered the SQL servers using 2005.  We finally had Elite 3.6 running live from the PDC in March 2007.  This is when we learned why Citrix permits you to silo applications.  Some of the processor-intensive applications such as conflicts searching and month-end reporting began to dominate the Citrix servers and slowed everything else down.  By carving out some Citrix servers and placing some of the Elite applications in a silo, we could dedicate more horsepower to the processor-intensive applications without adversely affecting other users.

The next major challenge we had to address was consolidating our two Exchange systems, one in Orlando and one in Miami, into one larger system in the PDC.  We employed additional consulting resources to help us design a fault-tolerant, clustered-server Exchange environment connected to a new Hitachi SAN.  Although this process was not simple, it was much easier than trying to move more than three terabytes of e-mail messages 1,200 miles with little or no user disruption.

After many design meetings, benchmark tests and consultations, we decided to use a host-based data-replication product to synchronize individual e-mail stores once we moved a snapshot of them to the PDC (an entire article could be written on this process alone).  Full Exchange synchronization of all 1,100-plus users was achieved by May 2007 with no lost e-mail messages and had very little downtime!

Our actual office deployments began in late May.  By July we had more than 500 users running all major applications via Citrix-published applications.  Unfortunately, there was a steep learning curve for all of us.  We decided to complete two more office deployments and then take a deep breath before our fiscal year end in October.  We now have over 650 of our 1,100-plus user population using Citrix.  After our most recent analysis and tuning exercises, we expect to resume deployments in November and finish some time in early 2008.

The Challenges
One would think building a new data center with leading-edge technology and operating that PDC from 1,200 miles away would have been our biggest challenge.  Although it certainly was a complex exercise that required long hours and steep learning curves, it proved not to be the toughest part of the project.

Getting the users to adjust to the new paradigm and getting the Citrix environment adjusted to our specific needs were by far the most difficult tasks.  Despite our focus groups, our e-mail messages, and our presentations at various attorney and staff meetings, the changes really did not sink in until our users sat down in front of their PCs and started working.

Lessons Learned
Don't put blind trust in consultants.  No matter how many times they tell you another firm has already done the same project successfully, every situation is different.

Things always take longer than planned.  It sounds like common sense, but it never ceases to amaze me how easy it is to fall back into thinking everything will go right the first time.

Don't let external forces make you commit to unrealistic goals.  If you know something will take 24 months to complete, it really does not make much sense to commit to do it in 18 months and then end up doing it in 24 anyway.

Conduct more focus groups, and plan for longer pilot testing.  Focus groups can be a good thing if done properly.  Carefully controlled pilot testing can be even more effective.

Allow more time between deployments for "tuning."  Make sure to take enough time to make the necessary adjustment before you barrel ahead according to the plan.

Never underestimate the user's resistance to change.  Most users don't live in our technology world, and they don't spend much time thinking about the benefits or necessity of the changes you are making.

Test Citrix printing over the WAN.  Do this with as much load as you can generate.  If it still does not handle printing as well as local print servers, you may need third-party printing products.

Consider running Internet Explorer traffic outside of the Citrix ICA stream.  Internet usage places a heavy load on the Citrix servers and impacts performance.

Consider running Windows Enterprise with 8GB of RAM on your Citrix servers.  We found that Citrix likes more memory.

About our author . . .

Mike Sooley has been the CTO at Akerman Senterfitt in Orlando, Florida since 2001.  In addition to his 15 years of law firm CIO/CTO experience, he has co-founded technology start-up companies, worked as a management consultant, an educator, a programmer and even run a custom carpet mill.  He serves on the Board of the Central Florida Chapter of SIM.  Mike holds a Master's degree in Management Science and a Bachelor's degree in Electrical Engineering and loves to ride motorcycles FAST!  He can be reached at michael.sooley@akerman.com.