Daryl Hunter – Hubby, Daddy, Mentor, Friend

So, a few weeks ago I tweeted that we are in serious negotiations for some large scale bandwidth redesign.  Just before Thanksgiving, we signed some contracts which is very exciting.  Just yesterday we informed the vendors that were not chosen, so I feel comfortable now blogging about the changing coming down the bend.

For the last 5 years or so, we've utilized a Cox Communications Transparent LAN setup between our two broadcast campuses – Edmond (EDM) and Oklahoma City (OKC).  We have a 100meg link.  It's great. Our Senior pastor, or whatever team teacher that's "on" that week, teaches primarily from the OKC Campus and then that message is transmitted around the world via Satellite.  BobbyG has discussed our satellite setup on Swerve a couple times – post1 and post2 – so I won't get into that.  If necessary, we could broacast from EDM too – thus, two broadcast campuses with the need for large bandwidth.

Another great reason we have large bandwidth between these two campuses is because our Central Campus, and our primary data center, is located at the EDM facility.  So are our Video guys.  Large bandwidth allows for a quick and easy transfer of data from one place to the other without having to dump to an external hard drive or some other "sneaker net" type of file transfer.

Also, a little more information, for the last several years, we've been using either Cox or AT&T for our Direct Internet Access (DIA) – but that has always come from the Central Campus data center.  We're currently using a 30meg DIA service and it's okay – no major complaints there either.  So, why change?

ENTER THE PRESENT

So, the truth of the matter is, our Church is based out of Edmond, Oklahoma.  Oklahoma is known for, among other things, crazy tornadoes.  I hate admitting this, but we have no real disaster recovery plan – other than the "in case of disaster, resign immediately and run away" plan.  (Just kidding bossman – really – ha ha).

Over the last couple years, we've added a few campuses in a few states, increased our global reach through strategic partnerships in several continents, added some staff, and that's been fun.  Technologically, we've entered a brand new world too.  We've gone from the normal "physical server with local storage for each process" philosophy into the "shared resources" philosophy.  We now have an EMC SAN with a bunch of shared storage.  We now are an ESX VI3.5 Virtual Cluster environment for our servers.  We have about 40 virtual boxes.  We have already been a Cisco CallManager/UNITY shop for phones + voicemail.

It's time to start distributing some of our resources to plan for the disaster that no one wants to talk about, but if/when it hits, will be really bad.

WHAT ABOUT THE FUTURE

So, enough of the wordiness.  Where are we going?

Well, strategically, we've identified our OKC Campus as our secondary / disaster site.  This is smart for three reasons – 1) it's big – we could physically house our Central/Edmond staff at OKC incase something really bad happens to the EDM Campus (fire, tornado, etc.) and 2) we have a great investment in a generator there and 3) we are always going to need high-speed network access there.

To help facilitate data redundancies, we're going to increase our Point to Point connectiivty between OKC and EDM to a screaming one (1) gigabit.  This is awesome!  We are going to move toward a distributed model for our Cisco CallManager appliances.  We are going to implement an ESX Cluster at OKC too – most likely using SAN Melody as our storage choice.  The ESX Cluster at OKC is going to primarily be for bringing up any VMs in the event of a disaster, but this is also a great place to store some monitoring tools like Solarwinds Orion, etc.

We're gonna upgrade our DIA service at EDM to a full 100meg burstable connection.  We have a process on Saturday night where we create a distributable image of Craig's sermon.  That image is created and uploaded to our Content Delivery Network (CDN) and then distributed around the world.  This 2gig (ish) image needs to be uploaded and distributed in a time window of less than 20 minutes (yes, that's a very tight window) – so the 100meg burstable connection will help facilitate that process.

That's not all, we are going to get a DIA service at OKC too – so if the Central Campus network connectivity dies, we can still get to resources across the gigabit connection between OKC and EDM.  This going to be a flexible, burstable connection – with a low CIR – so it's only for emergencies.  This is new for us… investing in a DIA connection that ONLY will be used in emergencies – it's smart technically, but a little harder to swallow $$$-wise.

We're gonna move half of our ASA 5520 cluster to OKC – to better facilitate the redundant DIA service.

We're obviously going to implement some sort of eBGP service – without BGP, there's no redundancy for that incoming internet connectivity.  Along with eBGP comes the need for HSRP internally to make sure our routing is weighted.

And, if we're gonna have gigabit, and two SANs, the best way I know of to replicate VMs is utilizing the Veeam family of tools.  So, look for that to be in the mix too.

BREAKDOWN

So, currently, our services look like this

Cox – 100meg TLAN (point to point / mesh) – OKC to EDM
Cox – 6meg DIA at EDM (Central) – currently unused
AT&T – 30meg DIA at EDM (Central)

When the new services are provisioned, 1Q2009, it will look like this

AT&T – 100meg Burstable DIA (Hi-CAP Flex Product – 95%-ile billing) at EDM (Central)
AT&T – 100meg Burstable DIA (Hi-CAP Flex Product – 95%-ile billing) at OKC (Oklahoma City)
AT&T – some eBGP service for incoming redundancy – either pure BGP or AT&T MARO
AT&T – 1gig Point to Point (Gigaman Product) between EDM and Central

The best part?  Our migration/upgrade will actually turn out to be a few hundred dollars total LESS Monthly Recurring Cost than what we have now.  Yes, we're getting MUCH more – for LESS money.  AT&T is a logical choice for us because of their worldwide reach.  I was negotating pretty hard – considering they were becoming our PRIMARY bandwidth vendor and effectively removing Cox from serious play outside some HFC (Cable) connections here and there.  We like Cox, always have, but they effectively placed themselves out of the game because of a) cost and b) no "burstable" products – they offer bursting, but at a per-incident rate which is not reasonable for us.

More details as they flesh out… it's exciting stuff, fresh off the presses, and hopefully implemented in the next 90 days!  🙂