Daryl Hunter – Hubby, Daddy, Mentor, Friend

It's a pretty exciting time at LifeChurch.tv – not just today, but all the time.  Last fall we began a long-term strategic project concerning our Internet connectivity and Private connectivity between our Central and OKC Campuses.  We have several goals:

• Faster Internet (Burstable) at the Central Office
• Some-sort of Internet (Burstable) at the OKC Campus
• Faster connection between Central and OKC
• Redundancy for Internet (outgoing)
• Redundancy for IPSec & Services (VPN for campuses, remote workers, email, etc.)
• Remote storage (OKC) to replicate critical XSAN data and other business data

Central Campus
Let's start with Central.  Our Central offices are in Edmond, Oklahoma at the same physical site as our Edmond Campus.  We've been using Cox and AT&T for years for various flavors of bandwidth and telephone service.  Just 5 years ago, when I first joined the LC.tv team, a single T1 was all the internet bandwidth we were using – and that was shared among our Edmond, Central and OKC Campuses.  We upgraded that to 6 meg about 4 years ago.  Then 3 years ago it went to 30 meg.  30meg is a lot, but it's being pushed to capacity in order to meet some business goals and data transfer timelines we have each week in order to "do business."

Central to OKC Campus
5 1/2 years ago or so, the OKC and Central campuses were connected via a single T1 – and we backhauled all interesting traffic to Central.  Everything changed in late 2003 – we partnered with Cox and brought in a 100meg TLAN circuit to connect OKC and Central.  It was a tipping point of what we do now on a weekly.  We started doing "live streaming" between the campuses, so Craig didn't have to drive back and forth and back and forth for the preaching.  The first time we used it was Christmas 2003.  We already had four campuses (we have 14 now), were averaging about 7,000 attendance (that's 25,000+ now) and were just starting to invest in better methods of network connectivity.  HOW EXCITING!  Again, 100meg is great.  That's fast.  However, as our methods have grown, and our file sizes have grown – it still can take a few hours to copy data back and forth between campuses.  We can plan ahead for the most part, but in a last-minute scenario – things can get sketchy.

OKC Campus
As mentioned, the OKC Campus is in, well, Oklahoma City.  It's physically about 7 miles due West of the Central Offices.  For as long as anyone can remember, the OKC Campus has not had Internet service.  It has always backhauled (in some fashion) network conectivity to wherever the staff was located – which is the Central Campus today.

This past Saturday I flew to OKC.  While there, I brought the new network connectivity live – and exorcised a demon that was plaguing us.  Here's the skinny on the new setup for you propeller heads.

Central Campus (Visio PDF) – We migrated from a legacy SBC 30meg service to an AT&T EaMIS Burstable 100meg service.  We also took our "old core" Cisco 4506 out and put in a "new to us core" Cisco 6509.  We only have a 30meg CIR – so we will continue to traffic shape to make sure our billing doesn't get out of control.  We still have our Cisco 2811 for Voice service, Cisco 3725 for SIP, ASA 5520 for firewall + IPS, and Cisco 3825 for Internet Routing.

OKC Campus (Visio PDF) – We added a new AT&T EaMIS Burstable 100meg service.  The "old core" Cisco 4506 from Central was gutted and reconfigured as a Cisco 4507R.  The OKC AT&T Service only has a simple T1 (1.5meg) CIR – so we will ONLY be using that in an emergency.  That's the point really, to have it if something really horrible happens at Central.  I provisioned a Cisco 2821 for Internet Routing, and a secondary ASA 5520 for firewall + IPS.  I haven't clustered the ASAs yet, but that'll happen very soon.

Disaster Recovery (Visio PDF) – We migrated FROM our COX TLAN (100meg) service TO an AT&T Gigaman – Gigabit Metro Ethernet service.  This is 10x the speed.  We're pleased.  We're "dot1q trunking" between the two sites.  We will be moving one of our redundant Cisco CallManagers from Central to OKC.  We'll also be using Veeam Backup to replicate some data from our Central EMC SAN to our redundant Nexenta SAN.  That Nexenta SAN also will house replicas of our XSAN data.  Here's a pic of all the redundant gear.

Phase Two
Well, I've already started Phase two.  Phase two is network redundancy.  I've implemented some simple HSRP on both the 4507 and 6509.  Here's a simplistic example on our VLAN 10 – the voice vlan that houses the Cisco CallManager servers

See how that works?  Both VLANs on both switches have their own IP (252 and 253) but they SHARE the gateway IP (254).  The 6509 has the highest priority (200) and is set to "preempt" which means that if for some reason the 6509 fails, and the 4507 takes over – the 6509 will immediately TAKE BACK OVER when it comes online.

I've setup HSRP for all relevant VLANs – some are preempted on the 6509.  Some are preempted on the 4507R.  It just depends on the need.  HSRP is great for INTERNAL routing.  It keeps you highly available by having your gateway exist on multiple devices.  But, we also need EXTERNAL highly available services.  That's BGP.  We should have our eBGP setup in the next few weeks.  When that happens, I'll cluster our ASA 5520 firewalls in such a way that network traffic can flow equally well incoming/outgoing through those.  Hooray!

Phase Three
Phase 3 will be a little trickier.  Phase three is data redundancy.  Basically we need to take all of our data at the Central Offices and replicate that across the Gigaman.  The problem is that it's over 100TB (terabytes) of ever-changing data on two separate SANs (EMC & Apple XSAN/XRAID) and there are no common management tools.  We've settled on a Nexenta SAN for the storage.  Now we need to make sure all the replication works.  Bakbone's Netvault On Demand Replicator for MacOS is what we hope to use for the XSAN.  I'll use Veeam of course for the EMC stuff.  More on that at a later time.

Are you still reading?  Tell me about some exciting technology stuff going on in your world!