Daryl Hunter – Hubby, Daddy, Mentor, Friend

Hello again!  Glad to have you back.  If you recall, LifeChurch.tv has been undergoing some significant network changes the last few months.  I blogged about some bandwidth changes last fall.  I blogged about our Data Center Cleanup.  I even blogged about our network design – including some Visio drawings.  As we've stepped through these from concept, contract, provisioning, implementation, troubleshooting, and then additional features, I thought it would be fun to blog through the process.  For some of you, this is tedious and boring.  Sorry.  For others of you, this may help you in future projects too.  I hope it helps.  At any rate, it's good for me to get things out of my head.

As you read in the "design" post linked above, we have redundant 100 Meg AT&T EaMIS Internet connections – one at our Central offices and one at our OKC Campus.  Each location is connected via AT&T Gigaman Metro Ethernet.  Each location has a Cisco ASA 5520 firewall.  I have the internet routers setup now with HSRP – along with my core routing switches – so I have "internal" redundancy.  My next steps include "external" redundancy – using HSRP/BGP on my Internet Circuits and "clustering / failover" on my ASA firewalls. So, anyway, today we finished up the clustering of our Cisco ASAs. Let's walk through that.

Each of my ASA 5520s, physically separated by 7 miles, have multiple interfaces – the breakdown is like this:

• Gig0/0 – Inside – "inside" VLAN setup for default routes/servers/etc.
• Gig0/1 – Outside – "outside" VLAN – My ISPs terminate directly to this VLAN
• Gig0/2 – "STATEful" failover – setup specifically for failover
• Gig0/3 – unused currently – maybe DMZ (or sub-DMZ) later
• Management0/0 – "LAN" failover – setup specifically for failover
• IPS – for Intrusion Prevention – on my "management" or "out of band" VLAN

Let's walk through the configuration…

Switch Configuration
At the Central Campus, each of the ASA interfaces are connected to our Core Cisco 6509.  At the OKC Campus, each of the ASA interfaces are connected to our Core Cisco 4507R.  This is just simple stuff here folks – I have a "STATE Failover" VLAN and placed Gig0/2 on both ASAs into a switchport that was an access port for that VLAN.  Same thing with the "LAN Failover" VLAN and Mgmt0/0 interfaces on the ASA.  Same thing again with the "inside" and "outside" interfaces.  That's easy.  No need to walk through that, right?

Central ASA Configuration
So, this is where it gets a little more complex – but not really.  Cisco has a really good document on this – you can download that here.  I'll walk through the setup on each relevant interface and we'll go from there:

So, I hope that makes sense.  The "failover" interfaces don't have any IP information set in the "interface" section of the config.  That comes later.  Similar to HSRP – you enter the IP Address of the interface, and then the "standby" IP Address – that is – the Address of the secondary ASA.

Great, now, let's focus on the actual failover configuration:

Okay, so, here you've identified THIS ASA as the "primary" cluster member.  You've identified the M0/0 inteface as the "lan failover" and Gig0/2 as the "state failover" interface.  Next, you've given those interfaces IP addresses and identified the "standby" IP address too.  Please remember, the "interfaces" don't have IP Addresses – you put IP Address configuration here and it applies it to the "failover" configuration.

Great.  Finally, you enable failover

OKC ASA Configuration
The configuration on the OKC ASA – the standby firewall – is much easier.  You'll setup the interfaces like this:

Remember, this is "pre" full failover.  So, the IP Addresses on Gig0/0 and Gig0/1 should correlate to the "standby" IP Addresses you set in the Central ASA.  Make sense?  You don't have "standby" setup here.  That's done magically later.  And, again, the "lan failover" and "state failover" interfaces don't have IP Addresses here.

Now, here's how you config the failover:

That's it.  Make sense?  You're identifying this ASA as the "secondary" unit.  You then tell it which interface handles the LAN failover (M0/0) and then setup the IP and standby in an IDENTICAL FASHION to the primary unit.  That's key.  Just use the exact line.  You don't have to do the "state" config – all of that will be replicated/sync'd between the ASAs.

Great, now enable failover:

Failover Happens
So, you've configured the switches.  You've configured both ASAs.  You enabled failover on both.  Now what?  Quick, before you miss it, head back to the console of the Central (primary) ASA.

Did you catch that?  It knows something is up.  When you enabled failover, it gave you a "no response" error because you hadn't enabled failover yet on the "standby" ASA.  Once you did, it started the config replication and the rest of the "magic" behind the scenes to allow failover to work. Let's verify that.

Failover Verification
Alrighty, wouldn't it be nice to know everything looks and feels right?  Go back to the Central (primary) ASA.  Do a "show failover" to look at what you've done.

That's a lot of greek.  Let's break it down.  Basically, you're verifying your failover interfaces (Mgmt0/0 and Gig0/2) are in the (up) status, and notice which hosts are Active and Standby.  We do need to upgrade the secondary IPS – it's a couple software revisions behind (6.0(3) vs. 5.1(6))

For completion, let's do a "show failover" on the OKC (secondary) ASA too:

Yup, same greek.  You should the details in the opposite location now – unit is Secondary – Standby, not Primary – Active.

Hooray!  We're done.  We now have two identical Cisco ASAs, separated physically by 7 miles, connected via a Gigabit Metro Ethernet connection.  They are setup in a cluster – and then feed the "next hop" which is an AT&T EaMIS connection.  Of course, we're not done with our complete network redesign…yet.  Once a small snag with AT&T provisioning is fixed, we'll get BGP all setup and happy and then have a fully redundant LAN/WAN setup.  Stay tuned…