Comcast SMC Gateway ARP Flash - High Avalibility

[Avestan]

Hello everyone,

I have a Comcast Business cable modem/router (SMC) with 13 Static IP Addresses. The SMC that Comcast provides a combo modem and router, so it's already running NAT. The SMC is connected to a LinkSys Switch in order to have more ports than what the SMC (4 ports) provides. I have logging into the web interface 10.1.10.1 and disabled the firewall in order to pass some of the IP addresses to my routers (Two Different Routers on Two of the public IP addresses). Everything has been working as expected until I decided to added a High Availability to a proxy which is on one of the public addresses. This is a Linux Box and I am using Heartbeat for HA.

The two proxy boxes work without any issue as an individual box. I am going to use Box One and Box Two notation in order to distinguish them here. Each boxes also have their own Public IP address XXX.XXX.XXX.XX7 and XXX.XXX.XXX.XX8 for accessing the box. The third IP address is the floating IP address XXX.XXX.XXX.XX9 which is obtained by the Master/Active Box while the Salve/Passive Box will be on hot standby. The two boxes are connected to LinkSys Switch which provides enough port (LinkSys 4124 - 24 port unmanaged switch) for the provided range of public IP addresses and Internet connectivity.

Initial condition is follow:
Box One – Master / Active
Box Two – Slave / Passive (Hot Standby)

On the box one, once the Heartbeat is stopped, the Floating IP address is released, Database is stopped, and finally the remaining services are shutdown in proper order. This event results the box two takes over the Floating IP address by assigning it to the eth0 under aliases eth0:0, starts MySQL database, and finally turn on the other services in proper order. At this point everything appears to be in
perfect working order.

The problem is that the the traffic from the internet cloud continues to be sent to the box one while the traffic from my other boxes connected to the switch have no problem getting to the box two via floating IP address! (eg. I can ssh to the floating IP address from any of the 13 public IP addresses provided by the comcast and get connected to box two while the ssh from an external IP address is sent to the box one. If I wait for 300seconds then everything works fine and traffic from outside get routed to the box two.

My disgnostic shows the issue is the ARP Cache of the Comcast Modem/Router and so far I haven't been able to use gratuitous ARP to update the ARP cache on the SMC.

I would like to know if you guys have come across this issue and would like to know what you think about it.

Thank you in advance for your help.

Avestan

[mrjester]

Not sure if you are still looking for a solution to this or not. I ran into the same problem, though I was using Vyatta as my internal router.

The problem isn't an ARP cache issue, it is a design 'feature' of the SMC and how CC saves 4 IPs by not providing transit. The SMC associates the static IP to a MAC addr so it knows were to forward traffic. To change the association you have to reboot the modem.

The problem lies in most implementations of VRRP or clustering the MAC address of the VIP or service IP is the same as the MAC of the active node interface. If on Box One, it will have the MAC of Box One eth0. If not Box Two, it will have the MAC of Box Two eth0.

In Vyatta there is a virtual interface called a pseudo-ethernet device or peth for short. (I believe this is based on macvlans) I configured a peth on each cluster member with the same MAC, but no addresses. When the cluster failed over the primary node gained all IPs and the secondary node effectively shuts down. This push the ARP movement to the switch instead of to SMC thus allowing you to perform failover behind the SMC.

I did a brief write up on the Vyatta forums when I did this originally. It may have some details I left out here.

Hopefully this helps.

[Avestan]

Hello Mr Jester,

Thank you for the reply. Many months has passed since my initial post and your kind and informative reply. But I am going to share my finding and solution for anyone else with the similar issue.

Interesting enough after lots of testing and scratching my head and running ngrep, wireshark, and other network packet catching/monitoring tools, I also came to the same conclusion as you mentioned in your posting. The issue is the MAC address difference between Primary Box (Box One) and the Secondary Box (Box Two).

I have to mention that at the time I was not running a vyatta as firewall/router. The problem that I was seeing was on two boxes running OpenSIPS which were configured as HA SIP-Proxy / LoadBalancer. I was using Heartbeat, Mon, and few other open-source software for system and service level monitoring.

I basically solved the problem by writing my own little script to generate a Floating MAC address which is assigned to the Network Interface of the Primary Box and is moved between the Primary and the Secondary in addition of the Floating IP address and everything started to work the way it was intended.

Funny enough, I have recently added a Vyatta box to my setup. Testing was done on a single vyatta box and after few glitches and issues that I had with Vyatta which almost resulted in dropping it and going for hard-core linux Firewall/Router solution using iptable, conntracker, etc, I decided to go Vyatta way.

Since my initial issue which resulted in my first posting, I have upgrading on my Comcast Service. Consequently, I have been given a fancier looking broadband modem. What I have right now is SMCD3G-CCR. I am still using the same little script for Floating MAC Address and everything works fine.

Going back to the discussion on the Vyatta solution. as I have added the second Vyatta Box to create a HA environment, I am seeing some very strange behavior which I would like to share.

What a small world. I have just realized that I have read many of your posting on the Vyatta Forum and they helped me tremendously, to understand and operate Vyatta as a solution for firewall/router in my setup.

In my setup I have to deal with SIP packets as I mentioned earlier. If I can break SIP packets into Signalling on ports 5060, 5061 and RTP/Voice on Ports 10000-20000.

The issues that I am seeing are:

* The network behavior is different between the Primary going down (Shutdown) and the Primary going standby (Not Shutdown).

* In Primary shutdown scenario, I only get signaling packet sends to the right box while no RTP/Voice packets are routed correctly (No Voice is heard).

* In Primary standby scenario, I get RTP/Voice packets but I am not able to send any "dtmf" (Voice is heard).

My initial investigation shows that I am losing the RTP packets when the Vyatta Primary Box gives up all its resources to the Secondary Box. Rebooting Vyatta secondary while the Vyatta Primary is down fixes the problem. I am not certain/sure what is going on at the moment. I have few thoughts and I have to do more thinking and packet inspection before being certain what the issue is this time. But I will certainly come back to post my finding and hopefully the FIX for the issue.

Anyways, as I mentioned earlier, this is a very strange behavior and since this appears to be Vyatta related I am going to continue this discussion on Vyatta Forum.

Thanks,

Avestan