Charter Layer 2 Connection drops when failing over to a secondary firewall

We recently solved a perplexing problem with a Layer2 connection for a client. This client has four sites that are connected with a Charter Layer 2 circuit. This dedicated connection is used to connect the client to our ADS Cloud Service. All of the connections are working fine, except for one location. All sites have Sonicwall High Availability Firewalls that are designed to failover to a secondary firewall if the primary firewall goes down or is rebooted.

In this location when the secondary firewall was active, the Charter Layer 2 connection would go down. Fortunately, we've configured the firewalls to automatically failover to a Virtual Private Network (VPN) so their connection to ADS Cloud stayed up even when the Layer 2 connection went down. We wanted to get the issue resolved, because a failover to the secondary firewall should not take down the Layer 2 connection.

This location had difficulty learning the new Media Access Control (MAC) addresses that were connected to the Layer2 circuit. We observed that if a new device was connected long enough (more than a few hours) it would magically start working. For this location Charter was using Level3 for the last mile carrier.

We setup a conference call with Charter and Level3. It turns out that Level3 was limiting the number of MAC addresses on their connection to only 50. Charter showed they had 93 active MAC addresses on the network. After Level3 increased the allowed MACs to 150 everthing started working. This made sense why it took a couple of hours for the new device to start working on the link because the device had to wait until other MACs in the Address Resolution Protocol (ARP) table expired so the new MAC could be registered.

If you run into a similar problem, ask the last mile carrier if they are limiting the number of MACs in their ARP table. It could save everyone a lot of troubleshooting time.