The simple answer is to make the CAM timer equal or slightly longer than the corresponding interface ARP timer, but there are at least three different options to select from...
Option 1: Lower all interface ARP Timers
This option works best if you have a decent-sized layer2 switched network, a reasonable number of ARP entries and few routed interfaces. This method also is preferable if you like to see PC mac entries age out of the topology quickly.
- On all IOS ethernet interfaces facing an ethernet switch:
arp timeout 240
- On all IOS ethernet interfaces facing an ethernet switch:
hold-queue 200 in and hold-queue 200 out to avoid dropping ARP packets during periodic ARP-refreshes (these limits could be higher, or lower depending on how many ARP refreshes you think that you'll need to handle at once). If you are adjusting Selective Packet Discard values, then you should follow the guidelines in the paper I linked.
This forces Cisco IOS to refresh the ARP table within four minutes, if it hasn't happened otherwise for a given ARP entry. The obvious disadvantage is that this doesn't scale well if you have lots of ARP entries... the limits vary by platform. I have used this with a few hundred ARPs per router on Catalyst 4500 / 6500 (the Layer3 SVIs) without any issues.
Option 2: Increase the switch CAM Timers
This option works best if you have a large number of ARP entries (i.e. thousands, such as an intense VMWare environment could see).
- On all IOS switches:
mac address-table aging-time 14400, or mac address-table aging-time 14400 vlan <vlan-id> for any Vlan that is of concern.
This change adjusts timers that most people assume are fixed at 300 seconds (on Cisco IOS), so be sure to include this in continuity docs. The side-effect of this is that CAM table entries linger for 4 hours after the PC is removed (which can be either good or bad, depending on your PoV). If 4 hours is too long, see the next option...
Option 3: Change both the interface ARP timers, and the switch CAM Timers
This option avoids hideously-long CAM timers in Option 2 at the expense of more configuration. You can choose whether you need 900 seconds, 1800 seconds, or whatever... just make sure your CAM and ARP timers match; thus, you will need to configure both Option 1 and Option 2 in your topologies.
As discussed in chat, PBX / SIP traffic is unique to an IP host route in your case. Therefore, you can remove PBR and use tracking objects on overlapping static routes, which go out different dialer interfaces to solve the problem.
ip route 89.123.45.10 255.255.255.255 Dial0 track 1 1 name PBX_Pri
ip route 89.123.45.10 255.255.255.255 Dial1 track 2 10 name PBX_Bak
ip route 0.0.0.0 0.0.0.0 Dial 1 track 2 1 name Data_Pri
ip route 0.0.0.0 0.0.0.0 Dial 0 track 1 10 name Data_Bak
Best Answer
You certainly can load balance at layer 3 with simple static routes. You'll want to configure
ip load-sharing per-packeton each interface. Be sure to enable Cisco Express Forwarding, CEF.