Catalyst 3850 – Rolling Stack Upgrade Guide

First of all you may want to remove the unnecessary spanning-tree configuration on your secondary uplink. STP calculates the cost metric based on the link speed. Your 100Mbit/sec CU uplink has a default cost of 19 and your primary 1Gbit/sec SM Fibre uplink got a cost of 4, so no need to set it manually.

You may also remove the "access vlan configuration" since your trunk configuration with native vlan 666 will be preferred. Also change your allowed vlan configuration to match the primary link. I do not know which vlans you actually use, but inconsistency may become a big pain in case your primary link fails.

 interface GigabitEthernet2/0/28
 description "downlink trunk BldgB 100M-Backup"
 switchport access vlan 666              -> unnecessary
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 666
 switchport trunk allowed vlan 2-99,101-109,111-4094
 switchport mode trunk
 switchport nonegotiate
 shutdown
 speed 100
 duplex full
 spanning-tree vlan 99 port-priority 128 -> unnecessary
 spanning-tree vlan 99 cost 50           -> unnecessary
end

An interesting counter from Gi1/0/28 (Site A) are the input errors on the interface. Might be a layer 1 issue but since you do not have any crc errors and the realibility is 255/255 it may be due to repluging cables.

Since there are no obvious configuration issues I would recommand re-enabling the port and posting the output of:

Site A:

show spanning-tree interface gi1/0/28
show spanning-tree detail

Site B:

show spanning-tree interface gi2/0/1
show spanning-tree detail

Last but not least could you provide the following information:

• Which IOS release are you running on the stacks?
• Does your Service Provider provide transparent L2 services (BPDUs received on both sides? ... you may check if CDP packets are received on both sites via show cdp neighbors)

I would approach it this way: fetch sysuptime, then calculate the boot date/time and the predicted next wrap date/time. Write both to the log. Calculate the next poll time and if it is after the predicted next wrap date/time by a little margin, write a 'wrap expected' bit to the log. Next time you fetch, look at the 'wrap expected' bit and the predicted next wrap time from the log, and if the bit is 1 and if the predicted wrap time and current boot times are pretty close (extremely close if your server and router are both using NTP) then you know it has wrapped and not rebooted. If not, you know it rebooted. If the wrap expected bit wasn't set, simply go back to the main script logic and calculate the new boot time and predicted wrap time, make your wrap prediction bit, and write it all to the log.

You are trying to guard against a reboot false positive by not anticipating the uptime wrap, AND a reboot true negative by assuming the counter wrapped when it actually rebooted. To do both you need pretty careful timing, and even then it just reduces the probability (down past 1 in 1,000,000), but it doesn't eliminate them.

If you want to go full tilt, you can do something like adding a second detection layer on top. For example look at the UDP traffic counter: since you are polling via snmp you will be constantly incrementing it a tiny bit. Since there probably arent many other SNMP polls taking place, it will not likely wrap very often (if at all compared to reboots for other reasons) so if you looked at sysuptime going down AND udp traffic count going down you can increase the confidence that you caught a reboot.