We recently had a sporadic reboot of a distribution switch for an unknown reason. No recent hardware/software changes have been made. Our log files don't indicate anything leading up to the incident.
The system is a 6509-E running software version 15.1(1)SY on both active and backup supervisor modules.
The system had a primary and backup supervisor card (SUP720) running in SSO. The system was actively functioning in Standby Hot, but the Backup RP never took over as the Active RP, which caused a complete system reboot.
The backup RP eventually booted back up and came back online with the most recent configuration, but the primary supervisor card slot won't boot any cards. We replaced the card with the same card model with no luck. The cards were all tested on a backup 6509 chassis, all of them booted without any issues; which eliminates the possibility of a faulty card.
When trying to boot the cards on this chassis, this is as far as we get:
System Bootstrap, Version 8.4(2) Release
Copyright (c) 1994-2005 by cisco Systems, Inc.
Cat6k-Sup720/SP processor with 524288 Kbytes of main memory
Autoboot executing command: "boot disk0:s72033-adventerprisek9-mz.151-1.SY.bin"
Loading image, please wait ...
Initializing ATA monitor library...
Self extracting the image... [OK]
Self decompressing the image : ########################################################################################]
Cisco IOS Software, s72033_sp Software (s72033_sp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:21 by prod_rel_team
Image text-base: 0x40101778, data-base: 0x427F0760
*Nov 19 17:31:23.407: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
*Nov 19 17:31:23.407: %PFREDUN-6-STANDBY: Initializing as STANDBY processor for this switch
*Nov 19 17:31:30.179: %SYS-SPSTBY-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
*Nov 19 17:31:23.607: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Firmware compiled 11-Sep-12 12:45 by integ Build [26624]
*Nov 19 17:31:30.179: %OIR-SPSTBY-6-CONSOLE: Changing console ownership to route processor
System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2005 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 524288 Kbytes of main memory
Download Start
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Download Completed! Booting the image.
Self decompressing the image : ########################################################################################]
<...Output Truncated...>
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:13 by prod_rel_team
Image text-base: 0x4010F6D8, data-base: 0x44F3E6C0
It doesn't get any further. When we check the active RP, it shows the card is attempting to come online but never fully boots.
router#show redundancy states
my state = 13 -ACTIVE
peer state = in progress to standby cold
Mode = Duplex
Unit = Secondary
Unit ID = 6
Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Communications = Up
client count = 145
client_notification_TMR = 30000 milliseconds
keep_alive TMR = 9000 milliseconds
keep_alive count = 1
keep_alive threshold = 19
RF debug mask = 0x0
Any suggestions on how to fix this? Or is the chassis defective?
show module reaffirms that the card isn't recognized by the system.
router#show module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAD10XXXXXX
2 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAD10XXXXXX
3 48 CEF720 48 port 1000mb SFP WS-X6748-SFP SAL10XXXXXX
4 48 CEF720 48 port 1000mb SFP WS-X6748-SFP SAL10XXXXXX
5 0 Supervisor-Other Unknown Unknown
6 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL12XXXXXX
7 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD11XXXXXX
8 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL10XXXXXX
9 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL10XXXXXX
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 000a.b866.74da to 000a.b866.74dd 2.4 12.2(14r)S5 15.1(1)SY Ok
2 0018.b9a6.a888 to 0018.b9a6.a88b 2.4 12.2(14r)S5 15.1(1)SY Ok
3 000a.b8cf.01e8 to 000a.b8cf.0217 1.7 12.2(14r)S5 15.1(1)SY Ok
4 000a.b80f.8da0 to 000a.b80f.8dcf 1.7 12.2(14r)S5 15.1(1)SY Ok
5 0000.0000.0000 to 0000.0000.0000 0.0 Unknown Unknown Unknown
6 000a.b86d.8df4 to 000a.b86d.8df7 5.7 8.5(2) 15.1(1)SY Ok
7 0007.0e0f.98ca to 0007.0e0f.98f9 2.6 12.2(14r)S5 15.1(1)SY Ok
8 0019.06a0.bb30 to 0019.06a0.bb5f 2.4 12.2(14r)S5 15.1(1)SY Ok
9 0019.2f2f.e280 to 0019.2f2f.e2af 2.4 12.2(14r)S5 15.1(1)SY Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC SAD10XXXXXX 3.0 Ok
2 Centralized Forwarding Card WS-F6700-CFC SAD10XXXXXX 3.0 Ok
3 Distributed Forwarding Card WS-F6700-DFC3A SAL10XXXXXX 2.5 Ok
4 Distributed Forwarding Card WS-F6700-DFC3A SAL10XXXXXX 2.5 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B SAD09XXXXXX 2.3 Other
6 Policy Feature Card 3 WS-F6K-PFC3B SAL12XXXXXX 2.4 Ok
6 MSFC3 Daughterboard WS-SUP720 SAL12XXXXXX 3.2 Ok
7 Centralized Forwarding Card WS-F6700-CFC SAD11XXXXXX 3.1 Ok
8 Centralized Forwarding Card WS-F6700-CFC SAL10XXXXXX 2.0 Ok
9 Centralized Forwarding Card WS-F6700-CFC SAL10XXXXXX 2.0 Ok
Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
4 Pass
5 Unknown
6 Pass
7 Pass
8 Pass
9 Pass
Running show redundancy gives a more detailed output of show redundancy states. Note that both active and backup are running the same software versions, 15.1(1)SY.
router#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 2 weeks, 4 days, 4 hours, 5 minutes
Switchovers system experienced = 0
Standby failures = 508
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = slot 6
Current Software state = ACTIVE
Uptime in current state = 2 weeks, 4 days, 4 hours, 3 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:13 by prod_rel_team
BOOT = disk0:s72033-adventerprisek9-mz.151-1.SY.bin,1;
CONFIG_FILE =
BOOTLDR =
Configuration register = 0x2102
Peer Processor Information :
----------------------------
Standby Location = slot 5
Current Software state = in progress to standby cold
Uptime in current state = 3 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:13 by prod_rel_team
BOOT = disk0:s72033-adventerprisek9-mz.151-1.SY.bin,1;
CONFIG_FILE =
BOOTLDR =
Configuration register = 0x2102
Best Answer
Based on what you describe and tested I'd expect the slot on the chassis to be broken.