Alarm: CB 2 Fabric Chip 1 Not Online – Juniper Troubleshooting

juniperjuniper-junosjuniper-mx

I am running MX960 for a while now, and I see the minor alarms. Please can someone help on how to clear them?

3 alarms currently active

Alarm time Class Description
2015-07-13 03:22:54 WAT Minor CB 2 Fabric Chip 1 Not Online
2015-07-13 03:22:53 WAT Minor CB 2 Fabric Chip 0 Not Online

Also, when I run the command show chassis fabric summary, I get the following output

hw_tanyu_n@domsc-dca-2.cm> show chassis hardware  
Hardware inventory:  
Item             Version  Part number  Serial number     Description  
Chassis                                JN11D7BCBAFA      MX960  
Midplane         REV 03   710-013698   ACAA3047          MX960 Backplane  
FPM Board        REV 03   710-014974   ZJ3196            Front Panel Display  
PDM              Rev 03   740-013110   QCS151450MM       Power Distribution Module  
PEM 0            Rev 10   740-013683   QCS151270XZ       DC Power Entry Module  
PEM 1            Rev 10   740-013683   QCS151270UE       DC Power Entry Module  
PEM 2            Rev 10   740-013683   QCS151270WT       DC Power Entry Module  
PEM 3            Rev 10   740-013683   QCS151270WL       DC Power Entry Module  
Routing Engine 0 REV 12   740-013063   9012000137        RE-S-2000  
Routing Engine 1 REV 15   740-013063   9009077464        RE-S-2000  
CB 0             REV 10   710-021523   ZG1033            MX SCB  
CB 1             REV 10   710-021523   ZG1045            MX SCB  
CB 2             REV 10   710-021523   ZG1061            MX SCB  
FPC 0            REV 18   750-021566   ZJ9129            DPCE 4x 10GE R  
  CPU            REV 04   710-022351   ZK1084            DPC PMB  
  PIC 0                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 03   740-014289   CB19BQ04J         XFP-10G-SR  
  PIC 1                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 03   740-014289   CB19BQ094         XFP-10G-SR  
  PIC 2                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 03   740-014289   CB19BQ04E         XFP-10G-SR  
  PIC 3                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 03   740-014289   T13L07468         XFP-10G-SR  
FPC 1            REV 18   750-021566   ZJ9110            DPCE 4x 10GE R  
  CPU            REV 04   710-022351   ZK1153            DPC PMB  
  PIC 0                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 03   740-014289   CB19BQ04D         XFP-10G-SR  
  PIC 1                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 03   740-014289   CB18BQ06P         XFP-10G-SR  
  PIC 2                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 03   740-014289   CB19BQ04C         XFP-10G-SR  
  PIC 3                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 01   740-014279   JNXFL099          XFP-10G-LR  
FPC 2            REV 23   750-021679   ZK2167            DPCE 40x 1GE R  
  CPU            REV 04   710-022351   ZK1143            DPC PMB  
  PIC 0                   BUILTIN      BUILTIN           10x 1GE(LAN)  
    Xcvr 0                NON-JNPR     PM32F7Q           SFP-SX  
    Xcvr 1                NON-JNPR     AM122429K12       SFP-SX  
    Xcvr 2       REV 01   740-031850   17T459600037      SFP-LX10  
    Xcvr 3       REV 01   740-031850   17T459600022      SFP-LX10  
    Xcvr 4       REV 02   740-013111   B192241           SFP-T  
    Xcvr 5       REV 02   740-013111   B192160           SFP-T  
    Xcvr 6                NON-JNPR     AN13342D5U        SFP-T  
    Xcvr 7                NON-JNPR     AN13342D5S        SFP-T  
    Xcvr 8       REV 01   740-013111   8268923           SFP-T  
    Xcvr 9       REV 02   740-014132   PGS4SS1           SFP-T  
  PIC 1                   BUILTIN      BUILTIN           10x 1GE(LAN)  
  PIC 2                   BUILTIN      BUILTIN           10x 1GE(LAN)  
    Xcvr 0       REV 02   740-014132   PGS4SPT           SFP-T  
    Xcvr 1                NON-JNPR     AN1209QYM5        SFP-T  
    Xcvr 2                NON-JNPR     AN141487EK        SFP-T  
    Xcvr 9                NON-JNPR     FNS15208975       SFP-LX10  
  PIC 3                   BUILTIN      BUILTIN           10x 1GE(LAN)  
    Xcvr 1                NON-JNPR     EA140300181683    SFP-LX10  
    Xcvr 2       OMPATI   NON-JNPR     A2161803          SFP-LX10  
    Xcvr 3       OMPATI   NON-JNPR     A2161790          SFP-LX10  
    Xcvr 4                NON-JNPR     EA151100120133    SFP-LX10  
FPC 3            REV 30   750-021566   CACX0766          DPCE 4x 10GE R  
  CPU            REV 08   710-022351   CADA0266          DPC PMB  
  PIC 0                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       REV 02   740-014279   URD0S29           XFP-10G-LR  
  PIC 1                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
  PIC 2                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
  PIC 3                   BUILTIN      BUILTIN           1x 10GE(LAN/WAN)  
    Xcvr 0       se-LR    NON-JNPR     1446P-10111       XFP-10G-LR  
FPC 4            REV 38   750-021679   CACK6943          DPCE 40x 1GE R  
  CPU            REV 08   710-022351   CABC1995          DPC PMB  
  PIC 0                   BUILTIN      BUILTIN           10x 1GE(LAN)  
    Xcvr 0                NON-JNPR     EC150900381704    SFP-LX10  
    Xcvr 1       REV 02   740-013111   JNGETF3068        SFP-T  
  PIC 1                   BUILTIN      BUILTIN           10x 1GE(LAN)  
  PIC 2                   BUILTIN      BUILTIN           10x 1GE(LAN)  
    Xcvr 0                NON-JNPR     EC150900380578    SFP-LX10  
  PIC 3                   BUILTIN      BUILTIN           10x 1GE(LAN)  
Fan Tray 0       REV 05   740-014971   VT0507            Fan Tray  
Fan Tray 1       REV 05   740-014971   VT0498            Fan Tray  

{master}  
hw_tanyu_n@domsc-dca-2.cm>  


hw_tanyu_n@domsc-dca-2.cm> show chassis fabric summary  
Plane   State    Uptime  
 0      Online   266 days, 6 hours, 43 minutes, 49 seconds  
 1      Online   266 days, 6 hours, 43 minutes, 49 seconds  
 2      Online   1465 days, 21 hours, 25 minutes, 34 seconds  
 3      Online   1465 days, 21 hours, 25 minutes, 34 seconds  
 4      Offline  
 5      Offline  

{master}  
hw_tanyu_n@domsc-dca-2.cm>

PS: I am not very familiar with Juniper

Best Answer

First, let's bang out some terminology.

DPC: Dense Port Concentrator - This is an older line card architecture, it predates MPC.

MPC: Modular Port Concentrator - This is the newer architecture for line cards, MPC's also have many "sub-architectures" as well (MPC Type 1, Type 2, etc.)

SCB: Switch Control Board - These are fabric cards, they are what connect the FPC's together.

Here's how the DPCs and SCBs fit together:

The boxes in green are the SCB's, they provide 2 fabric planes to each of the line PFE's on the line cards. DPCs/MPCs have either 1, 2, or 4 PFE's - obviously the example is assuming the DPC's have 4. Also note the 2 grey boxes, they act as spare planes should any others fail.

Juniper has a couple Knowledge Base (KB) articles that helps troubleshoot fabric problems that are a bit less obvious, this is just for future reference (I'll post them below).

Looking at the output you provided, it just looks like one of your SCB's rebooted, either by an administrator or just a potential hardware fault. If the same SCB has rebooted 2 or more times (and it wasn't an engineer intentionally rebooting it), you should open a case with JTAC and see what they have to say. You can verify why it rebooted by looking at the logs ("show log messages").

In addition to that, I see that two of the planes are offline (this corresponds to the alarm/offline planes), seeing as you have 3 SCB's, I believe you should see 6 (4 Active/2 Spare). From what I see, it appears that your 3rd SCB has been disabled, or is just down for another reason.

Plane   State    Uptime  
 0      Online   266 days, 6 hours, 43 minutes, 49 seconds    ##SCB0
 1      Online   266 days, 6 hours, 43 minutes, 49 seconds    ##SCB0
 2      Online   1465 days, 21 hours, 25 minutes, 34 seconds  ##SCB1
 3      Online   1465 days, 21 hours, 25 minutes, 34 seconds  ##SCB1
 4      Offline                                               ##SCB2
 5      Offline                                               ##SCB2

Your original question relates to planes 4 and 5, which correspond to SCB2. The second KB article will help with this more accurately. I'd strongly advise making sure another engineer isn't working on this, or that the SCB isn't offline intentionally - if it was offlined but another engineer due to errors then turning it back online might cause problems. If you don't see any evidence that it was offlined intentionally, work through the article and see what happens.

Related Solutions

Difference Between Fabric Plane and Switch Fabric in Juniper Networks

EDIT: I believe I tried to remain a bit too abstract in my answer, I've updated my answer to more acutely answer your questions.

I am struggling with the concept of 'Fabric Plane' and 'Switch Fabric'? I haven't found any clear cut difference between them?

At a very high level, SCBs contain Switch Fabrics, and Switch Fabrics contain Fabric Planes. The collective role of these components is to provide connectivity between all of the PFEs on each FPC installed in the chassis.

Switch Fabric: Switch Fabrics physical chips that are contained within the SCBs. Each SCB will have 2 Switch Fabrics.

Fabric Planes: Fabric Planes are more dynamic in nature, the number of Fabirc Planes per Switch Fabric will vary depending upon the MX chassis model.

Why fabric planes are different in MX960 compared with MX240 and MX480.

As previously stated, the role of the SCBs is to provide connectivity between the PFEs on the FPCs. This connectivity, must be organized into a full mesh, this means that each FPC will have the same amount of throughput capacity to every other FPC. Every time you add another FPC, the SCBs must be able to support that full mesh requirement.

Using the book's example, MX240s and MX480s have 2 SCB slots, which will result in 4 Switch Fabrics and 8 Fabric Planes. But, they also have a smaller number of available FPC slots. It simply comes down to the fact that the MX240 and MX480 only need half of the Fabric Planes in order to accomodate the full mesh requirement, leaving the remaining half to be used as spares (backup).

NOTE: To answer the comment you left on Benjamin's answer, yes, those grey boxes are the spare planes in that particular example, be aware, it can vary per FPC type.

If we look at the book's example of the MX960, even though it has an additional SCB slot, it also has more FPC slots (and therefore more PFEs). In order to maintain that full mesh requirement for the increased number of PFEs, half of the Fabric Planes would not be enough. So, it will "partition" the Fabric Planes a bit differently. Now, we end up with 6 total Fabric Planes (3 SCBs, with 2 Switch Fabrics each, and 1 Fabric Plane per Switch Fabric).

How SCB effect throughput?

There are additional complexities to complement what I mentioned above. First, is the fact that different model SCBs support different levels of maximum bandwidth. Second, FPCs also support varying levels of bandwidth. Finally, there are your redundancy and throughput requirements.

Using an MX960 as an example, utilizing fabric redundancy will keep 1 SCB as a hot standby in case that one of the others fail. This means that only 2 SCBs of capacity will ever be used at any one time. You have to account for your network capacity in that those 2 SCBs must be enough to handle the traffic on your router. On the other hand, if you would rather used "increased-bandwidth" mode, you can do so, which will enable all 3 SCBs to be utilized. However, if a SCB fails during a period of high network traffic, you may experience network congestion and performance impact.

MX Series Routers Fabric Resiliency

All of these factors must be considered when deciding what hardware to use.

One final note, there is a newer "2nd Edition" of the MX book that you may find helpful as well.

Juniper EX4300 – IRB Interface Cannot Ping Itself

I use the run show interfaces irb.2 terse, I get

> Interface       Admin       Link    Proto    Local    Remote irb.2    
>                 up          down    inet     10.10.11.110/24

You can see that the Interface status is "link down". So your interface is effectively down and this is perfectly normal that you can't ping its IP address.

You have to bring the interface UP, this imply to bring UP at least one physical interface on which the VLAN is configured. Either an interface with this VLAN configured in mode access, or a trunk port on which the VLAN is allowed.

Best Answer

Related Solutions

Difference Between Fabric Plane and Switch Fabric in Juniper Networks

Juniper EX4300 – IRB Interface Cannot Ping Itself

Related Topic