JUNIPER: Why is OSPF adjacency breaking when I enable FBF on an OSPF interface

juniperospfpbr

I've built a test lab where I am testing Filter Based Forwarding (FBF), aka Policy Based Routing. Question will follow below, but first, the details:

Below is the topology diagram:

enter image description here

OBJECTIVE: Any traffic destined to Staging from Site 1 should route via Link 2 into the WAN and NOT via Link 1. Since Link 1 will be saturated with replication traffic between the two data centers.

SW-1 & SW-2 are Juniper EX4200 switches
RTR-1 & RTR-2 are Juniper J4350's
PE-1 & PE-2 are Cisco 1841 routers running ISIS and MPLS VPN to simulate provider WAN backbone

SW-1, SW-2, RTR-1 and RTR-2 are all OSPF neighbors in Area 0. Both RTR-1 and RTR-2 are ASBR's and injecting BGP learned routes into OSPF. Each router is advertising routes into the WAN for its respective site (as well as pre-pended routes for the other site for redundancy).

Routing traffic from Site 1 to Staging at Site 2 is easily accomplished by simply redistributing the static route to Staging on SW-2 into OSPF with a higher metric. Since that route gets advertised by RTR-2 into the WAN, RTR-1 will learn that route and redistribute it into OSPF with a metric of 0. The OSPF route learned on SW-1 from SW-2 would have a higher metric, thus routing would be preferred over the WAN.

Return traffic from site 2 also needs to flow this way so that we avoid asymmetric routing.
FBF is applied on the inbound interface (Link 4) entering SW-2. This filter will take all traffic sourced from Staging (10.100.190 /24) and make the next-hop RTR-2. This portion of the FBF is working, as I have tested in the lab.

Since RTR-2's preferred route back to Site 1 is via Link 1, we need to apply FBF once again at the inbound LAN interface of RTR-2 (facing SW-2).

Here's the problem… When FBF is applied on that router, OSPF adjacency with SW-2 breaks.

QUESTION: Why is OSPF adjacency breaking between RTR-2 and SW-2??

Configuration for RTR-2 and SW-2 are attached:

RTR-2 Configs

root@RTR-2> show configuration interfaces | display set    
set interfaces ge-0/0/0 unit 0 family inet filter input FBF-TEST
deactivate interfaces ge-0/0/0 unit 0 family inet filter
set interfaces ge-0/0/0 unit 0 family inet address 10.100.254.2/24
set interfaces ge-0/0/3 description "Uplink to WAN"
set interfaces ge-0/0/3 unit 0 family inet address 200.200.200.2/30
set interfaces lo0 unit 0 family inet address 10.100.199.4/32

root@RTR-2> show configuration routing-options | display set 
set routing-options interface-routes rib-group inet STAGING-RIB
set routing-options rib-groups STAGING-RIB import-rib inet.0
set routing-options rib-groups STAGING-RIB import-rib PATH-1.inet.0
set routing-options rib-groups STAGING-RIB import-rib PATH-2.inet.0
set routing-options router-id 200.200.200.2
set routing-options autonomous-system 1

root@RTR-2> show configuration routing-instances | display set  
set routing-instances PATH-1 instance-type forwarding
set routing-instances PATH-1 routing-options static route 10.100.30.0/24 next-hop 200.200.200.1
set routing-instances PATH-1 routing-options static route 10.100.30.0/24 qualified-next-hop 10.100.254.1 preference 100
set routing-instances PATH-2 instance-type forwarding
set routing-instances PATH-2 routing-options static route 10.100.30.0/24 next-hop 10.100.254.1
set routing-instances PATH-2 routing-options static route 10.100.30.0/24 qualified-next-hop 200.200.200.1 preference 100

root@RTR-2> show configuration firewall | display set             
set firewall family inet filter FBF-TEST term TERM-1 from source-address 10.100.190.0/24
set firewall family inet filter FBF-TEST term TERM-1 from source-address 10.100.191.0/24
set firewall family inet filter FBF-TEST term TERM-1 then routing-instance PATH-1
set firewall family inet filter FBF-TEST term DEFAULT then routing-instance PATH-2

root@RTR-2> show configuration protocols | display set 
set protocols bgp path-selection cisco-non-deterministic
set protocols bgp log-updown
set protocols bgp group TEST type external
set protocols bgp group TEST local-address 200.200.200.2
set protocols bgp group TEST import REJECT
set protocols bgp group TEST export ADVERTISED
set protocols bgp group TEST peer-as 65000
set protocols bgp group TEST neighbor 200.200.200.1 preference 20
set protocols ospf rib-group STAGING-RIB
set protocols ospf export BGP-to-OSPF
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 priority 150
set protocols ospf area 0.0.0.0 interface lo0.0 passive

SW-2 Configs

root@SW-2> show configuration interfaces | display set 
set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.2/30
set interfaces ge-0/0/1 unit 0 family ethernet-switching port-mode access
set interfaces ge-0/0/1 unit 0 family ethernet-switching vlan members VLAN-254
set interfaces ge-0/0/11 description "Uplink to STAGING"
set interfaces ge-0/0/11 unit 0 family inet filter input FBF-TEST
set interfaces ge-0/0/11 unit 0 family inet address 10.100.100.1/30
set interfaces lo0 unit 0 family inet address 10.100.199.2/32
set interfaces vlan unit 2 family inet address 10.100.2.1/24
set interfaces vlan unit 251 family inet address 10.100.251.1/24
set interfaces vlan unit 254 family inet address 10.100.254.1/24

root@SW-2> show configuration routing-options | display set 
set routing-options nonstop-routing
set routing-options interface-routes rib-group inet STAGING-RIB
set routing-options static route 172.22.128.0/21 next-hop 10.22.76.1
set routing-options static route 10.22.20.0/24 next-hop 10.22.76.1
set routing-options static route 10.100.190.0/24 next-hop 10.100.100.2
set routing-options static route 10.100.191.0/24 next-hop 10.100.100.2
set routing-options rib-groups STAGING-RIB import-rib inet.0
set routing-options rib-groups STAGING-RIB import-rib PATH-1.inet.0
set routing-options rib-groups STAGING-RIB import-rib PATH-2.inet.0
set routing-options router-id 10.100.254.1

root@SW-2> show configuration routing-instances | display set  
set routing-instances PATH-1 instance-type forwarding
set routing-instances PATH-1 routing-options static route 10.100.30.0/24 next-hop 10.100.254.2
set routing-instances PATH-1 routing-options static route 10.100.30.0/24 qualified-next-hop 10.10.10.1 preference 100
set routing-instances PATH-2 instance-type forwarding
set routing-instances PATH-2 routing-options static route 10.100.30.0/24 next-hop 10.10.10.1
set routing-instances PATH-2 routing-options static route 10.100.30.0/24 qualified-next-hop 10.100.254.2 preference 100

root@SW-2> show configuration firewall | display set             
set firewall family inet filter FBF-TEST term TERM-1 from source-address 10.100.190.0/24
set firewall family inet filter FBF-TEST term TERM-1 from source-address 10.100.191.0/24
set firewall family inet filter FBF-TEST term TERM-1 then routing-instance PATH-1
set firewall family inet filter FBF-TEST term DEFAULT then routing-instance PATH-2

root@SW-2> show configuration protocols | display set   
set protocols ospf export ADVERTISED
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface vlan.2 passive
set protocols ospf area 0.0.0.0 interface vlan.251 passive
set protocols ospf area 0.0.0.0 interface vlan.254 priority 250

Best Answer

So, after working with JTAC yesterday, "I", as in I really didn't need JTAC since I figured out the issue on my own.. realized that my firewall filter was a bit redundant and was lacking a "permit any" statement.

OSPF adjacency was breaking because the firewall filter was taking the "else" traffic (term DEFAULT) and sending it to routing-instance PATH-2, which didn't help either way since it was sending traffic right back to SW-2, something a "then accept" statement would have done easily

So, to repair the issue..

New SW-2 & RTR-2 corrected configlets:

delete routing-instances PATH-2
delete firewall family inet filter FBF-TEST term DEFAULT
set firewall family inet filter FBF-TEST term PERMIT-ANY then accept

New config snips for SW-2:

routing-options {
    nonstop-routing;
    interface-routes {
        rib-group inet STAGING-RIB;
    }
    static {
        route 10.100.190.0/24 next-hop 10.100.100.2;
        route 10.100.191.0/24 next-hop 10.100.100.2;
    }
    rib-groups {
        STAGING-RIB {
            import-rib [ inet.0 PATH-1.inet.0 ];
        }
    }
    router-id 10.100.254.1;
}
firewall {
    family inet {
        filter FBF-TEST {
            term TERM-1 {
                from {
                    source-address {
                        10.100.190.0/24;
                        10.100.191.0/24;
                    }
                }
                then {
                    routing-instance PATH-1;
                }
            }
            term PERMIT-ANY {
                then accept;
            }
        }
    }
}
routing-instances {
    PATH-1 {
        instance-type forwarding;
        routing-options {
            static {
                route 10.100.30.0/24 {
                    next-hop 10.100.254.2;
                    qualified-next-hop 10.10.10.1 {
                        preference 100;
                    }
                }
            }
        }
    }
}

New config snips for RTR-2:

routing-options {
    interface-routes {
        rib-group inet STAGING-RIB;
    }
    rib-groups {
        STAGING-RIB {
            import-rib [ inet.0 PATH-1.inet.0 ];
        }
    }
    router-id 200.200.200.2;
    autonomous-system 1;
}
firewall {
    family inet {
        filter FBF-TEST {
            term TERM-1 {
                from {
                    source-address {
                        10.100.190.0/24;
                        10.100.191.0/24;
                    }
                }
                then {
                    routing-instance PATH-1;
                }
            }
            term PERMIT-ANY {
                then accept;
            }
        }
    }
}
routing-instances {
    PATH-1 {
        instance-type forwarding;
        routing-options {
            static {
                route 10.100.30.0/24 {
                    next-hop 200.200.200.1;
                    qualified-next-hop 10.100.254.1 {
                        preference 100;
                    }
                }
            }
        }
    }
}

Related Solutions

Cisco config example for Policy Based Routing

I would recommend using a route-map for only one purpose (one for nat, another to pbr); mixed use can make for a mess. For NAT, the match interface will apply post-routing -- handy to make conditional nat entries.

The route-map for PBR should use an ACL to match the traffic coming from LAN2 and then set the next-hop to the desired interface with set interface not set default -- or set ip next-hop dynamic dhcp as you won't know the actual gw address. This bypasses / overrides any routing logic that would otherwise send then traffic to your expensive WAN.

Routing – BGP remote-triggered blackhole (RTBH) filter for Juniper

There is no reason to limit customer to blackhole only /32, allow them to blackhole anything from them.

Something like this:

policy-statement AS42-IN {
    term blackhole {
        from {
            community BLACKHOLE;
            prefix-list-filter AS42 orlonger;
        }
        then {
            community add NO-EXPORT;
            next-hop 192.0.2.1;
            accept;
        }
    }
    term advertise {
        from {
            prefix-list AS42;
        }
        then {
            community add ADVERTISE;
            next-hop peer-address;
            accept;
        }
    }
    term reject {
        then reject;
    }
}

Remember to set 'accept-remote-nexthop' under BGP settings, so that the next-hop can be changed to something else than link address.

I also strongly support that providers start to support different blackhole communities than just 'full blackhole'. Especially if you operate in more than one country usually attack is from transit and often customer actually mostly wants to access domestic peerings, so it's useful to implement several levels of blackhhole:

Full
Outside local country (local IXP, whole own AS + customers seen)
Outside local area (if applicable, like for me it could be 'Nordics')
Include/Exclude specific peering router in blackhole

I'm happy to give example how to implement something like this with JNPR DCU/SCU, provided your peering routers are JNPR, but it's possible with just communities, just bit more awkward.

If you absolutely want to limit your customer's options you can add this:

policy-statement /32 {
    term /32 {
        from {
            route-filter 0.0.0.0/0 prefix-length-range /32-/32;
        }
        then accept;
    }
    term reject {
        then reject;
    }
}

policy-statement AS42-IN {
    term blackhole {
        from {
            community BLACKHOLE;
            policy /32;
            prefix-list-filter AS42 orlonger;
        }
..
}

This way it should be logical AND. But I really don't see the value in creating complexity to reduce expressiveness of configuration.

Best Answer

Related Solutions

Cisco config example for Policy Based Routing

Routing – BGP remote-triggered blackhole (RTBH) filter for Juniper

Related Topic