Today we are managing two big departments of our University and after a political decision, both departments are joining forces on the IT services. Due to this we are trying to unify our Core Network, but this wasn't easy as it should appear to be.

Both departments runs a huge routed infrastructure with public IPv4 address and only one of them have NAT for clients. But the issue isn't here. The issue is about our WAN links. They are "broken TCP network" in it's glory 🙂

To illustrate the issue. The WAN static addresses are:

• Dept#1: 192.0.2.195/26
• Dept#2: 192.0.2.196/26
• Common Gateway: 192.0.2.193/26

So as you can see both WAN links are on the same subnet, and this is an issue because on Dept#1 we have seven /24 routed networks and on Dept#2 we have three /24 routed networks pointing to the respective IP's as next hops on the routing tables.

Since TCP/IP networking does not allow two addresses of the same subnet in a single piece of hardware we cannot use those addresses on the same equipment. Basically we need to put some Layer 3 device to process the routes and after it merge the networks in a unified core.

The problem here is how to do this. Today we have two pfSense firewalls to do this, but we would prefer to merge those firewalls to put them in HA with both networks, but we're unable to do so.

I have some ideas using VRF on our Nexus 3048 Core Switches but I don't know if this a good idea or even if VRF should be used this way.

Here are the routing tables of both equipment:

Dept #1:

% interfaces
WAN (wan)       -> bce0        -> v4: 192.0.2.195/26
LAN (lan)       -> bce1        -> v4: 100.64.36.1/24

% netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.0.2.193        UGS        bce0
10.3.12.0/24       100.64.36.36       UGS        bce1
127.0.0.1          link#5             UH          lo0
192.0.2.192/26     link#1             U          bce0
192.0.2.195        link#1             UHS         lo0
100.64.36.0/24     link#2             U          bce1
100.64.36.1        link#2             UHS         lo0
100.64.37.0/24     100.64.36.36       UGS        bce1
100.64.40.0/26     100.64.36.36       UGS        bce1
100.64.40.64/26    100.64.36.36       UGS        bce1
100.64.40.128/26   100.64.36.36       UGS        bce1
100.64.40.192/26   100.64.36.36       UGS        bce1
100.64.136.0/22    100.64.36.36       UGS        bce1
198.51.100.0/24    100.64.36.35       UGS        bce1

Dept #2:

% interfaces
WAN (wan)       -> bce0        -> v4: 192.0.2.196/26
LAN (lan)       -> em0         -> v4: 172.16.0.1/21
MGMT (opt1)     -> em1         -> v4: 10.7.0.1/24
SRV (opt2)      -> em1_vlan29  -> v4: 100.64.29.1/24
VPN (opt3)      -> em1_vlan11  -> v4: 192.168.172.254/24
LIG (opt4)      -> em0_vlan666 -> v4: 172.26.66.30/27

% netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.0.2.193        UGS        bce0
10.7.0.0/24        link#3             U           em1
10.7.0.1           link#3             UHS         lo0
10.172.16.0/26     link#2             U           em0
10.172.16.1        link#2             UHS         lo0
127.0.0.1          link#6             UH          lo0
192.0.2.192/26     link#1             U          bce0
192.0.2.196        link#1             UHS         lo0
100.64.29.0/24     link#8             U      em1_vlan
100.64.29.1        link#8             UHS         lo0
100.64.30.0/26     link#2             U           em0
100.64.30.1        link#2             UHS         lo0
100.64.30.64/26    link#2             U           em0
100.64.30.65       link#2             UHS         lo0
100.64.30.128/26   link#2             U           em0
100.64.30.129      link#2             UHS         lo0
100.64.30.192/26   link#2             U           em0
100.64.30.193      link#2             UHS         lo0
172.16.0.0/21      link#2             U           em0
172.16.0.1         link#2             UHS         lo0
172.26.66.0/27     link#10            U      em0_vlan
172.26.66.30       link#10            UHS         lo0
192.168.172.0/24   link#9             U      em1_vlan
192.168.172.254    link#9             UHS         lo0

So as you can see we've a lot of networks. I changed the public IPv4's for the 100.64/10 range and the WAN links to the 192.0.2/24 range just to keep it generic.

If we could just process the WAN links in some device (in the Core Switches for instance) and them use the LAN addresses on a unified firewall the problem would be solved.

And the last thing, I can speculate how the WAN routing table looks like. It should be something like this:

Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
192.0.2.192/26     link#1             ?          switchport24
192.0.2.193        link#1             ?          loopback0
100.64.29.0/24     192.0.2.196        ?          switchport6
100.64.30.0/24     192.0.2.196        ?          switchport6
100.64.146.0/24    192.0.2.196        ?          switchport6
100.64.36.0/23     192.0.2.195        ?          switchport5
100.64.40.0/24     192.0.2.195        ?          switchport5
100.64.136.0/22    192.0.2.195        ?          switchport5

And this table cannot be changed since we don't have access to it.

Thanks in advance.

PS1: Talking with the guys responsible for the "WAN" architecture would be impossible right now. So we must find out a way without changing anything on the WAN address space.

PS2: Who needs routing protocols when everything is built with static routes… 🙁

EDIT: Drawning and long story short.

Today:

+----------------+       +------------------+       
|                | ----> | Firewall Dept #1 | -----------\
|                |       | 192.0.2.195/26   |            |
| Switch         |       +------------------+       +---------------+
| Layer 2 (WAN)  |                                  | Internal Core |
| 192.0.2.193/26 |       +------------------+       +---------------+
|                |       | Firewall Dept #2 |            |
|                | ----> | 192.0.2.196/26   | -----------/
+----------------+       +------------------+

What we want:

+----------------+       + -----------------+
| Layer 2 (WAN)  |       | Unified Firewall |       +---------------+
| 192.0.2.193/26 | ----> | 192.0.2.195/26   | ----- | Internal Core |
|                | ----> | 192.0.2.196/26   |       +---------------+
+----------------+       +------------------+

Routing Table – They are actually real public IPv4 network. I changed to 100.64/10 network to keep it generic.

100.64.29.0/24  gw 192.0.2.196
100.64.30.0/24  gw 192.0.2.196
100.64.36.0/23  gw 192.0.2.195
100.64.40.0/24  gw 192.0.2.195
100.64.136.0/22 gw 192.0.2.195
100.64.146.0/24 gw 192.0.2.196

Issues:

• I don't have management of the Layer 2 Switch.
• Our management start on the firewalls.
• I can't use one cable from the Layer 2 Switch to bind two IP's on the Firewall due to security policies on the Layer 2 Switch. So both cables must be connected with distinct networks.
• I can find use some addresses from our IPv4 pool on the Firewall, but to do this I must preroute before the firewall. I can use the "Internal Core" for this, but I don't know how.
• Separate routers solves this.
• We do have a Cisco Nexus 3048 Switch, so we can do some wizardry on this guy. That's why I mentioned VRF on the first place.