Routing – Cannot Load Internal Website over MPLS

firewalllanmplsrouting

We have an MPLS set up between two subnets. Everything (that is, I have RDP both directions as well as file-sharing) seems to be working correctly except for two things, which may be related.

Windows computers do not populate the "Network" with devices from the other subnet.
From the remote computer, we cannot load our internal website located on the host network.

WINS is enabled and working, and both computers know who is the DNS server and who is the WINS server.

The webserver is at 192.168.1.20 (Windows Server 2003) and the computer (Windows 7) in the remote subnet is at 192.168.2.249. File-sharing and RDP work both ways.

So, the host subnet is 192.168.0.0/23 and the remote subnet is 192.168.2.0/23. Each of the Windstream routers have two ports – one for MPLS and one for internet. Right now, the internet port at the remote is not connected. The internet port on the host router runs through our firewall router before entering the host network, but the MPLS port at the host plugs directly into the switch-trunk.

The two Windstream Routers each have an MPLS port:

192.168.1.2 = Host MPLS
192.168.2.2 = Remote MPLS

The Windstream routers also each have an open internet port to which I have attached a Firewall Router for filtering the internet, making the internet gateways:

192.168.1.1 = Host Gateway
192.168.2.1 = Remote Gateway

I read here that I needed to list the subnets in Active Directory Sites and Services, so I've created the two subnets as members of the one site.

For my testing, the firewalls are turned off on both computers plus the webserver.

The ISP (Windstream) confirms that MTU is set to 1500, and in their testing, their pings are always sent with a size of 1500.

So what can I try to get these two issues solved?

Here is a map:
enter image description here

[update]
When I run Wireshark on the webserver and watch the request for the webpage, I see alot of retransmissions on the http calls. Here is the report:
enter image description here

It looks like http traffic from the remote to the host is what is getting retransmissions. But I've got no equipment that can block it. The firewalls are all off.

So, I can telnet to the webserver from a machine on the same subnet, but I can't telnet to it from a machine on the remote subnet – it reports:

c:\>telnet 192.168.1.20 80
Connecting To 192.168.1.20...Could not open connection to the host, on port 80: Connect failed

Trace-Route from the webserver to the remote computer:
enter image description here

Trace-Route from the remote computer to the webserver:
enter image description here

[update]
I enabled IIS on the remote laptop, and I am able to pull that webpage from a computer at the host location. This has always been the case, however, in my tests. The http traffic, therefore, is only one way.

Best Answer

The two Windstream Routers each have an MPLS port:

192.168.1.2 = Host MPLS

192.168.2.2 = Remote MPLS

The Windstream routers also each have an open internet port to which I have attached a Firewall Router for filtering the internet, making the internet gateways:

192.168.1.1 = Host Gateway

192.168.2.1 = Remote Gateway

Problem Summary

Your problem is that you've got multiple routers on the same subnet:

The Internet gateway at 192.168.1.1
The Windstream MPLS router at 192.168.1.2

This is a faulty design; I completely understand that it "seems like" there is nothing wrong with this, but as you're discovering, this is a hard way to build the network.

Following up on our chat discussion, the exact problem is that SAINTJOSEPH's TCP SYN packets are delivered correctly; however, stateful inspection on your PaloAlto firewall drops SAINTSERVIUS' TCP SYN-ACK response, due to asymmetric routing in your environment. This is illustrated in the two following diagrams.

TCP SYN from SAINTJOSEPH to SAINTSERVIUS:

Your PaloAlto firewall drops the TCP SYN-ACK from SAINTSERVIUS to SAINTJOSEPH:

This tracert makes it very clear that SAINTSERVIUS defaults through 192.168.1.1 (the PaloAlto firewall):

Long Term Solutions

You should only have a single router next-hop for every subnet; however, you currently have two. This results in the drops shown in your wireshark screen capture (which indicates that TCP SYN-ACK packets never reach Windstream's MPLS network).

These are two long-term solutions which we discussed... I'm also including the quick hack we did to validate that the problem is stateful packet drops on the PaloAltos. I am assuming you are using multiple interfaces on the PaloAlto.

Option A: Keep the PaloAlto firewalls inline with your interoffice connections (more maintenance)
Option B: Move the PaloAlto firewalls in front of the internet connections (less maintenance, but also less comfortable)
Option C: Quick Windows static routing hack

Better Design, Option A (MPLS re-addressing required)

This is a another way to lay out the subnet for SAINTSERVIUS (retaining your numbering scheme with /23 subnets, although it's not required...). This option keeps interoffice routing on the PaloAlto firewalls, which feels more familiar to you right now.

This is a long-term solution. You would have to work with Windstream to readdress your infrastructure. This is a lot of work. In my opinion, keeping the firewalls in the middle of your interoffice traffic is less preferable.

Better Design, Option B (MPLS re-addressing required)

This is a another way to lay out the subnet for SAINTSERVIUS (retaining your numbering scheme with /23 subnets, although it's not required...). This option offloads interoffice LAN routing to your PowerConnect switches, and relies on the PaloAlto firewalls to protect against internet threats.

This is a long-term solution. You would have to work with Windstream to readdress your infrastructure. This is a lot of work, but it offers the advantage of not having to deal with firewalls for your interoffice communication.

Suboptimal Workaround, Option C (what you used after our chat)

Open a cmd.exe window and add this route on SAINTSERVIUS as the Administrator:

route ADD 192.168.2.0 MASK 255.255.254.0 192.168.1.2

Closing remarks

I should mention that you are one of the few people who followed through with sufficient details about your question When you started, we didn't have enough details to answer the question. Now, the issues are very clear; thank you for putting the time / effort into documenting the problem well.

Personal note: If you would like an electronic copies of the diagrams as SVG / Inkscape format, please feel free to contact me at my personal email (listed in my user profile).

Related Solutions

Vlan – Handling VLAN based MPLS circuits with site specific internet access

Depending on how the service is delivered by the SP will dictate how you can separate the services on your end.

Typical methods are either a port per service or a VLAN tag per service.

If the SP is tagging the traffic you can just set up your switch to trunk to the SP and then separate the traffic into two access ports (one to FW and one to LAN).

If it's a port per service then just create two VLANs with the services in different VLANs for isolation.

Cisco – traffic engineering – policies for BGP vs static routes

Policy-based routing works great for this. I recently did this to slowly cut over a hospital to a new internet edge one internal subnet at a time in exactly this way.

ip access-list extended PBR_SUBNETS
 permit ip 172.16.1.0 0.0.0.255 any
!
route-map POLICY_ROUTE permit 10
 match ip address PBR_SUBNETS
 set ip next-hop 192.168.1.2

The above will redirect all packets from 172.16.1.0/24 to the IP address specified in the route-map. Change both as needed to fit your situation.

Apply this policy to the router interface that's facing your internal subnets.

interface GigabitEthernet0/1
 ip policy route-map POLICY_ROUTE

This route-map will set the next hop for all permitted traffic, regardless of the routing table. (Use of the "default" keyword can get around this but doesn't sound like you need to) This means that your static route is probably not needed for this purpose, but you can leave it in there as a fallback in case BGP fails.

FYI - the way I did this was to enter the entire configuration, live, during the day, but with no entries in the access list. Since nothing matches, obviously, all traffic behaves normally, following the routing table. This provides for a very easy, testable, reversible framework for switching internal subnets over, since all it takes from this point is just to add them to the ACL.