Cisco – QoS woes – managed IP VPN

avayaciscoqos

(First of all, I'm sorry for this wall of text. I don't know how to make it any shorter without losing important information. I originally wanted to use the chat room for this, like we do on serverfault for these kind of questions, but there is nobody in the network engineering room).

We're a corporation with several daughter companies, where we have a rather large managed IP-VPN with about 70 different locations, varying from 2Mbps SHDSL to 100Mbps fiber. The IP-VPN carries multiple VPNs (or tunnels to be exact).

The priority of traffic is this, from a management and design standpoint:

VoIP (Avaya and Lync)
Video (Lync)
RDP
Internal services (fileservers, Active Directory, intranet etc)
Non-prioritized internal services (proxy servers for internet usage, windows update services, system center configuration management, antivirus update proxies etc)
The not matched traffic (internet)

VoIP is only used at certain offices, where there is a low amount of users. The biggest remote office that use VoIP right now has a 4mbps SHDSL with 5 employees and 5 avaya IP phones running the G.711 ALAW 64K codec. This should never bring the voip data traffic up to more than 320kbps.
I've verified that the phones use DSCP 46 for audio, and it's therefore correctly matched as EF (see config below). The signaling however is matched as DSCP 24, which I'm not sure if our QoS profile picks up..

All remote locations use RDP against several RDS farms at our HQ (2x100Mbit fiber). The bandwidth used for RDP is not so easy to figure out, since it basically uses everything it gets. We do have certain limitations set to make sure that it's not too resource hungry, but that is probably out of scope for this site. We do have some rather severe problems with RDP lately (https://serverfault.com/questions/515809/mouse-cursor-jumps-around-when-using-rdp), which is why I'm posting this on network engineering.

Lync uses DSCP 46 for audio and DSCP 34 for video. Internal services and non-prioritized internal services are just matched by subnets, and everything else is just match any.

Here is a copy of the latest QoS config revision, which I have modified slightly to hide certain names and IP addresses:

!
class-map match-any INTERNAL-PRI
 match access-group name CUST-INT-PRI
 match access-group name CUST-DMZ
class-map match-any INTERNAL-NOPRI
 match access-group name CUST-INT-NOPRI
class-map match-any REMOTEDESKTOP
 match access-group name RDP
class-map match-any ALL
 match any
class-map match-any NETWORK
 match ip precedence 6
 match ip precedence 7
class-map match-any EF
 match ip dscp ef
 match ip dscp cs5
class-map match-any AF-HIGH
 match ip dscp af41
 match ip dscp cs4
class-map match-any AF-MEDHI
 match ip dscp af31
 match ip dscp cs3
class-map match-any AF-MEDIUM
 match ip dscp af21
 match ip dscp cs2
class-map match-any AF-LOW
 match ip dscp af11
 match ip dscp cs1
class-map match-any BE
 match ip dscp default
!
!
policy-map setTos
 class EF
 class REMOTEDESKTOP
  set ip dscp af31
 class INTERNAL-PRI
  set ip dscp af21
 class INTERNAL-NONPRI
  set ip dscp af11
 class class-default
  set ip dscp default
policy-map useTos
 class EF
  priority percent 10
 class AF-HIGH
  bandwidth remaining percent 35
 class AF-MEDHI
  bandwidth remaining percent 25
 class AF-LOW
  bandwidth remaining percent 20
 class BE
  bandwidth remaining percent 10
 class NETWORK
policy-map QOS
 class ALL
  shape average 4096000
  service-policy useTos
!
!         
ip access-list standard CUST-DMZ
 permit 123.123.123.0 0.0.0.255
!
ip access-list standard CUST-INT-PRI
 permit 10.50.0.0 0.0.0.255
 permit 10.51.0.0 0.0.0.255
!
ip access-list standard CUST-INT-NOPRI
 permit 10.50.10.0 0.0.0.255
 permit 10.51.10.0 0.0.0.255
!
ip access-list extended RDP
 permit tcp any eq 3389 any
 permit tcp any any eq 3389
!

As you can see, it's a rather large QoS configuration. Note that we did not create this config our selves, it was all done by a previous employee at our IP-VPN provider. Note also that the shape value is changed according to what kind of connection it is (2mbps, 4mbps, 8mbps and 10mbps).

By now you're probably wondering – What's the question here? Here goes..

Like I mentioned earlier, we are drowning in complaints from RDP users about lag/user input not being recognized. Are we not prioritizing it correctly? Is it possible to make sure that RDP gets a minimum amount of packet loss, latency and jitter, but still being restricted in bandwith?
I'm not seeing any mention of queues in this config. I've read some Microsoft documentation, and they recommend to use priority queue on VoIP and WRED on video. How do I make this happen?
As the config shows, none of the AF classings use medium or high drop. What kind of services are safe to drop? RDP, video and voip does not work well with drops..
Are the bandwith percentages in order? It sums up to 100% usage

Any other suggestion(s) are welcome, as I'm desperate to get this sorted out. If you think it's too much to answer on a Q&A site I'll just bite the dust and hire a consultant from our Cisco Gold partner, which is financially OK – I just want to learn this if I can.

Best Answer

To answer your questions:

RDP traffic should get up to the 25% of the remaining bandwidth. Where the already reserved bandwidth is the 35% ( class-default gets 25% by default and EF get 10% ). So, if i'm right, you assigned ~665Kbps to RDP. Anyway you should check if you're dropping packets issuing the command below:

show policy-map <your wan interface> output class REMOTEDESKTOP

and checking for dropped packets.

Cisco assign a queue to each user-defined class that includes the bandwidth or police commands. To make a long-story simple those commands define the amount of bandwidth assigned to every queue during congestions.
In theory every TCP based stream should be OK with drops. In practice some of them aren't. Dropping precedence bits tell the routers what packets should be dropped, within a given class, before congestion happens. Since RDP is the only type of traffic defined in your REMOTEDESKTOP class, you should not worry about it.
Bandwidth percentage are not in order ( as Jeremy stated ).

That said, there are a lot of things that i would change in your configuration:

There are no matches between some of classes of the setTos and the useTos policy-map. For instance the one named AF-HIGH is processing no packets since no class in setTos sets DSCP to AF41.
BE class in setTos is somehow self-defeating since it makes the class-default class useless. Note that class-default is the only system-defined class and get the 25% of the bandwidth by default ( 100 - max-reserved-bandwidth ) .
bandwidth remaining percent is not the best options ( as Jeremy explained ). I would change it to bandwidth percent.
I would prefer to mark EF packets by myself and not to rely on the phones' settings.

Related Solutions

Cisco – How to reasonably verify the QoS configuration is working

Your question's pretty broad. There's a lot of different commands you can use to troubleshoot and monitor QoS, so I'll focus on the primary question you have, which is how to reasonably verify your QoS configuration is working and how to read the policy-map interface output.

The only true way to verify that QoS is working is to hook up a traffic generator and monitor your drop rate in various queues. Since that isn't typically feasible, particularly in a production environment, all you can really do is verify that the traffic is being marked and classified properly.

What you're really looking for, when it comes to verifying if your QoS configuration is working, is for the counters in the policy-map interface command to increment.

So, for example, in the output your provided:

Class-map: VOICE (match-any)
  3860628 packets, 1070196895 bytes
  5 minute offered rate 0 bps, drop rate 0 bps
  Match: protocol sip
    97348 packets, 49867304 bytes
    5 minute rate 0 bps
  Match: protocol rtp
    3763280 packets, 1020329591 bytes
    5 minute rate 0 bps
  Match: access-group name NEC-PBX
    0 packets, 0 bytes
    5 minute rate 0 bps
  Priority: 40% (340 kbps), burst bytes 8500, b/w exceed drops: 5

You can see that you're seeing packets under SIP and RTP, but not NEC-PBX. If you know you're getting SIP and RTP traffic across a link, you should see the packet counts increment and that's a reasonable way to know that your configuration is basically working.

Cisco ASA Routing Issue

Best practice wise - should I let the router or the ASA handle NAT (Overloading)?

In the most general of design best practices NAT is performed between an inside and outside network. NAT overloading is generally performed at the edge when there is limited public IP address space. You can learn more about NAT overloading, also known as Port Address Translation or PAT, in RFC 2663 (PAT is referred to as Network Address Port Translation (NAPT) in section 4.1.2).

In this particular scenario you can argue that you have two inside and outside networks and will need to perform some form of NAT on both the ASA (whether that is the NAT overloading you're using now, NAT exemption, static NAT, etc) and the Cisco Router.

I can ping the 172.16.2.2 interface but not 172.16.2.1 from a pc connected to one of the layer 2 switches (proves intervlan routing is working -- i have a 172.20.100.8 address on the PC). Why can't I ping 172.16.2.1 from a PC but I can from the Layer 3 Switch?

The ASA 172.16.2.2 is receiving the ICMP echo-request but does not have a route back to 172.20.100.0/27. The echo-reply is actually being forwarded to the Router 172.16.1.1 via the default route.

And most of all -- Why can't I get out to the Internet from the Layer 3 switch?

Currently your ASA and Cisco Router do not have routes to internal devices other than their connected routes.

Your ASA configuration:

route outside 0.0.0.0 0.0.0.0 172.16.1.1 1

This will provide a default route via the outside interface, but how will the ASA know how to reach subnets residing behind the Layer 3 Distribution Switch?

You'll need to add routes to the internal subnets via the inside interface using the Layer 3 Distribution Switch as the next-hop IP address.

ASA static routing example:

route inside 172.19.12.0 255.255.255.240 172.16.2.2
route inside 172.19.3.0 255.255.255.0 172.16.2.2
route inside 172.20.100.0 255.255.255.224 172.16.2.2

Your Cisco Router's configuration:

ip route 0.0.0.0 0.0.0.0 200.200.200.200

Additionally, how will your border router know how to reach subnets other than it's connected routes, and the catch all default route via the outside interface's next-hop address 200.200.200.200?

Router static routing example:

ip route 172.19.12.0 255.255.255.240 172.16.1.10
ip route 172.19.3.0 255.255.255.0 172.16.1.10
ip route 172.19.100.0 255.255.255.224 172.16.1.10
ip route 172.16.2.0 255.255.255.224 172.16.1.10