Downstream QoS Feasibility

qosvoip

I work for a VoIP provider, and one of the biggest hassles I deal with is downstream traffic management.

Our flagship product is QoS-enabled bonded T1 or SHDSL bundles configured to support our own VoIP service. We build QoS policies at our edge router that sort our voice traffic into the highest priority queue, and that product is very effective. When the circuit is built correctly, there's nothing the customer can do to interfere with their audio quality.

What's bothering me is that I've gotten a LOT of flack from other network engineers, IT techs, etc. who think that it's possible to make a service like this work without edge router QoS. I don't feel like this practical. I'd like to explain my understanding, and I hope somebody has the time to poke holes in it, if there are any. Pardon the lengthy question.

Let's take a Comcast circuit. I don't work for Comcast so I'm speculating on their network build, but I imagine it's something like this:

Comcast's fiber backs up to a Cisco edge router, and from there connects to their cable headend. Each customers IP is built out on the edge router with a given downstream bandwidth – say, 30mbit – which allocates them a queue of 30 megabits plus some percentage of overhead (I don't know how much this typically is).

Let's say the subscriber goes to download a 40mbyte file from a large enterprise server. They hit download, the TCP connection establishes and starts the slow-start process, but after a few seconds the server is pushing well over 30mbit of data. This fills up the queue on the edge router and it starts dropping packets.

At this point TCP congestion avoidance kicks in and starts slowing down the transfer to fit within the available bandwidth, but obviously voice is ruined already.

Now, I've been told that routers with inbound QoS policies are capable of leveraging the TCP congestion avoidance mechanism by dropping ACKs in order to rate-shape inbound traffic. I've never seen this done in practice, or at least never seen it work; is this a real thing? If so, is it effective? Wouldn't it result in a run of dropped packets every time that a new connection ramped up to speed, before the congestion algorithm could start slowing things down?

In addition to that, however, what if the traffic utilizing the circuit is largely UDP? If a request for a large amount of UDP data is sent to a server, it's naturally going to send the data as quickly as feasible, flood the queue and cause dropped voice packets – but maybe I'm overreacting to that scenario. Is UDP used for any high-throughput internet technologies these days?

More specifically, have you ever personally seen a QoS method that let VoIP work without constant quality problems on a frequently-saturated internet connection without ISP-provided QoS? Have you ever heard of a DSL or cable provider implementing edge router QoS for an application like this? I've only ever heard of that offered as part of MPLS products, is that pretty much a necessity if someone wants downstream QoS?

Best Answer

Now, I've been told that routers with inbound QoS policies are capable of leveraging the TCP congestion avoidance mechanism by dropping ACKs in order to rate-shape inbound traffic. I've never seen this done in practice, or at least never seen it work; is this a real thing? If so, is it effective? Wouldn't it result in a run of dropped packets every time that a new connection ramped up to speed, before the congestion algorithm could start slowing things down?

I doubt it, certainly for queues as these are relatively TCP friendly anway. Some routers might have tried this strategy to build tcp-friendly policers but I doubt this would be a decent solution. First of all, an ACK can be accompanied by data so you can't really only drop 'ACK's. Secondly, even if that is true for some streams, why would you drop only on the return path? For the TCP stream it's just as good to simply drop the original data and that way you don't waste bandwidth and processing resources between your edge router and receiver.

More likely is to have H-QoS, with a 30Mbps queue/scheduler on the circuit/user level and then multiple smaller-tier queues feeding into that. A simple setup could be to let both TCP and UDP feed into two different 25Mbps queues, so TCP can still leave 5Mbps for UDP and vice versa. Of course more complex variations exist.

Also, consider that ssthresh is usually adapted fairly quick towards your available bandwidth (here 30Mbps - your voice BW). While TCP does not stop increasing cwnd at ssthresh, it will only increase slowly afterwards and disturbing your voice stream only slightly when congestion is reached. However, at the start of the TCP connection, before ssthresh is adapted to a realistic value this will probably significantly disturb your voice flow (albeit shortly). Finally, while it might work okay with only a few TCP streams, when somebody starts 100s of TCP connections (e.g. P2P app) it gets more difficult. It's unlikely that all ssthresh and cwnd values stabilize (quickly) to guarantee fairness, even more if those connections are quite volatile (as is often the case with P2P).

Also, next to drops notice that a saturated connection also has a negative impact on your jitter, this might sometimes be just as problematic (or even worse) than a few packet drops.

In addition to that, however, what if the traffic utilizing the circuit is largely UDP? If a request for a large amount of UDP data is sent to a server, it's naturally going to send the data as quickly as feasible, flood the queue and cause dropped voice packets - but maybe I'm overreacting to that scenario. Is UDP used for any high-throughput internet technologies these days?

You're 100% correct, UDP has no fairness built-in. Many real-time multimedia applications (voip, video conferencing, ...) still use UDP. Tunnels (e.g. for VPN) also usually use UDP to avoid TCP-in-TCP retransmit timer conflicts. So yes, you certainly can have quite some high-throughput UDP traffic on a connection.

More specifically, have you ever personally seen a QoS method that let VoIP work without constant quality problems on a frequently-saturated internet connection without ISP-provided QoS? Have you ever heard of a DSL or cable provider implementing edge router QoS for an application like this? I've only ever heard of that offered as part of MPLS products, is that pretty much a necessity if someone wants downstream QoS?

Well, here the question is of course if many modern internet connections are frequently saturated. In my experience the available BW for a fixed connection often exceeds the average usage. There are always exceptions but usually the bulk of the people have loads of BW, it might still get saturated somewhere, but not that often on your direct access circuit. However, you might encounter it more often in mobile access as this is a relatively low-bandwidth low-reliability shared medium. Currently 4G helps a lot but that will only last this long as mobile BW usage is still growing heavily.

Yes, I have heard providers implementing QoS for multimedia applications. For which applications that might differ a lot from provider to provider. Assume they just qualify traffic as BE unless there is a good reason to do otherwise. But some might indifferently prioritize SIP/RTP.

MPLS is certainly not a necessity for QoS, any decent BNG today also includes quite significant QoS options.

Related Solutions

Cisco – QoS voice traffic on WAN of Cisco 1841

The short answer is no. Your queuing policy needs to be applied outbound at the point of bottleneck. In this case, the bottleneck is the WAN connection (20 Mbps) between the pfSense and the 1841. Thus, the proper location of your queuing policies are outbound on the LAN interface (a bit of a misnomer) of the pfSense and outbound on Fa0/01 of the 1841, which you have.

All that being said, your policy on the 1841 is flawed. In your example, you've reserved 1 Mbps of bandwidth for VoIP and applied the policy to a 100 Mbps interface. What happens when the PCs start pushing 70Mbps of traffic while trying to make a VoIP call? From a queuing standpoint on the 1841, the answer is nothing. There is only 70 Mbps of data flowing out Fa0/1. Therefore, there is always 1 Mbps available for VoIP. However, once this traffic reaches the pfSense, at least 50 Mbps of data and VoIP will be dropped. Your calls would be terrible at best.

Without getting into all of the details, and using your prior example, the policy should be something like this:

class-map match-any CM-VOICE-TRAFFIC
 match access-group 145
!
policy-map PM-PRIORITISE-VOICE-child
 class CM-VOICE-TRAFFIC
   set ip dscp ef
   priority 1000
 class class-default
   fair-queue
!
policy-map Shape-20Mb-parent
 class class-default
  shape average 20000000
  service-policy PM-PRIORITISE-VOICE-child
!
interface FastEthernet0/1
 service-policy output Shape-20Mb-parent
!
access-list 145 permit ip 10.0.59.0 0.0.0.255 any

This policy creates an artificial bandwidth limit on all traffic leaving the Fa0/1 on the 1841. Thus, the pfSense will never have the opportunity to drop traffic indiscriminately. Also, instead of the bandwidth command, the priority command should be used to limit latency and jitter. The tagging and the ACL change should be self-explanatory.

Cisco – QoS woes – managed IP VPN

To answer your questions:

RDP traffic should get up to the 25% of the remaining bandwidth. Where the already reserved bandwidth is the 35% ( class-default gets 25% by default and EF get 10% ). So, if i'm right, you assigned ~665Kbps to RDP. Anyway you should check if you're dropping packets issuing the command below:

show policy-map <your wan interface> output class REMOTEDESKTOP

and checking for dropped packets.

Cisco assign a queue to each user-defined class that includes the bandwidth or police commands. To make a long-story simple those commands define the amount of bandwidth assigned to every queue during congestions.
In theory every TCP based stream should be OK with drops. In practice some of them aren't. Dropping precedence bits tell the routers what packets should be dropped, within a given class, before congestion happens. Since RDP is the only type of traffic defined in your REMOTEDESKTOP class, you should not worry about it.
Bandwidth percentage are not in order ( as Jeremy stated ).

That said, there are a lot of things that i would change in your configuration:

There are no matches between some of classes of the setTos and the useTos policy-map. For instance the one named AF-HIGH is processing no packets since no class in setTos sets DSCP to AF41.
BE class in setTos is somehow self-defeating since it makes the class-default class useless. Note that class-default is the only system-defined class and get the 25% of the bandwidth by default ( 100 - max-reserved-bandwidth ) .
bandwidth remaining percent is not the best options ( as Jeremy explained ). I would change it to bandwidth percent.
I would prefer to mark EF packets by myself and not to rely on the phones' settings.

Best Answer

Related Solutions

Cisco – QoS voice traffic on WAN of Cisco 1841

Cisco – QoS woes – managed IP VPN

Related Topic