Intermittent periods of high packet loss, one hop beyond router

cablelayer1modempacket-losstroubleshooting

I've been trying to diagnose internet connection issues at a remote small business for which I am the system administrator. If this is not the proper Stack Exchange to post to, I'd really appreciate it if you could direct me to the correct forum. I come in peace!

Every few minutes, the internet drops out and no computer in the office can get a proper connection. The topography is as follows. All devices are connected by gigabit ethernet.

Time Warner Business Modem
└── Apple AirPort Extreme AC
    ├── Netgear Unmanaged Gigabit Switch 1
    │   ├── Apple iMac 1
    │   │   └── Canon Printer
    │   ├── Apple iMac 2
    │   ├── Apple iMac 3
    │   └── HP Ubuntu Laptop
    ├── Netgear Unmanaged Gigabit Switch 2
    │   ├── HP Ubuntu Desktop
    │   ├── Brother Printer 1
    │   └── Brother Printer 2
    └── Apple iMac 4

This problem began several weeks ago, and since then we have swapped the AirPort router for three other models, but the problem persisted.

To diagnose the issue, I've been running mtr (my traceroute) on several of the iMacs, pinging Google's Public DNS servers. I am also running mtr against the local network, having Apple iMac 1 ping HP Ubuntu Desktop, to try and rule out a local network issue with the wiring or Netgear switches. The local packets are never dropped, suggesting a fully stable connection. I'm including the IPv4 mtr session below. 10.0.0.1 is the AirPort's LAN IP, and the next IP address is Time Warner's server.

                        Packets               Pings
 Host                 Loss% Drop   Snt   Last   Avg  Best  Wrst StDev
 1. 10.0.0.1           0.0%    0 10722    0.4   0.3   0.2  10.6   0.8
 2. ##.###.##.##       7.9%  845 10722   22.1  21.7   7.7 292.0  14.2
 3. ##.###.##.##       7.9%  845 10722   13.8  20.2   8.4 459.2  13.2
 4. ##.###.##.##       7.9%  847 10722   22.3  22.0   8.9 374.5  13.2
 5. ##.###.##.##       7.9%  852 10722   29.1  24.7  10.4 290.1  13.4
 6. ##.###.##.##       7.9%  848 10722   23.3  25.2   8.3 643.9  15.3
 7. 66.109.6.163       7.9%  849 10722   15.1  23.3   9.0 554.9  17.2
 8. 66.110.96.53       7.9%  848 10722   26.3  21.3   7.9 467.2  14.4
 9. 72.14.195.232      7.9%  846 10721   28.0  21.9   8.9 402.2  14.7
10. ???
11. 108.170.238.201    7.9%  842 10721   19.1  21.8   8.8 498.9  14.5
12. 8.8.8.8            7.9%  844 10721   24.0  21.4   8.7 414.5  13.5

Every once in a while, the mtr sessions pinging Google DNS (on both IPv4 and IPv6) start dropping all packets for about a minute, after which everything starts working again. These periods of high packet loss coincide with the reported internet dropouts. This happens anywhere from 0-30 times per day. The packet loss begins with the first server beyond the router, in other words, the server that gives an IP address to the router from Time Warner Cable. Packets to the router are never dropped.

The Time Warner Modem is confirmed to be in bridge mode, and the AirPort successfully gets a WAN IP address, without any complaints (light is green, no double NAT, gets DNS servers, etc.). We pass all tests on test-ipv6.com.

Yesterday, we replaced the ethernet cable between the modem and the router because it appeared to be beaten up and kinked. The new cable is brand new, 10 gigabit with shielding, but the problem persists.

Last week, the mtr sessions reported only minimal packet loss, without any large periods of packet loss. The connection seemed stable. There doesn't seem to be any rhyme or reason to when it loses connectivity.

We have spoken with Time Warner Cable at length and they contend that they see no signal loss to our modem from their end, going back several months. Before this issue began several weeks ago, we have never had any issue with any of our equipment. All of the computers, routers, modems, and switches have been stable.

Now here's where it gets interesting — a month or so ago, the landlord installed a fire alarm backup power system in the same utility closet that houses the modem. It appears to power the strobe lights that would flash during a fire. The office manager reports that every few minutes, there is a distinct electrical clicking sound emanating from the power supply. I have not been on premises to witness this in person, but the office manager seems to believe that the clicking coincides with the internet dropout.

This is in part why I bought a shielded replacement cable, but it is my understanding that the two termination points have to be grounded for the shielding to work effectively, which they are not.

I am unsure of how to continue diagnosing this issue when there is no clear pattern. I am wondering if you know of any additional tests I should be running to rule out a local network issue (i.e. cabling, computers, router). I am comfortable with UNIX administration and can install and run whatever may be helpful (keep in mind the office is running macOS 10.13 and Ubuntu 14.04).

My second question is how likely is it that some sort of unknown electrical field is being discharged from the power supply, and if it is, is it more likely to be degrading the ethernet cable between the modem and router, or the modem itself? Is it worth putting together a makeshift Faraday cage to protect the modem? Should I be grounding the shielded cable on both ends? Or is this power supply a red herring? The router, I should note, is not in the same room, so is unlikely to be exposed to the same interference.

Unfortunately I am not able to travel to the office in the next few weeks, so I would prefer any advice on remote diagnosis.

Any ideas would be much appreciated. If anything is unclear, or you require more information, I'm happy to reply.

Best Answer

Shielded cable does not perform as well as unshielded cable if it is not properly grounded, and the shield will actually create problems by reflecting internal signals back into the cable. Unshielded cable should work correctly if you have more than 6" of separation from the power lines (probably a legal requirement, anyway).

You may have an electrical noise problem on the power for the modem, so a UPS would be in order to electrically isolate the modem and make sure it has clean power.

The cable modem itself is a consumer-grade device (off-topic here), and the ISP network is not under your direct control (off-topic here).

Related Topic