Measuring packet loss caused by bad reception or interference

However, when I move to the remote location, even though the wifi signal strength is either 3/4 or 4/4 bars, the packet loss is massive, and the latency varies wildly where for a stretch it will report pings of about 50ms, but then will drop to 800+ and about 10% are just timed out packets altogether.

if the signal strength is good, why would this distance create such horrible packet loss?

Signal strength is only one metric... also consider signal to noise ratio, which is often the problem for scenarios like this.

Wifi latency and packet loss are cousins of each other. 802.11 frames contain a sequence number that is ACK'd... if the sequence number isn't ACK'd (due to loss or a bit error in the original frame), then the sender attempts to retransmit the frame a certain number of times. These 802.11 retransmissions show up as increased latency or outright packet loss if the interference is bad enough.

I have literally seen 802.11g latency that is over 40 seconds (yes... seconds) when I'm only 50 feet from the LWAP. That particular environment had a lot of tools that also operated in the 2.4GHz bands, so obviously the potential for errors was quite high.

what could cause such interference? I'm in a residential neighborhood, but there is absolutely nothing, line-of-sight between the laptop and the signal repeater other than 2 walls (with whatever electrical wiring is in them).

Wifi operates in (mostly) open spectrum bands from the FCC... bluetooth, microwave ovens, phones, toy cars, we can only speculate about the source of the interference.

You could try using a directed antenna with a focused beam (i.e. a yagi, or a cantenna) on your stations... those might help if the interference is not in the direct path to your wifi source.

Finally, if you have a wireless sniffer or access to a linux system (suggestion: Backtrack Linux LiveCD) then you can diagnose your wifi problems with Wireshark / tshark. Cisco also has a good reference for Wireshark 802.11 display filters, which help filter out noise so you can focus on the problems at hand.

Interface Internal-Data0/0 "", is up, line protocol is up
     2749335943 input errors, 0 CRC, 0 frame, 2749335943 overrun, 0 ignored, 0 abort
                                              ^^^^^^^^^^^^^^^^^^
         0 output errors, 0 collisions, 0 interface resets

You show overruns on the InternalData interfaces, so you are dropping traffic through the ASA. With that many drops, it's not hard to imagine that this is contributing to problem. Overruns happen when the internal Rx FIFO queues overflow (normally because of some problem with load).

EDIT to respond to a question in the comments:

I don't understand why the firewall is overloaded, it is not close to using 10Gbps. Can you explain why we are seeing overruns even when the CPU and bandwidth are low? The CPU is about 5% and the bandwidth either direction never goes much higher than 1.4Gbps.

I have seen this happen over and over when a link is seeing traffic microbursts, which exceed either the bandwidth, connection-per-second, or packet-per-second horsepower of the device. So many people quote 1 or 5 minute statistics as if the traffic is relatively constant across that timeframe.

I would take a look at your firewall by running these commands every two or three seconds (run term pager 0 to avoid paging issues)...

show clock
show traffic detail | i ^[a-zA-Z]|overrun|packets dropped
show asp drop

Now graph out how much traffic you're seeing every few seconds vs drops; if you see massive spikes in policy drops or overruns when your traffic spikes, then you're closer to finding the culprit.

Don't forget that you can sniff directly on the ASA with this if you need help identifying what's killing the ASA... you have to be quick to catch this sometimes.

capture FOO circular-buffer buffer <buffer-size> interface <intf-name>

Netflow on your upstream switches could help as well.