I'm getting increased latency and StDev due to route congestion and packet loss, but the forward and reverse paths go over distinct networks (e.g. one being init7.net, the other one being he.net), so, it's very difficult to understand which network or host is responsible for the congestion, packet loss, jitter and increased latency.
Is there a way to narrow down the blame after forward and reverse mtr fails to pinpoint the exact culprit, and the NOC@ contacts either don't respond, or claim to suffer no loss on the path in question? (I'm using OpenBSD.)
I've even tried doing an mtr directly to some customers of both of the two networks that might be experiencing the congestion, but couldn't really find any problems that way, especially since, for example, he.net has many POPs, and oftentimes distinct routes are taken between a given entry and exit POP, so when I try to mtr their hosts (like the tserv) directly at the exit POP to which I might be losing packets in their network, a different he.net path is taken to reach the very same POP, and no packet loss occurs, which proves nothing of interest (other than a possible suggestion that they might indeed overload some routes, whilst ensuring others stay uncongested, all whilst ignoring NOC@ requests from non-customers).
Best Answer
One way to do this is ICMP Timestamp, which is milliseconds from midnight UTC. It has the added benefit that you don't necessarily need to control both ends, as long as the far-end is not firewalled, there is good chance it'll work.
However, to have reliable one-way measurements, you need reliably same time in both ends. As ICMP timestamp only have precision of 1ms (which is not nearly enough for many applications, but sufficient for this) it's reasonably easy to find even non-cooperating hosts where ICMP timestamp will provide useful data.
If you control both ends, be sure that you are synchronizing NTP to only 1 server and same server. The absolute clock is not very important, it's just important that you experience as closely same time as possible.
If ICMP timestamp is not sufficient, it's very easy to write 10 lines of ruby/perl/python or even C to do measurements when you control both ends.
I can't really suggest software for doing ICMP timestamp measurements unidirectionally, hping2 supports sending ICMP timestamp but for some reason does not output unidirectional values. I wrote patch for hping2 to display one way latencies.