Diagnosing your server’s internet issues with traceroute

The traceroute command, available on all Linux and OS X machines (or tracert on Windows), gives deep insight into how a packet travels from one location to another across the internet.

Each stop along the way is called a hop, and these hops—particularly when they fail—can be used to identify and troubleshoot networking errors. In fact, we request that support tickets for networking issues contain traceroute output for packets both to and from the server in question.

Failed hops are represented by three asterisks: * * *.

Here’s an example traceroute between my local machine and one of my SSD Nodes servers (anonymized for security):

traceroute 172.93.███.███
traceroute to 172.93.███.███ (172.93.███.███), 30 hops max, 60 byte packets
 1  gateway (192.168.1.1)  2.091 ms  2.245 ms  2.205 ms
 2  10.80.0.1 (10.80.0.1)  10.608 ms  11.540 ms  11.529 ms
 3  100.127.71.194 (100.127.71.194)  12.550 ms  12.506 ms  12.481 ms
 4  72.215.229.20 (72.215.229.20)  14.651 ms  14.609 ms  13.227 ms
 5  lag-157.bear2.Phoenix1.Level3.net (4.28.82.53)  17.735 ms  18.735 ms  18.685 ms
 6  * * *
 7  phx-b1-link.telia.net (62.115.57.89)  17.649 ms  14.004 ms  13.571 ms
 8  dls-b21-link.telia.net (62.115.136.218)  39.959 ms dls-b21-link.telia.net (62.115.135.14)  38.963 ms  39.808 ms
 9  modmission-ic-317651-dls-b21.c.telia.net (62.115.145.115)  39.834 ms  35.792 ms  36.715 ms
10  * * *
11  * * *
12  Dallas-TX.r19.Pwr-1xPDU-1xATS-2xUPS.incero.com (144.168.34.82)  41.062 ms  38.485 ms  37.538 ms
13  172.93.50.7 (172.93.50.7)  36.507 ms  40.201 ms  36.710 ms
14  172.93.███.███ (172.93.███.███)  36.714 ms  35.468 ms  35.240 ms

And the same destination, but while connected to my VPN:

$ traceroute 172.93.███.███
traceroute to 172.93.███.███ (172.93.███.███), 30 hops max, 60 byte packets
 1  gateway (10.8.0.1)  38.085 ms  38.327 ms  38.315 ms
 2  172.93.50.7 (172.93.50.7)  38.308 ms  38.299 ms  38.292 ms
 3  172.93.███.███ (172.93.███.███)  38.285 ms  38.277 ms  38.269 ms

As you can see, there are major differences between the two, and much to learn if something is going wrong, especially intermittently.

Adding in `mtr`

The mtr command adds a little more detail to traceroute, such as the time it takes for the packet to travel between hops, and the percentage of packet loss. This can help you understand where things are slowing down or stopping altogether.

To send 10 packets and report the results, use mtr --report IP-ADDRESS:

$ mtr --report 172.93.███.███
Start: Fri May  5 11:31:43 2017
HOST: localhost.localdomain       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- gateway                    0.0%    10    1.6   1.6   1.5   1.7   0.0
  2.|-- 10.80.0.1                  0.0%    10   11.3   9.8   8.3  11.6   1.1
  3.|-- 100.127.71.194             0.0%    10    8.6  10.1   8.6  13.8   1.3
  4.|-- 72.215.229.20              0.0%    10   11.7  13.0  11.2  19.2   2.2
  5.|-- lag-157.bear2.Phoenix1.Le 20.0%    10   12.4  13.5  12.1  18.5   2.0
  6.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  7.|-- phx-b1-link.telia.net      0.0%    10   14.1  13.9  13.3  14.9   0.5
  8.|-- dls-b21-link.telia.net     0.0%    10   34.5  36.6  34.2  42.5   3.1
  9.|-- modmission-ic-317651-dls-  0.0%    10   35.0  35.6  34.1  39.3   1.5
 10.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 11.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 12.|-- Dallas-TX.r19.Pwr-1xPDU-1  0.0%    10   36.1  36.9  35.6  39.7   1.0
 13.|-- 172.93.50.7                0.0%    10   38.1  36.5  34.9  38.1   0.8
 14.|-- 172.93.███.███             0.0%    10   36.4  36.2  35.2  37.0   0.3