Understanding Traceroute and How to Use It

Definitions:
ICMP - Internet Control Message Protocol is provided within IP which generates error messages to help in diagnosing IP delivery problems.
TTL - Time to live can be viewed as a hop counter. A datagram starts with 30 hops, if it doesn't reach its destination by 30 hops, the sending node gets notified that the node or Host is Unreachable.
Tracert - Windows/NT built-in traceroute utility which is run in the dos window.
Traceroute Utility - From sled you can type traceroute (ip address or fully qualified node name)

Fundamentals of Internet Communications:
To understand traceroute, you must understand something fundamental about the nature of Internet communications. As you may know, the Internet gets data from one place to another by formatting it as a "packet," which is simply a chunk of data along with some routing information and a few other things attached in a specific format. The pieces of extra information pertinent to this discussion are the source IP address, destination IP address, and Time-To-Live (TTL), which is simply a number specifying how many "hops" the packet is allowed to take during its journey across the Internet. All data is passed across the network in the form of packets, and it is the job of the Internet's many "routers" to decide how to get packets from one place to another. The way it works is best illustrated with a step-by-step walk-through: When one computer has some information to send (a web page, perhaps), it puts it in a packet (or multiple packets), and sets the TTL to some initial value (typically 30). The computer then sends the packet to a router (Router A) which is directly connected, whether it be by Ethernet, PPP (dialup), or some other form of network. Router A would be called the "default gateway" for that computer.

Generally, all routers will be connected in some form to one or more OTHER routers. Router A will examine the packet to determine the destination IP address and will, based on that address, choose one of these other routers which will get the packet closer to the destination. This new router (Router B) is referred to as the "Next Hop". There are many ways that the router might be able to choose the most appropriate Next Hop intelligently, but a discussion of "routing protocols" is beyond the scope of this article. At any rate, Router A will then decrement the TTL by one, and forward the packet to the new router, Router B.

This process of "hopping" repeats indefinitely, with Router B choosing a new Next Hop, decrementing the TTL, and forwarding the packet, etc. until one of two things happens: 1) The packet reaches its destination, or 2) the TTL decrements to zero, or "times out." It's this last bit that allows traceroute to do its magic. We are now at a point where we can explain just what it is that traceroute is supposed to do.

Traceroute's Mission:
Traceroute's mission in life is to show you the path a packet sent from your machine to some other machine on the network takes as it hops from router to router. It will show you the IP address (and the actual name, usually) of each router, line-by-line.

It's time for an example: The time you'd want to use traceroute is when you can't reach some remote site. For example, after a bit of searching, we've found that I can't currently reach 207.13.21.254, and I want to know why. Using Windows 95/NT's built-in traceroute utility (called "tracert"), we can see the following output:

C:\>tracert 207.13.21.254

   traceroute to 207.13.21.254 (207.13.21.254), 30 hops max, 40 byte packets
 1  rtr-cne-e.gsfc.nasa.gov (128.183.50.1)  2 ms  1 ms  1 ms
 2  rtr-wan1-cf.gsfc.nasa.gov (128.183.251.1)  2 ms  1 ms  1 ms
 3  rtr-internet-ef.gsfc.nasa.gov (192.43.240.36)  6 ms  4 ms  4 ms
 4  sl-mae-e-f0-0.sprintlink.net (192.41.177.241)  7 ms  25 ms  18 ms
 5  sl-bb5-dc-6-1-0.sprintlink.net (144.232.8.25)  41 ms  47 ms *
 6  sl-bb3-dc-4-0-0-155M.sprintlink.net (144.232.0.6)  29 ms *  48 ms
 7  144.232.8.113 (144.232.8.113)  51 ms  43 ms  40 ms
 8  sl-bb1-atl-4-0-0-155M.sprintlink.net (144.232.1.198)  40 ms *  61 ms
 9  sl-bb5-fw-1-0-0.sprintlink.net (144.232.8.98)  86 ms  84 ms  79 ms
10  sl-bb1-fw-4-0-0-155M.sprintlink.net (144.232.1.150)  86 ms  85 ms *
11  * * sl-gw13-fw-0-0.sprintlink.net (144.228.30.17)  87 ms
12  sl-comp-3-0.sprintlink.net (144.228.137.14)  91 ms  90 ms *
13  cisco3640-1.cmpu.net (204.96.11.48)  95 ms  88 ms  90 ms
14  cisco7505-1.cmpu.net (204.96.11.1)  149 ms * *
15  cisco3640-1.cmpu.net (204.96.11.48)  113 ms  114 ms  113 ms
.
.
28  cisco7505-1.cmpu.net (204.96.11.1)  149 ms * *
29  cisco3640-1.cmpu.net (204.96.11.48)  113 ms  114 ms  113 ms
30  * cisco7505-1.cmpu.net (204.96.11.1)  111 ms *       Trace complete.

First of all, please note that we edited out lines 16 through 27. As you can see, for each hop, tracert shows us the hop number, three "round-trip" times, the name of the router (if available), and the router's IP address. Let's skip the round-trip times for a moment and examine the other things tracert is telling us.

It shows us that the first hop my packet takes from my desktop computer is to rtr-cne-e.gsfc.nasa.gov (which is one of CNE's routers). Tracert then tells us that for this particular destination (207.13.21.254), our router determines that the best Next Hop  is rtr-wan1-cf.gsfc.nasa.gov.  That router passes the packet to rtr-internet-ef.gsfc.nasa.gov, then it traverses to Sprintnet,
where it reaches hop 13, cisco3640-1.cmpu.net (204.96.11.48)  , which immediately forwards it to  cisco7505-1.cmpu.net (204.96.11.1) , and THIS is where we start to see problems: cisco7505-1.cmpu.net (204.96.11.1) passes it BACK to cisco3640-1.cmpu.net (204.96.11.48) and again, and again ...! This circular routing continues until the end of the traceroute (30 hops), clearly showing us the location of the routing failure (in cisco7505-1.cmpu.net), and why the packet is never reaching its destination. Unfortunately, there's nothing we at CNE can do to fix this problem, as it is external to our network. Internal, yes, we can contact the owner of the offending router.

HOW DOES IT WORK???
Just a couple more important things about traceroute: First, HOW DOES IT WORK??? You may recall ICMP (Internet Control Message Protocol), and also that we said that traceroute relies on packet time-outs. To determine the first hop, traceroute simply sends a packet to the destination, but instead of setting the TTL to 30, as usual, it sets the TTL to 1, meaning the packet is only allowed to take one hop. It takes that hop, arriving at the "hop 1" router, which faithfully decrements the TTL and notices that it is now zero. Whenever a router sees the TTL go to zero, it does two things: it 1) drops the packet (NOT forwarding it at all), and 2) sends an ICMP packet BACK to the source IP address of the dropped packet. This ICMP message says that the "TTL Expired in Transit." The source IP address of the ICMP message IS the address of the router itself, so traceroute can now see the IP address of the router at hop 1. Windows 95/NT's tracert (like most) performs this little test three times for each hop (by default), also keeping track of how long it takes for each error message to return. This is how it gets the round-trip times (in milliseconds), which, like ping, can be used to gauge which routers may be slowing things down. These numbers should be taken with a grain of salt, because many routers assign a low priority to the ICMP reply, which can cause a short delay. Note also that these times are cumulative, so it's really the differences between the average for each router that's important. Incidentally, the "*" in hop 5 indicates a packet which received no ICMP response at all. This is a result of normal packet-loss on the network.

Example:

So, here is how it works:

Since hosts do not return ICMP "time exceeded" messages, traceroute needs another method for realizing it has reached the final destination. It sends UDP datagrams to a very high number port (33434 by default) that is unlikely to have an application running on it. This causes the destination host to generate an ICMP "port unreachable" error message, and traceroute then interprets this as the signal that it has reached the destination host.

The last caveat we should mention is this: the path BACK from the remote machine to you is NOT NECESSARILY the same! This situation is called asymmetric routing, and is usually not an issue. In cases where it does exist though, it can make troubleshooting much more difficult, because some problem may exist on the reverse path (preventing data from getting back to you) which traceroute cannot show you. Some traceroute implementations support a feature which allows to do a trick so that you CAN see this information (Hint: take a look at tracert's "-j" option).

So, if you encounter a problem with a node on-site, run a traceroute using tracert or some other utility. By doing a traceroute you can determine if a node is reachable, the general throughput of the network; and if the node is not reachable, where the failure is occurring. Additionally, if someone complains they can't get offsite try tracerouting a node offsite like 128.8.10.14, which is one of our backup nameservers. Also, try an external traceroute if someone complains they can't reach outside our network. Here's an url to use when doing an external traceroute

http://nucleus.nectec.or.th/pub/info/netinfo/routing.arbiter/tools/trace.html
 

Possible Problems:

Strange thing 1:
Traceroute Hangs:
Quick 'n' Dirty: traceroute -n

Normally when you do a traceroute to a domain name or IP address, traceroute will try to look up the
DNS hostnames of the IP address at each hop. If there is any problem with the DNS server, your
traceroute will "hang". This can throw you off track.

Instead, use traceroute -n. This means "IP numbers only", and any hangs or problems that show up in the
traceroute will be entirely due to routing -- which is what you're troubleshooting.
Here's an example:

           _fairy.tlg.net[~]> traceroute -n gw1-sj-tlg
              traceroute to gw1-sj-tlg.tlg.net (140.174.74.1), 30 hops max, 40 byte packets
               1  140.174.77.5  2.2 ms  2.119 ms  2.257 ms
               2  140.174.178.1  7.593 ms  4.177 ms  26.672 ms
               3  140.174.125.5  6.958 ms  49.766 ms  17.813 ms
               4  140.174.161.2  12.194 ms *  71.78 ms

Here's what the same traceroute looks like with DNS lookups:

              _fairy.tlg.net[~]> traceroute gw1-sj-tlg
              traceroute to gw1-sj-tlg.tlg.net (140.174.161.2), 30 hops max, 40 byte packets
               1  gw1-ms-tlg (140.174.77.5)  2.354 ms  2.191 ms  4.617 ms
               2  ln1_gw2-sf-tlg_ms (140.174.178.1)  29.144 ms  4.482 ms  4.305 ms
               3  border-sf-tlg (140.174.125.5)  5.013 ms  4.663 ms  5.071 ms
               4  border-sj-tlg (140.174.161.2)  13.748 ms *  12.346 ms

Notice that "border-sj-tlg" and "gw1-sj-tlg" are the same IP address. This is set up in the DNS for that IP address.

Strange thing 2:
Note that lines 2 & 3 are the same.  This is due to a buggy kernel
on the 2nd hop system that forwards packets with
a zero ttl (a bug in the distributed version of 4.3BSD).

             1  helios.ee.lbl.gov (128.3.112.1)  19 ms  19 ms  0 ms
             2  lilac-dmc.Berkeley.EDU (128.32.216.1)  39 ms  39 ms  19 ms
             3  lilac-dmc.Berkeley.EDU (128.32.216.1)  39 ms  39 ms  19 ms
             4  ccngw-ner-cc.Berkeley.EDU (128.32.136.23)  39 ms  40 ms  39

Strange thing 3:

!H, means that the router at that hop doesn't know anything about the target address.

!N, !P, means you got a  host, network or protocol unreachable, respectively.

!S or !F means the source route failed or fragmentation needed - neither of these
     should ever occur and the associated gateway is busted if you see
     one.

One last item:

Sometimes you will see a traceroute that display the following information:
 
 1  rtr-cne-e.gsfc.nasa.gov (128.183.50.1)  2 ms  1 ms  1 ms
 2  rtr-wan1-cf.gsfc.nasa.gov (128.183.251.1)  2 ms  1 ms  1 ms
 3  rtr-internet-ef.gsfc.nasa.gov (192.43.240.36)  6 ms  4 ms  4 ms
 4  sl-mae-e-f0-0.sprintlink.net (192.41.177.241)  7 ms  25 ms  18 ms
 5  sl-bb5-dc-6-1-0.sprintlink.net (144.232.8.25)  41 ms  47 ms *
 6  sl-bb3-dc-4-0-0-155M.sprintlink.net (144.232.0.6)  29 ms *  48 ms
 7  144.232.8.113 (144.232.8.113)  51 ms  43 ms  40 ms
 8  sl-bb1-atl-4-0-0-155M.sprintlink.net (144.232.1.198)  40 ms *  61 ms
 9  sl-bb5-fw-1-0-0.sprintlink.net (144.232.8.98)  86 ms  84 ms  79 ms
10  sl-bb1-fw-4-0-0-155M.sprintlink.net (144.232.1.150)  86 ms  85 ms *
11 * * *
12 * * *
13 * * *
14 * * *
15 sl-comp-3-0.sprintlink.net (144.228.137.14)  91 ms  90 ms *
16 cisco3640-1.cmpu.net (204.96.11.48)  95 ms  88 ms  90 ms

So, what's happening here. For some reason, probably the machine after sl-bbl-fw, the machine is not getting its ICMP packet back to the client requesting the traceroute. The traceroute circumvents the problem by bumping the ttl value up 1 hop until it gets a response or until the TTL reaches 30 hops.