When you see "!Z" or "!X" in a traceroute, what does that mean?
Where is it coming from, and how do I fix it?
Traceroute – Meaning of “!Z” and “!X” in Traceroute
firewalllinuxnetworking
Related Solutions
IP subnets exist to allow routers to choose appropriate destinations for packets. You can use IP subnets to break up larger networks for logical reasons (firewalling, etc), or physical need (smaller broadcast domains, etc).
Simply put, though, IP routers use your IP subnets to make routing decisions. Understand how those decisions work, and you can understand how to plan IP subnets.
Counting to 1
If you are already fluent in binary (base 2) notation you can skip this section.
For those of you who are left: Shame on you for not being fluent in binary notation!
Yes, that may be a bit harsh. It's really, really easy to learn to count in binary, and to learn shortcuts to convert binary to decimal and back. You really should know how to do it.
Counting in binary is so simple because you only have to know how to count to 1!
Think of a car's "odometer", except that unlike a traditional odometer each digit can only count up to 1 from 0. When the car is fresh from the factory the odometer reads "00000000".
When you've driven your first mile the odometer reads "00000001". So far, so good.
When you've driven your second mile the first digit of the odometer rolls back over to "0" (since its maximum value is "1") and the second digit of the odometer rolls over to "1", making the odometer read "00000010". This looks like the number 10 in decimal notation, but it's actually 2 (the number of miles you've driven the car so far) in binary notation.
When you've driven the third mile the odometer reads "00000011", since the first digit of the odometer turns again. The number "11", in binary notation, is the same as the decimal number 3.
Finally, when you've driven your fourth mile both digits (which were reading "1" at the end of the third mile) roll back over to zero position, and the 3rd digit rolls up to the "1" position, giving us "00000100". That's the binary representation of the decimal number 4.
You can memorize all of that if you want, but you really only need to understand how the little odometer "rolls over" as the number it's counting gets bigger. It's exactly the same as a traditional decimal odometer's operation, except that each digit can only be "0" or "1" on our fictional "binary odometer".
To convert a decimal number to binary you could roll the odometer forward, tick by tick, counting aloud until you've rolled it a number of times equal to the decimal number you want to convert to binary. Whatever is displayed on the odometer after all that counting and rolling would be the binary representation of the decimal number you counted up to.
Since you understand how the odometer rolls forward you'll also understand how it rolls backward, too. To convert a binary number displayed on the odometer back to decimal you could roll the odometer back one tick at a time, counting aloud until the odometer reads "00000000". When all that counting and rolling is done, the last number you say aloud would be the decimal representation of the binary number the odometer started with.
Converting values between binary and decimal this way would be very tedious. You could do it, but it wouldn't be very efficient. It's easier to learn a little algorithm to do it faster.
A quick aside: Each digit in a binary number is known as a "bit". That's "b" from "binary" and "it" from "digit". A bit is a bnary digit.
Converting a binary number like, say, "1101011" to decimal is a simple process with a handy little algorithm.
Start by counting the number of bits in the binary number. In this case, there are 7. Make 7 divisions on a sheet of paper (in your mind, in a text file, etc) and begin filling them in from right to left. In the rightmost slot, enter the number "1", because we'll always start with "1". In the next slot to the left enter double the value in the slot to the right (so, "2" in the next one, "4" in the next one) and continue until all the slots are full. (You'll end up memorizing these numbers, which are the powers of 2, as you do this more and more. I'm alright up to 131,072 in my head but I usually need a calculator or paper after that).
So, you should have the following on your paper in your little slots.
64 | 32 | 16 | 8 | 4 | 2 | 1 |
Transcribe the bits from the binary number below the slots, like so:
64 | 32 | 16 | 8 | 4 | 2 | 1 |
1 1 0 1 0 1 1
Now, add some symbols and compute the answer to the problem:
64 | 32 | 16 | 8 | 4 | 2 | 1 |
x 1 x 1 x 0 x 1 x 0 x 1 x 1
--- --- --- --- --- --- ---
+ + + + + + =
Doing all the math, you should come up with:
64 | 32 | 16 | 8 | 4 | 2 | 1 |
x 1 x 1 x 0 x 1 x 0 x 1 x 1
--- --- --- --- --- --- ---
64 + 32 + 0 + 8 + 0 + 2 + 1 = 107
That's got it. "1101011" in decimal is 107. It's just simple steps and easy math.
Converting decimal to binary is just as easy and is the same basic algorithm, run in reverse.
Say that we want to convert the number 218 to binary. Starting on the right of a sheet of paper, write the number "1". To the left, double that value (so, "2") and continue moving toward the left of the paper doubling the last value. If the number you are about to write is greater than the number being converted stop writing. otherwise, continue doubling the prior number and writing. (Converting a big number, like 34,157,216,092, to binary using this algorithm can be a bit tedious but it's certainly possible.)
So, you should have on your paper:
128 | 64 | 32 | 16 | 8 | 4 | 2 | 1 |
You stopped writing numbers at 128 because doubling 128, which would give you 256, would be large than the number being converted (218).
Beginning from the leftmost number, write "218" above it (128) and ask yourself: "Is 218 larger than or equal to 128?" If the answer is yes, scratch a "1" below "128". Above "64", write the result of 218 minus 128 (90).
Looking at "64", ask yourself: "Is 90 larger than or equal to 64?" It is, so you'd write a "1" below "64", then subtract 64 from 90 and write that above "32" (26).
When you get to "32", though, you find that 32 is not greater than or equal to 26. In this case, write a "0" below "32", copy the number (26) from above 32" to above "16" and then continue asking yourself the same question with the rest of the numbers.
When you're all done, you should have:
218 90 26 26 10 2 2 0
128 | 64 | 32 | 16 | 8 | 4 | 2 | 1 |
1 1 0 1 1 0 1 0
The numbers at the top are just notes used in computation and don't mean much to us. At the bottom, though, you see a binary number "11011010". Sure enough, 218, converted to binary, is "11011010".
Following these very simple procedures you can convert binary to decimal and back again w/o a calculator. The math is all very simple and the rules can be memorized with just a bit of practice.
Splitting Up Addresses
Think of IP routing like pizza delivery.
When you're asked to deliver a pizza to "123 Main Street" it's very clear to you, as a human, that you want to go to the building numbered "123" on the street named "Main Street". It's easy to know that you need to go to the 100-block of Main Street because the building number is between 100 and 199 and most city blocks are numbered in hundreds. You "just know" how to split the address up.
Routers deliver packets, not pizza. Their job is the same as a pizza driver: To get the cargo (packets) as close to the destination as possible. A router is connected to two or more IP subnets (to be at all useful). A router must examine destination IP addresses of packets and break those destination addresses up into their "street name" and "building number" components, just like the pizza driver, to make decisions about delivery.
Each computer (or "host") on an IP network is configured with a unique IP address and subnet mask. That IP address can be divided up into a "building number" component (like "123" in the example above) called the "host ID" and a "street name" component (like "Main Street" in the example above) called the "network ID". For our human eyes, it's easy to see where the building number and the street name are in "123 Main Street", but harder to see that division in "10.13.216.41 with a subnet mask of 255.255.192.0".
IP routers "just know" how to split up IP addresses into these component parts to make routing decisions. Since understanding how IP packets are routed hinges on understanding this process, we need to know how to break up IP addresses, too. Fortunately, extracting the host ID and the network ID out of an IP address and subnet mask is actually pretty easy.
Start by writing out the IP address in binary (use a calculator if you haven't learned to do this in your head just yet, but make a note learn how to do it-- it's really, really easy and impresses the opposite sex at parties):
10. 13. 216. 41
00001010.00001101.11011000.00101001
Write out the subnet mask in binary, too:
255. 255. 192. 0
11111111.11111111.11000000.00000000
Written side-by-side, you can see that the point in the subnet mask where the "1s" stop "lines up" to a point in the IP address. That's the point that the network ID and the host ID split. So, in this case:
10. 13. 216. 41
00001010.00001101.11011000.00101001 - IP address
11111111.11111111.11000000.00000000 - subnet mask
00001010.00001101.11000000.00000000 - Portion of IP address covered by 1s in subnet mask, remaining bits set to 0
00000000.00000000.00011000.00101001 - Portion of IP address covered by 0s in subnet mask, remaining bits set to 0
Routers use the subnet mask to "mask out" the bits covered by 1s in the IP address (replacing the bits that are not "masked out" with 0s) to extract the network ID:
10. 13. 192. 0
00001010.00001101.11000000.00000000 - Network ID
Likewise, by using the subnet mask to "mask out" the bits covered by 0s in the IP address (replacing the bits that are not "masked out" with 0s again) a router can extract the host ID:
0. 0. 24. 41
00000000.00000000.00011000.00101001 - Portion of IP address covered by 0s in subnet mask, remaining bits set to 0
It's not as easy for our human eyes to see the "break" between the network ID and the host ID as it is between the "building number" and the "street name" in physical addresses during pizza delivery, but the ultimate effect is the same.
Now that you can split up IP addresses and subnet masks into host ID's and network ID's you can route IP just like a router does.
More Terminology
You're going to see subnet masks written all over the Internet and throughout the rest of this answer as (IP/number). This notation is known as "Classless Inter-Domain Routing" (CIDR) notation. "255.255.255.0" is made up of 24 bits of 1s at the beginning, and it's faster to write that as "/24" than as "255.255.255.0". To convert a CIDR number (like "/16") to a dotted-decimal subnet mask just write out that number of 1s, split it into groups of 8 bits, and convert it to decimal. (A "/16" is "255.255.0.0", for instance.)
Back in the "old days", subnet masks weren't specified, but rather were derived by looking at certain bits of the IP address. An IP address starting with 0 - 127, for example, had an implied subnet mask of 255.0.0.0 (called a "class A" IP address).
These implied subnet masks aren't used today and I don't recommend learning about them anymore unless you have the misfortune of dealing with very old equipment or old protocols (like RIPv1) that don't support classless IP addressing. I'm not going to mention these "classes" of addresses further because it's inapplicable today and can be confusing.
Some devices use a notation called "wildcard masks". A "wildcard mask" is nothing more than a subnet mask with all 0s where there would be 1s, and 1s where there would be 0s. The "wildcard mask" of a /26 is:
11111111.11111111.11111111.11000000 - /26 subnet mask
00000000.00000000.00000000.00111111 - /26 "wildcard mask"
Typically you see "wildcard masks" used to match host IDs in access-control lists or firewall rules. We won't discuss them any further here.
How a Router Works
As I've said before, IP routers have a similar job to a pizza delivery driver in that they need to get their cargo (packets) to its destination. When presented with a packet bound for address 192.168.10.2, an IP router needs to determine which of its network interfaces will best get that packet closer to its destination.
Let's say that you are an IP router, and you have interfaces connected to you numbered:
- Ethernet0 - 192.168.20.1, subnet mask /24
- Ethernet1 - 192.168.10.1, subnet mask /24
If you receive a packet to deliver with a destination address of "192.168.10.2", it's pretty easy to tell (with your human eyes) that the packet should be sent out the interface Ethernet1, because the Ethernet1 interface address corresponds to the packet's destination address. All the computers attached to the Ethernet1 interface will have IP addresses starting with "192.168.10.", because the network ID of the IP address assigned to your interface Ethernet1 is "192.168.10.0".
For a router, this route selection process is done by building a routing table and consulting the table each time a packet is to be delivered. A routing table contains network ID and destination interface names. You already know how to obtain a network ID from an IP address and subnet mask, so you're on your way to building a routing table. Here's our routing table for this router:
- Network ID: 192.168.20.0 (11000000.10101000.00010100.00000000) - 24 bit subnet mask - Interface Ethernet0
- Network ID: 192.168.10.0 (11000000.10101000.00001010.00000000) - 24 bit subnet mask - Interface Ethernet1
For our incoming packet bound for "192.168.10.2", we need only convert that packet's address to binary (as humans -- the router gets it as binary off the wire to begin with) and attempt to match it to each address in our routing table (up to the number of bits in the subnet mask) until we match an entry.
- Incoming packet destination: 11000000.10101000.00001010.00000010
Comparing that to the entries in our routing table:
11000000.10101000.00001010.00000010 - Destination address for packet
11000000.10101000.00010100.00000000 - Interface Ethernet0
!!!!!!!!.!!!!!!!!.!!!????!.xxxxxxxx - ! indicates matched digits, ? indicates no match, x indicates not checked (beyond subnet mask)
11000000.10101000.00001010.00000010 - Destination address for packet
11000000.10101000.00001010.00000000 - Interface Ethernet1, 24 bit subnet mask
!!!!!!!!.!!!!!!!!.!!!!!!!!.xxxxxxxx - ! indicates matched digits, ? indicates no match, x indicates not checked (beyond subnet mask)
The entry for Ethernet0 matches the first 19 bits fine, but then stops matching. That means it's not the proper destination interface. You can see that the interface Ethernet1 matches 24 bits of the destination address. Ah, ha! The packet is bound for interface Ethernet1.
In a real-life router, the routing table is sorted in such a manner that the longest subnet masks are checked for matches first (i.e. the most specific routes), and numerically so that as soon as a match is found the packet can be routed and no further matching attempts are necessary (meaning that 192.168.10.0 would be listed first and 192.168.20.0 would never have been checked). Here, we're simplifying that a bit. Fancy data structures and algorithms make faster IP routers, but simple algorithms will produce the same results.
Static Routes
Up to this point, we've talked about our hypothetical router as having networks directly connected to it. That's not, obviously, how the world really works. In the pizza-driving analogy, sometimes the driver isn't allowed any further into the building than the front desk, and has to hand-off the pizza to somebody else for delivery to the final recipient (suspend your disbelief and bear with me while I stretch my analogy, please).
Let's start by calling our router from the earlier examples "Router A". You already know RouterA's routing table as:
- Network ID: 192.168.20.0 (11000000.10101000.00010100.00000000) - subnet mask /24 - Interface RouterA-Ethernet0
- Network ID: 192.168.10.0 (11000000.10101000.00001010.00000000) - subnet mask /24 - Interface RouterA-Ethernet1
Suppose that there's another router, "Router B", with the IP addresses 192.168.10.254/24 and 192.168.30.1/24 assigned to its Ethernet0 and Ethernet1 interfaces. It has the following routing table:
- Network ID: 192.168.10.0 (11000000.10101000.00001010.00000000) - subnet mask /24 - Interface RouterB-Ethernet0
- Network ID: 192.168.30.0 (11000000.10101000.00011110.00000000) - subnet mask /24 - Interface RouterB-Ethernet1
In pretty ASCII art, the network looks like this:
Interface Interface
Ethernet1 Ethernet1
192.168.10.1/24 192.168.30.254/24
__________ V __________ V
| | V | | V
----| ROUTER A |------- /// -------| ROUTER B |----
^ |__________| ^ |__________|
^ ^
Interface Interface
Ethernet0 Ethernet0
192.168.20.1/24 192.168.10.254/24
You can see that Router B knows how to "get to" a network, 192.168.30.0/24, that Router A knows nothing about.
Suppose that a PC with the IP address 192.168.20.13 attached to the network connected to router A's Ethernet0 interface sends a packet to Router A for delivery. Our hypothetical packet is destined for the IP address 192.168.30.46, which is a device attached to the network connected to the Ethernet1 interface of Router B.
With the routing table shown above, neither entry in Router A's routing table matches the destination 192.168.30.46, so Router A will return the packet to the sending PC with the message "Destination network unreachable".
To make Router A "aware" of the existence of the 192.168.30.0/24 network, we add the following entry to the routing table on Router A:
- Network ID: 192.168.30.0 (11000000.10101000.00011110.00000000) - subnet mask /24 - Accessible via 192.168.10.254
In this way, Router A has a routing table entry that matches the 192.168.30.46 destination of our example packet. This routing table entry effectively says "If you get a packet bound for 192.168.30.0/24, send it on to 192.168.10.254 because he knows how to deal with it." This is the analogous "hand-off the pizza at the front desk" action that I mentioned earlier-- passing the packet on to somebody else who knows how to get it closer to its destination.
Adding an entry to a routing table "by hand" is known as adding a "static route".
If Router B wants to deliver packets to the 192.168.20.0 subnet mask 255.255.255.0 network, it will need an entry in its routing table, too:
- Network ID: 192.168.20.0 (11000000.10101000.00010100.00000000) - subnet mask /24 - Accessible via: 192.168.10.1 (Router A's IP address in the 192.168.10.0 network)
This would create a path for delivery between the 192.168.30.0/24 network and the 192.168.20.0/24 network across the 192.168.10.0/24 network between these routers.
You always want to be sure that routers on both sides of such an "interstitial network" have a routing table entry for the "far end" network. If router B in our example didn't have a routing table entry for "far end" network 192.168.20.0/24 attached to router A our hypothetical packet from the PC at 192.168.20.13 would get to the destination device at 192.168.30.46, but any reply that 192.168.30.46 tried to send back would be returned by router B as "Destination network unreachable." One-way communication is generally not desirable. Always be sure you think about traffic flowing in both directions when you think about communication in computer networks.
You can get a lot of mileage out of static routes. Dynamic routing protocols like EIGRP, RIP, etc, are really nothing more than a way for routers to exchange routing information between each other that could, in fact, be configured with static routes. One large advantage to using dynamic routing protocols over static routes, though, is that dynamic routing protocols can dynamically change the routing table based on network conditions (bandwidth utilization, an interface "going down", etc) and, as such, using a dynamic routing protocol can result in a configuration that "routes around" failures or bottlenecks in the network infrastructure. (Dynamic routing protocols are WAY outside the scope of this answer, though.)
You Can't Get There From Here
In the case of our example Router A, what happens when a packet bound for "172.16.31.92" comes in?
Looking at the Router A routing table, neither destination interface or static route matches the first 24 bits of 172.18.31.92 (which is 10101100.00010010.00011111.01011100, by the way).
As we already know, Router A would return the packet to the sender via a "Destination network unreachable" message.
Say that there's another router (Router C) sitting at the address "192.168.20.254". Router C has a connection to the Internet!
Interface Interface Interface
Ethernet1 Ethernet1 Ethernet1
192.168.20.254/24 192.168.10.1/24 192.168.30.254/24
__________ V __________ V __________ V
(( heap o )) | | V | | V | | V
(( internet )) ----| ROUTER C |------- /// -------| ROUTER A |------- /// -------| ROUTER B |----
(( w00t! )) ^ |__________| ^ |__________| ^ |__________|
^ ^ ^
Interface Interface Interface
Ethernet0 Ethernet0 Ethernet0
10.35.1.1/30 192.168.20.1/24 192.168.10.254/24
It would be nice if Router A could route packets that do not match any local interface up to Router C such that Router C can send them on to the Internet. Enter the "default gateway" route.
Add an entry at the end of our routing table like this:
- Network ID: 0.0.0.0 (00000000.00000000.00000000.00000000) - subnet mask /0 - Destination router: 192.168.20.254
When we attempt to match "172.16.31.92" to each entry in the routing table we end up hitting this new entry. It's a bit perplexing, at first. We're looking to match zero bits of the destination address with... wait... what? Matching zero bits? So, we're not looking for a match at all. This routing table entry is saying, basically, "If you get here, rather than giving up on delivery, send the packet on to the router at 192.168.20.254 and let him handle it".
192.168.20.254 is a destination we DO know how to deliver a packet to. When confronted with a packet bound for a destination for which we have no specific routing table entry this "default gateway" entry will always match (since it matches zero bits of the destination address) and gives us a "last resort" place that we can send packets for delivery. You'll sometimes hear the default gateway called the "gateway of last resort."
In order for a default gateway route to be effective it must refer to a router that is reachable using the other entries in the routing table. If you tried to specify a default gateway of 192.168.50.254 in Router A, for example, delivery to such a default gateway would fail. 192.168.50.254 isn't an address that Router A knows how to deliver packets to using any of the other routes in its routing table, so such an address would be ineffective as a default gateway. This can be stated concisely: The default gateway must be set to an address already reachable by using another route in the routing table.
Real routers typically store the default gateway as the last route in their routing table such that it matches packets after they've failed to match all other entries in the table.
Urban Planning and IP Routing
Breaking up a IP subnet into smaller IP subnets is like urban planning. In urban planning, zoning is used to adapt to natural features of the landscape (rivers, lakes, etc), to influence traffic flows between different parts of the city, and to segregate different types of land-use (industrial, residential, etc). IP subnetting is really much the same.
There are three main reasons why you would subnet a network:
You may want to communicate across different unlike communication media. If you have a T1 WAN connection between two buildings IP routers could be placed on the ends of these connections to facilitate communication across the T1. The networks on each end (and possibly the "interstitial" network on the T1 itself) would be assigned to unique IP subnets so that the routers can make decisions about which traffic should be sent across the T1 line.
In an Ethernet network, you might use subnetting to limit the amount of broadcast traffic in a given portion of the network. Application-layer protocols use the broadcast capability of Ethernet for very useful purposes. As you get more and more hosts packed into the same Ethernet network, though, the percentage of broadcast traffic on the wire (or air, in wireless Ethernet) can increase to such a point as to create problems for delivery of non-broadcast traffic. (In the olden days, broadcast traffic could overwhelm the CPU of hosts by forcing them to examine each broadcast packet. That's less likely today.) Excessive traffic on switched Ethernet can also come in form of "flooding of frames to unknown destinations". This condition is caused by an Ethernet switch being unable to keep track of every destination on the network and is the reason why switched Ethernet networks can't scale to an infinite number of hosts. The effect of flooding of frames to unknown destinations is similar to the the effect of excess broadcast traffic, for the purposes of subnetting.
You may want to "police" the types of traffic flowing between different groups of hosts. Perhaps you have print server devices and you only want authorized print queuing server computers to send jobs to them. By limiting the traffic allowed to flow to the print server device subnet users can't configure their PCs to talk directly to the print server devices to bypass print accounting. You might put the print server devices into a subnet all to themselves and create a rule in the router or firewall attached to that subnet to control the list of hosts permitted to send traffic to the print server devices. (Both routers and firewalls can typically make decisions about how or whether to deliver a packet based on the source and destination addresses of the packet. Firewalls are typically a sub-species of router with an obsessive personality. They can be very, very concerned about the payload of packets, whereas routers typically disregard payloads and just deliver the packets.)
In planning a city, you can plan how streets intersect with each other, and can use turn-only, one-way, and dead-end streets to influence traffic flows. You might want Main Street to be 30 blocks long, with each block having up to 99 buildings each. It's pretty easy to plan your street numbering such that each block in Main Street has a range of street numbers increasing by 100 for each block. It's very easy to know what the "starting number" in each subsequent block should be.
In planning IP subnets, you're concerned with building the right number of subnets (streets) with the right number of available host ID's (building numbers), and using routers to connect the subnets to each other (intersections). Rules about allowed source and destination addresses specified in the routers can further control the flow of traffic. Firewalls can act like obsessive traffic cops.
For the purposes of this answer, building our subnets is our only major concern. Instead of working in decimal, as you would with urban planning, you work in binary to describe the bounds of each subnet.
Continued on: How does IPv4 Subnetting Work?
(Yes ... we reached the maximum size of an answer (30000 characters).)
That depend on your MPLS provider. so there are a patch for the traceroute program, that print the MPLS label using ICMP, if the LSR is configured to affich these labels ( but most of providers don't do). You may test the NANOG traceroute normalized in RFC 4950
Related Topic
- CPU Utilization – What Does ‘Nice’ Mean on CPU Utilization Graphs?
- Linux – What does Virtual memory size in top mean
- Linux – What Do the Colors in htop Status Bars Mean
- Linux Security – What Does ‘POSSIBLE BREAK-IN ATTEMPT!’ Mean in /var/log/secure?
- Linux – What Does ‘***’ Mean in Traceroute?
- Debconf: Delaying Package Configuration – Meaning and Solutions
- Traceroute continuing after destination
- Linux – What does having * * * mean in the command traceroute ? and How to you cope with that situation
Best Answer
!X means "communication administratively prohibited" and !Z "communication with destination host administratively prohibited" As far as I remember, you get !X on ipv4 and !Z on ipv6 and it should be documented in the man (8) pages.
Since Linux uses UDP for trace-routes, this can originate from a
--reject-with icmp-host-prohibitedrule at the destination. Some Linux distros have this as a default configuration. To fix this you need to reply with--reject-with icmp-port-unreachableon UDP ports 33434 through 33534.