Same ip, different route?

Question

There's a set of ip addresses that are hosted from the same ISP servers, but when i pick a random ip address from that set, it picks the right route or maybe the wrong one. I'll give you an example: ISP ip addresses: 94.0/38.xx.xx

If i do a Tracert to a random number like (94.2.55.112) it goes like this:

1- 84.111.xx.xx (ISP 1)
2- 84.111.xx.xx
3- 84.110.xx.xx 
4- 195.27.xx.xx (ISP 2)
5- 195.28.xx.xx
6- 99.34.xx.xx (ISP 3)
7- 94.2.55.112 (ISP 3)

However, if i do a tracert to that six route (99.34.xx.xx) it changes completely, and goes between so many ISPs before getting there.

Q: Why did the routing table change? Why wouldn't it follows the same route as mentioned above? That's so frustrating, this can't be true. I never knew that networking route would be so complicated like that.

Solution

It needs a couple of explanations:

First of all, how traceroute works:

1. Traceroute sends a package to the destination IP (94.2.55.112) with TLL=1
2. The receving router decrements TTL and because TTL=0 discards the package and returns a TTL exceeded
3. Traceroute sends a package to the destination IP (94.2.55.112) with TLL=2
4. The receving router decrements TTL and pass it to the next hop.
5. The next hop decrements TTL and because TTL=0 discards the package and returns a TTL exceeded
6. Traceroute sends a package to the destination IP (94.2.55.112) with TLL=3

. . .

And so on, until the package reach its destination (94.2.55.112).

---

Now, let's check how routers work:

When a router receives a packet, its only mission is to send it to the next hop as soon as possible.

To achieve this mission, the router makes a decision based on a routing table; the routing table for an internet table could be huge. Checking the table and making a decision is called process switching and is the slowest way to do it. So there are some tricks to speed it up.

• There is fast switching. With it, after a packet has been sent to the next hop, the routing information about how to get to the destination is stored in a fast cache. When the router receives another packet that is directed to the same destination it uses the cache. It's faster than the traditional way.

• Cisco has developed a technique called Cisco Express Forwarding or CEF, that is even faster. CEF is a propietary system that stores information in a way that can be queried very quickly.

---

Now let's get all together:

When the destination of a traceroute is a host (94.2.55.112, for example), on the first packet the routers will use the traditional way (process switching) to know where is the destination, but after that in the next packets the routers will use fast switching or CEF. From origin to destination there will be a fixed path.

When the destination of a traceroute is a router (99.34.xx.xx), the intermediate routers will do the same as before (first packet with process switching and from then cache), but the destination router won't do it because there are no next hop. So, the destination router will do process switching to respond to each received packet.

The big problem arises when the destination router has two or more connections to different ISPs and the connections are doing load balancing.

Load balancing means that to avoid that a single connection gets overloaded until the router uses the next one, it uses a mechanism as round-robin to evenly distribute the traffic among all its connections.

Now you have the explosive mix for your situation: A traceroute to a router, it will use process switching for every package, that is influenced by load balancing and you have every time a different path.

Sorry for the long explanation, but as the result is a sum of multiple mechanisms, I wanted to make them as clear as possible.

Comments

[ganesh]

Who says the routing table changed? ISP 1 may know a good route to IPS2 and dump packets for unknown destinations to ISP3. Maybe for no good reason. Maybe to balance a load. More importantly, why do you care? As long as it is outside your network and you get a good connection this should not influence you at all.

[CharlieRB]

It seems you have only given us what you think the problem might be, rather than what you are trying to resolve overall. Is this actually causing a problem? Or are you just frustrated because it is not taking the route you would like?

[Semphie94]

@CharlieRB Yes. And as a result, the packets goes back and forth between overseas which increase the ping/latency from 30-50ms(good route) to 240-250ms(bad route) and neither you, me or anyone would agree to this confusing behavior right? That's why it DOES NOT MAKE A SENSE.

[Daniel B]

Except a router is a host too and doesn’t have to do decide anything when it’s the destination. ;)

[Semphie94]

Well the destination didn't change. But the routing to the (Same) destination changes Only on some random numbers. It's like saying: instead of going from south to north to get to the station, we can take another subway road from south, east, middle, west to north. :/

[Daniel B]

Of course it did. First, you ran tracert 94.2.55.112, next you ran tracert 99.34.xx.xx. It really can’t get any more obvious.