In the early stages of learning about BGP, we learn to set Local Preference on routes we learn from other networks to control how outbound traffic leaves our network.
Then we learn to set AS Path prepending on routes we advertise to external networks to control how inbound traffic enters our network.
We manipulate the AS path by pre-pending additional instances of our own AS number to make the AS path for the route longer and therefore less attractive compared to the same route with a shorter AS path length.
AS Prepending Expectations
Let’s look at a fairly common scenario. An organization wants to multi-home to two different ISPs for redundancy. Traffic should come in primarily via one ISP. The second ISP should only be used if the link to the primary ISP fails.
We end up with a network that looks similar to this.
- Corp 1 advertises a network, 192.168.100.0/24 as follows:
- To ISP X with no prepends, a total AS Path length of 1.
- To ISP Y with 3 prepends, a total AS Path length of 4.
- ISP X advertises this network to its customers and other ISPs – ISP Z in this case.
- ISP Z advertises this network to its customers and on to ISP Y
- ISP Y receives two copies of a route for 192.168.100.0/24
- From ISP Z with AS Path length of 3
- From Corp 1 with AS Path length of 4
- ISP Y should prefer the path via ISP Z as it has a shorter AS Path length
Based on these advertisements, we expect the traffic to flow as per the below diagram.
AS Prepending Reality
In theory, traffic into Corp 1 should all come via ISP X. This, however, is not what happens.
In reality, the traffic flow is more likely to look like this.
Corp 1 is likely to receive traffic from both the connection to ISP X and the connection to ISP Y.
The route advertisements might look more like this.
Why Is AS Prepending Not Working?
The reason AS Path prepending doesn’t always have the desired outcome is that the networks we advertise the route to can choose to increase the Local Preference of the route when they receive it.
Local Preference takes precedence over AS path length in the BGP decision process.
In our example, both ISP X and ISP Y will likely increase the local preference of the 192.168.100.0/24 route they learn from Corp 1. Both are going to prefer to send traffic direct to Corp 1, rather than sending it via another network.
In this instance, the behavior of ISP X matches what Corp 1 wants to achieve. So this doesn’t cause any issues.
When ISP Y also increases the local preference for the Corp 1 route to prefer the direct path, this will mean any traffic destined for 192.168.100.0/24 that reaches ISP Y will go over this direct path.
I’ve also depicted ISP Z as preferring to send traffic via ISP X. It could instead choose to send the traffic via ISP Y.
The key takeaway is that AS Path prepending can influence how traffic enters our network, but it can’t control it.
Other organizations can make routing decisions based on business and technical reasons that suit their needs. These decisions can be at odds with our goals for how traffic should reach our networks.
Why Do ISPs Want To Send Traffic Direct To Customers?
Let’s loosely define a few terms to help with the next part of the explanation
- Customer – Someone who pays you to deliver traffic to them.
- Peering – A connection where two or more entities agree to share traffic without charging each other for it, each paying for their part of the connection.
- Transit – Someone you pay to deliver traffic for you.
From an ISP perspective, they want to increase their revenue and decrease costs.
They will prefer to deliver traffic over a customer connection first, because they get paid for it, increasing their revenue. It also keeps traffic away from peering or transit links, meaning these can be a smaller size, reducing costs.
Next, they will prefer peering, where the cost of traffic is cheaper than transit.
Transit is the last choice because it costs them money to pay someone else to deliver the traffic.
When we apply this to our example, we can see that ISP Y wants to deliver traffic directly to Corp 1 which pays them for that traffic. They will not want to send the traffic via ISP Z, where they pay for the traffic to be delivered.
What’s The Solution To Stop Traffic Coming In Via A Secondary Link?
So what is the solution to this problem? How does Corp 1 only have traffic arrive via ISP Y when the link to ISP X is down?
The best answer is to talk to ISP Y and find a workable solution. Corp 1 should contact their account manager at ISP Y or raise a support ticket. Advise ISP Y that the link is only intended to be a backup.
ISP Y may have a community string that Corp 1 can attach to the route which causes ISP Y’s routers not to change the local preference, or even to reduce it. This removes the need to use prepending.
It appears using community strings instead of prepending is becoming the preferred option in many cases.
What About Using More Specific Routes?
Another approach could be to advertise more specific routes to ISP X and less specific routes to ISP Y.
Let’s say Corp 1 has the following address space – 192.168.100.0/22.
It could advertise the four /24’s to ISP X, or two /23’s, and then the covering /22 to ISP Y.
Due to routing using the longest prefix match, the more specific routes should be preferred over the less specific route.
There are a couple of things to keep in mind with this approach.
The first is that advertising four networks instead of one increases someone’s costs. Those routes take up space in routing and forwarding tables. If everyone does this, people will need bigger routers to handle those additional routes.
The second is that other people can still manage their networks however they want. If an ISP expects to receive a customer route directly from the customer, it may choose to block learning those routes from a transit path. It could also reject more specific routes when it knows it has a less specific covering route available.
Summary
Other organizations can, and do, route traffic to achieve their specific goals.
The use of AS path prepending can be overridden by a provider using local preference.
The best solution is to talk to your provider(s) about their options to achieve your desired outcome.
Judging by this blog post at Cloudflare, the use of community strings to influence how a provider sets Local Preference is becoming the recommended option.