It can be difficult to troubleshoot the establishment of BGP sessions. Sometimes the issue is obvious, but other times it can be a vague and frustrating process.
In this post, we will run through some of the tools available, the BGP Finite State Machine (FSM), and a general approach for troubleshooting BGP establishment.
BGP Troubleshooting Tools
The specific tools available depend on which vendor you are using.
Generally, you should be able to view:
- A summary of BGP connections showing their current state.
- Information on a specific BGP peer.
- Log files or debugging output to help identify issues when the BGP sessions won’t establish.
- If BGP is established, what routes are being sent and received.
BGP Establishment Troubleshooting
Issues with BGP establishment tend to fall into one of two categories
- Basic connectivity issues that prevent the TCP connection from completing.
- Routing issues.
- Traffic filtering.
- Interface down.
- Authentication mismatch.
- BGP protocol-level issues, usually an issue in the Open message.
- A mismatch in AS numbers.
- Issues with hold time or optional parameters.
Once we know which category of issue we are dealing with, we can then focus on which things we need to look at to solve our problem.
Most issues with BGP come down to a misconfiguration of some kind. This is especially true when trying to solve BGP establishment issues.
To help understand how these issues might present themselves, we need to review the BGP states.
BGP States
BGP operates a Finite State Machine (FSM) on a per-peer basis. This means that the BGP peer can be in one of a limited number of states. The FSM will progress through the FSM states based on events that occur.
If you want to dig into the details of the BGP FSM, I recommend reading section 8 of RFC 4271 – BGP-4.
| State | Meaning |
| Idle | Initially BGP refuses all connections for this peer. After a start event, BGP will – initialize resources for the peer – Listen for a connection – Initiate a connection to the peer and move into the Connect state. |
| Connect | BGP is waiting for the TCP connection to be made. – If TCP connects send an Open message and move to the OpenSent state – If ConnectRetryTimer expires, stay in the Connect state and retry the TCP connection. – If the TCP connection fails, BGP could move to either Idle or Active states. |
| Active | BGP is listening for and accepting TCP connections. – When ConnectRetryTimer expires initiate TCP connection and move to Connect state – If there is a successful TCP connection, send an Open message and move to the OpenSent state |
| OpenSent | BGP is waiting to receive an Open message. – If HoldTimer expires, send a Notification and move to the Idle state. – If the TCP connection fails, close the connection, listen for a connection from the peer, and move to the Active state. – If an Open message is received, check for errors. If there are no errors, send a KeepAlive and move to the OpenConfirm state. – If the Open message has errors, send a Notification, drop the TCP connection, and move to the Idle state. – If a Notification is received, drop the TCP connection and move to the Idle state. |
| OpenConfirm | BGP is waiting for a KeepAlive or Notification message. – If KeepAlive is received, restart HoldTimer and move to the Established State. – If the Holdtimer expires, send a Notification and move to the Idle state. – If a Notification is received, tear the session down and move to the Idle state. |
| Established | BGP is established, it can send and receive Update, Notification, and Keepalive messages. – Receving an Update or Keepalive will restart the holdtimer – Sending or receiving a Notification message will tear the session down and BGP will go back to the Idle state. |
The BGP process, per peer, will work its way through these states.
The most common states a BGP session will be in are Idle, Active, or Established.
BGP Establishment Troubleshooting – A General Approach
The following approach is a good start to identifying the issue.
- Confirm BGP being down is actually an issue.
- Confirm what state BGP is in.
- Check the specific BGP peer
- Check the configuration is correct.
- Check logs for relevant information.
- If necessary, start looking at debug or traceoptions output.
Let’s look at each of these in more detail.
Confirm BGP Being Down Is Actually An Issue
The first thing to do is confirm that BGP being down is an issue.
This is especially important when we are troubleshooting another issue and have found a BGP peer that is down. We need to determine if this is related to the initial problem, or if this session was already down anyway.
There might be any number of reasons why a BGP session is down that don’t need any troubleshooting on our part.
- The session might be shut down on purpose.
- The session might be set to passive, and be waiting for the peer router to connect.
- There could be a planned change underway, and this is expected to occur.
- The session could be no longer required and the config hasn’t been removed yet.
On the other hand, indicators that it might be an issue are:
- Receiving an alert from a monitoring system
- A support ticket or event being raised.
- This is a new peer that we are trying to establish and it is not coming up.
The important thing with this step is to make sure we’re not wasting time trying to troubleshoot a BGP session that shouldn’t be established anyway.
Confirm What State BGP Is In
Once we have confirmed this is indeed an issue that needs to be solved, you want to look at what state the BGP session is in.
Based on what the state is, we can narrow down what we need to look at.
This is best achieved by looking at the BGP summary information. On Junos, this would be the show bgp summary command. Cisco has variations of show ip bgp summary depending on how the device is configured.
If the device has VRFs configured, you may need to specify additional parameters to see the BGP information you want.
It is best to run these commands several times in a row, spaced roughly 10 seconds apart. This is to determine whether the state stays the same, or if it changes over time. For example, does it change between Active, sit at Connect for a while, and then move back to Active?
Check The Specific BGP Peer
After checking the BGP summary information we can focus on the individual peer that is being problematic.
On Junos this is show bgp neighbor x.x.x.x and on Cisco it will be a variation of show ip bgp neighbor x.x.x.x, where you may need to specify the VRF if you’re using them.
From this detailed neighbor output, we’re trying to determine the state of the peer and any relevant information. What was the last error, what flags are set, and so forth? Specifically for eBGP are there any issues with finding an outgoing interface?
Check The Configuration Is Correct
Although it sounds basic, it is always a good idea to check the configuration is correct. Are the peer IP Addresses and AS numbers correct? If needed, has multihop been configured correctly?
Has the local address been set for iBGP sessions?
Check Logs For Relevant Information
Logs should be checked for any relevant information. Depending on the issue, there isn’t always a lot of information here.
Based on what you find in the log file, you may need to go back to a previous step such as checking the configuration is correct and the peer details are right.
Start Looking At Debug Or Traceoptions
Most vendors will have some form of debug or traceoptions output that can provide more detailed information about the BGP process or peer than the system logs do.
There is nothing wrong with using these tools, but they’re not the first thing I would check. The information isn’t necessarily hard to read, but there can be a lot of information to look through. It can sometimes be overwhelming and easy to miss a key detail.
Also, if you are not used to them, it can be hard to tell what is normal and what is not. This can lead to wasted time as you try to troubleshoot something that isn’t an issue.
If they are not already set up, they may require additional configuration to be set, which could require change control processes to be followed.
In some cases, they may not provide any additional useful information either.
Troubleshooting Specific BGP States
In the next few sections, we will look at troubleshooting specific BGP states.
BGP Stuck In Idle State
BGP being stuck in the Idle state can be hard to troubleshoot. In this state, BGP is waiting for a manual or automatic start event to trigger it into allocating resources for the peer and trying to connect to the peer.
When in the Idle state the BGP process isn’t necessarily doing a lot, so there may not be much in the way of logs or debug events to see.
We need to look at issues that could cause BGP to not even try to start the BGP process.
Some examples include
- When eBGP can’t determine the outgoing interface. EBGP by default uses a TTL of one, so if it can’t see the BGP peer on a directly connected interface it won’t even try to connect. This can be caused by an interface being in the down state.
- When using eBGP peering to loopback addresses without specifying multi-hop. EBGP will determine that it cannot connect due to TTL and won’t attempt a connection.
- The peer is shut down on the local router.
A quick check here is to look for a route in the routing table for the BGP peer.
BGP Stuck In Connect State
In this state, BGP is waiting for the TCP connection to complete. Troubleshooting should focus on what could prevent the TCP session from establishing.
- Could the traffic be filtered? On the local device, the remote device, or somewhere in the middle?
- Is there an authentication mismatch?
It is worth checking the logs to see if there is any further information available.
You can try to ping the remote peer using the relevant source address on your router. Be aware that the other side might be filtering ICMP traffic, or may allow ICMP and be blocking BGP traffic.
You might also try telnetting from the router to the remote peer on port 179 and seeing whether you get a connection or if it just times out. Note that you won’t likely get anything useful via the telnet session, it is only useful to get an idea of being able to connect.
BGP Stuck In Active State
Although it may appear that BGP is stuck in the Active state, it is more likely that it is cycling through several states and that our show commands are just catching it in the Active state.
The best option is to look through logs or debug information, specifically looking to see if we are sending or receiving Notification messages indicating that the BGP session is being torn down. This could be due to a misconfiguration.
For iBGP it can be useful to confirm you have a route to the BGP peer and the correct local address set.
Beyond that, you should be guided by what you see in the log or debug files.
What To Do If Nothing Worked?
Sometimes, it feels like we can’t solve the issue. BGP is not establishing and we don’t know why.
If you get stuck in the troubleshooting process, go back to the basics.
- Do I have a route to the BGP peer?
- What state is the BGP session in and what does that mean?
- Can I connect to TCP port 179 on the remote device?
- Do I have the right peer details and are they configured correctly?
- What error messages, if any, do I see?
Something else to keep in mind is that it is OK to ask for help. Instead of sitting stuck with an issue and looking for increasingly complex explanations, ask someone else to look at it with you. They might spot something you haven’t.
Sometimes just talking the issue through with someone else makes the answer obvious.
Troubleshooting Practice
The best way to get better at something is to practice it. To get better at BGP troubleshooting you need to practice it in a lab environment.
Practice doesn’t mean simply configuring BGP and fixing the mistakes that you happen to make as you build your lab.
You need to break things on purpose. When you break BGP in a certain way what do you see in the various show commands? What errors do you see in the log files? Which debug or traceoption output gives you the best information?
Once you’ve done that, set the lab up so it is all working, and then have someone else break the lab for you. Can you figure out what they broke? How efficiently did you find it? Was it something you had done previously or did they break it in a way you hadn’t seen before?
Putting in the time and effort to do this type of lab work will pay off when you have to troubleshoot BGP in a live environment where people make all sorts of weird and wonderful mistakes.