In this post, we will look at how to troubleshoot BGP establishment on Junos. First, we will look at what tools are available. Then, we will look at some common examples of why BGP might not be established
This follows on from the posts on general BGP establishment troubleshooting and the basics of how to configure BGP on Junos.
Junos BGP Troubleshooting Tools
The main troubleshooting tools for BGP establishment on Junos are:
- Show commands – show bgp summary and show bgp neighbor.
- Log files – such as the messages log file.
- Traceoptions – specific debug configuration to look at BGP events.
- Monitor traffic interface command – This is similar to a packet capture, it shows traffic destined for the router on a given interface.
Show Commands
The show bgp summary command gives an overview of the configured BGP sessions.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 107 108 0 0 46:48 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 78 78 0 0 34:33 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 77 76 0 0 33:44 Establ
inet.0: 0/3/3/0
lab@vMX-2>In the above output, you can see that all three BGP sessions are established.
The inet.0: 0/3/3/0 line is tells us the address family negotiated (inet.0 is the route table for IPv4 unicast routes) and how many routes are active, received, accepted, and damped. This specific entry tells us that the peer sent us three routes, we accepted all three, but none are active. In this case, the routes are not active because we prefer the same routes learned from one of the other peering sessions.
To look at one particular BGP peer in more detail, we can use the show bgp neigbor command and specify which neighbor we want to look at.
lab@vMX-2> show bgp neighbor 172.16.2.3
Peer: 172.16.2.3+179 AS 65001 Local: 172.16.1.2+53530 AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Options: <Multihop Preference LocalAddress AuthKey Ttl PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Local Address: 172.16.1.2 Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Peer ID: 172.16.2.3 Local ID: 172.16.1.2 Active Holdtime: 90
Keepalive Interval: 30 Group index: 2 Peer index: 0 SNMP index: 0
I/O Session Thread: bgpio-0 State: Enabled
BFD: disabled, down
NLRI for restart configured on peer: inet-unicast
NLRI advertised by peer: inet-unicast
NLRI for this session: inet-unicast
Peer supports Refresh capability (2)
Stale routes from peer are kept for: 300
Peer does not support Restarter functionality
Restart flag received from the peer: Notification
NLRI that restart is negotiated for: inet-unicast
NLRI of received end-of-rib markers: inet-unicast
NLRI of all end-of-rib markers sent: inet-unicast
Peer does not support LLGR Restarter functionality
Peer supports 4 byte AS extension (peer-as 65001)
Peer does not support Addpath
NLRI(s) enabled for color nexthop resolution: inet-unicast
Table inet.0 Bit: 20001
RIB State: BGP restart is complete
Send state: in sync
Active prefixes: 3
Received prefixes: 3
Accepted prefixes: 3
Suppressed due to damping: 0
Advertised prefixes: 1
Last traffic (seconds): Received 14 Sent 27 Checked 2330
Input messages: Total 88 Updates 2 Refreshes 0 Octets 1712
Output messages: Total 88 Updates 1 Refreshes 0 Octets 1704
Output Queue[1]: 0 (inet.0, inet-unicast)
lab@vMX-2> This output gives us a lot more detail about the peer. We can see information about the TCP connection, the state of the peering, and what options and address families have been negotiated.
We don’t always want to see all of this output, so we can use the CLI match option to show just the lines with state or error in them.
lab@vMX-2> show bgp neighbor 172.16.2.3 | match "State|error"
Type: External State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
I/O Session Thread: bgpio-0 State: Enabled
RIB State: BGP restart is complete
Send state: in sync
lab@vMX-2>This makes the output a little easier to read through.
Log Files
The main log file to look at is the messages log file. By default, BGP logs quite clear and useful information.
The command we want is show log messages. However, we don’t normally want to see the whole log file. We can limit it to either the last few lines, match specific entries, or combine these to show us the last few entries.
Showing just the last few lines.
lab@vMX-2> show log messages | last 10
Aug 18 03:00:37 vMX-2 rpd[8786]: L2CKT acquiring mastership for primary
Aug 18 03:00:37 vMX-2 rpd[8786]: RPD_COMMIT_SYNC_STATUS: RPD commit sync completed on MASTER
Aug 18 03:00:37 vMX-2 rpd[8786]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs started on MASTER
Aug 18 03:00:37 vMX-2 rpd[8786]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs completed on MASTER
Aug 18 03:00:37 vMX-2 mgd[9009]: UI_COMMIT_COMPLETED: : commit complete
Aug 18 03:00:52 vMX-2 rpd[8786]: RPD_RT_HWM_NOTICE: New RIB highwatermark for routes: 28 [2024-08-18 03:00:52]
Aug 18 03:01:22 vMX-2 rpd[8786]: RPD_RT_HWM_NOTICE: New RIB highwatermark for routes: 30 [2024-08-18 03:00:52]
Aug 18 03:18:49 vMX-2 mgd[9009]: UI_DBASE_LOGOUT_EVENT: User 'lab' exiting configuration mode
lab@vMX-2>We could instead match all entries that have the text ‘bgp’ and show the last 10 lines.
lab@vMX-2> show log messages | match bgp | last 10
Aug 18 02:56:50 vMX-2 rpd[8786]: bgp_pp_recv:4991: NOTIFICATION sent to 172.16.2.3+53299 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 172.16.2.3+53299 (proto) from AS 65001 found (eBGP connection attempt to an address that does not belong to the interface that peer connected to) in master(ge-0/0/2.0), dropping him
Aug 18 02:58:28 vMX-2 rpd[8786]: bgp_peer_delete:11369: NOTIFICATION sent to 192.168.2.1 (External AS 65001): code 6 (Cease) subcode 9 (Hard Reset) [code 6 (Cease) subcode 3 (Peer Unconfigured)], Reason: Peer Deletion
Aug 18 02:59:18 vMX-2 rpd[8786]: bgp_pp_recv:4991: NOTIFICATION sent to 172.16.2.3+63519 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 172.16.2.3+63519 (proto) from AS 65001 found (eBGP connection attempt to an address that does not belong to the interface that peer connected to) in master(ge-0/0/2.0), dropping him
lab@vMX-2> Sometimes we want to know how many entries are related to a certain message. We can match the text of ‘notification’ and count it.
lab@vMX-2> show log messages | match NOTIFICATION | count
Count: 80 lines
lab@vMX-2>Traceoptions
Traceoptions are the equivalent of debugging on Cisco IOS devices. A lot of the time it shouldn’t be necessary to configure traceoptions, as most of the information will be available via the messages log already.
However, traceoptions do provide more detail on what is happening at the BGP level.
When it comes to BGP we can configure traceoptions under the protocol, group, or neighbor level.
In this example, we will configure traceoptions under the my-internal-group. To save on typing, we will edit down the configuration hierarchy and then set the traceoptions.
[edit]
lab@vMX-2# edit protocols bgp group my-internal-group
[edit protocols bgp group my-internal-group]
lab@vMX-2# set traceoptions file bgp-my-int-group
[edit protocols bgp group my-internal-group]
lab@vMX-2# set traceoptions file size 1m
[edit protocols bgp group my-internal-group]
lab@vMX-2# set traceoptions file files 10
[edit protocols bgp group my-internal-group]
lab@vMX-2# set traceoptions flag all
[edit protocols bgp group my-internal-group]
lab@vMX-2# show
type internal;
traceoptions {
file bgp-my-int-group size 1m files 10;
flag all;
}
local-address 172.16.1.2;
authentication-key "$9$GSjkmz39O1hfT1hSr8LGDi"; ## SECRET-DATA
export [ fix-nhs exp-default ];
neighbor 172.16.1.1;
[edit protocols bgp group my-internal-group]
lab@vMX-2# top
[edit]
lab@vMX-2# show | compare
[edit protocols bgp group my-internal-group]
+ traceoptions {
+ file bgp-my-int-group size 1m files 10;
+ flag all;
+ }
[edit]
lab@vMX-2# commit
commit complete
[edit]
lab@vMX-2#This configuration creates a file called bgp-my-int-group in the /var/log directory. It will create up to 10 files of 1 megabyte each in size. Like other log files, it will append a .0, .1, .2 to the log file names each time it rolls the log files over due to size.
The flag directive determines what type of information is logged to the traceoptions file. In this case, we are logging everything.
We can view the file like we did the messages file with the show log command.
lab@vMX-2> show log bgp-my-int-group
Aug 18 03:59:40 trace_on: Tracing to "/var/log/bgp-my-int-group" started
Aug 18 03:59:40.552169 io-120-27-BGP_65002.172.16.1.1: Response processing done for queue 2, re-register handler for new msgs
Aug 18 04:00:05.119676 io-120-27-BGP_65002.172.16.1.1: Handling session msg for queue 2, call user to start job and deregister handler
Aug 18 04:00:05.119713 BGP peer 172.16.1.1 (Internal AS 65002): started read response process job 0xc042d70
Aug 18 04:00:05.119782 io-120-27-BGP_65002.172.16.1.1: Got inbound PDU, qidx=2, msgid=8, type=3, niov=1, nbytes=19, err=0
Aug 18 04:00:05.119788 bgp_read_resp_process_internal:1540: 172.16.1.1 (Internal AS 65002): npdus=1, pdubuf=0xda41000 iovbase=0xda41000, iovlen=19
Aug 18 04:00:05.119800
Aug 18 04:00:05.119800 BGP RECV 172.16.1.1+179 -> 172.16.1.2+62279
Aug 18 04:00:05.119807 BGP RECV message type 4 (KeepAlive) length 19
Aug 18 04:00:05.119886 io-120-27-BGP_65002.172.16.1.1: EOF read messages
Aug 18 04:00:05.119898 io-120-27-BGP_65002.172.16.1.1: Response processing done for queue 2, re-register handler for new msgs
lab@vMX-2>
As mentioned above, we can also apply traceoptions at the individual neighbor level. Here are the traceoptions applied to the two neighbors in the external peering group. I set these at the neighbor level so I know what output relates to which BGP peer. If I set the traceoptions at the group level it would have seen logging for both neighbors.
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
[edit]
lab@vMX-2#Monitor Traffic Interface Command
The monitor traffic interface command acts like a tcpdump on the given interface. Note that this command will only see traffic that is destined to the router, it won’t capture any transit traffic that is passing through the router.
This can be a quick and dirty way to see what is going on, or at least check that we’re seeing BGP traffic over the interface.
If you really need to see the details of what is happening, using traceoptions is a better idea.
In the following output, I deactivated the BGP peer on the remote end, vMX-1. After starting the monitor traffic command on vMX-2, I re-enabled the BGP peer on vMX-1. This meant I could capture the BGP session establishment.
lab@vMX-2> monitor traffic interface ge-0/0/0 size 1500 detail matching "tcp port 179"
Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ge-0/0/0, capture size 1500 bytes
Reverse lookup for 172.16.1.1 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.
04:07:45.507724 In IP (tos 0xc0, ttl 64, id 17617, offset 0, flags [none], proto: TCP (6), length: 80) 172.16.1.1.61244 > 172.16.1.2.bgp: S 1307293437:1307293437(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2153409145 0,nop,nop,md5 2fe3b11fae58e7cb407331d4db8a4559>
04:07:45.507776 Out IP (tos 0xc0, ttl 255, id 27850, offset 0, flags [DF], proto: TCP (6), length: 80) 172.16.1.2.bgp > 172.16.1.1.61244: S 2953203072:2953203072(0) ack 1307293438 win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2153405441 2153409145,md5 13e7f90b7aeb994a8d5112257bc4b514,eol>
04:07:45.513463 In IP (tos 0xc0, ttl 64, id 17620, offset 0, flags [none], proto: TCP (6), length: 72) 172.16.1.1.61244 > 172.16.1.2.bgp: . ack 1 win 16800 <nop,nop,timestamp 2153409152 2153405441,nop,nop,md5 7ac0a1663ba5e801e5f7cfb89e6851cf>
04:07:45.513467 In IP (tos 0xc0, ttl 64, id 17623, offset 0, flags [none], proto: TCP (6), length: 135) 172.16.1.1.61244 > 172.16.1.2.bgp: P 1:64(63) ack 1 win 16800 <nop,nop,timestamp 2153409152 2153405441,nop,nop,md5 821864d8ae875d0d8b945b0a1e6cabec>: BGP, length: 63
Open Message (1), length: 63
Version 4, my AS 65002, Holdtime 90s, ID 172.16.1.1
Optional parameters, length: 34
Option Capabilities Advertisement (2), length: 6
Multiprotocol Extensions (1), length: 4
AFI IPv4 (1), SAFI Unicast (1)
Option Capabilities Advertisement (2), length: 2
Route Refresh (Cisco) (128), length: 0
Option Capabilities Advertisement (2), length: 2
Route Refresh (2), length: 0
Option Capabilities Advertisement (2), length: 4
Graceful Restart (64), length: 2
Restart Flags: [none], Restart Time 120s
Option Capabilities Advertisement (2), length: 6
32-Bit AS Number (65), length: 4
4 Byte AS 65002
Option Capabilities Advertisement (2), length: 2
Unknown (71), length: 0
no decoder for Capability 71
04:07:45.515290 Out IP (tos 0xc0, ttl 255, id 27853, offset 0, flags [none], proto: TCP (6), length: 135) 172.16.1.2.bgp > 172.16.1.1.61244: P 1:64(63) ack 64 win 16384 <nop,nop,timestamp 2153405448 2153409152,nop,nop,md5 a915b8d3c4ba7658231888e82e143bd9>: BGP, length: 63
Open Message (1), length: 63
Version 4, my AS 65002, Holdtime 90s, ID 172.16.1.2
Optional parameters, length: 34
Option Capabilities Advertisement (2), length: 6
Multiprotocol Extensions (1), length: 4
AFI IPv4 (1), SAFI Unicast (1)
Option Capabilities Advertisement (2), length: 2
Route Refresh (Cisco) (128), length: 0
Option Capabilities Advertisement (2), length: 2
Route Refresh (2), length: 0
Option Capabilities Advertisement (2), length: 4
Graceful Restart (64), length: 2
Restart Flags: [none], Restart Time 120s
Option Capabilities Advertisement (2), length: 6
32-Bit AS Number (65), length: 4
4 Byte AS 65002
Option Capabilities Advertisement (2), length: 2
Unknown (71), length: 0
no decoder for Capability 71
04:07:45.621030 In IP (tos 0xc0, ttl 64, id 17630, offset 0, flags [none], proto: TCP (6), length: 72) 172.16.1.1.61244 > 172.16.1.2.bgp: . ack 64 win 16737 <nop,nop,timestamp 2153409259 2153405448,nop,nop,md5 4c129e49642d512f69152f33145ffc61>
04:07:45.718182 Out IP (tos 0xc0, ttl 255, id 27865, offset 0, flags [none], proto: TCP (6), length: 91) 172.16.1.2.bgp > 172.16.1.1.61244: P 64:83(19) ack 64 win 16384 <nop,nop,timestamp 2153405651 2153409259,nop,nop,md5 85d905066a5b1c4b12fba59e799450bc>: BGP, length: 19
Keepalive Message (4), length: 19
04:07:45.724784 In IP (tos 0xc0, ttl 64, id 17637, offset 0, flags [none], proto: TCP (6), length: 91) 172.16.1.1.61244 > 172.16.1.2.bgp: P 64:83(19) ack 64 win 16737 <nop,nop,timestamp 2153409360 2153405448,nop,nop,md5 0974c680bda32cf7a19035f277bbd1d9>: BGP, length: 19
Keepalive Message (4), length: 19
04:07:45.824655 Out IP (tos 0xc0, ttl 255, id 27874, offset 0, flags [none], proto: TCP (6), length: 72) 172.16.1.2.bgp > 172.16.1.1.61244: . ack 83 win 16384 <nop,nop,timestamp 2153405758 2153409360,nop,nop,md5 13c80241a1728b676a072746f208baf5>
04:07:45.826435 In IP (tos 0xc0, ttl 64, id 17645, offset 0, flags [none], proto: TCP (6), length: 72) 172.16.1.1.61244 > 172.16.1.2.bgp: . ack 83 win 16718 <nop,nop,timestamp 2153409464 2153405651,nop,nop,md5 e994a4d5270d06b71c71193c283571fe>
04:07:45.931124 In IP (tos 0xc0, ttl 64, id 17649, offset 0, flags [none], proto: TCP (6), length: 143) 172.16.1.1.61244 > 172.16.1.2.bgp: P 83:154(71) ack 83 win 16718 <nop,nop,timestamp 2153409569 2153405758,nop,nop,md5 f3bbfb9a76072b7a4d3d5bfa17da8fbe>: BGP, length: 71
Update Message (2), length: 48
Origin (1), length: 1, Flags [T]: IGP
AS Path (2), length: 0, Flags [T]: empty
Next Hop (3), length: 4, Flags [T]: 172.16.1.1
Local Preference (5), length: 4, Flags [T]: 100
Updated routes:
172.16.100.0/24
Update Message (2), length: 23
End-of-Rib Marker (empty NLRI)
04:07:45.932367 Out IP (tos 0xc0, ttl 255, id 27886, offset 0, flags [none], proto: TCP (6), length: 157) 172.16.1.2.bgp > 172.16.1.1.61244: P 83:168(85) ack 154 win 16384 <nop,nop,timestamp 2153405865 2153409569,nop,nop,md5 c4b5a480b6b817b2d249567601db36e4>: BGP, length: 85
Update Message (2), length: 62
Origin (1), length: 1, Flags [T]: IGP
AS Path (2), length: 6, Flags [T]: 65001
Next Hop (3), length: 4, Flags [T]: 172.16.1.2
Local Preference (5), length: 4, Flags [T]: 100
Updated routes:
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
Update Message (2), length: 23
End-of-Rib Marker (empty NLRI)
04:07:46.042676 In IP (tos 0xc0, ttl 64, id 17659, offset 0, flags [none], proto: TCP (6), length: 72) 172.16.1.1.61244 > 172.16.1.2.bgp: . ack 168 win 16633 <nop,nop,timestamp 2153409679 2153405865,nop,nop,md5 bac4ec5dcc6a292cf4ce10111e850180>
^C
19 packets received by filter
0 packets dropped by kernel
lab@vMX-2>In the above output, you can see that each BGP peer sends an open message to each other and after the session is established they send Update messages to share routing information.
Something interesting you might like to note is the format of the update messages. They list a set of path attributes – origin, AS path, next hop, and because this was an iBGP session, local preference. These path attributes are then followed by a list of routes, or prefixes, that share those path attributes.
Lab Environment
The lab is set up as follows.
- vMX-1 and vMX-2 belong to AS 65002.
- They peer using iBGP and peer between loopback interfaces.
- vMX-3 belongs to AS 65001.
- vMX-2 peers to vMX-3 in two ways.
- Direct eBGP peering over interface ge-0/0/1.
- eBGP peering to loopbacks over interface ge-0/0/2.
The use of both a direct and loopback-based eBGP peering between vMX-2 and vMX-3 is to show some specific issues that only occur over eBGP multi-hop sessions.
No Route To Neighbor (iBGP)
If we shut down the OSPF protocol between vMX-1 and vMX-2 they will no longer have routing information for each other’s loopback addresses.
On vMX-2 we can see the BGP session is in an active state.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
6 3 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 0 0 0 0 18 Active
172.16.2.3 65001 4 2 0 0 6 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 3 2 0 0 4 Establ
inet.0: 0/3/3/0
lab@vMX-2> If we look at the messages log, we see the following entries:
lab@vMX-2> show log messages | last 10
Aug 18 04:36:54 vMX-2 rpd[10510]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.1.1: Connecting to 172.16.1.1+179 failed: No route to host
Aug 18 04:36:54 vMX-2 rpd[10510]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.1.1 (Internal AS 65002) (instance master): No route to host
Aug 18 04:37:02 vMX-2 rpd[10510]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.1.1: Connecting to 172.16.1.1+179 failed: No route to host
Aug 18 04:37:02 vMX-2 rpd[10510]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.1.1 (Internal AS 65002) (instance master): No route to host
Aug 18 04:37:34 vMX-2 rpd[10510]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.1.1: Connecting to 172.16.1.1+179 failed: No route to host
Aug 18 04:37:34 vMX-2 rpd[10510]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.1.1 (Internal AS 65002) (instance master): No route to host
lab@vMX-2>
lab@vMX-2> show route 172.16.1.1
lab@vMX-2>The logs clearly tell us we don’t have a route to reach that BGP peer. This is backed up with a show route command showing no routes.
Looking at our traceoptions file, we see the following output repeating periodically.
Aug 18 04:39:42.609183 bgp_connect_timeout: BGP_65002.172.16.1.1_Connect
Aug 18 04:39:42.609211 bgp_connect_start: peer 172.16.1.1 (Internal AS 65002)
Aug 18 04:39:42.609219 bgp_event: peer 172.16.1.1 (Internal AS 65002) old state Active event ConnectRetry new state Connect
Aug 18 04:39:42.609521 task_get_socket: domain AF_INET type SOCK_STREAM protocol 0 socket 123
Aug 18 04:39:42.609537 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option RecvBuffer(0) value 16384
Aug 18 04:39:42.609542 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option SendBuffer(1) value 16384
Aug 18 04:39:42.609547 task_set_socket: task BGP_65002.172.16.1.1 socket 123
Aug 18 04:39:42.609567 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option NonBlocking(8) value 1
Aug 18 04:39:42.609573 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option ReUsePort(38) value 1
Aug 18 04:39:42.609578 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option ReUseAddress(3) value 1
Aug 18 04:39:42.609582 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option TcpMd5Authentication(72)
Aug 18 04:39:42.609589 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option PathMTUDiscovery(26) value 0
Aug 18 04:39:42.609595 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option TOS(16) value 192
Aug 18 04:39:42.609601 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option TcpAggressiveTransmission(59) value 0
Aug 18 04:39:42.609607 task_set_option_internal: task BGP_65002.172.16.1.1 socket 123 option EnableTcpNoDelay(63) value 0
Aug 18 04:39:42.609658 JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.1.1: Connecting to 172.16.1.1+179 failed: No route to host
Aug 18 04:39:42.609881 BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.1.1 (Internal AS 65002) (instance master): No route to host
Aug 18 04:39:42.609912 bgp_peer_close_and_restart: peer 172.16.1.1 (Internal AS 65002), state is 2 (Connect) event OpenFail, flags=0x0
Aug 18 04:39:42.609917 bgp_peer_close_and_restart: closing peer 172.16.1.1 (Internal AS 65002), state is 2 (Connect) event OpenFail
Aug 18 04:39:42.609922 bgp_send_deactivate:3449: 172.16.1.1 (Internal AS 65002) ,flags=0x0: removed from active list
Aug 18 04:39:42.609930 BGP peer 172.16.1.1 (Internal AS 65002) CLOSE: close socket 123
Aug 18 04:39:42.609935 task_close: close socket 123 task BGP_65002.172.16.1.1
Aug 18 04:39:42.609938 task_reset_socket: task BGP_65002.172.16.1.1 socket 123
Aug 18 04:39:42.610684 task_timer_delete: BGP_65002.172.16.1.1_Connect <Processing>
Aug 18 04:39:42.610698 bgp_peer_close_and_restart: 9278: BGP peer 172.16.1.1 (Internal AS 65002): state Connect task close fd -1
Aug 18 04:39:42.610703 bgp_event: peer 172.16.1.1 (Internal AS 65002) old state Connect event OpenFail new state Idle
Aug 18 04:39:42.610994 bgp_peer_post_close: Cleanup complete for peer 172.16.1.1 (last_flap NoEvent, state Idle, flags )
Aug 18 04:39:42.611004 bgp_delete_pending_set_impatient: peer 172.16.1.1 (Internal AS 65002) job-exist:no, flags:, need-job:no
Aug 18 04:39:42.611011 bgp_peer_post_close:8940 BGP Peer 172.16.1.1 (Internal AS 65002) CLOSE: post close cleanup - NEED restart, (last_flap NoEvent, state Idle, flags )
Aug 18 04:39:42.611025 task_timer_uset: timer BGP_65002.172.16.1.1_Down Peer Timer <Touched> set to interval 0 offset 2:00 with jitter 0 at 4:41:42.611011
Aug 18 04:39:42.611030 bgp_event: peer 172.16.1.1 (Internal AS 65002) old state Idle event Start new state Active
Aug 18 04:39:42.611048 task_timer_ucreate: created timer BGP_65002.172.16.1.1_Connect flags <>
Aug 18 04:39:42.611056 task_timer_uset: timer BGP_65002.172.16.1.1_Connect <Touched> set to interval 0 offset 32 with jitter 0 at 4:40:14.611048You can see that the peer moves from Active to Connect. The router tries to connect with TCP but gets a no route to host error. The peer then moves from Connect to Idle briefly, before moving to Active again.
If we re-enable OSPF we can see we now have a route to vMX-1’s loopback, and BGP establishes again.
Note: it did take a minute or two for BGP to establish after the router was available. Don’t expect BGP to immediately establish once you correct an issue. You need to give it a bit of time.
lab@vMX-2> show route 172.16.1.1
inet.0: 16 destinations, 19 routes (16 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
172.16.1.1/32 *[OSPF/10] 00:01:47, metric 1
> to 192.168.1.2 via ge-0/0/0.0
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 4 3 0 0 11 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 34 32 0 0 13:36 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 33 32 0 0 13:34 Establ
inet.0: 0/3/3/0
lab@vMX-2>Interface Shutdown (eBGP)
What happens if we shut down vMX-2’s interface ge-0/0/1? This interface is the one with the direct eBGP peering to vMX-3.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 15 13 0 0 4:54 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 14 14 0 0 5:12 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 0 0 0 0 5:22 Idle
lab@vMX-2>The state of the peer is Idle.
If we look at the messages log, there isn’t much useful information. The only BGP entries refer to one of the other peers.
lab@vMX-2> show log messages | last 10
Aug 18 04:59:11 vMX-2 rpd[10847]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.1.1: Connecting to 172.16.1.1+179 failed: No route to host
Aug 18 04:59:11 vMX-2 rpd[10847]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.1.1 (Internal AS 65002) (instance master): No route to host
Aug 18 04:59:13 vMX-2 rpd[10847]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.1.1: Connecting to 172.16.1.1+179 failed: No route to host
Aug 18 04:59:13 vMX-2 rpd[10847]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.1.1 (Internal AS 65002) (instance master): No route to host
Aug 18 04:59:23 vMX-2 rpd[10847]: RPD_RT_HWM_NOTICE: New RIB highwatermark for unique destinations: 25 [2024-08-18 04:59:21]
Aug 18 04:59:23 vMX-2 rpd[10847]: RPD_RT_HWM_NOTICE: New RIB highwatermark for routes: 26 [2024-08-18 04:59:21]
Aug 18 04:59:23 vMX-2 rpd[10847]: RPD_RT_HWM_NOTICE: New FIB highwatermark for routes: 9 [2024-08-18 04:59:21]
lab@vMX-2>Time to see what detail we have for this particular BGP peer. We’ll also do a show route for the peer address.
lab@vMX-2> show bgp neighbor 192.168.2.1
Peer: 192.168.2.1 AS 65001 Local: unspecified AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Idle Flags: <PeerInterfaceUnknown>
Last State: NoState Last Event: NoEvent
Last Error: None
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-direct size 1048576 files 10
lab@vMX-2> show route 192.168.2.1
lab@vMX-2>Now, this provides some useful information. The State is Idle, but there is a flag saying the peer interface is unknown. This combined with the lack of a route might point us towards looking at the interface.
lab@vMX-2> show interfaces ge* terse
Interface Admin Link Proto Local Remote
ge-0/0/0 up up
ge-0/0/0.0 up up inet 192.168.1.1/30
multiservice
ge-0/0/1 down down
ge-0/0/1.0 up down inet 192.168.2.2/30
multiservice
ge-0/0/2 up up
ge-0/0/2.0 up up inet 192.168.2.6/30
multiservice
ge-0/0/3 up up
ge-0/0/3.16386 up up
...
<<Output removed for brevity>>
...
lab@vMX-2>If we bring the interface up again, BGP will be reestablished.
lab@vMX-2> show interfaces ge-0/0/1* terse
Interface Admin Link Proto Local Remote
ge-0/0/1 up up
ge-0/0/1.0 up up inet 192.168.2.2/30
multiservice
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 31 30 0 0 12:17 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 31 30 0 0 12:35 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 3 3 0 0 15 Establ
inet.0: 0/3/3/0
lab@vMX-2>BGP Traffic Being Filtered.
This can be hard to identify. Let’s look at vMX-2 and an eBGP session to vMX-3.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 76 75 0 0 32:57 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 77 76 0 0 33:15 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 0 0 0 1 9:37 Connect
lab@vMX-2> show bgp neighbor 192.168.2.1
Peer: 192.168.2.1+179 AS 65001 Local: 192.168.2.2 AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Connect Flags: <TryConnect>
Last State: Connect Last Event: ConnectRetry
Last Error: Hold Timer Expired Error
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 1
Last flap event: HoldTime
Error: 'Hold Timer Expired Error' Sent: 1 Recv: 0
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-direct size 1048576 files 10
lab@vMX-2>We can see that the BGP session to 192.168.2.1 is in the Connect state and looking at the BGP neighbor output the flag is TryConnect.
If we look at the traceoptions, we see the following output:
Aug 18 05:32:44.564185 task_timer_reset: reset BGP_65001.192.168.2.1_Connect
Aug 18 05:32:44.564200 task_timer_set_oneshot_latest: timer BGP_65001.192.168.2.1_Connect interval set to 8
Aug 18 05:32:52.565582 bgp_connect_timeout: BGP_65001.192.168.2.1_Connect
Aug 18 05:32:52.565615 task_timer_reset: reset BGP_65001.192.168.2.1_Connect
Aug 18 05:33:59.563193 task_process_events_internal: connect ready for BGP_65001.192.168.2.1
Aug 18 05:33:59.563233 bgp_connect_complete: error connecting to 192.168.2.1 (External AS 65001): Socket is not connected
Aug 18 05:33:59.563239 BGP peer 192.168.2.1 (External AS 65001) CLOSE: close socket 124
Aug 18 05:33:59.563244 task_close: close socket 124 task BGP_65001.192.168.2.1
Aug 18 05:33:59.563248 task_reset_socket: task BGP_65001.192.168.2.1 socket 124
Aug 18 05:33:59.563284 bgp_event: peer 192.168.2.1 (External AS 65001) old state Connect event OpenFail new state Idle
Aug 18 05:33:59.563587 bgp_event: peer 192.168.2.1 (External AS 65001) old state Idle event Start new state Connect
Aug 18 05:33:59.563614 bgp_connect_start: peer 192.168.2.1 (External AS 65001)
Aug 18 05:33:59.563619 bgp_event: peer 192.168.2.1 (External AS 65001) old state Connect event ConnectRetry new state Connect
Aug 18 05:33:59.563646 task_get_socket: domain AF_INET type SOCK_STREAM protocol 0 socket 124
Aug 18 05:33:59.563657 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option RecvBuffer(0) value 16384
Aug 18 05:33:59.563662 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option SendBuffer(1) value 16384
Aug 18 05:33:59.563667 task_set_socket: task BGP_65001.192.168.2.1 socket 124
Aug 18 05:33:59.563677 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option NonBlocking(8) value 1
Aug 18 05:33:59.563683 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option ReUsePort(38) value 1
Aug 18 05:33:59.563687 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option ReUseAddress(3) value 1
Aug 18 05:33:59.563692 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option TcpMd5Authentication(72)
Aug 18 05:33:59.563698 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option PathMTUDiscovery(26) value 0
Aug 18 05:33:59.563704 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option TOS(16) value 192
Aug 18 05:33:59.563710 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option DontRoute(5) value 1
Aug 18 05:33:59.563715 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option IifRestrict(36) value 1
Aug 18 05:33:59.563720 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option TTL(15) value 1
Aug 18 05:33:59.563727 Eval BFD turn-on, currently off for peer 192.168.2.1 (External AS 65001)(fl:)instance master (kern-id 0)
Aug 18 05:33:59.563734 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option TcpAggressiveTransmission(59) value 0
Aug 18 05:33:59.563739 task_set_option_internal: task BGP_65001.192.168.2.1 socket 124 option EnableTcpNoDelay(63) value 0
Aug 18 05:33:59.563835 task_timer_reset: reset BGP_65001.192.168.2.1_Connect
Aug 18 05:33:59.563849 task_timer_set_oneshot_latest: timer BGP_65001.192.168.2.1_Connect interval set to 8
Aug 18 05:34:07.564900 bgp_connect_timeout: BGP_65001.192.168.2.1_Connect
Aug 18 05:34:07.564940 task_timer_reset: reset BGP_65001.192.168.2.1_ConnectWe can see that the connection is failing, and the peering is moving between Connect, briefly through Idle, and then back to Connect. There isn’t much useful in the messages log either.
Let’s test this with a ping to see if we can reach the peer device.
lab@vMX-2> ping 192.168.2.1 count 2
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=4.368 ms
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=37.689 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.368/21.029/37.689/16.660 ms
lab@vMX-2>We can ping the other device. This shows that ping can be an unreliable testing tool. Just because a ping works, doesn’t mean that BGP will establish. Likewise, if Ping doesn’t work, that doesn’t also mean that BGP won’t be established.
In this case, we at least know the peer address is reachable.
This is one of the times when monitoring the interface traffic might be useful. So let’s look at interface ge-0/0/1 facing the BGP peer we’re having issues with.
lab@vMX-2> monitor traffic interface ge-0/0/1 size 1500 matching "tcp port 179"
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ge-0/0/1, capture size 1500 bytes
Reverse lookup for 192.168.2.2 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.
05:41:32.575484 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159032509 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:41:35.775803 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159035709 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:41:38.975105 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159038909 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:41:42.175464 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159042109 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:41:45.375754 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159045309 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:41:51.575904 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159051509 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:42:03.775457 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159063709 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:42:27.975493 Out IP 192.168.2.2.60551 > 192.168.2.1.bgp: S 2670784345:2670784345(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159087909 0,nop,nop,md5 d7f28ba93f0ece6926f6affe494bd47c>
05:42:44.576544 Out IP 192.168.2.2.60754 > 192.168.2.1.bgp: S 3720392176:3720392176(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159104510 0,nop,nop,md5 a9c5fd4a7e0b4e6a7adb2e158fb0e07f>
05:42:47.576881 Out IP 192.168.2.2.60754 > 192.168.2.1.bgp: S 3720392176:3720392176(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2159107510 0,nop,nop,md5 a9c5fd4a7e0b4e6a7adb2e158fb0e07f>
^C
10 packets received by filter
0 packets dropped by kernel
lab@vMX-2>From this, we see that vMX-2 is trying to connect, it is sending traffic out, but it never gets a response back.
All of this combined – the BGP session connections timing out, the ping working, not seeing any return traffic – might point us toward filtering.
On Junos, we often apply a filter to the lo0 interface to protect the routing engine. Basically, we apply a filter here to filter all traffic destined to the router itself.
lab@vMX-2> show configuration interfaces lo0
unit 0 {
family inet {
filter {
input ff-re-protect;
}
address 172.16.1.2/32;
}
}
lab@vMX-2>
We can see there is a firewall filter applied, let’s see what it is configured for.
lab@vMX-2> show configuration firewall family inet filter ff-re-protect
term allow-ssh {
from {
protocol tcp;
port ssh;
}
then accept;
}
term allow-internal-nets {
from {
source-address {
172.16.1.1/32;
192.168.1.0/30;
}
}
then accept;
}
term allow-icmp {
from {
protocol icmp;
icmp-type [ echo-request echo-reply time-exceeded unreachable ];
}
then accept;
}
term allow-external-bgp {
from {
source-address {
192.168.2.5/32;
172.16.2.3/32;
}
protocol tcp;
port 179;
}
}
term final {
then {
reject;
}
}
lab@vMX-2>This filter will:
- Allow SSH
- Allow all traffic from 172.16.1.1 and 192.168.1.0/30
- Allow a subset of ICMP traffic
- Allow BGP from 192.168.2.5 and 172.16.2.3
- Block all other traffic.
Note: This isn’t a well-written or secure filter. It is just a quick example to show what happens when BGP traffic is filtered.
And we have found the issue. None of the terms in the firewall filter will allow BGP traffic from 192.168.2.1. This means vMX-2 can send traffic to vMX-3, but it will filter out the return traffic.
If we quickly update the allow-external-bgp term to include 192.168.2.1 it should allow BGP to establish.
lab@vMX-2> edit
Entering configuration mode
[edit]
lab@vMX-2# set firewall family inet filter ff-re-protect term allow-external-bgp from source-address 192.168.2.1
[edit]
lab@vMX-2# show | compare
[edit firewall family inet filter ff-re-protect term allow-external-bgp from source-address]
172.16.2.3/32 { ... }
+ 192.168.2.1/32;
[edit]
lab@vMX-2# commit and-quit
commit complete
Exiting configuration mode
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 121 120 0 0 53:08 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 122 121 0 0 53:26 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 3 3 0 1 8 Establ
inet.0: 0/3/3/0
lab@vMX-2>For clarity, I waited several minutes after the commit before running the show bgp summary. We need to give BGP time.
In this specific example, the traffic was being filtered on vMX-2. In other cases, it could be blocked by the other BGP peer or even something in the middle between the two BGP peers. Something in the middle is less likely with an eBGP session as they tend to be configured over the direct interface to the other peer. It might be slightly more likely when using iBGP where we mostly peer to loopback addresses.
Overall, it can be difficult to track down where traffic is being filtered.
Wrong Peer Address
If you configure the wrong peer address, the actual error you receive will depend on what type of BGP session it is and whether you have a route to that wrong address.
Any error messages you receive are unlikely to tell you that you have the wrong address set. Instead, you need to interpret the output and recognize that you might have made a typo or have wrong information.
Having the wrong address configured could appear to be a number of other issues, as you will see below.
To demonstrate this, I’ll configure some static routes on vMX-2 and some new BGP sessions.
- 172.16.5.1 is pointed to vMX-3 (192.168.2.1)
- 172.16.6.1 is pointed to vMX-1 (192.168.1.2)
- Configure eBGP peers to 172.16.5.1 and 172.16.5.2
- Configure iBGP peers to 172.16.6.1 and 172.16.6.2
Wrong Peer Address – eBGP With A Route
To test this, I’ll configure an eBGP peer of 172.16.5.1 under the existing my-external-group group.
[edit]
lab@vMX-2# show protocols bgp
group my-external-group {
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
neighbor 172.16.5.1;
}
group my-internal-group {
type internal;
traceoptions {
file bgp-my-int-group size 1m files 10;
flag all;
}
local-address 172.16.1.2;
authentication-key "$9$GSjkmz39O1hfT1hSr8LGDi"; ## SECRET-DATA
export [ fix-nhs exp-default ];
neighbor 172.16.1.1;
}
[edit]
lab@vMX-2#Next, we look at the BGP summary information.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 4 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 91 91 0 0 39:13 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 74 73 0 1 32:11 Establ
inet.0: 3/3/3/0
172.16.5.1 65001 0 0 0 0 1:40 Idle
192.168.2.1 65001 74 73 0 1 32:12 Establ
inet.0: 0/3/3/0
lab@vMX-2> The BGP peer is stuck in Idle. We can look at the neighbor information and see what it says.
lab@vMX-2> show bgp neighbor 172.16.5.1
Peer: 172.16.5.1 AS 65001 Local: unspecified AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Idle Flags: <PeerInterfaceError>
Last State: NoState Last Event: NoEvent
Last Error: None
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
lab@vMX-2> The key information here is the flag of PeerInterfaceError.
Next, we can look at the log file.
lab@vMX-2> show log messages | last 10
Aug 24 04:12:05 vMX-2 rpd[8797]: ted_client reset
Aug 24 04:12:05 vMX-2 rpd[8797]: L2VPN acquiring mastership for primary
Aug 24 04:12:05 vMX-2 rpd[8797]: BGP_SET_INTERFACE_FAILED: bgp_set_peer_if: BGP peer 172.16.5.1 (External AS 65001) interface not found. Leaving peer idled (instance master)
Aug 24 04:12:05 vMX-2 rpd[8797]: L2CKT acquiring mastership for primary
Aug 24 04:12:05 vMX-2 rpd[8797]: RPD_COMMIT_SYNC_STATUS: RPD commit sync completed on MASTER
Aug 24 04:12:05 vMX-2 rpd[8797]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs started on MASTER
Aug 24 04:12:05 vMX-2 rpd[8797]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs completed on MASTER
Aug 24 04:12:05 vMX-2 mgd[9081]: UI_COMMIT_COMPLETED: : commit complete
Aug 24 04:13:43 vMX-2 mgd[9081]: UI_DBASE_LOGOUT_EVENT: User 'lab' exiting configuration mode
lab@vMX-2>A couple of things worth noting in this output. First, we have the RPD telling us the peer interface could not be found and this message appears as part of other logs created when a commit is issued.
The second is that the log entry doesn’t repeat. BGP isn’t trying to connect at all, and won’t until we fix the issue.
In this case, we have the wrong address. But a similar error message can appear when trying to configure eBGP multi-hop, or if we have a misconfigured interface address.
Wrong Peer Address – eBGP Without A Route
To test this, I’ll configure an eBGP peer of 172.16.5.2 under the existing my-external-group group.
[edit]
lab@vMX-2# show protocols bgp
group my-external-group {
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
neighbor 172.16.5.2;
}
group my-internal-group {
type internal;
traceoptions {
file bgp-my-int-group size 1m files 10;
flag all;
}
local-address 172.16.1.2;
authentication-key "$9$GSjkmz39O1hfT1hSr8LGDi"; ## SECRET-DATA
export [ fix-nhs exp-default ];
neighbor 172.16.1.1;
}
[edit]
lab@vMX-2#The BGP summary and neighbor information look very similar to the previous example.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 4 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 126 127 0 0 55:03 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 108 107 0 1 48:01 Establ
inet.0: 3/3/3/0
172.16.5.2 65001 0 0 0 0 53 Idle
192.168.2.1 65001 109 109 0 1 48:02 Establ
inet.0: 0/3/3/0
lab@vMX-2> show bgp neighbor 172.16.5.2
Peer: 172.16.5.2 AS 65001 Local: unspecified AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Idle Flags: <PeerInterfaceError>
Last State: NoState Last Event: NoEvent
Last Error: None
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
lab@vMX-2>The log message is also the same.
lab@vMX-2> show log messages | last 10
Aug 24 04:28:42 vMX-2 rpd[8797]: ted_client reset
Aug 24 04:28:42 vMX-2 rpd[8797]: L2VPN acquiring mastership for primary
Aug 24 04:28:42 vMX-2 rpd[8797]: BGP_SET_INTERFACE_FAILED: bgp_set_peer_if: BGP peer 172.16.5.2 (External AS 65001) interface not found. Leaving peer idled (instance master)
Aug 24 04:28:42 vMX-2 rpd[8797]: L2CKT acquiring mastership for primary
Aug 24 04:28:42 vMX-2 rpd[8797]: RPD_COMMIT_SYNC_STATUS: RPD commit sync completed on MASTER
Aug 24 04:28:42 vMX-2 rpd[8797]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs started on MASTER
Aug 24 04:28:42 vMX-2 rpd[8797]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs completed on MASTER
Aug 24 04:28:42 vMX-2 mgd[9081]: UI_COMMIT_COMPLETED: : commit complete
Aug 24 04:29:32 vMX-2 mgd[9081]: UI_DBASE_LOGOUT_EVENT: User 'lab' exiting configuration mode
lab@vMX-2> The thing to take away here is that BGP remains idle as it cannot find an outgoing interface for this eBGP peer. By default, eBGP has the TTL set to 1 and expects the peer to be available directly over an interface.
Regardless of whether we have a route to the address or not, this interface check occurs first and causes both of these examples to have the same behavior.
Wrong Peer Address – iBGP With A Route
This time we will configure a BGP peer with the wrong address under the my-internal-group stanza. The vMX-2 router is configured with an iBGP peer of 172.16.6.1, which it does have a route for.
[edit]
lab@vMX-2# show protocols bgp
group my-external-group {
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
}
group my-internal-group {
type internal;
traceoptions {
file bgp-my-int-group size 1m files 10;
flag all;
}
local-address 172.16.1.2;
authentication-key "$9$GSjkmz39O1hfT1hSr8LGDi"; ## SECRET-DATA
export [ fix-nhs exp-default ];
neighbor 172.16.1.1;
neighbor 172.16.6.1;
}
[edit]
lab@vMX-2#We can now look at the BGP summary and neighbor information.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 4 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 140 141 0 0 1:01:35 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 123 122 0 1 54:33 Establ
inet.0: 3/3/3/0
172.16.6.1 65002 0 0 0 0 1:44 Connect
192.168.2.1 65001 123 123 0 1 54:34 Establ
inet.0: 0/3/3/0
lab@vMX-2> show bgp neighbor 172.16.6.1
Peer: 172.16.6.1+179 AS 65002 Local: 172.16.1.2 AS 65002
Group: my-internal-group Routing-Instance: master
Forwarding routing-instance: master
Type: Internal State: Connect Flags: <>
Last State: Connect Last Event: ConnectRetry
Last Error: None
Export: [ fix-nhs exp-default ]
Options: <Preference LocalAddress AuthKey Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Local Address: 172.16.1.2 Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Trace options: all
Trace file: /var/log/bgp-my-int-group size 1048576 files 10
lab@vMX-2> The iBGP session is stuck in the Connect state. Although there is no actual router configured with the address 172.16.6.1, because we have a route for it, vMX-2 thinks it can connect to it. It is stuck in Connect waiting for the TCP connection to complete.
This differs from eBGP because iBGP doesn’t have the default TTL of 1 limitation. It expects that iBGP sessions will be indirect.
The log messages file isn’t very helpful in this case either.
lab@vMX-2> show log messages | last 10
Aug 24 04:34:22 vMX-2 rpd[9874]: ted_client reset
Aug 24 04:34:22 vMX-2 ffp: "dynamic-profiles": No change to profiles
Aug 24 04:34:22 vMX-2 rpd[8797]: ted_client reset
Aug 24 04:34:22 vMX-2 rpd[8797]: L2VPN acquiring mastership for primary
Aug 24 04:34:22 vMX-2 rpd[8797]: L2CKT acquiring mastership for primary
Aug 24 04:34:22 vMX-2 rpd[8797]: RPD_COMMIT_SYNC_STATUS: RPD commit sync completed on MASTER
Aug 24 04:34:22 vMX-2 rpd[8797]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs started on MASTER
Aug 24 04:34:22 vMX-2 rpd[8797]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs completed on MASTER
Aug 24 04:34:22 vMX-2 mgd[9081]: UI_COMMIT_COMPLETED: : commit complete
Aug 24 04:35:21 vMX-2 mgd[9081]: UI_DBASE_LOGOUT_EVENT: User 'lab' exiting configuration mode
lab@vMX-2>If we look at the traceoptions output for our iBGP peers we do get some additional information.
Aug 24 04:45:04.543456 bgp_event: peer 172.16.6.1 (Internal AS 65002) old state Connect event OpenFail new state Idle
Aug 24 04:45:04.543901 bgp_event: peer 172.16.6.1 (Internal AS 65002) old state Idle event Start new state Active
Aug 24 04:47:19.933681 bgp_connect_timeout: BGP_65002.172.16.6.1_Connect
Aug 24 04:47:19.933884 bgp_connect_start: peer 172.16.6.1 (Internal AS 65002)
Aug 24 04:47:19.933894 bgp_event: peer 172.16.6.1 (Internal AS 65002) old state Active event ConnectRetry new state Connect
Aug 24 04:47:19.934307 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option RecvBuffer(0) value 16384
Aug 24 04:47:19.934313 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option SendBuffer(1) value 16384
Aug 24 04:47:19.934338 task_set_socket: task BGP_65002.172.16.6.1 socket 125
Aug 24 04:47:19.934352 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option NonBlocking(8) value 1
Aug 24 04:47:19.934358 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option ReUsePort(38) value 1
Aug 24 04:47:19.934363 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option ReUseAddress(3) value 1
Aug 24 04:47:19.934368 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option TcpMd5Authentication(72)
Aug 24 04:47:19.934375 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option PathMTUDiscovery(26) value 0
Aug 24 04:47:19.934382 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option TOS(16) value 192
Aug 24 04:47:19.934388 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option TcpAggressiveTransmission(59) value 0
Aug 24 04:47:19.934394 task_set_option_internal: task BGP_65002.172.16.6.1 socket 125 option EnableTcpNoDelay(63) value 0
Aug 24 04:47:19.934784 task_timer_reset: reset BGP_65002.172.16.6.1_Connect
Aug 24 04:47:19.934799 task_timer_set_oneshot_latest: timer BGP_65002.172.16.6.1_Connect interval set to 2:28
Aug 24 04:47:32.549182 task_process_events_internal: connect ready for BGP_65002.172.16.6.1
Aug 24 04:47:32.549253 bgp_connect_complete: error connecting to 172.16.6.1 (Internal AS 65002): Socket is not connected
Aug 24 04:47:32.549263 BGP peer 172.16.6.1 (Internal AS 65002) CLOSE: close socket 125
Aug 24 04:47:32.549268 task_close: close socket 125 task BGP_65002.172.16.6.1
Aug 24 04:47:32.549273 task_reset_socket: task BGP_65002.172.16.6.1 socket 125
Aug 24 04:47:32.549310 bgp_event: peer 172.16.6.1 (Internal AS 65002) old state Connect event OpenFail new state Idle
Aug 24 04:47:32.549912 bgp_event: peer 172.16.6.1 (Internal AS 65002) old state Idle event Start new state ActiveFrom the output, we can see that TCP is not connecting and the peer is cycling through several states.
The exact output the show bgp summary command shows will depend on when we run it. If we run it several times we should get slightly different states for this neighbor.
lab@vMX-2> show bgp summary | match 172.16.6.1 | refresh 10
---(refreshed at 2024-08-24 05:04:09 UTC)---
172.16.6.1 65002 0 0 0 0 29:46 Active
---(refreshed at 2024-08-24 05:04:19 UTC)---
172.16.6.1 65002 0 0 0 0 29:57 Active
---(refreshed at 2024-08-24 05:04:29 UTC)---
172.16.6.1 65002 0 0 0 0 30:07 Active
---(refreshed at 2024-08-24 05:04:39 UTC)---
172.16.6.1 65002 0 0 0 0 30:17 Connect
---(refreshed at 2024-08-24 05:04:49 UTC)---
172.16.6.1 65002 0 0 0 0 30:27 Active
---(refreshed at 2024-08-24 05:04:59 UTC)---
172.16.6.1 65002 0 0 0 0 30:37 Active
---(refreshed at 2024-08-24 05:05:10 UTC)---
172.16.6.1 65002 0 0 0 0 30:47 Active
---(refreshed at 2024-08-24 05:05:20 UTC)---
172.16.6.1 65002 0 0 0 0 30:57 Active
---(refreshed at 2024-08-24 05:05:30 UTC)---
172.16.6.1 65002 0 0 0 0 31:07 Active
---(*more 100%)---[abort]
lab@vMX-2>Note that none of the error messages or output tell us that we’ve misconfigured the neighbor statement.
Wrong Peer Address – iBGP Without A Route
In this example, we have changed the configuration and now have an iBGP peer for 172.16.6.2, which we do not have a route for.
[edit]
lab@vMX-2# show protocols bgp
group my-external-group {
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
}
group my-internal-group {
type internal;
traceoptions {
file bgp-my-int-group size 1m files 10;
flag all;
}
local-address 172.16.1.2;
authentication-key "$9$GSjkmz39O1hfT1hSr8LGDi"; ## SECRET-DATA
export [ fix-nhs exp-default ];
neighbor 172.16.1.1;
neighbor 172.16.6.2;
}
[edit]
lab@vMX-2#We’ll start by looking at the BGP summary and neighbor.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 4 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 213 214 0 0 1:34:35 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 196 195 0 1 1:27:33 Establ
inet.0: 3/3/3/0
172.16.6.2 65002 0 0 0 0 1:00 Active
192.168.2.1 65001 197 196 0 1 1:27:34 Establ
inet.0: 0/3/3/0
lab@vMX-2> show bgp neighbor 172.16.6.2
Peer: 172.16.6.2 AS 65002 Local: 172.16.1.2 AS 65002
Group: my-internal-group Routing-Instance: master
Forwarding routing-instance: master
Type: Internal State: Active Flags: <>
Last State: Idle Last Event: Start
Last Error: None
Export: [ fix-nhs exp-default ]
Options: <Preference LocalAddress AuthKey Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Local Address: 172.16.1.2 Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Trace options: all
Trace file: /var/log/bgp-my-int-group size 1048576 files 10
lab@vMX-2>The neighbor is idle but we still don’t have a lot of detail as to what is wrong here.
The messages log is more helpful in this case, telling us we don’t have a route.
lab@vMX-2> show log messages | last 10
Aug 24 05:09:11 vMX-2 rpd[8797]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.6.2: Connecting to 172.16.6.2+179 failed: No route to host
Aug 24 05:09:11 vMX-2 rpd[8797]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.6.2 (Internal AS 65002) (instance master): No route to host
Aug 24 05:09:43 vMX-2 rpd[8797]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.6.2: Connecting to 172.16.6.2+179 failed: No route to host
Aug 24 05:09:43 vMX-2 rpd[8797]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.6.2 (Internal AS 65002) (instance master): No route to host
Aug 24 05:10:15 vMX-2 rpd[8797]: JTASK_IO_CONNECT_FAILED: BGP_65002.172.16.6.2: Connecting to 172.16.6.2+179 failed: No route to host
Aug 24 05:10:15 vMX-2 rpd[8797]: BGP_CONNECT_FAILED: bgp_connect_start: connect 172.16.6.2 (Internal AS 65002) (instance master): No route to host
lab@vMX-2>Although the logs tell us something, they do not clearly tell us we have configured the wrong address. We need to know that we should be checking the address is configured correctly and that we have a route to that address.
Peer Set As Passive
When we set a BGP peer to passive the router will wait for a connection from the remote peer. This could be mistaken as a fault, as we generally expect the peers to be in the established state.
When set to passive it will not try and establish a connection itself. If both sides are set to passive the BGP session will not come up.
Setting a peer to be passive is often used when we don’t want our router to waste time and resources trying to connect to another device.
One example is service provider networks, where BGP is used between the service provider and the customer. The provider side can be set to passive so that it will wait for the customer side to initiate the BGP sessions. The customer could decide to turn their router off for the weekend. They could be suffering from a power outage, or they may have simply unplugged the connection at their end.
In these situations it can be a waste of resources to try establishing BGP to the customer’s router, instead, our router can patiently wait for the customer to sort their issues out and connect to us.
On vMX-2 we have set the 192.168.2.1 peer to be passive. The BGP summary and neighbor output are shown below.
Note: I have set the peer on the other side to passive as well, so neither side is trying to connect.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 256 257 0 0 1:53:40 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 239 238 0 1 1:46:38 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 1 0 0 2 10:44 Active
lab@vMX-2> show bgp neighbor 192.168.2.1
Peer: 192.168.2.1 AS 65001 Local: 192.168.2.2 AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Active Flags: <>
Last State: Active Last Event: Stop
Last Error: Hold Timer Expired Error
Options: <Preference Passive AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 2
Last flap event: RecvNotify
Error: 'Hold Timer Expired Error' Sent: 1 Recv: 0
Error: 'Cease' Sent: 0 Recv: 5
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-direct size 1048576 files 10
lab@vMX-2>There isn’t any information in the log messages output. Since we already have traceoptions turned on, we can check that log and see many entries similar to the following:
Aug 24 05:29:44.610612 bgp_connect_timeout: BGP_65001.192.168.2.1_Connect
Aug 24 05:29:44.610639 bgp_event: peer 192.168.2.1 (External AS 65001) old state Active event Stop new state Active
Aug 24 05:29:44.610695 task_timer_reset: reset BGP_65001.192.168.2.1_Connect
Aug 24 05:29:44.610706 task_timer_set_oneshot_latest: timer BGP_65001.192.168.2.1_Connect interval set to 8
Aug 24 05:29:52.617320 bgp_connect_timeout: BGP_65001.192.168.2.1_Connect
Aug 24 05:29:52.617349 bgp_event: peer 192.168.2.1 (External AS 65001) old state Active event Stop new state Active
Aug 24 05:29:52.617409 task_timer_reset: reset BGP_65001.192.168.2.1_Connect
Aug 24 05:29:52.617420 task_timer_set_oneshot_latest: timer BGP_65001.192.168.2.1_Connect interval set to 8
Aug 24 05:30:00.619485 bgp_connect_timeout: BGP_65001.192.168.2.1_Connect
Aug 24 05:30:00.619515 bgp_event: peer 192.168.2.1 (External AS 65001) old state Active event Stop new state Active
Aug 24 05:30:00.619616 task_timer_reset: reset BGP_65001.192.168.2.1_Connect
Aug 24 05:30:00.619631 task_timer_set_oneshot_latest: timer BGP_65001.192.168.2.1_Connect interval set to 8If we remove the passive statement from the neighbor configuration on vMX-3, so only vMX-2 is set to passive, we should see the neighbor come up on vMX-2.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 271 272 0 0 2:00:08 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 253 253 0 1 1:53:06 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 4 3 0 2 6 Establ
inet.0: 0/3/3/0
lab@vMX-2>If we look at the configuration, we can see that vMX-2 still has that peer set to passive.
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
passive;
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
[edit]
lab@vMX-2#BGP Peer Is Shutdown
In this scenario, we will configure vMX-2 to shut down the peer to 192.168.2.1
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
shutdown;
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
[edit]
lab@vMX-2#If we only look at the BGP summary information, we might see the peer being in Idle state and think it is an issue.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 281 282 0 0 2:04:45 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 263 263 0 1 1:57:43 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 12 11 0 3 45 Idle
lab@vMX-2> show bgp neighbor 192.168.2.1
Peer: 192.168.2.1 AS 65001 Local: 192.168.2.2 AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Idle Flags: <>
Last State: Established Last Event: Stop
Last Error: Cease
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv Shutdown>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 3
Last flap event: Stop
Error: 'Hold Timer Expired Error' Sent: 1 Recv: 0
Error: 'Cease' Sent: 1 Recv: 5
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-direct size 1048576 files 10
lab@vMX-2>In this case, it is intentional and we can see that under the detailed neighbor information, in particular the Shutdown option.
Where this does get interesting is with the other peer still configured. If we look at the log files, vMX-2 and vMX-3 show different results.
vMX-2:
lab@vMX-2> show log messages | last 10
Aug 24 05:39:28 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 192.168.2.1+51536 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 192.168.2.1+51536 (proto) from AS 65001 found (peer idled) in master(ge-0/0/1.0), dropping him
Aug 24 05:39:37 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 192.168.2.1+51381 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 192.168.2.1+51381 (proto) from AS 65001 found (peer idled) in master(ge-0/0/1.0), dropping him
Aug 24 05:40:09 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 192.168.2.1+64888 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 192.168.2.1+64888 (proto) from AS 65001 found (peer idled) in master(ge-0/0/1.0), dropping him
Aug 24 05:40:41 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 192.168.2.1+51993 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 192.168.2.1+51993 (proto) from AS 65001 found (peer idled) in master(ge-0/0/1.0), dropping him
lab@vMX-2> From this output, we can see vMX-2 sending a Notification message to vMX-3, saying the connection is rejected.
vMX-3:
lab@vMX-3> show log messages | last 10
Aug 24 05:34:32 vMX-3 rpd[17981]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs started on MASTER
Aug 24 05:34:32 vMX-3 rpd[17981]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs completed on MASTER
Aug 24 05:34:32 vMX-3 mgd[18761]: UI_COMMIT_COMPLETED: : commit complete
Aug 24 05:34:32 vMX-3 mgd[18761]: UI_DBASE_LOGOUT_EVENT: User 'lab' exiting configuration mode
Aug 24 05:38:33 vMX-3 rpd[17981]: bgp_handle_notify:4541: NOTIFICATION received from 192.168.2.2 (External AS 65002): code 6 (Cease) subcode 2 (Administratively Shutdown)
Aug 24 05:38:35 vMX-3 rpd[17981]: bgp_handle_notify:4541: NOTIFICATION received from 192.168.2.2 (External AS 65002): code 6 (Cease) subcode 5 (Connection Rejected)
Aug 24 05:38:57 vMX-3 last message repeated 5 times
Aug 24 05:40:42 vMX-3 last message repeated 7 times
lab@vMX-3>vMX-3 logs show it receiving these notifications. Also, note that one of the messages shows the neighbor is administratively down.
From vMX-3’s perspective, the neighbor is in the Active state. It will transition through this to the Connect state but after receiving the Notification it will move back to Active again.
lab@vMX-3> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.2 65002 279 277 0 0 2:04:05 Establ
inet.0: 1/1/1/0
192.168.2.2 65002 1 0 0 2 7:06 Active
lab@vMX-3>To get this peering session established again, we need to change vMX-2 to remove the shutdown command from the 192.168.2.2 neighbor.
Authentication Issue.
For this test, I have configured vMX-2 with the wrong password for the direct peer to vMX-3.
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
authentication-key "$9$.f5FB1hSyK36Cu0IrlKM8L7VsYoUjqKMds2gaJGDj"; ## SECRET-DATA
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
[edit]
lab@vMX-2#If we look at vMX-2, we can see the BGP session is in the Connect state.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 330 329 0 0 2:26:37 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 312 311 0 1 2:19:35 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 0 0 0 4 3:59 Connect
lab@vMX-2> show bgp neighbor 192.168.2.1
Peer: 192.168.2.1+179 AS 65001 Local: 192.168.2.2 AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Connect Flags: <TryConnect>
Last State: Connect Last Event: ConnectRetry
Last Error: Hold Timer Expired Error
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 4
Last flap event: HoldTime
Error: 'Hold Timer Expired Error' Sent: 2 Recv: 0
Error: 'Cease' Sent: 1 Recv: 5
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-direct size 1048576 files 10
lab@vMX-2>The messages log does show some useful information, mentioning a wrong MD5 Digest.
lab@vMX-2> show log messages | last 10
Aug 24 06:00:21 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:58339 wrong MD5 digest
Aug 24 06:00:25 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:179 wrong MD5 digest
Aug 24 06:00:45 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:58339 wrong MD5 digest
Aug 24 06:01:01 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:60648 wrong MD5 digest
Aug 24 06:01:04 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:60648 wrong MD5 digest
Aug 24 06:01:07 vMX-2 mgd[10542]: UI_DBASE_LOGOUT_EVENT: User 'lab' exiting configuration mode
Aug 24 06:01:07 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:60648 wrong MD5 digest
Aug 24 06:01:23 vMX-2 last message repeated 4 times
Aug 24 06:01:29 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:179 wrong MD5 digest
Aug 24 06:01:35 vMX-2 kernel: tcp_auth_ok: Packet from 192.168.2.1:60648 wrong MD5 digest
lab@vMX-2>This wrong digest, and the fact that at least some of the packets are from 192.168.2.1:179 show this is related to BGP.
vMX-3 shows very similar output.
lab@vMX-3> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.2 65002 319 318 0 0 2:22:37 Establ
inet.0: 1/1/1/0
192.168.2.2 65002 0 0 0 3 6:56 Connect
lab@vMX-3> show log messages | last 10
Aug 24 06:02:34 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:62239 wrong MD5 digest
Aug 24 06:02:35 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:61354 wrong MD5 digest
Aug 24 06:02:46 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:62239 wrong MD5 digest
Aug 24 06:03:11 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:62239 wrong MD5 digest
Aug 24 06:03:27 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:51410 wrong MD5 digest
Aug 24 06:03:33 vMX-3 last message repeated 2 times
Aug 24 06:03:37 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:51410 wrong MD5 digest
Aug 24 06:03:39 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:61354 wrong MD5 digest
Aug 24 06:03:40 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.2:51410 wrong MD5 digest
Aug 24 06:04:01 vMX-3 last message repeated 3 times
lab@vMX-3>To fix this, we need to ensure both sides use the same authentication method and key. Since I broke this by setting the wrong key at the neighbor level, I can remove that statement and let it use the right key, which is configured at the group level.
[edit]
lab@vMX-2# show | compare
[edit protocols bgp group my-external-group neighbor 192.168.2.1]
- authentication-key "$9$.f5FB1hSyK36Cu0IrlKM8L7VsYoUjqKMds2gaJGDj"; ## SECRET-DATA
[edit]
lab@vMX-2# commit and-quit
commit complete
Exiting configuration mode
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 341 340 0 0 2:31:47 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 324 323 0 1 2:24:45 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 6 4 0 4 1:10 Establ
inet.0: 0/3/3/0
lab@vMX-2> In this case, I knew which side had the wrong key set. In the real world, one or both sides might need to reset the authentication key to a known value. You can’t tell which side is wrong, only that they don’t match.
Wrong Peer AS
In this example, we’ll set the wrong peer AS on vMX-2 for the direct eBGP peering session facing vMX-3.
vMX-2
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
peer-as 65005;
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
[edit]
lab@vMX-2#The BGP summary and neighbor look as follows on vMX-2.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 1119 1112 0 0 8:20:54 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 1098 1094 0 1 8:13:52 Establ
inet.0: 3/3/3/0
192.168.2.1 65005 0 0 0 0 46 Active
lab@vMX-2> show bgp neighbor 192.168.2.1
Peer: 192.168.2.1 AS 65005 Local: 192.168.2.2 AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Active Flags: <>
Last State: Idle Last Event: Start
Last Error: Open Message Error
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Error: 'Open Message Error' Sent: 9 Recv: 0
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-direct size 1048576 files 10
lab@vMX-2>We can see the state is Active and there is an error under the neighbor output.
This is backed up by the messages log file. You can see we are sending a Notification message about bad peer AS.
lab@vMX-2> show log messages | last 10
Aug 24 11:55:53 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 192.168.2.1+53082 (proto): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: no group for 192.168.2.1+53082 (proto) from AS 65001 found (peer as mismatch) in master(ge-0/0/1.0), dropping him
Aug 24 11:56:17 vMX-2 rpd[8797]: bgp_process_open:4372: NOTIFICATION sent to 192.168.2.1 (External AS 65005): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: peer 192.168.2.1 (External AS 65005) claims 65001, 65005 configured
Aug 24 11:56:49 vMX-2 rpd[8797]: bgp_process_open:4372: NOTIFICATION sent to 192.168.2.1 (External AS 65005): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: peer 192.168.2.1 (External AS 65005) claims 65001, 65005 configured
Aug 24 11:56:49 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 192.168.2.1+53387 (proto): code 2 (Open Message Error) subcode 2 (bad peer AS number), Reason: no group for 192.168.2.1+53387 (proto) from AS 65001 found (peer as mismatch) in master(ge-0/0/1.0), dropping him
lab@vMX-2>From vMX-2’s perspective, vMX-3 claims to be AS 65001 but we have that peer configured as 65005. Since this doesn’t match, vMX-2 is sending a Notification and tearing down the BGP session.
If we look at vMX-3, we can see similar issues.
lab@vMX-3> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.2 65002 1102 1104 0 0 8:16:33 Establ
inet.0: 1/1/1/0
192.168.2.2 65002 1 0 0 4 3:26 Active
lab@vMX-3> show log messages | last 10
Aug 24 11:56:49 vMX-3 rpd[17981]: bgp_pp_recv:5258: NOTIFICATION sent to 192.168.2.2+56364 (proto): code 6 (Cease) subcode 7 (Connection Collision Resolution), Reason: dropping 192.168.2.2+56364 (proto), connection collision prefers 192.168.2.2 (External AS 65002)
Aug 24 11:56:49 vMX-3 rpd[17981]: bgp_handle_notify:4541: NOTIFICATION received from 192.168.2.2 (External AS 65002): code 2 (Open Message Error) subcode 2 (bad peer AS number)
Aug 24 11:57:21 vMX-3 rpd[17981]: bgp_handle_notify:4541: NOTIFICATION received from 192.168.2.2 (External AS 65002): code 2 (Open Message Error) subcode 2 (bad peer AS number)
Aug 24 11:57:54 vMX-3 rpd[17981]: bgp_pp_recv:5258: NOTIFICATION sent to 192.168.2.2+55720 (proto): code 6 (Cease) subcode 7 (Connection Collision Resolution), Reason: dropping 192.168.2.2+55720 (proto), connection collision prefers 192.168.2.2 (External AS 65002)
Aug 24 11:57:54 vMX-3 rpd[17981]: bgp_handle_notify:4541: NOTIFICATION received from 192.168.2.2 (External AS 65002): code 2 (Open Message Error) subcode 2 (bad peer AS number)
lab@vMX-3> There are a couple of things to note with these log entries. First, vMX-3 is receiving Notifications from vMX-2 about an Open message error that relates to the peer AS that is set. Secondly, vMX-3 is sending a Notification to resolve a connection collision.
The connection collision messages aren’t an issue. BGP only needs one TCP connection between the two peers. If both peers start at the same time, they will each try to create a TCP connection. When this happens a connection collision occurs and one of the TCP connections needs to be dropped.
In this situation, the session initiated by the router with the highest BGP identifier should be kept.
If we go back to vMX-2 and remove the bad AS number configured directly under the neighbor, it will use the correct peer AS that is defined at the group level in this case.
Once we do this, BGP will re-establish itself.
Various eBGP Multihop Issues
Configuring eBGP multi-hop can throw up some strange error messages. We need to configure the multi-hop setting, the TTL value, and the local address to use.
Initially, let’s set this incorrectly, with vMX-2 configured without multi-hop and vMX-3 with its configuration for this peer deactivated.
vMX-2:
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
}
[edit]
lab@vMX-2#And on vMX-3:
[edit]
lab@vMX-3# show protocols bgp
group my-external-group {
authentication-key "$9$35ou/AuRhr8X-O1X-VwaJ369"; ## SECRET-DATA
export exp-static;
peer-as 65002;
neighbor 192.168.2.2;
inactive: neighbor 172.16.1.2 {
multihop {
ttl 2;
}
local-address 172.16.2.3;
}
}
[edit]
lab@vMX-3#In the current state, vMX-2 is incorrectly configured for eBGP multihop, and vMX-3 has the peer deactivated so it won’t be trying to connect. On vMX-2 we see the following information.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 1189 1182 0 0 8:52:36 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 0 0 0 0 4:34 Idle
192.168.2.1 65001 29 28 0 0 12:05 Establ
inet.0: 3/3/3/0
lab@vMX-2> show bgp neighbor 172.16.2.3
Peer: 172.16.2.3 AS 65001 Local: unspecified AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Idle Flags: <PeerInterfaceError>
Last State: NoState Last Event: NoEvent
Last Error: None
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-loopback size 1048576 files 10
lab@vMX-2>This is similar to what we’ve seen with some of the other examples. The peer is Idle and the neighbor output shows a PeerInferfaceError which indicates BGP can’t determine the outgoing interface for this connection.
The interesting thing is when we activate the BGP peer config on vMX-3 causing it to start trying to establish the session to vMX-2.
vMX-3:
[edit]
lab@vMX-3# show | compare
[edit protocols bgp group my-external-group]
! active: neighbor 172.16.1.2 { ... }
[edit]
lab@vMX-3# show protocols bgp
group my-external-group {
authentication-key "$9$35ou/AuRhr8X-O1X-VwaJ369"; ## SECRET-DATA
export exp-static;
peer-as 65002;
neighbor 192.168.2.2;
neighbor 172.16.1.2 {
multihop {
ttl 2;
}
local-address 172.16.2.3;
}
}
[edit]
lab@vMX-3# commit and-quit
commit complete
Exiting configuration mode
lab@vMX-3> On vMX-3 we see the following:
lab@vMX-3> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.2 65002 1 0 0 0 1:17 Active
192.168.2.2 65002 38 36 0 4 15:37 Establ
inet.0: 1/1/1/0
lab@vMX-3> show bgp neighbor 172.16.1.2
Peer: 172.16.1.2 AS 65002 Local: 172.16.2.3 AS 65001
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Active Flags: <>
Last State: Idle Last Event: Start
Last Error: None
Export: [ exp-static ]
Options: <Multihop Preference LocalAddress AuthKey Ttl PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Local Address: 172.16.2.3 Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Error: 'Cease' Sent: 0 Recv: 11
lab@vMX-3> show log messages | last 10
Aug 24 12:29:24 vMX-3 rpd[17981]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs completed on MASTER
Aug 24 12:29:24 vMX-3 mgd[18761]: UI_COMMIT_COMPLETED: : commit complete
Aug 24 12:29:24 vMX-3 mgd[18761]: UI_DBASE_LOGOUT_EVENT: User 'lab' exiting configuration mode
Aug 24 12:29:26 vMX-3 rpd[17981]: bgp_handle_notify:4541: NOTIFICATION received from 172.16.1.2 (External AS 65002): code 6 (Cease) subcode 5 (Connection Rejected)
Aug 24 12:29:32 vMX-3 last message repeated 3 times
Aug 24 12:29:57 vMX-3 last message repeated 3 times
Aug 24 12:30:05 vMX-3 rpd[17981]: bgp_handle_notify:4541: NOTIFICATION received from 172.16.1.2 (External AS 65002): code 6 (Cease) subcode 5 (Connection Rejected)
Aug 24 12:30:29 vMX-3 last message repeated 3 times
lab@vMX-3>The peer is in an active state. There isn’t a lot of information about the neighbor itself, and in the log file, we can see Notifications are being received that the connection is rejected. To be clear, at this point, vMX-3 is correctly configured.
If we look at vMX-2 we can start to see some interesting stuff.
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 1201 1194 0 0 8:58:02 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 0 0 0 0 10:00 Idle
192.168.2.1 65001 41 41 0 0 17:31 Establ
inet.0: 3/3/3/0
lab@vMX-2> show bgp neighbor 172.16.2.3
Peer: 172.16.2.3 AS 65001 Local: unspecified AS 65002
Group: my-external-group Routing-Instance: master
Forwarding routing-instance: master
Type: External State: Idle Flags: <PeerInterfaceError>
Last State: NoState Last Event: NoEvent
Last Error: None
Options: <Preference AuthKey PeerAS Refresh>
Options: <GracefulShutdownRcv>
Authentication key is configured
Holdtime: 90 Preference: 170
Graceful Shutdown Receiver local-preference: 0
Number of flaps: 0
Trace options: all
Trace file: /var/log/bgp-ext-neighbor-loopback size 1048576 files 10
lab@vMX-2> show log messages | last 10
Aug 24 12:31:01 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 172.16.2.3+55714 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 172.16.2.3+55714 (proto) from AS 65001 found (eBGP connection attempt to an address that does not belong to the interface that peer connected to) in master(ge-0/0/2.0), dropping him
Aug 24 12:31:33 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 172.16.2.3+64435 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 172.16.2.3+64435 (proto) from AS 65001 found (eBGP connection attempt to an address that does not belong to the interface that peer connected to) in master(ge-0/0/2.0), dropping him
Aug 24 12:32:05 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 172.16.2.3+55187 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 172.16.2.3+55187 (proto) from AS 65001 found (eBGP connection attempt to an address that does not belong to the interface that peer connected to) in master(ge-0/0/2.0), dropping him
Aug 24 12:32:37 vMX-2 rpd[8797]: bgp_pp_recv:4991: NOTIFICATION sent to 172.16.2.3+50797 (proto): code 6 (Cease) subcode 5 (Connection Rejected), Reason: no group for 172.16.2.3+50797 (proto) from AS 65001 found (eBGP connection attempt to an address that does not belong to the interface that peer connected to) in master(ge-0/0/2.0), dropping him
lab@vMX-2>The BGP peer is still Idle. The neighbor still shows a PeerInterfaceError. We’re now getting some interesting log entries though.
The vMX-2 router is sending a Notification to vMX-3 rejecting the connection. What does “ eBGP connection attempt to an address that does not belong to the interface that peer connected to” mean?
It appears to be telling us that vMX-3 is trying to connect to vMX-2 on address 172.16.1.1. But that address doesn’t belong to interface ge-0/0/2 on vMX-2, it belongs to the loopback interface. It appears that vMX-2 currently thinks that vMX-3 is directly connected to ge-0/0/2 and should be trying to peer to the address of that interface.
Let’s see what happens when we configure the peer on vMX-2 for multihop.
VMX-2:
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
}
[edit]
lab@vMX-2#This is where something unexpected happened. The BGP session was actually established.
VMX-2:
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 1263 1256 0 0 9:25:54 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 5 3 0 2 37 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 102 102 0 0 45:23 Establ
inet.0: 0/3/3/0
lab@vMX-2>I was expecting the BGP session to remain active because we have not yet told vMX-2 to source traffic from its loopback interface, it should be sourcing the BGP connection from its outgoing interface of ge-0/0/2 with address 192.168.2.6. The vMX-3 router doesn’t have a peer configured for this address and should be rejecting the connection.
So what is happening here?
What I had not fully thought through is that vMX-3 is also trying to establish a connection to vMX-2 at the same time. It does have the correct peer configured (172.16.1.2) and is sourcing traffic from the correct address (172.16.2.3). Because the traffic was initiated to the 172.16.1.2 address on vMX-2, it completes the TCP connection using that address, allowing BGP to establish.
We can further prove this by changing vMX-3 to set the peer as passive, so it won’t try to initiate the connection.
vMX-3:
[edit]
lab@vMX-3# show protocols bgp
group my-external-group {
authentication-key "$9$35ou/AuRhr8X-O1X-VwaJ369"; ## SECRET-DATA
export exp-static;
peer-as 65002;
neighbor 192.168.2.2;
neighbor 172.16.1.2 {
multihop {
ttl 2;
}
local-address 172.16.2.3;
passive;
}
}
[edit]
lab@vMX-3#If we now clear the neighbor and leave it for a while, the BGP session will not establish.
vMX-2:
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
4 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 1281 1274 0 0 9:34:03 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 0 0 0 3 1:36 Connect
192.168.2.1 65001 121 121 0 0 53:32 Establ
inet.0: 3/3/3/0
lab@vMX-2>There is little to see in the messages log on vMX-2 and the traceoptions aren’t very helpful either.
Aug 24 13:09:30.800507 task_process_events_internal: connect ready for BGP_65001.172.16.2.3
Aug 24 13:09:30.800541 bgp_connect_complete: error connecting to 172.16.2.3 (External AS 65001): Socket is not connected
Aug 24 13:09:30.800547 BGP peer 172.16.2.3 (External AS 65001) CLOSE: close socket 122
Aug 24 13:09:30.800552 task_close: close socket 122 task BGP_65001.172.16.2.3
Aug 24 13:09:30.800556 task_reset_socket: task BGP_65001.172.16.2.3 socket 122
Aug 24 13:09:30.800587 bgp_check_and_release_local_port: peer 172.16.2.3 (External AS 65001) (instance master) local port 63367 released
Aug 24 13:09:30.800617 bgp_event: peer 172.16.2.3 (External AS 65001) old state Connect event OpenFail new state Idle
Aug 24 13:09:30.800847 bgp_event: peer 172.16.2.3 (External AS 65001) old state Idle event Start new state Connect
Aug 24 13:09:30.800867 bgp_connect_start: peer 172.16.2.3 (External AS 65001)
Aug 24 13:09:30.800871 bgp_event: peer 172.16.2.3 (External AS 65001) old state Connect event ConnectRetry new state Connect
Aug 24 13:09:30.800979 task_get_socket: domain AF_INET type SOCK_STREAM protocol 0 socket 122
Aug 24 13:09:30.800995 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option RecvBuffer(0) value 16384
Aug 24 13:09:30.801001 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option SendBuffer(1) value 16384
Aug 24 13:09:30.801005 task_set_socket: task BGP_65001.172.16.2.3 socket 122
Aug 24 13:09:30.801015 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option NonBlocking(8) value 1
Aug 24 13:09:30.801021 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option ReUsePort(38) value 1
Aug 24 13:09:30.801026 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option ReUseAddress(3) value 1
Aug 24 13:09:30.801030 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option TcpMd5Authentication(72)
Aug 24 13:09:30.801036 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option PathMTUDiscovery(26) value 0
Aug 24 13:09:30.801042 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option TOS(16) value 192
Aug 24 13:09:30.801047 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option TTL(15) value 2
Aug 24 13:09:30.801055 Eval BFD turn-on, currently off for peer 172.16.2.3 (External AS 65001)(fl:)instance master (kern-id 0)
Aug 24 13:09:30.801061 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option TcpAggressiveTransmission(59) value 0
Aug 24 13:09:30.801066 task_set_option_internal: task BGP_65001.172.16.2.3 socket 122 option EnableTcpNoDelay(63) value 0
Aug 24 13:09:30.801154 task_timer_reset: reset BGP_65001.172.16.2.3_Connect
Aug 24 13:09:30.801169 task_timer_set_oneshot_latest: timer BGP_65001.172.16.2.3_Connect interval set to 2
Aug 24 13:09:32.808757 bgp_connect_timeout: BGP_65001.172.16.2.3_Connect
Aug 24 13:09:32.808799 task_timer_reset: reset BGP_65001.172.16.2.3_ConnectWe can see that vMX-2 is trying to connect to 172.16.2.3 and is failing, but it isn’t entirely clear why.
The answer is a bit clearer on vMX-3 though when we look at its log file.
lab@vMX-3> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.2 65002 19 19 0 3 5:39 Active
192.168.2.2 65002 131 129 0 4 57:34 Establ
inet.0: 1/1/1/0
lab@vMX-3> show log messages | last 10
Aug 24 13:08:32 vMX-3 last message repeated 5 times
Aug 24 13:09:14 vMX-3 last message repeated 3 times
Aug 24 13:09:31 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.6:50761 unexpectedly has MD5 digest
Aug 24 13:09:37 vMX-3 last message repeated 2 times
Aug 24 13:10:29 vMX-3 last message repeated 6 times
Aug 24 13:10:46 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.6:51284 unexpectedly has MD5 digest
Aug 24 13:11:02 vMX-3 last message repeated 5 times
Aug 24 13:11:44 vMX-3 last message repeated 3 times
Aug 24 13:12:01 vMX-3 kernel: tcp_auth_ok: Packet from 192.168.2.6:49914 unexpectedly has MD5 digest
Aug 24 13:12:07 vMX-3 last message repeated 2 times
lab@vMX-3> The vMX-3 router also has the BGP session as Active. The log message is a little bit cryptic but shows some traffic being sourced from 192.168.2.6 which is the ge-0/0/2 interface of vMX-2. The MD5 digest is unexpected because vMX-3 expects it to be sourced from vMX-2’s loopback address, not the interface address.
We can confirm this from vMX-2 by looking at the monitor traffic interface command.
vMX-2:
lab@vMX-2> monitor traffic interface ge-0/0/2 no-resolve size 1500 matching "tcp port 179"
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is OFF.
Listening on ge-0/0/2, capture size 1500 bytes
13:15:29.234310 Out IP 192.168.2.6.55758 > 172.16.2.3.179: S 1852227832:1852227832(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2182370133 0,nop,nop,md5 9f0250bb30c5155ed7a8e83ae1f64689>
13:15:45.820340 Out IP 192.168.2.6.63345 > 172.16.2.3.179: S 3422264597:3422264597(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2182386719 0,nop,nop,md5 b02593d34854713fb9df40f1ada53bd6>
13:15:48.821534 Out IP 192.168.2.6.63345 > 172.16.2.3.179: S 3422264597:3422264597(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2182389720 0,nop,nop,md5 b02593d34854713fb9df40f1ada53bd6>
13:15:52.028911 Out IP 192.168.2.6.63345 > 172.16.2.3.179: S 3422264597:3422264597(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 2182392928 0,nop,nop,md5 b02593d34854713fb9df40f1ada53bd6>
^C
4 packets received by filter
0 packets dropped by kernel
lab@vMX-2>Traffic is being sent from 192.168.2.6 to 172.16.2.3 on port 179, which is the BGP port.
Let’s now configure vMX-2 to use the correct source address.
[edit]
lab@vMX-2# show protocols bgp group my-external-group
type external;
authentication-key "$9$7mdw2ZUH5Qn4aQn/CB17-V"; ## SECRET-DATA
peer-as 65001;
neighbor 192.168.2.1 {
traceoptions {
file bgp-ext-neighbor-direct size 1m files 10;
flag all;
}
}
neighbor 172.16.2.3 {
traceoptions {
file bgp-ext-neighbor-loopback size 1m files 10;
flag all;
}
multihop {
ttl 2;
}
local-address 172.16.1.2;
}
[edit]
lab@vMX-2#If we leave this for a moment BGP should establish.
vMX-2:
lab@vMX-2> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
7 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.1 65002 1302 1295 0 0 9:43:34 Establ
inet.0: 1/1/1/0
172.16.2.3 65001 3 3 0 0 26 Establ
inet.0: 3/3/3/0
192.168.2.1 65001 141 141 0 0 1:03:03 Establ
inet.0: 0/3/3/0
lab@vMX-2> As you can see, setting up eBGP multihop can create some non-obvious issues.
Triggering Prefix Limits
A BGP neighbor can be configured with a prefix limit. This prevents the neighbor from sending us too many routes.
Depending on whether you are looking at the router that has the prefix limit set, or the router that sent too many routes, you will see different results.
The router sending too many routes may just see the BGP session being reset. Depending on how the prefix limit is set on the other router, the BGP session may flap or just fail to establish.
This is covered in setting BGP prefix limits on Junos, so I won’t repeat it here.
Summary
In this post we have looked at the tools to troubleshoot BGP establishment and several common scenarios that prevent BGP from establishing.
Sometimes the cause of the issue is very obvious, and other times the error messages you see can be vague or unhelpful.
The best way to get better at troubleshooting BGP sessions is through experience. Setting it up in a lab environment and breaking it to see what the error messages look like.
For more general advice, look at troubleshooting BGP establishment.