In this post, we’re going to look at how to troubleshoot VRRP issues on a Junos device. If you haven’t already, you may like to review VRRP basics and how to configure VRRP on Junos.
What Does Good Look Like
Before we can start troubleshooting, we need to know what good looks like. For VRRP that means that all VRRP routers in a group agree on who the VRRP master is, and only the master router is sending VRRP advertisements. Hosts using the VIP as their gateway should be able to reach devices on remote networks too.
For Junos, the show vrrp output on the master should be similar to this:
lab@vMX-3> show vrrp
Interface State Group VR state VR Mode Timer Type Address
ge-0/0/2.0 up 50 master Active A 0.093 lcl 172.16.50.2
vip 172.16.50.1 For the backup devices, the show vrrp output should be similar to this:
lab@vMX-4> show vrrp
Interface State Group VR state VR Mode Timer Type Address
ge-0/0/2.0 up 50 backup Active D 0.872 lcl 172.16.50.3
vip 172.16.50.1
mas 172.16.50.2 A quick check to see whether VRRP packets are being sent and received is to use the show vrrp extensive command. In the below output, vMX-3 is the current master and vMX-4 is the backup. They have both recently had their vrrp statistics cleared.
VMX-3:
lab@vMX-3> show vrrp extensive
Interface: ge-0/0/2.0, Interface index :335, Groups: 1, Active :1
Interface VRRP PDU statistics
Advertisement sent :1299
Advertisement received :0
Packets received :0
No group match received :0
Interface VRRP PDU error statistics
Invalid IPAH next type received :0
Invalid VRRP TTL value received :0
Invalid VRRP version received :0
Invalid VRRP PDU type received :0
Invalid VRRP authentication type received:0
Invalid VRRP IP count received :0
Invalid VRRP checksum received :0
Physical interface: ge-0/0/2, Unit: 0, Address: 172.16.50.2/24
Index: 335, SNMP ifIndex: 548, VRRP-Traps: disabled, VRRP-Version: 2
Interface state: up, Group: 50, State: master, VRRP Mode: Active
Priority: 120, Advertisement interval: 0.100, Authentication type: md5
Advertisement threshold: 3, Computed send rate: 0
Preempt: yes, Accept-data mode: yes, VIP count: 1, VIP: 172.16.50.1
Advertisement Timer: 0.082s, Master router: 172.16.50.2
Virtual router uptime: 01:06:19, Master router uptime: 01:05:12
Virtual Mac: 00:00:5e:00:01:32
Preferred: yes
Tracking: enabled
Current priority: 120, Configured priority: 120
Priority hold time: disabled
Interface tracking: enabled, Interface count: 1
Tracked interface: ge-0/0/1.0
Interface state: up Speed: 1g
Incurred priority cost: 0
Threshold Priority cost Active
down 30
Route tracking: enabled, Route count: 1
Route VRF name Route state Priority cost
192.168.20.0/24 default up 30
Group VRRP PDU statistics
Advertisement sent :1299
Advertisement received :0
Group VRRP PDU error statistics
Bad authentication Type received :0
Bad password received :0
Bad MD5 digest received :0
Bad advertisement timer received :0
Bad VIP count received :0
Bad VIPADDR received :0
Group state transition statistics
Idle to master transitions :0
Idle to backup transitions :0
Backup to master transitions :0
Master to backup transitions :0 VMX-4:
lab@vMX-4> show vrrp extensive
Interface: ge-0/0/2.0, Interface index :335, Groups: 1, Active :1
Interface VRRP PDU statistics
Advertisement sent :0
Advertisement received :1295
Packets received :1295
No group match received :0
Interface VRRP PDU error statistics
Invalid IPAH next type received :0
Invalid VRRP TTL value received :0
Invalid VRRP version received :0
Invalid VRRP PDU type received :0
Invalid VRRP authentication type received:0
Invalid VRRP IP count received :0
Invalid VRRP checksum received :0
Physical interface: ge-0/0/2, Unit: 0, Address: 172.16.50.3/24
Index: 335, SNMP ifIndex: 548, VRRP-Traps: disabled, VRRP-Version: 2
Interface state: up, Group: 50, State: backup, VRRP Mode: Active
Priority: 100, Advertisement interval: 0.100, Authentication type: md5
Advertisement threshold: 3, Computed send rate: 0
Preempt: yes, Accept-data mode: yes, VIP count: 1, VIP: 172.16.50.1
Dead timer: 0.559s, Master priority: 120, Master router: 172.16.50.2
Virtual router uptime: 06:16:30
Preferred: yes
Tracking: disabled
Group VRRP PDU statistics
Advertisement sent :0
Advertisement received :1295
Group VRRP PDU error statistics
Bad authentication Type received :0
Bad password received :0
Bad MD5 digest received :0
Bad advertisement timer received :0
Bad VIP count received :0
Bad VIPADDR received :0
Group state transition statistics
Idle to master transitions :0
Idle to backup transitions :0
Backup to master transitions :0
Master to backup transitions :0 Again – it is only the VRRP master that sends the VRRP advertisements. So in this case, because we have recently cleared the counters, it is normal to see that vMX-3 is sending advertisements but hasn’t received any. Likewise, it is normal to see vMX4 only receiving advertisements and not sending any.
As the mastership changes, then that behavior would swap. The thing is to run the command several times and watch which counters are incrementing. You could consider using the Junos refresh command.
lab@vMX-3> show vrrp extensive | match Advertisement | refresh 5
---(refreshed at 2023-12-09 10:54:43 UTC)---
Interface VRRP PDU statistics
Advertisement sent :4535
Advertisement received :0
Priority: 120, Advertisement interval: 0.100, Authentication type: md5
Advertisement threshold: 3, Computed send rate: 0
Advertisement Timer: 0.086s, Master router: 172.16.50.2
Group VRRP PDU statistics
Advertisement sent :4535
Advertisement received :0
Bad advertisement timer received :0
---(refreshed at 2023-12-09 10:54:48 UTC)---
Interface VRRP PDU statistics
Advertisement sent :4590
Advertisement received :0
Priority: 120, Advertisement interval: 0.100, Authentication type: md5
Advertisement threshold: 3, Computed send rate: 0
Advertisement Timer: 0.051s, Master router: 172.16.50.2
Group VRRP PDU statistics
Advertisement sent :4590
Advertisement received :0
Bad advertisement timer received :0
---(*more 100%)---[abort]What Does Bad Look Like
For VRRP, bad looks like the VRRP routers disagreeing about who the master is. Both will show their VR state as master and if you look at a packet capture, both will be advertising VRRP packets.
These problems generally fall into one of two categories:
- VRRP packets not being received
- VRRP packets are being received, but ignored or discarded for some reason.
We’ll look at these two categories in more detail below, but first, what tools do we have to diagnose VRRP problems?
- The various show vrrp and related commands
- Log files
- Monitor traffic interface command to look at traffic
- Turning on VRRP traceoptions.
We’ll look at these tools as we progress through the troubleshooting steps below.
VRRP Packets Not Being Received
If a VRRP router is not receiving VRRP packets from another VRRP router, it will have to assume it is the master and start acting in that role.
Packets not being received usually come down to one of two issues – a network issue or a filtering issue.
The first thing to consider is whether this is VRRP behaving as expected during a network event. Is there a failure in the network path between the two VRRP routers?
Consider the following diagram, where VRRP traffic must go from Router 1, through two switches, to Router 2.
- Are the right vlans presented to the right interfaces or are the interfaces in the right vlans?
- Has the inter-switch link failed?
- Are the right vlans being carried over the inter-switch link?
A quick test is to try pinging from the Router 1 interface to the Router 2 interface. If this ping succeeds, then great, it appears the network path is in place. If the ping fails then it might be a network path issue, or it could be a misleading result. It depends on whether ping is normally allowed between the Router 1 and 2 devices.
This brings us to the second possible issue for packets not being received – packet filtering. On Junos it is very common to deploy a lo0 filter to protect the routing engine by limiting what traffic can be sent to the router itself. If this is a new VRRP deployment, and a lo0 filter is already in place, it may need editing to allow VRRP traffic. Even if VRRP has been working for a while, it is worth checking that no one has made a recent change to the filter that may affect VRRP traffic.
Another thing to be aware of is the interaction of lo0 filters and routing instances or logical systems. Make sure you understand which filter, if any, applies to the VRRP traffic you are looking at.
VRRP Packets Received But Ignored
The next set of issues we are going to look at is where VRRP advertisements are received by both VRRP routers, but they are ignored for one reason or another.
The best approach is to start with the simple steps first. These are checking the show vrrp extensive output and looking at the messages log for errors. A good method is to use show log messages | last 10 to limit the amount of output you get from the messages file.
From the show vrrp extensive output, we have the following types of errors listed:
Group VRRP PDU error statistics
Bad authentication Type received :0
Bad password received :0
Bad MD5 digest received :0
Bad advertisement timer received :0
Bad VIP count received :0
Bad VIPADDR received :0 If any of the above are incrementing, then you should correct the cause.
Bad authentication Type received
This counter increments when the authentication type – none, simple, or md5 is not the same across all VRRP routers. It generates log entries similar to the following:
Dec 9 11:05:35 vMX-4 vrrpd[8676]: VRRPD_AUTH_INFO_INVALID: VRRP ad packet from 172.16.50.2 received by interface ge-0/0/2.0 for group 50 was invalid: authentication-type-mismatchBad password received
This counter increments when the authentication type is set to simple and the authentication-key is set to different values. It generates log entries similar to the following:
Dec 9 11:10:03 vMX-4 vrrpd[8676]: VRRPD_AUTH_INFO_INVALID: VRRP ad packet from 172.16.50.2 received by interface ge-0/0/2.0 for group 50 was invalid: authentication-key-mismatchBad MD5 digest received
This counter increments when the authentication type is set to md5, and the authentication-key is set to different values. Oddly this did not seem to generate a specific log entry.
Bad advertisement timer received
This counter increments when the advertisement intervals differ between the VRRP routers. It generates log entries similar to the following:
Dec 9 11:21:53 vMX-4 vrrpd[8676]: VRRPD_ADVERT_TIME_MISMATCH: Packet received by ge-0/0/2.0 for VRRP group 50 had advertisement time 1000 ms instead of required 100 msBad VIP count received
This counter increments when the number of virtual addresses configured for the same vrrp group differ between the VRRP routers. For example one is configured with two VIP addresses and the other only one. It generates log entries similar to the following:
Dec 9 11:26:34 vMX-3 vrrpd[8678]: VRRPD_VIP_COUNT_MISMATCH: Packet received by ge-0/0/2.0 for VRRP group 50 had 2 virtual IP addresses instead of required 1Bad VIPADDR received
This counter increments when the configured virtual address differs for the configured VRRP group. It generates log entries similar to the following:
Dec 9 11:32:01 vMX-4 vrrpd[8676]: VRRPD_MISSING_VIP: Packet received by ge-0/0/2.0 for VRRP group 50 did not include required virtual IP address 172.16.50.1Configuring Traceoptions
Traceoptions can be a double-edged sword. While it can provide more detail of what is happening, the output can be quite large and overwhelming to try and search through. This is especially true if you have a lot of VRRP groups configured.
To configure traceoptions for VRRP, it is configured under the protocols stanza, as shown below.
lab@vMX-4> show configuration protocols vrrp
traceoptions {
file vrrp-trace size 1m files 10;
flag all;
}
lab@vMX-4> show configuration protocols vrrp | display set
set protocols vrrp traceoptions file vrrp-trace
set protocols vrrp traceoptions file size 1m
set protocols vrrp traceoptions file files 10
set protocols vrrp traceoptions flag allThe possible types of flags you can set are:
lab@vMX-4# set protocols vrrp traceoptions flag ?
Possible completions:
all Trace all events
database Trace database
general Trace general events
interfaces Trace interface messages
normal Trace normal events
packets Trace packets
ppm VRRP PPM related messages
state Trace state transitions
timer Trace timer eventsOutside of a lab environment, setting flag all may generate quite a bit of logging. You may need to experiment to see which flags are most useful for you and your environment.
Monitor Traffic Interface Command
Another tool that can be helpful for a quick check is the monitor traffic interface command, which acts like a tcpdump for packets destined to the router itself. An example is shown below. Note the traffic is destined to the VRRP multicast address of 224.0.0.18 and not the router’s interface address which is 172.16.50.3.
lab@vMX-4> monitor traffic interface ge-0/0/2
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ge-0/0/2, capture size 96 bytes
Reverse lookup for 172.16.50.2 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.
12:25:26.818469 In IP 172.16.50.2 > 224.0.0.18: AH(spi=2880154539,seq=0x2f9c223a): VRRPv2-advertisement 20: vrid=50 prio=120 authtype=ah intvl=2
12:25:29.004656 In IP 172.16.50.2 > 224.0.0.18: AH(spi=2880154539,seq=0x2f9c223a): VRRPv2-advertisement 20: vrid=50 prio=120 authtype=ah intvl=2
12:25:30.545318 In IP 172.16.50.2 > 224.0.0.18: AH(spi=2880154539,seq=0x2f9c223a): VRRPv2-advertisement 20: vrid=50 prio=120 authtype=ah intvl=2
12:25:32.220893 In IP 172.16.50.2 > 224.0.0.18: AH(spi=2880154539,seq=0x2f9c223a): VRRPv2-advertisement 20: vrid=50 prio=120 authtype=ah intvl=2
^C
4 packets received by filter
0 packets dropped by kernelThis can provide quick validation of what is being seen on the interface. Something to be aware of however is the ppmd process. On some devices, mostly the MX series, the periodic packet manager daemon can “distribute” the work of generating and responding to periodic packets to the Packet Forwarding Engine (PFE). In this case, because the monitor traffic command works on the RE and the PFE is handling the advertisements, they may not appear in this command.
Use the show ppm adjacencies detail command to see whether “Distributed” is set to True or False.
VRRP Mastership Switching When There Are No Network Events
Let’s assume a situation where VRRP is configured between two routers and pre-emption is enabled. The VRRP mastership is switching between the two routers. There are no network events to cause the loss of VRRP packets or cause tracking to reduce the priority value.
In this case, I would look at the value of the advertisement interval and the threshold that has been set. If the timers are too aggressive and many VRRP groups are configured then the VRRP router may not be able to keep up with sending or receiving all of the VRRP advertisements. This is a similar problem to setting BFD settings too low.
The easiest way to see if this is the issue is to increase the advertisement interval. Initially change it back to the default of 1 second and see whether that makes a difference to the behavior you see.
Weird Issues
At the start, I said that VRRP issues tend to fall into two categories – not receiving VRRP advertisements at all, or receiving them but ignoring them.
There is a third category that I like to call weirdness. This is where everything looks like VRRP should be working happily. There is a network path between the devices, they seem to be receiving packets OK, and all the settings are correct. But still, both VRRP routers think they should be the master.
I saw this once when connecting two older SRXs together to test something in a lab. Both were running what is now an old version of Junos. I spent time trying to figure out why it was not working as I expected and even had someone else check my thinking as to what could be wrong.
In the end, restarting both the SRX devices resolved the issue. In hindsight, simply restarting the vrrp with the restart vrrp gracefully command might have been a quicker fix.
There is no good reason why restarting the SRX or the VRRP daemon should have fixed the issue. Sometimes things just behave weirdly.
In production environments, it can be hard to test for weirdness. It is also not the first issue I would start thinking about. That said, if everything looks right and you’ve had someone else check it and come to the same conclusion that it should be working, then it might be time to think about restarting the VRRP process.
Summary
Overall, VRRP is a pretty simple protocol, and there aren’t too many places where things can go wrong.
If you’re having VRRP issues, start by confirming that VRRP packets are getting through between the VRRP routers. Check there is a usable network path and that VRRP isn’t being filtered somewhere.
After confirming VRRP packets are being received, it is usually a configuration issue, so check the logs and show vrrp extensive output error counters.
If VRRP mastership is changing for no apparent reason, check the timers and thresholds and consider increasing these values to see if that changes the behavior.
Happy troubleshooting.