Exploring MTU

Network Engineering

In this post, we are going to look at what MTU is. We’re also going to see what it looks like on the network with practical examples.

What is MTU

MTU stands for Maximum Transmission Unit and represents the largest frame or packet that can be sent over a link. We are typically concerned with MTU at layer 2 and layer 3 of the OSI network model.

I’m mainly going to be talking about Ethernet for Layer 2 and IP(IPv4) for Layer 3.

Since everyone loves a bad analogy, try to think of it this way.

Ethernet frames are like a delivery truck that can only take one package at a time, and only up to a certain size. If the truck is too big to fit down a certain road (network link), the truck cannot be broken into smaller pieces and the package is not delivered. In other words, you cannot fragment an Ethernet frame. If it meets a link with a too small MTU, it will be dropped.

IP Packets are like the package that goes into the delivery truck. If the package is too big to fit into the truck either the sender (end host) or the shipping company (e.g. a router) can break the large package into smaller packages and send it in multiple trucks. In technical terms, this is IP Fragmentation.

Continuing with IP fragmentation for a moment, some receivers do not like packages that have been broken up and will refuse to receive them. The sender can also say the package is not allowed to be broken down into smaller ones. Again, in technical terms, some networks will filter out and block IP fragments. The sender can also set the Don’t Fragment bit to disallow fragmentation as the IP packet travels the network.

This makes it sound like Ethernet frames are all the same size, which they are not. They will be made as big as necessary, up to the MTU size, to accommodate the payload. If you look at a network packet capture, you will see Ethernet frames of varying sizes.

Why Do We Care About MTU Size?

One reason we care is efficiency. If there is a link with a small MTU in the middle of the network that is causing IP fragmentation, then it results in more packets being sent over the network and increases the work to fragment and re-assemble the packets.

Using a larger MTU allows for the same amount of data to be transported in fewer packets (or frames), which reduces the network overhead.

Larger MTUs will take longer to transmit though, and may delay “smaller” packets.

Normal MTU sizing

The standard MTU for most Ethernet-based networks is a layer 2 frame of 1514 bytes. This is made up of a payload of 1500 bytes plus 14 bytes of layer 2 overhead. In most cases, the Ethernet trailer, or Frame Check Sequence, of 4 bytes is not counted. Neither are things like the start of frame delimiter.

In switched networks, trunk ports have their MTU increased by an additional 4 bytes to account for the VLAN tag. Giving an MTU of 1518 bytes for a tagged or trunk interface. This increase is done automatically when the port is configured as a trunk or tagged port.

The layer 3 (IP) MTU is based on the layer 2 MTU and is sized to fit in the payload portion of the layer 2 frame. So in most cases, the layer 3 MTU is 1500 bytes, including the IP header.

What About Jumbo Frames?

Most network vendors support “jumbo frames” around the 9Kb range. When configuring these you need to be aware of how your vendor is counting the MTU – whether the MTU you set includes the layer 2 overhead or not.

You also need to be aware of whether you are setting the layer 2 or layer 3 MTU, as these can sometimes be specified separately.

How is MTU displayed?

MTU is displayed differently depending on which vendor, device family, or sometimes even which configuration you have applied.

In some cases, the displayed MTU will count the layer 2 overhead, in other cases it won’t. Juniper typically counts the layer 2 overhead, whereas Cisco IOS doesn’t. If you’re unsure you should check your vendor documentation.

Let’s look at a vQFX. What happens when we take an interface set for family ethernet-switching and change it from an access port to a trunk port?

{master:0}
lab@vqfx-1> show configuration interfaces xe-0/0/0  
unit 0 {
    family ethernet-switching {
        interface-mode access;
        vlan {
            members 20;
        }
    }
}

{master:0}
lab@vqfx-1> show interfaces xe-0/0/0 | match mtu 
  Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 10Gbps,
    Protocol eth-switch, MTU: 1514



{master:0}
lab@vqfx-1> show configuration interfaces xe-0/0/0    
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members 20;
        }
    }
}

{master:0}
lab@vqfx-1> show interfaces xe-0/0/0 | match mtu      
  Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 10Gbps,
    Protocol eth-switch, MTU: 1514

The MTU is 1514 as an access port, and confusingly also 1514 when a trunk port. This is surprising as I would expect it to be 1518 due to the normal 1514 MTU plus 4 bytes for the VLAN tag. This behavior seems to be normal on the EX and QFX switches when working with family ethernet-switching.

What happens if we try a layer 3 interface on the vQFX. Note when we set the interface to VLAN tagging, we need to add a VLAN id under the unit as well.

{master:0}
lab@vqfx-1> show configuration interfaces xe-0/0/0  
unit 0 {
    family inet;
}

{master:0}
lab@vqfx-1> show interfaces xe-0/0/0 | match mtu 
  Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 10Gbps,
    Protocol inet, MTU: 1500



{master:0}
lab@vqfx-1> show configuration interfaces xe-0/0/0    
vlan-tagging;
unit 0 {
    vlan-id 10;
    family inet;
}

{master:0}
lab@vqfx-1> show interfaces xe-0/0/0 | match mtu      
  Link-level type: Ethernet, MTU: 1518, LAN-PHY mode, Speed: 10Gbps,
    Protocol inet, MTU: 1500

When using family inet it behaves as expected. We have an Ethernet MTU of 1514 without VLAN tagging, and 1518 when VLAN tagging is turned on. The IP (inet) MTU remains the same at 1500.

What Reduces The Usable MTU?

The thing that usually reduces the usable MTU is additional encapsulation overhead. This is where a normal IP packet is further encapsulated, or has additional headers applied to it, before being sent out an interface. These additional headers reduce the amount of space available for the “original” IP packet.

In enterprise and home networks, this might look like some form of tunneling or VPN solution.

In service provider networks there might be several MPLS labels that need to be added, or possibly QinQ tagging.

In data center environments it could be additional MPLS or VXLAN encapsulation.

To work around this, the Layer 2 MTU is usually increased where possible in the “core” side of a network to accommodate any additional overheads or encapsulations. I say where possible because not all devices support jumbo frames.

What Should I Set The MTU To?

The short answer is, it depends. In general, it is best to leave MTU size alone unless you know what you are doing or have a clear reason for changing it.

For most home or office style networks it can be left alone.

Telco and Service Provider style networks are likely to increase MTU to as large as possible in their core. They may offer jumbo frame support to their clients as well. They will need to consider the largest MTU they can set, and what the largest usable MTU will be that they can offer to their customers.

Data center networks may increase MTU to account for either additional encapsulation or to improve efficiency for storage traffic if that is being carried over the network. For example, iSCSI traffic may see a performance improvement when jumbo frames are in use.

Example Network Setup

The network setup used for the following examples is this:

This was built on Eve-NG using vQFX, vMX, and Ubuntu Linux images.

At a logical level, Linux-1 and Linux-2 are on the same network. Linux-3 is on another network several hops away.

At a “physical” layer (since this was all virtualized), these are all connected using a vQFX switch, and port mirroring is enabled to send mirrored traffic to Linux-Capture.

Example 1 – Ping A Host On The Same Subnet

Now that we know a bit more about MTU, it’s time to show a few examples. Starting with Linux-1 pinging Linux-2. The IP MTU is the default 1500 bytes.

First up we’ll run a ping and run a packet capture to see what it looks like “on the wire”.

eve@linux-1:~$ ping 192.168.20.6 -c 1
PING 192.168.20.6 (192.168.20.6) 56(84) bytes of data.
64 bytes from 192.168.20.6: icmp_seq=1 ttl=64 time=114 ms

--- 192.168.20.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 113.757/113.757/113.757/0.000 ms

From our capturing host we see the following:

03:51:26.343140 IP (tos 0x0, ttl 64, id 2033, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.20.5 > 192.168.20.6: ICMP echo request, id 5, seq 1, length 64
03:51:26.445063 IP (tos 0x0, ttl 64, id 25932, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.6 > 192.168.20.5: ICMP echo reply, id 5, seq 1, length 64

Take note that there is only one packet from 192.168.20.5 to 192.168.20.6 and one in the return direction.

So what happens if we use a ping of size 2000 which is larger than our MTU of 1500 – clearly that won’t work will it?

eve@linux-1:~$ ping 192.168.20.6 -c 1 -s 2000
PING 192.168.20.6 (192.168.20.6) 2000(2028) bytes of data.
2008 bytes from 192.168.20.6: icmp_seq=1 ttl=64 time=24.3 ms

--- 192.168.20.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 24.292/24.292/24.292/0.000 ms

So what happened here? Why did this work? The answer is IP fragmentation. Since the ping packet was too big to fit within a single IP packet, it was broken into two fragments which can be seen in the packet capture below. You’ll notice that even though we only sent one ping, the packet capture shows two packets from 192.168.20.5 to 192.168.20.6 and two in return. Linux-1 fragmented the packet, as it knew its local interface only had a layer 3 MTU of 1500.

03:52:16.373951 IP (tos 0x0, ttl 64, id 11511, offset 0, flags [+], proto ICMP (1), length 1500)
    192.168.20.5 > 192.168.20.6: ICMP echo request, id 6, seq 1, length 1480
03:52:16.374456 IP (tos 0x0, ttl 64, id 11511, offset 1480, flags [none], proto ICMP (1), length 548)
    192.168.20.5 > 192.168.20.6: icmp
03:52:16.386059 IP (tos 0x0, ttl 64, id 26381, offset 0, flags [+], proto ICMP (1), length 1500)
    192.168.20.6 > 192.168.20.5: ICMP echo reply, id 6, seq 1, length 1480
03:52:16.389445 IP (tos 0x0, ttl 64, id 26381, offset 1480, flags [none], proto ICMP (1), length 548)
    192.168.20.6 > 192.168.20.5: icmp

One useful method for trying to find MTU issues can be to use ping and set the don’t fragment flag so that devices can’t fragment the ping packet. On Linux we can use ‘-M do’ to turn on Path MTU Discovery (PMTUD) and the ‘do’ says to not allow fragmentation, even on the local interface.

Now let’s try this again with a ping of size 1500, this has to work right?

eve@linux-1:~$ ping 192.168.20.6 -c 1 -M do -s 1500
PING 192.168.20.6 (192.168.20.6) 1500(1528) bytes of data.
ping: local error: message too long, mtu=1500

--- 192.168.20.6 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

So why didn’t this work? We didn’t capture any packets because we didn’t send anything, the ping failed on the local host, but why?

The answer is in the first line of output, which shows ‘1500(1528)’ and requires us to understand the tool that we’re using. The ‘-s’ parameter on the Linux ping command specifies the number of data bytes to be sent, and not the overall size of the packet. We need to account for the 8 bytes of ICMP header and 20 bytes of IP header. We need to set the size to 1472. To prove this, let’s try a ping first with 1473, then another set to 1472.

Note that the ping command on other platforms might behave differently, sometimes the size option for the ping command will be to set the overall packet size including ICMP and IP headers.

eve@linux-1:~$ ping 192.168.20.6 -c 1 -M do -s 1473
PING 192.168.20.6 (192.168.20.6) 1473(1501) bytes of data.
ping: local error: message too long, mtu=1500

--- 192.168.20.6 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

eve@linux-1:~$ ping 192.168.20.6 -c 1 -M do -s 1472
PING 192.168.20.6 (192.168.20.6) 1472(1500) bytes of data.
1480 bytes from 192.168.20.6: icmp_seq=1 ttl=64 time=11.0 ms

--- 192.168.20.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 10.986/10.986/10.986/0.000 ms

The above output matches our expectations, showing the largest ICMP data payload we could get through was 1472, giving us a single 1500byte overall packet.

This is supported by the packet capture.

04:11:27.673961 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.20.5 > 192.168.20.6: ICMP echo request, id 9, seq 1, length 1480
04:11:27.674164 IP (tos 0x0, ttl 64, id 32977, offset 0, flags [none], proto ICMP (1), length 1500)
    192.168.20.6 > 192.168.20.5: ICMP echo reply, id 9, seq 1, length 1480

Example 2 – Ping A Host On A Remote Subnet

So what happens if we make things a bit more complicated by pinging across the network, rather than a host on the same local network?

Let’s try pinging from Linux-1 to Linux-3, requiring traffic to pass over R1, R2 and R3. I’ll show the output from Linux-1 and two packet captures. One from VLAN 20, and the other from VLAN 50.

eve@linux-1:~$ ping 172.16.50.10 -c 1 -M do -s 1472
PING 172.16.50.10 (172.16.50.10) 1472(1500) bytes of data.
1480 bytes from 172.16.50.10: icmp_seq=1 ttl=61 time=365 ms

--- 172.16.50.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 365.431/365.431/365.431/0.000 ms

The above output looks as expected, our ping worked fine. The below packet capture shows VLAN 20 that Linux-1 is connected to.

03:40:50.483565 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 1, seq 1, length 1480
03:40:50.703999 IP (tos 0x0, ttl 61, id 31440, offset 0, flags [none], proto ICMP (1), length 1500)
    172.16.50.10 > 192.168.20.5: ICMP echo reply, id 1, seq 1, length 1480

The below capture is from VLAN 50 which Linux-3 is connected to.

03:40:50.667160 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 1, seq 1, length 1480
03:40:50.681565 IP (tos 0x0, ttl 64, id 31440, offset 0, flags [none], proto ICMP (1), length 1500)
    172.16.50.10 > 192.168.20.5: ICMP echo reply, id 1, seq 1, length 1480

The two captures look the same, which is what I’d expect at this stage.

The next step is to lower the MTU on the interfaces between R1 and R2 to see what happens when we make the middle of the path smaller than the MTU of the end hosts.

eve@linux-1:~$ ping 172.16.50.10 -c 1 -M want -s 1472
PING 172.16.50.10 (172.16.50.10) 1472(1500) bytes of data.
From 192.168.20.1 icmp_seq=1 Frag needed and DF set (mtu = 986)

--- 172.16.50.10 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

We can see here that R1 responded to say the packet needed to be fragmented, but we’ve set the don’t fragment bit so R1 could not fragment the packet.

This is also seen on the packet capture for VLAN 20. I captured this separately and looked at it in Wireshark. The ICMP message is a Type 3 (Unreachable) Code 4 (Fragmentation Needed) message.

04:57:48.068659 IP (tos 0x0, ttl 64, id 48736, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 23, seq 1, length 1480
04:57:48.371243 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.20.1 > 192.168.20.5: ICMP 172.16.50.10 unreachable - need to frag (mtu 986), length 36
	IP (tos 0x0, ttl 64, id 48736, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 23, seq 1, length 1480

The packet capture for VLAN 50 didn’t show any traffic, as no ping packets made it that far, they effectively stopped at R1.

Let’s repeat this, but without using the -M command to set the DF flag.

Now – at this stage, we need to take a short diversion, as the next part didn’t work as I anticipated. Once the Linux-1 device received the ICMP Unreachable, Fragmentation Needed, it “remembered” that for future connections. To “forget” this, I ran ‘sudo netplan apply’ to re-apply the network configuration.

The first ping I then ran without the -M switch failed, receiving the same ICMP Unreachable message again. It appears that it was setting the DF bit anyway, which I had not previously noticed. Repeating this command a second time worked.

You can see this below.

eve@linux-1:~$ ping 172.16.50.10 -c 1 -s 1472
PING 172.16.50.10 (172.16.50.10) 1472(1500) bytes of data.
From 192.168.20.1 icmp_seq=1 Frag needed and DF set (mtu = 986)

--- 172.16.50.10 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

eve@linux-1:~$ ping 172.16.50.10 -c 1 -s 1472
PING 172.16.50.10 (172.16.50.10) 1472(1500) bytes of data.
1480 bytes from 172.16.50.10: icmp_seq=1 ttl=61 time=192 ms

--- 172.16.50.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 191.872/191.872/191.872/0.000 ms

The packet capture on VLAN 20 shows:

05:15:48.639090 IP (tos 0x0, ttl 64, id 47887, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 27, seq 1, length 1480
05:15:48.663714 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.20.1 > 192.168.20.5: ICMP 172.16.50.10 unreachable - need to frag (mtu 986), length 36
	IP (tos 0x0, ttl 64, id 47887, offset 0, flags [DF], proto ICMP (1), length 1500)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 27, seq 1, length 1480

and then

05:16:03.512352 IP (tos 0x0, ttl 64, id 48389, offset 0, flags [+], proto ICMP (1), length 980)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 28, seq 1, length 960
05:16:03.517890 IP (tos 0x0, ttl 64, id 48389, offset 960, flags [none], proto ICMP (1), length 540)
    192.168.20.5 > 172.16.50.10: icmp
05:16:03.695956 IP (tos 0x0, ttl 61, id 52105, offset 0, flags [+], proto ICMP (1), length 980)
    172.16.50.10 > 192.168.20.5: ICMP echo reply, id 28, seq 1, length 960
05:16:03.696774 IP (tos 0x0, ttl 61, id 52105, offset 960, flags [none], proto ICMP (1), length 540)
    172.16.50.10 > 192.168.20.5: icmp

The packet capture for VLAN 50 showed nothing for the first ping packet, and then this for the second:

05:16:03.551068 IP (tos 0x0, ttl 61, id 48389, offset 0, flags [+], proto ICMP (1), length 980)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 28, seq 1, length 960
05:16:03.651001 IP (tos 0x0, ttl 61, id 48389, offset 960, flags [none], proto ICMP (1), length 540)
    192.168.20.5 > 172.16.50.10: icmp
05:16:03.664095 IP (tos 0x0, ttl 64, id 52105, offset 0, flags [none], proto ICMP (1), length 1500)
    172.16.50.10 > 192.168.20.5: ICMP echo reply, id 28, seq 1, length 1480

Notice anything weird here? Linux-3 (172.16.50.10) is receiving a fragmented ping request, and responding with a single ICMP echo reply with an overall size of 1500 bytes.

The keen-eyed amongst you will have noticed the capture for VLAN 20 shows Linux-1 receiving a fragmented echo reply.

If we repeat this process and do a capture on VLAN 31 between R1 and R2 we see this:

05:26:23.558244 IP (tos 0x0, ttl 63, id 22949, offset 0, flags [+], proto ICMP (1), length 980)
    192.168.20.5 > 172.16.50.10: ICMP echo request, id 30, seq 2, length 960
05:26:23.560378 IP (tos 0x0, ttl 63, id 22949, offset 960, flags [none], proto ICMP (1), length 540)
    192.168.20.5 > 172.16.50.10: icmp
05:26:23.589040 IP (tos 0x0, ttl 62, id 36333, offset 0, flags [+], proto ICMP (1), length 980)
    172.16.50.10 > 192.168.20.5: ICMP echo reply, id 30, seq 2, length 960
05:26:23.592898 IP (tos 0x0, ttl 62, id 36333, offset 960, flags [none], proto ICMP (1), length 540)
    172.16.50.10 > 192.168.20.5: icmp

This shows both a fragmented echo request and echo reply.

At first, this seems like a bit of a head-scratcher – what is going on here? Based on what we’re seeing I interpret this as follows:

  • Linux-1 sends a ping with an overall size of 1500 bytes, crucially the DF bit is set, even though we didn’t use the -M flag.
  • R1 can’t send this over the link to R2, as the MTU is too small. R1 can’t fragment the packet either as the DF bit is set. R1 sends ICMP Unreachable, Fragmentation Needed back to Linux-1
  • Linux-1 notes this first echo request as failing and sends a second request and fragments it before sending it to R1. It is now small enough to go over the link from R1 to R2.
  • Linux-3 receives the fragmented echo request, and since the response will fit within its MTU of 1500, Linux-3 sends a single echo reply packet.
  • R2 receives the single 1500 byte echo reply destined for Linux-1. This is too big to send as is to R1. Because Linux-3 did NOT set the DF bit, R2 can fragment the packet, which it does and sends along.
  • Linux-1 successfully receives the fragmented echo reply, completing the overall process.

This behavior was not what I expected. I had assumed that Linux-1 would send the echo request without the DF bit set, and R1 would simply fragment the packet on its way to R2. In return, I assumed Linux-3 would fragment the response, but it didn’t.

Example 3 – Using Tracepath To Discover MTU

Up until now, we’ve been using ping to try to find the size of the MTU for the path between Linux-1 and Linux-3. But is there a better method? Well, we could try a tool like tracepath, let’s look at that next. Again we’ll show the output from Linux-1 and packet captures for vlan 20 and 50.

I’ve used the -n switch so tracepath doesn’t try to resolve addresses to hostnames.

eve@linux-1:~$ tracepath -n 172.16.50.10
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.20.1                                        120.961ms 
 1:  192.168.20.1                                         17.116ms 
 2:  192.168.20.1                                         29.838ms pmtu 986
 2:  10.100.1.1                                          316.893ms 
 3:  10.100.2.2                                          516.290ms 
 4:  172.16.50.10                                        632.750ms reached
     Resume: pmtu 986 hops 4 back 4

The path MTU of 986 makes sense. I set the interface between R1 and R2 at 1000, so taking off 14 bytes of layer 2 overhead gives 986.

The packet capture for VLAN 20 is below. You can see that the source and destination of the packets are the same, the DF bit is set, but the ttl is being incremented up much like a traceroute command would. The tracepath command appears to be looking out for both time exceeded and ICMP Unreachable, Fragmentation Needed messages.

06:02:28.139991 IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
    192.168.20.5.54354 > 172.16.50.10.44444: UDP, length 1472
06:02:28.145604 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.20.1 > 192.168.20.5: ICMP time exceeded in-transit, length 36
	IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
    192.168.20.5.54354 > 172.16.50.10.44444: UDP, length 1472
06:02:28.158345 IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
    192.168.20.5.54354 > 172.16.50.10.44445: UDP, length 1472
06:02:28.162536 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.20.1 > 192.168.20.5: ICMP time exceeded in-transit, length 36
	IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
    192.168.20.5.54354 > 172.16.50.10.44445: UDP, length 1472
06:02:28.174143 IP (tos 0x0, ttl 2, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
    192.168.20.5.54354 > 172.16.50.10.44446: UDP, length 1472
06:02:28.192335 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.20.1 > 192.168.20.5: ICMP 172.16.50.10 unreachable - need to frag (mtu 986), length 36
	IP (tos 0x0, ttl 2, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
    192.168.20.5.54354 > 172.16.50.10.44446: UDP, length 1472
06:02:28.201059 IP (tos 0x0, ttl 2, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44447: UDP, length 958
06:02:28.509164 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    10.100.1.1 > 192.168.20.5: ICMP time exceeded in-transit, length 36
	IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44447: UDP, length 958
06:02:28.516712 IP (tos 0x0, ttl 3, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44448: UDP, length 958
06:02:29.025635 IP (tos 0x0, ttl 253, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    10.100.2.2 > 192.168.20.5: ICMP time exceeded in-transit, length 36
	IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44448: UDP, length 958
06:02:29.035718 IP (tos 0x0, ttl 4, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44449: UDP, length 958
06:02:29.658494 IP (tos 0xc0, ttl 61, id 16859, offset 0, flags [none], proto ICMP (1), length 576)
    172.16.50.10 > 192.168.20.5: ICMP 172.16.50.10 udp port 44449 unreachable, length 556
	IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44449: UDP, length 958

The capture for VLAN 50 shows:

06:02:29.250071 IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44449: UDP, length 958
06:02:29.351808 IP (tos 0xc0, ttl 64, id 16859, offset 0, flags [none], proto ICMP (1), length 576)
    172.16.50.10 > 192.168.20.5: ICMP 172.16.50.10 udp port 44449 unreachable, length 556
	IP (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 986)
    192.168.20.5.54354 > 172.16.50.10.44449: UDP, length 958

VLAN 50 only sees the packets with a ttl big enough to make it that far, which is why it sees fewer packets than VLAN 20. The ttl for the packet from Linux-1 (192.168.20.5) has decremented down to 1 as it passed through the network.

Example 4 – HTTP Get From Host On A Remote Subnet

In the previous examples, we’ve been testing the MTU with pings. But how does a TCP session behave? In the next example, we will use curl on Linux-1 to fetch a webpage from Linux-3. The HTML page being fetched is a cut-down version of the default webpage for an Ubuntu installation, shortened to be around 2Kb in size.

Initially let’s set the MTU back to its normal settings to get a baseline view of what it looks like.

From Linux-1 we see the following. Note I have removed some of the output for brevity.

eve@linux-1:~$ curl http://172.16.50.10/old.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <!--
    Modified from the Debian original for Ubuntu
    Last updated: 2022-03-22
    See: https://launchpad.net/bugs/1966004
  -->
  <head>
 *** OUTPUT OMITTED FOR BREVITY ***
  </head>
  <body>
  *** OUTPUT OMITTED FOR BREVITY ***
  </body>
</html>

The VLAN 20 packet capture shows the TCP three-way handshake establishing, the HTTP request, the HTTP response, and the connection being closed.

09:28:15.389727 IP (tos 0x0, ttl 64, id 53997, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [S], cksum 0x5bf4 (correct), seq 1934198875, win 64240, options [mss 1460,sackOK,TS val 585728332 ecr 0,nop,wscale 7], length 0
09:28:16.099116 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.40488: Flags [S.], cksum 0xdbb0 (correct), seq 813536623, ack 1934198876, win 65160, options [mss 1460,sackOK,TS val 4242389455 ecr 585728332,nop,wscale 7], length 0
09:28:16.202631 IP (tos 0x0, ttl 64, id 53998, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0x0445 (correct), ack 1, win 502, options [nop,nop,TS val 585729047 ecr 4242389455], length 0
09:28:16.205480 IP (tos 0x0, ttl 64, id 53999, offset 0, flags [DF], proto TCP (6), length 136)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [P.], cksum 0x5206 (correct), seq 1:85, ack 1, win 502, options [nop,nop,TS val 585729047 ecr 4242389455], length 84: HTTP, length: 84
	GET /old.html HTTP/1.1
	Host: 172.16.50.10
	User-Agent: curl/7.81.0
	Accept: */*
	
09:28:16.574040 IP (tos 0x0, ttl 61, id 61398, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.40488: Flags [.], cksum 0x00ba (correct), ack 85, win 509, options [nop,nop,TS val 4242390271 ecr 585729047], length 0
09:28:16.574040 IP (tos 0x0, ttl 61, id 61399, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.40488: Flags [.], cksum 0x403e (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4242390294 ecr 585729047], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 09:28:16 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***

09:28:16.576919 IP (tos 0x0, ttl 61, id 61400, offset 0, flags [DF], proto TCP (6), length 1318)
    172.16.50.10.http > 192.168.20.5.40488: Flags [P.], cksum 0xdf30 (correct), seq 1449:2715, ack 85, win 509, options [nop,nop,TS val 4242390294 ecr 585729047], length 1266: HTTP
09:28:16.583140 IP (tos 0x0, ttl 64, id 54000, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0xf927 (correct), ack 1449, win 501, options [nop,nop,TS val 585729522 ecr 4242390294], length 0
09:28:16.583141 IP (tos 0x0, ttl 64, id 54001, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0xf435 (correct), ack 2715, win 501, options [nop,nop,TS val 585729522 ecr 4242390294], length 0
09:28:16.592381 IP (tos 0x0, ttl 64, id 54002, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [F.], cksum 0xf433 (correct), seq 85, ack 2715, win 501, options [nop,nop,TS val 585729523 ecr 4242390294], length 0
09:28:17.297690 IP (tos 0x0, ttl 61, id 61401, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.40488: Flags [F.], cksum 0xf2c0 (correct), seq 2715, ack 86, win 509, options [nop,nop,TS val 4242390656 ecr 585729523], length 0
09:28:17.398713 IP (tos 0x0, ttl 64, id 54003, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0xeff5 (correct), ack 2716, win 501, options [nop,nop,TS val 585730246 ecr 4242390656], length 0

The VLAN 50 capture is as follows, it is basically the same as that taken in VLAN 20.

09:28:15.693286 IP (tos 0x0, ttl 61, id 53997, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [S], cksum 0x5bf4 (correct), seq 1934198875, win 64240, options [mss 1460,sackOK,TS val 585728332 ecr 0,nop,wscale 7], length 0
09:28:15.794227 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.40488: Flags [S.], cksum 0xdbb0 (correct), seq 813536623, ack 1934198876, win 65160, options [mss 1460,sackOK,TS val 4242389455 ecr 585728332,nop,wscale 7], length 0
09:28:16.507733 IP (tos 0x0, ttl 61, id 53998, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0x0445 (correct), ack 1, win 502, options [nop,nop,TS val 585729047 ecr 4242389455], length 0
09:28:16.510008 IP (tos 0x0, ttl 61, id 53999, offset 0, flags [DF], proto TCP (6), length 136)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [P.], cksum 0x5206 (correct), seq 1:85, ack 1, win 502, options [nop,nop,TS val 585729047 ecr 4242389455], length 84: HTTP, length: 84
	GET /old.html HTTP/1.1
	Host: 172.16.50.10
	User-Agent: curl/7.81.0
	Accept: */*
	
09:28:16.538502 IP (tos 0x0, ttl 64, id 61398, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.40488: Flags [.], cksum 0x00ba (correct), ack 85, win 509, options [nop,nop,TS val 4242390271 ecr 585729047], length 0
09:28:16.545906 IP (tos 0x0, ttl 64, id 61399, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.40488: Flags [.], cksum 0x403e (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4242390294 ecr 585729047], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 09:28:16 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***

09:28:16.549640 IP (tos 0x0, ttl 64, id 61400, offset 0, flags [DF], proto TCP (6), length 1318)
    172.16.50.10.http > 192.168.20.5.40488: Flags [P.], cksum 0xdf30 (correct), seq 1449:2715, ack 85, win 509, options [nop,nop,TS val 4242390294 ecr 585729047], length 1266: HTTP
09:28:16.891298 IP (tos 0x0, ttl 61, id 54000, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0xf927 (correct), ack 1449, win 501, options [nop,nop,TS val 585729522 ecr 4242390294], length 0
09:28:16.892584 IP (tos 0x0, ttl 61, id 54001, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0xf435 (correct), ack 2715, win 501, options [nop,nop,TS val 585729522 ecr 4242390294], length 0
09:28:16.894399 IP (tos 0x0, ttl 61, id 54002, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [F.], cksum 0xf433 (correct), seq 85, ack 2715, win 501, options [nop,nop,TS val 585729523 ecr 4242390294], length 0
09:28:16.992434 IP (tos 0x0, ttl 64, id 61401, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.40488: Flags [F.], cksum 0xf2c0 (correct), seq 2715, ack 86, win 509, options [nop,nop,TS val 4242390656 ecr 585729523], length 0
09:28:17.703350 IP (tos 0x0, ttl 61, id 54003, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.40488 > 172.16.50.10.http: Flags [.], cksum 0xeff5 (correct), ack 2716, win 501, options [nop,nop,TS val 585730246 ecr 4242390656], length 0

Next, I dropped the MTU on the interfaces between R1 and R2 again, back to 1000, and repeated the test.

From Linux-1 CLI, things look the same.

eve@linux-1:~$ curl http://172.16.50.10/old.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <!--
    Modified from the Debian original for Ubuntu
    Last updated: 2022-03-22
    See: https://launchpad.net/bugs/1966004
  -->
  <head>
 *** OUTPUT OMITTED FOR BREVITY ***
  </head>
  <body>
 *** OUTPUT OMITTED FOR BREVITY ***
  </body>
</html>

The VLAN 20 capture is shown below. From the Linux-1 perspective, it looks very similar to the first attempt. If you look closely though you will notice that the responses back from Linux-3 are smaller. In particular, the IP Packet containing the first part of the HTTP Response is 986 bytes instead of 1500 bytes.

09:39:30.101341 IP (tos 0x0, ttl 64, id 44432, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [S], cksum 0x0b4c (correct), seq 1948226610, win 64240, options [mss 1460,sackOK,TS val 586402819 ecr 0,nop,wscale 7], length 0
09:39:30.918939 IP (tos 0x0, ttl 64, id 44433, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [S], cksum 0x075c (correct), seq 1948226610, win 64240, options [mss 1460,sackOK,TS val 586403827 ecr 0,nop,wscale 7], length 0
09:39:31.019361 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.38754: Flags [S.], cksum 0xd695 (correct), seq 1942665969, ack 1948226611, win 65160, options [mss 1460,sackOK,TS val 4243064168 ecr 586402819,nop,wscale 7], length 0
09:39:31.122246 IP (tos 0x0, ttl 64, id 44434, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xfd78 (correct), ack 1, win 502, options [nop,nop,TS val 586403967 ecr 4243064168], length 0
09:39:31.122246 IP (tos 0x0, ttl 64, id 44435, offset 0, flags [DF], proto TCP (6), length 136)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [P.], cksum 0x4b3a (correct), seq 1:85, ack 1, win 502, options [nop,nop,TS val 586403967 ecr 4243064168], length 84: HTTP, length: 84
	GET /old.html HTTP/1.1
	Host: 172.16.50.10
	User-Agent: curl/7.81.0
	Accept: */*
	
09:39:31.455351 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.38754: Flags [S.], cksum 0xd363 (correct), seq 1942665969, ack 1948226611, win 65160, options [mss 1460,sackOK,TS val 4243064986 ecr 586402819,nop,wscale 7], length 0
09:39:31.564842 IP (tos 0x0, ttl 64, id 44436, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xfb70 (correct), ack 1, win 502, options [nop,nop,TS val 586404403 ecr 4243064168], length 0
09:39:31.569788 IP (tos 0x0, ttl 61, id 1551, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.38754: Flags [.], cksum 0xf91a (correct), ack 85, win 509, options [nop,nop,TS val 4243065195 ecr 586403967], length 0
09:39:31.718444 IP (tos 0x0, ttl 61, id 1554, offset 0, flags [DF], proto TCP (6), length 986)
    172.16.50.10.http > 192.168.20.5.38754: Flags [.], cksum 0x2638 (correct), seq 1:935, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 934: HTTP, length: 934
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 09:39:31 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
09:39:31.723790 IP (tos 0x0, ttl 61, id 1555, offset 0, flags [DF], proto TCP (6), length 986)
    172.16.50.10.http > 192.168.20.5.38754: Flags [P.], cksum 0xf322 (correct), seq 935:1869, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 934: HTTP
09:39:31.729680 IP (tos 0x0, ttl 61, id 1556, offset 0, flags [DF], proto TCP (6), length 898)
    172.16.50.10.http > 192.168.20.5.38754: Flags [P.], cksum 0xe7cd (correct), seq 1869:2715, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 846: HTTP
09:39:31.841854 IP (tos 0x0, ttl 64, id 44437, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xf2c0 (correct), ack 935, win 501, options [nop,nop,TS val 586404666 ecr 4243065196], length 0
09:39:31.843129 IP (tos 0x0, ttl 64, id 44438, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xef19 (correct), ack 1869, win 501, options [nop,nop,TS val 586404667 ecr 4243065196], length 0
09:39:31.843569 IP (tos 0x0, ttl 64, id 44439, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xebc2 (correct), ack 2715, win 501, options [nop,nop,TS val 586404676 ecr 4243065196], length 0
09:39:31.843938 IP (tos 0x0, ttl 64, id 44440, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [F.], cksum 0xebc1 (correct), seq 85, ack 2715, win 501, options [nop,nop,TS val 586404676 ecr 4243065196], length 0
09:39:32.539977 IP (tos 0x0, ttl 61, id 1557, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.38754: Flags [F.], cksum 0xe8fd (correct), seq 2715, ack 86, win 509, options [nop,nop,TS val 4243065895 ecr 586404676], length 0
09:39:32.641173 IP (tos 0x0, ttl 64, id 44441, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xe5d9 (correct), ack 2716, win 501, options [nop,nop,TS val 586405488 ecr 4243065895], length 0

The VLAN 50 capture is shown below. This is where we can see the real difference when the MTU in the middle is reduced. The first time it sends the HTTP response back, the IP packet is 1500 bytes large, and Linux-3 receives an ICMP Unreachable, Fragmentation Needed packet. It then re-sends the HTTP response, but this time the IP packet is 986 bytes big, which matches what was seen on the VLAN 20 capture.

09:39:30.406306 IP (tos 0x0, ttl 61, id 44432, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [S], cksum 0x0b4c (correct), seq 1948226610, win 64240, options [mss 1460,sackOK,TS val 586402819 ecr 0,nop,wscale 7], length 0
09:39:30.508875 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.50.1 tell 172.16.50.10, length 28
09:39:30.715983 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.38754: Flags [S.], cksum 0xd695 (correct), seq 1942665969, ack 1948226611, win 65160, options [mss 1460,sackOK,TS val 4243064168 ecr 586402819,nop,wscale 7], length 0
09:39:31.225957 IP (tos 0x0, ttl 61, id 44433, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [S], cksum 0x075c (correct), seq 1948226610, win 64240, options [mss 1460,sackOK,TS val 586403827 ecr 0,nop,wscale 7], length 0
09:39:31.330247 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.38754: Flags [S.], cksum 0xd363 (correct), seq 1942665969, ack 1948226611, win 65160, options [mss 1460,sackOK,TS val 4243064986 ecr 586402819,nop,wscale 7], length 0
09:39:31.432173 IP (tos 0x0, ttl 61, id 44434, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xfd78 (correct), ack 1, win 502, options [nop,nop,TS val 586403967 ecr 4243064168], length 0
09:39:31.434657 IP (tos 0x0, ttl 61, id 44435, offset 0, flags [DF], proto TCP (6), length 136)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [P.], cksum 0x4b3a (correct), seq 1:85, ack 1, win 502, options [nop,nop,TS val 586403967 ecr 4243064168], length 84: HTTP, length: 84
	GET /old.html HTTP/1.1
	Host: 172.16.50.10
	User-Agent: curl/7.81.0
	Accept: */*
	
09:39:31.441634 IP (tos 0x0, ttl 64, id 1551, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.38754: Flags [.], cksum 0xf91a (correct), ack 85, win 509, options [nop,nop,TS val 4243065195 ecr 586403967], length 0
09:39:31.443836 IP (tos 0x0, ttl 64, id 1552, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.38754: Flags [.], cksum 0x35b9 (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 09:39:31 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***

09:39:31.445770 IP (tos 0x0, ttl 64, id 1553, offset 0, flags [DF], proto TCP (6), length 1318)
    172.16.50.10.http > 192.168.20.5.38754: Flags [P.], cksum 0xd7a7 (correct), seq 1449:2715, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 1266: HTTP
09:39:31.665673 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    10.100.2.1 > 172.16.50.10: ICMP 192.168.20.5 unreachable - need to frag (mtu 986), length 36
	IP (tos 0x0, ttl 63, id 1552, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.38754:  [|tcp]
09:39:31.666471 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    10.100.2.1 > 172.16.50.10: ICMP 192.168.20.5 unreachable - need to frag (mtu 986), length 36
	IP (tos 0x0, ttl 63, id 1553, offset 0, flags [DF], proto TCP (6), length 1318)
    172.16.50.10.http > 192.168.20.5.38754:  [|tcp]
09:39:31.674297 IP (tos 0x0, ttl 64, id 1554, offset 0, flags [DF], proto TCP (6), length 986)
    172.16.50.10.http > 192.168.20.5.38754: Flags [.], cksum 0x2638 (correct), seq 1:935, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 934: HTTP, length: 934
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 09:39:31 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***

09:39:31.674912 IP (tos 0x0, ttl 64, id 1555, offset 0, flags [DF], proto TCP (6), length 986)
    172.16.50.10.http > 192.168.20.5.38754: Flags [P.], cksum 0xf322 (correct), seq 935:1869, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 934: HTTP
09:39:31.675048 IP (tos 0x0, ttl 64, id 1556, offset 0, flags [DF], proto TCP (6), length 898)
    172.16.50.10.http > 192.168.20.5.38754: Flags [P.], cksum 0xe7cd (correct), seq 1869:2715, ack 85, win 509, options [nop,nop,TS val 4243065196 ecr 586403967], length 846: HTTP
09:39:31.693365 IP (tos 0x0, ttl 61, id 44436, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xfb70 (correct), ack 1, win 502, options [nop,nop,TS val 586404403 ecr 4243064168], length 0
09:39:32.134109 IP (tos 0x0, ttl 61, id 44437, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xf2c0 (correct), ack 935, win 501, options [nop,nop,TS val 586404666 ecr 4243065196], length 0
09:39:32.134110 IP (tos 0x0, ttl 61, id 44438, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xef19 (correct), ack 1869, win 501, options [nop,nop,TS val 586404667 ecr 4243065196], length 0
09:39:32.134110 IP (tos 0x0, ttl 61, id 44439, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xebc2 (correct), ack 2715, win 501, options [nop,nop,TS val 586404676 ecr 4243065196], length 0
09:39:32.134110 IP (tos 0x0, ttl 61, id 44440, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [F.], cksum 0xebc1 (correct), seq 85, ack 2715, win 501, options [nop,nop,TS val 586404676 ecr 4243065196], length 0
09:39:32.241777 IP (tos 0x0, ttl 64, id 1557, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.38754: Flags [F.], cksum 0xe8fd (correct), seq 2715, ack 86, win 509, options [nop,nop,TS val 4243065895 ecr 586404676], length 0
09:39:32.945692 IP (tos 0x0, ttl 61, id 44441, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.38754 > 172.16.50.10.http: Flags [.], cksum 0xe5d9 (correct), ack 2716, win 501, options [nop,nop,TS val 586405488 ecr 4243065895], length 0

For the next test, I’ve configured R3 to drop inbound ICMP traffic on its ge-0/0/1.0 interface, which should prevent ICMP traffic from R1 or R2 from reaching Linux-3. Note that I’ve run sudo netplan apply on both Linux-1 and Linux-3 to clear them from “remembering” that they need to fragment packets.

On Linux-1 we see the following CLI output. Note that it took quite a while between the command being run and the connection being reset message appearing.

eve@linux-1:~$ curl http://172.16.50.10/old.html
curl: (56) Recv failure: Connection reset by peer
eve@linux-1:~$

The VLAN 20 capture shows the establishment of the TCP session and the HTTP request being sent successfully. It never receives the HTTP Response and eventually gets an RST (Reset) from Linux-3 to reset the connection.

10:01:19.383590 IP (tos 0x0, ttl 64, id 47028, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [S], cksum 0x3913 (correct), seq 3588669462, win 64240, options [mss 1460,sackOK,TS val 587712122 ecr 0,nop,wscale 7], length 0
10:01:20.248333 IP (tos 0x0, ttl 64, id 47029, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [S], cksum 0x351a (correct), seq 3588669462, win 64240, options [mss 1460,sackOK,TS val 587713139 ecr 0,nop,wscale 7], length 0
10:01:20.351026 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.57700: Flags [S.], cksum 0x1df4 (correct), seq 2314506412, ack 3588669463, win 65160, options [mss 1460,sackOK,TS val 4244373464 ecr 587712122,nop,wscale 7], length 0
10:01:20.449555 IP (tos 0x0, ttl 64, id 47030, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0x44bd (correct), ack 1, win 502, options [nop,nop,TS val 587713296 ecr 4244373464], length 0
10:01:20.450159 IP (tos 0x0, ttl 64, id 47031, offset 0, flags [DF], proto TCP (6), length 136)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [P.], cksum 0x927e (correct), seq 1:85, ack 1, win 502, options [nop,nop,TS val 587713296 ecr 4244373464], length 84: HTTP, length: 84
	GET /old.html HTTP/1.1
	Host: 172.16.50.10
	User-Agent: curl/7.81.0
	Accept: */*
	
10:01:20.780781 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.57700: Flags [S.], cksum 0x1aa5 (correct), seq 2314506412, ack 3588669463, win 65160, options [mss 1460,sackOK,TS val 4244374311 ecr 587712122,nop,wscale 7], length 0
10:01:20.880685 IP (tos 0x0, ttl 64, id 47032, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0x42ba (correct), ack 1, win 502, options [nop,nop,TS val 587713727 ecr 4244373464], length 0
10:01:20.880686 IP (tos 0x0, ttl 61, id 65030, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x4047 (correct), ack 85, win 509, options [nop,nop,TS val 4244374515 ecr 587713296], length 0
10:01:26.067987 IP (tos 0x0, ttl 61, id 65034, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [F.], cksum 0x2072 (correct), seq 2715, ack 85, win 509, options [nop,nop,TS val 4244379518 ecr 587713727], length 0
10:01:26.169007 IP (tos 0x0, ttl 64, id 47033, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xa965 (correct), ack 1, win 502, options [nop,nop,TS val 587719016 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:02:26.213079 IP (tos 0x0, ttl 64, id 47034, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xbe7a (correct), ack 1, win 502, options [nop,nop,TS val 587779155 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:02:27.024317 IP (tos 0x0, ttl 61, id 65040, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x1e47 (correct), ack 85, win 509, options [nop,nop,TS val 4244440319 ecr 587719016], length 0
10:03:27.140174 IP (tos 0x0, ttl 64, id 47035, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xd079 (correct), ack 1, win 502, options [nop,nop,TS val 587840083 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:03:27.869567 IP (tos 0x0, ttl 61, id 65042, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x3067 (correct), ack 85, win 509, options [nop,nop,TS val 4244501214 ecr 587719016], length 0
10:04:28.585607 IP (tos 0x0, ttl 64, id 47036, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xe078 (correct), ack 1, win 502, options [nop,nop,TS val 587901523 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:04:29.305162 IP (tos 0x0, ttl 61, id 65043, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x4064 (correct), ack 85, win 509, options [nop,nop,TS val 4244562656 ecr 587719016], length 0
10:05:30.024435 IP (tos 0x0, ttl 64, id 47037, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xf077 (correct), ack 1, win 502, options [nop,nop,TS val 587962963 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:05:30.743172 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.50.10.http > 192.168.20.5.57700: Flags [R], cksum 0x00c2 (correct), seq 2314506413, win 0, length 0

The VLAN 50 capture shows the initial TCP establishment and Linux-3 receiving the HTTP Request from Linux-1. The response is being sent with the DF bit set and because ICMP messages are being blocked at the router Linux-3 never learns that it should fragment the packet. Thus it keeps retrying before finally sending an RST to reset the connection.

10:01:19.702684 IP (tos 0x0, ttl 61, id 47028, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [S], cksum 0x3913 (correct), seq 3588669462, win 64240, options [mss 1460,sackOK,TS val 587712122 ecr 0,nop,wscale 7], length 0
10:01:20.045004 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.57700: Flags [S.], cksum 0x1df4 (correct), seq 2314506412, ack 3588669463, win 65160, options [mss 1460,sackOK,TS val 4244373464 ecr 587712122,nop,wscale 7], length 0
10:01:20.549866 IP (tos 0x0, ttl 61, id 47029, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [S], cksum 0x351a (correct), seq 3588669462, win 64240, options [mss 1460,sackOK,TS val 587713139 ecr 0,nop,wscale 7], length 0
10:01:20.650429 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.57700: Flags [S.], cksum 0x1aa5 (correct), seq 2314506412, ack 3588669463, win 65160, options [mss 1460,sackOK,TS val 4244374311 ecr 587712122,nop,wscale 7], length 0
10:01:20.752073 IP (tos 0x0, ttl 61, id 47030, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0x44bd (correct), ack 1, win 502, options [nop,nop,TS val 587713296 ecr 4244373464], length 0
10:01:20.753223 IP (tos 0x0, ttl 61, id 47031, offset 0, flags [DF], proto TCP (6), length 136)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [P.], cksum 0x927e (correct), seq 1:85, ack 1, win 502, options [nop,nop,TS val 587713296 ecr 4244373464], length 84: HTTP, length: 84
	GET /old.html HTTP/1.1
	Host: 172.16.50.10
	User-Agent: curl/7.81.0
	Accept: */*
	
10:01:20.761612 IP (tos 0x0, ttl 64, id 65030, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x4047 (correct), ack 85, win 509, options [nop,nop,TS val 4244374515 ecr 587713296], length 0
10:01:20.763979 IP (tos 0x0, ttl 64, id 65031, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x84f0 (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4244374517 ecr 587713296], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:01:21 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
10:01:20.766249 IP (tos 0x0, ttl 64, id 65032, offset 0, flags [DF], proto TCP (6), length 1318)
    172.16.50.10.http > 192.168.20.5.57700: Flags [P.], cksum 0x1ed3 (correct), seq 1449:2715, ack 85, win 509, options [nop,nop,TS val 4244374517 ecr 587713296], length 1266: HTTP
10:01:21.182525 IP (tos 0x0, ttl 61, id 47032, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0x42ba (correct), ack 1, win 502, options [nop,nop,TS val 587713727 ecr 4244373464], length 0
10:01:22.916604 IP (tos 0x0, ttl 64, id 65033, offset 0, flags [DF], proto TCP (6), length 1318)
    172.16.50.10.http > 192.168.20.5.57700: Flags [P.], cksum 0x14bb (correct), seq 1449:2715, ack 85, win 509, options [nop,nop,TS val 4244376670 ecr 587713727], length 1266: HTTP
10:01:25.761325 IP (tos 0x0, ttl 64, id 65034, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [F.], cksum 0x2072 (correct), seq 2715, ack 85, win 509, options [nop,nop,TS val 4244379518 ecr 587713727], length 0
10:01:26.475852 IP (tos 0x0, ttl 61, id 47033, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xa965 (correct), ack 1, win 502, options [nop,nop,TS val 587719016 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:01:26.483676 IP (tos 0x0, ttl 64, id 65035, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x5840 (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4244380237 ecr 587719016], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:01:21 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
10:01:29.826427 IP (tos 0x0, ttl 64, id 65036, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x4b2f (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4244383582 ecr 587719016], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:01:21 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
10:01:36.226581 IP (tos 0x0, ttl 64, id 65037, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x322f (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4244389982 ecr 587719016], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:01:21 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
10:01:48.769989 IP (tos 0x0, ttl 64, id 65038, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x012f (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4244402526 ecr 587719016], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:01:21 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
10:02:14.626412 IP (tos 0x0, ttl 64, id 65039, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x9c2e (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4244428382 ecr 587719016], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:01:21 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
10:02:26.557867 IP (tos 0x0, ttl 61, id 47034, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xbe7a (correct), ack 1, win 502, options [nop,nop,TS val 587779155 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:02:26.668002 IP (tos 0x0, ttl 64, id 65040, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x1e47 (correct), ack 85, win 509, options [nop,nop,TS val 4244440319 ecr 587719016], length 0
10:03:05.827243 IP (tos 0x0, ttl 64, id 65041, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0xd42d (correct), seq 1:1449, ack 85, win 509, options [nop,nop,TS val 4244479582 ecr 587719016], length 1448: HTTP, length: 1448
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:01:21 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
	
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	  <!--
	    Modified from the Debian original for Ubuntu
	    Last updated: 2022-03-22
	    See: https://launchpad.net/bugs/1966004
	  -->
	  <head>
*** OUTPUT OMITTED FOR BREVITY ***
10:03:27.453118 IP (tos 0x0, ttl 61, id 47035, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xd079 (correct), ack 1, win 502, options [nop,nop,TS val 587840083 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:03:27.554676 IP (tos 0x0, ttl 64, id 65042, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x3067 (correct), ack 85, win 509, options [nop,nop,TS val 4244501214 ecr 587719016], length 0
10:04:28.893956 IP (tos 0x0, ttl 61, id 47036, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xe078 (correct), ack 1, win 502, options [nop,nop,TS val 587901523 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:04:29.001727 IP (tos 0x0, ttl 64, id 65043, offset 0, flags [DF], proto TCP (6), length 52)
    172.16.50.10.http > 192.168.20.5.57700: Flags [.], cksum 0x4064 (correct), ack 85, win 509, options [nop,nop,TS val 4244562656 ecr 587719016], length 0
10:05:30.338337 IP (tos 0x0, ttl 61, id 47037, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [.], cksum 0xf077 (correct), ack 1, win 502, options [nop,nop,TS val 587962963 ecr 4244374515,nop,nop,sack 1 {2715:2716}], length 0
10:05:30.442233 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    172.16.50.10.http > 192.168.20.5.57700: Flags [R], cksum 0x00c2 (correct), seq 2314506413, win 0, length 0

So why show you this example of having a smaller MTU in the middle of a path and blocking ICMP from reaching a webserver? Because some common practices cause this to occur on the Internet today. Most firewall admins and some network admins block ICMP entirely from entering their network. This prevents the discovery of the Path MTU and causes breakage.

From an end-user perspective, it can be quite frustrating. The connection establishes, and the initial request being small makes it through the network. The larger response can’t get through causing the connection to time out or get reset as seen above.

And now for one final example using TCP-MSS clamping. Leaving the network as it was, with a small MTU in the middle and R3 blocking ICMP traffic, we can make one small change. On R1 we set the TCP MSS on the ge-0/0/2 interface.

lab@vMX-1> show configuration interfaces ge-0/0/2  
mtu 1000;
mac aa:bb:cc:dd:00:11;
unit 0 {
    family inet {
        tcp-mss 800;
        address 10.100.1.2/24;
    }
}

Let’s see the CLI output from Linux-1 when we try the curl command.

eve@linux-1:~$ curl http://172.16.50.10/old.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <!--
    Modified from the Debian original for Ubuntu
    Last updated: 2022-03-22
    See: https://launchpad.net/bugs/1966004
  -->
  <head>
*** OUTPUT OMITTED FOR BREVITY ***
  </head>
  <body>
*** OUTPUT OMITTED FOR BREVITY ***
  </body>
</html>

Looking at the VLAN 20 capture from the previous test and the current test we see the following.

10:01:20.248333 IP (tos 0x0, ttl 64, id 47029, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [S], cksum 0x351a (correct), seq 3588669462, win 64240, options [mss 1460,sackOK,TS val 587713139 ecr 0,nop,wscale 7], length 0
10:01:20.351026 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.57700: Flags [S.], cksum 0x1df4 (correct), seq 2314506412, ack 3588669463, win 65160, options [mss 1460,sackOK,TS val 4244373464 ecr 587712122,nop,wscale 7], length 0

and

10:47:44.630885 IP (tos 0x0, ttl 64, id 22202, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.46534 > 172.16.50.10.http: Flags [S], cksum 0x2450 (correct), seq 3745085006, win 64240, options [mss 1460,sackOK,TS val 590497574 ecr 0,nop,wscale 7], length 0
10:47:45.447726 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.46534: Flags [S.], cksum 0xa46d (correct), seq 1292546637, ack 3745085007, win 65160, options [mss 1460,sackOK,TS val 4247158714 ecr 590497574,nop,wscale 7], length 0

Notice how in both cases the MSS setting for TCP is 1460.

Now let’s look at the VLAN 50 capture of both the previous and the current test.

10:01:19.702684 IP (tos 0x0, ttl 61, id 47028, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.57700 > 172.16.50.10.http: Flags [S], cksum 0x3913 (correct), seq 3588669462, win 64240, options [mss 1460,sackOK,TS val 587712122 ecr 0,nop,wscale 7], length 0
10:01:20.045004 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.57700: Flags [S.], cksum 0x1df4 (correct), seq 2314506412, ack 3588669463, win 65160, options [mss 1460,sackOK,TS val 4244373464 ecr 587712122,nop,wscale 7], length 0

and

10:47:44.952716 IP (tos 0x0, ttl 61, id 22202, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.5.46534 > 172.16.50.10.http: Flags [S], cksum 0x26e4 (correct), seq 3745085006, win 64240, options [mss 800,sackOK,TS val 590497574 ecr 0,nop,wscale 7], length 0
10:47:45.062566 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.16.50.10.http > 192.168.20.5.46534: Flags [S.], cksum 0xa46d (correct), seq 1292546637, ack 3745085007, win 65160, options [mss 1460,sackOK,TS val 4247158714 ecr 590497574,nop,wscale 7], length 0

You can see how the Linux-1 packet now has TCP MSS set to 800. This was changed by R1 as the packet passed through the network. In this scenario we have only applied TCP-MSS clamping to one side of the R1 to R2 connection so this works in one direction but wouldn’t work in the other. So the MSS clamping is best applied to both sides of a connection.

This leads to the following packet for the HTTP Response from Linux-3 to Linux-1. Note how the overall length of the IP packet is 840 bytes, which will be the TCP MSS plus TCP header plus IP header.

10:47:45.953313 IP (tos 0x0, ttl 64, id 49528, offset 0, flags [DF], proto TCP (6), length 840)
    172.16.50.10.http > 192.168.20.5.46534: Flags [.], cksum 0x1d9c (correct), seq 1:789, ack 85, win 509, options [nop,nop,TS val 4247159700 ecr 590498395], length 788: HTTP, length: 788
	HTTP/1.1 200 OK
	Date: Sat, 11 Nov 2023 10:47:46 GMT
	Server: Apache/2.4.52 (Ubuntu)
	Last-Modified: Sun, 05 Nov 2023 11:37:02 GMT
	ETag: "99d-609662801e940"
	Accept-Ranges: bytes
	Content-Length: 2461
	Vary: Accept-Encoding
	Content-Type: text/html
*** OUTPUT OMITTED FOR BREVITY ***

So is TCP MSS clamping the right answer? For this scenario, I tend to say no. MSS clamping is usually applied when configuring tunneling, e.g. an IPSec or VPN connection where we know we need to reduce the MSS size.

If we assume that the R1 to R2 link is outside of our control, then we don’t know what we should set the MSS clamping to. And even if we did what if it changes in the future? In this case, I think it is far better to let ICMP do its job and ensure we don’t break Path MTU Discovery (PMTUD) by blocking ICMP traffic entirely.

Conclusions

So what do we take away from all of this exploring of MTU?

  • MTU issues can be difficult to diagnose and are made up of several possible causes. The fact that the initial connection can establish and make a request but responses fail can make it feel like it is not a network issue.
  • When adding additional encapsulation such as using a VPN, TCP-MSS Clamping may be required for applications to work well.
  • A lot of initial packets are sent with the DF bit set to trigger PMTUD.
  • IP Fragmentation can occur either on the end hosts or be done by routers along the network path.
  • Know your equipment – what does their MTU output mean? Does it include the layer 2 headers or not?
  • Know your tools – when you specify something like size, what exactly are you specifying the size of? Is it the payload size or the overall packet size?

Leave a Comment

Your email address will not be published. Required fields are marked *