Debugging WireGuard MTU Bottlenecks and TCP MSS Clamping

5 min read cybersecurity, Development

The Black-Hole Connection Symptoms

A WireGuard tunnel interface can appear fully active while dropping application payloads. Initial TCP connection setups succeed. An SSH handshake completes, curl fetches small HTTP headers without error, and DNS queries return immediate IP addresses. Troubleshooting breaks down when transferring larger payloads: scp file transfers freeze after sending the initial block, git pushes stall indefinitely, and web requests time out before receiving the response body.

This failure pattern stems from packet size discrepancies. Small control frames pass through the tunnel without issue because their overall size stays well below interface limits. The TCP handshake uses SYN, SYN-ACK, and ACK packets that rarely exceed 60 bytes. Once payload transfer starts, TCP scales up frame sizes to match the negotiated Maximum Segment Size. If an outgoing packet exceeds the path Maximum Transmission Unit (MTU), the transmitting router must fragment the frame or return an ICMP Fragmentation Needed notification.

When middleboxes, WAN firewalls, or internet service providers block ICMP Type 3 Code 4 messages, Path MTU Discovery (PMTUD) fails. The sender never receives notification that the packet was dropped for exceeding path size. TCP retransmits the full-sized frame repeatedly until socket timeout kills the connection, creating a black-hole drop.

Calculating WireGuard Overhead Across Encapsulations

WireGuard defaults to an MTU of 1420 bytes on standard Linux installations. This default assumes an outer WAN interface MTU of 1500 bytes running over IPv4. The encapsulation math breaks down when transport layers change or intermediate ISP framing adds headers.

Each WireGuard packet adds explicit protocol headers before entering the outer network interface:

  • Standard IPv4 header: 20 bytes
  • Outer UDP header: 8 bytes
  • WireGuard message header and authentication tag: 32 bytes
  • Outer IPv6 header (when tunneling over IPv6 transport): 40 bytes
  • PPPoE framing (common on fiber or DSL WAN links): 8 bytes

Over IPv4 transport, total encapsulation overhead equals 60 bytes. Subtracting 60 bytes from a standard 1500-byte WAN MTU yields an allowable payload MTU of 1440 bytes. Setting the WireGuard interface MTU to 1420 bytes provides a 20-byte safety margin.

Problems emerge when transport switches to IPv6 or passes through PPPoE connections. IPv6 transport adds 20 extra header bytes, increasing WireGuard overhead to 80 bytes. PPPoE reduces the WAN interface MTU from 1500 to 1492 bytes. A WireGuard tunnel operating over IPv6 across a PPPoE WAN connection leaves a maximum payload MTU of 1412 bytes. If wg0.conf retains the default 1420 MTU, every maximum-sized TCP segment exceeds the physical path limit by 8 bytes.

Tracing Dropped Frames with tcpdump and ICMP Filter Checks

Identifying path MTU bottlenecks requires monitoring interface flags and ICMP feedback on the gateway server while sending payload traffic.

Blocking ICMP Type 3 Code 4 messages breaks TCP Path MTU Discovery completely. When a network device drops an oversized frame without sending an ICMP response, the transmitting host cannot adjust its Maximum Segment Size.

Run tcpdump on both the physical WAN interface and the WireGuard tunnel interface while reproducing a stalled file transfer:

# Capture ICMP error signals and TCP SYN/ACK flags on the physical WAN interface
tcpdump -nn -i eth0 'icmp or icmp6 or (tcp[tcpflags] & (tcp-syn|tcp-ack) != 0)'

# Monitor WireGuard interface payload packet sizes
tcpdump -nn -i wg0 -s 0 'ip[2:2] > 1400'

If tcpdump shows outbound WireGuard UDP packets exceeding 1432 bytes on eth0 without corresponding ICMP unreachable responses returning from intermediate nodes, path MTU discovery is failing. You can confirm interface MTU configuration on Linux nodes using ip link:

# Inspect active link MTUs across physical and virtual interfaces
ip -d link show type wireguard

Fixing Path MTU Bottlenecks with MSS Clamping and Interface Tuning

Resolving silent WireGuard drops requires either reducing the tunnel interface MTU or enforcing TCP MSS clamping on routed traffic.

Direct MTU adjustment in wg0.conf fixes client-generated traffic. Lowering MTU to 1380 bytes provides full clearance across IPv4 transport, IPv6 transport, and PPPoE encapsulations without relying on external ICMP feedback:

[Interface]
PrivateKey = [SERVER_PRIVATE_KEY]
Address = 10.0.0.1/24
ListenPort = 51820
MTU = 1380

PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

For network setups where connected subnets send traffic through WireGuard, tuning wg0.conf alone will not affect client hosts that advertise 1460-byte TCP MSS options. Enforce TCP MSS clamping on the router forwarding path to automatically rewrite SYN packet MSS fields to match the outgoing tunnel path MTU.

# nftables rule to clamp TCP MSS to Path MTU automatically
nft add chain inet filter forward { type filter hook forward priority 0 \; }
nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu

# Legacy iptables command for WireGuard interface wg0
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -o wg0 -j TCPMSS --clamp-mss-to-pmtu

Setting an MTU of 1380 in wireguard configuration files combined with forwarding-path MSS clamping eliminates payload stalls across multi-hop network paths.

Press Cmd K to search برای جستجوی سایت از Cmd+K استفاده کنید