AWS PrivateLink Tunnel Endpoints Push the Routing Tax onto Your Kernel
Every auditor I have ever sat across from hates cross-account VPC peering for the same reason: it opens a bidirectional pipe between two routing tables that do not trust each other. Until recently, the only clean escape on AWS was PrivateLink interface endpoints, and those fall apart the moment a vendor needs to talk to an ephemeral cluster where nodes come and go every twenty minutes. Now InfoQ has detailed AWS PrivateLink tunnel endpoints, a feature that shares a whole CIDR block across accounts without peering. The pitch sounds like the missing middle between a single-target NLB and a full mesh. Look at the packet path, though, and you will find that AWS moved the complexity out of the control plane and dumped it directly onto your Linux network stack.
You do not point a socket at a tunnel endpoint IP and send plain TCP. The endpoint expects incoming packets wrapped in GENEVE (RFC 8926) over UDP port 6081, and it only accepts them from the same Availability Zone. When the encapsulated frame hits the tunnel endpoint, AWS strips the outer header, checks that the inner destination IP sits inside the CIDR block shared via Resource Access Manager, rewrites the source IP to the provider resource gateway, and drops a plain packet into the target VPC. That solves overlapping RFC 1918 space in one stroke, because both sides can run ten-dot-zero networks without colliding on the wire. It also enforces strict unidirectional initiation: the customer network cannot open a socket back into the vendor VPC.
Self-managed Kafka and Aerospike clusters are the obvious winners here. Both protocols hand back broker metadata containing raw node IPs after the initial bootstrap connection. Under classic PrivateLink, exposing a 30-broker cluster meant maintaining thirty separate target configurations or playing ugly port-mapping tricks on a Network Load Balancer. Sharing a /24 subnet configuration lets the client talk to whatever broker IP the cluster metadata returns, including replacement nodes spun up by an autoscaler three hours after the share was created.
Why the client side gets ugly in production
Ordinary application containers do not speak RFC 8926 on their own.
The host kernel has to do the heavy lifting. To keep the application unmodified, you build a virtual tunnel interface, encapsulate traffic bound for the remote CIDR, and steer it toward the local AZ endpoint IP while isolating that interface inside a dedicated Linux network namespace so an overlapping route does not hijack the instance’s own metadata service or local VPC resolver. Here is the exact setup on a consumer host:
ip netns add privatelink-tun
ip link add geneve0 type geneve id 0 remote 10.20.1.45 dstport 6081
ip link set geneve0 netns privatelink-tun
ip -n privatelink-tun link set lo up
ip -n privatelink-tun link set geneve0 mtu 8950 up
ip -n privatelink-tun route add 172.16.40.0/24 dev geneve0
iptables -t mangle -A POSTROUTING -o geneve0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
That MSS clamp at the bottom is not optional. Tunnel endpoints drop IP fragments outright, and they block all ICMP traffic, including Type 3 Code 4 fragmentation warnings. Because UDP port 53 is the only non-TCP traffic allowed through the tunnel, Path MTU Discovery is blind. If a worker process inside your container tries to push a full 9001-byte jumbo frame or even a standard 1500-byte packet without accounting for the 50-byte GENEVE overhead, the connection hangs after the TLS handshake and nobody gets an ICMP error back to explain why. And if both VPCs share the exact same address range, as Ryo Suzuki demonstrated in his Classmethod tests, you cannot drop a broad route into the main table without breaking local subnet reachability; you end up running a root-level relay daemon, maintaining /32 host routes per target, or sticking an HAProxy box in front as a forwarder.
Blind spots at the resource gateway
The provider side has its own operational traps. Because the resource gateway rewrites the source IP of every inner packet to its own subnet address before forwarding it into the target VPC, the destination database or broker never sees the consumer’s private IP. VPC Flow Logs and gateway access logs tell you which consumer endpoint sent the bytes, but they carry no field for the inner source address.
Never approve a default-route CIDR share on a resource gateway attached to a routed VPC. Because the gateway forwards whatever matches its local route table, an unrestricted range turns a vendor tunnel into an unmonitored transitive bridge across your Direct Connect and VPN links, while source NAT strips your ability to attribute which workload inside the vendor account pulled the trigger.
Security groups on the target workloads have to allow the resource gateway’s entire subnet or security group ID rather than specific caller hosts. Worse, if a provider engineer takes the lazy path and shares a quad-zero route instead of a tight /24, the consumer can route packets out through the provider’s NAT gateway to the public internet, or pivot across transitive peering links attached to that VPC. I would use tunnel endpoints tomorrow for a SaaS control plane that needs to poll a customer’s self-hosted Kafka brokers inside a dedicated /25 subnet where no transit gateway exists. For anything broader, paying double data-processing fees (PrivateLink per-GB rates on the consumer side plus VPC Lattice resource gateway per-GB rates on the provider side) just to babysit kernel GENEVE namespaces and lose source-IP attribution is a worse deal than running a pair of locked-down WireGuard relays you can inspect with standard tcpdump.