HOWTO check step-by-step a LVS-TUN setup for problems

IPVS supports different tunnel modes:
- IPv4 over IPIP: the default, supported from years
- IPv6 over IPv6: 2.6.* kernels
- IPv4 over IPv6, IPv6 over IPv4: Linux 3.18+
- IPv4/IPv6 over IPv4/6+UDP+GUE: Linux 5.2+
- IPv4/IPv6 over IPv4/6+UDP+GUE with checksums: Linux 5.3+
- IPv4/IPv6 over IPv4/6+GRE: Linux 5.3+

Example settings for real server:

# Command: "add" or "del"
cmd=add

# External device where IPIP or GRE traffic is received
edev=eth0

# Public IPs
rip="10.0.0.1/24"
vip="172.16.0.1/32"
# For IPv6
rip6="XXXX/64"
vip6="XXXX/128"

vport=80

# Add iptables rules to filter IPIP traffic on $edev, allow from director IPs

# Add public RIP (main IP, used by director as remote tunnel endpoint)
ip addr $cmd "$rip" brd + dev $edev
# For IPv6
ip -6 addr $cmd "$rip6" dev $edev

# Do not report VIP on $edev, set it before adding VIP
echo 1 > /proc/sys/net/ipv4/conf/$edev/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/$edev/arp_announce
# alternatively, unofficial (patch):
echo 1 > /proc/sys/net/ipv4/conf/all/hidden
echo 1 > /proc/sys/net/ipv4/conf/lo/hidden

# Configure VIP on loopback device (just one IP, no subnet)
ip addr $cmd "$vip" dev lo
ip -6 addr $cmd "$vip6" dev lo

# Start tunnels to accept traffic from director

# IPIP tunnel
modprobe ipip
ifconfig tunl0 0.0.0.0 up
# tunl0 is not a primary path for client_ip, so remove RP filter protection
echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter

# IPv6 tunnel
modprobe ip6_tunnel
ifconfig ip6tnl0 up

# IPv4 GRE tunnel
modprobe ip_gre
ip tunnel change gre0 mode gre icsum
ifconfig gre0 0.0.0.0 up

# IPv4/FOU/GUE
modprobe fou
ip fou add port 4589 gue

Example Director settings:

# IPv4 (IPIP)
ipvsadm -a -t "$vip:$vport" -r "$rip" -i

# IPv4/GRE+CSUM
ipvsadm -a -t "$vip:$vport" -r "$rip" -i --tun-type gre --tun-csum

# IPv4/UDP/GUE
ipvsadm -a -t "$vip:$vport" -r "$rip" -i --tun-type gue --tun-port 4589
# IPv4/UDP/GUE+CSUM
ipvsadm -a -t "$vip:$vport" -r "$rip" -i --tun-type gue --tun-port 4589 --tun-remcsum


Q.1 The director must use unique address when talking to the real server

	director# ip route get RIP

		If  the  above command  returns  as preferred  source IP
	address  the VIP or another IP address already configured on the
	real server then the IPIP packets will be dropped there.

Q.2 The real server's routing drops the decapsulated packets?

	rs# ip route get from CIP to VIP iif tunl0

	If  the  above command  returns error  then  may be  the request
	packets   are   dropped   from   rp_filter   protection.   Check
	tunl0/rp_filter.   It should be 0 to  allow the packets to reach
	the  real server's sockets.  Note that  in such case the reverse
	path   protection   can   not  detect   spoofed   packets  after
	decapsulation, so proper firewall rules should be added instead.

	The  packets can  be dropped also  when the tunl0  device is not
	configured properly for IP. To setup the real server use:

	echo 1 > /proc/sys/net/ipv4/conf/all/hidden
	ifconfig tunl0 0.0.0.0 up
	echo 1 > /proc/sys/net/ipv4/conf/tunl0/hidden
	ip addr add VIP dev tunl0

Q.3 The real server's gateway passes the packets to the client?

	rs# arp -d GW ; traceroute -n -s VIP CIP

	As  result, after some  time, we should see  on the director the
	ICMP  port unreachable replies  sent from the  client. Note that
	these  replies should be received in  the director. At least, we
	should see on the client the UDP packets generated from the real
	server.

	Note  that not  all ISPs  allow such  kind of  IP spoofing. Some
	ISPs  filter the outgoing traffic  sent with incorrect source IP
	address (VIP in our case).

Q.4 Other trivial reasons such as:

	- firewall filtering on the real server or on the director

	- the  real server's gateway allows spoofed IP traffic (see Q.3)
	but  drops ARP probes "who-has GW tell  VIP". In such case it is
	recommended  to make sure  the VIP is not  advertised in the ARP
	probes (for example, by hiding the VIP).


Some useful tcpdump commands that can help in the debugging process:

director# tcpdump -ln -i INDEV host CIP
director# tcpdump -ln -i OUTDEV host RIP -vvv
real server# tcpdump -ln -i IN_ETH host DIP -vvv
real server# tcpdump -ln -i tunl0 host CIP
real server# tcpdump -ln -i OUT_DEV host CIP


Document history:
02 Apr 2020: add configuration examples
03 Jan 2002: initial troubleshooting info


Julian Anastasov