Issue with TCP traffic and NAT running Free5GC in OpenStack

Q/A
#1

Hi,

We are working on deploying Free5G inside an Ubuntu virtual machine running on OpenStack infrastructure. We are connecting the UPF to a virtual router in order to access the Internet from simulated UEs. The simulation of UEs is performed by another virtual machine that runs a custom software that performs the registration and PDU session establishment for a new UE and then captures IP traffic from user applications (ping, wget, etc.) in a third machine. It encapsulates it in the GTP tunnel to send the traffic to the UPF. The scenario is basically the one seen in the diagram that I am attaching.

The deployment is currently working properly for ICMP or UDP traffic, but we are experiencing issues with the NAT MASQUERADE rule when sending TCP traffic to the Internet. It seems that most of the TCP packets are not matching the rule for some reason, and they are sent to the gateway router with the original IP from the application generating the traffic, and consequently are not being routed properly. Nevertheless, some of the packets (usually, retransmissions from the original packet) are actually being translated, routed, and answered by the remote Internet server. I am also attaching a tcpdump capture on the core machine interface connected to the gateway. Packets with 192.168.100.1 as source IP are not being translated while packets with source IP 192.168.1.189 are using the NAT rule. The capture is a wget to google.com.

We have previously tested this exact same scenario in a Virtualbox environment, without experiencing these issues with TCP traffic.

Would anyone have an idea why this could be happening?

tcp-wget-google.pcap (81.4 KB)

openstack-scenario
openstack-scenario3336×2035 477 KB

UE not able to reach internet in UERANSIM
UPF not forwarding packets received in GTP tunnel with UERANSIM
#2

Hi @driveraupm,
have you checked the setting of the floating IP to the instance?

#3

Hi,

Is your User App and Connector (GTP-U encapsulator) on separate VMs or on the same VM ?

If they are on the same VM, what happens is that the user app’s packet gets encapsulated inside the GTP-U tunnel and the checksum for user-app packet is not calculated yet (because checksum is typically offloaded to the NIC driver/card). When this packet comes out of GTP-U tunnel at UPF, the packet does not hit the MASQUERADE rule (because its checksum is wrong) and hence goes out without NAT happenning. Solution is to turn off the checksum offloading using ‘ethtool’ command.

Regds…

#4

Hi! thank you both for your responses. The connector is running on a separate VM but I will look into the checksum calculation anyway, as it sounds like something that could be happening (maybe packets are not being properly encapsulated in the connector or something like this). I Will let you know if this fixes the issue or at least gives us a clue.

#5

We tested it and definitely the problem is related with the checksum validation in the core VM. We have now turned off the checksum in the packets generated in the user app VM and the NAT started working. I still don’t know why we weren’t experiencing this issue in Virtualbox, but in any case, this has helped us a lot, thank you!

#6