Running TNGFUE on another subnet

Q/A
#1

Hello,

I’ve installed free5GC v3.4.3 on a machine in a LAN (IP: 10.x.x.x). Then I’ve followed the instructions to setup TNGF on free5GC (changing the parameters IPSecInterfaceAddress, IPSecTunnelAddress and UEIPAddressRange to another IP range to avoid conflicts) and to setup the Wi-Fi AP (which is in the same LAN as free5GC). According to the instructions, you have to enable the DHCP server on OpenWrt. I did that, and another subnet was created (IP: 192.168.1.x).

That said, on the step “Set wifi interface IP to the same LAN with free5GC”, that currently displays:

ip a add 192.168.1.202/24 brd 192.168.1.255 dev <YOUR_WIFI_INTERFACE_NAME>
ip r add default via 192.168.1.202 dev <YOUR_WIFI_INTERFACE_NAME>

Why do I have to set a fixed IP address (first line) on the Wi-Fi interface (given that we have DHCP enabled on the AP)?

And which IP is the gateway (second line)? Shouldn’t it be 192.168.1.1 (AP’s IP)?

I tried to set the route to 192.168.1.1, however TNGFUE fails to connect with the message “the mac is not equal dude…”, the virtual network interfaces are created however it’s not possible to get the replies back on ping. I have double checked that the data on webconsole matches the configuration files (see attachments)

So, the network configuration currently is:

free5GC (10.0.0.110) <-> (10.0.0.1) Switch (10.0.0.1) <-> (10.0.1.12) Wi-Fi AP (192.168.1.1) <-> (192.168.1.202) TNGFUE

PS: The UE is on a laptop machine. All OSes used are Ubuntu 20.04.6 LTS

TNGFUE-attachments.zip (342.3 KB)

Regards,
Leo.

#2

  1. You should set a IP address for your TNGFUE wifi interface, and all the packets will be sent through this interface. That is the purpose of this command: ip r add default via 192.168.1.202 dev <YOUR_WIFI_INTERFACE_NAME>. The interface will send packets directly to its gateway.

  2. The issue of “the mac is not equal dude…” means that 5g-nas packet was modified in some way, causing it to fail the security integrity check. You can capture and compare the packets from both free5gc and tngfue to analyze 5g-nas packets.

You can add -dd option when executing tngfue, i.g. sudo ./wpa_supplicant -c ../wpa_supplicant.conf -i <YOUR_WIFI_INTERFACE_NAME> -dd , to get more debug information. By the way, check your wifi status is open but disconnected to any AP before you start TNGFUE.

#3

Hello @blackcat-118,

First of all, thanks for the prompt reply.

I think now I get it. After putting the IP addresses as the instructions suggest (screenshot on attached file), the message regarding mac disappeared. Anyway, I’m still curious about why do we have to enable DHCP on the AP if we set the WiFi device IP manually.

So, even after correcting the route, TNGFUE doesn’t work. The logs from free5GC and TNGFUE (now in debug mode) are attached to this message. From the logs, it’s possible to see that the UE completed the authentication successfully (which means the parameters match on both sides):

EAPOL: External notification - portValid=1
EAPOL: SUPP_PAE entering state AUTHENTICATED
EAPOL: Supplicant port status: Authorized
nl80211: Set supplicant port authorized for [REDACTED-MAC-ADDR]
EAPOL authentication completed - result=SUCCESS

However it’s stuck on the message

Try to receive packet in IKE

And if I try to send SIGTERM, the output is:

eloop: could not process SIGINT or SIGTERM in two seconds. Looks like there
is a bug that ends up in a busy loop that prevents clean shutdown.
Killing program forcefully.

On free5GC I didn’t find any errors that might interfere with TNGFUE’s setup, except for a warning regarding a timer that timeouts during authentication.

Please, what else can I test to help debug this?

TIA,
Leo.

TNGFUE-attachments2.zip (222.0 KB)

#4

Hi,

I tried disabling DHCP and it worked well. I will schedule a time to update the setup instruction on the official site. Thank you!

For your connection issue, I notice the error message in your free5gc log showing registration accept message timeout. I’ve encountered this issue before, but it can be caused by various factor, most commonly due to the communication problem in the direction from TNGFUE to free5gc.

Because I only tested tngfue in the same LAN of free5gc. The setup is TNGFUE: 192.168.1.202, free5gc: 192.168.1.10x, AP: 192.168.1.1. I believe one possible reason for the failure is that your TNGFUE is on a different LAN than free5GC. If you want to run TNGFUE on a different subnet, some additional configuration may be required.

I can help you to find the problem if you provide the pcap files captured from both free5gc and TNGFUE. You can use tcpdump or wireshark to captured them.

#6

Hello,

A quick update on the situation.

As I’ve mentioned, I rebuilt my testing environment from scratch. Now I could successfully reproduce the results from the TNGF docs page using the setup you’ve suggested on the quoted message (free5GC and TNGFUE on the same LAN).

In the process, I could improve the proposed run script (see new commits on the TNGFUE repo PR) and the docs page.

By the way, what do you think of adding a new section to the tested hardware page listing the Wi-Fi APs used to test TNGF/TNGFUE?

The next step is to try to figure it out what is happening when I install free5GC on another subnet. I’m planning to make this setup again, capture the packets and update this forum post next week.

Kind regards,
Leo.

#7

Hi @oliveiraleo,

That helps us (especially me) sooo much. Thank you!!

I think that’s unnecessary, as it seems most AP devices can be used successfully here.

I’ve tried to deploy free5GC in a Docker private network and observed that the ARP routing issues between TNGFUE and free5GC caused connection failures. I think that can be consider as deploying across different subnets.

At the beginning of the registration step, TNGFUE will send an ARP packet to request the TNGF MAC address. You can verify this by checking the captured packets from TNGFUE.

Registration procedure: 3GPP SPEC 23.502

image
image776×477 73 KB

In my case, TNGFUE cannot get TNGF MAC address. The solution is to use arp proxy on free5GC host.

image
image845×464 115 KB

If you make any progress or encounter any issues, I’d be happy to hear about it.

Best,

#5

Hello,

I tried disabling DHCP and it worked well. I will schedule a time to update the setup instruction on the official site. Thank you!

Don’t worry, I can do it for us because I’ve already planned to update these docs (see the TODO from this issue here).

Because I only tested tngfue in the same LAN of free5gc. The setup is TNGFUE: 192.168.1.202, free5gc: 192.168.1.10x, AP: 192.168.1.1. I believe one possible reason for the failure is that your TNGFUE is on a different LAN than free5GC. If you want to run TNGFUE on a different subnet, some additional configuration may be required.

I see. I’m gonna start form scratch and report here all the steps I took on a new round of tests.

I can help you to find the problem if you provide the pcap files captured from both free5gc and TNGFUE. You can use tcpdump or wireshark to captured them.

All right, thank you for that. I’ll capture the packets and update this post on the coming days.

Regards,
Leo.

#9

Hi, @oliveiraleo,

I find that TNGFUE still not receive the response of free5GC MAC address.

image
image1918×680 323 KB

In my case, free5GC-Compose runs on a host machine, so I configured proxy ARP on the host. After that, when the host receives an ARP broadcast request, it will send a proxy reply to the ARP request.

Proxy ARP reference: link

Proxy ARP is a technique by which a proxy server on a given network answers the Address Resolution Protocol (ARP) queries for an IP address that is not on that network.

Since ARP requests use link layer, the routes you set are ineffective. Could you try configuring the proxy on the switch instead? Alternatively, handling this issue by other means would also work.

#8

Hello,

First of all, thanks for helping me out.

I tried to enable the ARP proxy as suggested but, unfortunately, that didn’t solve the problem.

I have captured packets, copied the logs and configuration files from TNGFUE and free5GC and uploaded to that Google Drive folder. In TNGFUE I’m using the values suggested on docs for sec.conf and wpa_supplicant.conf (the same I’ve used when successfully testing the setup on the same LAN).

The setup still is:

free5GC (10.0.0.110) <-> (10.0.0.1) Switch (10.0.0.1) <-> (10.0.1.10) Wi-Fi AP (192.168.1.1) <-> (192.168.1.202) TNGFUE

Extra details: free5GC is installed on a VM with a network interface in bridge mode, the switch is running freeBSD, the AP OpenWrt, and the TNGFUE is installed on a Ubuntu 20.04 bare metal laptop. I have added the routes on the switch (so it could send the packets meant from 192.168.1.x to 10.0.1.10 (a.k.a. AP’s WAN interface) instead of to its default gateway/internet) and disabled the OpenWrt firewall (which means that when I connect other device to the AP, it’s possible to successfully ping free5GC and vice versa).

If any other testing is required, please, let me know.

OK

Regards,
Leo.

#10

Hello @blackcat-118,

I could run some other tests. As suggested, the ARP proxy was configured on the switch (I’ve used choparp for that. Some extra information and an usage example can be found on that page). This wasn’t enough to get the ARP responses on TNGFUE. Then I’ve realized that I’d need to enable ARP proxying on the AP too (because the 10.x.x.x network was on its WAN port and the 192.168.1.x on the LAN/WLAN interfaces which I figure it out that they were isolated). Then ARP started to work (see screenshot below)

Screenshot: ARP OK

arp-ok-screenshot
arp-ok-screenshot962×810 168 KB

However I’d to like to add two other findings: 1- The message “the mac is not equal dude…” still appears (even when I run TNGFUE on the same LAN of free5GC and TNGFUE gets a successful connection); and 2- Even after establishing a successful connection between TNGFUE and free5GC on the different sub-networks setup data can’t be transmitted on the tunnel.

For the first point, I’ve captured the packets to analyze and then I realized that TNGFUE wasn’t capturing any NAS packets (filter used on Wireshark: nas-5gs), so I couldn’t compare them to try to debug this message. (Perhaps were they encapsulated by another protocol?)

For the second, I’ve captured the packets and could confirm that the ARP answers were reaching TNGFUE, but the ping replies were not. Perhaps I need to configure something else? Another route (or this should be automatically handled by IPsec?)?

I’ve uploaded all capture files to this Google Drive sub-folder. An explanation of the naming pattern I’ve used is below:

Namming Pattern

The files that contain “same-LAN” on their title, were captured deploying free5GC and TNGFUE on the same LAN (so Wi-Fi AP is the network gateway)

The files that contain “no-nas-pkt” on their title, were captured on the setup described on OP (2 sub-nets)

The “ping-fail” refers to the second point (data transmission fails to be transmitted on the tunnel)

BTW, I’m gonna finish working on the TNGFUE docs soon. By then I’ll make the PR ready for review.

Best Regards,
Leo.

#11

Hi, @oliveiraleo,

Let’s address the questions one by one.

  1. Running on different subnet

I’ve encountered this situation before, where the ESP packets from TNGFUE to free5GC go missing. I’m still unsure what happens between them, so I ensure the ESP packets arrive correctly by setting a forwarding rule. If you have any insights into the issue, I’d be happy to hear them.
(ESP packets is in IPsec tunnel mode)

After free5GC can receive all the ESP packets, I think the data transmission will be successful.
image
Updated:

// I’ll address the other issues later. Just wanted to share the latest discovery to help speed up our progress.

#12

Hi, @oliveiraleo,

Thank you for your contribution to TNGFUE workflow!!

I occasionally see a ‘mac not equal’ message even when the UE connection is successful. I’ll need some time to figure out what’s happening, but if you have any insights, please feel free to share so we can discuss it further.

Best regards,
blackcat

#13

Hello

Thank you for confirming this information. I’m planning to try to continue debugging this issue later and will update here if I find something interesting.

For future reference (in case someone else needs it), the setup I had imagined when deploying the TNGFUE on another LAN was related to the use case where free5GC is installed in a given environment (e.g. external/3rd party cloud or another department) and the APs and TNGFUEs are running locally (or vice-versa). However, perhaps this setup should involve a VLAN (so all the hosts involved appear to be on the same LAN and potentially the ARP requests won’t be lost - see the example in Figure 4-7 from this page here).

Ok, sure.

You are welcome :slight_smile:

Ok.

Regards,
Leo.