The problem of gtp5g?

Q/A
1

Hello @free5gc team or whomever concerned,

I am using free5gc-compose.https://github.com/free5gc/free5gc-compose
The version of free5gc is v3.0.6 and the version of gtp5g is v0.4.0.
On the performance measurement test , when I placed the load on host PC or generate a lot of traffic, the host pc often goes down.

As you can see in the kernel log I attached below, there are some errors related to gtp5g kernel module.

log.txt (4.2 KB)

Do you have any information about this issue?
It will be helpful if you have any solutions or workaround for this problem.

Thank you for your help.

4

@NK636,

Could you please provide the complete kernel log? I have to check what happened at the particular time

11

Hello @muthuramanecs03g,

Thank you for your help.
I’m preparing to send you the complete kernel log.
I will do it as soon as possible.
I’m sorry for the delay.

2

Check out this: https://github.com/PrinzOwO/gtp5g/issues/7

Bests,

3

FYI

You can post your bug or report on https://github.com/free5gc/gtp5g

5

Hello @abousselmi,

Thank you for your reply.

I checked the issue which you mentioned. I’m afraid but I think it may be not same issue.
This issue said that “The kernel is crashed by the zero-divide.”, but this time, it’s probably in kfree_skb.

I attached additional error log which showed before log.txt.

[ 1865.356613] perf: interrupt took too long (2553 > 2500), lowering kernel.perf_event_max_sample_rate to 78250
[ 1867.605264] upfgtp:[gtp5g] ip_xmit: Failed to send skb to ip layer
[ 1867.605265] upfgtp:[gtp5g] gtp5g_fwd_skb_encap: Failed to transmit skb through ip_xmit
[ 1867.605266] upfgtp:[gtp5g] gtp5g_encap_recv: GTP packet has been dropped
[ 1867.605270] ------------[ cut here ]------------
[ 1867.605270] kernel BUG at /build/linux-hwe-zHO4ZF/linux-hwe-5.0.0/mm/slub.c:305!
[ 1867.605292] invalid opcode: 0000 [#1] SMP NOPTI
[ 1867.605304] CPU: 11 PID: 3504 Comm: handler40 Kdump: loaded Tainted: G OE 5.0.0-23-generic #24~18.04.1-Ubuntu
[ 1867.605341] Hardware name: HP HP EliteDesk 800 G5 TWR/8591, BIOS R01 Ver. 02.04.02 12/27/2019
[ 1867.605373] RIP: 0010:kfree+0x16e/0x180
[ 1867.605390] Code: 74 05 41 0f b6 72 51 4c 89 d7 e8 fd cd f8 ff eb 81 41 b8 01 00 00 00 48 89 d9 48 89 da 4c 89 d6 e8 17 f6 ff ff e9 68 ff ff ff <0f> 0b 48 8b 3d 09 88 39 01 e9 c5 fe ff ff 0f 1f 40 00 0f 1f 44 00
[ 1867.605432] RSP: 0018:ffff9cdd9f4c3b38 EFLAGS: 00010246
[ 1867.605445] RAX: ffff9cdc7e2c8800 RBX: ffff9cdc7e2c8800 RCX: ffff9cdc7e2c8800
[ 1867.605460] RDX: 00000000000001dd RSI: ffff9cdd9f4e7180 RDI: ffff9cdd9ec06d00
[ 1867.605476] RBP: ffff9cdd9f4c3b50 R08: 000000000000040c R09: ffffffff834a328e
[ 1867.605492] R10: fffffac70bf8b200 R11: 0000000000000001 R12: ffff9cdc7e2c8ef0
[ 1867.605516] R13: ffffffff834a328e R14: ffff9cdc7e2c8ec0 R15: ffff9cdd9288fc00
[ 1867.605540] FS: 00007fcf477fe700(0000) GS:ffff9cdd9f4c0000(0000) knlGS:0000000000000000
[ 1867.605568] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1867.605593] CR2: 00007fc78a13a400 CR3: 00000002d2126001 CR4: 00000000003626e0
[ 1867.605621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1867.605647] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

If you have any idea to solve this problem or if it has already solved, could you tell me?

I really appreciate your help.

7

Which kernel version do you use? Please use uname -a to check it.

8

Hello @free5GC,

Thank you for your reply.
I’m using the following kernel version.

$uname -a
Linux rd5gc2 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:12:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

9

Which version of gtp5g you are using ?

10

Hi @abousselmi,

I’m using gtp5g v0.4.0.
As a supplement information, I tried to use gtp5g v0.4.1, but upf couldn’t transport packet normally when I used. So I didn’t upgrade it.

12

Hi @NK636,

It looks like ip_local_out is failing. Try to increase your system’s udp buffer size and redo the test.

13

Hi @abousselmi

Thank you for your advice. I tried to increase the udp buffer size from 212992 as default to 26214400.

Before changing, when I executed the following command from UE without executing iperf -s -u on the server, the kernel crash was happened as soon as it was executed.
iperf -c 10.100.203.22 -u -B 60.60.0.1 -l 1470 -b 1000M

After changing the udp buffer, it didn’t be happened immediately. (I will try whether it is happend when I send larger amount of traffic.)
Thank you for your help.

I have a question about this solution.
Could you tell me why you find the failure of ip_local_out, and why this failure cause the kernel crash?

I think changing the udp buffer might be good way to avoid the kernel crash, but I wonder that it is not a fundamental solution.
So I would like to report this for gtp5g issue.

I appreciate for your help.

14

Hi @NK636

Actually the first line of the error message is just a warning. The second line is the beginning of the actual error, and it says: ip_xmit: Failed to send skb to ip layer which is triggered when ip_local_out(dev_net(gtp_dev), sk, skb) < 0 is true. ip_local_out is a function of the kernel networking code ip_output.c - net/ipv4/ip_output.c - Linux source code (v5.15.11) - Bootlin that handles sockets and socket buffers to send traffic out. This is when one might think that probably we may have a buffering issue. You can take a look to this UDP Tuning to maximise performance – The Linux Cluster which describes some other things that you can do to improve the performance of your system.
Bests,

15

Hello @abousselmi,

Thank you for your explanation.
I can understand the reason why you thought that this is buffering issue.

Thank you again for your help.

6

Hello @muthuramanecs03g,

Thank you for your help.
I will try to post the issue on there.