N3IWF config when communicate with the distributed AMF/UPF

peggie · November 17, 2020, 1:58am

Hi free5gc team,
I want to make N3IWF in free5gc talk with my own AMF and UPF, AMF and UPF are deployed in the same machine, but their N2 of AMF and N3/N4 of UPF are bind to different INCs port. N3IWF’s N3 and N2 IP will be bind to one different NIC, we use the external switch to make them talk with each other. now, my SMF can talk with my UPF, but foe N3IWF, how to change the config in n3iwf.conf to make it can connect with AMF and UPF, I see that

AMFSCTPAddresses:
    - IP:
      - 192.179.120.191    // here is the my own AMF N2 ip

 GTPBindAddress:
    192.179.120.180     //here is my own upf N3 ip.

I config one ip 192.179.120.170 on interface enp24s0f0, like below:

<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b4:96:91:0d:eb:a8 brd ff:ff:ff:ff:ff:ff
    inet **192.179.120.170/23** brd 192.179.121.255 scope global enp24s0f0

So my question is where I can bind this ip and interface to N3IWF N2/N3 interface ip? since currently when I run N3IWF Individually, there is the error like this:

2020-11-17T05:49:43-05:00 [WARN][N3IWF][Context] No certificate file path specified, load default certificate...
2020-11-17T05:49:43-05:00 [INFO][N3IWF][NGAP] [SCTP] AMF SCTP address: %+v192.179.120.191:38412
2020-11-17T05:49:43-05:00 [INFO][N3IWF][NGAP] [SCTP] N3IWF SCTP address: %+v:0
2020-11-17T05:49:43-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T05:49:43-05:00 [INFO][N3IWF][NGAP] Retry to connect AMF after 1 second...
2020-11-17T05:49:44-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T05:49:44-05:00 [INFO][N3IWF][NGAP] Retry to connect AMF after 1 second...
2020-11-17T05:49:45-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T05:49:45-05:00 [ERRO][N3IWF][NGAP] Failed to connect to AMF.
2020-11-17T05:49:45-05:00 [ERRO][N3IWF][Init] Start NGAP service failed: NGAP service

Thanks very much!
looking forward your reply!

free5GC · November 17, 2020, 2:33am

2020-11-17T05:49:43-05:00 [INFO][N3IWF][NGAP] [SCTP] N3IWF SCTP address: %+v:0

It seems that IP and port didn’t have been parse successfully. Can you provide your config file of N3IWF?

free5GC · November 17, 2020, 6:02am

2020-11-17T05:49:43-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused

Is there any firewall on AMF? Can you try to log the pcap file on AMF.

peggie · November 17, 2020, 3:26am

YES,I think so, I think the SCTP IP and port should be input from the conf file, but how to do, and the source code also need change maybe, I check the code in src/n3iwf/ngap/service/service.go:
will read the conf file to get the ip and port, then setup the sctp connection.
func Run() error {
// n3iwf context
n3iwfSelf := context.N3IWFSelf()
// load amf SCTP address slice
amfSCTPAddresses := n3iwfSelf.AMFSCTPAddresses
localAddr := new(sctp.SCTPAddr) // here the localAddr will be the N3IWF sctp addr? I think it’s the null. am I right??

n3iwfcfg.conf (1.3 KB)

free5GC · November 17, 2020, 5:23am

You seem to have modified the code, can you change back to the original code and set the log level to debug

on free5GC.conf

  N3IWF:
    debugLevel: debug # <- change it to debug or trace
    ReportCaller: false

Then can you provide the log to us again.

free5GC · November 17, 2020, 5:26am

Also, we found that you have set the GTPBindAddress as your UPF’s IP. But it should be the GTP Binding Address that your N3IWF wanted to bind on (N3IWF N3 IP).

peggie · November 17, 2020, 5:49am

YES, you are right, I update the GTPBindAddress to be the n3iwf N3 ip now. and run ./bin/n3iwf, then the debug log is like below:

root@ubuntu:/home/test/free5gc# ./bin/n3iwf
2020-11-17T09:38:47-05:00 [INFO][N3IWF][App] n3iwf
2020-11-17T09:38:47-05:00 [INFO][N3IWF][App] N3IWF version:
Not specify ldflags (which link version) during go build
go version: go1.14.4 linux/amd64
CommonConfig file: config/free5GC.conf
2020-11-17T09:38:47-05:00 [INFO][NAS][Message] set log level : info
2020-11-17T09:38:47-05:00 [INFO][NAS][Message] set report call : false
2020-11-17T09:38:47-05:00 [INFO][LIB][FSM] set log level : info
2020-11-17T09:38:47-05:00 [INFO][LIB][FSM] set report call : false
2020-11-17T09:38:47-05:00 [INFO][LIB][NGAP] set log level : info
2020-11-17T09:38:47-05:00 [INFO][LIB][NGAP] set report call : false
2020-11-17T09:38:47-05:00 [INFO][OAPI][NamfComm] set log level : info
2020-11-17T09:38:47-05:00 [INFO][OAPI][NamfComm] set report call : false
2020-11-17T09:38:47-05:00 [INFO][OAPI][NamfEvent] set log level : info
2020-11-17T09:38:47-05:00 [INFO][OAPI][NamfEvent] set report call : false
2020-11-17T09:38:47-05:00 [INFO][OAPI][NsmfPDUSess] set log level : info
2020-11-17T09:38:47-05:00 [INFO][OAPI][NsmfPDUSess] set report call : false
2020-11-17T09:38:47-05:00 [INFO][OAPI][NudrDataRepo] set log level : info
2020-11-17T09:38:47-05:00 [INFO][OAPI][NudrDataRepo] set report call : false
2020-11-17T09:38:47-05:00 [INFO][LIB][OAPI] set log level : info
2020-11-17T09:38:47-05:00 [INFO][LIB][OAPI] set report call : false
2020-11-17T09:38:47-05:00 [INFO][LIB][Aper] set log level : info
2020-11-17T09:38:47-05:00 [INFO][LIB][Aper] set report call : false
2020-11-17T09:38:47-05:00 [INFO][CommonTest][Comm] set log level : info
2020-11-17T09:38:47-05:00 [INFO][CommonTest][Comm] set report call : false
2020-11-17T09:38:47-05:00 [INFO][N3IWF][Init] Successfully initialize configuration config/n3iwfcfg.conf
2020-11-17T09:38:47-05:00 [INFO][N3IWF][Init] Log level is set to [debug] level
2020-11-17T09:38:47-05:00 [INFO][N3IWF][Init] Server started
2020-11-17T09:38:47-05:00 [WARN][N3IWF][Context] No private key file path specified, load default key file…
2020-11-17T09:38:47-05:00 [WARN][N3IWF][Context] Parse PKCS8 private key failed: x509: failed to parse private key (use ParsePKCS1PrivateKey instead for this key format)
2020-11-17T09:38:47-05:00 [INFO][N3IWF][Context] Parse using PKCS1…
2020-11-17T09:38:47-05:00 [WARN][N3IWF][Context] No certificate authority file path specified, load default CA certificate…
2020-11-17T09:38:47-05:00 [WARN][N3IWF][Context] No certificate file path specified, load default certificate…
2020-11-17T09:38:47-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T09:38:47-05:00 [INFO][N3IWF][NGAP] Retry to connect AMF after 1 second…
2020-11-17T09:38:48-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T09:38:48-05:00 [INFO][N3IWF][NGAP] Retry to connect AMF after 1 second…
2020-11-17T09:38:49-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T09:38:49-05:00 [DEBU][N3IWF][NGAP] [SCTP] AMF SCTP address: 192.179.120.191:38412
2020-11-17T09:38:49-05:00 [ERRO][N3IWF][NGAP] Failed to connect to AMF.
2020-11-17T09:38:49-05:00 [ERRO][N3IWF][Init] Start NGAP service failed: NGAP service run failed
n3iwfcfg.conf (1.4 KB)

free5GC · November 17, 2020, 5:53am

Hi,

Can you set the log level of ngap to debug and provide the log again.
And save the pcap file on n3iwf host to let us check the connection.

Thanks

peggie · November 17, 2020, 5:55am

Currently my question is where can I config the n3iwf N2 interface ip and bind it in config file, now, when setup the sctp connection, only AMF SCTP ip and port, no n3iwf’s, do you understand me?
in config/n3iwfcfg.conf, there is the string AMFSCTPAddresses:
- IP:
- 192.179.120.191
to indicate the amf ip, and in the code service.go, will exract it from conf, here:
func Run() error {
// n3iwf context
n3iwfSelf := context.N3IWFSelf()
// load amf SCTP address slice
amfSCTPAddresses := n3iwfSelf.AMFSCTPAddresses
localAddr := new(sctp.SCTPAddr) // here is the ip of n3iwf, but it’s null,

free5GC · November 17, 2020, 5:59am

Now N3IWF’s N2 interface is listen on 0.0.0.0 and have no config option in config file. That’s correct if it prints as null.

peggie · November 17, 2020, 6:07am

I set the log level to be debug, looks like no more information printed out, could you please help to check the ip config in my machine, i config the AMF n2 interface ip and the N3IWF N2 N3 interface ip in to one pysical int, like

eno2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a4:bf:01:72:e4:18 brd ff:ff:ff:ff:ff:ff
    inet 192.179.120.191/23 brd 192.179.121.255 scope global eno2
       valid_lft forever preferred_lft forever
    inet 192.179.120.1/24 scope global eno2
       valid_lft forever preferred_lft forever
    inet 192.179.120.190/23 brd 192.179.121.255 scope global secondary eno2
       valid_lft forever preferred_lft forever
    inet 192.179.120.73/23 brd 192.179.121.255 scope global secondary eno2
       valid_lft forever preferred_lft forever
    inet 192.179.120.97/23 brd 192.179.121.255 scope global secondary eno2
       valid_lft forever preferred_lft forever
    inet 192.179.120.2/24 scope global secondary eno2
       valid_lft forever preferred_lft forever
    inet6 fe80::a6bf:1ff:fe72:e418/64 scope link
       valid_lft forever preferred_lft forever

the pcap file is attached, but the sctp message is from 192.179.120.191 to 192.179.120.191, strange, why?
log.txt (3.2 KB) test.pcap (9.8 KB)

free5GC · November 17, 2020, 8:15am

Hi,

Because N3IWF’s ngap is binding on 0.0.0.0, the host will auto pick one IP on host and use it. As the pcap shows, your N3IWF’s ngap are binding on 192.179.120.191:32874.

If you want to change the binding IP of N3IWF’s ngap, you can change the IPAddrs of localAddr

github.com

free5gc/n3iwf/blob/1ccc4178cbd142acf8733bb741e9a58399325c01/ngap/service/service.go#L32


	ngapLog = logger.NgapLog
}

// Run start the N3IWF SCTP process.
func Run() error {
	// n3iwf context
	n3iwfSelf := context.N3IWFSelf()
	// load amf SCTP address slice
	amfSCTPAddresses := n3iwfSelf.AMFSCTPAddresses

	localAddr := new(sctp.SCTPAddr)

	for _, remoteAddr := range amfSCTPAddresses {
		errChan := make(chan error)
		go listenAndServe(localAddr, remoteAddr, errChan)
		if err, ok := <-errChan; ok {
			ngapLog.Errorln(err)
			return errors.New("NGAP service run failed")
		}
	}

As the pcap file shows, the AMF have return ABORT back to N3IWF, can you check the log on AMF why it ABORT N3IWF?

peggie · November 17, 2020, 6:17am

the amf and n3iwf are installed in one machine, i capture the pkt using tcpdump -i any, so I think the pkt in amf are also included the pcap file i sent to you.

peggie · November 17, 2020, 8:22am

OK, clear, now I update your code to let n3iwf bind one ip in conf file:
I config it like this:
LocalSCTPAddress:
IP:
- 192.179.120.1
port:
2000
and add some code in n3iwf module to parse it, now the log like this:
2020-11-17T12:10:39-05:00 [INFO][N3IWF][NGAP] [SCTP] AMF SCTP address: %+v192.179.120.191:38412
2020-11-17T12:10:39-05:00 [INFO][N3IWF][NGAP] [SCTP] N3IWF SCTP address: %+v192.179.120.1:2000
2020-11-17T12:10:39-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T12:10:39-05:00 [INFO][N3IWF][NGAP] Retry to connect AMF after 1 second…
2020-11-17T12:10:40-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T12:10:40-05:00 [INFO][N3IWF][NGAP] Retry to connect AMF after 1 second…
2020-11-17T12:10:41-05:00 [ERRO][N3IWF][NGAP] [SCTP] DialSCTP(): connection refused
2020-11-17T12:10:41-05:00 [DEBU][N3IWF][NGAP] [SCTP] AMF SCTP address: 192.179.120.191:38412
2020-11-17T12:10:41-05:00 [ERRO][N3IWF][NGAP] Failed to connect to AMF.
2020-11-17T12:10:41-05:00 [ERRO][N3IWF][Init] Start NGAP service failed: NGAP service run failed

but still failed in sctp connection.

free5GC · November 17, 2020, 8:36am

The AMF has refuse N3IWF’s SCTP connection.

You should check the error log on AMF to see why it ABORT n3iwf’s sctp init.

calee · November 25, 2020, 6:57am

Hi @peggie ,

Can I ask that how do you overcome the SCTP ABORT issue?

peggie · November 18, 2020, 8:07am

does free5gc use the kernel sctp module? i see that go pkg like DialSCTP when setup the sctp connection with amf. now i deploy the free5gc and amf in different machine, sctp can setup normally, but looks like that the DialSCTP not return, it hanging at the place of conn, err = sctp.DialSCTP(“sctp”, localAddr, remoteAddr), do you know why???

func listenAndServe(localAddr *sctp.SCTPAddr, remoteAddr *sctp.SCTPAddr, errChan chan<- error) {
    var conn *sctp.SCTPConn
    var err error

    // Connect the session
    for i := 0; i < 3; i++ {
        conn, err = sctp.DialSCTP("sctp", localAddr, remoteAddr)
        if err != nil {
            ngapLog.Errorf("[SCTP] DialSCTP(): %+v", err)
        } else {
            break
        }
    }
}

free5GC · November 18, 2020, 2:56pm

The SCTP is base on this repo.

https://git.cs.nctu.edu.tw/calee/sctp

And base on our understanding, this sctp will call linux’s sctp to send the packet.

peggie · November 23, 2020, 4:27am

Thank you for the nice feedback.
now I use the UE simulator and N3IWF in free5gc to connect to our own core, now there is no authentication and security procedure in registration procedure. When UE registration request to AMF, AMF we give the registration accept. I use the non3gpp_test.go to simulate the UE, but when received the downlink NAS transport with registration accept message from AMF, the case is crashed like this:
I think there are many steps in non3gpp_test.go to handle the Authentication and Security messages, so for my case, I need remove them, so could you please tell me which part I need remove?

2020-11-23T18:51:34Z [INFO][N3IWF][NGAP] [N3IWF] Handle Downlink NAS Transport
2020-11-23T18:51:34Z [TRAC][N3IWF][NGAP] Create new logical UE-associated NG-connection
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Encoding IKE payloads
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][EAP] marshal(): Start marshalling
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][EAP][Expanded] marshal(): Start marshalling
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Encoding IKE message
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE] Start encoding IKE message
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Encoding IKE payloads
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][Encrypted] marshal(): Start marshalling
2020-11-23T18:51:34Z [TRAC][N3IWF][IKE] Encoded 108 bytes
2020-11-23T18:51:34Z [TRAC][N3IWF][IKE] [IKE] Send IKE message to UE
2020-11-23T18:51:34Z [TRAC][N3IWF][IKE] [IKE] Encoding...
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Encoding IKE message
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE] Start encoding IKE message
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Encoding IKE payloads
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][Encrypted] marshal(): Start marshalling
2020-11-23T18:51:34Z [TRAC][N3IWF][IKE] Encoded 108 bytes
2020-11-23T18:51:34Z [TRAC][N3IWF][IKE] [IKE] Sending...
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Decoding IKE message
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Decoding IKE payloads
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][Encrypted] unmarshal(): Start unmarshalling received bytes
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Encoding IKE message
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE] Start encoding IKE message
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Encoding IKE payloads
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][Encrypted] marshal(): Start marshalling
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] Decoding IKE payloads
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][EAP] unmarshal(): Start unmarshalling received bytes
2020-11-23T18:51:34Z [INFO][N3IWF][IKE] [IKE][EAP][Expanded] unmarshal(): Start unmarshalling received bytes
--- FAIL: TestNon3GPPUE (0.26s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xcd4797]

goroutine 107 [running]:
testing.tRunner.func1.1(0x12a8620, 0x20f4ad0)
        /usr/local/go/src/testing/testing.go:940 +0x2f5
testing.tRunner.func1(0xc0003447e0)
        /usr/local/go/src/testing/testing.go:943 +0x3f9
panic(0x12a8620, 0x20f4ad0)
        /usr/local/go/src/runtime/panic.go:969 +0x166
free5gc/src/test.TestNon3GPPUE(0xc0003447e0)
        /root/free5gc/src/test/non3gpp_test.go:775 +0x2a57
testing.tRunner(0xc0003447e0, 0x14da478)
        /usr/local/go/src/testing/testing.go:991 +0xdc
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:1042 +0x357
exit status 2

dayal007 · May 19, 2021, 10:14am

@peggie What UE did you use to communicate to the n3iwf?