Getting wrong iperf3 results when using pasta and podman 4.4.2 [core@ip-10-1-36-63 ~]$ pasta --version pasta 0^20230310.g70c0765-1.fc37.x86_64 [core@ip-10-1-36-63 ~]$ podman run --rm -it --network pasta alpine / # apk add iperf3 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz (1/1) Installing iperf3 (3.12-r0) Executing busybox-1.35.0-r29.trigger OK: 7 MiB in 16 packages / # iperf3 -c 88.86.XX.YY -p 5203 Connecting to host 88.86.XX.YY, port 5203 [ 5] local 10.1.36.63 port 40416 connected to 88.86.XX.YY port 5203 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 656 KBytes 5.37 Mbits/sec 3 8.00 KBytes [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 3 8.00 KBytes [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 3 8.00 KBytes [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 0 8.00 KBytes [ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 3 8.00 KBytes [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.00 KBytes [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 0 8.00 KBytes [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 8.00 KBytes [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 8.00 KBytes [ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 3 8.00 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 656 KBytes 537 Kbits/sec 15 sender [ 5] 0.00-10.11 sec 140 KBytes 113 Kbits/sec receiver iperf Done.
Created attachment 6 [details] Packet capture shared by reporter
I could reproduce something similar with the public iperf3 server nyfiosspeed4.west.verizon.net, also with slirp4netns, but not with any other public iperf3 server I tried (mostly located in Europe) -- there I can just fill the available uplink. With nyfiosspeed4.west.verizon.net, a RST segment from the remote side closes the data connection unexpectedly, the control connection terminates properly, so the results are actually accurate -- what I can't explain is why the iperf3 server decides to terminate the data connection after, at least in my case, 192 test packets. I'm now comparing captures taken on the host in the two cases. I couldn't spot any obvious difference so far.
Created attachment 8 [details] pasta capture of two-seconds iperf3 test to Verizon public server After frame 50, the client would expect an ACK, but we send a FIN, ACK more than a second later.
Created attachment 9 [details] ...and matching capture outside namespace Here, frame 190 would allow us to ACK frame 50 from the namespace capture: its relative sequence is 163873. We don't, the client holds until frame 191, and because of this gap the server (probably configured with --rcv-timeout < 1000 [ms] ?) closes the connection. I'm now looking for the reason why we are holding that ACK off for so long. All the other ones seems to come in due time.
Created attachment 10 [details] Don't reset the ACK_TO_TAP_DUE flag on any ACK we send I have a fix I still need to evaluate and test properly, but it solves the issue for me: $ ./pasta --config-net # iperf3 -c nyfiosspeed4.west.verizon.net -w 16M Connecting to host nyfiosspeed4.west.verizon.net, port 5201 [ 5] local 88.198.0.164 port 47858 connected to 206.124.86.196 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 32.5 MBytes 273 Mbits/sec 0 20.9 MBytes [ 5] 1.00-2.00 sec 110 MBytes 922 Mbits/sec 0 46.2 MBytes [ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 46.2 MBytes [ 5] 3.00-4.00 sec 114 MBytes 954 Mbits/sec 0 46.2 MBytes [ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 46.2 MBytes [ 5] 5.00-6.00 sec 114 MBytes 954 Mbits/sec 0 46.2 MBytes [ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 46.2 MBytes [ 5] 7.00-8.00 sec 114 MBytes 954 Mbits/sec 0 46.2 MBytes [ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 46.2 MBytes [ 5] 9.00-10.00 sec 114 MBytes 954 Mbits/sec 0 46.2 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.02 GBytes 879 Mbits/sec 0 sender [ 5] 0.00-10.10 sec 1.01 GBytes 862 Mbits/sec receiver iperf Done. ...which is limited by my uplink and the same behaviour as I observe directly from the same host.
Created attachment 11 [details] Static x86_64 RPM build with fix from #c5 ...as well as a statically linked RPM build including the fix, if you want to try it out. Of course, this binary is not signed in any way, so don't run in production or confidential settings -- ideally just in a throw-away, contained environment.
Fixed in 2023_03_21.1ee2f7c, Fedora 37 build: https://bodhi.fedoraproject.org/updates/FEDORA-2023-0972b1a0a9 (passt-0^20230321.g1ee2f7c-1.fc37). https://passt.top/passt/commit/?id=1ee2f7cada9e6f739a00d39bb9821f1ce3493d92 tcp: Don't reset ACK_TO_TAP_DUE on any ACK, reschedule timer as needed