Bug 54 - pasta significantly underperforms slirp4netns when connecting to a genuinely remote host
Summary: pasta significantly underperforms slirp4netns when connecting to a genuinely ...
Status: RESOLVED FIXED
Alias: None
Product: passt
Classification: Unclassified
Component: general (show other bugs)
Version: unspecified
Hardware: All Linux
: Normal normal
Assignee: nobody
URL:
Depends on:
Blocks:
 
Reported: 2023-06-08 05:08 UTC by David Gibson
Modified: 2023-06-15 04:51 UTC (History)
1 user (show)

See Also:


Attachments

Description David Gibson 2023-06-08 05:08:32 UTC
When measuring TCP throughput with iperf3 from namespace to host, pasta performs much better than slirp4netns.  However, when I tried putting the iperf3 server on a truly remote machine (attached via gigabit ethernet and some switches), pasta now underperforms slirp4netns.

Steps to reproduce:
    1. Have host1 & host2 connected via gigabit physical ethernet
    2. On host2 run "iperf3 -s"
    3. On host1 run "iperf3 -c host2" as a baseline
    4. On host1 create a network namespace with "unshare -Unr", and get its $PID with "echo $$"
    5. On host1 connect the namespace to the network with pasta:
         $ pasta -d --config-net $PID
    6. In the namespace run "iperf3 -c host2"
    7. Kill pasta
    8. Now connect the namespace with slirp4netns:
        $ slirp4netns -c $PID tap0
    9. In the namepace run "iperf3 -c host2"
    10.  Compare throughput from (3), (6) and (9)

Expected results:

pasta should perform at least as well as slirp4netns, and given that gigabit ethernet is fairly slow by modern standards, it should be quite close to the direct host to host throughput.

Actual results:

Direct host to host,

$ iperf3 -c 192.168.17.5
Connecting to host 192.168.17.5, port 5201
[  5] local 192.168.17.26 port 40224 connected to 192.168.17.5 port 5201
[...]
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   409 MBytes   343 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   408 MBytes   342 Mbits/sec                  receiver

[This seems a bit slow for gigabit ethernet, but there are a couple of switches and a USB3 NIC in there.  Probably doesn't matter where the bottleneck is for the purpose of this bug]

With pasta,

$ pasta --config-net -- iperf3 -c 192.168.17.5
Connecting to host 192.168.17.5, port 5201
[  5] local 192.168.17.26 port 48636 connected to 192.168.17.5 port 5201
[...]
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   110 MBytes  91.9 Mbits/sec   19             sender
[  5]   0.00-10.00  sec   109 MBytes  91.6 Mbits/sec                  receiver

With slirp4netns,

$ unshare -Unr
# echo $$
2037185

[on the host] $ slirp4netns -c 2034858 tap0

[back in the ns]
# iperf3 -c 192.168.17.5
Connecting to host 192.168.17.5, port 5201
[  5] local 10.0.2.100 port 37844 connected to 192.168.17.5 port 5201
[...]
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   410 MBytes   344 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   409 MBytes   343 Mbits/sec                  receiver


To summarize, slirp4netns gets near-host throughput, ~345Mbps, whereas pasta only gets a third of that ~90Mbps.

Additional info:

* For comparison if the iperf3 server is on the same host as the namespace (but using a different IP address), then host to host direct gets ~65Gbps, slirp4netns gets ~1.5Gbps and pasta gets ~38Gbps.

* If I repeat the pasta tests with -d, the numbers don't change much, but I get continuous
debug messages from pasta:

43.9226: TCP: index 1: ACK_TO_TAP_DUE
43.9226: TCP: index 1, timer expires in 0.010s
43.9327: TCP: index 1: ACK_TO_TAP_DUE dropped
43.9327: TCP: index 1, timer expires in 7200.000s

These don't occur when the iperf3 server is on the pasta host.
Comment 1 Stefano Brivio 2023-06-08 06:44:47 UTC
See also https://bugs.passt.top/show_bug.cgi?id=44 -- but I probably missed some case with those calculations.

What are the RTTs and MSSes involved? We seem to have some extreme bandwidth * delay products here, because of how much this affects libslirp throughput (64 KiB window, which makes me think of ~2ms RTT if you get ~300mbps with, again my estimate, 1460 bytes MSS).

By the way, I don't think the summary of this issue is true in general, I use pasta to connect namespaces to different remote hosts (from same data centre with microseconds RTT up to 200ms "far") and I can happily fill up gigabit links. That's not the case with the 64 KiB of libslirp.

If you set ACK_INTERVAL in tcp.c to 2 (ms) I would expect that pasta will also fill up your USB sort-of-gigabit link, but that's probably low enough as to warrant one of the possible adaptive approaches I mentioned in commit 1ee2f7cada9e ("tcp: Don't reset ACK_TO_TAP_DUE on any ACK, reschedule timer as needed").
Comment 2 David Gibson 2023-06-15 04:51:25 UTC
So, I belatedly realized that I was using the Fedora packaged pasta, rather than the git version when I tested this.  The problem appears to be fixed already with current git.  So I guess this could just be a dupe of bug 44.

In any case, closing.

Note You need to log in before you can comment on or make changes to this bug.