Bug 51 - SPICE server replies aren't relayed back to the client
Summary: SPICE server replies aren't relayed back to the client
Status: RESOLVED INVALID
Alias: None
Product: passt
Classification: Unclassified
Component: pasta (show other bugs)
Version: unspecified
Hardware: All Linux
: Normal normal
Assignee: David Gibson
URL:
Depends on:
Blocks:
 
Reported: 2023-05-26 09:19 UTC by Justin Jereza
Modified: 2023-06-01 04:08 UTC (History)
2 users (show)

See Also:


Attachments
Pasta SPICE packet capture (380 bytes, application/vnd.tcpdump.pcap)
2023-05-26 09:19 UTC, Justin Jereza
Details
Slirp4netns SPICE packet capture (12.88 KB, application/vnd.tcpdump.pcap)
2023-05-26 09:20 UTC, Justin Jereza
Details
Pasta SPICE packet capture w/ replies (694 bytes, application/vnd.tcpdump.pcap)
2023-05-26 21:41 UTC, Justin Jereza
Details
Slirp4netns SPICE packet capture w/ replies (290.58 KB, application/vnd.tcpdump.pcap)
2023-05-26 21:42 UTC, Justin Jereza
Details
Pasta SPICE container-side packet capture (888 bytes, application/vnd.tcpdump.pcap)
2023-05-26 22:22 UTC, Justin Jereza
Details

Description Justin Jereza 2023-05-26 09:19:31 UTC
Created attachment 12 [details]
Pasta SPICE packet capture

`remote-viewer` is unable to connect to the SPICE server of a QEMU container executed with `podman run --network pasta --publish "[::1]:52143:5900"` with the following message:

```
(remote-viewer:36379): GSpice-WARNING **: 09:12:41.724: incomplete link header (0/16)
```

It works as expected when `slirp4netns` is used.

Two packet captures are attached, one for when `pasta` is used where only the initial client link message is seen and another using `slirp4netns` where a connection is established.

This has only been observed with SPICE. Similar port forwarding while listening on ::1 works fine with `nc` and `nginx:alpine`.
Comment 1 Justin Jereza 2023-05-26 09:20:19 UTC
Created attachment 13 [details]
Slirp4netns SPICE packet capture
Comment 2 Justin Jereza 2023-05-26 09:36:41 UTC
Pasta was executed by podman as follows:

```
/usr/bin/pasta --config-net -t ::1/52143-52143:5900-5900 -u none -T none -U none --no-map-gw --netns /run/user/1000/netns/netns-e18209d0-8de2-ee20-3acc-2ba2194264b6
```
Comment 3 Stefano Brivio 2023-05-26 16:25:13 UTC
(In reply to Justin Jereza from comment #0)
> Two packet captures are attached, one for when `pasta` is used where only
> the initial client link message is seen and another using `slirp4netns`
> where a connection is established.

I'm confused. In the slirp4netns capture, I don't see any TCP connection being established, actually: the client sends a SYN segment, continues with an ACK, without ever receiving a SYN, ACK, then proceeds sending data. I don't see a single packet from port 52143, so I don't understand why the client would even proceed.

The capture in the pasta case is similar, but the client just stops after the third segment.

Could it be that the capture files you shared are incomplete? How did you obtain those?
Comment 4 Justin Jereza 2023-05-26 21:41:51 UTC
Created attachment 14 [details]
Pasta SPICE packet capture w/ replies
Comment 5 Justin Jereza 2023-05-26 21:42:45 UTC
Created attachment 15 [details]
Slirp4netns SPICE packet capture w/ replies
Comment 6 Justin Jereza 2023-05-26 21:53:47 UTC
(In reply to Stefano Brivio from comment #3)
> Could it be that the capture files you shared are incomplete? How did you
> obtain those?

Hello Stefano. Sorry, I didn't realize my packet filtering was too aggressive and I didn't notice that when I skimmed the output before posting them here.

I was doing `tcpdump -i lo 'dst port 52143'` and forgot that that will only capture packets sent to the server but not the replies. Somehow my brain thought that tcpdump would function like a stateful firewall and include the replies as well. ????

I've uploaded a couple of new files captured with just `tcpdump -i lo 'port 52143'` that replace the previous captures.
Comment 7 Justin Jereza 2023-05-26 22:22:10 UTC
Created attachment 16 [details]
Pasta SPICE container-side packet capture
Comment 8 Justin Jereza 2023-05-26 22:26:54 UTC
That is strange. I've uploaded a packet capture created using `--network pasta:--pcap,/tmp/spice-container-pasta.pcap` as you've suggested on IRC and it only contains IPv6 packets. None of the packets seen on the host `lo` interface seem to even make it inside.

By the way, the pasta version I'm testing this all on is 0^20230222.g4ddbcb9-2.el9_2.x86_64. I haven't had the chance to test it on the latest packages available from Fedora.
Comment 9 David Gibson 2023-06-01 02:48:38 UTC
So, the fact that nothing related to your TCP connection appears on pasta's packet dump (which is, more or less, a packet dump of the tap interface) is, strangely enough, expected.

Remember that pasta, unlike slirp doesn't NAT everything - the container gets the same IP as the host.  For most things that works more smoothly, but it does mean we have no way to address the host from the guest.  The map-gw option re-introduces limited NAT to allow that, but podman has disabled that here.  This means that there's simply no way for pasta to forward packets from ::1 -> ::1 across the tap interface: it has nothing it could use as a source address.  Hence, nothing on the tap packet capture.

We do have a hack that is supposed to handle this specific case - "spliced" connections.  These use a socket-to-socket forward between loopback sockets on the host and loopback sockets in the container.  I'm not yet sure why that's not working in this case.

Are you able to take a packet capture on 'lo' within the container?  This should show "spliced" packets.
Comment 10 Justin Jereza 2023-06-01 04:08:37 UTC
> Remember that pasta, unlike slirp doesn't NAT everything - the container
> gets the same IP as the host.  For most things that works more smoothly, but
> it does mean we have no way to address the host from the guest.  The map-gw
> option re-introduces limited NAT to allow that, but podman has disabled that
> here.  This means that there's simply no way for pasta to forward packets
> from ::1 -> ::1 across the tap interface: it has nothing it could use as a
> source address.  Hence, nothing on the tap packet capture.

Hello David.

Your comment made me think about how qemu listens inside the container. It seems I took it for granted that slirp does something similar to NAT64 and that pasta wouldn't behave in the same manner. It turns out qemu was only listening on `0.0.0.0:5900`. Adding `-spice ipv6=on` got it listening on `*:5900` after which remote-viewer started working as expected.

This is definitely a case of user error that's making me facepalm. Thanks to you and Stefano for spending time to look into this.

Note You need to log in before you can comment on or make changes to this bug.