If I attempt to use pasta with the IPv6 loopback address on a machine that doesn't have a default IPv6 route, I get the following error: ``` Error: failed to start pasta: No external routable interface for IPv6 Failed to bind any port for '-t ::1/57143-57143:5900-5900', exiting ``` This prevents the use of IPv6 addresses on a completely offline machine. Aside from the loopback address, unique local addresses may be bound as well to the loopback device in offline situations. I don't think port binding should be predicated by the existence of any route at all. If any check must be done, it should only check whether the OS supports IPv6 and whether that IPv6 support is enabled or not.
Justin, thanks for reporting this. Unfortunately, a side effect of this commit: https://passt.top/passt/commit/?id=a7359f09489803e501c85c7158b9462c6b3df465 should have been to cover that case as well: if there's no IPv6 default gateway, proceed, while disabling the functionality that maps the address of the default gateway (because there isn't one) to the parent namespace/host. However, there's a typo there: instead of not checking the default gateway, we skip the check on the "main" IPv6 address (the one that would be assigned). The fix is quite simple, if you feel like trying that out before I formally submit the patch: in conf_ip6(), conf.c, don't return 0 on IN6_IS_ADDR_UNSPECIFIED(&ip6->gw). I'll wait a bit in case you can test this in your setup, first. Meanwhile, as a workaround, you can pass something like '-g 2001:db8::1', that should work. (In reply to Justin Jereza from comment #0) > I don't think port binding should be predicated by the existence of any > route at all. If any check must be done, it should only check whether the OS > supports IPv6 and whether that IPv6 support is enabled or not. I see your point about routes, and coincidentally that's what I wanted to fix with that change (which works as intended for IPv4, not for IPv6). On the other hand, if IPv6 is enabled but we don't even have a single address on any interface (and you don't give one with -a), this breaks a few assumptions, such as having a link-local address we can use to communicate with the namespace or guest, and having one address that (by default) we can use to assign. We can probably cut down a bit further on these requirements (strictly speaking, we can pretend we have a link-local address), but we need to find reasonable default values for those cases.
(In reply to Stefano Brivio from comment #1) > The fix is quite simple, if you feel like trying that out before I formally > submit the patch: in conf_ip6(), conf.c, don't return 0 on > IN6_IS_ADDR_UNSPECIFIED(&ip6->gw). I'll wait a bit in case you can test this > in your setup, first. Heya Stefano. I've tried modifying the code from HEAD as you suggested and the problem seems more complex. I've gone as far as making the following changes: ``` diff --git a/conf.c b/conf.c index d8414fe..87cb3b2 100644 --- a/conf.c +++ b/conf.c @@ -705,13 +705,17 @@ static unsigned int conf_ip6(unsigned int ifi, memcpy(&ip6->addr_seen, &ip6->addr, sizeof(ip6->addr)); memcpy(&ip6->addr_ll_seen, &ip6->addr_ll, sizeof(ip6->addr_ll)); - if (MAC_IS_ZERO(mac)) + if (MAC_IS_ZERO(mac)) { nl_link(0, ifi, mac, 0, 0); + // hack to just return a valid `ifi` no matter what + // since the mac address of `lo` is zero. + return 1; + } - if (IN6_IS_ADDR_UNSPECIFIED(&ip6->gw) || - IN6_IS_ADDR_UNSPECIFIED(&ip6->addr_ll) || - MAC_IS_ZERO(mac)) - return 0; +// if (IN6_IS_ADDR_UNSPECIFIED(&ip6->gw) || +// IN6_IS_ADDR_UNSPECIFIED(&ip6->addr_ll) || +// MAC_IS_ZERO(mac)) +// return 0; return ifi; } ``` This still results in the same error: ``` + NETWORK='--network pasta' + podman run --rm -it --network pasta --publish '[::1]:52143:5900' alpine:latest Error: failed to start pasta: No external routable interface for IPv6 Failed to bind any port for '-t ::1/52143-52143:5900-5900', exiting ``` > Meanwhile, as a workaround, you can pass something like '-g 2001:db8::1', > that should work. I've tried this as well and the result is the same as not specifying a gateway at all. ``` + NETWORK='--network pasta:-g,2001:db8::1' + podman run --rm -it --network pasta:-g,2001:db8::1 --publish '[::1]:52143:5900' alpine:latest Error: failed to start pasta: No external routable interface for IPv6 Failed to bind any port for '-t ::1/52143-52143:5900-5900', exiting ``` Just to make sure I'm actually running the modified code, `pasta --version` returns `pasta unknown version` as expected. > On the other hand, if IPv6 is enabled but we don't even have a single > address on any interface (and you don't give one with -a), this breaks a few > assumptions, such as having a link-local address we can use to communicate > with the namespace or guest, and having one address that (by default) we can > use to assign. I see what you mean. I'm thinking about the following cases: * [::1] - This should only exist on loopback interfaces so maybe just check for that? * [::] - Whether or not an IPv6 address is assigned to any interface, listening should succeed I think. I'm assuming the kernel needs to only have IPv6 support. Whether it is reachable or not is the user's problem. * Any other IPv6 address - As long as the address (even a link-local address) exists on any interface, we can bind a port to it. I'm still trying to figure out how the network namespaces work with pasta and how that affects your concern with the availability of link-local addresses. With `--pcap /tmp/foo.cap`, I'm seeing a router solicitation from `c2:fa:c7:ec:db:eb` but I can't find that interface. I'm assuming though that whatever virtual interface is used to communicate with the namespace will be able to acquire a link-local address and that it's only a question of whether it will just be IPv4 or also IPv6 depending on kernel support. > We can probably cut down a bit further on these requirements (strictly > speaking, we can pretend we have a link-local address), but we need to find > reasonable default values for those cases. Wouldn't the same algorithm used to generate link-local addresses for any other interface be sufficient since it's tied to a particular interface anyway?
After looking at the source some more, could the problem be caused by the `ifi` of `lo` being 0? I see a number of `if (c->ifi6) ...` statements in the code so a valid interface would be ignored.
Hi Justin, Stefano is extremely busy with other things, so I'm taking a look at this. There's a fair bit to analyze, but first things first: 1) I'm preparing a proper patch to merge fixing the bug Stefano describes in comment 1. It's definitely a bug, although it's not the only thing going wrong here. 2) Index of 'lo' 'lo' has index 1, not index 0, so that's not an issue. 3) Host connectivity I only have a fuzzy picture of what exactly the network setup is on your host. Could you send the output of 'ip -6 addr' from the host to make those details clear?
A few additional questions that will help me to understand what's going on here: 4) Does it behave differently if you use '-t ::/57143-57143:5900-5900' or just '-t 57143-57143:5900-5900' instead of including the explicit ::1 bind address? 5) What address is the server within the container listening on?
Hello David, > Stefano is extremely busy with other things, so I'm taking a look at this. > There's a fair bit to analyze, but first things first: > > 1) I'm preparing a proper patch to merge fixing the bug Stefano describes in > comment 1. It's definitely a bug, although it's not the only thing going > wrong here. > > 2) Index of 'lo' > > 'lo' has index 1, not index 0, so that's not an issue. Understood on all of the above. > 3) Host connectivity > > I only have a fuzzy picture of what exactly the network setup is on your > host. Could you send the output of 'ip -6 addr' from the host to make those > details clear? This is a CoreOS VM that doesn't have globally unique IPv6 addresses and does not have an IPv6 gateway available. The output of `ip -6 addr`: ``` 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000 inet6 fe80::3135:4288:a4f8:5829/64 scope link noprefixroute valid_lft forever preferred_lft forever ``` The output of `ip -6 route`: ``` ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev enp1s0 proto kernel metric 1024 pref medium ``` > 4) Does it behave differently if you use '-t ::/57143-57143:5900-5900' or > just '-t 57143-57143:5900-5900' instead of including the explicit ::1 bind > address? I tested both with and without just the gateway check from HEAD: ``` diff --git a/conf.c b/conf.c index d8414fe..5d790cd 100644 --- a/conf.c +++ b/conf.c @@ -708,8 +708,7 @@ static unsigned int conf_ip6(unsigned int ifi, if (MAC_IS_ZERO(mac)) nl_link(0, ifi, mac, 0, 0); - if (IN6_IS_ADDR_UNSPECIFIED(&ip6->gw) || - IN6_IS_ADDR_UNSPECIFIED(&ip6->addr_ll) || + if (IN6_IS_ADDR_UNSPECIFIED(&ip6->addr_ll) || MAC_IS_ZERO(mac)) return 0; ``` In both cases, `-t ::/57143-57143:5900-5900` and `-t ::1/57143-57143:5900-5900` resulted in the same failure: ``` Error: failed to start pasta: No external routable interface for IPv6 Failed to bind any port for '-t ::/57143-57143:5900-5900', exiting ``` Just `-t 57143-57143:5900-5900` succeeds. > 5) What address is the server within the container listening on? There is currently no server listening within the container since I figured whether or not there is something listening, the address/port should bind. If I do test with a listening process, it's just with `nc -lp 5900` which will listen on `*:5900`. The failure to start pasta happens before nc can even be executed though. I've been testing all of this with `alpine:latest` but attempting to start with `fedora:38` results in the same failure.
The pasta arguments generated by podman: ``` DEBU[0000] pasta arguments: --config-net -t ::1/57143-57143:5900-5900 -u none -T none -U none --no-map-gw --netns /run/user/1000/netns/netns-99155890-8697-52d3-2740-fb1ac67f7c3b ```
(In reply to Justin Jereza from comment #6) [snip] > > 3) Host connectivity > > > > I only have a fuzzy picture of what exactly the network setup is on your > > host. Could you send the output of 'ip -6 addr' from the host to make those > > details clear? > > This is a CoreOS VM that doesn't have globally unique IPv6 addresses and > does not have an IPv6 gateway available. > > The output of `ip -6 addr`: > > ``` > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000 > inet6 ::1/128 scope host > valid_lft forever preferred_lft forever > 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000 > inet6 fe80::3135:4288:a4f8:5829/64 scope link noprefixroute > valid_lft forever preferred_lft forever > ``` Ok, so no global IPv6 addresses at all, just a scope host and a scope link. > The output of `ip -6 route`: > > ``` > ::1 dev lo proto kernel metric 256 pref medium > fe80::/64 dev enp1s0 proto kernel metric 1024 pref medium > ``` No default route, and also no routes to any global addresses, either. > > 4) Does it behave differently if you use '-t ::/57143-57143:5900-5900' or > > just '-t 57143-57143:5900-5900' instead of including the explicit ::1 bind > > address? > > I tested both with and without just the gateway check from HEAD: > > ``` > diff --git a/conf.c b/conf.c > index d8414fe..5d790cd 100644 > --- a/conf.c > +++ b/conf.c > @@ -708,8 +708,7 @@ static unsigned int conf_ip6(unsigned int ifi, > if (MAC_IS_ZERO(mac)) > nl_link(0, ifi, mac, 0, 0); > > - if (IN6_IS_ADDR_UNSPECIFIED(&ip6->gw) || > - IN6_IS_ADDR_UNSPECIFIED(&ip6->addr_ll) || > + if (IN6_IS_ADDR_UNSPECIFIED(&ip6->addr_ll) || > MAC_IS_ZERO(mac)) > return 0; > ``` > > In both cases, `-t ::/57143-57143:5900-5900` and `-t > ::1/57143-57143:5900-5900` resulted in the same failure: > > ``` > Error: failed to start pasta: > No external routable interface for IPv6 > Failed to bind any port for '-t ::/57143-57143:5900-5900', exiting > ``` > > Just `-t 57143-57143:5900-5900` succeeds. Huh.. that's weird. -t 1234 and -t ::/1234 are supposed to be equivalent. > > 5) What address is the server within the container listening on? > > There is currently no server listening within the container since I figured > whether or not there is something listening, the address/port should bind. > If I do test with a listening process, it's just with `nc -lp 5900` which > will listen on `*:5900`. The failure to start pasta happens before nc can > even be executed though. > > I've been testing all of this with `alpine:latest` but attempting to start > with `fedora:38` results in the same failure. Ok, understood.
<some analysis later> So, first thing to note is that similar to my description in bug 51 comment 9, the only way this scenario can work at all is because of the "splicing" semantics. There's simply no way to forward packets to the container from the host across the tap interface, because we have nothing we can use as a source address. That could be worked around by using both -a and -g options, but I'm not sure how that would need to be configured at the podman level. So, I think here it's actually correct for pasta to fail if given -t ::/1234, or indeed -t <addr>/1234 for any address that's not loopback. Obviously the error message should be more helpful, though. So that leaves two things to debug: a) Why is it appearing to succeed with -t 1234. (Initial guess: it can do something meaningful here for IPv4 and so is ignoring the IPv6 failure) b) Why is it failing with -t ::1/1234, which should be possible via splicing. I'm looking into these now..
Ok. First level answer to both the questions from comment 9 is pretty simple. In this scenario (AFAICT) pasta is not being given an explicit interface to use (the "template interface"). So, it tries to set the interface to the one with the default route for both IPv4 and IPv6. For IPv4 it succeeds (I'm assuming there *is* an IPv4 default route). For IPv6 it fails, and therefore disables IPv6 entirely. Essentially as soon as the "No external routable interface for IPv6" message is printed we set c->ifi6 to 0 and it's all over for IPv6. a) Basically as guessed. The logic is a bit complicated but roughly speaking we consider the forwarding setup successful if we managed to bind any ports. Since we bound IPv4 ports, we don't give any error. b) As above. Or with more detail, when we try to bind the ::1 address we call into tcp_sock_init() which explicitly fails because c->ifi6 is 0. Next stop, whaet can we do about it...
* Long term, I'd like to get rid of the need for a template interface, which should make this possible. However, while I have plans for that it's a reasonable way off. It also can't really work without some sort of NAT-to-host address (a generalized version of --map-gw), which has its own complications * Short term, the only real answer here is to get pasta to pick an interface for IPv6 despite not having a default route. That can't really be 'lo' (or at least that introduces further complications that I don't think we can solve quickly). However, it should be possible for it to use enp1s0, even though that has no (global scope) IPv6 routes. * The most obvious approach is to explicitly tell it to use enp1s0. At the pasta level that's easy (-i enp1s0), but I'm not sure if or how that's possible via podman (still need to look into this). * The next question is can we have pasta pick some fallback interface if it doesn't find a default route. In this case it's the only non-lo interface, but that rule doesn't generalize that well. Not sure if there's anything else sensible we could do. Ok, so WORKAROUND 1 (you might be able to use this one right now) Fake a default route on the host so that pasta will do close enough to the right thing. 1. Pick an arbitrary address in the RFC4193/8190 range fc00::/7, call it $ADDR 2. # ip link add type dummy 3. # ip addr add $ADDR dev dummy0 4. # ip link set dummy0 up 5. # ip -6 route add default dev dummy0 6. Start podman+pasta WORKAROUND 1a (if you don't have root on the host) Create your own network sandbox with "unshare -Unr", then use WORKAROUND 1. You might need to also fake connectivity for IPv4. Still researching... if there's a way to make podman give a -i option to pasta.
WORKAROUND 2 (explicitly set the interface) So, I looked at how podman invokes pasta. There's no built in way to set the interface, but looks like you can add explicit additional pasta options in containers.conf. I think if you add to that: . . . [network] pasta_options = [ "-i", "enp1s0"] . . . That should do the trick (for this host, anyway).
Stefano pointed out to me that you can do WORKAROUND 2 more easily from the command line with: --net=pasta:-i,enp1s0
Thanks for doing an in-depth investigation on this. (In reply to David Gibson from comment #10) > In this scenario (AFAICT) pasta is not being given an explicit interface to > use (the "template interface"). So, it tries to set the interface to the > one with the default route for both IPv4 and IPv6. For IPv4 it succeeds > (I'm assuming there *is* an IPv4 default route). For IPv6 it fails, and > therefore disables IPv6 entirely. Essentially as soon as the "No external > routable interface for IPv6" message is printed we set c->ifi6 to 0 and it's > all over for IPv6. I can confirm that an IPv4 address and default route is configured on the machine. I got curious and shut down `enp1s0` resulting in `lo` being the only active interface. In this particular case, `podman run --rm -it --network pasta --publish 57143:5900 alpine:latest` fails with the following: ``` Error: failed to start pasta: No external routable interface for IPv4 No external routable interface for IPv6 External interface not usable ``` This seems to confirm your theory that `-p 57143:5900` succeeds on IPv4 and silently fails on IPv6. Indeed with `enp1s0` active and without an IPv6 default route, `nc 127.0.0.1 57143` successfully sends packets to the container whereas connecting with `nc "::1" 57143` is refused. (In reply to David Gibson from comment #11) > * Long term, I'd like to get rid of the need for a template interface, > which should make this possible. However, while I have plans for that it's > a reasonable way off. It also can't really work without some sort of > NAT-to-host address (a generalized version of --map-gw), which has its own > complications It's nice to know that there's a plan to get rid of the reliance on the state of host interfaces besides loopback but understandably, it's going to be quite complicated since I'm guessing it's going to require substantial re-architecture. > WORKAROUND 1 (you might be able to use this one right now) > > Fake a default route on the host so that pasta will do close enough to the > right thing. > > 1. Pick an arbitrary address in the RFC4193/8190 range fc00::/7, call it > $ADDR > 2. # ip link add type dummy > 3. # ip addr add $ADDR dev dummy0 > 4. # ip link set dummy0 up > 5. # ip -6 route add default dev dummy0 > 6. Start podman+pasta This works but I'm wary that giving the host a default IPv6 route that doesn't actually go anywhere might result in unknown ramifications. (In reply to David Gibson from comment #13) > Stefano pointed out to me that you can do WORKAROUND 2 more easily from the > command line with: > --net=pasta:-i,enp1s0 I think this is the winner as far as workarounds go. It shouldn't have any side-effects on the host that we can't foresee and it doesn't require root. As a note for the case I mentioned above where the only active interface is `lo`, the following can also be done to make it work: ``` # ip link add type dummy # ip link set dummy0 up # dummy0 is given an IPv6 link-local address which is sufficient for our needs $ podman run --rm -it --network pasta:-i,dummy0 --publish "[::1]:57143:5900" alpine:latest nc -lp 5900 ``` The lack of an IPv4 address on the dummy0 interface above means that only IPv6 connections will work when `--publish 57143:5900` is used.
(In reply to Justin Jereza from comment #14) > Thanks for doing an in-depth investigation on this. > > (In reply to David Gibson from comment #10) > > In this scenario (AFAICT) pasta is not being given an explicit interface to > > use (the "template interface"). So, it tries to set the interface to the > > one with the default route for both IPv4 and IPv6. For IPv4 it succeeds > > (I'm assuming there *is* an IPv4 default route). For IPv6 it fails, and > > therefore disables IPv6 entirely. Essentially as soon as the "No external > > routable interface for IPv6" message is printed we set c->ifi6 to 0 and it's > > all over for IPv6. > > I can confirm that an IPv4 address and default route is configured on the > machine. I got curious and shut down `enp1s0` resulting in `lo` being the > only active interface. In this particular case, `podman run --rm -it > --network pasta --publish 57143:5900 alpine:latest` fails with the following: > > ``` > Error: failed to start pasta: > No external routable interface for IPv4 > No external routable interface for IPv6 > External interface not usable > ``` > > This seems to confirm your theory that `-p 57143:5900` succeeds on IPv4 and > silently fails on IPv6. Indeed with `enp1s0` active and without an IPv6 > default route, `nc 127.0.0.1 57143` successfully sends packets to the > container whereas connecting with `nc "::1" 57143` is refused. Thanks for the confirmation. > (In reply to David Gibson from comment #11) > > * Long term, I'd like to get rid of the need for a template interface, > > which should make this possible. However, while I have plans for that it's > > a reasonable way off. It also can't really work without some sort of > > NAT-to-host address (a generalized version of --map-gw), which has its own > > complications > > It's nice to know that there's a plan to get rid of the reliance on the > state of host interfaces besides loopback but understandably, it's going to > be quite complicated since I'm guessing it's going to require substantial > re-architecture. Kind of, yes. I basically have a bunch of related changes to make which clean up and generalize NAT handling, port forwarding configuration and interface management. This allows a bunch of interesting usecases. I think this can be done fairly incrementally, with benefits from each step. However, there's kind of an order they have to happen in, and removing the "template interface" concept is a fair way down the track. > > WORKAROUND 1 (you might be able to use this one right now) > > > > Fake a default route on the host so that pasta will do close enough to the > > right thing. > > > > 1. Pick an arbitrary address in the RFC4193/8190 range fc00::/7, call it > > $ADDR > > 2. # ip link add type dummy > > 3. # ip addr add $ADDR dev dummy0 > > 4. # ip link set dummy0 up > > 5. # ip -6 route add default dev dummy0 > > 6. Start podman+pasta > > This works but I'm wary that giving the host a default IPv6 route that > doesn't actually go anywhere might result in unknown ramifications. That's fair, and it also obviously requires root on the host, which arguably defeats the point of pasta in the first place. > (In reply to David Gibson from comment #13) > > Stefano pointed out to me that you can do WORKAROUND 2 more easily from the > > command line with: > > --net=pasta:-i,enp1s0 > > I think this is the winner as far as workarounds go. It shouldn't have any > side-effects on the host that we can't foresee and it doesn't require root. Ok, good to hear. Stefano has pointed out that it might be possible to have a fallback selection of "template interface" when there is no default route: the obviously choice would be to pick any (non lo) interface with a global address, and if that fails pick any (non lo) interface with any address at all. I believe that would allow your case to work out of the box. > As a note for the case I mentioned above where the only active interface is > `lo`, the following can also be done to make it work: > > ``` > # ip link add type dummy > # ip link set dummy0 up # dummy0 is given an IPv6 link-local address which > is sufficient for our needs > $ podman run --rm -it --network pasta:-i,dummy0 --publish "[::1]:57143:5900" > alpine:latest nc -lp 5900 > ``` > > The lack of an IPv4 address on the dummy0 interface above means that only > IPv6 connections will work when `--publish 57143:5900` is used. Ok. That's a useful workaround to know about. I don't think we'll attempt to allow pasta to "naturally" work with no external interface, at least until that happens naturally a fair way down the chain of NAT/forwarding/interface changes.