fix port forwarding with ipv6.disable=1 #2635

AkihiroSuda · 2021-04-26T06:31:54Z

Make docker run -p 80:80 functional again on environments with kernel boot parameter ipv6.disable=1.

Fix moby/moby#42288
fixes #2629 Network issue with IPv6 following update to version 20.10.6

Tested on Ubuntu 21.04.

Previously, docker run -p 80:80 with ipv6.disable=1 was failing with an error:
docker: Error response from daemon: driver failed programming external connectivity on endpoint naughty_moore (038e9ed4b5ea77e1c52462d6d04ad001fbad9beb185a6511aadc217c8a271608): Error starting userland proxy: listen tcp6 [::]:80: socket: address family not supported by protocol.

cpuguy83 · 2021-04-26T18:02:37Z

drivers/bridge/port_mapping.go

@@ -50,6 +51,12 @@ func (n *bridgeNetwork) allocatePortsInternal(bindings []types.PortBinding, cont
 			bs = append(bs, bIPv4)
 		}

+		// skip adding v6 addr when the kernel was booted with `ipv6.disable=1`
+		// https://github.com/moby/moby/issues/42288
+		if !IsV6Listenable() {


The only case we need to fix here (I think...) is when 0.0.0.0 is passed.
If someone specifies a v6 address and we can't do that I don't think we should ignore it.

I don't understand what was fixed.
I think the check for IsV6Listenable() (or whwatever its called now) should go into https://github.com/moby/libnetwork/pull/2635/files#diff-5e6a1e1b6a7570d4362c2cfd293b0f05b409b6c9b1c8157b9832f427f6e0cf56L109 explicitly where we are checking if the bind address is 0.0.0.0

isV6Binding := c.HostIP != nil && c.HostIP.To4() == nil is set to false when the bind address is 0.0.0.0, and set to true when the bind address is set to IPv6 explicitly, so the issue was fixed IIUC.

Test matrix:

docker run -p 80:80: listens on 0.0.0.0:80, without an error

docker run -p 0.0.0.0:80:80: listens on 0.0.0.0:80, without an error

docker run -p [::]:80:80: raises an error

My point is we are already checking for 0.0.0.0 when validating the bind address, why not add the check when we do that?

It's also a bit less confusing because it is checking net.IPv4zero specifically.

I tried that, but couldn't make it work

Sorry to be a stickler but I don't understand the check you have in place and how it works.
Meanwhile: https://play.golang.org/p/jtQGBQ4yzWd

It should be able to implicitBinding := hostIP != nil && hostIp.Equal(net.IPv4zero)

@tianon helped me unbreak my brain here.
This seems like it's ok because we are checking if it is a v4 binding and only continuing if it is a v6-only binding which makes sense. It may also be that we actually have a nil host IP...

Make `docker run -p 80:80` functional again on environments with kernel boot parameter `ipv6.disable=1`. Fix moby/moby issue 42288 Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda · 2021-04-27T07:02:52Z

@moby/libnetwork @thaJeztah @tiborvass PTAL

cpuguy83

Maybe a unit test for this?

Probably need to hijack the v6ListenableOnce in the test.

arkodg · 2021-05-03T18:21:36Z

libnetwork/drivers/bridge/port_mapping.go

Line 118 in b5dc370

}

might be a better place to reuse some of the validation above and then check IsV6Listenable

AkihiroSuda · 2021-05-06T09:07:14Z

Help wanted for testing.

Anyway, would it be possible to release v20.10.7 to provide the quick fix, and revisit the testing plan later?
cc @thaJeztah @tiborvass

thaJeztah · 2021-05-06T19:13:17Z

@AkihiroSuda could you have a look at @arkodg 's suggestion? If we're able to move the check inside that function, we can probably add a unit test for it at least

docker-engine 20.10.6 broke container port forwarding for hosts without IPv6 support: docker: Error response from daemon: driver failed programming external connectivity on endpoint naughty_moore (038e9ed4b5ea77e1c52462d6d04ad001fbad9beb185a6511aadc217c8a271608): Error starting userland proxy: listen tcp6 [::]:80: socket: address family not supported by protocol. Add a libnetwork patch from an upstream pull request to fix this, after adjusting the patch to apply to docker-engine (which has libnetwork vendored under vendor/github.com/docker/libnetwork): - moby/libnetwork#2635, - moby/moby#42322 Signed-off-by: Peter Korsgaard <[email protected]>

docker-engine 20.10.6 broke container port forwarding for hosts without IPv6 support: docker: Error response from daemon: driver failed programming external connectivity on endpoint naughty_moore (038e9ed4b5ea77e1c52462d6d04ad001fbad9beb185a6511aadc217c8a271608): Error starting userland proxy: listen tcp6 [::]:80: socket: address family not supported by protocol. Add a libnetwork patch from an upstream pull request to fix this, after adjusting the patch to apply to docker-engine (which has libnetwork vendored under vendor/github.com/docker/libnetwork): - moby/libnetwork#2635, - moby/moby#42322 Signed-off-by: Peter Korsgaard <[email protected]> (cherry picked from commit 2fd3390) Signed-off-by: Peter Korsgaard <[email protected]>

thaJeztah

LGTM to get the fix in, but we should have a look at cleaning up in a follow up

thaJeztah · 2021-05-20T19:17:55Z

@cpuguy83 @arkodg ok with you to merge this?

cpuguy83

Did some manual validation here to at least check the logic.

thaJeztah · 2021-05-25T09:06:42Z

let me bring this one in and open a vendor PR in moby

Since moby/libnetwork#2635 has been merged, allocatePortsInternal() checks if IPv6 is enabled by calling IsV6Listenable(). This function calls `net.Listen("tcp6", "[::1]:0")` and returns false when net.Listen() fails. TestPortMappingV6Config() starts by setting up a new net ns to run into it. The loopback interface is not bring up in this net ns, thus net.Listen() fails and IsV6Listenable() returns false. This change takes care of bringing loopback iface up right after moving to the new net ns. This test has been reported has flaky on s390x in moby#42468. For some reason, this test seems to be consistently green on the CI (on amd64 arch) and when running `hack/test/unit` locally. However it consistently fails when running `TESTFLAGS='-shuffle on' hack/test/unit` locally. Signed-off-by: Albin Kerouanton <[email protected]>

Since moby/libnetwork#2635 has been merged, allocatePortsInternal() checks if IPv6 is enabled by calling IsV6Listenable(). This function calls `net.Listen("tcp6", "[::1]:0")` and returns false when net.Listen() fails. TestPortMappingV6Config() starts by setting up a new net ns to run into it. The loopback interface is not bring up in this net ns, thus net.Listen() fails and IsV6Listenable() returns false. This change takes care of bringing loopback iface up right after moving to the new net ns. This test has been reported has flaky on s390x in #42468. For some reason, this test seems to be consistently green on the CI (on amd64 arch) and when running `hack/test/unit` locally. However it consistently fails when running `TESTFLAGS='-shuffle on' hack/test/unit` locally. Signed-off-by: Albin Kerouanton <[email protected]> Upstream-commit: c721bad8ccddeb353e71d4b4b26ad878d1ae8bee Component: engine

Since moby/libnetwork#2635 has been merged, allocatePortsInternal() checks if IPv6 is enabled by calling IsV6Listenable(). This function calls `net.Listen("tcp6", "[::1]:0")` and returns false when net.Listen() fails. TestPortMappingV6Config() starts by setting up a new net ns to run into it. The loopback interface is not bring up in this net ns, thus net.Listen() fails and IsV6Listenable() returns false. This change takes care of bringing loopback iface up right after moving to the new net ns. This test has been reported has flaky on s390x in moby#42468. For some reason, this test seems to be consistently green on the CI (on amd64 arch) and when running `hack/test/unit` locally. However it consistently fails when running `TESTFLAGS='-shuffle on' hack/test/unit` locally. Signed-off-by: Albin Kerouanton <[email protected]>

docker-engine 20.10.6 broke container port forwarding for hosts without IPv6 support: docker: Error response from daemon: driver failed programming external connectivity on endpoint naughty_moore (038e9ed4b5ea77e1c52462d6d04ad001fbad9beb185a6511aadc217c8a271608): Error starting userland proxy: listen tcp6 [::]:80: socket: address family not supported by protocol. Add a libnetwork patch from an upstream pull request to fix this, after adjusting the patch to apply to docker-engine (which has libnetwork vendored under vendor/github.com/docker/libnetwork): - moby/libnetwork#2635, - moby/moby#42322 Signed-off-by: Peter Korsgaard <[email protected]> (cherry picked from commit 2fd3390) Signed-off-by: Peter Korsgaard <[email protected]>

AkihiroSuda mentioned this pull request Apr 26, 2021

fix port forwarding with ipv6.disable=1 moby/moby#42322

Closed

AkihiroSuda closed this Apr 26, 2021

AkihiroSuda reopened this Apr 26, 2021

AkihiroSuda force-pushed the ipv6disable branch 2 times, most recently from a4a3ac7 to 5e6322b Compare April 26, 2021 07:14

cpuguy83 requested changes Apr 26, 2021

View reviewed changes

fix port forwarding with ipv6.disable=1

7b9c290

Make `docker run -p 80:80` functional again on environments with kernel boot parameter `ipv6.disable=1`. Fix moby/moby issue 42288 Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda force-pushed the ipv6disable branch from 5e6322b to 7b9c290 Compare April 27, 2021 06:46

AkihiroSuda requested a review from cpuguy83 April 27, 2021 06:47

cpuguy83 requested changes Apr 30, 2021

View reviewed changes

thaJeztah approved these changes May 20, 2021

View reviewed changes

cpuguy83 approved these changes May 24, 2021

View reviewed changes

thaJeztah merged commit 64b7a45 into moby:master May 25, 2021

chienhanlin mentioned this pull request Jul 30, 2021

Filter out ipv6 port forwards when an ipv6 default route doesn't exist aws/amazon-ecs-agent#2855

Closed

akerouanton mentioned this pull request Nov 14, 2021

Flaky test: libnetwork TestPortMappingV6Config (broken on ppc64le, s390x)? moby/moby#42468

Closed

akerouanton mentioned this pull request Nov 14, 2021

Fix flaky TestPortMappingV6Config moby/moby#43017

Merged

rulatir mentioned this pull request Sep 9, 2022

Latest image fails to start nginx concrete5-community/docker5#24

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix port forwarding with ipv6.disable=1 #2635

fix port forwarding with ipv6.disable=1 #2635

AkihiroSuda commented Apr 26, 2021 •

edited by thaJeztah

Loading

cpuguy83 Apr 26, 2021

AkihiroSuda Apr 27, 2021

cpuguy83 Apr 27, 2021

AkihiroSuda Apr 28, 2021

cpuguy83 Apr 29, 2021

cpuguy83 Apr 29, 2021

AkihiroSuda Apr 30, 2021

cpuguy83 Apr 30, 2021

cpuguy83 Apr 30, 2021

cpuguy83 Apr 30, 2021

AkihiroSuda commented Apr 27, 2021

cpuguy83 left a comment

arkodg commented May 3, 2021

AkihiroSuda commented May 6, 2021

thaJeztah commented May 6, 2021

thaJeztah left a comment

thaJeztah commented May 20, 2021

cpuguy83 left a comment

thaJeztah commented May 25, 2021

fix port forwarding with ipv6.disable=1 #2635

fix port forwarding with ipv6.disable=1 #2635

Conversation

AkihiroSuda commented Apr 26, 2021 • edited by thaJeztah Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Apr 27, 2021

cpuguy83 left a comment

Choose a reason for hiding this comment

arkodg commented May 3, 2021

AkihiroSuda commented May 6, 2021

thaJeztah commented May 6, 2021

thaJeztah left a comment

Choose a reason for hiding this comment

thaJeztah commented May 20, 2021

cpuguy83 left a comment

Choose a reason for hiding this comment

thaJeztah commented May 25, 2021

AkihiroSuda commented Apr 26, 2021 •

edited by thaJeztah

Loading