Flaky test: libnetwork TestPortMappingV6Config (broken on ppc64le, s390x)?

[thaJeztah]

Seen failing on: https://ci-next.docker.com/public/job/moby/job/PR-42071/20/execution/node/349/log/

Bundles:
ppc64le-integration-bundles.tar.gz

=== RUN   TestPortMappingV6Config
time="2021-06-04T05:10:59Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: , error: exec: \"modprobe\": executable file not found in $PATH"
time="2021-06-04T05:10:59Z" level=warning msg="running inside docker container, ignoring missing kernel params: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
time="2021-06-04T05:10:59Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge/dummy/ is not added to the store"
time="2021-06-04T05:10:59Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
time="2021-06-04T05:11:00Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
--- FAIL: TestPortMappingV6Config (0.92s)
    port_mapping_test.go:157: Failed to store the port bindings into the sandbox info. Found: [{udp 172.19.0.11 400 0.0.0.0 54000 54000} {tcp 172.19.0.11 500 0.0.0.0 65000 65000} {sctp 172.19.0.11 500 0.0.0.0 65000 65000}]

See https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-42325/6/tests

Looks like modprobe is not installed on the ppc64le machines (may have to install https://packages.debian.org/buster/kmod), causing this
test to fail. I think this was added in the libnetwork migration, so we should
fix this (either in the Jenkinsfile, or ideally update the machine configurations)

=== RUN   TestPortMappingV6Config
time="2021-06-07T21:23:04Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: , error: exec: \"modprobe\": executable file not found in $PATH"
time="2021-06-07T21:23:04Z" level=warning msg="running inside docker container, ignoring missing kernel params: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
time="2021-06-07T21:23:04Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge/dummy/ is not added to the store"
time="2021-06-07T21:23:04Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
time="2021-06-07T21:23:05Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
--- FAIL: TestPortMappingV6Config (0.81s)
    port_mapping_test.go:157: Failed to store the port bindings into the sandbox info. Found: [{udp 172.19.0.11 400 0.0.0.0 54000 54000} {tcp 172.19.0.11 500 0.0.0.0 65000 65000} {sctp 172.19.0.11 500 0.0.0.0 65000 65000}]

On s390x, the test is also failing, but not showing the modprobe error:

=== RUN   TestPortMappingV6Config
time="2021-06-07T21:18:20Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge/dummy/ is not added to the store"
time="2021-06-07T21:18:20Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
time="2021-06-07T21:18:20Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
--- FAIL: TestPortMappingV6Config (1.54s)
    port_mapping_test.go:157: Failed to store the port bindings into the sandbox info. Found: [{udp 172.19.0.11 400 0.0.0.0 54000 54000} {tcp 172.19.0.11 500 0.0.0.0 65000 65000} {sctp 172.19.0.11 500 0.0.0.0 65000 65000}]

[SteampunkEngine]

Hi, I'm new here and I was running the unit tests in Ubuntu 20.04. This unit test failed for me as well:

=== Failed
=== FAIL: libnetwork/drivers/bridge TestPortMappingV6Config (0.23s)
time="2021-08-02T17:29:40Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge/dummy/ is not added to the store"
time="2021-08-02T17:29:40Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
time="2021-08-02T17:29:40Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
    port_mapping_test.go:157: Failed to store the port bindings into the sandbox info. Found: [{udp 172.19.0.11 400 0.0.0.0 54000 54000} {tcp 172.19.0.11 500 0.0.0.0 65000 65000} {sctp 172.19.0.11 500 0.0.0.0 65000 65000}]

I noticed that the if statement on line 156 is checking for 6 port mappings, but portBindings only has 3 elements.

TestPortMappingConfig only checks for 3 port mappings and it works fine. Is there a reason that TestPortMappingV6Config would need to check for 6 port mappings or was that unintentional?

I hope that helps and feel free to correct me if I missed anything. I haven't looked into the warnings from this error message yet, but I'm not sure if they're related to the test failure.

[thaJeztah]

TestPortMappingConfig only checks for 3 port mappings and it works fine. Is there a reason that TestPortMappingV6Config would need to check for 6 port mappings or was that unintentional?

I think in tis case, it's because mappings for both IPv4 and IPv6 are enabled, so each port will be published twice

[SteampunkEngine]

Yeah, I knew I missed something. That fix would have been too easy.

[akerouanton]

On few occasions TestPortMappingV6Config passes on my laptop (amd64 cpu) but many times it fails. After looking at the code, I think I found why it breaks:

1. The test starts with testutils.SetupTestOSContext(t) to use a new net ns ;
2. Then the following calls happen: d.ProgramExternalConnectivity() -> network.allocatePorts() -> n.allocatePortsInternal()
3. n.allocatePortsInternal() checks if IPv6 is available by calling IsV6Listenable(), since fix port forwarding with ipv6.disable=1 libnetwork#2635
4. IsV6Listenable() tries to listen on [::1]:0 to check if IPv6 is available but it actually fails because loopback interface is not up

I think this test passes only when executed in a certain order although I don't get why since the test always run in its own net ns. For instance, using TESTFLAGS='-shuffle on' hack/test/unit, this test always fail.

[EGuesnet]

Hi,

I am able to reproduce the error on Ubuntu 20.04 ppc64le and Ubuntu 20.04 x86 (Qemu VM) using branch master.

# make test-unit

=== Failed
=== FAIL: libnetwork/drivers/bridge TestPortMappingV6Config (0.13s)
time="2022-03-15T13:05:49Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge/dummy/ is not added to the store"
time="2022-03-15T13:05:49Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
time="2022-03-15T13:05:49Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
    port_mapping_test.go:163: Failed to store the port bindings into the sandbox info. Found: [{udp 172.19.0.11 400 0.0.0.0 54000 54000} {tcp 172.19.0.11 500 0.0.0.0 65000 65000} {sctp 172.19.0.11 500 0.0.0.0 65000 65000}]

As reported by @thaJeztah , it seems some IPv6 networks are attempted, but not found. I confirm IPv6 is not activated in docker:

# docker network ls --format "{{.Name}} {{.Driver}} {{.IPv6}}"
bridge bridge false
[...]

If I create a new IPv6 docker network, following https://docs.docker.com/config/daemon/ipv6/ first and next:

docker network create  bridge_ipv6 --ipv6 --subnet "2001:db8:1::/64"

I can run tests using this network:

# docker run --rm -i --privileged  -e DOCKER_CROSSPLATFORMS -e BUILD_APT_MIRROR -e BUILDFLAGS -e KEEPBUNDLE -e DOCKER_BUILD_ARGS -e DOCKER_BUILD_GOGC -e DOCKER_BUILD_OPTS -e DOCKER_BUILD_PKGS -e DOCKER_BUILDKIT -e DOCKER_BASH_COMPLETION_PATH -e DOCKER_CLI_PATH -e DOCKER_DEBUG -e DOCKER_EXPERIMENTAL -e DOCKER_GITCOMMIT -e DOCKER_GRAPHDRIVER -e DOCKER_LDFLAGS -e DOCKER_PORT -e DOCKER_REMAP_ROOT -e DOCKER_ROOTLESS -e DOCKER_STORAGE_OPTS -e DOCKER_TEST_HOST -e DOCKER_USERLANDPROXY -e DOCKERD_ARGS -e TEST_FORCE_VALIDATE -e TEST_INTEGRATION_DIR -e TEST_SKIP_INTEGRATION -e TEST_SKIP_INTEGRATION_CLI -e TESTDEBUG -e TESTDIRS -e TESTFLAGS -e TESTFLAGS_INTEGRATION -e TESTFLAGS_INTEGRATION_CLI -e TEST_FILTER -e TIMEOUT -e VALIDATE_REPO -e VALIDATE_BRANCH -e VALIDATE_ORIGIN_BRANCH -e VERSION -e PLATFORM -e DEFAULT_PRODUCT_LICENSE -e PRODUCT -v "/home/ubuntu/moby/bundles:/go/src/github.com/docker/docker/bundles" -v "/home/ubuntu/moby/.git:/go/src/github.com/docker/docker/.git" -v docker-dev-cache:/root/.cache -v docker-mod-cache:/go/pkg/mod/ --network bridge_ipv6   -t "docker-dev" hack/test/unit

and test TestPortMappingV6Config passes.

I have a side effect, the test TestDNSOptions fails, but it was related to /etc/resolv.conf configuration.

So, as far as I know, this error is not related to docker itself, but to the machine configuration, and especially the fact IPv6 is activated or not in docker. PR 43017 does not seems to be enough to deal with this issue.