cannot create listening socket [:::2375] after #108

[jscheytt]

Hi there,

we are running docker-socket-proxy in Docker Swarm (docker engine v23). Since the newest nightly build, we get this error on startup:

[WARNING] 340/080535 (1) : config : missing timeouts for backend 'docker-events'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 340/080535 (1) : Can't open global server state file '/var/lib/haproxy/server-state': No such file or directory
Proxy dockerbackend started.
Proxy docker-events started.
[NOTICE] 340/080535 (1) : haproxy version is 2.2.31-d69335f
[ALERT] 340/080535 (1) : Starting frontend dockerfrontend: cannot create listening socket [:::2375]

This is likely related to #108.

We have not set up any IPv6 configuration in the Docker daemon nor in our networks. Do we need to do that in order for docker-socket-proxy to keep working?

This is roughly how we deploy docker-socket-proxy in Docker Swarm currently:

version: "3.9"
services:
  docker-socket-proxy:
    deploy:
      mode: global
      update_config:
        order: start-first
      resources:
        limits:
          memory: "104857600"
          pids: 1000
        reservations:
          cpus: "0.02"
          memory: "104857600"
      restart_policy:
        condition: on-failure
        max_attempts: 5
      placement:
        constraints:
        - node.role == manager
    environment:
      CONTAINERS: "1"
      NETWORKS: "1"
      NODES: "1"
      POST: "1"
      SERVICES: "1"
      SWARM: "1"
      TASKS: "1"
    healthcheck:
      test:
      - CMD-SHELL
      - wget -O - http://127.0.0.1:2375/_ping
      interval: 5s
      retries: 10
    image: tecnativa/docker-socket-proxy:nightly@sha256:46148a17aad9df1c21e4b7f173b8c0b5efe213ab90b193cec583d5adcaa16965
    logging:
      options:
        tag: '{{.Name}}'
    networks:
      internal: null
    volumes:
    - type: bind
      source: /var/run/docker.sock
      target: /var/run/docker.sock
      read_only: true
networks:
  internal: {}

[pedrobaeza]

Any advice, @saltydk @yajo ?

[saltydk]

We could move the IPv6 bind to an ENV toggle like with the socket capabilities?

[saltydk]

I suppose we could add a second bind

frontend dockerfrontend
    bind :2375
    bind :::2375 v6only

I will do some tests with that.

[saltydk]

@jscheytt Could you try this image?

saltydk/docker-socket-proxy:edge

[jscheytt]

@saltydk Still getting the same error ...

[saltydk]

Wonder why your setup doesn't work. I've tested both the current version and mine on a IPv4 only system.

[saltydk]

Are you just looking for the error or actually testing if the proxy works with the other image?

[jscheytt]

I don't understand your last question. What I did is: I changed the stack I pasted at the top to use the image you pasted. I rolled out this change via docker stack up --prune -c FILENAME STACKNAME. I then checked the logs and they contained the same text again as at the top.

Maybe some more system context helps: Host OS is RHEL v8. Docker daemon config is this:

{
  "metrics-addr": "0.0.0.0:9323",
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "*REDACTED*"
  },
  "no-new-privileges": true,
  "userland-proxy": false
}

[saltydk]

I meant actually using the proxy, assuming the container doesn't stop due to the error.

[jscheytt]

The container unfortunately does stop due to the error, so I cannot try using it ...

[saltydk]

@pedrobaeza Not sure how to fix this outside of having the entrypoint toggle the bind based on an ENV like USE_IPV6 or the like since haproxy seems unable to handle it internally.

[yajo]

Have you tried this?

frontend dockerfrontend
    bind [::]:2375 v4v6

Docs: https://www.haproxy.com/documentation/haproxy-configuration-tutorials/core-concepts/frontends/#listen-on-multiple-ip-addresses-and-ports

[saltydk]

Have you tried this?

frontend dockerfrontend
    bind [::]:2375 v4v6

Docs: https://www.haproxy.com/documentation/haproxy-configuration-tutorials/core-concepts/frontends/#listen-on-multiple-ip-addresses-and-ports

I've built an image with that although I'd be surprised if it changes anything.

saltydk/docker-socket-proxy:edge

[jscheytt]

Unfortunately still the same error ...

[yajo]

Then it'll have to be based on an option, or we'll have to roll back the ipv6 support.

[saltydk]

You okay with a custom entrypoint so we can do some more templating?

…

On Tue, 12 Dec 2023, 11:13 Jairo Llopis, ***@***.***> wrote: Then it'll have to be based on an option, or we'll have to roll back the ipv6 support. — Reply to this email directly, view it on GitHub <#109 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSIMLTKS7FNGYVSKKN25KTYJAU5PAVCNFSM6AAAAABAKWQWK2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJRG4ZTKMRRHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[saltydk]

I have two docker-entrypoint.sh you can choose from:

Use envsubst:

#!/bin/sh
set -e

# Normalize the input for ENABLE_IPV6 to lowercase
ENABLE_IPV6_LOWER=$(echo "$ENABLE_IPV6" | tr '[:upper:]' '[:lower:]')

# Check for different representations of 'true' and set BIND_CONFIG
case "$ENABLE_IPV6_LOWER" in
    1|true|yes)
        BIND_CONFIG="[::]:2375 v4v6"
        ;;
    *)
        BIND_CONFIG=":2375"
        ;;
esac

# Export BIND_CONFIG for envsubst
export BIND_CONFIG

# Process the HAProxy configuration template
envsubst < /usr/local/etc/haproxy/haproxy.cfg.template > /usr/local/etc/haproxy/haproxy.cfg

# Command-line argument handling
if [ "${1#-}" != "$1" ]; then
    set -- haproxy "$@"
fi

if [ "$#" -eq 0 ] || [ "$1" = 'haproxy' ]; then
    if [ "$1" = 'haproxy' ]; then
        shift
    fi
    set -- haproxy -W -db -f /usr/local/etc/haproxy/haproxy.cfg "$@"
fi

exec "$@"

or use sed:

#!/bin/sh
set -e

# Normalize the input for ENABLE_IPV6 to lowercase
ENABLE_IPV6_LOWER=$(echo "$ENABLE_IPV6" | tr '[:upper:]' '[:lower:]')

# Check for different representations of 'true' and set BIND_CONFIG
case "$ENABLE_IPV6_LOWER" in
    1|true|yes)
        BIND_CONFIG="[::]:2375 v4v6"
        ;;
    *)
        BIND_CONFIG=":2375"
        ;;
esac

# Process the HAProxy configuration template using sed
sed "s/\${BIND_CONFIG}/$BIND_CONFIG/g" /usr/local/etc/haproxy/haproxy.cfg.template > /usr/local/etc/haproxy/haproxy.cfg

# Command-line argument handling
if [ "${1#-}" != "$1" ]; then
    set -- haproxy "$@"
fi

if [ "$#" -eq 0 ] || [ "$1" = 'haproxy' ]; then
    if [ "$1" = 'haproxy' ]; then
        shift
    fi
    set -- haproxy -W -db -f /usr/local/etc/haproxy/haproxy.cfg "$@"
fi

exec "$@"

Which both modify the template file which would look like:

global
    log stdout format raw daemon "${LOG_LEVEL}"

    pidfile /run/haproxy.pid
    maxconn 4000

    # Turn on stats unix socket
    server-state-file /var/lib/haproxy/server-state

defaults
    mode http
    log global
    option httplog
    option dontlognull
    option http-server-close
    option redispatch
    retries 3
    timeout http-request 10s
    timeout queue 1m
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    timeout http-keep-alive 10s
    timeout check 10s
    maxconn 3000

    # Allow seamless reloads
    load-server-state-from-file global

    # Use provided example error pages
    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

backend dockerbackend
    server dockersocket $SOCKET_PATH

backend docker-events
    server dockersocket $SOCKET_PATH
    timeout server 0

frontend dockerfrontend
    bind ${BIND_CONFIG}
    http-request deny unless METH_GET || { env(POST) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers/[a-zA-Z0-9_.-]+/((stop)|(restart)|(kill)) } { env(ALLOW_RESTARTS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers/[a-zA-Z0-9_.-]+/start } { env(ALLOW_START) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers/[a-zA-Z0-9_.-]+/stop } { env(ALLOW_STOP) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/auth } { env(AUTH) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/build } { env(BUILD) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/commit } { env(COMMIT) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/configs } { env(CONFIGS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers } { env(CONTAINERS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/distribution } { env(DISTRIBUTION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/events } { env(EVENTS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/exec } { env(EXEC) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/grpc } { env(GRPC) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/images } { env(IMAGES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/info } { env(INFO) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/networks } { env(NETWORKS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/nodes } { env(NODES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/_ping } { env(PING) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/plugins } { env(PLUGINS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/secrets } { env(SECRETS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/services } { env(SERVICES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/session } { env(SESSION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/swarm } { env(SWARM) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/system } { env(SYSTEM) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/tasks } { env(TASKS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/version } { env(VERSION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/volumes } { env(VOLUMES) -m bool }
    http-request deny
    default_backend dockerbackend

    use_backend docker-events if { path,url_dec -m reg -i ^(/v[\d\.]+)?/events }

With the following Dockerfile to override the upstream CMD

FROM haproxy:2.2-alpine

EXPOSE 2375
ENV ALLOW_RESTARTS=0 \
    ALLOW_STOP=0 \
    ALLOW_START=0 \
    AUTH=0 \
    BUILD=0 \
    COMMIT=0 \
    CONFIGS=0 \
    CONTAINERS=0 \
    DISTRIBUTION=0 \
    ENABLE_IPV6=0 \
    EVENTS=1 \
    EXEC=0 \
    GRPC=0 \
    IMAGES=0 \
    INFO=0 \
    LOG_LEVEL=info \
    NETWORKS=0 \
    NODES=0 \
    PING=1 \
    PLUGINS=0 \
    POST=0 \
    SECRETS=0 \
    SERVICES=0 \
    SESSION=0 \
    SOCKET_PATH=/var/run/docker.sock \
    SWARM=0 \
    SYSTEM=0 \
    TASKS=0 \
    VERSION=1 \
    VOLUMES=0
RUN apk add --no-cache gettext # remove if using sed
COPY docker-entrypoint.sh /
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg.template

ENTRYPOINT ["/docker-entrypoint.sh"]

sed option results in shaving 2-3MB off the image size but is less flexible if you want to append stuff to the template later but either way works for this particular edit.

[saltydk]

@jscheytt test image for the above is:

saltydk/docker-socket-proxy:entrypoint

[jscheytt]

@saltydk Works like a charm! Thank you 😊

[saltydk]

I've made a PR with the suggested fix, please comment there if you want to switch the chosen implementation, like if you don't want to install gettext.

[jscheytt]

Hey there, just wanted to check back if there is any chance to get the PR by @saltydk merged? Would very much appreciate the fix. @yajo Does this need approval e.g. from you or another maintainer?

[yajo]

I don't maintain this anymore, it must be somebody from @Tecnativa.

[sammcj]

Any update on getting this merged? It's currently broken for my setup without it.

[jscheytt]

Newest version works for me when setting DISABLE_IPV6=1 (see #111 (comment))