Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When files.block() starts its marker with a tab char, the block is inserted every execution #1258

Open
jlamoure opened this issue Dec 23, 2024 · 0 comments
Open

When files.block() starts its marker with a tab char, the block is inserted every execution #1258

jlamoure opened this issue Dec 23, 2024 · 0 comments

Comments

@jlamoure
Copy link

Describe the bug

When using files.block() with a custom marker starting with a tab character, the operation is no longer idempotent and will insert the block text every time the deploy is executed.

To Reproduce

Run on a Debian 12 system, using python-poetry to manage the environment, with Pyinfra 3.1.1.

System: Linux
Platform: Linux-6.1.0-26-amd64-x86_64-with-glibc2.36
Release: 6.1.0-26-amd64
Machine: x86_64
pyinfra: v3.1.1
click: v8.1.7
configparser: v7.1.0
distro: v1.9.0
gevent: v24.2.1
jinja2: v3.1.4
packaging: v24.2
paramiko: v3.5.0
python-dateutil: v2.9.0.post0
pywinrm: v0.5.0
setuptools: v75.3.0
typeguard: v4.4.0
typing-extensions: v4.12.2
Executable: /home/j/.cache/pypoetry/virtualenvs/censored/bin/pyinfra
Python: 3.11.2 (CPython, GCC 12.2.0)

Run the following operation:

grafana_nginx_config = """
        # This is required to proxy Grafana Live WebSocket connections.
        map $http_upgrade $connection_upgrade {
                default upgrade;
                '' close;
        }

        upstream grafana {
                server localhost:3000;
        }
        
        server {
                listen 443 ssl;
                root /usr/share/nginx/html;
                index index.html index.htm;
        
                location / {
                        proxy_set_header Host $host;
                        proxy_pass https://grafana;
                }
        
                # Proxy Grafana Live WebSocket connections.
                location /api/live/ {
                        proxy_http_version 1.1;
                        proxy_set_header Upgrade $http_upgrade;
                        proxy_set_header Connection $connection_upgrade;
                        proxy_set_header Host $host;
                        proxy_pass https://grafana;
                }
        }"""
files.block(
    path='/etc/nginx/nginx.conf',
    content=grafana_nginx_config,
    line='http {',
    after=True,
    escape_regex_characters=True,
    marker='    # {mark} PYINFRA BLOCK',
    _sudo=True)

Note that on copy-paste, actual tabs here have been converted to spaces.

Expected behavior

When this operation is run against the host for the first time, the block is inserted into the specified configuration file.
When this operation is run again, no changes occur.

Actual behavior

The text block is inserted into the configuration file each time the operation is run.
This behavior goes away and idempotency restored when the single tab character is removed from the beginning of the marker.

However, the end software requires this configuration be indented, thus I must push the whole file as a workaround instead of modifying in place.

output with -vv and --debug on first run

--> Starting operation: roles/configure-grafana.py | files.block (path=/etc/nginx/nginx.conf, content=
	# This is required to proxy Grafana Live WebSocket connections.
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'' close;
	}

	upstream grafana {
		server localhost:3000;
	}
	
	server {
		listen 443 ssl;
		root /usr/share/nginx/html;
		index index.html index.htm;
	
		location / {
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	
		# Proxy Grafana Live WebSocket connections.
		location /api/live/ {
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	}, line=http {, after=True, escape_regex_characters=True, marker=	# {mark} PYINFRA BLOCK)
    [pyinfra.api.operations] Starting operation {'roles/configure-grafana.py | files.block'} on grafana
    The `files.block` operation is currently in beta!
    [pyinfra.api.facts] Getting fact: files.Block (begin=None, end=None, marker=	# {mark} PYINFRA BLOCK, path=/etc/nginx/nginx.conf) (ensure_hosts: None)
    [pyinfra.connectors.ssh] Running command on grafana: (pty=False) env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-6mtQgixA9rXa *** sudo -H -A -k sh -c 'awk '"'"'/ # END PYINFRA BLOCK/{ f=0} f; / # BEGIN PYINFRA BLOCK/{ f=1} '"'"' /etc/nginx/nginx.conf || (find /etc/nginx/nginx.conf -type f > /dev/null && echo __pyinfra_exists_/etc/nginx/nginx.conf || echo __pyinfra_missing_/etc/nginx/nginx.conf )'
[grafana] >>> env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-6mtQgixA9rXa *** sudo -H -A -k sh -c 'awk '"'"'/ # END PYINFRA BLOCK/{ f=0} f; / # BEGIN PYINFRA BLOCK/{ f=1} '"'"' /etc/nginx/nginx.conf || (find /etc/nginx/nginx.conf -type f > /dev/null && echo __pyinfra_exists_/etc/nginx/nginx.conf || echo __pyinfra_missing_/etc/nginx/nginx.conf )'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [grafana] Loaded fact files.Block (begin=None, end=None, marker=	# {mark} PYINFRA BLOCK, path=/etc/nginx/nginx.conf)
    [pyinfra.connectors.ssh] Running command on grafana: (pty=False) env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-6mtQgixA9rXa *** sudo -H -A -k sh -c 'OUT="$(TMPDIR=/tmp mktemp -t pyinfra.XXXXXX)" &&  awk '"'"'BEGIN {x=ARGV[2]; ARGV[2]=""} { print } f!=1 && /^.*http \{.*$/ { print x; f=1} END {if (f==0) print ARGV[2] } '"'"' /etc/nginx/nginx.conf "	# BEGIN PYINFRA BLOCK

	# This is required to proxy Grafana Live WebSocket connections.
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'"'"''"'"' close;
	}

	upstream grafana {
		server localhost:3000;
	}
	
	server {
		listen 443 ssl;
		root /usr/share/nginx/html;
		index index.html index.htm;
	
		location / {
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	
		# Proxy Grafana Live WebSocket connections.
		location /api/live/ {
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	}
	# END PYINFRA BLOCK" > $OUT && chmod $(stat -c %a /etc/nginx/nginx.conf 2>/dev/null || stat -f %Lp /etc/nginx/nginx.conf ) $OUT &&  (chown $(stat -c "%U:%G" /etc/nginx/nginx.conf 2>/dev/null) $OUT ||  chown -n $(stat -f "%u:%g" /etc/nginx/nginx.conf ) $OUT)  && mv "$OUT" /etc/nginx/nginx.conf'
[grafana] >>> env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-6mtQgixA9rXa *** sudo -H -A -k sh -c 'OUT="$(TMPDIR=/tmp mktemp -t pyinfra.XXXXXX)" &&  awk '"'"'BEGIN {x=ARGV[2]; ARGV[2]=""} { print } f!=1 && /^.*http \{.*$/ { print x; f=1} END {if (f==0) print ARGV[2] } '"'"' /etc/nginx/nginx.conf "	# BEGIN PYINFRA BLOCK

	# This is required to proxy Grafana Live WebSocket connections.
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'"'"''"'"' close;
	}

	upstream grafana {
		server localhost:3000;
	}
	
	server {
		listen 443 ssl;
		root /usr/share/nginx/html;
		index index.html index.htm;
	
		location / {
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	
		# Proxy Grafana Live WebSocket connections.
		location /api/live/ {
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	}
	# END PYINFRA BLOCK" > $OUT && chmod $(stat -c %a /etc/nginx/nginx.conf 2>/dev/null || stat -f %Lp /etc/nginx/nginx.conf ) $OUT &&  (chown $(stat -c "%U:%G" /etc/nginx/nginx.conf 2>/dev/null) $OUT ||  chown -n $(stat -f "%u:%g" /etc/nginx/nginx.conf ) $OUT)  && mv "$OUT" /etc/nginx/nginx.conf'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [grafana] Success

output with -vv and --debug on second run

--> Starting operation: roles/configure-grafana.py | files.block (path=/etc/nginx/nginx.conf, content=
	# This is required to proxy Grafana Live WebSocket connections.
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'' close;
	}

	upstream grafana {
		server localhost:3000;
	}
	
	server {
		listen 443 ssl;
		root /usr/share/nginx/html;
		index index.html index.htm;
	
		location / {
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	
		# Proxy Grafana Live WebSocket connections.
		location /api/live/ {
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	}, line=http {, after=True, escape_regex_characters=True, marker=	# {mark} PYINFRA BLOCK)
    [pyinfra.api.operations] Starting operation {'roles/configure-grafana.py | files.block'} on grafana
    The `files.block` operation is currently in beta!
    [pyinfra.api.facts] Getting fact: files.Block (begin=None, end=None, marker=	# {mark} PYINFRA BLOCK, path=/etc/nginx/nginx.conf) (ensure_hosts: None)
    [pyinfra.connectors.ssh] Running command on grafana: (pty=False) env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-7JJOuIQgZP82 *** sudo -H -A -k sh -c 'awk '"'"'/ # END PYINFRA BLOCK/{ f=0} f; / # BEGIN PYINFRA BLOCK/{ f=1} '"'"' /etc/nginx/nginx.conf || (find /etc/nginx/nginx.conf -type f > /dev/null && echo __pyinfra_exists_/etc/nginx/nginx.conf || echo __pyinfra_missing_/etc/nginx/nginx.conf )'
[grafana] >>> env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-7JJOuIQgZP82 *** sudo -H -A -k sh -c 'awk '"'"'/ # END PYINFRA BLOCK/{ f=0} f; / # BEGIN PYINFRA BLOCK/{ f=1} '"'"' /etc/nginx/nginx.conf || (find /etc/nginx/nginx.conf -type f > /dev/null && echo __pyinfra_exists_/etc/nginx/nginx.conf || echo __pyinfra_missing_/etc/nginx/nginx.conf )'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [grafana] Loaded fact files.Block (begin=None, end=None, marker=	# {mark} PYINFRA BLOCK, path=/etc/nginx/nginx.conf)
    [pyinfra.connectors.ssh] Running command on grafana: (pty=False) env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-7JJOuIQgZP82 *** sudo -H -A -k sh -c 'OUT="$(TMPDIR=/tmp mktemp -t pyinfra.XXXXXX)" &&  awk '"'"'BEGIN {x=ARGV[2]; ARGV[2]=""} { print } f!=1 && /^.*http \{.*$/ { print x; f=1} END {if (f==0) print ARGV[2] } '"'"' /etc/nginx/nginx.conf "	# BEGIN PYINFRA BLOCK

	# This is required to proxy Grafana Live WebSocket connections.
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'"'"''"'"' close;
	}

	upstream grafana {
		server localhost:3000;
	}
	
	server {
		listen 443 ssl;
		root /usr/share/nginx/html;
		index index.html index.htm;
	
		location / {
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	
		# Proxy Grafana Live WebSocket connections.
		location /api/live/ {
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	}
	# END PYINFRA BLOCK" > $OUT && chmod $(stat -c %a /etc/nginx/nginx.conf 2>/dev/null || stat -f %Lp /etc/nginx/nginx.conf ) $OUT &&  (chown $(stat -c "%U:%G" /etc/nginx/nginx.conf 2>/dev/null) $OUT ||  chown -n $(stat -f "%u:%g" /etc/nginx/nginx.conf ) $OUT)  && mv "$OUT" /etc/nginx/nginx.conf'
[grafana] >>> env SUDO_ASKPASS=/tmp/pyinfra-sudo-askpass-7JJOuIQgZP82 *** sudo -H -A -k sh -c 'OUT="$(TMPDIR=/tmp mktemp -t pyinfra.XXXXXX)" &&  awk '"'"'BEGIN {x=ARGV[2]; ARGV[2]=""} { print } f!=1 && /^.*http \{.*$/ { print x; f=1} END {if (f==0) print ARGV[2] } '"'"' /etc/nginx/nginx.conf "	# BEGIN PYINFRA BLOCK

	# This is required to proxy Grafana Live WebSocket connections.
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'"'"''"'"' close;
	}

	upstream grafana {
		server localhost:3000;
	}
	
	server {
		listen 443 ssl;
		root /usr/share/nginx/html;
		index index.html index.htm;
	
		location / {
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	
		# Proxy Grafana Live WebSocket connections.
		location /api/live/ {
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;
			proxy_pass https://grafana;
		}
	}
	# END PYINFRA BLOCK" > $OUT && chmod $(stat -c %a /etc/nginx/nginx.conf 2>/dev/null || stat -f %Lp /etc/nginx/nginx.conf ) $OUT &&  (chown $(stat -c "%U:%G" /etc/nginx/nginx.conf 2>/dev/null) $OUT ||  chown -n $(stat -f "%u:%g" /etc/nginx/nginx.conf ) $OUT)  && mv "$OUT" /etc/nginx/nginx.conf'
    [pyinfra.connectors.ssh] Waiting for exit status...
    [pyinfra.connectors.ssh] Command exit status: 0
    [grafana] Success
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jlamoure