Skip to content

Commit

Permalink
Merge branch 'ease_of_use_features' into 'master'
Browse files Browse the repository at this point in the history 
More ease of use features

See merge request evernym/utilities/devlab!15
  • Loading branch information
@absltkaos
absltkaos committed Aug 2, 2022
2 parents e320961 + c98d102 commit 31b5cdf
Show file tree
Hide file tree
Showing 5 changed files with 73 additions and 32 deletions.
10 changes: 9 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ pip3 install pyyaml
The configuration file has the following base structure:
```
{
"min_devlab_version": "",
"components": {},
"domain": "",
"foreground_component": {},
Expand All @@ -115,6 +116,7 @@ All Keys that are in **bold** are required to be in the config
| domain | String | The domain name to assign to all component's hostname inside the container |
| **components** | Hash of Hashes | Defines the components to start up. The First level key is a string of the name of the container. Structure conforms to the [Component Config Structure](#component-config-structure) |
| foreground_component | Hash | Defines a component that will be startup up after ***all*** other components and will run in the foreground. After the process exits, all ther components will be stopped. Same structure as [Component Config Structure](#component-config-structure) with one additional key `name` to indicate the name of the foreground component |
| min_devlab_version | String | The minimum version of devlab that the project requires. Useful when taking advantage of new features and ensuring users of your project are updated to a version that supports the features you're using |
| network | Hash | Defines a docker network to create and/or attach components to. Structure conforms to [Network Config Structure](#network-config-structure) |
| **paths** | Hash | Defines the persistence directory for components, as well as files that should be deleted during the [reset](#reset-action) action. Structure conforms to [Paths Config Structure](#paths-config-structure) |
| **project_filter** | String | A unique docker label that is used to identify containers and images that belong to the project. |
Expand All @@ -130,6 +132,8 @@ The structure looks like this:
"systemd_support": false,
"systemd_tmpfs_args": "",
"enabled": false,
"env": {},
"env_file": "",
"cmd": "",
"ports": [],
"mounts": [],
Expand Down Expand Up @@ -157,7 +161,9 @@ All Keys that are in **bold** are required
| systemd_tmpfs_args | String | If `systemd_support` is set to `true`, and this argument is set, then the value is appended to the tmpfs mounts as arguments for systemd support. This way you can specify things like: `rw`, `exec`, etc... |
| **enabled** | Boolean | Whether or not the component should be brought [up](#up-action) and images [built](#build-action) |
| **_name_** | String | This is only supported for `foreground_components` but required. It indicates the name of the component |
| type | String | This only only supported for `foreground_components`, but can be either `host` or `container`. If set to host then `cmd` is executed on the local system instead of a container |
| type | String | This is only supported for `foreground_components`, but can be either `host` or `container`. If set to host then `cmd` is executed on the local system instead of a container |
| env | Hash | Key value pairs of environment variables to set for the component |
| env_file | String | Path to a file containing environment variables for the component. This fills in the `--env-file` option for the `docker run` command
| cmd | String | This is the command passed to the container as part of the `docker run` command. If `type` is set to `host` then the command is executed on the local system |
| ports | List of Strings | The ports that should be "published" using the same notation as the `--publish` option to `docker run` |
| mounts | List of Strings | List of mounts in the format `SOURCE_ON_HOST:DESTINATION_IN_CONTAINER`. If using a relative path then the paths are relative to the project's root |
Expand Down Expand Up @@ -220,6 +226,7 @@ The structure looks like this:
"tag": ""|[],
"docker_file": "",
"build_opts": [],
"skip_pull": BOOL,
"ordinal": {
"group": INT,
"number": INT
Expand All @@ -233,6 +240,7 @@ All Keys that are in **bold** are required
| **tag** | String or List of Strings | This is a tag that should be applied to the image. If a list is passed, the first tag becomes a primary identifier. |
| **docker_file** | String | Path to the docker file, relative to the project's root to use when building the image. ***[NOTE]*** The build context will be the parent directory of the dockerfile's path |
| build_opts | List of Strings | Additional options to pass to the `docker build` command. Each CLI arg must be it's own element. For example: `[ '--build-arg', 'foo=bar' ]` would become `docker build --build-arg foo=bar PATH...` etc... |
| skip_pull | Boolean | Whether or not a forced pull does anything. There are cases where an image is built locally used elsewhere, so a pull will fail since it isn't on docker hub. If this is `true`, then even when a build is requesting a `pull` it will skip it for this image |
| ordinal | Hash | This is used indicate the order of the images to build. When parallel execution is supported, the `group` key indicates the image that can be built at the same time, `number` indicates the order inside the group to start up |

_**[NOTE]**_ Devlab supports a special label (`last_modified`).
Expand Down
17 changes: 17 additions & 0 deletions devlab
View file
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,23 @@ if __name__ == '__main__':
LOGGER.error("No configured components found!... aborting")
sys.exit(1)

#Check min devlab version if set
if devlab_bench.CONFIG.get('min_devlab_version', None):
MIN_DEVLAB_VERSION = devlab_bench.CONFIG['min_devlab_version']
#Assume that "master" version is newer that min version and only if the version doesn't match
if __VERSION__ not in ["master", MIN_DEVLAB_VERSION]:
VERS_SORT = sorted(
[
__VERSION__,
MIN_DEVLAB_VERSION
],
key=devlab_bench.helpers.common.human_keys
)
if VERS_SORT[-1] != __VERSION__:
LOGGER.error("This devlab project requuires a minimum version of: '%s' Found: '%s' installed. Please upgrade", MIN_DEVLAB_VERSION, __VERSION__)
sys.exit(1)
LOGGER.debug("Current version of devlab: '%s' matches or excedes required minimum version: '%s'", __VERSION__, MIN_DEVLAB_VERSION)

#Create our DockerHelper Object
devlab_bench.helpers.docker.DOCKER = DockerHelper(
filter_label=devlab_bench.CONFIG['project_filter'],
Expand Down
27 changes: 17 additions & 10 deletions devlab_bench/actions/build.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -135,16 +135,23 @@ def action(images='*', clean=False, no_cache=False, pull=False, skip_pull_images
log.debug(line)
log.debug("Successfully removed image: %s", image)
if pull:
with open(images_dict[image]['docker_file_full_path']) as dfile:
local_image = False
for line in dfile.readlines():
if line.startswith('FROM '):
if line.split()[1].split(':')[0] in images_to_build + skip_pull_images:
local_image = True
log.debug("Skipping pull, as devlab manages this image's base image")
break
if not local_image:
images_dict[image]['build_opts'].append('--pull')
local_image = False
if image in skip_pull_images:
log.info("Image: %s was explicitly excluded from being pulled from a calling function, Skipping", image)
local_image = True
elif image not in base_images_to_build and config['runtime_images'][image].get('skip_pull', False):
log.info("Runtime image: %s is explicitely set to not be pulled in Devlabconfig, skipping pull argument when building", image)
local_image = True
else:
with open(images_dict[image]['docker_file_full_path']) as dfile:
for line in dfile.readlines():
if line.startswith('FROM '):
if line.split()[1].split(':')[0] in images_to_build + skip_pull_images:
local_image = True
log.debug("Skipping pull, as devlab manages this image's base image")
break
if not local_image:
images_dict[image]['build_opts'].append('--pull')
if no_cache:
images_dict[image]['build_opts'].append('--no-cache')
log.info("Building image: %s", image_n_tag)
Expand Down
39 changes: 20 additions & 19 deletions devlab_bench/helpers/common.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
#Python2/3 compatibility
try:
#Python2
text_input = raw_input #pylint: disable=invalid-name
text_input = globals()['__builtins__'].raw_input #pylint: disable=invalid-name
from pipes import quote #pylint: disable=unused-import
try:
from pathlib2 import Path #pylint: disable=unused-import
Expand Down Expand Up @@ -262,21 +262,6 @@ def get_ordinal_sorting(components, config_components):
num = 100
ordinals['{}:{}|{}'.format(grp, num, comp)] = comp
log.debug("Ordinals found for components: %s", ordinals)
def human_keys(astr):
"""
Sorts keys based on human order.. IE 1 is less than 10 etc..
alist.sort(key=human_keys) sorts in human order
"""
keys = []
for elt in re.split(r'(\d+)', astr):
elt = elt.swapcase()
try:
elt = int(elt)
except ValueError:
pass
keys.append(elt)
return keys
#Get the list of ordinals, and human sort them
ordinal_list = sorted(tuple(ordinals.keys()), key=human_keys)
log.debug("Sorted list of ordinals: '%s'", ', '.join(ordinal_list))
Expand Down Expand Up @@ -328,7 +313,7 @@ def get_proj_root(start_dir=None):
start_dir = os.path.abspath(start_dir)
cur_dir = start_dir
found = False
while cur_dir != None:
while cur_dir is not None:
if os.path.basename(cur_dir) != 'defaults':
for cfile_name in devlab_bench.CONFIG_FILE_NAMES:
if os.path.isfile('{}/{}'.format(cur_dir, cfile_name)):
Expand Down Expand Up @@ -368,6 +353,22 @@ def get_shell_components(filter_list):
"""
return get_components(filter_list=filter_list, virtual_components=('adhoc',))

def human_keys(astr):
"""
Sorts keys based on human order.. IE 1 is less than 10 etc..
alist.sort(key=human_keys) sorts in human order
"""
keys = []
for elt in re.split(r'(\d+)', astr):
elt = elt.swapcase()
try:
elt = int(elt)
except ValueError:
pass
keys.append(elt)
return keys

def is_valid_hostname(hostname):
"""
Takes a hostname and tries to determine if it is valid or not
Expand Down Expand Up @@ -540,8 +541,6 @@ def script_runner(script, name, ignore_nonzero_rc=False, interactive=True, log_o
if '=' in script_arg:
if not script_end_env:
log.debug("Found environment variable for script: '%s'", script_arg)
script_run_opts.append('-e')
script_run_opts.append(script_arg)
e_var, e_val = script_arg.split('=')
env_map[e_var] = e_val
continue
Expand Down Expand Up @@ -570,6 +569,7 @@ def script_runner(script, name, ignore_nonzero_rc=False, interactive=True, log_o
mounts=[
'{}:/devlab'.format(devlab_bench.PROJ_ROOT)
],
env=env_map,
background=False,
interactive=interactive,
cmd=script_stripped,
Expand All @@ -595,6 +595,7 @@ def script_runner(script, name, ignore_nonzero_rc=False, interactive=True, log_o
name=name,
background=False,
interactive=interactive,
env=env_map,
cmd=script_stripped,
ignore_nonzero_rc=ignore_nonzero_rc,
logger=log,
Expand Down
12 changes: 10 additions & 2 deletions devlab_bench/helpers/docker.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -452,7 +452,7 @@ def rm_image(self, name):
logger=self.log
).run()
return cmd_ret
def run_container(self, image, name, network=None, ports=None, background=True, interactive=False, ignore_nonzero_rc=False, cmd=None, logger=None, mounts=None, systemd_support=False, systemd_tmpfs_args=None, run_opts=None, **kwargs): #pylint: disable=too-many-arguments
def run_container(self, image, name, network=None, ports=None, background=True, env=None, env_file=None, interactive=False, ignore_nonzero_rc=False, cmd=None, logger=None, mounts=None, systemd_support=False, systemd_tmpfs_args=None, run_opts=None, **kwargs): #pylint: disable=too-many-arguments
"""
Run a docker_container
Expand All @@ -461,6 +461,8 @@ def run_container(self, image, name, network=None, ports=None, background=True,
name: str, The name of the container (this also sets the hostname)
network: str, docker network to attach
cmd: str, Command to run inside the container. (OPTIONAL)
env: dict, key/values of environment vars to set (OPTIONAL)
env_file: str, path to a file to set environment vars (OPTIONAL)
ports: list/tuple, of ports to publish to the host. (OPTIONAL)
background: Run the container in the background. (OPTIONAL)
interactive: bool, whether or not the docker command could require
Expand Down Expand Up @@ -499,9 +501,15 @@ def run_container(self, image, name, network=None, ports=None, background=True,
opts.append("--detach")
if network:
opts.append("--network={}".format(network))
if env:
for e_var, e_val in env.items():
opts.append("--env")
opts.append("{}={}".format(e_var, e_val))
if env_file:
opts.append("--env-file={}".format(env_file))
if systemd_support:
if systemd_tmpfs_args:
systemd_tmpfs_args=':{}'.format(systemd_tmpfs_args)
systemd_tmpfs_args = ':{}'.format(systemd_tmpfs_args)
opts += [
'--tmpfs=/run{}'.format(systemd_tmpfs_args),
'--tmpfs=/run/lock{}'.format(systemd_tmpfs_args),
Expand Down

0 comments on commit 31b5cdf

Please sign in to comment.