Skip to content
This repository has been archived by the owner on Aug 18, 2023. It is now read-only.
/ bamboo-specs Public archive

A programme for specifying Atlassian Bamboo plans as YAML files

License

Notifications You must be signed in to change notification settings

reecetech/bamboo-specs

Repository files navigation

Bamboo Plan and Deployment Configuration

Build Status

This project provides a tool that uses YAML files to specify how Bamboo plans and deployment projects should be configured.

Sample configuration files

See the specifications directory for sample files (patersoa TODO: add more examples)

Credentials and Authentication

Before running this program you need to configure an admin user to the user the program will run as (ie. your Bamboo credentials) You do this by creating a .credentials file with the following contents:

username=<admin user username>
password=<admin user password>

Don't check this into a repository.

Building and Running

Build the code with:

mvn package

Run with:

java -jar target/bamboo-specs-reece-2.0.0.jar permissions.yaml plan.yaml deployment-project.yaml

You can test your YAML using the -t switch passed to any of those commands, for example:

java -jar target/bamboo-specs-reece-2.0.0.jar -t plan.yaml

This will just parse the YAML and not deploy it to Bamboo.

The commands all accept multiple yaml files to process:

java -jar target/bamboo-specs-reece-2.0.0.jar configs/plan-*.yaml

Authorizing Bamboo to access Stash

Visit this URL: https://${bamboo-server}/bamboo/admin/configureLinkedRepositories!doDefault.action

and then click Add Repository and select BitBucket Server/Stash. If you haven't authorized it already, you will be prompted to enter your credentials.

Java SSL keystore fix

If you get this error (or anything mentioning a certificate) when running the jar files, you need to add your corporate CA cert to your Java keystore:

INFO [BambooServer] An error occurred while publishing plan AS-BK8SD

On Ubuntu:
----------
Visit https://bamboo.reecenet.org in Chrome
Use developer tools - Security - View Certificate - details tab - export
This will save a copy of the public certificate to a file
(vicpjdt01.reecenet.org in the example below)
Import that file using:
keytool -import -alias vicpjdt01.reecenet.org -keystore cacerts \
-trustcacerts -file ~dev/vicpjdt01.reecenet.org
This will prompt for a password; the default is "changeit"

On Windows:
-----------
keytool -importcert -alias vicpjdt01.reecenet.org \
-file vicpjdt01.reecenet.org.cer -keystore "C:\Program Files (x86)\Java\jre1.8.0_131\lib\security\cacerts"

On macOS:
------------
Download the certificates `reecenet-ca.crt` and `reecenet-intermediate.crt` from https://stash.reecenet.org/projects/DBI/repos/rhel7_java_base/browse and place them under `/tmp`.

Find the Java home:
$ /usr/libexec/java_home

Then import the cert:
$ cd /Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre/lib/security
$ sudo cp cacerts cacerts.orig
$ sudo keytool -trustcacerts -keystore cacerts -noprompt -alias reecenet-base -importcert -file /tmp/reecenet-ca.crt -storepass changeit
$ sudo keytool -trustcacerts -keystore cacerts -noprompt -alias reecenet-intermediate -importcert -file /tmp/reecenet-intermediate.crt -storepass changeit 

Controlling Permissions

Create a permissions.yaml file:

specType: permissions
bambooServer: https://bamboo.reecenet.org/bamboo
projects:
- plans: [BST-ST, SPAM-IT]
  permissions:
  - groups: [Cyborg_Team]
    users: [islaa, tobind]
    grant: [VIEW, EDIT, BUILD, CLONE, ADMIN]
- plans: [BST-ST]
  permissions:
  - users: [dooleyj]
    grant: [VIEW]
- plans: [SPAM-IT]
  permissions:
  - allLoggedInUsers: true
    grant: [VIEW]
deployments:
- name: Diary Notes Python Shared Service
  permissions:
  - users: [dooleyj, poultonj]
    grant: [VIEW, EDIT]
  environments:
  - names: [Production (AU + NZ), TEST AU, TEST NZ]
    permissions:
    - groups: [Cyborg_Team]
      grant: [VIEW, EDIT, BUILD]
- name: Customer Prices Service
  permissions:
  - users: [dooleyj, poultonj]
    grant: [VIEW, EDIT]
  environments:
  - names: [Production (AU + NZ)]
    permissions:
    - groups: [Cyborg_Team]
      grant: [VIEW, EDIT, BUILD]

There can be many entries in the permissions: list, each of which specifies groups and/or users and permissions to grant to them.

The projects list contains plans identified by key pairs (project key, plan key) so the first list has the ST plan in the BST project, and the IT plan in the SPAM project. These key pairs are at the end of the URL when viewing a project's plan, eg:

https://bamboo.reecenet.org/bamboo/browse/DNSS-DNPSM

The optional deployments list contains deployments and environments identified by their label in the Bamboo interface. Note that permissions granted to a deployment are only for administrating the deployment project settings in Bamboo and do not affect the access controls for each of the environments.

Each permission will be granted to each group and user in each plan, deployment project or environment for a given permissions entry. So for the first group above, the complete set of permissions will be granted to the Cyborg_Team group and users islaa and tobind in the BST-ST and SPAM-IT plans.

The allowed permissions for each section are:

  • plans: VIEW, EDIT, BUILD, CLONE, ADMIN
  • deployments: VIEW, EDIT
  • environments: VIEW, EDIT, BUILD

The admin user used to make the changes (see Credentials and Authentication) is hard-coded to be granted admin user permission regardless of the other settings in the permissions yaml, to prevent that user from having that permission removed (which would break the program).

Build and Test Plans

Plans have a lot more options. The required minimum is:

specType: plan
bambooServer: https://bamboo.reecenet.org/bamboo
projectKey: BST
projectName: Bamboo Spec Testing
planKey: ST
planName: Spec Testing
description: This is a test plan for bamboo specs
maximumConcurrentBuilds: 1

If the Plan or Project do not exist in Bamboo they will be created, so please double-check that the projectKey and planKey are correct.

The rest of the configuration is all optional chunks, though some will depend on others (VCS tasks would require a repository, for example).

If you have arbitrary variables stored on a plan you may set them as key-value pairs like so:

variables:
  major_version_number: 1
  target_name: bamboo-spec-testing

Variables defined here (and others defined by Bamboo for you) may be reference in SCRIPT task body texts using ${bamboo.major_version_number} or ${bamboo.target_name} using the above example settings.

You can also add labels to your build plan by simply adding a list of strings you would like your build tagged with:

labels:
   - awesome
   - very_cool

Source Repositories

If there are repositories used then include as either linked repositories (shared between plans):

linkedRepositories: [Bamboo Spec Test Project, Other Repository]

The linked repository is typically added when a plan is created. Alternatively you can use a locally (to this plan) defined repositories:

repositories:
- name: Bamboo Spec Test Project
  projectKey: SAN
  repositorySlug: bamboo-spec-test-project
  branch: development
  triggerPattern: /files/i/care/about/*
  shallowClone: false
- name: PACT Specification
  gitURL: https://github.com/pact-foundation/pact-specification.git
  branch: version-1.1
  path: pact-spec-version-1.1

So the two types of repositories currently supported are:

  1. A repository in the Reece Stash instance. It's identified by the projectKey and repositorySlug from the repository URL like so:

    https://stash.reecenet.org/projects/\<projectKey>/repos/<repositorySlug>/browse

    The "branch" is optional (default is "master"). Additionally, you can specify a 'trigger pattern' that will cause the build to fire only if the changed files match the pattern (a regex).

    You can also specify whether or not the clone operation is 'shallow'. By default, the clone is shallow.

  2. An arbitrary git repository identified by gitURL. It must specify a path that the repository will be cloned to, and optionally a branch.

Plan branches are local configurations based on branches in the repository and the strategy for synchronising the two are controlled with:

branchManagement:
  createStrategy: MANUALLY

The creation strategy is one of: MANUALLY, ON_PULL_REQUEST, ON_NEW_BRANCH or ON_BRANCH_PATTERN. The last will create on new branches matching a name pattern regular expression:

branchManagement:
  createStrategy: ON_BRANCH_PATTERN
  branchPattern: feature/.*

The issueLinkingEnabled option enables automatic linking of the plan branch to the Jira issue related to the repository branch, which is enabled by default. Cleaning up plan branches is defaulted to 7 days after the repository branch is deleted. The default is to never clean up branches that are simply inactive. These options may be modified in the branchManagement section:

branchManagement:
  issueLinkingEnabled: false
  delayCleanAfterDelete: 2
  delayCleanAfterInactivity: 60

Triggers

Plans may also have a triggers section to indicate the specific circumstances in which they are to be triggered (that is, their tasks should be executed), say running unit tests after commits to the stash repository:

triggers:
- type: AFTER_STASH_COMMIT
  description: Trigger from stash changes

Or perhaps trigger a deploy from a successful build:

triggers:
- type: AFTER_SUCCESSFUL_BUILD_PLAN
  description: Deploy main plan branch (master)

You may also trigger only off certain non-master branches:

triggers:
- type: AFTER_SUCCESSFUL_BUILD_PLAN
  branch: development
  description: Deploy development branch

Or scheduled using a variety of periods:

triggers:
- type: SCHEDULED
  description: Test every 4 hours
  everyNumHours: 4
  dailyAt: 5:00
  weeklyAt: Saturday 12:59
  montlyAt: 15 12:59
  cron: "0 0/30 9-19 ? * MON-FRI"

If the plan has dependent plans which are to be triggered when this plan completes they may be specified (as "dependencies"):

dependencies:
  requiresPassing: true
  plans: [USRSRV-UPSDB]

If there are no dependencies you may leave this section out, though if the plan previously had dependencies you will need to explicitly clear them with:

dependencies:
  none: true

If you don't then you'll get an error "Plan import is blocked to prevent deleting your plan dependencies silently."

Notifications

Notifications on plan completion are supported:

notifications:
- when: PLAN_COMPLETED
  slack: https://hooks.slack.com/services/...the rest of the URL...|#cyborg-dev
  recipientGroups: [Cyborg_Team]
  recipientUsers: [dooleyj, poultonj]
  responsibleUser: "true"

At least one of the notification targets is required: slack, recipientGroups, responsibleUser or recipientUsers. The when values are PLAN_COMPLETED, PLAN_FAILED, STATUS_CHANGED, DEPLOYMENT_FAILEDandDEPLOYMENT_FINISHED` which mirror the options of the same name in the Bamboo UI.

Stages, Jobs and Tasks

Your plan may have multiple stages, which each have jobs, and each job may have tasks and final tasks (tasks to run even if the other tasks fail).

Stages and jobs may be defined:

stages:
- name: Default Stage
  jobs:
  - name: Run Tests
    key: JOB1
    description: Run Python Unit Tests
    requirements:
    - name: system.docker.executable
    - name: DOCKER
    - name: LINUX
    artifacts:
    - name: PACT Contracts
      pattern: "**"
      location: pacts
    - name: Coverage Report
      pattern: "**"
      location: htmlcov
    - name: Docker Image
      pattern: built-image.docker
      location: .
      shared: true

The job key is arbitrary and unique inside a plan. Requirements and artifacts are optional.

The requirements are restrictions on which Bamboo agents may be used to run the plan's jobs. The actual list of requirements possible is available in the Bamboo UI, though the precise key to be used in the list above is unclear for the requirements build into Bamboo. For example, in the Bamboo UI you may select the built-in "Docker" requirement, which in the list above is "system.docker.executable". The UI also lists "DOCKER" which is represented in the list above with the same name. I recommend adding the requirement through the UI and using the "View plan as Bamboo Specs" option under Plan Configuration Actions menu to determine the actual string to use in the requirements list in YAML.

Artifacts must be shared: true if you wish to use them in other jobs. You must then subscribe to the artifact from the other job using artifactSubscriptions:

stages:
- name: Second Stage
  jobs:
  - name: Run More Tests
    artifactSubscriptions:
    - name: Docker Image
      destination: .

A job may then have a list of tasks:

stages:
- name: Default Stage
  jobs:
  - name: Run Tests
    ...
    tasks:
    - type: VCS
      description: Checkout Default Repository
      cleanCheckout: true
    - type: SCRIPT
      description: Build docker image
      body: |
        set -ex
        scripts/test_image.sh bamboo/${bamboo.target_name}
    - type: SCRIPT
      description: Run tests
      body: |
        set -ex
        scripts/run_tests.sh

Here you can see we refer to the bamboo variable we defined way up above so that the script body may be the same across multiple projects.

VCS Task

The VCS task has a number of options. By default it will check out the default repository for the plan. If you wish to check out other repositories you may list them (and optionally include the default repository also):

- type: VCS
  description: Checkout All Repositories
  defaultRepository: true
  repositories:
  - name: Running Man
  - name: Running Man Properties
    path: properties
  cleanCheckout: true

The default repository will always be checked out first (if used), and then the other repositories in the order specified.

If you wish to force a clean checkout of the repositories on or off use cleanCheckout.

SCRIPT Tasks

These are pretty simple, just bash scripts that contain a body to run.

If you have multiple repositories be setup, then you will need subWorkingDirectory to support your script tasks. Here is the example

- type: SCRIPT
  description: Run unit tests
  body: |
   echo "Do your actions here."
  workingDirectory: your_sub_directory

DOCKER Tasks

Currently only the run docker task is supported. It requires the image property to be specified, but also allows all the other options:

- type: DOCKER
  description: Run unit tests
  image: dockerrepo.reecenet.org:4433/cyborg/tox-tests
  environmentVariables: JAVA_OPTS="-Xmx256m -Xms128m"
  cmdLineArguments: -u 1000
  container:
    workingDirectory: /app
    environmentVariables: PACT_DIR=/app/pacts
    command: tox
  volumeMappings:
  - local: ${bamboo.working.directory}
    container: /app

The cmdLineAguments are additional arguments for the "docker run" command. Note the distinction between environment variables set outside the docker container and those set inside the container.

Also supported is running docker containers in the background with port mappings:

- type: DOCKER
  description: Run unit tests
  image: dockerrepo.reecenet.org:4433/cyborg/some-server
  detach: true
  container:
    name: detached-server
    workingDirectory: /app
  portMappings:
  - local: 8080
    container: 8001
  serviceStartCheck:
    url: http://localhost:${docker.port}
    timeout: 120

Note that the container -> name property is required for detached containers. The docker.port variable will provide the first exposed port for creating the service check URL.

INJECT Tasks

While running tasks you may write values to a properties file which the INJECT task makes available to other tasks. To specify the file, use:

- type: INJECT
  description: Store variables for use in other tasks
  namespace: inject
  propertiesFile: inject-properties.ini
  scope: RESULT

The file must exist when this task is run and uses a 'key=value' format. You must provide a relative path to the property file. The values will be available in the "bamboo." variable namespace, so given this properties file:

key=value
version=1.2.3

These values will be available in other tasks (scripts, etc) as the variables "${bamboo.inject.key}" and "${bamboo.inject.version}".

The variables will be discarded at the end of the Job if the scope is "LOCAL" and retained for other Jobs if the scope is "RESULT" (the default).

CUCUMBER_REPORT Tasks

This task is used to showing the cucumber test json report.

- type: CUCUMBER_REPORT
  description: Cucumber JSON test report
  reportPath: "test/cucumber./reports/json/*.json"

ARTIFACTORY Tasks

This task is used to deploy a generic artefact to Artifactory

- type: ARTIFACTORY
  uploadSpec: |
    { * JSON upload spec here * }

For reference on building the 'upload spec', please consult the Artifactory documentation: https://www.jfrog.com/confluence/display/RTF/Using+File+Specs

Artefact Download

- type: ARTEFACT
  description: Download Artefacts

This will download all artefacts generated by the build into the current directory.

Specific Artefacts Download

- type: SPECIFIC_ARTEFACTS
  description: This is the description section (~˘▾˘)~
  artifacts:
    - name: Helm Package
      path: .
    - name: Coverage Report
      path: .

This will download a specific list of artefacts generated by the build into the current directory. Also the name of artifact has to be matched to the name defined in source plan.

Final Tasks

Final tasks are tasks that are always run after the other tasks, regardless of whether they were successful. These could be cleanup tasks, or more commonly including a JUnit parser to parse the results of the tests which may have failed:

finalTasks:
- type: JUNIT
  description: Include XML test results
  resultFrom: "**/unittest-report/xml/*.xml"

Reusable Jobs

If you have the same Job run across many different plans you can craft a YAML file that contains just the Job specification, for example "include/library-unit-tests.yaml":

name: Run Library Unit Tests
key: UT
description: Run tox unit tests and check setup.py version against existing git tags
requirements:
- name: system.docker.executable
tasks:
- type: VCS
...

And then in your plan YAML file you may include that (noting that you may also declare other plan-specific jobs):

stages:
- name: Default Stage
  jobs:
  - include: include/library-unit-tests.yaml
  - name: Build package
    key: BUILD
    description: Build Python package
    requirements:
    ...

In this example the include file is in a separate subdirectory - this makes it easier to use globbing to process multiple files in a single directory, like:

java -jar target/bamboo-specs-reece-1.0.5.jar plan bamboo-configs/Cyborg/unit-tests/*.yaml

Deployment Projects

Deployment projects look a lot like build and test plans, and even share some of the same sections, but do have a different preamble and structure.

Top Level Settings

At the top of the file you need to identify the deployment by name, and then the build plan that it belongs to:

specType: deployment
bambooServer: https://bamboo.reecenet.org/bamboo
name: Diary Notes Python Shared Service
buildProject: DNSS
buildPlan: DNPSDB
description: This is a deployment plan for the Diary Notes Python Shared Service

You should also define the release naming scheme:

releaseNaming:
  pattern: ${bamboo.version_major_number}.${bamboo.buildNumber}
  retainNamingStrategyForBranches: false

If you would like to set variables across all environments you can set variables in the preamble:

variables:
  target_name: diary-notes-service

This will have the effect of setting the variable in each of the environments (Bamboo does not offer variables at this level).

Permissions

Deployment project permissions may be set in this file with a section at the top level which has the same basic structure as the previous permissions settings (users, groups, grants), just that it has two sections:

permissions:
  project:
    users: [dooleyj, poultonj]
    groups: [Cyborg_Team]
    grant: [VIEW, EDIT]
  environment:
    users: [dooleyj, poultonj]
    groups: [Cyborg_Team]
    grant: [VIEW, EDIT, BUILD]

The first section applies to the the project itself, and the second applies to all environments in the project. There is currently no support for per-environment permissions in this file.

Environments Structure

This is a list of environments that you will deploy to. Each environment will have a name and description followed by requirements, notifications, variables and tasks that are constructed exactly the same as in build plans. So for example:

environments:
- environment: Production (AU + NZ)
  description: Deployment plan for the Diary Notes Python Shared Service to production
  requirements:
  - name: system.docker.executable
  notifications:
  - when: DEPLOYMENT_FINISHED
    slack: https://hooks.slack.com/services/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|#cyborg-dev
  variables:
    deployment_script: cutover.py
  tasks:
  - type: VCS
    description: Running Man
    repositories:
    - name: Running Man
    - name: Running Man Properties
      path: properties
  - type: SCRIPT
    description: Cutover Blue to Green - Training
    body: |
      ${deployment_script} ${bamboo.target_name} training_nz ${bamboo.version_major_number}.${bamboo.buildNumber}
      ${deployment_script} ${bamboo.target_name} training_au ${bamboo.version_major_number}.${bamboo.buildNumber}
  - type: SCRIPT
    description: Cutover Blue to Green - Production
    body: |
      ${deployment_script} ${bamboo.target_name} prod_au ${bamboo.version_major_number}.${bamboo.buildNumber}
      ${deployment_script} ${bamboo.target_name} prod_nz ${bamboo.version_major_number}.${bamboo.buildNumber}
  triggers:
  - type: AFTER_SUCCESSFUL_BUILD_PLAN
    description: Deploy main plan branch (master)

Using Included Environments

If you have the same environments appearing in multiple deployments you may save them off in a separate YAML file (say, environments.yaml) which has the exact structure of the above environments structure sample (ie. environments: at the top level) and then each of the named environments may be included in your deployment project yaml like so:

includeEnvironments:
  from: environments.yaml
  environments: [
    Production (AU + NZ),
    POS1 TEST AU, POS1 TEST NZ, POS2 TEST AU, POS2 TEST NZ,
    POS2 UAT AU, POS2 UAT NZ
  ]

Using included tasks inside environments

If you have multiple environments specified and all or some of them are using the same set of tasks you can make use of includedTasks. Specify the set of tasks in a separate yaml file and then specify the reference in the environment. Note you can have both the tasks and includedTasks property specified, the tasks will be added first, then the included tasks.

Specify tasks in separate yaml file:

- type: CLEAN
  description: Clean working directory
- type: ARTEFACT
  description: Download Helm chart
- type: VCS
  description: Git Checkout
  repositories:
    - name: oyster-scripts
      path: oyster-scripts

Use included tasks like this:

- environment: trstst05
  description: Inventory UAT NZ
  requirements:
    - name: pegasus
  notifications:
    - when: DEPLOYMENT_FAILED
      slack: https://hooks.slack.com/services/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
  variables:
    k8s_cluster: non-production-internal-cluster
  tasks:
    - type: CLEAN
      description: Clean working directory
  # Path is relative to the root yaml file  
  includedTasks: ../include/deployment-tasks.yaml
  triggers:
    - type: AFTER_SUCCESSFUL_BUILD_PLAN
      description: Deploy development branch

Note that the path for the included tasks is relative to the root yaml file, in this case the actual deployment template yaml.

Version History

2.3.18

Add support for retaining the release naming strategy for branches.
Commonly known as The Checkbox of Salvation.

2.3.17

Add support for including tasks, remove support for Artifactory

2.3.16

Upgrade to Bamboo 7.0.4

2.3.15

Add support to include tasks for multiple deployment environments to reduce the repeating task entries in the environments specified inside a deployment template.
Works for both inline environments and included environments inside the deployment template.

2.3.14

Log errors to STDOUT

2.3.13

Add support for customising maximum concurrent builds

2.3.12

Add support for SPECIFIC_ARTEFACTS type which supports optional artifact download

2.3.11

Add support for submodules in repo settings

2.3.10

Add support for cucumber test JSON report task

2.3.9

Add support for 'responsible user' notification

2.3.8

Upgrade Bamboo to 6.10.4

2.3.9

Add support for 'responsible user' notification

2.3.8

Upgrade Bamboo to 6.10.4

2.3.7

Add support for checking out into a different directory

2.3.6

Add Apache license

2.3.5

Various cleanup, typo corrections, etc.

2.3.4

Add a label to 'specs-built' plans for identification/analysis purposes

2.3.3

Retry when receiving HTTP 302 from Bamboo

2.3.2

Add support for triggering plans based on regex, enable/disable shallow cloning

2.3.1

Upgrade to Bamboo 6.7.1

2.3.0

???

2.2.1

Add support for running in a Docker container

2.2.0

Add support for ARTIFACTORY upload tasks

2.1.5

Add support for TestNG final tasks

2.1.4

Add support for build status change notifications

2.1.3

Fix a bug in Integer parsing for scheduled tasks

2.1.2

Fix a defect where YAML syntax errors did not result in 'failing' specs files

2.1.1

Fix a defect whereby runs fail if the file already exists for the output

2.1.0

Added support for outputting (in JUnit format) results

2.0.1

Added support for labeling plans

2.0.0

Added support for automatic parsing of plans added to the repository

1.1.9

Add ability to specify the branch to trigger after successful build on.

1.1.8

Added INJECT task type.

1.1.7

Bugfix: honor the branch name in repository settings.
Removed the defaulting of branch cleanup after 30 days.
Added ability to set permissions to all logged in users.

1.1.6

Added support for scheduled triggers.

1.1.4

Added support for arbitrary git repositories in test plans.

1.1.1 - 1.1.3

???

1.1.0

Clarified docker container configuration by splitting various aspects into sub-groups in the
YAML, allowing clear distinction between the two environmentVariables settings. Also, clean up
the config around detached containers.

1.0.0

Initial release

About

A programme for specifying Atlassian Bamboo plans as YAML files

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published