Release 7.12.0

[taylorsilva]

Creating this issue to discuss making a new release since I saw Rui make the milestone for one.

Are there any blockers at this point?
@xtremerui (not sure if there's anyone else on the Broadcom side I should tag too)

Steps for a new major/minor release:

• Add this issue to the v<M.m.x> milestone

• Bump the appropriate versions for resource types. Go through the list of resource types below and take a look at which resource type repositories have had new commits or PRs by comparing the last release to the master branch. Take a look at what those changes entail and bump the version in their respective pipeline in ci.concourse-ci.org.

  • If the changes were only README or repo restructuring changes with no user impact, you don't need to bump the version

  • If the changes were small bug fixes or changes, you can do a patch version bump

  • If the changes were adding of features, you can do a minor version bump

  • If the changes involve a breaking changes, that should be a major version bump

  • git: concourse/git-resource@v...master

  • registry-image: concourse/registry-image-resource@v...master

  • s3: concourse/s3-resource@v...master

  • time: concourse/time-resource@v...master

  • semver: concourse/semver-resource@v...master

  • pool: concourse/pool-resource@v...master

  • mock: concourse/mock-resource@v...master

  • github-release: concourse/github-release-resource@v...master

  • hg: concourse/hg-resource@v...master

  • docker-image: concourse/docker-image-resource@v...master

  • bosh-io-stemcell: concourse/bosh-io-stemcell-resource@v...master

  • bosh-io-release: concourse/bosh-io-release-resource@v...master

  • tracker: concourse/tracker-resource@v...master Repo archived since Tracker is EOL

• Create your release branch on the concourse/concourse github repo with the following format release/M.m.x (M being the major version and m being the minor version)

• Create the release branch on concourse/concourse-docker repository.

• Create the release branch on the concourse/concourse-bosh-release repository. Make any missing changes to the spec of web or worker depending on if the release contains any changes that adds or modifies any flags.

  • Any changes you make on the branch will not get automatically merged back to master so try to make the changes on master and then create the branch from there.
  • We should really have something that will merge the branch back to master. (like we do for the concourse/concourse branches)

• Create the release branch on concourse/concourse-bosh-deployment repository.

• Create the release branch on concourse/concourse-chart repository from the dev branch. Make any missing changes to values.yaml or templates/web-deployment.yaml for changes to flags on web or templates/worker-deployment.yaml for changes to flags on the worker.

• Add your release pipeline to the reconfigure-pipeline

• Go through all the needs-documentation PRs in the release page for your milestone https://project.concourse-ci.org/releases/concourse?milestone=v<M.m.p> and make sure that everything has proper documentation within concourse/docs (if needed). You can organize which PRs by clicking on the button to add whichever label best fits that PR.

  • If it is already documented within concourse/docs, add a release/documented label
  • If there is no documentation and the changes have user impact that should be documented, add the documentation to concourse/docs(or delegate) then add a release/documented label after finished. E.g. the addition of a new step type ( set_pipeline step).
  • If there is no documentation and the changes have user impact that do not need to be documented, add a release/undocumented label. E.g. an experimental feature.
  • If there is no documentation and the changes do not have user impact, add a release/no-impact label. E.g. refactors.

• Once the all source code changes are finalized, Concourse RC version should be deployed to CI

  • including all the external workers (pr-worker, ci-topgun-worker, darwin-worker & BOSH deployed windows worker)

• If you are doing a major release (or a release that involves a risky large feature), consider creating a drills environment for some stress testing to ensure that the release does not involve any performance regressions.

• Once the final commit has made it through the pipeline, the create-draft-release job can be triggered. This job will create a draft release within the concourse GitHub release page where you can make any final adjustments or arrangements to the generated release notes. PLEASE NOTE that any manual changes made on the draft release WILL BE OVERWRITTEN if you retrigger the create-draft-release job. Please be sure to only make manual edits AFTER you are sure this is the final run of the job.

  • If you would like to edit the content, you can directly edit the PRs that it was generated from. The title is used for each PR and also the body within the Release Note header in the PR. After you have made your edits within the PR, you can rerun the create-draft-release job in order to regenerate a new release note.
  • If you would like to change the arrangement of the PRs within the release note, you can make the edits directly on the release note of the draft release.

• Once everything is ready, the shipit job can be triggered. The publish-binaries job will convert your draft release into a final release including the body of your draft release (which will persist any changes you made to the draft release body). Subsequently, the promote concourse job will run automatically. The publish-docs job runs automatically.

• The helm-chart pipeline is used to bump & then publish the chart.

  • First, run the merge-dev-into-master job | I believe this job no longer exists and we must manually do this ourselves, probably to fix any conflicts?
  • Next, run the concourse-app-bump job (bumps the app version and image to point to the latest release)
  • Finally, run the publish-chart-{major|minor|patch} job, depending on what has changed in the chart
  • If you make a major bump, be sure to update the CHANGELOG.md in the concourse-chart repo

[xtremerui]

@taylorsilva thank you for creating this issue. We have an epic for this release. Whoever got assigned this issue will do the release.

[taylorsilva]

Sorry to folks seeing this issue still open. Haven't had time to start cutting the release. You know, life.

[marco-m-pix4d]

@taylorsilva @xtremerui is there any way external people like me can help?

[wayneadams]

Hi @marco-m-pix4d , thanks for the ping and sorry for the delay. Last week, we started to work though the steps in the issue template, but we needed to put it down temporarily for other priorities. The plan is that we (I) pick this up again early next week.

[marco-m-pix4d]

@wayneadams thanks a lot for the update, appreciated!

[taylorsilva]

@xtremerui Looks like Vaults' cert just expired

[taylorsilva]

Chipping away at getting a new cluster setup: https://ci.concourse-oss.org/ (it's setup with Github auth to the concourse org)

Next step is updating the pipelines. Planning to remove all the bosh jobs from the release pipeline because it's too heavy to keep maintaining. Focus will be on getting the release pipelines setup and running and pushing 7.12.0 out the door

---

For those out of the loop, the infrastructure at https://ci.concourse-ci.org/ is maintained by Broadcom. Only Broadcom employees have access to the GCP account where all the infrastructure is running. They currently have no ETA as to when they'll be able to get it fixed.

I've decided to go ahead and setup a new Concourse cluster for the community to start releasing from. Goal is to get us back on track and releasing regularly again. Once 7.12.0 is out I'm going to focus on getting the PRs pipeline working.

[marco-m-pix4d]

@taylorsilva for sure I am happy and what to thank you. But, I also want to say one thing: please be careful not to step into the slope towards open source maintainer burn out! In case of doubt, protect yourself. 💪

Also if I may ask: the infra will have a cost, who is going to fit the bill? Maybe it would be worthwhile to enable GitHub sponsorship ?

[taylorsilva]

please be careful not to step into the slope towards open source maintainer burn out! In case of doubt, protect yourself. 💪

Thanks for the reminder. I am constantly aware of this as I've read many of the same maintainer burnout stories that you likely have. I'm doing my best to not push myself very hard and just do one small thing at a time.

For infra cost, I'm footing it right now. I'm in a good financial position that I can do this, though I am trying to stay thrifty to stretch my dollars! I do want to start doing Github sponsors or OpenCollective. I'll do that once a release is out and folks can see that we are able to keep pushing releases out. I want people to feel confident giving money to the project and I think that means seeing some results first in the form of a new release.

[suhlig]

GitHub sponsors would be nice as it would give us a way to help ensure the future of Concourse.

[taylorsilva]

A little update for everyone because I'm going on vacation so won't be working on this over the next week and a bit.

I've got the main pipelines setup and running, but not everything is passing yet. I'm focusing on the main concourse pipeline right now. I want to get that pipeline fully green before making a release/7.12.0 pipeline. I also need to get all the resource-type pipelines green too.

So order right now is:

• https://ci.concourse-oss.org/teams/main/pipelines/concourse?group=all
• https://ci.concourse-oss.org/?search=team%3A%22main%22%20group%3A%22resource%22
• https://ci.concourse-oss.org/?search=team%3A%22main%22%20group%3A%22release%22

Most of my time has been spent getting tests to pass on the new infrastructure. The workers are running on Fedora 40 which only has cgroups v2 enabled. I only ran into issues with running container-in-container tests (like DinD). There's also a test failing right now that I'm pretty sure is due to Hetzner's NIC having a max MTU of 1450 instead of 1500 (see here).

Here's a snapshot of all the jobs passing right now:

I've still got to get some infra setup like an S3 bucket for storing some artifacts and for testing the S3 resource-type.
I also need to modify a bunch of the k8s jobs to do one of the following:

• Run a small k8s (k3s or minikube?) on the worker and run tests against that. I'm worried that'll be a very slow test though and possibly flakey because it might overload the workers CPU
• Create a k8s cluster for the duration of the job/test (thinking Linode as they have the fastest startup times for a k8s cluster) and run the tests against that cluster. Likely to be the most robust option and the one I'll likely go with. It'll mean more $$$ but I think it'll be worth it.

[taylorsilva]

Progress!

[taylorsilva]

Was able to continue to keep everything on Hetzner when setting up the S3 bucket. Hetzner has a new S3-compatible service available. It's in beta but I was able to get in and start using it.

Next few sets of tests will require some thinking as they use terraform to deploy a conocurse. Going to see if I can port those to run on Hetzner instead.

[marco-m-pix4d]

Hetzner has a new S3-compatible service available. It's in beta but I was able to get in and start using it.

Funny I missed that! I will use that for my own projects :-) Thanks for mentioning it!

Next few sets of tests will require some thinking as they use terraform to deploy a conocurse. Going to see if I can port those to run on Hetzner instead.

I think you know this, but mentioning for completeness: Hetzner has the notion of projects, this might be useful if you want to automate terraforming and have some separation.

[taylorsilva]

So close!

k8s-topgun is giving me issues. It's SUCH a flakey test suite. I've been tuning it to run well on Linode's k8s cluster. Had to adjust account resources limits on their end as the test suite would request too many PVC's 😅

[taylorsilva]

🎉

(seems like this was the main cause of flakes in k8s-topgun: 6be6435)

Can finally move onto the resource pipelines!

[taylorsilva]

In the final stretch:

Need to bump some Go dependencies to resolve CVE's in the main binary. Might try to squeeze this in too: concourse/flag#11

I'm not going to get any other features or fixes in for this release though. Anything else will go into the next release, 7.13.0

[marco-m-pix4d]

I'm not going to get any other features or fixes in for this release though

Totally agree

[taylorsilva]

Currently testing the latest RC build (7.12.0-rc.2) on https://ci.concourse-oss.org/

If anyone else wants to give it a spin they can pull that image

docker pull concourse/concourse-rc:7.12.0-rc.2

I'm going to let it run for a few days and then make the official release. Let me know if you run it and if there's any issues. If everything looks good drop a 🚀 emoji on this message.

[norman-abramovitz]

I am checking if FiveTwenty can take the concourse rc for a spin.

[taylorsilva]

We did it!
https://github.com/concourse/concourse/releases/tag/v7.12.0

The binaries and container images are available for everyone to download.

A few lingering tasks:

• The bosh publishing job had some issues, but I think I know what I broke there and can get it fixed. I'll do that this weekend and then the bosh release will also be available for those that use it
• Need to publish the updated helm-chart still. Will probably prioritize that over the bosh stuff since most folks use helm from what I recall on the last community survey

[marco-m-pix4d]

Thanks @taylorsilva !
We will incorporate the changes in our fork, deploy it and report back, but be warned that it will take some time on our side. Thanks again.

[analytically]

Something seems off with the windows package?

[analytically]

It also seems that the binary_params option doesn't seem to work correctly in our preliminary testing:

This lead me to lib/pq#528

[taylorsilva]

@analytically Thanks for reporting those issues! I'll see what happened to the Windows release. Sadly I don't have a windows worker setup so that was one area of testing I had to drop.

For the binary_params I guess that's something we could fix. Do you think we'd run into something similar with concourse/flag#11? I've noticed I have a gap here with understanding how new params affect these db drivers.

[analytically]

I think your comment on concourse/flag#11 regarding libpq support is valid. Though I believe binary_params with jsonb is a Go specific thing, and easily solved, for example see https://github.com/go-testfixtures/testfixtures/pull/84/files

[kcbimonte]

While the Bosh Release Repository was updated with the new variables tied to new functionality, the Bosh Deployment repository was not updated to include the new Ops Files. This repo does lag behind a bit so there was a gap with AWS SSM Support that is entirely added in concourse/concourse-bosh-deployment#260 along with the new shared path functionality. All properties should map the spec entries for both web and worker.

[taylorsilva]

Figured out the issue with the windows artifacts. The script that builds the binary mostly worked, it was the last step that zipped everything together that was incorrect and resulted in a zip file that contained an empty folder: concourse/ci@a4c001d

Can see it zipping everything correctly now:

[taylorsilva]

Thanks @kcbimonte for the PR! I'll review it once I figure out the other bosh-stuff in the release pipeline. I ended up cutting too much out when I was removing the bosh related jobs. I'm trying to figure out how I can upload bosh releases without a director right now (bosh upload-release).

[taylorsilva]

Windows package actually contains stuff now! Thanks for initially pointing that out @analytically

[taylorsilva]

Bosh release is now up! Had to fork and re-publish a resource to get it working (old resource was using bosh-cli 3.0.8, bumped that to the latest version). @kcbimonte Since it sounds like you use bosh, give the new release a spin and let us know if there are any issues.

https://bosh.io/releases/github.com/concourse/concourse-bosh-release

The windows package also looks good. I had to hack the pipeline a bit to get the bosh release job pushing out a 7.12.0 version. If you manually open the bosh blobs you'll notice that RC1 got packaged instead of RC2, what we actually released on GitHub. There's no difference between the two RC's though. They're built off the exact same commit and use the same resource-type versions.

Working on the helm chart release next!

[o-orand]

It seems that re-uploading messes up the versioning.
I tried to update to 7.12.0 today and got 7.12.1. I used the bosh-concourse-deployment, which points to 7.12.0. (see concourse/concourse-bosh-deployment@781baf3?diff=split&w=0#diff-267701a4444b19e808e619ff53d61b2a5eba5838724c6be13e85dbecbe7526c0R9)

Version displayed on concourse:

Warning from fly:

fly version (7.11.2) is out of sync with the target (7.12.1). to sync up, run the following:

Concourse linux binary seems consistent:

tar xzvf concourse-7.12.0-linux-amd64.tgz concourse/bin/concourse
concourse/bin/concourse --version 
7.12.0

[taylorsilva]

@o-orand damn! I was worried this would be an issue with the bosh-release. It wasn't easy to get that part working and I think running the job post-release messed with the pipelines version-ing logic, as you've shown.

sigh

I'm not gonna fix that right now. Right now I'm going to focus on working with the Broadcom folks that have reached out to me and get things working back on https://ci.concourse-ci.org/

I'll likely push out a 7.12.1 from their infrastructure once we're back on their stuff, since they have the infra to test bosh properly. Sadly, releasing bosh from an unsupported bosh-IAAS proved to be very difficult task. I don't like that I couldn't test the bosh release at all. Anyways, that'll all be fixed with 7.12.1

[cfryanr]

@taylorsilva Thanks for the new Concourse release!

I was wondering if there is still a plan for a corresponding release of concourse-chart? From your comments above, it sounds like that's the last remaining item for this GitHub issue.

[taylorsilva]

@cfryanr I was about to write some long answer about why I'm blocked, but then realized that I am unblocked from releasing the chart!

I wasn't able to previously release a new version of the chart because the chart repo points to a bucket that is owned by Broadcom. Thankfully the Broadcom folks and I are talking now and I'm able to release using the original pipeline infrastructure (ci.concourse-ci.org)

The main development pipelines are still blocked (expired Bosh certs 😞), but PR pipelines are working again!

Anyways, chart version 7.14.0 is released! 🎉

I'm going to close this issue now. I know there are outstanding issues with the bosh release (sorry folks!), but I can't do anything about those until Broadcom gets the Bosh deployment fixed. I think the majority of people are satisfied now.

I'll have more time to focus on Concourse in the new year. More news on that to come 😉