From 60381583fa373975fd22570e7cd0a76f53adf25d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Pra=C5=BCak?= Date: Mon, 12 Feb 2024 14:14:45 +0100 Subject: [PATCH] Update website, Justfile and CONTRIBUTING.md (#388) - fix right sidebar in basics page - minor editing corrections - update release instructions --- CONTRIBUTING.md | 45 +++++++++++++++++++------- Justfile | 2 ++ website/README.md | 22 ++++++------- website/docs/api_reference.md | 2 +- website/docs/basics.md | 26 +++++++-------- website/docs/constructors.md | 10 +++--- website/docs/getting_started.md | 2 +- website/docs/intro.md | 6 ++-- website/docs/logging.md | 2 +- website/docs/tutorial.md | 43 +++++++++++++++--------- website/src/remark/codeblockVersion.js | 2 +- 11 files changed, 98 insertions(+), 64 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 19216ac2..4d93bdb7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -193,15 +193,24 @@ It is recommended to use `just power-wash` before publishing a release: just power-wash ``` -#### Bump Besom version +#### Prerequisites + +Publish SDKs locally to test and provide fresh dependencies for scripts: + +```bash +just publish-local-all +``` + +#### Bump Besom version (skip for `SNAPSHOT` re-release) To bump Besom version in all `project.scala` and `version.txt` files: ```bash +export GITHUB_TOKEN=$(gh auth token) just cli version bump X.Y.Z ``` -#### Create release branch +#### Create release branch (skip for `SNAPSHOT`) ```bash git checkout -b release/v$(cat version.txt) @@ -213,13 +222,14 @@ git push --set-upstream origin release/v$(cat version.txt) This is most useful for `examples` and `templates`, and integration tests: ```bash +export GITHUB_TOKEN=$(gh auth token) just cli version update ``` -#### Update versions in all other places +#### Update versions in all other places (skip for `SNAPSHOT`) Manually update versions in all other places, specifically documentation and website, using find&replace. -Remember about `website/docusaurus.config.js:40` +Remember about `website/docusaurus.config.js:40` and `website/src/remark/codeblockVersion.js:5` Manually update branch names in all places. @@ -233,7 +243,6 @@ just upsert-gh-release #### Publish core and language host ```bash -just publish-local-all just publish-maven-all just publish-language-plugins-all ``` @@ -258,7 +267,7 @@ export GITHUB_TOKEN=$(gh auth token) just clean-out cli packages maven-all ``` -Tip: it's safer to publish the packages on-by-one due to how Maven Central behaves. +Tip: it's safer to publish the packages on-by-one or in batches due to how Maven Central behaves. In case of any issues, you can try to resolve the issues manually at https://oss.sonatype.org/index.html#stagingRepositories. @@ -268,6 +277,13 @@ Finish the release on GitHub manually, make sure the changelog is correct and co According to our Git branching and versioning strategy, the release branch should be created after the tag is created. +If releasing a `SNAPSHOT` make sure to bump the git tag. + +```bash +git tag -f v$(cat version.txt) +git push -f origin v$(cat version.txt) +``` + #### After the release After the release, you can bump the version to the next `-SNAPSHOT` version: @@ -472,6 +488,16 @@ rm pulumi.rb ### Compilation issues +Remove `.scala-build`, e.g.: +```bash +rm -rf core/.scala-build +``` + +To restart `bloop` compilation server: +```bash +scala-cli bloop exit +``` + To clean the builds: ```bash just clean-all @@ -482,11 +508,6 @@ If a deep cleaning needed: just power-wash ```` -To restart `bloop` compilation server: -```bash -scala-cli bloop exit -``` - To set `bloop` verbosity: ```bash scala-cli setup-ide -v -v -v . @@ -499,7 +520,7 @@ scala-cli compile -S 3.nightly . To increase Scala compiler verbosity: ```bash -scala-cli compile --javac-opt=-verbose . +scala-cli compile --scalac-option -verbose . ``` To inspect a running JVM byt its PID use `jcmd`, e.g.: diff --git a/Justfile b/Justfile index 0713fbfa..e7e9e547 100644 --- a/Justfile +++ b/Justfile @@ -417,6 +417,8 @@ upsert-gh-release: power-wash: clean-all rm -rf ~/.ivy2/local/org.virtuslab/ rm -rf ~/.m2/repository/org/virtuslab/ + rm -rf ~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/virtuslab/ # Linux + rm -rf ~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/virtuslab/ # Mac git clean -i -d -x -e ".idea" killall -9 java diff --git a/website/README.md b/website/README.md index aaba2fa1..d33902b9 100644 --- a/website/README.md +++ b/website/README.md @@ -1,25 +1,25 @@ # Website -This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator. +This website is built using [Docusaurus 2](https://docusaurus.io/docs/2.x), a modern static website generator. ### Installation -``` -$ yarn +```bash +yarn ``` ### Local Development -``` -$ yarn start +```bash +yarn start ``` This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server. ### Build -``` -$ yarn build +```bash +yarn build ``` This command generates static content into the `build` directory and can be served using any static contents hosting service. @@ -28,14 +28,14 @@ This command generates static content into the `build` directory and can be serv Using SSH: -``` -$ USE_SSH=true yarn deploy +```bash +USE_SSH=true yarn deploy ``` Not using SSH: -``` -$ GIT_USER= yarn deploy +```bash +GIT_USER= yarn deploy ``` If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch. diff --git a/website/docs/api_reference.md b/website/docs/api_reference.md index 11f395e8..d789cd04 100644 --- a/website/docs/api_reference.md +++ b/website/docs/api_reference.md @@ -2,4 +2,4 @@ title: API Reference --- -### Coming soon! \ No newline at end of file +## Coming soon! \ No newline at end of file diff --git a/website/docs/basics.md b/website/docs/basics.md index 8614ffc8..7b250fea 100644 --- a/website/docs/basics.md +++ b/website/docs/basics.md @@ -7,14 +7,14 @@ import Version from '@site/src/components/Version'; Before we dive into the [details of Besom](architecture.md), let's take a look at the basics of Pulumi. This page offers an executive summary of [Pulumi's concepts](https://www.pulumi.com/docs/concepts/). -### What is Pulumi? +## What is Pulumi? Pulumi is a modern infrastructure as code platform. It leverages existing programming languages and their native ecosystem to interact with cloud resources through the Pulumi SDK. Pulumi is a registered trademark of [Pulumi Corporation](https://pulumi.com). -### What is Besom? +## What is Besom? Besom is a **Pulumi SDK for Scala 3**. It allows you to use Scala to define your infrastructure in a type-safe and functional way. @@ -25,7 +25,7 @@ Besom **does NOT depend on Pulumi Java SDK**, it is a completely separate implem Please pay attention to your dependencies, **only use `org.virtuslab::besom-*`** and not `com.pulumi:*`.
::: -### Concepts +## Concepts It is important to understand the basic concepts of Pulumi before we dive into the details of Besom. We strongly advise to get acquainted with [Pulumi's concepts](https://www.pulumi.com/docs/concepts/) @@ -38,7 +38,7 @@ For more detailed information see [how Pulumi works](https://www.pulumi.com/docs/concepts/how-pulumi-works/#how-pulumi-works) documentation section. -#### Projects +### Projects A [Pulumi project](https://www.pulumi.com/docs/concepts/projects/) consists of: @@ -69,7 +69,7 @@ name: Example Besom project file with only required attributes runtime: scala ``` -#### Programs +### Programs A Pulumi program, written in a general-purpose programming language, is a collection of [resources](#resources) that are deployed to form a [stack](#stacks). @@ -98,7 +98,7 @@ A minimal Besom program consists of: Pass [`Context`](context.md) everywhere you are using Besom outside of `Pulumi.run` block with `(using besom.Context)`. ::: -#### Stacks +### Stacks [Pulumi stack](https://www.pulumi.com/docs/concepts/stack/) is a separate, isolated, independently configurable instance of a Pulumi [program](#programs), and can be updated and referred to independently. @@ -141,14 +141,14 @@ To export values from a stack in Besom, use the [`Stack.exports`](exports.md) fu ##### Stack References -[Stack References](https://www.pulumi.com/docs/concepts/stack/#stackreferences) allow you to use outputs from other [stacks](#stacks) in your [program](#programs). +[Stack Reference](https://www.pulumi.com/docs/concepts/stack/#stackreferences) allows you to use outputs from other [stacks](#stacks) in your [program](#programs). To reference values from another stack, create an instance of the `StackReference` type using the fully qualified name of the stack as an input, and then read exported stack outputs by their name. `StackReference` is not implemented yet in Besom, coming soon. -#### Resources +### Resources Resources are the primary [construct of Pulumi](https://www.pulumi.com/docs/concepts/resources/) programs. Resources represent the fundamental units that make up your infrastructure, such as a compute instance, @@ -196,7 +196,7 @@ is not managed by Pulumi. Here's an example of how to use it: } ``` -#### Inputs and Outputs +### Inputs and Outputs Inputs and Outputs are the primary [asynchronous data types in Pulumi](https://www.pulumi.com/docs/concepts/inputs-outputs/), @@ -287,7 +287,7 @@ val https: Output[String] = p"https://$host:$port/api/" We encourage you to learn more about relationship between [resources](#resources) and [outputs](#inputs-and-outputs) in the [Resource constructors and asynchronicity](constructors.md) section. -#### Configuration and Secrets +### Configuration and Secrets [Configuration](https://www.pulumi.com/docs/concepts/config) or [Secret](https://www.pulumi.com/docs/concepts/secrets/) is a set of key-value pairs that influence the behavior of a Pulumi program. @@ -328,10 +328,10 @@ Secrets in Besom differ in behavior from other Pulumi SDKs. In other SDKs, if yo secret, you will obtain it as plaintext (and due to [a bug](https://github.com/pulumi/pulumi/issues/7127) you won't even get a warning). We choose to do the right thing in Besom and **return all configs as Outputs** so that we can handle failure in pure, -functional way, and **automatically** mark secret values**as [secret Outputs](https://www.pulumi.com/docs/concepts/secrets/#how-secrets-relate-to-outputs)**. +functional way, and **automatically** mark secret values **as [secret Outputs](https://www.pulumi.com/docs/concepts/secrets/#how-secrets-relate-to-outputs)**. ::: -#### Providers +### Providers A [resource provider](https://www.pulumi.com/docs/concepts/resources/providers/) is a plugin that handles communications with a cloud service to create, read, update, and delete the resources you define in your Pulumi [programs](#programs). @@ -349,7 +349,7 @@ using [provider configuration](https://www.pulumi.com/docs/concepts/resources/pr It is recommended to [disable default providers](https://www.pulumi.com/blog/disable-default-providers/) if not for [all providers, at least for Kubernetes](https://www.pulumi.com/docs/concepts/config#pulumi-configuration-options). ::: -#### State +### State State is a snapshot of your [project](#projects) [resources](#resources) that is stored in a [backend](https://www.pulumi.com/docs/concepts/state/#deciding-on-a-state-backend) with [Pulumi Service](https://www.pulumi.com/docs/intro/cloud-providers/pulumi-service/) being the default. diff --git a/website/docs/constructors.md b/website/docs/constructors.md index bf44a155..d3934882 100644 --- a/website/docs/constructors.md +++ b/website/docs/constructors.md @@ -4,15 +4,15 @@ title: Resource constructors and asynchronicity import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -### Resources +## Resources Resources are the [primary construct of Pulumi](basics.md#resources) programs. -### Outputs +## Outputs Outputs are the [primary asynchronous data structure of Pulumi](basics.md#inputs-and-outputs) programs. -### Resource constructor syntax +## Resource constructor syntax Most Pulumi SDKs expect you to create resource objects by invoking their constructors with `new` keyword, for example in TypeScript: @@ -45,7 +45,7 @@ We have retained the CamelCase naming convention of resource constructors for pa You can always expect resource constructor names to start with capital letter. ::: -### Resource asynchronicity +## Resource asynchronicity Resources in Besom have an interesting property related to the fact that Pulumi's runtime is asynchronous. One could suspect that in following snippet resources are created sequentially due to monadic syntax: @@ -161,7 +161,7 @@ A good observer will notice that all these forks have to be awaited somehow and does await for all spawned Outputs to be resolved before finishing the run. ::: -### Compile time checking +## Compile time checking Besom tries to catch as many errors as possible at compile time, examples of our compile time checks are: - A component resource must register a correct type name with the base constructor in format: `::` diff --git a/website/docs/getting_started.md b/website/docs/getting_started.md index 5b30a178..0b5f9d78 100644 --- a/website/docs/getting_started.md +++ b/website/docs/getting_started.md @@ -76,7 +76,7 @@ To start your adventure with infrastructure-as-code with Scala follow these step pulumi destroy -y ``` -### Choice of build tool and IDE +## Choice of build tool and IDE --- Besom uses [Scala-CLI](https://scala-cli.virtuslab.org/) for project compilation and execution. diff --git a/website/docs/intro.md b/website/docs/intro.md index d99a5b7b..251a74b3 100644 --- a/website/docs/intro.md +++ b/website/docs/intro.md @@ -17,7 +17,7 @@ our implementation with the rest of Pulumi ecosystem, finding and solving issues ergonomics and feature completeness. ::: -#### Mission +### Mission Besom's mission is to introduce **rich type safety** to the domain of cloud and platform engineering. We want to allow our users to benefit from Scala's compiler support to **catch problems and mistakes before the execution** @@ -28,7 +28,7 @@ Besom follows the general model of Pulumi SDKs and operates using the **same bas It is therefore strongly advised to get acquainted with [**Pulumi's basics**](basics.md) section as a fast way to get up to speed with the general concepts. -#### Uniqueness +### Uniqueness Besom is unique in the fact that it's meant to support all Scala's technological ecosystems transparently: * Scala's [Future](https://docs.scala-lang.org/overviews/core/futures.html) and [Akka](https://akka.io/) / [Apache Pekko](https://pekko.apache.org/) @@ -42,7 +42,7 @@ written in functional style, to name two: - Stack along with its exports are the return value of the main function of the program, - smaller chunks of the program have to be composed into the main flow of the program to be executed. -#### Next steps: +### Next steps - [Getting started](getting_started.md) section helps you **get your hands dirty** and figure out the details as you go - [Basics](basics.md) offers an **executive summary** of Pulumi's key concepts and how they are implemented in Besom diff --git a/website/docs/logging.md b/website/docs/logging.md index 86a4dcfe..0b601fce 100644 --- a/website/docs/logging.md +++ b/website/docs/logging.md @@ -14,7 +14,7 @@ logger by writing `log` with a following severity level used as a logging method Logging is an asynchronous, effectful operation and therefore returns an `Output`. This means that all logging statements need to be composed into other values that will eventually be either passed as `Stack` arguments or exports. This is similar to how logging frameworks for `cats` or `ZIO` behave (eg.: [log4cats](https://github.com/typelevel/log4cats)). -### Why not simply `println`? +## Why not simply `println`? Given that you're working with CLI you might be tempted to just `println` some value, but that will have no visible effect. That's because Besom's Scala code is being executed in a different process than Pulumi. It's Pulumi that drives the diff --git a/website/docs/tutorial.md b/website/docs/tutorial.md index 93c51c58..9ffe7b5d 100644 --- a/website/docs/tutorial.md +++ b/website/docs/tutorial.md @@ -2,10 +2,9 @@ title: Tutorial --- -### Introduction +## Introduction --- - Besom, the Pulumi SDK for Scala allows you to create, deploy and manage cloud resources such as databases, serverless functions and services quickly and safely. In this tutorial your will do exactly that - you will deploy a very basic yet functional serverless application built in Scala 3. The target environment will be @@ -58,7 +57,7 @@ already packaged into zip files. These artifacts can be found in `./pre-built/` directory of the repository. -### Architecture +## Architecture --- The application you are going to deploy to AWS is called CatPost. It's a simple app from simpler times - its only @@ -74,7 +73,7 @@ in a DynamoDB table and the pictures of cats are stored in a publicly available page with post feed, second handles creation of new posts, uploads data to S3 and DynamoDB. Ok, let's deploy! -### Infrastructure-as-Scala-code +## Infrastructure-as-Scala-code --- Your infrastructure will live inside the `./besom` directory. Once you open the directory in your IDE (remember: @@ -95,10 +94,12 @@ import besom.* } ``` +To learn more about projects and programs in Besom refer to the [Projects section of the Basics page](./basics.md#projects). + This is the minimal Besom program that doesn't do anything beside logging a message. Let's change that and add some AWS components! -### AWS S3 Bucket +## AWS S3 Bucket --- The first step that you are going to do is to set up data storage for our app. The most important part of application @@ -145,14 +146,20 @@ val feedBucket = s3.Bucket( ) ``` -Unfortunately bucket names in AWS are **globally unique**, and therefore it would be very easy to have a name clash with +### Resource names, identifiers and URNs + +Since bucket names in AWS are **globally unique**, it would be very easy to have a name clash with another user. When you omit the explicit name of a resource via it's resource arguments Pulumi uses the name of Pulumi -resource and suffixes it with a random string to generate a unique name. This distinction is quite important -because Pulumi resource name is a part of [Pulumi URN](https://www.pulumi.com/docs/concepts/resources/names/), but -the actual bucket name is a part of its resource ID in AWS infrastructure. For now let's just remember that first +resource and suffixes it with a random string to generate a unique name (auto-name). This distinction is quite important +because Pulumi resource name is a part of [Pulumi URN](https://www.pulumi.com/docs/concepts/resources/names/), but the actual bucket name is a part of its resource ID in AWS infrastructure. +To learn more please refer to the [Resources section of the Basics page](./basics.md#resources). + +For now let's just remember that first argument of Besom resource constructor function is always the Pulumi resource name and resource names for cloud provider to use are always provided via resource arguments. +### Create the resources stack + You can try and create your bucket now. To do this execute `pulumi up` in `./besom` directory: ```bash cd besom @@ -164,7 +171,7 @@ stacks commonly match application environments. You might be also asked to provi actually quite important because Pulumi uses that password to encrypt sensitive data related to your deployment. As you are just playing you can use an empty password, but it's *very important* to use a proper password for actual work. Pulumi will subsequently print a preview of changes between current state of affairs and the changes your code will -introduce to the cloud environment. +introduce to the cloud environment. To learn more about stacks refer to the [Stacks section of the Basics page](./basics.md#stacks) Select `yes` when asked whether you want to deploy this set of changes to cloud. @@ -178,6 +185,8 @@ Remember that in actual production environment you can't just `pulumi destroy` t out it will be handy however because it's a tutorial. If you get into any trouble you can just start over by destroying everything and then applying your program from ground up. +### Make the bucket public + There's one huge problem with our bucket now - all S3 buckets are private by default. Let's make it public. Making an S3 bucket public is quite involved because turning a private bucket public can deal an enormous amount of damage @@ -256,7 +265,7 @@ Stack(feedBucket, feedBucketPublicAccessBlock, feedBucketPolicy) If you run your program now you will have an empty but publicly accessible S3 bucket! -### DynamoDB Table +## DynamoDB Table --- In scope of the lambda app at `./lambda/dynamodb.scala` you can find the case class used to model a row of @@ -303,7 +312,7 @@ val catPostTable = dynamodb.Table( That's all! DynamoDB is that easy to set up. Add the resource to the final `Stack` and run it. -### AWS Lambdas +## AWS Lambdas Ok, now you're getting to the point. You will now deploy both [AWS Lambdas](https://aws.amazon.com/lambda/). For lambdas to run they have to have a correct set of permissions aggregated as an AWS IAM Role. @@ -389,7 +398,7 @@ If you chose to rebuild the lambdas on your own you have to adjust the path so t Add this to your program, add both lambdas to the `Stack` at the end of your program, run `pulumi up` and that's it, AWS Lambdas deployed. -### AWS API Gateway +## AWS API Gateway Last thing to do - you have to make your app accessible from the Internet. To do this you have to deploy [API Gateway](https://aws.amazon.com/api-gateway/) with a correct configuration. @@ -619,6 +628,8 @@ Two things that need attention here is that API deployment has to be sequenced w [`deleteBeforeReplace` resource option](https://www.pulumi.com/docs/concepts/options/deletebeforereplace/) makes an appearance. These are necessary for AWS to correctly handle the deployment of these resources. +## Export stack outputs + Ok, that's it. Add *all* of these resources to Stack and then modify the [`exports`](./exports.md) so that it matches this: ```scala @@ -642,7 +653,7 @@ Current stack outputs (2):

-### Addendum A - debugging: +## Addendum A - debugging If you run into problems with CatPost lambdas when hacking on this tutorial you might want to create Cloudwatch `LogGroup`s for them so that you can track what's in their logs: @@ -674,7 +685,7 @@ val addLambdaLogs = addLambda.name.flatMap { addName => ``` Again, remember to add them to the Stack at the end of the program. -### Addendum B - final `Stack` block: +## Addendum B - final `Stack` block Here's how the final `Stack` block looks like in a complete deployment: ```scala @@ -706,7 +717,7 @@ Stack( ) ``` -### Addendum C - complete branch: +## Addendum C - complete branch If you run into problems during your tutorial run be sure to check out the `complete` branch of the [`besom-tutorial` repo](https://github.com/VirtusLab/besom-tutorial). diff --git a/website/src/remark/codeblockVersion.js b/website/src/remark/codeblockVersion.js index 3f733845..06efcd2b 100644 --- a/website/src/remark/codeblockVersion.js +++ b/website/src/remark/codeblockVersion.js @@ -2,7 +2,7 @@ const visit = require('unist-util-visit'); const codeblockVersion = () => async (ast) => { visit(ast, 'code', (node) => { - node.value = node.value.replace('$version', '0.1.0') // TODO read from env: process.env.BESOM_VERSION + node.value = node.value.replace('$version', '0.2.0') // TODO read from env: process.env.BESOM_VERSION }) }