Fuzzbucket in CI #211

kgartland-rstudio · 2023-09-12T15:54:18Z

Intent

(don't worry most of those 223 files are from adding all our jumpstart examples. feel free to ignore all files in test/sample-content/)

This PR introduces Connect in Fuzzbucket so we can run deploy tests to a real installation of Connect.

In the pull-request.yaml workflow, this will add a start-connect job that will create the fuzzbucket instance. It saves a connect_ip artifact to be re-used by other jobs for our other OS'es. That will likely be removed if we get a domain for our fuzzbucket in the future.
It adds an environment directory that will contain various environment files where we will store flags and other variables we may want to test against in the future. Currently, we only have a .basic environment but I imagine we'll have many other flag combinations we'll need to test against.
deployment_helper which currently just crawls the sample-content directory so we can iterate over any content type we want. this will only run a single deploy test in CI for now but once we get logs in the UI, i'll update it to test all python content.
if you want to run the tests locally, you can run just test/init-connect-and-publish darwin-arm64 which will start Connect in fuzzbucket and run the publishing tests against it (replace darwin-arm64 with whatever architecture you're running, you will need to just build first).
due to some issues accessing environment variables in Windows, I had to add a second target in web/justfile and web/package.json specifically for Windows.
the cypress tests added are still under home.cy.ts, they don't do much now because I can't yet access the deployment URL to actually test the deployment on Connect. right now we just make sure we can reach Connect and see that something has been deployed. i'll likely be adding a different file/folder structure once we're ready to add more tests.

Type of Change

Bug Fix
New Feature
Breaking Change
CI

Approach

Automated Tests

these are them

Directions for Reviewers

make sure you have fuzzbucket-client installed (pip install fuzzbucket-client)
and are logged in: fuzzubucket-client login
have the variables stored
FUZZBUCKET_URL=https://g3z4r6ej53.execute-api.us-east-1.amazonaws.com/prod
FUZZBUCKET_CREDENTIALS=[github_username]:[password] (password is retrieved during the login step)

run the tests locally
just test/init-connect-and-publish darwin-arm64

to run against all the deployments:
TEST_SCENARIO=all_content just test/init-connect-and-publish darwin-arm64

Checklist

I have updated CHANGELOG.md to cover notable changes.

tdstein · 2023-09-21T16:11:22Z

@kgartland-rstudio - If fuzzbucket goes down, will the CI/CD fail? I'm generally wary of depending on external services as part of pull request checks.

Should we consider isolating these tests to run via a different mechanism? For instance, only on main or on a schedule?

kgartland-rstudio · 2023-09-21T16:35:34Z

@kgartland-rstudio - If fuzzbucket goes down, will the CI/CD fail? I'm generally wary of depending on external services as part of pull request checks.

Should we consider isolating these tests to run via a different mechanism? For instance, only on main or on a schedule?

Understood. We've been running tests in fuzzbucket elsewhere for a few years now. I haven't seen fuzzbucket itself fail anywhere but I see your concern. I'd be concerned about merging a PR to main that wasn't tested in Windows specifically since a large percentage of our users will likely be running Windows and I'd rather not have to test every PR against Windows manually.

I do like the idea of running a larger set of longer running tests on a schedule rather than in CI but I think we need a way to test with Windows regularly on each PR and I couldn't find a way to do that with Docker in GH Actions.

kgartland-rstudio · 2023-09-21T16:38:33Z

I also removed ux-e2e-tests since those were redundant with the linux testing we do later on. Does anyone have any objections to that? If not, I'll remove ux-e2e-tests from the pipeline requirements.

kgartland-rstudio · 2023-09-21T17:03:11Z

Should we consider isolating these tests to run via a different mechanism? For instance, only on main or on a schedule?

Reconsidering this...
if we only run the CLI/UI tests (without actually publishing) in pull_request.yaml and then ran the full deploy tests in the main.yaml, would it be a big issue if they failed on the main run? Would a new PR would have to be opened? Would that get messy?

mmarchetti · 2023-09-21T17:18:11Z

Could we run the PR branches with

CLI/UI tests for all platforms
End to end publishing tests only on Linux

and run the nightly test with end to end publishing tests only on all platforms with fuzzbucket?

tdstein · 2023-09-21T17:23:03Z

Should we consider isolating these tests to run via a different mechanism? For instance, only on main or on a schedule?

Reconsidering this... if we only run the CLI/UI tests (without actually publishing) in pull_request.yaml and then ran the full deploy tests in the main.yaml, would it be a big issue if they failed on the main run? Would a new PR would have to be opened? Would that get messy?

This is my preferred approach. Considering these "acceptance tests" for the sake of conversation, we can make acceptance tests a pre-condition for releases, a typical CI/CD pattern.

If running these tests on pull requests makes more sense, I'm open to doing so. I want to ensure that we have a "break the glass" option if fuzzbucket or AWS goes down.

This afternoon, I would like to start discussing our release process so that we can begin producing pre-alpha distributions.

kgartland-rstudio · 2023-09-21T17:26:47Z

Could we run the PR branches with

CLI/UI tests for all platforms

End to end publishing tests only on Linux

and run the nightly test with end to end publishing tests only on all platforms with fuzzbucket?

We could. For this PR, maybe I'll rip out Connect and the publishing tests from pull_request.yaml and add them to a nightly run. In a follow-up PR, I'll bring Docker back in and run Connect in that for the Linux tests.

kgartland-rstudio · 2023-09-21T20:13:10Z

This PR now...

runs the ui-only tests in the pull_request.yaml
the full deploy tests with Connect in fuzzbucket in the new nightly.yaml

nightly workflow: https://github.com/rstudio/publishing-client/actions/runs/6266164364
pull_request workflow: https://github.com/rstudio/publishing-client/actions/runs/6266186318

In a follow-up PR, I'll add Connect in Docker for Linux builds in the main.yaml workflow

dotNomad · 2023-09-27T22:24:44Z

I also removed ux-e2e-tests since those were redundant with the linux testing we do later on. Does anyone have any objections to that? If not, I'll remove ux-e2e-tests from the pipeline requirements.

If I am tracking this correctly in our test justfile we do just test/ui-client ${os} which will run the e2e tests on each OS. It makes sense to remove the extra ux-e2e-tests then so we don't do it twice.

dotNomad · 2023-09-27T22:31:14Z

CLI/UI tests for all platforms

I don't think it makes sense to run our UI unit tests on all platforms each time?

For unit tests in the frontend our environment is jsdom which shouldn't change between OSes.

For our e2e tests browser based tests will be dependent on what browser is being used (and really what browser / rendering engine is being used). However we don't really split up e2e tests for browser specific features and full features across the UI and CLI. The CLI seems like the dependent-on-the-OS part of this project.

Feel free to let me know if I'm off base though.

kgartland-rstudio · 2023-09-28T12:55:17Z

I don't think it makes sense to run our UI unit tests on all platforms each time?

This PR doesn't impact the unit tests. Those are run once in linux I believe. These are the integration tests which start the UI / CLI and runs Cypress tests and CLI tests respectively on each OS. We've been bitten by Windows bugs in rsconnect-python because we didn't have an automated way to run against a Windows environments.

For our e2e tests browser based tests will be dependent on what browser is being used (and really what browser / rendering engine is being used). However we don't really split up e2e tests for browser specific features and full features across the UI and CLI. The CLI seems like the dependent-on-the-OS part of this project.

File paths are treated differently in Windows vs Unix systems so I think it's best to cover that in both the CLI and UI, though admittedly both will use the same files api. However, since we have multiple OSes up for the CLI, we may as well ensure the UI works properly in each binary too.

mmarchetti

Could you add a summary or table in the README of what tests are performed on which OSes for each of the GHA workflows? I'm a little unclear on that. We have agent (go) unit tests, UI unit tests, CLI deployment tests, and e2e deployment tests (at least... I feel like I'm missing another kind of UI test?).

mmarchetti · 2023-09-28T20:16:33Z

test/sample-content/python/fastapi-simple/.posit/deployments/default/id.json

@@ -0,0 +1 @@
+{"account_name":"","server_type":"","server_url":"","content_id":"","content_name":"","bundle_id":null,"deployed_at":null}


I'm wondering if it's correct to commit this metadata. In real user workflows, it's probably correct to allow for collaborators to redeploy to the same target. But, this target data is blank, and we probably want a new deployment anyway.

That was not intended to be added. I'll remove it.

mmarchetti · 2023-09-28T20:20:54Z

.github/workflows/nightly.yaml

+#     - UX e2e tests
+
+jobs:
+  build-docker-image:


It might be worth looking at how Taylor was able to reuse GHA parts across workflows. Maybe some of this can be factored out? Does not need to be part of this PR.

It looks like Taylor's release PR is in progress. I'll hold off on this PR and use his approach once that lands. It looks like some additional changes will be required for the tests to work with the refactor anyway.

mmarchetti · 2023-09-28T20:22:11Z

.github/workflows/nightly.yaml

+          name: web-ux
+          path: ./web/dist
+
+  validate-codebase:


Some steps might not be needed in the nightly test since they run on PRs and on main? Like validate-codebase or agent-tests.

dotNomad · 2023-09-29T20:52:28Z

Thanks for the explanation @kgartland-rstudio!

kgartland-rstudio · 2023-11-02T15:25:14Z

Closing in favor of #335

Kevin Gartland added 30 commits September 11, 2023 15:49

use fuzzbucket

06ae1a8

fuzzbucket ci

6282eaf

fixes

1d6ae2d

fix linux

d649864

fix linux

69ad8b0

fix windows

79e94e2

pass connect_ips

cc72448

pass connect_ips

86da697

pass connect_ips

9f76552

teardown connect

dac6d7c

fix script

25f47f5

fixes for windows

236392d

windows fix

1e17014

windows fix again

b114191

remove baseurl so we can test Connect URL

db4ab8d

null out the baseurl for Connect

31b8780

add separate command for windows

a46f539

typo

fdd5ae8

typo

41f1499

ugh

b84d2ce

re-enable linux-arm

f82ccb6

save servers file as backup

70736ad

log don't print

17e3c34

speed up ci

78680ac

speed up ci

c7220db

do that for linux-arm too

17e49f2

don't teardown the instance

0cc1580

run all python tests

1f66090

forgot deploy helper

401bf36

need build

7067356

kgartland-rstudio requested review from dotNomad, mmarchetti, sagerb and tdstein as code owners September 21, 2023 16:03

Merge branch 'main' into kg-fuzzbucket

9442ac3

udpate README

634589f

Kevin Gartland added 9 commits September 21, 2023 14:14

separate run for ui tests

794d34c

typo

05d864d

typo

511be72

remove CONNECT_SERVER env var

f015659

forgot to build

d119e74

forgot to build

def29f4

force a nightly run

6d19538

force a nightly run

12f9da3

revert to nightly

b576a5e

mmarchetti reviewed Sep 28, 2023

View reviewed changes

kgartland-rstudio closed this Nov 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fuzzbucket in CI #211

Fuzzbucket in CI #211

kgartland-rstudio commented Sep 12, 2023 •

edited

Loading

tdstein commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023 •

edited

Loading

mmarchetti commented Sep 21, 2023 •

edited

Loading

tdstein commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

dotNomad commented Sep 27, 2023

dotNomad commented Sep 27, 2023

kgartland-rstudio commented Sep 28, 2023 •

edited

Loading

mmarchetti left a comment

mmarchetti Sep 28, 2023

kgartland-rstudio Oct 11, 2023

mmarchetti Sep 28, 2023

kgartland-rstudio Oct 11, 2023

mmarchetti Sep 28, 2023

dotNomad commented Sep 29, 2023

kgartland-rstudio commented Nov 2, 2023

		@@ -0,0 +1 @@
		{"account_name":"","server_type":"","server_url":"","content_id":"","content_name":"","bundle_id":null,"deployed_at":null}

Fuzzbucket in CI #211

Fuzzbucket in CI #211

Conversation

kgartland-rstudio commented Sep 12, 2023 • edited Loading

Intent

Type of Change

Approach

Automated Tests

Directions for Reviewers

Checklist

tdstein commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023 • edited Loading

mmarchetti commented Sep 21, 2023 • edited Loading

tdstein commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

kgartland-rstudio commented Sep 21, 2023

dotNomad commented Sep 27, 2023

dotNomad commented Sep 27, 2023

kgartland-rstudio commented Sep 28, 2023 • edited Loading

mmarchetti left a comment

Choose a reason for hiding this comment

mmarchetti Sep 28, 2023

Choose a reason for hiding this comment

kgartland-rstudio Oct 11, 2023

Choose a reason for hiding this comment

mmarchetti Sep 28, 2023

Choose a reason for hiding this comment

kgartland-rstudio Oct 11, 2023

Choose a reason for hiding this comment

mmarchetti Sep 28, 2023

Choose a reason for hiding this comment

dotNomad commented Sep 29, 2023

kgartland-rstudio commented Nov 2, 2023

kgartland-rstudio commented Sep 12, 2023 •

edited

Loading

kgartland-rstudio commented Sep 21, 2023 •

edited

Loading

mmarchetti commented Sep 21, 2023 •

edited

Loading

kgartland-rstudio commented Sep 28, 2023 •

edited

Loading