Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New OnCall initialization process #4363

Closed
wants to merge 109 commits into from
Closed
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
0a63ceb
implement basic backend plugin and Api path config on the frontend
brojd May 20, 2024
03a3f8c
Merge branch 'dev' into brojd/backend-plugin-starter
brojd May 20, 2024
3b05cbe
update
brojd May 20, 2024
f3eb32c
update
brojd May 20, 2024
5de963a
Merge branch 'dev' into brojd/backend-plugin-starter
brojd May 21, 2024
c4817a4
update
brojd May 21, 2024
73c6631
add missing permission, move flag to env var
brojd May 21, 2024
f89dddf
update
brojd May 21, 2024
465066c
Enable drone build on branch
mderynck May 21, 2024
d98230d
Add build command
mderynck May 21, 2024
9d60c68
Split backend plugin build into separate step
mderynck May 21, 2024
d4aa462
Add build step
mderynck May 21, 2024
0fbf33e
Fix path
mderynck May 21, 2024
7e43bec
Enable externalServiceAccounts in docker compose dev
mderynck May 21, 2024
1a1f688
Merge branch 'dev' into brojd/backend-plugin-starter
brojd May 22, 2024
b9632c2
update
brojd May 22, 2024
a96d72d
Merge branch 'brojd/backend-plugin-starter' of github.com:grafana/onc…
brojd May 22, 2024
b9cf918
read useBackendPlugin from jsonData
brojd May 23, 2024
11b3b9a
update
brojd May 23, 2024
ce3ab85
remove unused import
brojd May 23, 2024
2243223
Read useBackendPlugin from jsonData (#4385)
brojd May 23, 2024
d9d0541
Basic working backend proxy
mderynck May 23, 2024
5885402
Merge
mderynck May 23, 2024
c990689
Add getting user id and permissions in backend plugin
mderynck May 24, 2024
61344c9
Add teams, handle case for non-rbac instance
mderynck May 27, 2024
9eb862a
Add install handler, disable token write from frontend
mderynck May 28, 2024
bedd61f
Revert provisioning change, lint
mderynck May 28, 2024
a4d112f
Revert provisioning
mderynck May 28, 2024
87890f5
Remove test for unused endpoints
mderynck May 28, 2024
210623c
Merge pull request #4392 from grafana/mderynck/backend-plugin-proxy
mderynck May 28, 2024
04f2bdc
Fix body not being sent by proxy
mderynck May 28, 2024
c6185f8
Merge branch 'dev' into brojd/backend-plugin-starter
brojd May 29, 2024
f42d253
Oncall initialization frontend cleanup (#4420)
brojd Jun 3, 2024
79d39d0
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 3, 2024
b550e68
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 4, 2024
9c5ce13
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 4, 2024
e9664f4
First e2e tests for plugin initialization (#4465)
brojd Jun 7, 2024
6e49e3c
Merge dev
mderynck Jun 7, 2024
7bfc867
Assemble data for user, team and permission sync in backend plugin
mderynck Jun 8, 2024
e1d1f52
Fix avatarUrl name
mderynck Jun 8, 2024
5a4b9f3
Move proxy add config for other plugin features
mderynck Jun 11, 2024
88a3a71
Merge pull request #4491 from grafana/mderynck/backend-plugin-2
mderynck Jun 11, 2024
d8011c4
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 12, 2024
b1c9e07
Fix user lookup and grafana URL for 10.1.7, fix sync check for rbac
mderynck Jun 12, 2024
f26e0a0
Merge branch 'brojd/backend-plugin-starter' of github.com:grafana/onc…
mderynck Jun 12, 2024
cce4918
WIP
mderynck Jun 15, 2024
6b05e01
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 17, 2024
410056b
Merge branch 'brojd/backend-plugin-starter' of github.com:grafana/onc…
brojd Jun 17, 2024
da1130e
WIP basic OSS install
mderynck Jun 18, 2024
ca5e8f8
WIP basic OSS status
mderynck Jun 18, 2024
b6b4013
WIP remove test permission
mderynck Jun 18, 2024
a9b1fb9
WIP remove test command
mderynck Jun 18, 2024
5441b14
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 18, 2024
036d773
fix rbac check, add grafanaUrl to provisioning file, fix Tiltfile (#4…
brojd Jun 18, 2024
682a5f4
Lint
mderynck Jun 18, 2024
6f6ea02
Merge pull request #4551 from grafana/mderynck/plugin-init-install-st…
mderynck Jun 18, 2024
670714f
Plugin configuration UI, full page error (#4507)
brojd Jun 19, 2024
8a06b66
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 19, 2024
c89d4d7
Fix install endpoint
mderynck Jun 19, 2024
b0a6205
Use provisioned grafana url for now instead of appurl
mderynck Jun 19, 2024
f4a58e5
Set provisioned grafana url to http://grafana:3000
mderynck Jun 19, 2024
7b71a2e
resetConnectionStatus
brojd Jun 20, 2024
20bd24d
more e2e tests
brojd Jun 20, 2024
b819fbe
bring back previous timeout
brojd Jun 20, 2024
e2a39f5
Fix error status code used for /install (#4571)
mderynck Jun 21, 2024
c11af49
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 21, 2024
42ea1f4
Merge branch 'brojd/backend-plugin-starter' into brojd/plugin-initial…
brojd Jun 21, 2024
012419a
improve refreshing applied url
brojd Jun 21, 2024
b575010
implement health check in /status and /install
brojd Jun 21, 2024
3925677
update
brojd Jun 21, 2024
11209fe
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 24, 2024
054e9e9
Merge branch 'brojd/backend-plugin-starter' into brojd/plugin-initial…
brojd Jun 24, 2024
f6effc4
set ONCALL_API_URL value in provisioning file based on env var (#4558)
brojd Jun 24, 2024
bb03583
Add backend plugin org sync and rework existing sync
matiasb Jun 24, 2024
ed890f0
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jun 27, 2024
048f002
Merge branch 'brojd/backend-plugin-starter' into brojd/plugin-initial…
brojd Jun 27, 2024
ebb084b
Merge branch 'brojd/backend-plugin-starter' into matiasb/engine-sync-…
brojd Jun 27, 2024
de41691
fix health check
brojd Jun 27, 2024
6a48ecd
improve error messages
brojd Jun 27, 2024
55d1db0
remove test that breaks configuration on the backend
brojd Jun 27, 2024
28ffbb3
update
brojd Jun 27, 2024
1a4a180
fix test
brojd Jun 27, 2024
8afa118
Merge pull request #4581 from grafana/matiasb/engine-sync-organization
brojd Jun 27, 2024
57c4c05
Merge branch 'brojd/backend-plugin-starter' into brojd/plugin-initial…
brojd Jun 27, 2024
771d582
handle sync_teams is None
brojd Jun 27, 2024
154f287
update
brojd Jun 27, 2024
a589a80
Merge pull request #4567 from grafana/brojd/plugin-initialization-imp…
brojd Jun 27, 2024
7f70ce1
install Mage, Go and build plugin backend on Github CI
brojd Jul 8, 2024
11694eb
Merge dev
matiasb Jul 8, 2024
fd4b23b
Fix merge issues
matiasb Jul 8, 2024
2347a2a
Fix issues post-merge
matiasb Jul 8, 2024
e6e5b1c
Sync teams on user creation during first login
matiasb Jul 1, 2024
2bcd01c
Add authentication for sync endpoint engine-side
matiasb Jul 3, 2024
d365c27
Update codeowners
matiasb Jul 3, 2024
df42162
Add tests
matiasb Jul 3, 2024
6f5a884
Update v2 sync org/stack check
matiasb Jul 4, 2024
e578df7
Add a few tests
matiasb Jul 5, 2024
72ad172
Merge dev
matiasb Jul 8, 2024
e5c729c
Merge pull request #4604 from grafana/matiasb/user-header-including-t…
matiasb Jul 8, 2024
6af98f8
Merge pull request #4620 from grafana/brojd/new-init-process-dev-depl…
brojd Jul 9, 2024
2a0db2f
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jul 9, 2024
fd5ba8a
use more Grafana versions in PRs e2e tests
brojd Jul 9, 2024
72fc09f
update
brojd Jul 9, 2024
2926837
Check for external service account before setting teams in header
matiasb Jul 9, 2024
049b85a
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jul 10, 2024
dbae55c
add comment
brojd Jul 10, 2024
e9a091a
Merge pull request #4639 from grafana/brojd/run-e2e-tests-on-prs-agai…
brojd Jul 10, 2024
289a24d
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jul 10, 2024
9a2fbb0
Merge branch 'dev' into brojd/backend-plugin-starter
brojd Jul 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/e2e-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,11 @@ jobs:

# ---------- Expensive e2e tests steps start -----------
- name: Install Go
if: inputs.run-expensive-tests
uses: actions/setup-go@v4
with:
go-version: "1.21.5"

- name: Install Mage
if: inputs.run-expensive-tests
run: go install github.com/magefile/[email protected]

- name: Get Vault secrets
Expand Down
30 changes: 20 additions & 10 deletions .github/workflows/linting-and-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ env:
jobs:
lint-entire-project:
name: "Lint entire project"
runs-on: ubuntu-latest
runs-on: ubuntu-latest-16-cores
steps:
- name: Checkout project
uses: actions/checkout@v4
Expand All @@ -26,7 +26,7 @@ jobs:

lint-test-and-build-frontend:
name: "Lint, test, and build frontend"
runs-on: ubuntu-latest
runs-on: ubuntu-latest-16-cores
steps:
- name: Checkout project
uses: actions/checkout@v4
Expand All @@ -38,7 +38,7 @@ jobs:

test-technical-documentation:
name: "Test technical documentation"
runs-on: ubuntu-latest
runs-on: ubuntu-latest-16-cores
steps:
- name: "Check out code"
uses: "actions/checkout@v4"
Expand All @@ -55,7 +55,7 @@ jobs:

lint-migrations-backend-mysql-rabbitmq:
name: "Lint database migrations"
runs-on: ubuntu-latest
runs-on: ubuntu-latest-16-cores
services:
rabbit_test:
image: rabbitmq:3.12.0
Expand Down Expand Up @@ -86,7 +86,7 @@ jobs:

unit-test-helm-chart:
name: "Helm Chart Unit Tests"
runs-on: ubuntu-latest
runs-on: ubuntu-latest-16-cores
steps:
- name: Checkout project
uses: actions/checkout@v4
Expand All @@ -98,6 +98,16 @@ jobs:
- name: Run tests
run: helm unittest ./helm/oncall

unit-test-backend-plugin:
name: "Backend Tests: Plugin"
runs-on: ubuntu-latest-16-cores
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v4
with:
go-version: "1.21.5"
- run: cd grafana-plugin && go test ./pkg/...

unit-test-backend-mysql-rabbitmq:
name: "Backend Tests: MySQL + RabbitMQ (RBAC enabled: ${{ matrix.rbac_enabled }})"
runs-on: ubuntu-latest-16-cores
Expand Down Expand Up @@ -201,7 +211,7 @@ jobs:

unit-test-migrators:
name: "Unit tests - Migrators"
runs-on: ubuntu-latest
runs-on: ubuntu-latest-16-cores
steps:
- name: Checkout project
uses: actions/checkout@v4
Expand All @@ -215,7 +225,7 @@ jobs:

mypy:
name: "mypy"
runs-on: ubuntu-latest
runs-on: ubuntu-latest-16-cores
steps:
- name: Checkout project
uses: actions/checkout@v4
Expand All @@ -229,8 +239,8 @@ jobs:
name: Standard e2e tests
uses: ./.github/workflows/e2e-tests.yml
with:
# TODO: fix issues with running e2e tests against Grafana v10.2.x and v10.3.x
grafana_version: 10.1.7
# grafana_version: 10.3.3
# TODO: fix issues with running e2e tests against Grafana v10.1.x, v10.2.x
# grafana_version: 10.1.7
grafana_version: 10.3.3
run-expensive-tests: false
browsers: "chromium"
26 changes: 22 additions & 4 deletions Tiltfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ if not running_under_parent_tiltfile:
# Load the custom Grafana extensions
v1alpha1.extension_repo(
name="grafana-tilt-extensions",
ref="v1.2.0",
ref="v1.4.2",
url="https://github.com/grafana/tilt-extensions",
)
v1alpha1.extension(
Expand Down Expand Up @@ -63,7 +63,7 @@ docker_build_sub(
if is_ci:
local_resource(
"build-ui",
labels=["OnCallUI"],
labels=["OnCallPluginUI"],
dir="grafana-plugin",
cmd="yarn build",
allow_parallel=True,
Expand All @@ -81,13 +81,30 @@ if not is_ci:
allow_parallel=True,
)

local_resource(
'build-oncall-plugin-backend',
labels=["OnCallPluginBackend"],
dir="./grafana-plugin",
cmd="mage buildAll",
deps=['grafana-plugin/pkg/plugin']
)

local_resource(
'restart-oncall-plugin-backend',
labels=["OnCallPluginBackend"],
dir="./dev/scripts",
cmd="chmod +x ./restart_backend_plugin.sh && ./restart_backend_plugin.sh",
resource_deps=["grafana", "build-oncall-plugin-backend"],
deps=['grafana-plugin/pkg/plugin']
)

local_resource(
"e2e-tests",
labels=["E2eTests"],
cmd=e2e_tests_cmd,
trigger_mode=TRIGGER_MODE_MANUAL,
auto_init=is_ci,
resource_deps=["build-ui", "grafana", "grafana-oncall-app-provisioning-configmap", "engine", "celery"]
resource_deps=["build-ui", "grafana", "grafana-oncall-app-provisioning-configmap", "engine", "celery", "build-oncall-plugin-backend"]
)

cmd_button(
Expand Down Expand Up @@ -155,11 +172,12 @@ if not running_under_parent_tiltfile:
context="grafana-plugin",
plugin_files=["grafana-plugin/src/plugin.json"],
namespace="default",
deps=["grafana-oncall-app-provisioning-configmap", "build-ui"],
deps=["grafana-oncall-app-provisioning-configmap", "build-ui", "build-oncall-plugin-backend"],
extra_env={
"GF_SECURITY_ADMIN_PASSWORD": "oncall",
"GF_SECURITY_ADMIN_USER": "oncall",
"GF_AUTH_ANONYMOUS_ENABLED": "false",
"GF_FEATURE_TOGGLES_ENABLE": "externalServiceAccounts",
},
)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ apps:
jsonData:
stackId: 5
orgId: 100
license: OpenSource
onCallApiUrl: http://oncall-dev-engine:8080
grafanaUrl: http://grafana:3000
2 changes: 1 addition & 1 deletion dev/helm-local.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ grafana:
- name: DATABASE_PASSWORD
value: oncallpassword
env:
GF_FEATURE_TOGGLES_ENABLE: topnav
GF_FEATURE_TOGGLES_ENABLE: topnav,externalServiceAccounts
GF_SECURITY_ADMIN_PASSWORD: oncall
GF_SECURITY_ADMIN_USER: oncall
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: grafana-oncall-app
Expand Down
23 changes: 23 additions & 0 deletions dev/scripts/restart_backend_plugin.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/bash

# Find a grafana pod
pod=$(kubectl get pods -l app.kubernetes.io/name=grafana -o=jsonpath='{.items[0].metadata.name}')

if [ -z "$pod" ]; then
echo "No pod found with the specified label."
exit 1
fi

# Exec into the pod
kubectl exec -it "$pod" -- /bin/bash <<'EOF'

# Find and kill the process containing "gpx_grafana" (plugin backend process)
process_id=$(ps aux | grep gpx_grafana | grep -v grep | awk '{print $1}')
echo $process_id
if [ -n "$process_id" ]; then
echo "Killing process $process_id"
kill $process_id
else
echo "No process containing 'gpx_grafana' in COMMAND found."
fi
EOF
1 change: 1 addition & 0 deletions docker-compose-developer.yml
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,7 @@ services:
GF_SECURITY_ADMIN_USER: oncall
GF_SECURITY_ADMIN_PASSWORD: oncall
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: grafana-oncall-app
GF_FEATURE_TOGGLES_ENABLE: externalServiceAccounts
env_file:
- ./dev/.env.${DB}.dev
ports:
Expand Down
16 changes: 7 additions & 9 deletions engine/apps/auth_token/auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,16 +149,14 @@ def _get_user(request: Request, organization: Organization) -> User:
except (ValueError, TypeError):
raise exceptions.AuthenticationFailed("Grafana context must be JSON dict.")

if "UserId" not in context and "UserID" not in context:
raise exceptions.AuthenticationFailed("Invalid Grafana context.")

try:
user_id = context["UserId"]
except KeyError:
user_id = context["UserID"]

try:
return organization.users.get(user_id=user_id)
user_id = context.get("UserId", context.get("UserID"))
if user_id is not None:
return organization.users.get(user_id=user_id)
elif "Login" in context:
return organization.users.get(username=context["Login"])
else:
raise exceptions.AuthenticationFailed("Grafana context must specify a User or UserID.")
except User.DoesNotExist:
logger.debug(f"Could not get user from grafana request. Context {context}")
raise exceptions.AuthenticationFailed("Non-existent or anonymous user.")
Expand Down
4 changes: 2 additions & 2 deletions engine/apps/grafana_plugin/helpers/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -242,8 +242,8 @@ def get_users_permissions(self) -> typing.Optional[UserPermissionsDict]:
return all_users_permissions

def is_rbac_enabled_for_organization(self) -> bool:
_, resp_status = self.api_head(self.USER_PERMISSION_ENDPOINT)
return resp_status["connected"]
response, resp_status = self.api_get(self.USER_PERMISSION_ENDPOINT)
return response is not None and len(response) > 0 and resp_status["connected"]

def get_users(self, rbac_is_enabled_for_org: bool, **kwargs) -> GrafanaUsersWithPermissions:
users_response, _ = self.api_get("api/org/users", **kwargs)
Expand Down
105 changes: 105 additions & 0 deletions engine/apps/grafana_plugin/serializers/sync_data.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
from dataclasses import asdict
from typing import Dict, List

from rest_framework import serializers

from apps.grafana_plugin.sync_data import SyncData, SyncPermission, SyncSettings, SyncTeam, SyncUser


class SyncPermissionSerializer(serializers.Serializer):
action = serializers.CharField()

def create(self, validated_data):
return SyncPermission(**validated_data)

def to_representation(self, instance):
return asdict(instance)


class SyncUserSerializer(serializers.Serializer):
id = serializers.IntegerField()
name = serializers.CharField(allow_blank=True)
login = serializers.CharField()
email = serializers.EmailField()
role = serializers.CharField()
avatar_url = serializers.CharField()
permissions = SyncPermissionSerializer(many=True, allow_empty=True, allow_null=True)
teams = serializers.ListField(child=serializers.IntegerField(), allow_empty=True, allow_null=True)

def create(self, validated_data):
return SyncUser(**validated_data)

def to_representation(self, instance):
return asdict(instance)


class SyncTeamSerializer(serializers.Serializer):
team_id = serializers.IntegerField()
name = serializers.CharField()
email = serializers.EmailField()
avatar_url = serializers.CharField()

def create(self, validated_data):
return SyncTeam(**validated_data)

def to_representation(self, instance):
return asdict(instance)


class TeamMemberMappingField(serializers.Field):
def to_representation(self, value: Dict[int, List[int]]):
return {str(k): v for k, v in value.items()}

def to_internal_value(self, data):
if not isinstance(data, dict):
raise serializers.ValidationError("Expected a dictionary")
try:
return {int(k): v for k, v in data.items()}
except ValueError:
raise serializers.ValidationError("All keys must be convertible to integers")


class SyncOnCallSettingsSerializer(serializers.Serializer):
stack_id = serializers.IntegerField()
org_id = serializers.IntegerField()
license = serializers.CharField()
oncall_api_url = serializers.CharField()
oncall_token = serializers.CharField(allow_blank=True)
grafana_url = serializers.CharField()
grafana_token = serializers.CharField()
rbac_enabled = serializers.BooleanField()
incident_enabled = serializers.BooleanField()
incident_backend_url = serializers.CharField(allow_blank=True)
labels_enabled = serializers.BooleanField()

def create(self, validated_data):
return SyncSettings(**validated_data)

def to_representation(self, instance):
return asdict(instance)


class SyncDataSerializer(serializers.Serializer):
users = serializers.ListField(child=SyncUserSerializer())
teams = serializers.ListField(child=SyncTeamSerializer(), allow_null=True, allow_empty=True)
team_members = TeamMemberMappingField()
settings = SyncOnCallSettingsSerializer()

def create(self, validated_data):
return SyncData(**validated_data)

def to_representation(self, instance):
return asdict(instance)

def to_internal_value(self, data):
data = super().to_internal_value(data)
users = data.get("users")
if users:
data["users"] = [SyncUser(**user) for user in users]
teams = data.get("teams")
if teams:
data["teams"] = [SyncTeam(**team) for team in teams]
settings = data.get("settings")
if settings:
data["settings"] = SyncSettings(**settings)
return data
Loading
Loading