-
Notifications
You must be signed in to change notification settings - Fork 56
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add test plan & results template for kruize rel 0.2
- Loading branch information
1 parent
b9e71c4
commit 23a6546
Showing
1 changed file
with
144 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| # KRUIZE TEST PLAN RELEASE 0.2 | ||
|
|
||
| - [INTRODUCTION](#introduction) | ||
| - [FEATURES TO BE TESTED](#features-to-be-tested) | ||
| - [BUG FIXES TO BE TESTED](#bug-fixes-to-be-tested) | ||
| - [TEST ENVIRONMENT](#test-environment) | ||
| - [TEST DELIVERABLES](#test-deliverables) | ||
| - [New Test Cases Developed](#new-test-cases-developed) | ||
| - [Regression Testing](#regresion-testing) | ||
| - [SCALABILITY TESTING](#scalability-testing) | ||
| - [RELEASE TESTING](#release-testing) | ||
| - [TEST METRICS](#test-metrics) | ||
| - [RISKS AND CONTINGENCIES](#risks-and-contingencies) | ||
| - [APPROVALS](#approvals) | ||
|
|
||
| ----- | ||
|
|
||
| ## INTRODUCTION | ||
|
|
||
| This document describes the test plan for Kruize remote monitoring release 0.2 | ||
|
|
||
| ---- | ||
|
|
||
| ## FEATURES TO BE TESTED | ||
|
|
||
| * Webhook implementation | ||
| * Add time range filter for bulk API | ||
| * Bulk API error handling with New JSON | ||
| * Database updates for authentication | ||
| * Add test script for authentication | ||
| * Bulk test cases | ||
| * Datasource Exception handler | ||
|
|
||
|
|
||
| ------ | ||
|
|
||
| ## BUG FIXES TO BE TESTED | ||
|
|
||
| * Time_range fix | ||
| * Fix datasource missing issue on pod restart | ||
| * Environment variable error found in the log | ||
| * Removed the autotune job from the PR check workflow | ||
|
|
||
| --- | ||
|
|
||
| ## TEST ENVIRONMENT | ||
|
|
||
| * Minikube Cluster | ||
| * Openshift Cluster | ||
|
|
||
| --- | ||
|
|
||
| ## TEST DELIVERABLES | ||
|
|
||
| ### New Test Cases Developed | ||
|
|
||
| | # | ISSUE (NEW FEATURE) | TEST DESCRIPTION | TEST DELIVERABLES | RESULTS | COMMENTS | | ||
| |---|---------------------------------------|----------------------------------------------------------------------|-------------------|---------|----------| | ||
| | 1 | Webhook implementation | | | | | | ||
| | 2 | Add time range filter for bulk API | | | | | | ||
| | 3 | Bulk API error handling with New JSON | | | | | ||
| | 4 | Database updates for authentication | New tests added [1307](https://github.com/kruize/autotune/pull/1307) | | | | | ||
| | 5 | Datasource Exception handler | | | | | | ||
|
|
||
|
|
||
| ### Regression Testing | ||
|
|
||
| | # | ISSUE (BUG/NEW FEATURE) | TEST CASE | RESULTS | COMMENTS | | ||
| |---|-----------------------------------------------------|-----------|---------|----------| | ||
| | 1 | Fix datasource missing issue on pod restart | | | | | ||
| | 2 | Time_range fix | | | | | ||
| | 3 | Environment variable error found in the log | | | | | ||
| | 4 | Removed the autotune job from the PR check workflow | | | | | ||
|
|
||
| --- | ||
|
|
||
| ## SCALABILITY TESTING | ||
|
|
||
| Evaluate Kruize Scalability on OCP, with 5k experiments by uploading resource usage data for 15 days and update recommendations. | ||
| Changes do not have scalability implications. Short scalability test will be run as part of the release testing | ||
|
|
||
| Short Scalability run | ||
| - 5K exps / 15 days of results / 2 containers per exp | ||
| - Kruize replicas - 10 | ||
| - OCP - Scalelab cluster | ||
|
|
||
| | Kruize Release | Exps / Results / Recos | Execution time | Latency (Max/ Avg) in seconds | | | Postgres DB size(MB) | Kruize Max CPU | Kruize Max Memory (GB) | | ||
| |----------------|------------------------|----------------|-------------------------------|---------------|----------------------|----------------------|----------------|------------------------| | ||
| | | | | UpdateRecommendations | UpdateResults | LoadResultsByExpName | | | | | ||
| | 0.1 | 5K / 72L / 3L | 5h 02 mins | 0.97 / 0.55 | 0.16 / 0.14 | 0.52 / 0.36 | 21757 | 7.3 | 33.67 | | ||
| | 0.2 | 5K / 72L / 3L | | | | | | | | | ||
|
|
||
| ---- | ||
| ## RELEASE TESTING | ||
|
|
||
| As part of the release testing, following tests will be executed: | ||
| - [Kruize Remote monitoring Functional tests](/tests/scripts/remote_monitoring_tests/Remote_monitoring_tests.md) | ||
| - [Fault tolerant test](/tests/scripts/remote_monitoring_tests/fault_tolerant_tests.md) | ||
| - [Stress test](/tests/scripts/remote_monitoring_tests/README.md) | ||
| - [DB Migration test](/tests/scripts/remote_monitoring_tests/db_migration_test.md) | ||
| - [Recommendation and box plot values validation test](https://github.com/kruize/kruize-demos/blob/main/monitoring/remote_monitoring_demo/recommendations_infra_demo/README.md) | ||
| - [Scalability test (On openshift)](/tests/scripts/remote_monitoring_tests/scalability_test.md) - scalability test with 5000 exps / 15 days usage data | ||
| - [Kruize remote monitoring demo (On minikube)](https://github.com/kruize/kruize-demos/blob/main/monitoring/remote_monitoring_demo/README.md) | ||
| - [Kruize local monitoring demo (On openshift)](https://github.com/kruize/kruize-demos/blob/main/monitoring/local_monitoring_demo) | ||
| - [Kruize local monitoring Functional tests](/tests/scripts/local_monitoring_tests/Local_monitoring_tests.md) | ||
|
|
||
|
|
||
| | # | TEST SUITE | EXPECTED RESULTS | ACTUAL RESULTS | COMMENTS | | ||
| |---|------------------------------------------------|-----------------------------------------|-----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | 1 | Kruize Remote monitoring Functional testsuite | TOTAL - 359, PASSED - 316 / FAILED - 43 | TOTAL - 359, PASSED - 316 / FAILED - 43 | Intermittent issue seen [1281](https://github.com/kruize/autotune/issues/1281), existing issues - [559](https://github.com/kruize/autotune/issues/559), [610](https://github.com/kruize/autotune/issues/610) | | ||
| | 2 | Fault tolerant test | PASSED | | | | ||
| | 3 | Stress test | PASSED | | | | ||
| | 4 | Scalability test (short run) | | | | | ||
| | 5 | DB Migration test | PASSED | | | | ||
| | 6 | Recommendation and box plot values validations | PASSED | | | | ||
| | 7 | Kruize remote monitoring demo | PASSED | | | | ||
| | 8 | Kruize Local monitoring demo | PASSED | | | | ||
| | 9 | Kruize Local Functional tests | TOTAL - 78, PASSED - 75 / FAILED - 3 | | | | ||
|
|
||
| --- | ||
|
|
||
| ## TEST METRICS | ||
|
|
||
| ### Test Completion Criteria | ||
|
|
||
| * All must_fix defects identified for the release are fixed | ||
| * New features work as expected and tests have been added to validate these | ||
| * No new regressions in the functional tests | ||
| * All non-functional tests work as expected without major issues | ||
| * Documentation updates have been completed | ||
|
|
||
| ---- | ||
|
|
||
| ## RISKS AND CONTINGENCIES | ||
|
|
||
| * None | ||
|
|
||
| ---- | ||
| ## APPROVALS | ||
|
|
||
| Sign-off | ||
|
|
||
| ---- | ||
|
|