Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🌟 [WIP] DB Migration Tool #487

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

🌟 [WIP] DB Migration Tool #487

wants to merge 1 commit into from

Conversation

MontaGhanmy
Copy link
Collaborator

No description provided.

@MontaGhanmy MontaGhanmy self-assigned this Apr 23, 2024
@github-actions GitHub Actions
Copy link

Coverage Report

  • Status: ✅ Passed
Click to view remaining coverage report
Status File Stmts Branch Funcs Lines
total 67.17 48.25 62.72 67.15
/usr/src/app/src/version.ts 100 100 100 100
/usr/src/app/src/core/config/index.ts 100 100 100 100
/usr/src/app/src/core/crypto/index.ts 95.65 100 100 95.45
/usr/src/app/src/core/crypto/legacy.ts 85.71 57.14 50 85.71
/usr/src/app/src/core/crypto/v1.ts 86.36 71.42 100 86.36
/usr/src/app/src/core/crypto/v2.ts 88 71.42 100 88
/usr/src/app/src/core/platform/platform.ts 80 100 50 80
/usr/src/app/src/core/platform/framework/configuration.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/event-bus.ts 93.75 83.33 100 93.75
/usr/src/app/src/core/platform/framework/execution-storage.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/factory.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/index.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/logger.ts 100 66.66 100 100
/usr/src/app/src/core/platform/framework/api/application-configuration.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/api/component.ts 92.59 81.81 100 92.3
/usr/src/app/src/core/platform/framework/api/constants.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/api/container.ts 70.37 50 66.66 76
/usr/src/app/src/core/platform/framework/api/crud-service.ts 96.29 83.33 90.9 95.55
/usr/src/app/src/core/platform/framework/api/index.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/api/service-options.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/api/service-state.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/api/service.ts 58.73 68.75 77.77 58.73
/usr/src/app/src/core/platform/framework/decorators/consumes.ts 100 0 100 100
/usr/src/app/src/core/platform/framework/decorators/index.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/decorators/prefix.ts 100 0 100 100
/usr/src/app/src/core/platform/framework/decorators/service-name.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/decorators/skip.ts 92.85 66.66 100 92.3
/usr/src/app/src/core/platform/framework/decorators/realtime/created.ts 100 80 100 100
/usr/src/app/src/core/platform/framework/decorators/realtime/deleted.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/decorators/realtime/index.ts 90 57.14 75 94.44
/usr/src/app/src/core/platform/framework/decorators/realtime/saved.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/decorators/realtime/updated.ts 100 100 100 100
/usr/src/app/src/core/platform/framework/utils/component-utils.ts 73.8 11.11 100 75.6
/usr/src/app/src/core/platform/framework/utils/loader.ts 76.19 60 85.71 73.68
/usr/src/app/src/core/platform/services/auth/index.ts 100 100 100 100
/usr/src/app/src/core/platform/services/auth/service.ts 85.71 50 66.66 85.71
/usr/src/app/src/core/platform/services/auth/web/index.ts 100 100 100 100
/usr/src/app/src/core/platform/services/auth/web/jwt.ts 100 88.88 100 100
/usr/src/app/src/core/platform/services/auth/web/routes.ts 62.5 100 33.33 71.42
/usr/src/app/src/core/platform/services/counter/index.ts 100 100 100 100
/usr/src/app/src/core/platform/services/counter/provider.ts 85.71 80 100 85.71
/usr/src/app/src/core/platform/services/cron/index.ts 34.78 0 28.57 34.78
/usr/src/app/src/core/platform/services/database/index.ts 74.07 0 75 72
/usr/src/app/src/core/platform/services/database/services/connector-factory.ts 66.66 25 100 66.66
/usr/src/app/src/core/platform/services/database/services/index.ts 77.77 50 60 77.77
/usr/src/app/src/core/platform/services/database/services/orm/manager.ts 84.05 70 76.92 87.87
/usr/src/app/src/core/platform/services/database/services/orm/utils.ts 84.61 73.91 90 86.48
/usr/src/app/src/core/platform/services/database/services/orm/connectors/abstract-connector.ts 66.66 100 33.33 50
/usr/src/app/src/core/platform/services/database/services/orm/connectors/cassandra/cassandra.ts 76.21 55.73 70.68 76.26
/usr/src/app/src/core/platform/services/database/services/orm/connectors/cassandra/pagination.ts 22.22 0 0 25
/usr/src/app/src/core/platform/services/database/services/orm/connectors/cassandra/query-builder.ts 92.3 73.07 84.61 93.47
/usr/src/app/src/core/platform/services/database/services/orm/connectors/cassandra/typeTransforms.ts 81.25 80.3 100 81.81
/usr/src/app/src/core/platform/services/database/services/orm/connectors/mongodb/mongodb.ts 7.61 0 0 8
/usr/src/app/src/core/platform/services/database/services/orm/connectors/mongodb/query-builder.ts 14.28 0 0 15.38
/usr/src/app/src/core/platform/services/database/services/orm/connectors/mongodb/typeTransforms.ts 12.28 0 0 9.25
/usr/src/app/src/core/platform/services/database/services/orm/connectors/postgres/postgres-data-transform.ts 72.58 77.55 100 72.88
/usr/src/app/src/core/platform/services/database/services/orm/connectors/postgres/postgres-query-builder.ts 95.87 80 100 95.4
/usr/src/app/src/core/platform/services/database/services/orm/connectors/postgres/postgres.ts 47.24 14.81 42.85 47.96
/usr/src/app/src/core/platform/services/database/services/orm/decorators/column.ts 100 100 100 100
/usr/src/app/src/core/platform/services/database/services/orm/decorators/entity.ts 100 50 100 100
/usr/src/app/src/core/platform/services/database/services/orm/decorators/index.ts 100 100 100 100
/usr/src/app/src/core/platform/services/database/services/orm/repository/manager.ts 84.61 100 100 84.61
/usr/src/app/src/core/platform/services/database/services/orm/repository/repository.ts 83.78 77.77 76.92 87.87
/usr/src/app/src/core/platform/services/email-pusher/index.ts 63.88 40.74 100 63.38
/usr/src/app/src/core/platform/services/message-queue/api.ts 81.25 57.14 77.77 82.6
/usr/src/app/src/core/platform/services/message-queue/bus.ts 83.33 100 66.66 83.33
/usr/src/app/src/core/platform/services/message-queue/factory.ts 83.33 50 100 83.33
/usr/src/app/src/core/platform/services/message-queue/index.ts 80.95 50 75 80
/usr/src/app/src/core/platform/services/message-queue/processor.ts 58.33 20 50 58.33
/usr/src/app/src/core/platform/services/message-queue/proxy.ts 52.63 23.8 40 53.57
/usr/src/app/src/core/platform/services/message-queue/amqp/client.ts 80.95 60 84.21 80.48
/usr/src/app/src/core/platform/services/message-queue/amqp/constants.ts 100 100 100 100
/usr/src/app/src/core/platform/services/message-queue/amqp/index.ts 73.07 100 55.55 73.07
/usr/src/app/src/core/platform/services/message-queue/amqp/manager.ts 76.74 33.33 66.66 76.74
/usr/src/app/src/core/platform/services/message-queue/amqp/pubsub.ts 84.21 47.82 83.33 84.21
/usr/src/app/src/core/platform/services/message-queue/amqp/pubsubclient.ts 54.83 38.88 46.15 58.62
/usr/src/app/src/core/platform/services/message-queue/local/index.ts 15.62 0 0 15.62
/usr/src/app/src/core/platform/services/push/index.ts 63.15 16.66 75 58.82
/usr/src/app/src/core/platform/services/push/connectors/fcm/service.ts 18.75 0 0 18.75
/usr/src/app/src/core/platform/services/realtime/bus.ts 100 100 100 100
/usr/src/app/src/core/platform/services/realtime/index.ts 80 100 50 78.57
/usr/src/app/src/core/platform/services/realtime/types.ts 94.44 66.66 75 94.11
/usr/src/app/src/core/platform/services/realtime/services/entity-manager.ts 81.81 0 66.66 81.81
/usr/src/app/src/core/platform/services/realtime/services/room-manager.ts 11.47 0 11.11 10.71
/usr/src/app/src/core/platform/services/search/api.ts 100 100 100 100
/usr/src/app/src/core/platform/services/search/index.ts 63.04 22.22 58.33 61.9
/usr/src/app/src/core/platform/services/search/repository.ts 88.88 33.33 100 88.46
/usr/src/app/src/core/platform/services/search/adapters/abstract.ts 54.54 100 28.57 54.54
/usr/src/app/src/core/platform/services/search/adapters/ascii-folder.ts 64.7 30 50 91.66
/usr/src/app/src/core/platform/services/search/adapters/utils.ts 76.92 0 77.77 77.27
/usr/src/app/src/core/platform/services/search/adapters/elasticsearch/elastic-open-search-adapter.ts 73.64 60.16 76.92 73.01
/usr/src/app/src/core/platform/services/search/adapters/elasticsearch/search.ts 90.16 81.6 100 90.16
/usr/src/app/src/core/platform/services/search/adapters/mongosearch/index.ts 10.41 0 0 10.41
/usr/src/app/src/core/platform/services/search/adapters/mongosearch/search.ts 11.11 0 0 11.42
/usr/src/app/src/core/platform/services/storage/index.ts 84.52 67.3 91.66 83.75
/usr/src/app/src/core/platform/services/storage/connectors/S3/s3-service.ts 70.17 58.82 83.33 71.69
/usr/src/app/src/core/platform/services/storage/connectors/local/service.ts 16.66 0 0 16.66
/usr/src/app/src/core/platform/services/tracker/index.ts 30.43 0 30 28.57
/usr/src/app/src/core/platform/services/tracker/adapters/segment.ts 11.11 0 0 11.53
/usr/src/app/src/core/platform/services/webserver/error.ts 100 100 100 100
/usr/src/app/src/core/platform/services/webserver/index.ts 81.94 9.09 81.25 83.09
/usr/src/app/src/core/platform/services/websocket/index.ts 93.75 100 75 92.85
/usr/src/app/src/core/platform/services/websocket/services/index.ts 36.11 50 41.66 36.11
/usr/src/app/src/services/global-resolver.ts 97.36 50 100 97.36
/usr/src/app/src/services/applications/index.ts 92.3 100 75 90.9
/usr/src/app/src/services/applications/realtime.ts 50 100 0 50
/usr/src/app/src/services/applications-api/index.ts 58.53 50 50 57.89
/usr/src/app/src/services/applications-api/web/index.ts 100 100 100 100
/usr/src/app/src/services/applications-api/web/routes.ts 72.22 0 25 76.47
/usr/src/app/src/services/applications-api/web/schemas.ts 100 100 100 100
/usr/src/app/src/services/applications-api/web/controllers/index.ts 15.68 0 0 15.68
/usr/src/app/src/services/applications/entities/application.ts 45.45 100 0 33.33
/usr/src/app/src/services/applications/services/applications.ts 50 0 25 60
/usr/src/app/src/services/applications/services/company-applications.ts 17.39 0 25 17.39
/usr/src/app/src/services/applications/services/hooks.ts 25.8 0 25 25.8
/usr/src/app/src/services/applications/web/index.ts 100 100 100 100
/usr/src/app/src/services/applications/web/routes.ts 100 100 100 100
/usr/src/app/src/services/applications/web/schemas.ts 100 100 100 100
/usr/src/app/src/services/applications/web/controllers/applications.ts 17.39 0 0 17.39
/usr/src/app/src/services/applications/web/controllers/company-applications.ts 27.27 0 0 27.27
/usr/src/app/src/services/console/client-factory.ts 37.5 0 0 37.5
/usr/src/app/src/services/console/index.ts 92.3 100 75 90.9
/usr/src/app/src/services/console/service.ts 88.23 100 50 88.23
/usr/src/app/src/services/console/clients/internal.ts 16.21 0 0 16.66
/usr/src/app/src/services/console/clients/remote-jwks-verifier.ts 10.81 0 0 10.81
/usr/src/app/src/services/console/clients/remote.ts 9.89 0 0 10
/usr/src/app/src/services/console/web/controller.ts 10.9 1.08 5.88 10.9
/usr/src/app/src/services/console/web/index.ts 100 100 100 100
/usr/src/app/src/services/console/web/routes.ts 86.66 100 33.33 86.66
/usr/src/app/src/services/console/web/schemas.ts 100 100 100 100
/usr/src/app/src/services/documents/const.ts 100 100 100 100
/usr/src/app/src/services/documents/index.ts 93.75 100 80 92.3
/usr/src/app/src/services/documents/types.ts 100 100 100 100
/usr/src/app/src/services/documents/utils.ts 84.09 74.85 90.9 83.51
/usr/src/app/src/services/documents/entities/drive-file.search.ts 100 83.33 100 100
/usr/src/app/src/services/documents/entities/drive-file.ts 100 100 100 100
/usr/src/app/src/services/documents/entities/drive-tdrive-tab.ts 68.75 100 0 100
/usr/src/app/src/services/documents/entities/file-version.ts 63.33 100 0 100
/usr/src/app/src/services/documents/entities/missed-drive-file.ts 58.82 100 0 100
/usr/src/app/src/services/documents/services/access-check.ts 72.27 61.88 75.86 71.77
/usr/src/app/src/services/documents/services/drive-file-dto-builder.ts 100 90.9 100 100
/usr/src/app/src/services/documents/services/index.ts 69.17 54.33 74.24 68.9
/usr/src/app/src/services/documents/services/engine/extract-keywords.ts 100 100 100 100
/usr/src/app/src/services/documents/services/engine/index.ts 96.55 100 100 96.42
/usr/src/app/src/services/documents/services/engine/save-keywords.ts 80.95 83.33 80 80.95
/usr/src/app/src/services/documents/web/index.ts 100 100 100 100
/usr/src/app/src/services/documents/web/routes.ts 100 100 100 100
/usr/src/app/src/services/documents/web/schemas.ts 100 100 100 100
/usr/src/app/src/services/documents/web/controllers/documents.ts 55.85 28.04 59.52 57.37
/usr/src/app/src/services/documents/web/controllers/index.ts 100 100 100 100
/usr/src/app/src/services/files/index.ts 92.3 100 75 90.9
/usr/src/app/src/services/files/entities/file.ts 86.36 100 50 100
/usr/src/app/src/services/files/services/index.ts 53.75 45.88 50 53.25
/usr/src/app/src/services/files/services/preview.ts 81.81 50 80 85
/usr/src/app/src/services/files/web/index.ts 100 100 100 100
/usr/src/app/src/services/files/web/routes.ts 90.47 100 33.33 89.47
/usr/src/app/src/services/files/web/controllers/files.ts 42.1 61.9 33.33 41.81
/usr/src/app/src/services/files/web/controllers/index.ts 100 100 100 100
/usr/src/app/src/services/general/index.ts 92.85 100 75 91.66
/usr/src/app/src/services/general/languages.ts 100 100 100 100
/usr/src/app/src/services/general/web/index.ts 100 100 100 100
/usr/src/app/src/services/general/web/routes.ts 58.82 0 14.28 58.82
/usr/src/app/src/services/i18n/i18n-service.ts 80 100 60 80
/usr/src/app/src/services/i18n/index.ts 100 100 100 100
/usr/src/app/src/services/previews/utils.ts 100 100 100 100
/usr/src/app/src/services/previews/services/files/engine/clear.ts 40 0 25 42.85
/usr/src/app/src/services/previews/services/files/engine/index.ts 100 100 100 100
/usr/src/app/src/services/previews/services/files/engine/service.ts 95.23 66.66 83.33 95.12
/usr/src/app/src/services/previews/services/files/processing/image.ts 86.36 83.33 100 86.36
/usr/src/app/src/services/previews/services/files/processing/pdf.ts 76.19 0 100 75
/usr/src/app/src/services/previews/services/files/processing/service.ts 92.85 100 100 92.85
/usr/src/app/src/services/previews/services/files/processing/video.ts 58.33 12.5 69.23 57.44
/usr/src/app/src/services/statistics/index.ts 83.33 100 66.66 83.33
/usr/src/app/src/services/statistics/types.ts 100 100 100 100
/usr/src/app/src/services/statistics/entities/statistics.ts 91.66 100 0 90
/usr/src/app/src/services/statistics/service/index.ts 47.05 44.44 50 47.05
/usr/src/app/src/services/tags/index.ts 93.33 100 80 91.66
/usr/src/app/src/services/tags/entities/index.ts 100 100 100 100
/usr/src/app/src/services/tags/entities/tags.ts 100 100 100 100
/usr/src/app/src/services/tags/services/tags.ts 91.66 100 100 91.66
/usr/src/app/src/services/tags/web/index.ts 100 100 100 100
/usr/src/app/src/services/tags/web/routes.ts 100 100 100 100
/usr/src/app/src/services/tags/web/controllers/index.ts 100 100 100 100
/usr/src/app/src/services/tags/web/controllers/tags.ts 85.29 83.33 100 85.29
/usr/src/app/src/services/user/index.ts 92.3 100 75 90.9
/usr/src/app/src/services/user/realtime.ts 71.42 100 42.85 71.42
/usr/src/app/src/services/user/utils.ts 100 62.68 100 100
/usr/src/app/src/services/user/entities/company.ts 100 100 100 100
/usr/src/app/src/services/user/entities/company_user.ts 100 100 100 100
/usr/src/app/src/services/user/entities/device.ts 100 100 100 100
/usr/src/app/src/services/user/entities/external_company.ts 100 100 100 100
/usr/src/app/src/services/user/entities/external_user.ts 100 100 100 100
/usr/src/app/src/services/user/entities/user.search.ts 100 88.88 100 100
/usr/src/app/src/services/user/entities/user.ts 88.37 40 75 87.8
/usr/src/app/src/services/user/services/companies.ts 50 27.38 41.66 51.85
/usr/src/app/src/services/user/services/external_links/index.ts 61.11 0 50 61.11
/usr/src/app/src/services/user/services/users/service.ts 51.87 42.62 53.65 54.1
/usr/src/app/src/services/user/web/controller.ts 78.78 65.78 80.76 77.65
/usr/src/app/src/services/user/web/index.ts 100 100 100 100
/usr/src/app/src/services/user/web/routes.ts 95.83 0 100 95.65
/usr/src/app/src/services/user/web/schemas.ts 100 100 100 100
/usr/src/app/src/services/user/web/types.ts 100 100 100 100
/usr/src/app/src/services/workspaces/index.ts 92.3 100 75 90.9
/usr/src/app/src/services/workspaces/realtime.ts 87.5 75 75 87.5
/usr/src/app/src/services/workspaces/entities/workspace.ts 100 100 100 100
/usr/src/app/src/services/workspaces/entities/workspace_counters.ts 100 100 100 100
/usr/src/app/src/services/workspaces/entities/workspace_invite_domain.ts 90.9 100 0 88.88
/usr/src/app/src/services/workspaces/entities/workspace_invite_tokens.ts 92.3 100 0 90.9
/usr/src/app/src/services/workspaces/entities/workspace_pending_users.ts 91.66 100 0 90
/usr/src/app/src/services/workspaces/entities/workspace_user.ts 100 66.66 100 100
/usr/src/app/src/services/workspaces/services/workspace.ts 54.89 37.79 62 55.02
/usr/src/app/src/services/workspaces/web/index.ts 100 100 100 100
/usr/src/app/src/services/workspaces/web/routes.ts 72.82 24.24 64.7 71.91
/usr/src/app/src/services/workspaces/web/schemas.ts 100 100 100 100
/usr/src/app/src/services/workspaces/web/controllers/workspace-invite-tokens.ts 8.33 0 0 8.47
/usr/src/app/src/services/workspaces/web/controllers/workspace-users.ts 55.47 44.64 62.85 57.14
/usr/src/app/src/services/workspaces/web/controllers/workspaces.ts 80.43 76.66 86.95 80.43
/usr/src/app/src/utils/coalesce.ts 25 0 0 25
/usr/src/app/src/utils/company.ts 82.6 75 50 82.6
/usr/src/app/src/utils/counters.ts 100 100 100 100
/usr/src/app/src/utils/files.ts 88.67 100 80 89.47
/usr/src/app/src/utils/handleError.ts 33.33 0 0 33.33
/usr/src/app/src/utils/mime.ts 100 100 100 100
/usr/src/app/src/utils/password-encoder.ts 50 0 50 50
/usr/src/app/src/utils/pick.ts 100 100 100 100
/usr/src/app/src/utils/types.ts 100 100 100 100
/usr/src/app/src/utils/users.ts 85.71 73.33 100 88.46
/usr/src/app/src/utils/uuid-reducer.ts 40 0 0 33.33
/usr/src/app/src/utils/workspace.ts 33.33 14.28 25 33.33
/usr/src/app/test/e2e/utils.api.ts 93.75 100 83.33 93.33
/usr/src/app/test/e2e/utils.prepare.db.ts 78.49 76.92 65.21 79.77
/usr/src/app/test/e2e/common/user-api.ts 93.63 64.51 90.9 94.39
/usr/src/app/test/e2e/common/entities/mock_entities.ts 100 100 100 100
/usr/src/app/test/e2e/documents/utils.ts 100 100 100 100
/usr/src/app/test/e2e/setup/index.ts 88.88 50 100 88.88
/usr/src/app/test/unit/core/services/database/services/orm/connectors/postgres/utils.ts 79.16 100 37.5 100
Total 67.15%
Change from base: 0%
Covered Lines: 4880
Total Lines: 7267

ericlinagora
ericlinagora reviewed Jul 19, 2024
View reviewed changes
tdrive/backend/node/src/cli/cmds/db.ts
visit: commandModule => commandModule.default,
}),
// eslint-disable-next-line @typescript-eslint/no-empty-function
handler: () => {},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the plan here ?

ericlinagora
ericlinagora reviewed Jul 19, 2024
View reviewed changes
tdrive/backend/node/src/cli/cmds/db_cmds/migrate-db.ts
name: {
default: "",
type: "string",
description: "Entity name to migrate",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think migrations should be per-entity, they can mutate together inter-dependently, I propose we remove the option all together. If we want a limit in migrations it should be per version thereof

ericlinagora
ericlinagora reviewed Jul 19, 2024
View reviewed changes
tdrive/backend/node/src/cli/cmds/db_cmds/migrate-db.ts
decimal: "numeric", // Assuming decimal numbers are stored as numeric in PostgreSQL
encoded_json: "text", // Assuming encoded JSON objects are stored as text in PostgreSQL
};
console.log("\n ##################################### \n");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this feels like something that would be in the pgsql adapter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericlinagora ericlinagora ericlinagora left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@MontaGhanmy MontaGhanmy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MontaGhanmy @ericlinagora