From 30c19c49cd8c35ce67f367265b86ee2e06cdbb66 Mon Sep 17 00:00:00 2001 From: John Yaist Date: Thu, 3 Oct 2024 15:57:02 -0700 Subject: [PATCH] add sample for adding object store --- .../add_object_store_to_server.ipynb | 513 ++++++++++++++++++ 1 file changed, 513 insertions(+) create mode 100644 samples/03_org_administrators/add_object_store_to_server.ipynb diff --git a/samples/03_org_administrators/add_object_store_to_server.ipynb b/samples/03_org_administrators/add_object_store_to_server.ipynb new file mode 100644 index 000000000..0ae5f1be2 --- /dev/null +++ b/samples/03_org_administrators/add_object_store_to_server.ipynb @@ -0,0 +1,513 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "118e599d-535e-44b7-9eef-cfe465bfa054", + "metadata": {}, + "source": [ + "# Add an Object Store to a Web GIS Deployment" + ] + }, + { + "cell_type": "markdown", + "id": "20849eda-da40-4106-be2b-5fa07cf92cba", + "metadata": {}, + "source": [ + "## Introduction" + ] + }, + { + "cell_type": "markdown", + "id": "229a1894-715c-44f7-918d-81f3cbc203a3", + "metadata": {}, + "source": [ + "With the October 2024 ArcGIS Enterprise 11.4 release, the [tile cache data store](https://enterprise.arcgis.com/en/portal/latest/administer/windows/arcgis-data-store-terms.htm#ESRI_SECTION1_307B3EB511274C39B5BF0135E0EEB682) is deprecated. In a future release, it will no longer be supported, which means to [publish hosted scene layers](https://enterprise.arcgis.com/en/portal/latest/use/publish-scenes.htm) or [hosted 3D tiles layers](https://enterprise.arcgis.com/en/portal/latest/use/publish-3d-tiles.htm) in an 11.4 ArcGIS Enterprise deployment, you must configure an object store. This sample notebook demonstrates a workflow for adding an object store with the ArcGIS API for Python. The code will work in a Python script as well. See the [What's new in ArcGIS Data Store 11.4](https://enterprise.arcgis.com/en/portal/latest/administer/windows/whats-new-data-store.htm#ESRI_SECTION1_239941E4BC1D4DA1ACC0B9F8DAA18BA2) for full details on Data Store." + ] + }, + { + "cell_type": "markdown", + "id": "a640f425-1c53-4998-adbe-515d1bd61ccb", + "metadata": {}, + "source": [ + "## Import Libraries" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "7619fd92-7c53-463f-8a50-5cebba30a8fe", + "metadata": {}, + "outputs": [], + "source": [ + "from arcgis.gis import GIS" + ] + }, + { + "cell_type": "markdown", + "id": "5b99c46f-871d-4687-89df-b03757239496", + "metadata": {}, + "source": [ + "## Connect to your ArcGIS Enterprise Deployment" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "63113763-87fa-4a79-aab6-ce407e2a6e1c", + "metadata": {}, + "outputs": [], + "source": [ + "gis = GIS(profile=\"your_enterprise_admin_profile\")" + ] + }, + { + "cell_type": "markdown", + "id": "7c00963b-91e2-46a0-b8e5-5455df91cf80", + "metadata": {}, + "source": [ + "> **Note:** See [Storing your credentials locally](/python/latest/guide/working-with-different-authentication-schemes/#storing-your-credentials-locally) for instructions on creating a _profile_." + ] + }, + { + "cell_type": "markdown", + "id": "d3c27b99-8579-417a-bca2-575f06271943", + "metadata": {}, + "source": [ + "## List the Federated Servers\n", + "\n", + "When connecting as an admin, the *admin* property returns access to the [PortalAdminManager](python/latest/api-reference/arcgis.gis.admin.html#portaladminmanager), and the [*servers*](/python/latest/api-reference/arcgis.gis.admin.html#arcgis.gis.admin.PortalAdminManager.servers) property on that returns a [ServerManager](/python/latest/api-reference/arcgis.gis.server.html#arcgis.gis.server.ServerManager) object to administer the Enterprise's servers:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "cb1b6cf0-6672-422e-b013-53d97504f0e4", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "< ServerManager @ https://example.site.com/portal/portaladmin >" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "server_mgr = gis.admin.servers\n", + "server_mgr" + ] + }, + { + "cell_type": "markdown", + "id": "ac4bcb8d-55bc-4cf8-a616-4bb8cf22b712", + "metadata": {}, + "source": [ + "Use the [*list*](/python/latest/api-reference/arcgis.gis.server.html#arcgis.gis.server.ServerManager.list) method to return all the servers federated with this GIS:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "31361c20-fbbb-4753-9a1f-49eb489b628f", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[< Server @ https://example.site.com:6443/arcgis/admin >]" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "servers = server_mgr.list()\n", + "servers" + ] + }, + { + "cell_type": "markdown", + "id": "f4cd7aa5-ae6c-4666-89fc-9d3ee56d1ab5", + "metadata": {}, + "source": [ + "### Retrieve specific Server\n", + "\n", + "Use list indexing to return the specific [server](/python/latest/api-reference/arcgis.gis.server.html#arcgis.gis.server.Server) object to which you'll add the data store:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "22317968-b16c-445b-a969-ad6505ffaebf", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "arcgis.gis.server.admin.administration.Server" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "gis_server = servers[0]\n", + "type(gis_server)" + ] + }, + { + "cell_type": "markdown", + "id": "e6905c71-f0d6-406a-8448-13ef3359b83f", + "metadata": {}, + "source": [ + "## Get the Server's Data Store Manager\n", + "With the *server* object, the [*datastores*](/python/latest/api-reference/arcgis.gis.server.html#arcgis.gis.server.Server.datastores) property returns a [DataStoreManager](/python/latest/api-reference/arcgis.gis.server.html#arcgis.gis.server.DataStoreManager) object that provides access to adminstrative capabilities for data holdings of a server. These items retain important information pertinent to publishing in a Web GIS deployment. See the [Data](/rest/enterprise-administration/server/data/) documentation for full details.\n", + "\n", + "Let's demonstrate using the *DataStoreManager* to add an object store to our GIS. First we'll get the *DataStoreManager*, then construct an object representing the object store, then pass that object to the [add()](/python/latest/api-reference/arcgis.gis.server.html#arcgis.gis.server.DataStoreManager.add) method to regtister the object store with the server." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "5f0c5b3b-f84d-4032-aa39-5ed80795b79f", + "metadata": {}, + "outputs": [], + "source": [ + "datastore_mgr = gis_server.datastores" + ] + }, + { + "cell_type": "markdown", + "id": "0aa65762-aaf0-4b01-8f6f-b9b60339c679", + "metadata": {}, + "source": [ + "### Construct the Object Store Connection object" + ] + }, + { + "cell_type": "markdown", + "id": "cada8b53-46ee-42b9-aa29-41a3a0f3a229", + "metadata": {}, + "source": [ + "We'll use a Python dictionary to create an object that holds all the necessary information about the object store to add it to the server. See [Object store](https://developers.arcgis.com/rest/enterprise-administration/server/registerdataitem/#object-store) and [Object store examples](https://developers.arcgis.com/rest/enterprise-administration/server/registerdataitem/#object-store-examples) for specific details and requirements for creating the object." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "89e20714-e814-4eb2-8dcb-8c2f4535829d", + "metadata": {}, + "outputs": [], + "source": [ + "object_str_config: dict = {\n", + " \"path\": \"/cloudStores/api_object_store\",\n", + " \"type\": \"objectStore\",\n", + " \"provider\": \"amazon\",\n", + " \"info\": {\n", + " \"isManaged\": True,\n", + " \"systemManaged\": False,\n", + " \"isManagedData\": True,\n", + " \"purposes\": [\"feature-tile\", \"scene\"],\n", + " \"connectionString\": \"{\\\"accessKeyId\\\":\\\"\\\",\\\"secretAccessKey\\\":\\\"\\\",\\\"regionEndpointUrl\\\":\\\"https://s3.us-west-2.amazonaws.com\\\",\\\"defaultEndpointsProtocol\\\":\\\"https\\\",\\\"credentialType\\\":\\\"accesskey\\\",\\\"region\\\":\\\"us-west-2\\\"}\",\n", + " \"objectStore\": \"s3bucket/1091cache\"\n", + " }\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "d42c04b3-3583-4c9a-9a2c-400249a142e5", + "metadata": {}, + "source": [ + "### Add the Object Store to the server" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "f36371cc-f4bd-448d-a071-1b7bd450e44c", + "metadata": {}, + "outputs": [], + "source": [ + "object_data_store = datastore_mgr.add(\n", + " item=object_str_config\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "08589fb4-e585-47ed-be8e-b49fad912dbb", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "object_data_store" + ] + }, + { + "cell_type": "markdown", + "id": "c7362371-c021-46fa-990b-a3b00ba1425e", + "metadata": {}, + "source": [ + "## Validate the Data Store" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "6e327b77-bf84-436a-bd7f-e9affde88b56", + "metadata": {}, + "outputs": [ + { + "ename": "KeyError", + "evalue": "'status'", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[21], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m object_data_store\u001b[38;5;241m.\u001b[39mvalidate()\n", + "File \u001b[0;32m~/opt/anaconda3/envs/geosaurus_dev_env/lib/python3.11/site-packages/arcgis/gis/server/admin/_data.py:255\u001b[0m, in \u001b[0;36mDatastore.validate\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 252\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 253\u001b[0m res \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_con\u001b[38;5;241m.\u001b[39mpost(path, params, verify_cert\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m)\n\u001b[0;32m--> 255\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m res[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mstatus\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124msuccess\u001b[39m\u001b[38;5;124m\"\u001b[39m\n", + "\u001b[0;31mKeyError\u001b[0m: 'status'" + ] + } + ], + "source": [ + "object_data_store.validate()" + ] + }, + { + "cell_type": "markdown", + "id": "e9005794-699a-43d1-a6c4-6bb104971b29", + "metadata": {}, + "source": [ + "## Updating the Configuration" + ] + }, + { + "cell_type": "markdown", + "id": "22d0cb37-7ab5-4f4b-8763-935a5073cf56", + "metadata": {}, + "source": [ + "The validation failed. There may be something in the object store construction that is causing the failure. Let's update the data store configuration." + ] + }, + { + "cell_type": "markdown", + "id": "e1f6af8f-d982-4420-8e44-04cdac439ff3", + "metadata": {}, + "source": [ + "### Get the Configuration\n", + "The *properties* property on the object store returns the configuration." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "12909869-9c0e-4d81-9c2d-4be422cdf142", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{\n", + " \"path\": \"/cloudStores/api_object_store\",\n", + " \"type\": \"objectStore\",\n", + " \"id\": \"2eaad7a5-3be6-47ef-bd78-ae0e26788d53\",\n", + " \"systemManaged\": false,\n", + " \"provider\": \"amazon\",\n", + " \"info\": {\n", + " \"isManaged\": true,\n", + " \"systemManaged\": false,\n", + " \"isManagedData\": true,\n", + " \"purposes\": [\n", + " \"feature-tile\",\n", + " \"scene\"\n", + " ],\n", + " \"connectionString\": \"{crypt}B3R2/3FKfR45nMTX/Xcoj3AE5V9N2JXtYKe4DyxN0GhZOjYr6sNUWh2JBOOXh2eayeidble/AlEVlJcUhZRNGubB9VqoE4av8/BidobKtO11Fxtzt2UJqn1Q7v7uRJbHRB7drioDhNfYil0KCrNCqbtNFR4awURk8gX2tEy4GYDqXnxKkpwCi/qDr/8Xm8azV0K2NnQrlFcVsw0BT0O+2u/w8Q7GM5ZZjKx1r/rUEv1McgeLydaVveIw8WLxd+n/eboX/6Pab3Bj4HAmYba5N14/mJoerQSHmNFwFdkNCv4Yy2u/QHJQMzAhXJPazDSN4WJyzirWd9k=\",\n", + " \"objectStore\": \"s3bucket/1091cache\"\n", + " }\n", + "}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "obj_store_config = object_data_store.properties\n", + "obj_store_config" + ] + }, + { + "cell_type": "markdown", + "id": "8f56f685-aa08-404c-bd8a-2c0e7183a2bf", + "metadata": {}, + "source": [ + "### Update the configuration information\n", + "\n", + "First, update the properties of the dictionary that represents the object data store connection information." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "3942a8e2-d827-47dd-8d0f-95d0888e8057", + "metadata": {}, + "outputs": [], + "source": [ + "obj_store_config.update({\n", + " \"path\": \"/cloudStores/api_object_store\",\n", + " \"type\": \"objectStore\",\n", + " \"provider\": \"amazon\",\n", + " \"info\": {\n", + " \"isManaged\": True,\n", + " \"systemManaged\": False,\n", + " \"isManagedData\": True,\n", + " \"purposes\": [\"feature-tile\", \"scene\"],\n", + " \"connectionString\": \"{\\\"accessKeyId\\\":\\\"\\\",\\\"secretAccessKey\\\":\\\"\\\",\\\"regionEndpointUrl\\\":\\\"https://s3.us-west-2.amazonaws.com\\\",\\\"defaultEndpointsProtocol\\\":\\\"https\\\",\\\"credentialType\\\":\\\"accesskey\\\",\\\"region\\\":\\\"us-west-2\\\"}\",\n", + " \"objectStore\": \"1091cache\",\n", + " }\n", + "})" + ] + }, + { + "cell_type": "markdown", + "id": "dd1ccc0b-ac43-4bd7-a44f-78ae475a1b0a", + "metadata": {}, + "source": [ + "### Update the data store configuration on the server\n", + "Use the [*update()*](/python/latest/api-reference/arcgis.gis.server.html#arcgis.gis.server.Datastore.update) method to edit the data store connection information." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "9f3dd567-7d44-49a6-858b-04bda6237983", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "object_data_store.update(obj_store_config)" + ] + }, + { + "cell_type": "markdown", + "id": "6b4bbb0d-8e97-4825-8f07-b6a9f107d3c5", + "metadata": {}, + "source": [ + "### Validate the updated Data Store\n", + "\n", + "Run the validation again, and then confirm the properties are valid." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "efc56afc-0d50-439a-afa8-d9cdc6abc7f9", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 31, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "object_data_store.validate()" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "90af5b7f-d082-49df-9215-ef0ed610fe35", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{\n", + " \"path\": \"/cloudStores/api_object_store\",\n", + " \"type\": \"objectStore\",\n", + " \"id\": \"2eaad7a5-3be6-47ef-bd78-ae0e26788d53\",\n", + " \"systemManaged\": false,\n", + " \"provider\": \"amazon\",\n", + " \"info\": {\n", + " \"isManaged\": true,\n", + " \"systemManaged\": false,\n", + " \"isManagedData\": true,\n", + " \"purposes\": [\n", + " \"feature-tile\",\n", + " \"scene\"\n", + " ],\n", + " \"connectionString\": \"{\\\"accessKeyId\\\":\\\"\\\",\\\"secretAccessKey\\\":\\\"\\\",\\\"regionEndpointUrl\\\":\\\"https://s3.us-west-2.amazonaws.com\\\",\\\"defaultEndpointsProtocol\\\":\\\"https\\\",\\\"credentialType\\\":\\\"accesskey\\\",\\\"region\\\":\\\"us-west-2\\\"}\",\n", + " \"objectStore\": \"1091cache\"\n", + " }\n", + "}" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "object_data_store.properties" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}