diff --git a/heatwave-lakehouse-cdp/configuring-fn/configuring-fn.md b/heatwave-lakehouse-cdp/configuring-fn/configuring-fn.md new file mode 100644 index 00000000..e7a42684 --- /dev/null +++ b/heatwave-lakehouse-cdp/configuring-fn/configuring-fn.md @@ -0,0 +1,187 @@ +# Configuring and using OCI Functions to consume streamming data from OCI Streaming Topic + +### Introduction + +This lab focus on creating the Function ,deploying the function script and getting it connected to OCI service connector hub to get automatically triggered once the streaming data falls on the OCI streaming topic.Here the function is used to perform the below tasks : + +- Connect to OCI streaming topic to receive the data streams. + +- Pass the data stream to ML model API and write the streaming data along with prediction score to + Object Storage and MySQL table. + +_Estimated Lab Time_: 75 minutes + +### Objectives + +- Create a Function Application +- Configure and Invoke a Function +- Create Service Connector + +### Prerequisites + +- Completion of the previous labs. + +## +## Task 1: Create a Function Application + +1. From the OCI services menu click **Policies** under **Identity** , Create "FunctionApplicationPolicies" into the name section and type "Allow functions to work" in the description section. Scroll down to the Policy statements section. Click the **+ Another Statement**. + + ![function policies](images/func-policy.png) + + Copy and paste the following as policy statement. ``` + + allow service FAAS to read repos in tenancy + allow service datascience to use virtual-network-family in compartment id ocid1.compartment.oc1..xyz + allow dynamic-group datascience to manage data-science-family in compartment id ocid1.compartment.oc1..xyz + allow dynamic-group datascience to manage object-family in compartment id ocid1.compartment.oc1..xyz + Allow group Administrators to manage functions-family in compartment id ocid1.compartment.oc1..xyz + Allow group Administrators to manage stream-family in compartment id ocid1.compartment.oc1..xyz + Allow group Administrators to manage stream-pull in compartment id ocid1.compartment.oc1..xyz + Allow group Administrators to manage streams in compartment id ocid1.compartment.oc1..xyz + Allow group Administrators to manage objects in compartment id ocid1.compartment.oc1..xyz + ``` + +2. From the OCI services menu click **Functions** under **Developer Services** to create the Fn. + + ![fuction developer check](images/func-navigate.png) + +3. Click **Create Application** and fill out the dialog box. + + - **NAME**: streamhwapp + - **VCN in** : Choose the compartment where your VCN was created + - **SUBNETS** : Choose your VCN's public subnet + - **SHAPE** : Choose your VCN's public subnet + - **LOGGING POLICY**: GENERIC_X86 + + Click **Create**. + + ![create function](images/func-creating.png) + +4. Scroll down and click on **Getting Started** and follow the steps on the OCI console. + + ![create function](images/func-create.png) + +## Task 2: Configure and Invoke a Function + +As a continuation of task 1 , folllow the steps shown in the OCI console to configure the Function. +Here we will be performing the below steps to Launch Cloud Console ,Create, deploy, and invoke your python function. + +1. Follow steps No. 1 to No. 7 given on the OCI console to set up OCI cloud shell and listing the Applications. + + ![Getting Started ](images/func-cloudshell.png) + +2. Create, deploy, and invoke your function + Create the Function : (Step No 8 on OCI console) ``` + + fn init --runtime python streamhwfunc + ``` + +3. Switch into the generated directory (Step No 9 on OCI console) ``` + + cd streamhwfunc + ``` + +4. Now change the content of **func.py** and **requirements.txt** file. + + Function Script for function : + Scripts to be changed and moved to the OCI cloud console - func.py ,func.yaml,requirements.txt + Link to download the Function script and Dependencies : [`MYSQLLakehouse_labfiles.zip`](https://objectstorage.us-ashburn-1.oraclecloud.com/p/RPka_orWclfWJmKN3gTHfEiv-uPckBJTZ3FV0sESZ3mm3PDCQcVDCT-uM2dsJNGf/n/orasenatdctocloudcorp01/b/MYSQLLakehouse\_labfiles/o/MYSQLLakehouse_labfiles.zip) + + ### Download Referece Path: MYSQLLakehouse_labfiles\Lab5\OCI Function\func.py + + [Refer to downloadbled MYSQLLakehouse_labfiles.zip] + Changes in the func.py script is as below: + ``` + + Line No -33 - In the endpoint replace the ML model deployment URL. + endpoint = "http://IP:5000/predict" # change here for the lab with your compute ipfor Lab + + ``` + + ``` + + Line No -51 & 52 - Replace these lines with the OCI Streaming Enpoint URL and stream OCID. + ociMessageEndpoint = https://cell-1.streaming.us-.com # change here for Lab + ociStreamOcid = "ocid1.stream.oc1.phx.xyz" # change here for Lab + + ``` + + ``` + + Line No -65 & 66 - Replace these lines with the oci Objectstore details for the demo. + namespace_name="" # change here for Lab + bucket_name="e2e-demo-events-raw-mysqlhw" # change here for Lab + + ``` + ``` + + Line No -76,77,78,79 - Replace these lines login credentials of MYSQL HW DB + DB_HOST = '1x.x.x.xx' # change here for Lab + DB_USER = 'admin' + DB_PASSWORD = 'xxxxx' # change here for Lab + DB_NAME = 'MFG_SENSOR' + + ``` + + ``` + + fn update context registry phx.ocir.io//streamhwrepo + + ``` + +5. Change the content of the requirement.txt file. The sample requirement.txt can be downloaded from the demo co-laterals as well. + + ``` + + fdk>=0.1.75 + requests + pymysql + oci + + ``` +6. Upload these files to the function directory and replace the existing func.py and requirement.txt files and move them to the funtion directory. + ![Set up ](images/function_file_upload.png) + Execute the below commands to deploy and invoke the Function. + + ``` + + fn -v deploy --app streamhwapp + fn invoke streamhwapp streamhwfunc + + ``` +Congratulations! You've just created, deployed, and invoked your first function using Oracle Functions! + +### connect to service connector Hub and trigger the Fn. + +## Task 3: Create Service Connector + +For simple archiving operations, we don’t need to write a single line of code. Instead, we just create a service connector and point it at the source (stream) and destination (bucket). Navigate to the Service Connector Hub via the burger menu (or by searching for it). + +1. Open the navigation menu and click ***Analytics & AI***. Under ***Messaging***, click ***Service Connector Hub***. + +2. You can choose a compartment you have permission to work in (on the left side of the page). The page updates to display only the resources in that compartment. For this lab, we will work in the tenancy root compartment. + +3. Click ***Create Service Connector***. + +4. Name the connector ***streaming-to-Fn*** . + +5. Provide the description ***Moving streaming data to object storage using Fn***. + +6. For the compartment, choose the demo compartment. + +![Service Connector](./images/oci-service-connector.png) + +7. Now to configure your service connector, Choose ’Streaming’ as the ***source***, and ‘function’ as the ***target***. Chose the compartment where the stream pool resides, choose the stream pool, and the stream. You can choose to read from either the ‘Latest’ offset or ’Trim Horizon’ (the oldest non-committed offset). + +![Service Connector](./images/func-eventhub.png) + +![Service Connector](./images/func-eventhub2.png) +8. Click ***Create*** and the service connector is ready to collect the streams. + +You may now **proceed to the next lab** + +## Acknowledgements + +* **Author** - Biswanath Nanda - Master Principal Cloud Architect +* **Contributors** - Lovelesh Saxena - Principal Cloud Architect, Biswanath Nanda -Master Principal Cloud Architect, Bhushan Arora - Principal Cloud Architect +* **Last Updated By/Date** - Biswanath Nanda, November 2024 diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/allow_dg_to_manage.png b/heatwave-lakehouse-cdp/configuring-fn/images/allow_dg_to_manage.png new file mode 100644 index 00000000..c455b80a Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/allow_dg_to_manage.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/desktop.ini b/heatwave-lakehouse-cdp/configuring-fn/images/desktop.ini new file mode 100644 index 00000000..e4df371a --- /dev/null +++ b/heatwave-lakehouse-cdp/configuring-fn/images/desktop.ini @@ -0,0 +1,3 @@ +[LocalizedFileNames] +function_file_upload.png=@function_file_upload.png,0 +oci-service-connector.png=@oci-service-connector.png,0 diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/func-cloudshell.png b/heatwave-lakehouse-cdp/configuring-fn/images/func-cloudshell.png new file mode 100644 index 00000000..3356c0d7 Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/func-cloudshell.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/func-create.png b/heatwave-lakehouse-cdp/configuring-fn/images/func-create.png new file mode 100644 index 00000000..3bbf1566 Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/func-create.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/func-creating.png b/heatwave-lakehouse-cdp/configuring-fn/images/func-creating.png new file mode 100644 index 00000000..e0e5b13e Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/func-creating.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/func-eventhub.png b/heatwave-lakehouse-cdp/configuring-fn/images/func-eventhub.png new file mode 100644 index 00000000..1d724301 Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/func-eventhub.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/func-eventhub2.png b/heatwave-lakehouse-cdp/configuring-fn/images/func-eventhub2.png new file mode 100644 index 00000000..bfdb4210 Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/func-eventhub2.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/func-navigate.png b/heatwave-lakehouse-cdp/configuring-fn/images/func-navigate.png new file mode 100644 index 00000000..e6c19cef Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/func-navigate.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/func-policy.png b/heatwave-lakehouse-cdp/configuring-fn/images/func-policy.png new file mode 100644 index 00000000..014d7691 Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/func-policy.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/function_file_upload.png b/heatwave-lakehouse-cdp/configuring-fn/images/function_file_upload.png new file mode 100644 index 00000000..ac13b120 Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/function_file_upload.png differ diff --git a/heatwave-lakehouse-cdp/configuring-fn/images/oci-service-connector.png b/heatwave-lakehouse-cdp/configuring-fn/images/oci-service-connector.png new file mode 100644 index 00000000..e2d166ec Binary files /dev/null and b/heatwave-lakehouse-cdp/configuring-fn/images/oci-service-connector.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/create-compute-with-mysqlsh.md b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/create-compute-with-mysqlsh.md new file mode 100644 index 00000000..c86ad948 --- /dev/null +++ b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/create-compute-with-mysqlsh.md @@ -0,0 +1,208 @@ +# Create Compute and MySQL Shell + +### Introduction + +In this lab we will be creating a compute instance which will solve the below purpose for the workshop : + +1. Hosting IoT Simulator Code: The compute instance will serve as a virtual machine to host the custom Python generator codebase, which will be used to create IoT datasets for OCI streaming. This involves setting up the simulator code framework on the instance. + +2. The instance will act as a Bastion Host, a resource that sits between the private resource and the endpoint, providing access to the private network. It will function as a "jump box," allowing you to log in to the private resource through protocols like SSH. Additionally, we will use this compute instance for configuring the MySQL Shell client. This Bastion Host will be set up within a Virtual Cloud Network and will include a compute instance that can connect with the MySQL DB Systems. + +3. This compute instance is also leveraged for building and deploying custom Machine Learning model which identifies the probablity of device failures . + +Note - MySQL HeatWave on OCI is an example of a service that is only accessible through private networks. Since the service is fully managed, we keep it siloed away from the internet to help protect your data from potential attacks and vulnerabilities. It’s a good practice to limit resource exposure as much as possible, but at some point, you’ll likely want to connect to those resources. That’s where Compute Instance, also known as a Bastion host, enters the picture. + +_Estimated Lab Time:_ 45 minutes + +### Objectives + +In this lab, you will be guided through the following tasks: + +- Create SSH Key on OCI Cloud +- Copy public SSH key value to Notepad +- Create Compute Instance +- Connect to Compute and Install MySQl Shell + +### Prerequisites + +- An Oracle Trial or Paid Cloud Account + +## +## Task 1: Create SSH Key on OCI Cloud Shell + +The Cloud Shell machine is a small virtual machine running a Bash shell which you access through the Oracle Cloud Console (Homepage). You will start the Cloud Shell and generate a SSH Key to use for the Bastion session. + +1. To start the Oracle Cloud shell, go to your Cloud console and click the cloud shell icon at the top right of the page. This will open the Cloud Shell in the browser, the first time it takes some time to generate it. + + ![Open Cloudshell](./images/cloudshell-open.png "cloudshellopen ") + + ![Cloudshell Message](./images/cloudshell-welcome.png "cloudshell welcome ") + + *Note: You can use the icons in the upper right corner of the Cloud Shell window to minimize, maximize, restart, and close your Cloud Shell session. + +2. Once the cloud shell has started, create the SSH Key using the following command: + + ```bash + ssh-keygen -t rsa + ``` + + Press just enter for each question, here is what it should look like: + + ![Generate SSH Key](./images/ssh-keygen.png "ssh keygen ") + +3. The SSH keys are stored as follows: + - public SSH key stored in ~/.ssh/id_rsa.pub + - private SSH keys stored in ~/.ssh/id_rsa + +4. Examine the two files that you just created. + + ```bash + cd .ssh + ``` + + ```bash + ls + ``` + + ![SSH files list](./images/ssh-list.png "ssh list ") + + **Note** in the output there are two files, a *private key:`id_rsa` and a public key: `id_rsa.pub`. Keep the private key safe and don't share its content with anyone. The public key will be needed for various activities and can be uploaded to certain systems as well as copied and pasted to facilitate secure communications in the cloud. + +5. To asign the right permissions to your SSH keys, run the following command: + + ```bash + chmod 600 id_rsa.pub + ``` + + ```bash + chmod 600 id_rsa + ``` + +## Task 2: Copy public SSH key value to Notepad + +You will need a compute Instance to connect to your brand new MySQL database. + +1. Before creating the Compute instance open a notepad. + +2. Do the followings steps to copy the public SSH key to the notepad. + + Enter the following command + + ```bash + cat ~/.ssh/id_rsa.pub + ``` + + ![Display SSH Key](./images/cloudshell-cat.png "cloudshell cat") + +3. Copy the above output to a notepad. + + Your notepad should look like this + ![Completed SSH key](./images/notepad-rsa-key.png "notepad rsa key ") + +## Task 3: Create Compute instance + + +1. To launch a Linux Compute instance, go to + Navigation Menu + Compute + Instances + ![Compute Template](./images/compute-launch.png "compute launch ") + +2. On Instances in **e2e-demo-specialist-eng** Compartment, click **Create Instance**. + ![Create Compute button](./images/compute-create.png "compute create") + +3. On Create Compute Instance. + + Enter Name + + ```bash + heatwave-client + ``` + +4. Make sure **e2e-demo-specialist-eng** compartment is selected. + +5. On Placement, keep the selected Availability Domain. + +6. Keep the selected Image, Oracle Linux 8 + + ![Use Linux OS](./images/compute-oracle-linux.png "compute oracle linux") + +7. Change the Instance Shape: + - Click **Change shape** button as in above screen shot + - Click **Virtual Machine** box + - Click **Specialty and previous generation** box + + + ![New Shape](./images/compute-shape-select.png "compute shape select") + + - Click **VM.Standard.E2.2** + - Click the **Select Shape** button + + ![Final Shape](./images/final-compute-shape-select.png "final shape select") + +8. On Networking, make sure '**vcn_phoenix**' is selected and Select the **subnet** + + 'Automatically assign public IPv4 address' checkbox should be enabled under 'Public IPv4 address' as in following: + + ![Select compuete VCN](./images/compute-vcn-public-ip.png "compute vcn") + +9. On Add SSH keys, paste the public key from the notepad. + + ![Add SSH key](./images/compute-id-ssh-paste.png "compute id rsa paste") + +10. Click '**Create**' to finish creating your Compute Instance. + +11. The New Virtual Machine will be ready to use after a few minutes. The state 'Running' indicates that the Virtual Machine is ready to use. + + ![Completed Compute](./images/compute-running-status.png "compute running") + +## Task 4: Connect to Compute and Install MySQL Shell + +1. Copy the public IP address of the active Compute Instance to your notepad + + - Go to Navigation Menu + Compute + Instances + ![Navigate to Compute ](./images/compute-list-e2e.png "compute list") + + - Click the `heatwave-cient` Instance link + + - Copy the `Public IP Address` to the notepad + + ![Compute Instance List](./images/compute-running-copy-e2e.png "compute public ip") + +2. Now go to your OCI Console Cloud Shell, which we used earlier in the lab + +3. Now from OCI Cloud Shell, ssh to **heatwave-client** compute instance + + Enter the username **opc** and the Public **IP Address**. + + + ```bash + ssh -i ~/.ssh/id_rsa opc@ + ``` + + ![SSH Connected](./images/connect-ssh-bastion.png "connect signin") + +4. Install MySQL Shell on the Compute Instance + + You will need a MySQL client tool to connect to your new MySQL DB System from your client machine. + + Install MySQL Shell with the following command (enter y for each question) + + **[opc@…]$** + + ```bash + sudo yum install mysql-shell -y + ``` + + ![MySQL Shell Install](./images/mysql-shell-install.png "connect shell") + + +You may now **proceed to the next lab** + + +## Acknowledgements +* **Author** - Bhushan Arora, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Bhushan Arora ,Principal Cloud Architect, Biswanath Nanda, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Bhushan Arora, March 2024 diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-cat.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-cat.png new file mode 100644 index 00000000..98775e0d Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-cat.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-open.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-open.png new file mode 100644 index 00000000..e00f0c4c Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-open.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-welcome.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-welcome.png new file mode 100644 index 00000000..ec20970e Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/cloudshell-welcome.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-create.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-create.png new file mode 100644 index 00000000..e23fec3b Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-create.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-id-ssh-paste.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-id-ssh-paste.png new file mode 100644 index 00000000..f6820c15 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-id-ssh-paste.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-launch.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-launch.png new file mode 100644 index 00000000..30177f2d Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-launch.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-list-e2e.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-list-e2e.png new file mode 100644 index 00000000..55e628aa Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-list-e2e.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-oracle-linux.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-oracle-linux.png new file mode 100644 index 00000000..5d5e6938 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-oracle-linux.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-running-copy-e2e.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-running-copy-e2e.png new file mode 100644 index 00000000..4aafd667 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-running-copy-e2e.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-running-status.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-running-status.png new file mode 100644 index 00000000..749856bb Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-running-status.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-shape-select.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-shape-select.png new file mode 100644 index 00000000..05669a47 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-shape-select.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-vcn-public-ip.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-vcn-public-ip.png new file mode 100644 index 00000000..109daa50 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/compute-vcn-public-ip.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/connect-ssh-bastion.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/connect-ssh-bastion.png new file mode 100644 index 00000000..0f030ee3 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/connect-ssh-bastion.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/final-compute-shape-select.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/final-compute-shape-select.png new file mode 100644 index 00000000..c86035b3 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/final-compute-shape-select.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/mysql-shell-install.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/mysql-shell-install.png new file mode 100644 index 00000000..09207a07 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/mysql-shell-install.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/notepad-rsa-key.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/notepad-rsa-key.png new file mode 100644 index 00000000..b01a3bf0 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/notepad-rsa-key.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/ssh-keygen.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/ssh-keygen.png new file mode 100644 index 00000000..9f25f437 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/ssh-keygen.png differ diff --git a/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/ssh-list.png b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/ssh-list.png new file mode 100644 index 00000000..4c889012 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-compute-with-mysqlsh/images/ssh-list.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/create-heatwave-vcn-db.md b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/create-heatwave-vcn-db.md new file mode 100644 index 00000000..d1a13958 --- /dev/null +++ b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/create-heatwave-vcn-db.md @@ -0,0 +1,160 @@ +# Create VCN, Subnet and Allow Incoming Connections for the Workshop. + +### Introduction + +In this lab, you will create a Compartment and Virtual Cloud Network (VCN) to connect your OCI resources. + +_Estimated Lab Time_: 15 minutes + +### Objectives + +In this lab, you will be guided through the following tasks: + +- Create a Virtual Cloud Network +- Configure a security list to allow MySQL incoming connections +- Configure security list to allow HTTP incoming connections + +### Prerequisites + +- An Oracle Trial or Paid Cloud Account +- Some Experience with MySQL Shell +- Login to OCI to land on OCI Dashboard + +![INTRO](./images/oci-dashboard.png "land on oci dashboard") + +## +## Task 1: Create Virtual Cloud Network + +1. Click Navigation Menu + Select Networking + Select Virtual Cloud Networks + ![VCN Menu](./images/menuvcn.png "show vcn menu") + +2. Click **Start VCN Wizard** + ![VCN Wizard](./images/networking-main.png "show networking main dialog") + +3. Select 'Create VCN with Internet Connectivity' + + Click 'Start VCN Wizard' + ![Use VCN Wizard](./images/vcn-wizard-start.png "start vcn wizard") + +4. Create a VCN with Internet Connectivity + + On Basic Information, complete the following fields: + + VCN Name: + + ```bash + vcn_phoenix + ``` + + Compartment: Please Select - **e2e-demo-specialist-eng** + + The VCN name can be customized but ensure you use the same VCN through out the demo where ever required. + Your screen should look similar to the following + ![Configure VCN](./images/vcn-internet-connect-config.png "Configured VCN internet connection ") + +5. Click 'Next' at the bottom of the screen + +6. Review Oracle Virtual Cloud Network (VCN), Subnets, and Gateways + + Click 'Create' to create the VCN + +7. The Virtual Cloud Network creation is completing + ![Completed VCN](./images/vcn-wizard-review.png "Review complete vcn wizard work") + +8. Click 'View VCN' to display the created VCN + ![Created VCN](./images/wizard-view-vcn.png "display the created") + +## Task 2: Configure security list to allow MySQL incoming connections + +1. On phoenix-vcn page under 'Subnets in lakehouse Compartment', click '**Crete subnet**' + +2. Create a subnet **subnetB** + ![Subnet creations](./images/subnet-create-a.png "Subnet creation") + *** Assign the IPV4-CIDR Block ip -10.0.2.0/24 + ![Subnet creation](./images/subnet-create-b.png "Subnet creation") + + ![Subnet creation](./images/subnetcreate-c.png "Subnet creation") +3. Delete the other two subnets created by default . + +4. On vcn_phoenix page under 'Subnets in lakehouse Compartment', click '**subnetB**' + ![VCN Details](./images/vcn-details.png "Show VCN Details") + +5. On Private subnetB page under 'Security Lists', click '**Security List for subnetB**' + ![VCN Security list](./images/vcn-security-list.png "Show Security Lists") + +6. On Security List for Private subnetB page under 'Ingress Rules', click '**Add Ingress Rules**' + ![VCN Ingress Rule](./images/vcn-mysql-ingress.png "Prepare for add Add Ingress Rules") + +7. On Add Ingress Rules page under Ingress Rule + + Add an Ingress Rule with Source CIDR + + ```bash + 0.0.0.0/0 + ``` + + Destination Port Range + + ```bash + 3306,33060 + ``` + + Description + + ```bash + MySQL Port Access + ``` + +8. Click 'Add Ingress Rule' + ![Add VCN Ingress Rule](./images/vcn-mysql-add-ingress.png "Save MySQL Ingress Rule entries") + +9. On Security List for Private subnetB page, the new Ingress Rules will be shown under the Ingress Rules List + ![View VCN Ingress Rule](./images/vcn-mysql-ingress-completed.png "view MySQL Ingress Rules") + +## Task 3: Configure security list to allow HTTP incoming connections + +1. Navigation Menu > Networking > Virtual Cloud Networks + +2. Open vcn_phoenix + +3. Click public subnetB + +4. Click Default Security List for vcn_phoenix + +5. Click Add Ingress Rules page under Ingress Rule + + Add an Ingress Rule with Source CIDR + + ```bash + 0.0.0.0/0 + ``` + + Destination Port Range + + ```bash + 80,443,5000 + ``` + + Description + + ```bash + Allow HTTP connections + ``` + +6. Click 'Add Ingress Rule' + + ![Add VCN HTTP](./images/vcn-ttp-add-ingress.png "Add HTTP Ingress Rule") + +7. On Security List for Default Security List for vcn_phoenix page, the new Ingress Rules will be shown under the Ingress Rules List + + ![View VCN HTTP](./images/vcn-ttp-ingress-completed.png"View VCN Completed HTTP Ingress rules") + +You may now **proceed to the next lab** + +## Acknowledgements + +- **Author** - Biswanath Nanda,Bhushan Arora Principal Cloud Architect, North America Cloud Infrastructure - Engineering +- **Contributors** - Biswanath Nanda, Bhushan Arora +- **Last Updated By/Date** - Biswanath Nanda, November 2024 diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/db-list.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/db-list.png new file mode 100644 index 00000000..4a3cd144 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/db-list.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/menuvcn.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/menuvcn.png new file mode 100644 index 00000000..86ea01e6 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/menuvcn.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-avail-domain.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-avail-domain.png new file mode 100644 index 00000000..a955858b Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-avail-domain.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-backup.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-backup.png new file mode 100644 index 00000000..645260a8 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-backup.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configuration-version.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configuration-version.png new file mode 100644 index 00000000..6d83a44b Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configuration-version.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configure-hardware-button.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configure-hardware-button.png new file mode 100644 index 00000000..69e21b05 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configure-hardware-button.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configure-hardware.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configure-hardware.png new file mode 100644 index 00000000..429aa9e4 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-configure-hardware.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create-button.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create-button.png new file mode 100644 index 00000000..aa3927a0 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create-button.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create-select-option.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create-select-option.png new file mode 100644 index 00000000..3ea022d0 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create-select-option.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create.png new file mode 100644 index 00000000..de0eb50b Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-create.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-data-import.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-data-import.png new file mode 100644 index 00000000..1f66bc60 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-data-import.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-active.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-active.png new file mode 100644 index 00000000..cbf66a35 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-active.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-basic.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-basic.png new file mode 100644 index 00000000..f7ca308c Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-basic.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-connection-tab.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-connection-tab.png new file mode 100644 index 00000000..e7536288 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-connection-tab.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-creating.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-creating.png new file mode 100644 index 00000000..1b56d416 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-creating.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-system-selection.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-system-selection.png new file mode 100644 index 00000000..59fa74b4 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-heatwave-system-selection.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-host.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-host.png new file mode 100644 index 00000000..d9d8fafb Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-host.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-menu.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-menu.png new file mode 100644 index 00000000..499dfbcb Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-menu.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-password.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-password.png new file mode 100644 index 00000000..d27c2cc8 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-password.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-vcn.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-vcn.png new file mode 100644 index 00000000..33bfd0a4 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/mysql-vcn.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/networking-main.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/networking-main.png new file mode 100644 index 00000000..f9c2c113 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/networking-main.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/oci-dashboard.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/oci-dashboard.png new file mode 100644 index 00000000..c6e6e4b9 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/oci-dashboard.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/regionSelector.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/regionSelector.png new file mode 100644 index 00000000..62f5b77f Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/regionSelector.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-a.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-a.png new file mode 100644 index 00000000..544f98e5 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-a.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-aa.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-aa.png new file mode 100644 index 00000000..0599517d Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-aa.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-b.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-b.png new file mode 100644 index 00000000..80f260c6 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnet-create-b.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnetcreate-c.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnetcreate-c.png new file mode 100644 index 00000000..9f4f39a0 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/subnetcreate-c.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-details.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-details.png new file mode 100644 index 00000000..111e98a2 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-details.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-internet-connect-config.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-internet-connect-config.png new file mode 100644 index 00000000..d57dd2e0 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-internet-connect-config.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-add-ingress.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-add-ingress.png new file mode 100644 index 00000000..913043cd Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-add-ingress.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-ingress-completed.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-ingress-completed.png new file mode 100644 index 00000000..d8239f26 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-ingress-completed.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-ingress.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-ingress.png new file mode 100644 index 00000000..a1e3d441 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-mysql-ingress.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-security-list.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-security-list.png new file mode 100644 index 00000000..883966a0 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-security-list.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-ttp-add-ingress.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-ttp-add-ingress.png new file mode 100644 index 00000000..e8b331ed Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-ttp-add-ingress.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-ttp-ingress-completed.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-ttp-ingress-completed.png new file mode 100644 index 00000000..ae4400bc Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-ttp-ingress-completed.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-wizard-review.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-wizard-review.png new file mode 100644 index 00000000..e35dde33 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-wizard-review.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-wizard-start.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-wizard-start.png new file mode 100644 index 00000000..a1dab7ad Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/vcn-wizard-start.png differ diff --git a/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/wizard-view-vcn.png b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/wizard-view-vcn.png new file mode 100644 index 00000000..a70f8f1b Binary files /dev/null and b/heatwave-lakehouse-cdp/create-heatwave-vcn-db/images/wizard-view-vcn.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/create-mysql-database-system.md b/heatwave-lakehouse-cdp/create-mysql-database-system/create-mysql-database-system.md new file mode 100644 index 00000000..ab273674 --- /dev/null +++ b/heatwave-lakehouse-cdp/create-mysql-database-system/create-mysql-database-system.md @@ -0,0 +1,74 @@ +# Create MySQL Database HeatWave + +## Introduction + +This lab walks you through the steps to create **MySQL Database System** in Oracle Cloud Infrastructure. + +_Estimated Lab Time_: 40 minutes + +### Objectives + +In this lab, you will: +- Be able to launch MySQL Heatwave Database System in Oracle Cloud. + +### Prerequisites + +This lab assumes you have: +- An Oracle account +- Privileges to use Oracle Cloud resources +- Compartment and VCN has been created. + +Note - This lab "**2c -Create MySQL Database HeatWave**" is independent of the previous lab **2b**.Once the above Prerequisites are met the below steps can be executed. + +## Task 1: Create MySQL Database System in Oracle Cloud + +1. Login into OCI console, Navigate to **Databases** menu and click on DB Systems under **MySQL HeatWave**. + + ![login-into-oci-console](images/navigation-b1.png) + +2. Click on "**Create DB System**" + + ![create-db](images/db-system-navigation-b2.png) + +3. Select **Production** , **Compartment** , **DB System Name** and provide **username**/**password**, confirm password + + ![create-db-01](images/compartment-user-b3.png) + +4. Select **Standalone** then select VCN/Subnet in the corresponding compartment: + + ![username](images/standalone-vcn-b4.png) + +5. (a) **Enable MySQL Heatwave** and Click on Change Shape as in following screen shot + + + ![changeshape](images/shape-storage-b5-a-new.png) + + + (b) Select OCPU , Choose the Shape (16 OCPU/512 GB Memory) then click on "Select a shape" button + + ![ocpu](images/select-ocpu-b5-b-new.png) + + (c) Following options you can keep as default, storage and backup: + + ![backup-default](images/backup-default-b5-c-new.png) + + (d) Then Click on “advanced options” –> Connections -> give hostname “eemysql” and click on “**Create**” to initiate creation process. + + **Note : The host name must be unique within the subnet. If it is not unique, the DB system will fail to provision. It will be used later in the live lab. + + ![adv-hostname](images/advanced-hostname-b5-d-new.png) + +6. Once Instance is launched, it will be Active in few minutes: + + ![active-db](images/db-active-status.png) + +7. Check the MySQL Endpoint(Hostname/Address), which we will use later to connect: + + ![endpoints-db](images/endpoints.png) + +You may now proceed to the next lab. + +## Acknowledgements +* **Author** - Bhushan Arora, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Bhushan Arora ,Principal Cloud Architect,Biswanath Nanda, Master Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Bhushan Arora, November 2024 diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/active-db.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/active-db.png new file mode 100644 index 00000000..bfb940c4 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/active-db.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/advanced-hostname-b5-d-new.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/advanced-hostname-b5-d-new.png new file mode 100644 index 00000000..5ce01b73 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/advanced-hostname-b5-d-new.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/availability.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/availability.png new file mode 100644 index 00000000..8b8a6d90 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/availability.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/backup-default-b5-c-new.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/backup-default-b5-c-new.png new file mode 100644 index 00000000..34f7fcc4 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/backup-default-b5-c-new.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/code_9.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/code_9.png new file mode 100644 index 00000000..f48bf417 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/code_9.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/compartment-user-b3.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/compartment-user-b3.png new file mode 100644 index 00000000..384244dd Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/compartment-user-b3.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/compartment_user_b3.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/compartment_user_b3.png new file mode 100644 index 00000000..c3382556 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/compartment_user_b3.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/create-db-01.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/create-db-01.png new file mode 100644 index 00000000..c07ddfc5 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/create-db-01.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/create-db.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/create-db.png new file mode 100644 index 00000000..e62c6929 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/create-db.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/create.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/create.png new file mode 100644 index 00000000..a79ac3a5 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/create.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/db-active-status.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db-active-status.png new file mode 100644 index 00000000..3b90a247 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db-active-status.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/db-system-navigation-b2.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db-system-navigation-b2.png new file mode 100644 index 00000000..510a8450 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db-system-navigation-b2.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/db_active_b6.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db_active_b6.png new file mode 100644 index 00000000..cdea1fcf Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db_active_b6.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/db_system_navigation_b2.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db_system_navigation_b2.png new file mode 100644 index 00000000..82362348 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/db_system_navigation_b2.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/endpoints.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/endpoints.png new file mode 100644 index 00000000..58ee063f Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/endpoints.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/endpoints_b7.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/endpoints_b7.png new file mode 100644 index 00000000..82f6b902 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/endpoints_b7.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/login-oci-console.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/login-oci-console.png new file mode 100644 index 00000000..0b79baf6 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/login-oci-console.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/navigation-b1.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/navigation-b1.png new file mode 100644 index 00000000..1ee4f200 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/navigation-b1.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/navigation_b1.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/navigation_b1.png new file mode 100644 index 00000000..1ee4f200 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/navigation_b1.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/select-ocpu-b5-b-new.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/select-ocpu-b5-b-new.png new file mode 100644 index 00000000..aa7634f9 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/select-ocpu-b5-b-new.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/shape-storage-b5-a-new.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/shape-storage-b5-a-new.png new file mode 100644 index 00000000..aeb1ad80 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/shape-storage-b5-a-new.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/shape_storage_b5.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/shape_storage_b5.png new file mode 100644 index 00000000..08dfc296 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/shape_storage_b5.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/standalone-vcn-b4.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/standalone-vcn-b4.png new file mode 100644 index 00000000..86daee83 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/standalone-vcn-b4.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/standalone_vcn_b4.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/standalone_vcn_b4.png new file mode 100644 index 00000000..1ceed61f Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/standalone_vcn_b4.png differ diff --git a/heatwave-lakehouse-cdp/create-mysql-database-system/images/username.png b/heatwave-lakehouse-cdp/create-mysql-database-system/images/username.png new file mode 100644 index 00000000..76565a6f Binary files /dev/null and b/heatwave-lakehouse-cdp/create-mysql-database-system/images/username.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/create-stream.md b/heatwave-lakehouse-cdp/create-stream/create-stream.md new file mode 100644 index 00000000..b84a8205 --- /dev/null +++ b/heatwave-lakehouse-cdp/create-stream/create-stream.md @@ -0,0 +1,111 @@ +# Creating the OCI stream service and build the synthetic data generator. + +### Introduction + +In this lab, you will be creating the OCI stream service and will be deploying the Python Streaming Simulator code base to the Compute VM created in ***Lab2B*** + +_Estimated Lab Time_: 30 minutes + +### Objectives + +- Create OCI stream service +- Deploy the generator codebase to compute VM +- Perform script refactoring and execute the code + +### Pre-Requisites + +* Completed ***Lab 2B - Task 3: Create Compute instance*** .Once the copute instance has been created , the below tasks can be executed. + +## +## Task 1: Create an OCI Stream + +1. Open the navigation menu and click ***Analytics & AI***. Under ***Messaging***, click ***Streaming***. A list of existing streams is displayed. +![OCI Stream console](./images/stream-console.png) + +2. Click ***Create Stream*** at the top of the screen.Make sure to create the stream in the same demo compartment. +![OCI Stream console](./images/strem-select.png) + +3. ***Stream Name:*** Specify a friendly name for the stream and create the streaming app using default pool option "Auto-Create a default stream pool" . + + Streaming application name example - "e2e-stream-mysqlhw". +![OCI Stream console](./images/streaming-pool.png) + + +4. Click ***Create***. + +5. In a few seconds the streaming application gets created . +![OCI Stream console](./images/stream-create.png) + + + +## Task 2: Deploy the Python generator script on Compute VM +1. Login to the compute VM using user ***OPC*** (Using and SCP tool like WINSCP or Mobaextern). + +2. Download the Labfiles and navigate to the below folder location to collect the codepump.zip file. + + Download file [`MYSQLLakehouse_labfiles.zip`](https://objectstorage.us-ashburn-1.oraclecloud.com/p/RPka_orWclfWJmKN3gTHfEiv-uPckBJTZ3FV0sESZ3mm3PDCQcVDCT-uM2dsJNGf/n/orasenatdctocloudcorp01/b/MYSQLLakehouse_labfiles/o/MYSQLLakehouse_labfiles.zip) + + *** Python Framework Location in the Zip file - \_MYSQLLakehouse_labfiles\_Lab4b\_ProducerCodebase + ![Python Generator code loc](images/generator-zip.png" ") + +3. Upload the codepump.zip to the compute VM in /home/opc + +4. Ensure python 3.8 and OCI library is installed in the compute VM + + + Run the below commands to get the packages installed. + + ``` + + sudo yum install python38 + sudo yum install oci + python3.8 --version + ``` +5. unzip the zip file + + ![Python Generator code loc](images/unzip-file.png " ") + +6. Naviagte to the below location + ![Python Generator code loc](images/navigate-gen.png " ") + +7. Navigate to Keys directory and paste your OCI keys in the below directory + *** Flollow the below link ,which shows the steps to configure your private and public keys. + + **[Click here for key generation ]( https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm)** + + ![Python Generator code loc](images/key-gen.png" ") + + ![Python Generator code loc](images/key-file-content.png" ") + +8. Navigate to the ***Code*** directory. + Change the contents of the code - ProducerCodeLoop_ETC.py (in VI editor) + ![Python Generator code loc](images/code-script.png" ") + + ![OCI streaming application connection details from Producer code](images/streaming-connection-check.png " ") + + *** Pick the parameters (ociMessageEndpoint and ociStreamOcid) from the OCI streaming instance created and parameters shown in the above snapshot . + *** Pick the parameters (device_file_path and ociConfigFilePath) from the configured Compute server. + + ``` + + device_file_path = '/home/opc/codepump/data/datapumper-input-file-2.csv' - Copy the path of the file from the compute server path. + ociMessageEndpoint = "https://cell-1.streaming.us-xyz" + ociStreamOcid = "ocid....xyz" + ociConfigFilePath = "/home/opc/codepump/keys/config" + ociProfileName = "DEFAULT" + ``` + + +9. Save and run the python code + + ``` + + python3 ProducerCodeLoop_ETC.py + ``` + +You may now **proceed to the next lab** + +## Acknowledgements +* **Author** - Biswanath Nanda, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Biswanath Nanda, November 2024 \ No newline at end of file diff --git a/heatwave-lakehouse-cdp/create-stream/image.png b/heatwave-lakehouse-cdp/create-stream/image.png new file mode 100644 index 00000000..c8a8e3b5 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/image.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/DG_Navigation1.png b/heatwave-lakehouse-cdp/create-stream/images/DG_Navigation1.png new file mode 100644 index 00000000..fe18dd18 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/DG_Navigation1.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/Picture 1.png b/heatwave-lakehouse-cdp/create-stream/images/Picture 1.png new file mode 100644 index 00000000..cf728f02 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/Picture 1.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/allow_dg_to_manage.png b/heatwave-lakehouse-cdp/create-stream/images/allow_dg_to_manage.png new file mode 100644 index 00000000..c455b80a Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/allow_dg_to_manage.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/code-script.png b/heatwave-lakehouse-cdp/create-stream/images/code-script.png new file mode 100644 index 00000000..d5524ba7 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/code-script.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/compute-select-vm.png b/heatwave-lakehouse-cdp/create-stream/images/compute-select-vm.png new file mode 100644 index 00000000..83fa30dc Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/compute-select-vm.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/create_dg3.png b/heatwave-lakehouse-cdp/create-stream/images/create_dg3.png new file mode 100644 index 00000000..3fd782db Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/create_dg3.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/create_dg_matching_rule.png b/heatwave-lakehouse-cdp/create-stream/images/create_dg_matching_rule.png new file mode 100644 index 00000000..1d8c7717 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/create_dg_matching_rule.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/create_policy_1.png b/heatwave-lakehouse-cdp/create-stream/images/create_policy_1.png new file mode 100644 index 00000000..25d6d414 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/create_policy_1.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/create_policy_builder.png b/heatwave-lakehouse-cdp/create-stream/images/create_policy_builder.png new file mode 100644 index 00000000..d5059cb6 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/create_policy_builder.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/default_domain.png b/heatwave-lakehouse-cdp/create-stream/images/default_domain.png new file mode 100644 index 00000000..0279eddc Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/default_domain.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/dg_navigation.png b/heatwave-lakehouse-cdp/create-stream/images/dg_navigation.png new file mode 100644 index 00000000..fe18dd18 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/dg_navigation.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/generator-zip.png b/heatwave-lakehouse-cdp/create-stream/images/generator-zip.png new file mode 100644 index 00000000..6b380d18 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/generator-zip.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/key-file-content.png b/heatwave-lakehouse-cdp/create-stream/images/key-file-content.png new file mode 100644 index 00000000..8bd12c85 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/key-file-content.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/key-gen.png b/heatwave-lakehouse-cdp/create-stream/images/key-gen.png new file mode 100644 index 00000000..2122b345 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/key-gen.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/navigate-gen.png b/heatwave-lakehouse-cdp/create-stream/images/navigate-gen.png new file mode 100644 index 00000000..c3fa2c88 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/navigate-gen.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/stream-console.png b/heatwave-lakehouse-cdp/create-stream/images/stream-console.png new file mode 100644 index 00000000..8f2edddf Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/stream-console.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/stream-create.png b/heatwave-lakehouse-cdp/create-stream/images/stream-create.png new file mode 100644 index 00000000..347585f6 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/stream-create.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/streaming-connection-check.png b/heatwave-lakehouse-cdp/create-stream/images/streaming-connection-check.png new file mode 100644 index 00000000..c4a3469d Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/streaming-connection-check.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/streaming-pool.png b/heatwave-lakehouse-cdp/create-stream/images/streaming-pool.png new file mode 100644 index 00000000..1c32f024 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/streaming-pool.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/strem-select.png b/heatwave-lakehouse-cdp/create-stream/images/strem-select.png new file mode 100644 index 00000000..154f119c Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/strem-select.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/unzip-file.png b/heatwave-lakehouse-cdp/create-stream/images/unzip-file.png new file mode 100644 index 00000000..0c5b9fc3 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/unzip-file.png differ diff --git a/heatwave-lakehouse-cdp/create-stream/images/upload-codebase.png b/heatwave-lakehouse-cdp/create-stream/images/upload-codebase.png new file mode 100644 index 00000000..18400ed8 Binary files /dev/null and b/heatwave-lakehouse-cdp/create-stream/images/upload-codebase.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/dataflow.md b/heatwave-lakehouse-cdp/dataflow/dataflow.md new file mode 100644 index 00000000..8b3f454e --- /dev/null +++ b/heatwave-lakehouse-cdp/dataflow/dataflow.md @@ -0,0 +1,212 @@ + +## Setting up Oracle Cloud Infrastructure Data Flow + +### Introduction +The labs shows how we can set up the OCI Data Flow PySpark application and extract data from ***demo-events-raw-mysqlhw*** bucket ,consolidate the datasets and load the data to ***demo-events-silver-mysqlhw*** bucket . + +Setting up OCI Data Flow +- Provision the OCI Data flow in the tenancy +- Deploy the PySpark application to run using Data flow. +- Create PAR Link for the bucket + +*Estimated Lab Time*: 30 mins + +### Objectives + +In this lab you will learn about Oracle Data Flow , how to set the service ,dependent policies and run PySpark application. + +Before you can create, manage and execute applications in Data Flow, the tenant administrator (or any user with elevated privileges to create buckets and modify IAM) must create specific storage buckets and associated policies in IAM. These set up steps are required in Object Store and IAM for Data Flow functioning . + +This lab will set the foundation for future labs to follow. + +- Provision the OCI Data flow in the tenancy +- Deploy the PySpark application to run using Data flow. +- Create PAR Link for the bucket. + +### Prerequisites + +Before you Begin with Data Flow lab, you must have: + +* An Oracle Cloud Infrastructure account. Trial accounts can be used to demo Data Flow. +* A Service Administrator role for your Oracle Cloud services. When the service is activated, Oracle sends the credentials and URL to the designated Account Administrator. The Account Administrator creates an account for each user who needs access to the service. +* A supported browser, such as: + * Microsoft Internet Explorer 11.x+ + * Mozilla Firefox ESR 38+ + * Google Chrome 42+ +* Familiarity with Object Storage Service. + +## Task 1: Object Storage Bucket : Validation Check + +Before running the application in the Data Flow service. + +Please ensure you have already created the bucket called **dataflow-logs** and **Dataflowbucket** in the Object Store service as previously mentioned in **Lab 1 -> Task2** + + +## Task 2(a): Identity: Policy Set Up + +A user's permissions to access services comes from the _groups_ to which they belong. The permissions for a group are defined by policies. Policies define what actions members of a group can perform, and in which compartments. Users can access services and perform operations based on the policies set for the groups of which they are members. + +We'll create a user, a group, and policies to understand the concept. + + +1. **User Policies** : Data Flow requires policies to be set in IAM to access resources in order to manage and run applications. We categorize the Data Flow users into two groups for clear separation of authority administrator and users: + + * From the OCI Services menu, click `Identity and Security` -> `Domains` -> `Default domain` and `Groups` + + ![groups](images/df-groups001.png " ") + + * Create a group in your identity service called `dataflow-admin` + + ![admin](images/dataflowadminuser.png " ") + + * Click on your new group to display it. Your new group is displayed. Assign user to the groups + + ![admin user](images/addusertodfadmingroup.png " ") + + ![add users](images/addusers.png " ") + + * From the OCI Services menu, click `Identity and Security` and in identity click `Policies` + + ![new policy](images/policies-01.png " ") + + * Create a policy called `dataflow-admin` in your `compartment` and add the following statements: + + ![policy](images/createadadminpolicy.png " ") + + ``` + + ALLOW GROUP dataflow-admin TO READ buckets IN TENANCY + + ``` + + ``` + + ALLOW GROUP dataflow-admin TO MANAGE dataflow-family IN TENANCY + + ``` + ``` + + ALLOW GROUP dataflow-admin TO MANAGE objects IN TENANCY WHERE ALL + {target.bucket.name='dataflow-logs', any {request.permission='OBJECT_CREATE', + request.permission='OBJECT_INSPECT'}} + + ``` + * Create a group in your identity service called dataflow-users and add users to this group. + + ![group](images/dfgroup.png " ") + + * Create a policy called dataflow-users and add the following statements: + + ![policy](images/dfcreateuserspolicies.png " ") + + ``` + + ALLOW GROUP dataflow-users TO READ buckets IN TENANCY + + ``` + ``` + + ALLOW GROUP dataflow-users TO USE dataflow-family IN TENANCY + + ``` + ``` + + ALLOW GROUP dataflow-users TO MANAGE dataflow-family IN TENANCY WHERE ANY {request.user.id = target.user.id, request.permission = 'DATAFLOW_APPLICATION_CREATE', request.permission = 'DATAFLOW_RUN_CREATE'} + + ``` + *Note: Replace with the name of your tenancy* + +2. **Service Policies** : The Data Flow service needs permission to perform actions on behalf of the user or group on objects within the tenancy.To set it up, create a policy called `dataflow-service` and add the following statement: + + ``` + + ALLOW SERVICE dataflow TO READ objects IN tenancy WHERE target.bucket.name='dataflow-logs' + + + ``` +## Task 2(b): Deploy the PySpark Application to the data flow +In this task we will be deploying the PySpark Application to OCI Objectstore and will be creating a Data flow application. +1. Modify the PySpark script - Open the script locall +- Download the Labfiles and navigate to the below folder location to collect the codepump.zip file. + + Download file [`MYSQLLakehouse_labfiles.zip`](https://objectstorage.us-ashburn-1.oraclecloud.com/p/RPka_orWclfWJmKN3gTHfEiv-uPckBJTZ3FV0sESZ3mm3PDCQcVDCT-uM2dsJNGf/n/orasenatdctocloudcorp01/b/MYSQLLakehouse_labfiles/o/MYSQLLakehouse_labfiles.zip) + + *** Python Framework Location in the Zip file - MYSQLLakehouse_labfiles\_Lab6a + + ![Spark Application code](images/spark-code.png " ") + + ``` + + line 26: change the namespace of the bucket path + line36: change the namespace of the bucket path + + + ``` + ![Spark Application code](images/script-chnage.png " ") + +2. Upload the main.py script to Objectstore bucket ***Dataflowbucket*** + + ![Spark Application code](images/script-upload.png " ") + +3. Create a PySpark application on Data Flow. + Navigate to DataFlow + + ![Spark Application code](images/df-navigate.png " ") + + Create the Data Flow Application + + DataFlow Application Name - + ``` + + dfPipeline + + + ``` + *** Note - Select the parameters based on the below snapshot provided. + ![Spark Application Navigate](images/df-create-app.png " ") + + ![Spark Application Create](images/create-one.png " ") + + ![Spark Application Allocate Resources](images/create-two.png " ") + + ![Spark Application Allocate Resources](images/df-app-executor.png" ") + + ![Spark Application Allocate Resources](images/df-app-selection.png" ") + +4. Run the PySpark App + ![Spark Application Run](images/run-app.png " ") + +## Task 3: Create the PAR Link for the "e2e-demo-events-silver-mysqlhw" bucket + +1. Create a PAR URL for all of the **e2e-demo-events-silver-mysqlhw** objects with a prefix + + - a. From your OCI console, navigate to your e2e-demo-events-silver-mysqlhw bucket in OCI. + + ![Go to Bucket](./images/bucket-details.png "Go to Bucket folder") + - b. Click the three vertical dots of "e2e-demo-events-silver-mysqlhw" bucket + + ![Create Par](./images/bucket-three-dots.png "create par") + + - c. Click on ‘Create Pre-Authenticated Request’ + - d. Click to select the ‘Objects with prefix’ option under ‘Pre-Authenticated Request Target’. + - e. Leave the ‘Access Type’ option as-is: ‘Permit object reads on those with the specified prefix’. + - f. Click to select the ‘Enable Object Listing’ checkbox. + - g. Click the ‘Create Pre-Authenticated Request’ button. + + ![Create Par Configuration](./images/par-with-prefix-object-listing.png "Par Configuration") + + - h. Click the ‘Copy’ icon to copy the PAR URL, it will not be shown again. + - i. Save the generated PAR URL; you will need it later. + - j. You can test the URL out by pasting it in your browser. It should return output like this: + + ![List par files](./images/list-par-file.png "par files list") + +2. Save the generated PAR URL; you will need it in the next task + + +You may now **proceed to the next lab** +## Acknowledgements +* **Author** - Biswanath Nanda, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect,Sharmistha das ,Master Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Biswanath Nanda, November 2024 + diff --git a/heatwave-lakehouse-cdp/dataflow/images/Picture 1.png b/heatwave-lakehouse-cdp/dataflow/images/Picture 1.png new file mode 100644 index 00000000..6c2eeb4c Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/Picture 1.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/addusers.png b/heatwave-lakehouse-cdp/dataflow/images/addusers.png new file mode 100644 index 00000000..fc1fd010 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/addusers.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/addusertodfadmingroup.png b/heatwave-lakehouse-cdp/dataflow/images/addusertodfadmingroup.png new file mode 100644 index 00000000..bb7c64e7 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/addusertodfadmingroup.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/alter-table-load.png b/heatwave-lakehouse-cdp/dataflow/images/alter-table-load.png new file mode 100644 index 00000000..e4bcf9ff Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/alter-table-load.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/bucket-details.png b/heatwave-lakehouse-cdp/dataflow/images/bucket-details.png new file mode 100644 index 00000000..17ea5c3e Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/bucket-details.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/bucket-three-dots.png b/heatwave-lakehouse-cdp/dataflow/images/bucket-three-dots.png new file mode 100644 index 00000000..62a47a4e Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/bucket-three-dots.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/create-one.png b/heatwave-lakehouse-cdp/dataflow/images/create-one.png new file mode 100644 index 00000000..729f22a2 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/create-one.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main copy.png b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main copy.png new file mode 100644 index 00000000..15a5c898 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main copy.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main.png b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main.png new file mode 100644 index 00000000..4437273d Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main_rem.textClipping b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main_rem.textClipping new file mode 100644 index 00000000..f195ac84 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main_rem.textClipping differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main_remove.png b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main_remove.png new file mode 100644 index 00000000..caac71b8 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/create-table-output-main_remove.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/create-two.png b/heatwave-lakehouse-cdp/dataflow/images/create-two.png new file mode 100644 index 00000000..49e0256e Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/create-two.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/createadadminpolicy.png b/heatwave-lakehouse-cdp/dataflow/images/createadadminpolicy.png new file mode 100644 index 00000000..c7456ab9 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/createadadminpolicy.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/createbucket001.png b/heatwave-lakehouse-cdp/dataflow/images/createbucket001.png new file mode 100644 index 00000000..67c7ba62 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/createbucket001.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/dataflowadminuser.png b/heatwave-lakehouse-cdp/dataflow/images/dataflowadminuser.png new file mode 100644 index 00000000..37cf4056 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/dataflowadminuser.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-app-executor.png b/heatwave-lakehouse-cdp/dataflow/images/df-app-executor.png new file mode 100644 index 00000000..5a30a6aa Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-app-executor.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-app-selection.png b/heatwave-lakehouse-cdp/dataflow/images/df-app-selection.png new file mode 100644 index 00000000..5760aaaf Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-app-selection.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-bucket-01.png b/heatwave-lakehouse-cdp/dataflow/images/df-bucket-01.png new file mode 100644 index 00000000..d189a91d Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-bucket-01.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-create-app.png b/heatwave-lakehouse-cdp/dataflow/images/df-create-app.png new file mode 100644 index 00000000..83148841 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-create-app.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-groups001.png b/heatwave-lakehouse-cdp/dataflow/images/df-groups001.png new file mode 100644 index 00000000..4deaebb0 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-groups001.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-navigate.png b/heatwave-lakehouse-cdp/dataflow/images/df-navigate.png new file mode 100644 index 00000000..8b758599 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-navigate.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-object-store01.png b/heatwave-lakehouse-cdp/dataflow/images/df-object-store01.png new file mode 100644 index 00000000..7ec05639 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-object-store01.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-usergroup.png b/heatwave-lakehouse-cdp/dataflow/images/df-usergroup.png new file mode 100644 index 00000000..e365a44a Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-usergroup.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/df-warehouse.png b/heatwave-lakehouse-cdp/dataflow/images/df-warehouse.png new file mode 100644 index 00000000..e17eec3e Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/df-warehouse.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/dfcreateuserspolicies.png b/heatwave-lakehouse-cdp/dataflow/images/dfcreateuserspolicies.png new file mode 100644 index 00000000..ea3e5ab6 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/dfcreateuserspolicies.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/dfgroup.png b/heatwave-lakehouse-cdp/dataflow/images/dfgroup.png new file mode 100644 index 00000000..e6f1207e Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/dfgroup.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/heatwave-load-op1.png b/heatwave-lakehouse-cdp/dataflow/images/heatwave-load-op1.png new file mode 100644 index 00000000..21744366 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/heatwave-load-op1.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/heatwave-load-op2.png b/heatwave-lakehouse-cdp/dataflow/images/heatwave-load-op2.png new file mode 100644 index 00000000..95b88083 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/heatwave-load-op2.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/list-par-file.png b/heatwave-lakehouse-cdp/dataflow/images/list-par-file.png new file mode 100644 index 00000000..e86899d0 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/list-par-file.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/log-create-table-output.png b/heatwave-lakehouse-cdp/dataflow/images/log-create-table-output.png new file mode 100644 index 00000000..54fb4532 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/log-create-table-output.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/par-with-prefix-object-listing.png b/heatwave-lakehouse-cdp/dataflow/images/par-with-prefix-object-listing.png new file mode 100644 index 00000000..33dd510b Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/par-with-prefix-object-listing.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/policies-01.png b/heatwave-lakehouse-cdp/dataflow/images/policies-01.png new file mode 100644 index 00000000..dcc97d5c Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/policies-01.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/run-app.png b/heatwave-lakehouse-cdp/dataflow/images/run-app.png new file mode 100644 index 00000000..c5be4ca0 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/run-app.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/sample-five-rows.png b/heatwave-lakehouse-cdp/dataflow/images/sample-five-rows.png new file mode 100644 index 00000000..3128ee42 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/sample-five-rows.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/script-chnage.png b/heatwave-lakehouse-cdp/dataflow/images/script-chnage.png new file mode 100644 index 00000000..ba0ea2ea Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/script-chnage.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/script-upload.png b/heatwave-lakehouse-cdp/dataflow/images/script-upload.png new file mode 100644 index 00000000..e7ad452a Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/script-upload.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/set-db-list.png b/heatwave-lakehouse-cdp/dataflow/images/set-db-list.png new file mode 100644 index 00000000..be096a53 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/set-db-list.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/set-load-table-examples.png b/heatwave-lakehouse-cdp/dataflow/images/set-load-table-examples.png new file mode 100644 index 00000000..a36e61d7 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/set-load-table-examples.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/set-options.png b/heatwave-lakehouse-cdp/dataflow/images/set-options.png new file mode 100644 index 00000000..996a9273 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/set-options.png differ diff --git a/heatwave-lakehouse-cdp/dataflow/images/spark-code.png b/heatwave-lakehouse-cdp/dataflow/images/spark-code.png new file mode 100644 index 00000000..011d5305 Binary files /dev/null and b/heatwave-lakehouse-cdp/dataflow/images/spark-code.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/dg-policy.md b/heatwave-lakehouse-cdp/dg-policy/dg-policy.md new file mode 100644 index 00000000..dca8f726 --- /dev/null +++ b/heatwave-lakehouse-cdp/dg-policy/dg-policy.md @@ -0,0 +1,73 @@ +# Creating dynamic group to access streams/objects + +### Introduction + +In this lab we will create the dynamic group + +_Estimated Lab Time_: 5 minutes + +### Objectives +In this lab, you will: +- Create Dynamic Group +- Create Policies + +## +## Task 1 Create Dynamic Group + +1. Open the navigation menu and click ***Domains*** + + ![Domain Navigation](./images/dg_navigation.png) + +2. Click on ***Default***(Current Domain) + + ![OCI Default Domain](./images/default_domain.png) + +3. Click on ***Dynamic Group*** -> ***Create dynamic group***: + + ![OCI Stream console](./images/create_dg3.png) + + +4. Give Details as in following screen shot: + + + ![Create Dynamic Group Details](./images/create_dg_matching_rule.png) + + ``` + + ALL {resource.type = 'fnfunc', resource.compartment.id = 'ocid1.compartment.oc1……xyz’} + + + ``` + +5. Open the navigation Menu -> Go to Identity -> Policies -> Create Policy + + ![create_policies](./images/create_policy_1.png) + +6. Click on Create Policy and give it name **function-dynamic-group** + + In following copy & Replace the OCID of compartment **e2e-demo-specialist-eng** + + ``` + + allow dynamic-group function-dynamic-group to manage stream-family in compartment id resource.compartment.id = ‘ocid1.compartment.oc1……xyz’ + allow dynamic-group function-dynamic-group to manage stream-pull in compartment id resource.compartment.id = ‘ocid1.compartment.oc1……xyz’ + allow dynamic-group function-dynamic-group to manage streams in compartment id resource.compartment.id = ‘ocid1.compartment.oc1……xyz’ + allow dynamic-group function-dynamic-group to manage stream-pools in compartment id resource.compartment.id = ‘ocid1.compartment.oc1……xyz’ + allow dynamic-group function-dynamic-group to manage objects in compartment id resource.compartment.id = ‘ocid1.compartment.oc1……xyz’ + + ``` + + Above Create Policy should like as in following screen shots: + + ![create_policies_allow](./images/create_policy_builder.png) + +7. Now you should see **function-dynamic-group** has been created + + ![allow-dynamic-group](./images/allow_dg_to_manage.png) + +You may now **proceed to the next lab** + +## Acknowledgements +* **Author** - Bhushan Arora, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Master Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect, Lovelesh Saxena, Principal Cloud Architect +* **Last Updated By/Date** - Bhushan Arora, November 2024 \ No newline at end of file diff --git a/heatwave-lakehouse-cdp/dg-policy/image.png b/heatwave-lakehouse-cdp/dg-policy/image.png new file mode 100644 index 00000000..c8a8e3b5 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/image.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/DG_Navigation1.png b/heatwave-lakehouse-cdp/dg-policy/images/DG_Navigation1.png new file mode 100644 index 00000000..fe18dd18 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/DG_Navigation1.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/Picture 1.png b/heatwave-lakehouse-cdp/dg-policy/images/Picture 1.png new file mode 100644 index 00000000..cf728f02 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/Picture 1.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/allow_dg_to_manage.png b/heatwave-lakehouse-cdp/dg-policy/images/allow_dg_to_manage.png new file mode 100644 index 00000000..c455b80a Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/allow_dg_to_manage.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/code-script.png b/heatwave-lakehouse-cdp/dg-policy/images/code-script.png new file mode 100644 index 00000000..d5524ba7 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/code-script.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/compute-select-vm.png b/heatwave-lakehouse-cdp/dg-policy/images/compute-select-vm.png new file mode 100644 index 00000000..83fa30dc Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/compute-select-vm.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/create_dg3.png b/heatwave-lakehouse-cdp/dg-policy/images/create_dg3.png new file mode 100644 index 00000000..3fd782db Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/create_dg3.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/create_dg_matching_rule.png b/heatwave-lakehouse-cdp/dg-policy/images/create_dg_matching_rule.png new file mode 100644 index 00000000..1d8c7717 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/create_dg_matching_rule.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/create_policy_1.png b/heatwave-lakehouse-cdp/dg-policy/images/create_policy_1.png new file mode 100644 index 00000000..25d6d414 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/create_policy_1.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/create_policy_builder.png b/heatwave-lakehouse-cdp/dg-policy/images/create_policy_builder.png new file mode 100644 index 00000000..d5059cb6 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/create_policy_builder.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/default_domain.png b/heatwave-lakehouse-cdp/dg-policy/images/default_domain.png new file mode 100644 index 00000000..0279eddc Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/default_domain.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/dg_navigation.png b/heatwave-lakehouse-cdp/dg-policy/images/dg_navigation.png new file mode 100644 index 00000000..fe18dd18 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/dg_navigation.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/generator-zip.png b/heatwave-lakehouse-cdp/dg-policy/images/generator-zip.png new file mode 100644 index 00000000..6b380d18 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/generator-zip.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/key-file-content.png b/heatwave-lakehouse-cdp/dg-policy/images/key-file-content.png new file mode 100644 index 00000000..8bd12c85 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/key-file-content.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/key-gen.png b/heatwave-lakehouse-cdp/dg-policy/images/key-gen.png new file mode 100644 index 00000000..2122b345 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/key-gen.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/navigate-gen.png b/heatwave-lakehouse-cdp/dg-policy/images/navigate-gen.png new file mode 100644 index 00000000..c3fa2c88 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/navigate-gen.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/stream-console.png b/heatwave-lakehouse-cdp/dg-policy/images/stream-console.png new file mode 100644 index 00000000..8f2edddf Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/stream-console.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/stream-create.png b/heatwave-lakehouse-cdp/dg-policy/images/stream-create.png new file mode 100644 index 00000000..347585f6 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/stream-create.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/streaming-connection-check.png b/heatwave-lakehouse-cdp/dg-policy/images/streaming-connection-check.png new file mode 100644 index 00000000..c4a3469d Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/streaming-connection-check.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/streaming-pool.png b/heatwave-lakehouse-cdp/dg-policy/images/streaming-pool.png new file mode 100644 index 00000000..1c32f024 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/streaming-pool.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/strem-select.png b/heatwave-lakehouse-cdp/dg-policy/images/strem-select.png new file mode 100644 index 00000000..154f119c Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/strem-select.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/unzip-file.png b/heatwave-lakehouse-cdp/dg-policy/images/unzip-file.png new file mode 100644 index 00000000..0c5b9fc3 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/unzip-file.png differ diff --git a/heatwave-lakehouse-cdp/dg-policy/images/upload-codebase.png b/heatwave-lakehouse-cdp/dg-policy/images/upload-codebase.png new file mode 100644 index 00000000..18400ed8 Binary files /dev/null and b/heatwave-lakehouse-cdp/dg-policy/images/upload-codebase.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/enable-heatwave-cluster.md b/heatwave-lakehouse-cdp/enable-heatwave-cluster/enable-heatwave-cluster.md new file mode 100644 index 00000000..4520cc2a --- /dev/null +++ b/heatwave-lakehouse-cdp/enable-heatwave-cluster/enable-heatwave-cluster.md @@ -0,0 +1,155 @@ +# Enable MySQL HeatWave Lakehouse in MySQL Database System + +### Introduction + +This lab walks you through the steps to enable **MySQL HeatWave Lakehouse** Cluster in MySQL Database System that we have created in earlier lab + +_Estimated Lab Time_: 20 minutes + +### Objectives + +In this lab, you will: +* Be able to enable **MySQL HeatWave Lakehouse** in Oracle Cloud. + +### Prerequisites + +This lab assumes you have: +- An Oracle account +- You have enough privileges to use Oracle Cloud resources +- Previous lab has been successfully completed + +## +## Task 1: Enable MySQL HeatWave Lakehouse in MySQL Database System + +1. Go to Navigation Menu Databases MySQL + ![click-on-db-systems](images/click-on-db-systems.png) + +2. Click the heatwave-db Database System link + ![click-on-db-systems](images/db-system-link.png) + +3. All DB Systems are created with point-in-time-recovery enabled by default. + + ***Note***: Prior to MySQL 8.3.0-u2 to enable Lakehouse,you must disable point-in-time-recovery. As here we are using the prior version; so we will disable the point-in-time-recovery. + + + As of now, you will see Point-in-time recovery is enabled and proceed to next steps: + + ![enabled-status-pitr](images/enabled-status-pitr.png) + + +4. Now Click on **Edit** button above to disable "Point-in-time recovery". In Following, uncheck **Point-in-time recovery** and click on **Save Changes** + + ![disable-pitr](images/disable-pitr.png) + +5. Click on **Lakehouse: Enable Button** under HeatWave + + ![enable-lakehouse](images/enable-lakehouse.png) + +6. There will be a popup, now click on enable button: + + ![confirm-again-enable](images/confirm-enable-lakehouse.png) + +7. It will take around 2 minutes then lakehouse status will be enabled as following: + + + ![lakehouse-heatwave-active](images/lakehouse-heatwave-active.png) + + +## Task 2: Connect to your MySQL HeatWave system using Cloud Shell + +1. If not already connected with SSH, connect to the Compute instance using OCI Cloud Shell... be sure replace the "private key file" and the "new compute instance ip" + + ```bash + ssh -i private_key_file opc@new_compute_instance_ip + ``` + +2. Connect to MySQL Database using the MySQL Shell client tool with the following command: + + ```bash + mysqlsh -uadmin -p -h 10.x.x.x --sql + ``` + + ![MySQL Shell Connect](./images/mysql-shell-login.png " mysql shell login") + +3. List schemas in your heatwave instance + + ```bash + show databases; + ``` + + ```bash + create database MFG_SENSOR; + ``` + + Now you should see a new database: + + ```bash + show databases; + ``` + + ![Databse Schemas](./images/show-databases.png "list schemas after") + +4. Change to the MFG\_SENSOR database + + Enter the following command at the prompt + + ```bash + USE MFG_SENSOR; + ``` + ![Use Schemas](./images/use-mfg.png "Use schemas") + +5. Create Table "T\_LIVE\_EVENT\_FEED\_DATA" for Real Time data: + + ```bash + CREATE TABLE MFG_SENSOR.T_LIVE_EVENT_FEED_DATA ( + TIMESTAMP VARCHAR(26) , + DEVICE_ID VARCHAR(26), + PROD_TYPE VARCHAR(26) , + DEVICE_TEMP DECIMAL(38,1), + ROTN_SPEED DECIMAL(38,1), + TORQUE DECIMAL(38,1), + TOOL_WEAR DECIMAL(38,1), + PROBABILITY_SCORE DECIMAL(20,7) + ); + ``` + + ```bash + ALTER TABLE MFG_SENSOR.T_LIVE_EVENT_FEED_DATA add (event_id INT AUTO_INCREMENT PRIMARY KEY); + ``` + + ```bash + ALTER TABLE MFG_SENSOR.T_LIVE_EVENT_FEED_DATA SECONDARY_ENGINE=RAPID; + ``` + + Result of above commands: + + ![Create Real Time Table](./images/create-real-time-table.png "real time table creation") + +6. Now load the data into heatwave: + + ```bash + ALTER TABLE MFG_SENSOR.T_LIVE_EVENT_FEED_DATA SECONDARY_LOAD; + ``` + + Result of above command: + + ![Secondary Load](./images/secondary-load-ream-time-table.png "secondary real time table load") + + +7. To see a list of the tables available in the MFG\_SENSOR schema + + Enter the following command at the prompt + + ```bash + show tables; + ``` + ![show tables](./images/show-schema-table.png "show tables") + + + +You may now proceed to the next lab. + +## Acknowledgements +* **Author** - Bhushan Arora, Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Contributors** - Bhushan Arora ,Principal Cloud Architect,Biswanath Nanda, Master Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Bhushan Arora, November 2024 diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/add-hw-cluster-lakehouse.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/add-hw-cluster-lakehouse.png new file mode 100644 index 00000000..c24ec7e6 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/add-hw-cluster-lakehouse.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/click-on-db-systems.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/click-on-db-systems.png new file mode 100644 index 00000000..3a252632 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/click-on-db-systems.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/confirm-enable-lakehouse.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/confirm-enable-lakehouse.png new file mode 100644 index 00000000..27618876 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/confirm-enable-lakehouse.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/create-real-time-table.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/create-real-time-table.png new file mode 100644 index 00000000..c100017b Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/create-real-time-table.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/db-system-link.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/db-system-link.png new file mode 100644 index 00000000..2d5c0543 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/db-system-link.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/disable-pitr.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/disable-pitr.png new file mode 100644 index 00000000..4a56546d Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/disable-pitr.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/enable-lakehouse.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/enable-lakehouse.png new file mode 100644 index 00000000..3d25c61d Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/enable-lakehouse.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/enabled-status-pitr.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/enabled-status-pitr.png new file mode 100644 index 00000000..23d9fff5 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/enabled-status-pitr.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/lakehouse-heatwave-active.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/lakehouse-heatwave-active.png new file mode 100644 index 00000000..d26f15f0 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/lakehouse-heatwave-active.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/more-actions.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/more-actions.png new file mode 100644 index 00000000..bde96a53 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/more-actions.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/mysql-shell-login.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/mysql-shell-login.png new file mode 100644 index 00000000..83c0b3f7 Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/mysql-shell-login.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/secondary-load-ream-time-table.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/secondary-load-ream-time-table.png new file mode 100644 index 00000000..d05aaa4c Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/secondary-load-ream-time-table.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/show-databases.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/show-databases.png new file mode 100644 index 00000000..8e59fe2d Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/show-databases.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/show-schema-table.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/show-schema-table.png new file mode 100644 index 00000000..6405818b Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/show-schema-table.png differ diff --git a/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/use-mfg.png b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/use-mfg.png new file mode 100644 index 00000000..c12d369f Binary files /dev/null and b/heatwave-lakehouse-cdp/enable-heatwave-cluster/images/use-mfg.png differ diff --git a/heatwave-lakehouse-cdp/insights/files/FishSurvey.ipynb-not-used b/heatwave-lakehouse-cdp/insights/files/FishSurvey.ipynb-not-used new file mode 100644 index 00000000..21e5bdb8 --- /dev/null +++ b/heatwave-lakehouse-cdp/insights/files/FishSurvey.ipynb-not-used @@ -0,0 +1,171 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "awful-brother", + "metadata": {}, + "source": [ + "# OCI Data Science - Fish Survey\n", + "\n", + "This is your Jupyter Notebook to make all the magic." + ] + }, + { + "cell_type": "markdown", + "id": "surgical-decrease", + "metadata": {}, + "source": [ + "### Import the libraries you need" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "suited-excuse", + "metadata": {}, + "outputs": [], + "source": [ + "import mysql.connector\n", + "import pandas as pd\n", + "import seaborn as sns\n", + "import matplotlib.pyplot as plt\n", + "sns.set_theme(style=\"whitegrid\")" + ] + }, + { + "cell_type": "markdown", + "id": "second-columbus", + "metadata": {}, + "source": [ + "### Create the connection with MySQL Database Service\n", + "\n", + "**REMEMBER:** Change the `PRIVATE_IP` before run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "pleasant-macro", + "metadata": {}, + "outputs": [], + "source": [ + "cnx = mysql.connector.connect(\n", + " host=\"PRIVATE_IP\",\n", + " user=\"root\",\n", + " passwd=\"R2d2&C3po!\",\n", + " database=\"nature\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "unlimited-profile", + "metadata": {}, + "source": [ + "### Read data from MySQL" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "celtic-integration", + "metadata": {}, + "outputs": [], + "source": [ + "df = pd.read_sql('SELECT * FROM fish', con=cnx)" + ] + }, + { + "cell_type": "markdown", + "id": "nearby-hunter", + "metadata": {}, + "source": [ + "### How the data looks like" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "posted-atlas", + "metadata": {}, + "outputs": [], + "source": [ + "df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "athletic-wagner", + "metadata": {}, + "source": [ + "### TODO, clean this data" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "retired-delivery", + "metadata": {}, + "outputs": [], + "source": [ + "df.total = pd.to_numeric(df.total, errors='coerce').fillna(0).astype(np.int64)\n", + "df.depth = pd.to_numeric(df.depth, errors='coerce').fillna(0).astype(np.int64)" + ] + }, + { + "cell_type": "markdown", + "id": "classical-frank", + "metadata": {}, + "source": [ + "### Plot some data" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "greenhouse-jamaica", + "metadata": {}, + "outputs": [], + "source": [ + "f, ax = plt.subplots(figsize=(6.5, 6.5))\n", + "sns.despine(f, left=True, bottom=True)\n", + "clarity_ranking = [\"I1\", \"SI2\", \"SI1\", \"VS2\", \"VS1\", \"VVS2\", \"VVS1\", \"IF\"]\n", + "sns.scatterplot(x=\"total\", y=\"depth\",\n", + " hue=\"total\", size=\"depth\",\n", + " palette=\"ch:r=-.2,d=.3_r\",\n", + " hue_order=clarity_ranking,\n", + " sizes=(1, 8), linewidth=0,\n", + " data=df, ax=ax);" + ] + }, + { + "cell_type": "markdown", + "id": "documentary-presence", + "metadata": {}, + "source": [ + "# Time to think about what data you want to use" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/heatwave-lakehouse-cdp/insights/images/activesession-create.png b/heatwave-lakehouse-cdp/insights/images/activesession-create.png new file mode 100644 index 00000000..4f64f9d6 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/activesession-create.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/datascience-active.png b/heatwave-lakehouse-cdp/insights/images/datascience-active.png new file mode 100644 index 00000000..3a49734f Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/datascience-active.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/datascience-creating.png b/heatwave-lakehouse-cdp/insights/images/datascience-creating.png new file mode 100644 index 00000000..3b8d8f68 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/datascience-creating.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/datascience-workshop.png b/heatwave-lakehouse-cdp/insights/images/datascience-workshop.png new file mode 100644 index 00000000..88aa8700 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/datascience-workshop.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-banner.jpg b/heatwave-lakehouse-cdp/insights/images/ds-banner.jpg new file mode 100644 index 00000000..ca1037a8 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-banner.jpg differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-conda-env-install.png b/heatwave-lakehouse-cdp/insights/images/ds-conda-env-install.png new file mode 100644 index 00000000..3210d772 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-conda-env-install.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-conda-install-ext.png b/heatwave-lakehouse-cdp/insights/images/ds-conda-install-ext.png new file mode 100644 index 00000000..663376a2 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-conda-install-ext.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-conda-install-next.png b/heatwave-lakehouse-cdp/insights/images/ds-conda-install-next.png new file mode 100644 index 00000000..f475386d Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-conda-install-next.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-conda-installation.png b/heatwave-lakehouse-cdp/insights/images/ds-conda-installation.png new file mode 100644 index 00000000..47e54d7b Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-conda-installation.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-conda-publish.png b/heatwave-lakehouse-cdp/insights/images/ds-conda-publish.png new file mode 100644 index 00000000..b506dc2f Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-conda-publish.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-conda-publishh.png b/heatwave-lakehouse-cdp/insights/images/ds-conda-publishh.png new file mode 100644 index 00000000..06b29552 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-conda-publishh.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-condapublish-objstore.png b/heatwave-lakehouse-cdp/insights/images/ds-condapublish-objstore.png new file mode 100644 index 00000000..debad599 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-condapublish-objstore.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-create.png b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-create.png new file mode 100644 index 00000000..e75803aa Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-create.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-creating.png b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-creating.png new file mode 100644 index 00000000..b0851f0a Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-creating.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-open.png b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-open.png new file mode 100644 index 00000000..156961d8 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook-open.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-create-notebook.png b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook.png new file mode 100644 index 00000000..ba616756 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-create-notebook.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-create-project-button.png b/heatwave-lakehouse-cdp/insights/images/ds-create-project-button.png new file mode 100644 index 00000000..128d6aa2 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-create-project-button.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-create-project.png b/heatwave-lakehouse-cdp/insights/images/ds-create-project.png new file mode 100644 index 00000000..5f036a1d Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-create-project.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-create-button.png b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-create-button.png new file mode 100644 index 00000000..be525e46 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-create-button.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-create.png b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-create.png new file mode 100644 index 00000000..0b3ee5d5 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-create.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-menu.png b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-menu.png new file mode 100644 index 00000000..6636ed2e Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-menu.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-review.png b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-review.png new file mode 100644 index 00000000..01ef5fa6 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-dynamic-group-review.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-execution-mode.png b/heatwave-lakehouse-cdp/insights/images/ds-execution-mode.png new file mode 100644 index 00000000..1292faa9 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-execution-mode.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-kernel.png b/heatwave-lakehouse-cdp/insights/images/ds-kernel.png new file mode 100644 index 00000000..493db201 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-kernel.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-menu.png b/heatwave-lakehouse-cdp/insights/images/ds-menu.png new file mode 100644 index 00000000..3da631c8 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-menu.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-navigation.png b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-navigation.png new file mode 100644 index 00000000..832aa01b Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-navigation.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-url.jpg b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-url.jpg new file mode 100644 index 00000000..a6f96360 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-url.jpg differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-url.png b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-url.png new file mode 100644 index 00000000..a6f96360 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment-url.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-model-deployment.png b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment.png new file mode 100644 index 00000000..907863eb Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-model-deployment.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-move.png b/heatwave-lakehouse-cdp/insights/images/ds-move.png new file mode 100644 index 00000000..8391fba9 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-move.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-navigate.png b/heatwave-lakehouse-cdp/insights/images/ds-navigate.png new file mode 100644 index 00000000..9243e9b7 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-navigate.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-exp.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-exp.png new file mode 100644 index 00000000..07a56780 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-exp.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-head.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-head.png new file mode 100644 index 00000000..665ed5a9 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-head.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-plot.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-plot.png new file mode 100644 index 00000000..a677eed5 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-plot.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-run.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-run.png new file mode 100644 index 00000000..6010b167 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook-run.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook.png new file mode 100644 index 00000000..78a055ed Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-fish-notebook.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-login.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-login.png new file mode 100644 index 00000000..befd9dc2 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-login.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-install.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-install.png new file mode 100644 index 00000000..c1e81890 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-install.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-launcher-new.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-launcher-new.png new file mode 100644 index 00000000..14002d57 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-launcher-new.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-new.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-new.png new file mode 100644 index 00000000..28bb9e6f Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal-new.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal.png new file mode 100644 index 00000000..3d77a287 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-terminal.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-notebook-upload.png b/heatwave-lakehouse-cdp/insights/images/ds-notebook-upload.png new file mode 100644 index 00000000..e47f5d76 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-notebook-upload.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-policies-create-button.png b/heatwave-lakehouse-cdp/insights/images/ds-policies-create-button.png new file mode 100644 index 00000000..6f60415f Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-policies-create-button.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-policies-create-review.png b/heatwave-lakehouse-cdp/insights/images/ds-policies-create-review.png new file mode 100644 index 00000000..5a38b35f Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-policies-create-review.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-policies-create.png b/heatwave-lakehouse-cdp/insights/images/ds-policies-create.png new file mode 100644 index 00000000..b5050dd6 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-policies-create.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-policies-menu.png b/heatwave-lakehouse-cdp/insights/images/ds-policies-menu.png new file mode 100644 index 00000000..075ee8ea Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-policies-menu.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-project.png b/heatwave-lakehouse-cdp/insights/images/ds-project.png new file mode 100644 index 00000000..b49c0fbf Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-project.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-publish-cmplete.png b/heatwave-lakehouse-cdp/insights/images/ds-publish-cmplete.png new file mode 100644 index 00000000..95971f43 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-publish-cmplete.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-runall.png b/heatwave-lakehouse-cdp/insights/images/ds-runall.png new file mode 100644 index 00000000..bb2c525a Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-runall.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-select-kernel.png b/heatwave-lakehouse-cdp/insights/images/ds-select-kernel.png new file mode 100644 index 00000000..ad155d02 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-select-kernel.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-select.png b/heatwave-lakehouse-cdp/insights/images/ds-select.png new file mode 100644 index 00000000..679d5a5d Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-select.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-session-kernel-activate.png b/heatwave-lakehouse-cdp/insights/images/ds-session-kernel-activate.png new file mode 100644 index 00000000..53d44055 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-session-kernel-activate.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-setting-close.png b/heatwave-lakehouse-cdp/insights/images/ds-setting-close.png new file mode 100644 index 00000000..8c1d49ee Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-setting-close.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ds-settings.png b/heatwave-lakehouse-cdp/insights/images/ds-settings.png new file mode 100644 index 00000000..bbc3e4f5 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ds-settings.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/identity-policies-menu.png b/heatwave-lakehouse-cdp/insights/images/identity-policies-menu.png new file mode 100644 index 00000000..c0c592bd Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/identity-policies-menu.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/md-model-endpoint.png b/heatwave-lakehouse-cdp/insights/images/md-model-endpoint.png new file mode 100644 index 00000000..153c5d1e Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/md-model-endpoint.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/ml-deployment.png b/heatwave-lakehouse-cdp/insights/images/ml-deployment.png new file mode 100644 index 00000000..90af54ef Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/ml-deployment.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/notebook-active.png b/heatwave-lakehouse-cdp/insights/images/notebook-active.png new file mode 100644 index 00000000..7cc0f21e Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/notebook-active.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/notebook-create-end.png b/heatwave-lakehouse-cdp/insights/images/notebook-create-end.png new file mode 100644 index 00000000..2798a257 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/notebook-create-end.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/notebook-create-start.png b/heatwave-lakehouse-cdp/insights/images/notebook-create-start.png new file mode 100644 index 00000000..ba1b5d9e Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/notebook-create-start.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/notebook-create.png b/heatwave-lakehouse-cdp/insights/images/notebook-create.png new file mode 100644 index 00000000..1be6d20a Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/notebook-create.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/notebook-interface.png b/heatwave-lakehouse-cdp/insights/images/notebook-interface.png new file mode 100644 index 00000000..85ad55de Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/notebook-interface.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/notebook-open.png b/heatwave-lakehouse-cdp/insights/images/notebook-open.png new file mode 100644 index 00000000..ea51a1c8 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/notebook-open.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/notebook-session-create.png b/heatwave-lakehouse-cdp/insights/images/notebook-session-create.png new file mode 100644 index 00000000..f2ec1998 Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/notebook-session-create.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/oracle-cloud-account-notebook-dashboard.png b/heatwave-lakehouse-cdp/insights/images/oracle-cloud-account-notebook-dashboard.png new file mode 100644 index 00000000..9997be1b Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/oracle-cloud-account-notebook-dashboard.png differ diff --git a/heatwave-lakehouse-cdp/insights/images/workshop-notebook.png b/heatwave-lakehouse-cdp/insights/images/workshop-notebook.png new file mode 100644 index 00000000..130a5cba Binary files /dev/null and b/heatwave-lakehouse-cdp/insights/images/workshop-notebook.png differ diff --git a/heatwave-lakehouse-cdp/insights/insights.md b/heatwave-lakehouse-cdp/insights/insights.md new file mode 100644 index 00000000..d3be7b5b --- /dev/null +++ b/heatwave-lakehouse-cdp/insights/insights.md @@ -0,0 +1,108 @@ +# Build and Deploy the custom Machine learning model to evaluate the probablity of asset failure + +![Data Science](images/ds-banner.jpg) + +### Introduction + +In this task, we will be using already created compute instance and will be using the training data set +to build and Deploy a predictive ML model on OCI platform which will identify the probablity of failure of a sensor device . The ML API URL once generated will be used by the Fn function code to evaluate the probablity of device when exposed to Live sensor datasets from stereaming layer . + + +_Estimated Lab Time_: 30 Minutes + +### Objectives + +In this section, you will: + + +- Build and Deploy a custom ML model on Compute VM to evaluate the device failure probability. + + +### Prerequisites + +- All previous sections have been successfully completed. + +## +## Task 1: Open the port + +1. Login to your same compute machine as opc, which we created earlier: + + ``` + sudo firewall-cmd --zone=public --add-port=5000/tcp --permanent + ``` + + +2. Check the python version: + + ``` + python3 --version + ``` + Output: Python 3.8.17 + + +3. Install the pip + + ``` + sudo dnf install python3-pip + ``` + +4. Install virtualenv + + ``` + pip install virtualenv + ``` + +## Task 2: Create Virtual Environment and Install Dependencies + +1. Create a Virtual Environment + + ``` + virtualenv datasciecne_venv + ``` +2. Activate the Virtual Environment + + ``` + source datasciecne_venv/bin/activate + ``` + +3. Installing Required Packages + + ``` + pip install flask pandas scikit-learn joblib + ``` + +4. Creating a Directory + + ``` + mkdir /home/opc/model + ``` +5. Changing to the New Directory + + ``` + cd /home/opc/model + ``` +6. Download the Lab Files and scp the files(devices.csv & model_deployement.py) from Lab3 to /home/opc/model + + ``` + ls + ``` + + output : + + devices.csv model_deployement.py + +7. Now run the ML code in virtual environment and keep this running in the terminal: + + ``` + python3 model_deployement.py + ``` +8. ML Endpoint URL : Copy the generated ML Model API endpoint which will be used in the Fn code. + + ``` + http://:5000/predict + ``` +You may now **proceed to the next lab** +## Acknowledgements +* **Author** - Biswanath Nanda, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect,Sharmistha das ,Master Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Biswanath Nanda, November 2024 \ No newline at end of file diff --git a/heatwave-lakehouse-cdp/introduction/images/architectturehw.png b/heatwave-lakehouse-cdp/introduction/images/architectturehw.png new file mode 100644 index 00000000..2705e0fe Binary files /dev/null and b/heatwave-lakehouse-cdp/introduction/images/architectturehw.png differ diff --git a/heatwave-lakehouse-cdp/introduction/images/data-sets.png b/heatwave-lakehouse-cdp/introduction/images/data-sets.png new file mode 100644 index 00000000..aa4fcd8c Binary files /dev/null and b/heatwave-lakehouse-cdp/introduction/images/data-sets.png differ diff --git a/heatwave-lakehouse-cdp/introduction/introduction.md b/heatwave-lakehouse-cdp/introduction/introduction.md new file mode 100644 index 00000000..aea708c0 --- /dev/null +++ b/heatwave-lakehouse-cdp/introduction/introduction.md @@ -0,0 +1,59 @@ +# Introduction + +### MySQL HeatWave - Cloud Data Platform Architecture + + ![heatwave](images/architectturehw.png) + +### About this Workshop + +The workshop focuses on demonstrating how Oracle Manufacturing Lakehouse helps the business in monitoring the Operational matrices and optimizing the supply chain using Oracle Cloud Data Platform. The Workshop has been designed to get a hands-on experience in Manufacturing Lakehouse and learn to leverage the next-age Data Platform services for customer demonstrations, designing POC and as an implementation accelerator for MySQL Heatwave Lakehouse. + +A data Lakehouse is a modern, open architecture that enables you to store, understand, and analyze all your data. It combines the power and richness of data warehouses with the breadth and flexibility of the most popular open-source data technologies you use today. + +### Dataset Brief + + The below figure shows the feature vectors of manufacturing sensor data streams .The events are being generated by the producer code hosted in a compute VM. A single event contains features such as timestamp ,asset\_id,asset\_type ,process\_temp,rotation speed ,torque and tool\_wear . Based on the historical data sets and occurrence of failures, we have built a training data set which is used as a source for the ML model built on OCI. + + ![Data set](images/data-sets.png) + +_Estimated Workshop Time_: 6 Hours + +### Objectives + +In this workshop, you will learn how to: + +* Develop end to end OCI Cloud Data Platform using MySQL HW +* Provision OCI Streaming Service and OCI Connector Hub - for ingesting streaming work loads +* Develop Custom Machine Learning Model to predict device failures +* Provision OCI Functions +* Provision HeatWave cluster on OCI +* Provision OCI Data FLow for bulk data inserts +* Load Sample data into HeatWave Cluster +* Provision Oracle Analytics Cloud and build Dashboards +* Provision OCI Data Catalog + +### About MySQL Heatwave + +HeatWave is a massively parallel, high performance, in-memory query accelerator that accelerates MySQL performance by orders of magnitude for analytics workloads, mixed workloads, and machine learning. + +HeatWave consists of a MySQL DB System and HeatWave nodes. Analytics queries that meet certain prerequisites are automatically offloaded from the MySQL DB System to the HeatWave Cluster for accelerated processing. With a HeatWave Cluster, you can run online transaction processing (OLTP), online analytical processing (OLAP), and mixed workloads from the same MySQL database without requiring extract, transfer, and load (ETL), and without modifying your applications. + +MySQL HeatWave Lakehouse processes data in a variety of file formats, such as CSV, Parquet, Avro, and exports from other databases. You can query data in object storage and, optionally, combine it with transactional data in MySQL databases. Applications can use large language models to interact with HeatWave Lakehouse in natural language. Data loaded into the HeatWave cluster for processing is automatically transformed into the HeatWave in-memory format, and object storage data is not copied to the MySQL database. + +### Prerequisites + +This lab assumes you have: + +* You have an Oracle account. + +## Livelab Files location + + Download [`MYSQLLakehouse_labfiles.zip`](https://objectstorage.us-ashburn-1.oraclecloud.com/p/RPka_orWclfWJmKN3gTHfEiv-uPckBJTZ3FV0sESZ3mm3PDCQcVDCT-uM2dsJNGf/n/orasenatdctocloudcorp01/b/MYSQLLakehouse_labfiles/o/MYSQLLakehouse_labfiles.zip) and save to a folder on your laptop or workstation. + +## Acknowledgements + +* **Author** - Bhushan Arora - Principal Cloud Architect, Biswanath Nanda - Master Principal Cloud Architect (North America Cloud Infrastructure - Engineering Team) +* **Lead By** - Bhushan Arora - Principal Cloud Architect +* **Contributors** - Biswanath Nanda - Master Principal Cloud Architect, Bhushan Arora - Principal Cloud Architect , Sharmistha das - Master Principal Cloud Architect, Lovelesh Saxena - Principal Cloud Architect +* **Supported By** - Bhaskar Sudarshan - Director, Balasubramanian Ramamoorthy - Director, Anviksha Shukla - Manager +* **Last Updated By/Date** - Bhushan Arora, November 2024 diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/Picture 1.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/Picture 1.png new file mode 100644 index 00000000..6c2eeb4c Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/Picture 1.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/addusers.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/addusers.png new file mode 100644 index 00000000..61d800b9 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/addusers.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/addusertodfadmingroup.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/addusertodfadmingroup.png new file mode 100644 index 00000000..5b37a8c4 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/addusertodfadmingroup.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/alter-table-load.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/alter-table-load.png new file mode 100644 index 00000000..e4bcf9ff Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/alter-table-load.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/bucket-details.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/bucket-details.png new file mode 100644 index 00000000..17ea5c3e Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/bucket-details.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/bucket-three-dots.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/bucket-three-dots.png new file mode 100644 index 00000000..92d103c6 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/bucket-three-dots.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-one.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-one.png new file mode 100644 index 00000000..6247f30e Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-one.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main copy.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main copy.png new file mode 100644 index 00000000..15a5c898 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main copy.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main.png new file mode 100644 index 00000000..4437273d Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main_rem.textClipping b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main_rem.textClipping new file mode 100644 index 00000000..f195ac84 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main_rem.textClipping differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main_remove.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main_remove.png new file mode 100644 index 00000000..caac71b8 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-table-output-main_remove.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-two.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-two.png new file mode 100644 index 00000000..bf05958c Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/create-two.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/createadadminpolicy.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/createadadminpolicy.png new file mode 100644 index 00000000..62fa09fe Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/createadadminpolicy.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/createbucket001.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/createbucket001.png new file mode 100644 index 00000000..67c7ba62 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/createbucket001.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/dataflowadminuser.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/dataflowadminuser.png new file mode 100644 index 00000000..b8e17793 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/dataflowadminuser.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-app-executor.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-app-executor.png new file mode 100644 index 00000000..c957fda5 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-app-executor.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-app-selection.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-app-selection.png new file mode 100644 index 00000000..88d02600 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-app-selection.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-bucket-01.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-bucket-01.png new file mode 100644 index 00000000..caa80b6c Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-bucket-01.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-create-app.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-create-app.png new file mode 100644 index 00000000..30321b69 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-create-app.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-groups001.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-groups001.png new file mode 100644 index 00000000..d9171d3e Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-groups001.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-navigate.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-navigate.png new file mode 100644 index 00000000..8b758599 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-navigate.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-object-store01.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-object-store01.png new file mode 100644 index 00000000..7ec05639 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-object-store01.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-usergroup.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-usergroup.png new file mode 100644 index 00000000..e365a44a Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-usergroup.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-warehouse.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-warehouse.png new file mode 100644 index 00000000..e17eec3e Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/df-warehouse.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/dfcreateuserspolicies.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/dfcreateuserspolicies.png new file mode 100644 index 00000000..22db232e Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/dfcreateuserspolicies.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/heatwave-load-op1.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/heatwave-load-op1.png new file mode 100644 index 00000000..21744366 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/heatwave-load-op1.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/heatwave-load-op2.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/heatwave-load-op2.png new file mode 100644 index 00000000..95b88083 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/heatwave-load-op2.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/list-par-file.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/list-par-file.png new file mode 100644 index 00000000..637e2e60 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/list-par-file.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/log-create-table-output.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/log-create-table-output.png new file mode 100644 index 00000000..54fb4532 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/log-create-table-output.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/par-with-prefix-object-listing.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/par-with-prefix-object-listing.png new file mode 100644 index 00000000..faa96060 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/par-with-prefix-object-listing.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/policies-01.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/policies-01.png new file mode 100644 index 00000000..3e3cce33 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/policies-01.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/run-app.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/run-app.png new file mode 100644 index 00000000..df4f4aa2 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/run-app.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/sample-five-rows.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/sample-five-rows.png new file mode 100644 index 00000000..3128ee42 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/sample-five-rows.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/script-chnage.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/script-chnage.png new file mode 100644 index 00000000..ba0ea2ea Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/script-chnage.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/script-upload.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/script-upload.png new file mode 100644 index 00000000..0a138e7e Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/script-upload.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-db-list.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-db-list.png new file mode 100644 index 00000000..be096a53 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-db-list.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-load-table-examples.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-load-table-examples.png new file mode 100644 index 00000000..a36e61d7 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-load-table-examples.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-options.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-options.png new file mode 100644 index 00000000..996a9273 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/set-options.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/images/spark-code.png b/heatwave-lakehouse-cdp/load-into-lakehouse/images/spark-code.png new file mode 100644 index 00000000..011d5305 Binary files /dev/null and b/heatwave-lakehouse-cdp/load-into-lakehouse/images/spark-code.png differ diff --git a/heatwave-lakehouse-cdp/load-into-lakehouse/load-into-lakehouse.md b/heatwave-lakehouse-cdp/load-into-lakehouse/load-into-lakehouse.md new file mode 100644 index 00000000..512eaaa4 --- /dev/null +++ b/heatwave-lakehouse-cdp/load-into-lakehouse/load-into-lakehouse.md @@ -0,0 +1,135 @@ + +## Load datasets from demo-events-silver-mysqlhw to MySQL HW + +### Introduction + +The labs shows how to load the datasets from demo-events-silver-mysqlhw to MYSQL HW +- Run Autoload to infer the schema and estimate capacity +- Load complete T\_AGGR\_DEVICE\_PRED\_HOUR table from Object Store into MySQL HeatWave + + +_Estimated Lab Time_: 30 mins + +### Objectives + +In this lab you will learn about how to run Autoload to infer the data + + +- Run Autoload to infer the schema +- Load the T\_AGGR\_DEVICE\_PRED\_HOUR table from Object Store into MySQL HeatWave. + + + +### Prerequisites + +Before you Begin with Data Flow lab, you must have: + +- Previous Labs have been completed +- We will use the PAR URL Created in previous lab for the bucket e2e-demo-events-silver-mysqlhw + +## Task 1: Run Autoload to infer the schema and estimate capacity required to upload the aggregate device data + +1. Aggregated data information is in the csv files in the object store for which we have created a PAR URL in the earlier task. Now run the following commands: + +2. This sets the schema, in which we will load table data into. + + ```bash + SET @db_list = '["MFG_SENSOR"]'; + ``` + ![autopilot set dblist example](./images/set-db-list.png "autopilot set dblist example") + + +3. This sets the parameters for the table name we want to load data into and other information about the source file in the object store. Substitute the **PAR URL** below with the one you generated in the previous **Lab 6(a) Task 3**: + ```bash + SET @dl_tables = '[{ + "db_name": "MFG_SENSOR", + "tables": [{ + "table_name": "T_AGGR_DEVICE_PRED_HOUR", + "dialect": + { + "format": "csv", + "field_delimiter": ",", + "record_delimiter": "\\n" + }, + "file": [{"par": "PAR URL"}] + }] }]'; + ``` + Be sure to include the PAR URL inside the quotes " " and above should look like as in following: + + ![autopilot set table example](./images/set-load-table-examples.png "autopilot set table example") + +4. This command populates all the options needed by Autoload: + + ```bash + SET @options = JSON_OBJECT('mode', 'dryrun', 'policy', 'disable_unsupported_columns', 'external_tables', CAST(@dl_tables AS JSON)); + ``` + ![autopilot set option](./images/set-options.png "autopilot set option example") + +5. Run this Autoload command: + + ```bash + CALL sys.heatwave_load(@db_list, @options); + ``` + + +6. Once Autoload completes running, its output has several pieces of information: + - a. Whether the table exists in the schema you have identified. + - b. Auto schema inference determines the number of columns in the table. + - c. Auto schema sampling samples a small number of rows from the table and determines the number of rows in the table and the size of the table. + - d. Auto provisioning determines how much memory would be needed to load this table into HeatWave and how much time loading this data take. + + ![Load Script](./images/heatwave-load-op1.png "load script dryrun") + + ![Load Script log](./images/heatwave-load-op2.png "load script log dryrun") + +7. Autoload also generated a statement like the one below. Execute this statement now. + + ```bash + SELECT log->>"$.sql" AS "Load Script" FROM sys.heatwave_autopilot_report WHERE type = "sql" ORDER BY id; + ``` + + Output of above command: + + ![log table output](./images/log-create-table-output.png "log table output dryrun") + + +8. The execution result contains the SQL statements needed to create the table. As there was no header in the csv file generated, now execute the following **CREATE TABLE** command by replacing your **PAR Value**. In following create table we have mentioned the required column name for this lab. + + **NOTE: Ensure to Replace you PAR VALUE after copying the following command** + + + ```bash + CREATE TABLE `MFG_SENSOR`.`T_AGGR_DEVICE_PRED_HOUR`( `year` year NOT NULL, `month` tinyint unsigned NOT NULL, `Day` tinyint unsigned NOT NULL, `Hour` tinyint unsigned NOT NULL, `device_id` varchar(6) NOT NULL COMMENT 'RAPID_COLUMN=ENCODING=VARLEN', `prod_type` varchar(1) NOT NULL COMMENT 'RAPID_COLUMN=ENCODING=VARLEN', `air_temp_mean` decimal(17,14) NOT NULL, `air_temp_max` decimal(4,1) NOT NULL, `air_temp_min` decimal(4,1) NOT NULL, `device_temp_mean` decimal(17,14) NOT NULL, `device_temp_max` decimal(4,1) NOT NULL, `device_temp_min` decimal(4,1) NOT NULL, `rotn_speed_mean` decimal(17,13) NOT NULL, `rotn_speed_max` decimal(5,1) NOT NULL, `rotn_speed_min` decimal(5,1) NOT NULL, `torque_mean` decimal(17,15) NOT NULL, `torque_max` decimal(3,1) NOT NULL, `torque_min` decimal(3,1) NOT NULL, `tool_wear_mean` decimal(17,14) NOT NULL, `tool_wear_max` decimal(4,1) NOT NULL, `tool_wear_min` decimal(4,1) NOT NULL, `prob_mean` decimal(20,19) NOT NULL, `prob_max` decimal(4,3) NOT NULL, `prob_min` decimal(4,3) NOT NULL, `prob_freq_over60` tinyint unsigned NOT NULL, `prob_freq_over90` tinyint unsigned NOT NULL) ENGINE=lakehouse SECONDARY_ENGINE=RAPID ENGINE_ATTRIBUTE='{"file": [{"par": "PAR_URL"}], "dialect": {"format": "csv", "field_delimiter": ",", "record_delimiter": "\\n"}}'; + ``` + + +9. The above create command and result should look like this + + ![Create Table Main](./images/create-table-output-main.png "create table main") + +## Task 2: Load the data from Object Store into MySQL HeatWave Table + + +1. Now load the data from the Object Store into the table + + ```bash + ALTER TABLE /*+ AUTOPILOT_DISABLE_CHECK */ `MFG_SENSOR`.`T_AGGR_DEVICE_PRED_HOUR` SECONDARY_LOAD; + ``` + ![Load Table](./images/alter-table-load.png "load aggr table") + + +2. View a sample of the data in the table. + + ```bash + select * from T_AGGR_DEVICE_PRED_HOUR limit 5; + ``` + ![Sample Result](./images/sample-five-rows.png "Sample Result") + + + +You may now **proceed to the next lab** +## Acknowledgements +* **Author** - Bhushan Arora, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect +* **Last Updated By/Date** - Bhushan Arora, November 2024 + diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/Saurabh Architecture.png b/heatwave-lakehouse-cdp/oac-provisioning/images/Saurabh Architecture.png new file mode 100644 index 00000000..afd06ea7 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/Saurabh Architecture.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate-connection.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate-connection.png new file mode 100644 index 00000000..c0f32b1d Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate-connection.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate-data.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate-data.png new file mode 100644 index 00000000..fda55945 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate-data.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate.png new file mode 100644 index 00000000..f52c1409 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-navigate.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-oac.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-oac.png new file mode 100644 index 00000000..82e8017d Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics-oac.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashbaord1.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashbaord1.png new file mode 100644 index 00000000..931ba222 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashbaord1.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashbaord2.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashbaord2.png new file mode 100644 index 00000000..2e1847ef Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashbaord2.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashboardreload.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashboardreload.png new file mode 100644 index 00000000..430a825f Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_dashboardreload.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_mysqlconnection.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_mysqlconnection.png new file mode 100644 index 00000000..58aa3df6 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_mysqlconnection.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_realtime.png b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_realtime.png new file mode 100644 index 00000000..369135d7 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/analytics_realtime.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/banner.jpg b/heatwave-lakehouse-cdp/oac-provisioning/images/banner.jpg new file mode 100644 index 00000000..0334013d Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/banner.jpg differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/banner.png b/heatwave-lakehouse-cdp/oac-provisioning/images/banner.png new file mode 100644 index 00000000..739aa066 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/banner.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac-creating.PNG b/heatwave-lakehouse-cdp/oac-provisioning/images/oac-creating.PNG new file mode 100644 index 00000000..2d30497f Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac-creating.PNG differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_create_button.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_create_button.png new file mode 100644 index 00000000..4838879e Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_create_button.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_creating.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_creating.png new file mode 100644 index 00000000..9ed01fe5 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_creating.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_dashbaord.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_dashbaord.png new file mode 100644 index 00000000..e0b7127a Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_dashbaord.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_form.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_form.png new file mode 100644 index 00000000..24aaab96 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_form.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_import.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_import.png new file mode 100644 index 00000000..44a98476 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_import.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_cloud_account_name.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_cloud_account_name.png new file mode 100644 index 00000000..4138ad2e Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_cloud_account_name.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_idp.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_idp.png new file mode 100644 index 00000000..bf3c829d Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_idp.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_user_password.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_user_password.png new file mode 100644 index 00000000..7b60c632 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_login_user_password.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_menu.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_menu.png new file mode 100644 index 00000000..c8627651 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_menu.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oac_profile_federated.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_profile_federated.png new file mode 100644 index 00000000..e54d8d03 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oac_profile_federated.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oacpac1.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oacpac1.png new file mode 100644 index 00000000..537c3ab2 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oacpac1.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/images/oacpac2.png b/heatwave-lakehouse-cdp/oac-provisioning/images/oacpac2.png new file mode 100644 index 00000000..f0df44e6 Binary files /dev/null and b/heatwave-lakehouse-cdp/oac-provisioning/images/oacpac2.png differ diff --git a/heatwave-lakehouse-cdp/oac-provisioning/oac-provisioning.md b/heatwave-lakehouse-cdp/oac-provisioning/oac-provisioning.md new file mode 100644 index 00000000..b77b28de --- /dev/null +++ b/heatwave-lakehouse-cdp/oac-provisioning/oac-provisioning.md @@ -0,0 +1,105 @@ +# Provision Analytics Cloud + + + +### Introduction + +During this lab you will deploy an Oracle Analytics Cloud instance on Oracle Cloud Infrastructure. + +_Estimated Lab Time_: 15 minutes (OAC provisioning time may vary) + +### Objectives + +In this lab, you will: + +- Login as a federated user. +- Create an Oracle Analytics Cloud Instance. + +### Prerequisites + +- Oracle Free Trial Account. + +## +## Task 1: Create an Oracle Analytics Cloud (OAC) Instance + +1. Return to the Home Page and go to the **Menu** > **Analytics & AI** > **Analytics Cloud**. + + ![OAC Menu](images/analytics-oac.png) + +2. Create the OAC instance in the Demo compartment and click **Create Instance**. + + ![OAC Create Button](images/oac_create_button.png) + +3. (a) Fill the web form with the following information and click **Create**: + + - **Compartment**: `e2e-demo-specialist-eng` compartment, unless you have permissions and experience selecting a different one. + - **Name**: `WorkshopAnalytics` + - **Description**: `Analytics Instance for Data Science workshop` + - **Capacity**: `OCPU` and `1 - Non Production` + - **License Type**: `License Included` and `Enterprise Analytics` + + ![OAC Form](images/oac_form.png) + + Your Analytics Instance will start provisioning. + + ![pic3](images/oac_creating.png) + + (b) Now Configure Private Access Channel of OAC. + Go to the private Access Channel under Resources and Configure Private Access Channel: + + ![oacpac_1](images/oacpac1.png) + + (c) Fill the details and click on configure button: + + ![oacpac_2](images/oacpac2.png) + +4. Deploy the DVX file to OAC project + + - Collect the OAC dashboard extract file (.DVA) file from this location.[MYSQLLakehouse_labfiles\Lab5\OAC] + - Upload these files to OAC console and Visualize Dashboards . + - Predictive Streaming Analytics.dva + - **DVA password - Admin123 + + ![DVA File Deployment](images/oac_dashbaord.png) + ![DVA File Deployment](images/oac_import.png) + +5. Changing the connection for MYSQL Heatwave for Live Dashboard + + - Navigate to `OAC Home page` and click on Navigate `option` -> `Data` as shown below. + + ![Navigation](images/analytics-navigate.png) + + - Click on `Data` + + ![Connection Navigation](images/analytics-navigate-data.png) + + - In the below window click on `Data` then on the existing LiveDs, click on vertical dots and then click on Inspect option to re-point the existing MYSQLHW connection to the MYSQL HW instance in your Demo environment. + + ![Connection Navigation](images/analytics-navigate-connection.png) + + - Click on `Inspect` and edit the MYSQL connections and click on Save. + + ![Connection Navigation](images/analytics_mysqlconnection.png) + +6. Navigate to the `Workbooks and Reports` and double click on the `Predictive Streaming Analytics` icon and reload all the demo Dashboard. + + - Live Dashboard - Contains the live data being sourced from the SQL table to OAC. + + ![Live Dashboard](images/analytics_realtime.png) + + - There are a couple more static dashboards for your reference which shows visualization on different sensor KPI and fabricated Operational datasets.These dashboards are designed for your reference only. + + ![Live Dashboard](images/analytics_dashbaord1.png) + ![Live Dashboard](images/analytics_dashbaord2.png) + +- NOTE: Provisioning an Oracle Analytics Cloud instance can take from 10 (most likely) to 40 minutes. + + We will get back to your Oracle Analytics Cloud instance later in the workshop. + +You may now **proceed to the next lab** + +## Acknowledgements + +* **Author** - Biswanath Nanda, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect,Sharmistha das ,Master Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Biswanath Nanda, November 2024 \ No newline at end of file diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/catalog-instance.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/catalog-instance.png new file mode 100644 index 00000000..704757b7 Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/catalog-instance.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/copy-ocid.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/copy-ocid.png new file mode 100644 index 00000000..dc547eda Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/copy-ocid.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-asset-hw.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-asset-hw.png new file mode 100644 index 00000000..8958ded6 Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-asset-hw.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-asset-objectstore.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-asset-objectstore.png new file mode 100644 index 00000000..79fcdd7a Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-asset-objectstore.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-profile.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-profile.png new file mode 100644 index 00000000..599e2cb3 Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-profile.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-profile_create_asset.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-profile_create_asset.png new file mode 100644 index 00000000..3421cf59 Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/data-catalog-profile_create_asset.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/hostname-hw.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/hostname-hw.png new file mode 100644 index 00000000..f6d8c118 Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/hostname-hw.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/navigate-catalog.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/navigate-catalog.png new file mode 100644 index 00000000..16260e87 Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/navigate-catalog.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/seach-operation.png b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/seach-operation.png new file mode 100644 index 00000000..c84b9f25 Binary files /dev/null and b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/images/seach-operation.png differ diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/index.html b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/index.html new file mode 100644 index 00000000..aaac634b --- /dev/null +++ b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/index.html @@ -0,0 +1,62 @@ + + + + + + + + + Oracle LiveLabs + + + + + + + + + + + + +
+
+
+
+
+
+
+
+ + + + + diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/manifest.json b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/manifest.json new file mode 100644 index 00000000..a50c391f --- /dev/null +++ b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/manifest.json @@ -0,0 +1,14 @@ +{ + "workshoptitle": "LiveLabs Sprints", + "help": "livelabs-help-db_us@oracle.com", + "tutorials": [ + { + "title": "How do I find a Data Catalog instance OCID?", + "type": "freetier", + "description": " ", + "filename": "./../sprint-find-data-catalog-ocid/sprint-find-data-catalog-ocid.md" + } + ], + "task_type": "Sections", + "hide_button": "true" +} diff --git a/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/sprint-find-data-catalog-ocid.md b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/sprint-find-data-catalog-ocid.md new file mode 100644 index 00000000..0d462ee1 --- /dev/null +++ b/heatwave-lakehouse-cdp/sprint-find-data-catalog-ocid/sprint-find-data-catalog-ocid.md @@ -0,0 +1,58 @@ +# Provision the OCI Data Catalog and connect to MYSQL HW cluster and Objectstorage + +### Introduction + +In this lab you will be creating a Data catalog instance and connecting it to MYSQL HW cluster and Objectstore. In this workshop data catalog is used to browse metadata across the Lakehouse. + +Oracle Cloud Infrastructure (OCI) Data Catalog is a metadata management service that helps data professionals discover data and support data governance. Designed specifically to work well with the Oracle ecosystem, it provides an inventory of assets, a business glossary, and a common metastore for data lakes. + +_Estimated Lab Time_: 15 minutes + +### Objectives +- Create the Data Catalog +- Create assets and Harvest metadata +- Browse for metadata + +### Prerequisites +* An Oracle Cloud Account. + +## +## Task 1: Create a data catalog + +1. Sign in to the Oracle Cloud Infrastructure Console. + +2. Open the **Navigation** menu and click **Analytics & AI**. Under **Data Lake**, click **Data Catalog**. + +3. On the **Data Catalog Overview** page, click **Go to Data Catalogs**. + ![data ctalog navigate](./images/navigate-catalog.png" ") + +4. Click on **Create Data Catalog** as shown in the below snapshot + + Name of Catalog - LakehouseCatalog + ![data catalog navigate](./images/catalog-instance.png" ") + +## Task 2: Create assets and Harvest metadata + +1. Harvest metadata from Oracle Object Storage. + Navigate to Create new Data Asset and select on ObjectStorage . Connect to the demo-events-raw-mysqlhw and demo-events-silver-mysqlhw + ![Create Table Main](./images/data-catalog-asset-objectstore.png "MYSQL Lakehouse Host name") + +## Task 3: Search Metadata + +1. Search and discover your data + The metadata across different storage location and types can be searched from the OCI Data Catalog. + ![Create Table Main](./images/seach-operation.png "Search Operations.") + +## Learn More + +* [Signing In to the Console](https://docs.cloud.oracle.com/en-us/iaas/Content/GSG/Tasks/signingin.htm). +* [Get Started with Data Catalog](https://docs.oracle.com/en-us/iaas/data-catalog/using/index.htm) +* [Data Catalog Overview](https://docs.oracle.com/en-us/iaas/data-catalog/using/overview.htm) +* [Oracle Cloud Infrastructure Documentation](https://docs.oracle.com/en-us/iaas/Content/GSG/Concepts/baremetalintro.htm) + +You may now **proceed to the next lab** + +## Acknowledgements +* **Author** - Biswanath Nanda, Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect,Sharmistha das ,Master Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Biswanath Nanda, November 2024 \ No newline at end of file diff --git a/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/bucket.png b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/bucket.png new file mode 100644 index 00000000..674fef4f Binary files /dev/null and b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/bucket.png differ diff --git a/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/compartment-create.png b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/compartment-create.png new file mode 100644 index 00000000..5fbcc697 Binary files /dev/null and b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/compartment-create.png differ diff --git a/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/compartment-create1.png b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/compartment-create1.png new file mode 100644 index 00000000..da0ecaf5 Binary files /dev/null and b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/compartment-create1.png differ diff --git a/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/obj-storage-01.png b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/obj-storage-01.png new file mode 100644 index 00000000..1a9a6abe Binary files /dev/null and b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/images/obj-storage-01.png differ diff --git a/heatwave-lakehouse-cdp/upload-demo-data-objectstore/upload-demo-data-objectstore.md b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/upload-demo-data-objectstore.md new file mode 100644 index 00000000..c46595b2 --- /dev/null +++ b/heatwave-lakehouse-cdp/upload-demo-data-objectstore/upload-demo-data-objectstore.md @@ -0,0 +1,90 @@ +# Create Compartment and Required Buckets for the Workshop + +### Introduction + +In this lab we be creating the Demo compartment and OCI Object-storage buckets used for the workshop + +_Estimated Lab Time_: 10 minutes + +### Objectives + +In this lab, you will be guided through the following tasks: + +- Create Compartment +- Create Required Buckets +- Setting Up Object Stroage + +### Prerequisites + +- An Oracle Trial or Paid Cloud Account + + +## +## Task 1: Create Demo Compartment + +1. Click the **Navigation Menu** in the upper left, navigate to **Identity & Security** and select **Compartments**. + +2. On the Compartments page, click **Create Compartment**. + +3. In the Create Compartment dialog box, complete the following fields: + + Name: + + ```bash + e2e-demo-specialist-eng + ``` + + Description: + + ```bash + Compartment for end to end workshop + ``` + *** You can use any other compartment name based up on your needs .Please ensure to create all the OCI services under the same compartment and region which you created in the step No 3. +4. The **Parent Compartment** should be **root** and click **Create Compartment** + ![VCN](./images/compartment-create.png "create the compartment") + +## Task 2: Create the following buckets in object storage before proceeding. +1. RAW Bucket : + Bucket Name - **demo-events-raw-mysqlhw** + + Bucket Purpose - The the streaming payloads will be continously written to this bucket . +2. Silver Bucket - **demo-events-silver-mysqlhw** + + Bucket Purpose - This bucket is used by the Data flow to write the aggregated data from RAW bucket . + +3. OCI Data flow buckets - **Dataflowbucket** and **dataflow-logs** + + Bucket Purpose - These buckets will be containing the PySpark script and data flow application logs respectively. + + +## Task 3: Object Store: Setting Up Storage +The below steps shows ,how to create the ObjectStorage buckets. In this task we will be creating the below ObjectStores . + +1. Before running any applications the below storage buckets needs to be created in the demo compartment. + + * From the OCI Services menu, click **Storage** and then click **Buckets** under Object Storage + + ![](images/obj-storage-01.png " ") + + * Click **Create Bucket** + ![](images/bucket.png " ") + + **NOTE:** Ensure the correct Compartment is selected under COMPARTMENT list + **NOTE** Ensure for the workshop make the below buckets ***PUBLIC*** .For customized set up the buckets can be marked private . +2. Bucket Details: + + * **Buckets for OCI Streaming Sink** + * **e2e-demo-events-raw-mysqlhw** - Store raw data set. + + * **Buckets for OCI Dataflow set up** + * **Dataflowbucket** - host the main.py file ( Spark Application file) + * **dataflow-logs** - Store the dataflow Application log + * **Bucket for storing Historical Data** + * **demo-events-silver-mysqlhw** -Store the Aggregated streaming data sets + +You may now **proceed to the next lab** + +## Acknowledgements +* **Author** - Biswanath Nanda, Master Principal Cloud Architect, North America Cloud Infrastructure - Engineering +* **Contributors** - Biswanath Nanda, Principal Cloud Architect,Bhushan Arora ,Principal Cloud Architect,Sharmistha das ,Master Principal Cloud Architect,North America Cloud Infrastructure - Engineering +* **Last Updated By/Date** - Biswanath Nanda, November 2024 \ No newline at end of file diff --git a/heatwave-lakehouse-cdp/workshops/freetier/index.html b/heatwave-lakehouse-cdp/workshops/freetier/index.html new file mode 100644 index 00000000..6acdb69d --- /dev/null +++ b/heatwave-lakehouse-cdp/workshops/freetier/index.html @@ -0,0 +1,62 @@ + + + + + + + + + Oracle LiveLabs + + + + + + + + + + + + +
+
+
+
+
+
+
+
+ + + + + diff --git a/heatwave-lakehouse-cdp/workshops/freetier/manifest.json b/heatwave-lakehouse-cdp/workshops/freetier/manifest.json new file mode 100644 index 00000000..a93ec970 --- /dev/null +++ b/heatwave-lakehouse-cdp/workshops/freetier/manifest.json @@ -0,0 +1,38 @@ +{ + "workshoptitle": "Get Started with MySQL-HeatWave and Feel the Difference", + "help": "livelabs-help-db_us@oracle.com", + "tutorials": [ + { + "title": "Introduction", + "description": "MySQL Database HeatWave Cloud Services Workshop for OCI Lab Introduction.", + "filename": "../../introduction/introduction.md" + }, + { + "title": "Getting Started", + "description": "This is the prerequisites for customers using Free Trial and Paid tenancies, and Always Free accounts (if applicable). The title of the lab and the Contents Menu title (the title above) match for Prerequisite lab. This lab is always first.", + "filename": "https://oracle-livelabs.github.io/common/labs/cloud-login/pre-register-free-tier-account.md" + }, + { + "title": "Lab 1: Create MySQL Database System in Oracle Cloud", + "description": "Create MySQL Database System first so that later we can enable HeatWave Cluster", + "filename": "../../create-mysql-database-system/create-mysql-database-system.md" + }, + { + "title": "Lab 2: Enable HeatWave Cluster in MySQL Database System", + "filename": "../../enable-heatwave-cluster/enable-heatwave-cluster.md" + }, + { + "title": "Lab 3: Upload Data into MySQL HeatWave Cluster Engine", + "filename": "../../upload-data-into-heatwave/upload-data-into-heatwave.md" + }, + { + "title": "Lab 4: Execute SQL Query to View the Performance Differences", + "filename": "../../execute-sql-queries/execute-sql-queries.md" + }, + { + "title": "Need Help?", + "description": "Solutions to Common Problems and Directions for Receiving Live Help", + "filename":"https://oracle-livelabs.github.io/common/labs/need-help/need-help-freetier.md" + } + ] +} diff --git a/heatwave-lakehouse-cdp/workshops/livelabs/index.html b/heatwave-lakehouse-cdp/workshops/livelabs/index.html new file mode 100644 index 00000000..77be06e3 --- /dev/null +++ b/heatwave-lakehouse-cdp/workshops/livelabs/index.html @@ -0,0 +1,61 @@ + + + + + + + + + Oracle LiveLabs + + + + + + + + + + + +
+
+
+
+
+
+
+
+ + + + + diff --git a/heatwave-lakehouse-cdp/workshops/livelabs/manifest.json b/heatwave-lakehouse-cdp/workshops/livelabs/manifest.json new file mode 100644 index 00000000..a27f4297 --- /dev/null +++ b/heatwave-lakehouse-cdp/workshops/livelabs/manifest.json @@ -0,0 +1,84 @@ +{ + "workshoptitle": "MySQL Heatwave Lakehouse: Realtime Asset Maintenance", + "help": "livelabs-help-db_us@oracle.com", + "tutorials": [ + { + "title": "Introduction", + "description": "Explore how Oracle MySQL Heatwave Lakehouse help the business in monitoring the Operational metrics and optimizing the supply chain using the Out of the box AI and ML capabilities of OCI. The workshop covers the exceptional features of MySQL Heatwave Lakehouse which helps customers to focus on data platform innovations.", + "filename": "../../introduction/introduction.md" + }, + { + "title": "Get Started", + "description": "This is the prerequisites for customers using Free Trial and Paid tenancies, and Always Free accounts (if applicable). The title of the lab and the Contents Menu title (the title above) match for Prerequisite lab. This lab is always first.", + "filename": "https://oracle-livelabs.github.io/common/labs/cloud-login/cloud-login.md" + }, + { + "title": "Lab 1: Create Demo Compartment and ObjectStore Buckets and upload Data", + "description": "Create Demo Compartment and ObjectStore Buckets and upload Data", + "filename": "../../upload-demo-data-objectstore/upload-demo-data-objectstore.md" + }, + { + "title": "Lab 2a: Create VCN and Compartment for workshop", + "description": "This lab walks you through user setup and configuration and puts the foundation in place to build the Demo on OCI", + "filename": "../../create-heatwave-vcn-db/create-heatwave-vcn-db.md" + }, + { + "title": "Lab 2b: Create Compute instance for workshop and configure MySQL Shell", + "description": "Create compute instance for workshop and configure MySQL Shell.", + "filename": "../../create-compute-with-mysqlsh/create-compute-with-mysqlsh.md" + }, + { + "title": "Lab 2c: Create MySQL Database System in Oracle Cloud", + "description": "Create MySQL Database System first so that later we can enable HeatWave Cluster", + "filename": "../../create-mysql-database-system/create-mysql-database-system.md" + }, + { + "title": "Lab 2d: Enable Lakehouse and Create MFG_SENSOR Database/Table in MySQL Database System", + "description": "Create a OCI Data Science notebook and Deploy the ML Model", + "filename": "../../enable-heatwave-cluster/enable-heatwave-cluster.md" + }, + { + "title": "Lab 3: Build ML model API to evalaute the probablity of device faiures", + "filename": "../../insights/insights.md" + }, + { + "title": "Lab 4a: Set up OCI Stream service & Execute Sensor Data Simulator", + "description": "set up OCI stream service Execute Sensor Data Simulator", + "filename": "../../create-stream/create-stream.md" + }, + { + "title": "Lab 4b: Setup Dynamic Group for function to manage stream/objects", + "description": "set up OCI stream service Execute Sensor Data Simulator", + "filename": "../../dg-policy/dg-policy.md" + }, + { + "title": "Lab 5: Set up and Deploy OCI function and connect to service connector Hub ", + "description": "Set up and Deploy OCI function and connect to service connector Hub", + "filename": "../../configuring-fn/configuring-fn.md" + }, + { + "title": "Lab 6a: Set up OCI Dataflow and deploy a Pyspark Application and load the datasets from demo-events-silver-mysqlhw to MYSQL HW", + "description": "Set up OCI Dataflow and deploy a Pyspark Application", + "filename": "../../dataflow/dataflow.md" + }, + { + "title": "Lab 6b: Load the datasets from demo-events-silver-mysqlhw to MYSQL HW", + "description": "Load the datasets from demo-events-silver-mysqlhw to MYSQL HW", + "filename": "../../load-into-lakehouse/load-into-lakehouse.md" + }, + { + "title": "Lab 7: Design and Develop OAC Dashboards ", + "filename": "../../oac-provisioning/oac-provisioning.md" + }, + { + "title": "Lab 8: Set up OCI Data Catalog ", + "filename": "../../sprint-find-data-catalog-ocid/sprint-find-data-catalog-ocid.md" + }, + { + "title": "Need Help?", + "description": "Solutions to Common Problems and Directions for Receiving Live Help", + "filename":"https://oracle-livelabs.github.io/common//labs/need-help/need-help-freetier.md" + } + ] +} +