This end-to-end (E2E) sample is based on the guide "Fine-Tune and Integrate Custom Phi-3 Models with Prompt Flow in Azure AI Studio" from the Microsoft Tech Community. It introduces the processes of fine-tuning, deploying, and integrating custom Phi-3 models with Prompt flow in Azure AI Studio. Unlike the E2E sample, "Fine-Tune and Integrate Custom Phi-3 Models with Prompt Flow", which involved running code locally, this tutorial focuses entirely on fine-tuning and integrating your model within the Azure AI / ML Studio.
In this E2E sample, you will learn how to fine-tune the Phi-3 model and integrate it with Prompt flow in Azure AI Studio. By leveraging Azure AI / ML Studio, you will establish a workflow for deploying and utilizing custom AI models. This E2E sample is divided into three scenarios:
Scenario 1: Set up Azure resources and Prepare for fine-tuning
Scenario 2: Fine-tune the Phi-3 model and Deploy in Azure Machine Learning Studio
Scenario 3: Integrate with Prompt flow and Chat with your custom model in Azure AI Studio
Here is an overview of this E2E sample.
-
Scenario 1: Set up Azure resources and Prepare for fine-tuning
-
Scenario 2: Fine-tune Phi-3 model and Deploy in Azure Machine Learning Studio
-
Scenario 3: Integrate with Prompt flow and Chat with your custom model in Azure AI Studio
-
Type azure machine learning in the search bar at the top of the portal page and select Azure Machine Learning from the options that appear.
-
Select + Create from the navigation menu.
-
Select New workspace from the navigation menu.
-
Perform the following tasks:
- Select your Azure Subscription.
- Select the Resource group to use (create a new one if needed).
- Enter Workspace Name. It must be a unique value.
- Select the Region you'd like to use.
- Select the Storage account to use (create a new one if needed).
- Select the Key vault to use (create a new one if needed).
- Select the Application insights to use (create a new one if needed).
- Select the Container registry to use (create a new one if needed).
-
Select Review + Create.
-
Select Create.
In this tutorial, you will learn how to fine-tune and deploy a Phi-3 model, using GPUs. For fine-tuning, you will use the Standard_NC24ads_A100_v4 GPU, which requires a quota request. For deployment, you will use the Standard_NC6s_v3 GPU, which also requires a quota request.
Note
Only Pay-As-You-Go subscriptions (the standard subscription type) are eligible for GPU allocation; benefit subscriptions are not currently supported.
-
Visit Azure ML Studio.
-
Perform the following tasks to request Standard NCADSA100v4 Family quota:
-
Select Quota from the left side tab.
-
Select the Virtual machine family to use. For example, select Standard NCADSA100v4 Family Cluster Dedicated vCPUs, which includes the Standard_NC24ads_A100_v4 GPU.
-
Select the Request quota from the navigation menu.
-
Inside the Request quota page, enter the New cores limit you'd like to use. For example, 24.
-
Inside the Request quota page, select Submit to request the GPU quota.
-
-
Perform the following tasks to request Standard NCSv3 Family quota:
- Select Quota from the left side tab.
- Select the Virtual machine family to use. For example, select Standard NCSv3 Family Cluster Dedicated vCPUs, which includes the Standard_NC6s_v3 GPU.
- Select the Request quota from the navigation menu.
- Inside the Request quota page, enter the New cores limit you'd like to use. For example, 24.
- Inside the Request quota page, select Submit to request the GPU quota.
To fine-tune and deploy your models, you must first create a User Assigned Managed Identity (UAI) and assign it the appropriate permissions. This UAI will be used for authentication during deployment
-
Type managed identities in the search bar at the top of the portal page and select Managed Identities from the options that appear.
-
Select + Create.
-
Perform the following tasks:
- Select your Azure Subscription.
- Select the Resource group to use (create a new one if needed).
- Select the Region you'd like to use.
- Enter the Name. It must be a unique value.
-
Select Review + create.
-
Select + Create.
-
Navigate to the Managed Identity resource that you created.
-
Select Azure role assignments from the left side tab.
-
Select +Add role assignment from the navigation menu.
-
Inside Add role assignment page, Perform the following tasks:
- Select the Scope to Resource group.
- Select your Azure Subscription.
- Select the Resource group to use.
- Select the Role to Contributor.
-
Select Save.
-
Type storage accounts in the search bar at the top of the portal page and select Storage accounts from the options that appear.
-
Select the storage account that associated with the Azure Machine Learning workspace that you created. For example, finetunephistorage.
-
Perform the following tasks to navigate to Add role assignment page:
- Navigate to the Azure Storage account that you created.
- Select Access Control (IAM) from the left side tab.
- Select + Add from the navigation menu.
- Select Add role assignment from the navigation menu.
-
Inside Add role assignment page, Perform the following tasks:
- Inside the Role page, type Storage Blob Data Reader in the search bar and select Storage Blob Data Reader from the options that appear.
- Inside the Role page, select Next.
- Inside the Members page, select Assign access to Managed identity.
- Inside the Members page, select + Select members.
- Inside Select managed identities page, select your Azure Subscription.
- Inside Select managed identities page, select the Managed identity to Manage Identity.
- Inside Select managed identities page, select the Manage Identity that you created. For example, finetunephi-managedidentity.
- Inside Select managed identities page, select Select.
-
Select Review + assign.
-
Type container registries in the search bar at the top of the portal page and select Container registries from the options that appear.
-
Select the container registry that associated with the Azure Machine Learning workspace. For example, finetunephicontainerregistry
-
Perform the following tasks to navigate to Add role assignment page:
- Select Access Control (IAM) from the left side tab.
- Select + Add from the navigation menu.
- Select Add role assignment from the navigation menu.
-
Inside Add role assignment page, Perform the following tasks:
- Inside the Role page, Type AcrPull in the search bar and select AcrPull from the options that appear.
- Inside the Role page, select Next.
- Inside the Members page, select Assign access to Managed identity.
- Inside the Members page, select + Select members.
- Inside Select managed identities page, select your Azure Subscription.
- Inside Select managed identities page, select the Managed identity to Manage Identity.
- Inside Select managed identities page, select the Manage Identity that you created. For example, finetunephi-managedidentity.
- Inside Select managed identities page, select Select.
- Select Review + assign.
To download the datasets needed for fine-tuning, you will set up a local environment.
In this exercise, you will
- Create a folder to work inside it.
- Create a virtual environment.
- Install the required packages.
- Create a download_dataset.py file to download the dataset.
-
Open a terminal window and type the following command to create a folder named finetune-phi in the default path.
mkdir finetune-phi -
Type the following command inside your terminal to navigate to the finetune-phi folder you created.
cd finetune-phi
-
Type the following command inside your terminal to create a virtual environment named .venv.
python -m venv .venv -
Type the following command inside your terminal to activate the virtual environment.
.venv\Scripts\activate.bat
Note
If it worked, you should see (.venv) before the command prompt.
-
Type the following commands inside your terminal to install the required packages.
pip install datasets==2.19.1
Note
Complete folder structure:
└── YourUserName
. └── finetune-phi
. └── download_dataset.py
-
Open Visual Studio Code.
-
Select File from the menu bar.
-
Select Open Folder.
-
Select the finetune-phi folder that you created, which is located at C:\Users\yourUserName\finetune-phi.
-
In the left pane of Visual Studio Code, right-click and select New File to create a new file named download_dataset.py.
In this exercise, you will run the download_dataset.py file to download the ultrachat_200k datasets to your local environment. You will then use this datasets to fine-tune the Phi-3 model in Azure Machine Learning.
In this exercise, you will:
- Add code to the download_dataset.py file to download the datasets.
- Run the download_dataset.py file to download datasets to your local environment.
-
Open the download_dataset.py file in Visual Studio Code.
-
Add the following code into download_dataset.py file.
import json import os from datasets import load_dataset def load_and_split_dataset(dataset_name, config_name, split_ratio): """ Load and split a dataset. """ # Load the dataset with the specified name, configuration, and split ratio dataset = load_dataset(dataset_name, config_name, split=split_ratio) print(f"Original dataset size: {len(dataset)}") # Split the dataset into train and test sets (80% train, 20% test) split_dataset = dataset.train_test_split(test_size=0.2) print(f"Train dataset size: {len(split_dataset['train'])}") print(f"Test dataset size: {len(split_dataset['test'])}") return split_dataset def save_dataset_to_jsonl(dataset, filepath): """ Save a dataset to a JSONL file. """ # Create the directory if it does not exist os.makedirs(os.path.dirname(filepath), exist_ok=True) # Open the file in write mode with open(filepath, 'w', encoding='utf-8') as f: # Iterate over each record in the dataset for record in dataset: # Dump the record as a JSON object and write it to the file json.dump(record, f) # Write a newline character to separate records f.write('\n') print(f"Dataset saved to {filepath}") def main(): """ Main function to load, split, and save the dataset. """ # Load and split the ULTRACHAT_200k dataset with a specific configuration and split ratio dataset = load_and_split_dataset("HuggingFaceH4/ultrachat_200k", 'default', 'train_sft[:1%]') # Extract the train and test datasets from the split train_dataset = dataset['train'] test_dataset = dataset['test'] # Save the train dataset to a JSONL file save_dataset_to_jsonl(train_dataset, "data/train_data.jsonl") # Save the test dataset to a separate JSONL file save_dataset_to_jsonl(test_dataset, "data/test_data.jsonl") if __name__ == "__main__": main()
-
Type the following command inside your terminal to run the script and download the dataset to your local environment.
python download_dataset.py -
Verify that the datasets were saved successfully to your local finetune-phi/data directory.
Note
In this tutorial, you use only 1% of the dataset (split='train[:1%]'). This significantly reduces the amount of data, speeding up both the upload and fine-tuning processes. You can adjust the percentage to find the right balance between training time and model performance. Using a smaller subset of the dataset reduces the time required for fine-tuning, making the process more manageable for a tutorial.
In this exercise, you will fine-tune the Phi-3 model in Azure Machine Learning Studio.
In this exercise, you will:
- Create computer cluster for fine-tuning.
- Fine-tune the Phi-3 model in Azure Machine Learning Studio.
-
Visit Azure ML Studio.
-
Select Compute from the left side tab.
-
Select Compute clusters from the navigation menu.
-
Select + New.
-
Perform the following tasks:
- Select the Region you'd like to use.
- Select the Virtual machine tier to Dedicated.
- Select the Virtual machine type to GPU.
- Select the Virtual machine size filter to Select from all options.
- Select the Virtual machine size to Standard_NC24ads_A100_v4.
-
Select Next.
-
Perform the following tasks:
- Enter Compute name. It must be a unique value.
- Select the Minimum number of nodes to 0.
- Select the Maximum number of nodes to 1.
- Select the Idle seconds before scale down to 120.
-
Select Create.
-
Visit Azure ML Studio.
-
Select the Azure Macnine Learning workspace that you created.
-
Perform the following tasks:
- Select Model catalog from the left side tab.
- Type phi-3-mini-4k in the search bar and select Phi-3-mini-4k-instruct from the options that appear.
-
Select Fine-tune from the navigation menu.
-
Perform the following tasks:
- Select Select task type to Chat completion.
- Select + Select data to upload Traning data.
- Select the Validation data upload type to Provide different validation data.
- Select + Select data to upload Validation data.
[!TIP]
You can select Advanced settings to customize configurations such as learning_rate and lr_scheduler_type to optimize the fine-tuning process according to your specific needs.
-
Select Finish.
-
In this exercise, you successfully fine-tuned the Phi-3 model using Azure Machine Learning. Please note that the fine-tuning process can take a considerable amount of time. After running the fine-tuning job, you need to wait for it to complete. You can monitor the status of the fine-tuning job by navigating to the Jobs tab on the left side of your Azure Machine Learning Workspace. In the next series, you will deploy the fine-tuned model and integrate it with Prompt flow.
To integrate the fine-tuned Phi-3 model with Prompt flow, you need to deploy the model to make it accessible for real-time inference. This process involves registering the model, creating an online endpoint, and deploying the model.
In this exercise, you will:
- Register the fine-tuned model in the Azure Machine Learning workspace.
- Create an online endpoint.
- Deploy the registered fine-tuned Phi-3 model.
-
Visit Azure ML Studio.
-
Select the Azure Macnine Learning workspace that you created.
-
Select Models from the left side tab.
-
Select + Register.
-
Select From a job output.
-
Select the job that you created.
-
Select Next.
-
Select Model type to MLflow.
-
Ensure that Job output is selected; it should be automatically selected.
-
Select Next.
-
Select Register.
-
You can view your registered model by navigating to the Models menu from the left side tab.
-
Navigate to the Azure Macnine Learning workspace that you created.
-
Select Endpoints from the left side tab.
-
Select Real-time endpoints from the navigation menu.
-
Select Create.
-
select the registered model that you created.
-
Select Select.
-
Perform the following tasks:
- Select Virtual machine to Standard_NC6s_v3.
- Select the Instance count you'd like to use. For example, 1.
- Select the Endpoint to New to create an endpoint.
- Enter Endpoint name. It must be a unique value.
- Enter Deployment name. It must be a unique value.
-
Select Deploy.
Warning
To avoid additional charges to your account, make sure to delete the created endpoint in the Azure Machine Learning workspace.
-
Navigate to Azure Machine Learning workspace that you created.
-
Select Endpoints from the left side tab.
-
Select the endpoint that you created.
-
On this page, you can manage the endpoints during the deployment process.
Note
Once the deployment is complete, ensure that Live traffic is set to 100%. If it is not, select Update traffic to adjust the traffic settings. Note that you cannot test the model if the traffic is set to 0%.
After successfully deploying your fine-tuned model, you can now integrate it with Prompt Flow to use your model in real-time applications, enabling a variety of interactive tasks with your custom Phi-3 model.
In this exercise, you will:
- Create Azure AI Studio Hub.
- Create Azure AI Studio Project.
- Create Prompt flow.
- Add a custom connection for the fine-tuned Phi-3 model.
- Set up Prompt flow to chat with your custom Phi-3 model
Note
You can also integrate with Promptflow using Azure ML Studio. The same integration process can be applied to Azure ML Studio.
You need to create a Hub before creating the Project. A Hub acts like a Resource Group, allowing you to organize and manage multiple Projects within Azure AI Studio.
-
Visit Azure AI Studio.
-
Select All hubs from the left side tab.
-
Select + New hub from the navigation menu.
-
Perform the following tasks:
- Enter Hub name. It must be a unique value.
- Select your Azure Subscription.
- Select the Resource group to use (create a new one if needed).
- Select the Location you'd like to use.
- Select the Connect Azure AI Services to use (create a new one if needed).
- Select Connect Azure AI Search to Skip connecting.
-
Select Next.
-
In the Hub that you created, select All projects from the left side tab.
-
Select + New project from the navigation menu.
-
Enter Project name. It must be a unique value.
-
Select Create a project.
To integrate your custom Phi-3 model with Prompt flow, you need to save the model's endpoint and key in a custom connection. This setup ensures access to your custom Phi-3 model in Prompt flow.
-
Visit Azure ML Studio.
-
Navigate to the Azure Machine learning workspace that you created.
-
Select Endpoints from the left side tab.
-
Select endpoint that you created.
-
Select Consume from the navigation menu.
-
Copy your REST endpoint and Primary key.
-
Visit Azure AI Studio.
-
Navigate to the Azure AI Studio project that you created.
-
In the Project that you created, select Settings from the left side tab.
-
Select + New connection.
-
Select Custom keys from the navigation menu.
-
Perform the following tasks:
- Select + Add key value pairs.
- For the key name, enter endpoint and paste the endpoint you copied from Azure ML Studio into the value field.
- Select + Add key value pairs again.
- For the key name, enter key and paste the key you copied from Azure ML Studio into the value field.
- After adding the keys, select is secret to prevent the key from being exposed.
-
Select Add connection.
-
Perform the following tasks to add the custom Phi-3 model's key:
You have added a custom connection in Azure AI Studio. Now, let's create a Prompt flow using the following steps. Then, you will connect this Prompt flow to the custom connection so that you can use the fine-tuned model within the Prompt flow.
-
Navigate to the Azure AI Studio project that you created.
-
Select Prompt flow from the left side tab.
-
Select + Create from the navigation menu.
-
Select Chat flow from the navigation menu.
-
Enter Folder name to use.
-
Select Create.
You need to integrate the fine-tuned Phi-3 model into a Prompt flow. However, the existing Prompt flow provided is not designed for this purpose. Therefore, you must redesign the Prompt flow to enable the integration of the custom model.
-
In the Prompt flow, perform the following tasks to rebuild the existing flow:
-
Select Raw file mode.
-
Delete all existing code in the flow.dag.yml file.
-
Add the following code to flow.dag.yml file.
inputs: input_data: type: string default: "Who founded Microsoft?" outputs: answer: type: string reference: ${integrate_with_promptflow.output} nodes: - name: integrate_with_promptflow type: python source: type: code path: integrate_with_promptflow.py inputs: input_data: ${inputs.input_data}
-
Select Save.
-
-
Add the following code to integrate_with_promptflow.py file to use the custom Phi-3 model in Prompt flow.
import logging import requests from promptflow import tool from promptflow.connections import CustomConnection # Logging setup logging.basicConfig( format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.DEBUG ) logger = logging.getLogger(__name__) def query_phi3_model(input_data: str, connection: CustomConnection) -> str: """ Send a request to the Phi-3 model endpoint with the given input data using Custom Connection. """ # "connection" is the name of the Custom Connection, "endpoint", "key" are the keys in the Custom Connection endpoint_url = connection.endpoint api_key = connection.key headers = { "Content-Type": "application/json", "Authorization": f"Bearer {api_key}" } data = { "input_data": { "input_string": [ {"role": "user", "content": input_data} ], "parameters": { "temperature": 0.7, "max_new_tokens": 128 } } } try: response = requests.post(endpoint_url, json=data, headers=headers) response.raise_for_status() # Log the full JSON response logger.debug(f"Full JSON response: {response.json()}") result = response.json()["output"] logger.info("Successfully received response from Azure ML Endpoint.") return result except requests.exceptions.RequestException as e: logger.error(f"Error querying Azure ML Endpoint: {e}") raise @tool def my_python_tool(input_data: str, connection: CustomConnection) -> str: """ Tool function to process input data and query the Phi-3 model. """ return query_phi3_model(input_data, connection)
Note
For more detailed information on using Prompt flow in Azure AI Studio, you can refer to Prompt flow in Azure AI Studio.
-
Select Chat input, Chat output to enable chat with your model.
-
Now you are ready to chat with your custom Phi-3 model. In the next exercise, you will learn how to start Prompt flow and use it to chat with your fine-tuned Phi-3 model.
Note
The rebuilt flow should look like the image below:
Now that you have fine-tuned and integrated your custom Phi-3 model with Prompt flow, you are ready to start interacting with it. This exercise will guide you through the process of setting up and initiating a chat with your model using Prompt flow. By following these steps, you will be able to fully utilize the capabilities of your fine-tuned Phi-3 model for various tasks and conversations.
- Chat with your custom Phi-3 model using Prompt flow.
-
Select Start compute sessions to start Prompt flow.
-
Select Validate and parse input to renew parameters.
-
Select the Value of the connection to the custom connection you created. For example, connection.
-
Select Chat.
-
Here's an example of the results: Now you can chat with your custom Phi-3 model. It is recommended to ask questions based on the data used for fine-tuning.
