Skip to content

Commit

Permalink
Refactor effdet notebook (#845)
Browse files Browse the repository at this point in the history 
refactor EffDet notebook
* Add to notebook: TPC, KPI & MP config
* revert removing PostProcess from model for MCT
* switch installation to MCT-nightly (needed for using commits not yet in latest release)

---------

Co-authored-by: elad cohen <[email protected]>
  • Loading branch information
@elad-c
elad-c and elad cohen authored Oct 31, 2023
1 parent fc72c09 commit 74f4b3f
Show file tree
Hide file tree
Showing 6 changed files with 176 additions and 103 deletions.
173 changes: 140 additions & 33 deletions tutorials/notebooks/example_keras_effdet_lite0.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@
"\n",
"## Overview\n",
"\n",
"In this notebook, we'll demonstrate the post-training quantization using MCT for a pre-trained object detection model in Keras. In addition, we'll integrate a post-processing custom layer from [sony-custom-layers](https://github.com/sony/custom_layers) into the model. This integration aligns with the imx500 target platform capabilities.\n",
"In this notebook, we'll demonstrate the post-training quantization using MCT for a pre-trained object detection model in Keras. In addition, we'll integrate a post-processing custom layer from [sony-custom-layers](https://github.com/sony/custom_layers) into the model. This custom layer is supported by the imx500 target platform capabilities.\n",
"\n",
"In this example we will use an existing pre-trained EfficientDet model taken from [efficientdet-pytorch](https://github.com/rwightman/efficientdet-pytorch). We will convert the model to a Keras functional model that includes the custom [PostProcess Layer](https://github.com/sony/custom_layers/blob/main/sony_custom_layers/keras/object_detection/ssd_post_process.py). Further, we will quantize the model using MCT post training quantization and evaluate the performance of the floating point model and the quantized model on the COCO dataset.\n",
"\n",
"We'll use the [timm](https://github.com/huggingface/pytorch-image-models)'s data loader and evaluation capabilities used for the original pytorch pretrained model. The conversion to the Keras model will not be covered. You can go over the conversion [here](https://github.com/sony/model_optimization/tree/main/tutorials/resources/efficientdet)\n",
"We'll use the [timm](https://github.com/huggingface/pytorch-image-models)'s data loader and evaluation capabilities used for the original PyTorch pretrained model. The conversion to the Keras model will not be covered. You can go over the conversion [here](https://github.com/sony/model_optimization/tree/main/tutorials/resources/efficientdet).\n",
"\n",
"Steps:\n",
"* **Setup environment**: install relevant packages, import them\n",
"* **Init dataset**: Download the COCO evaluation and prepare the evaluation code\n",
"* **Setup the environment**: install relevant packages, import them\n",
"* **Initialize the dataset**: Download the COCO evaluation set and prepare the evaluation code\n",
"* **Keras float model**: Create the Keras model, assign the pretrained weights and evaluate it\n",
"* **Quantize Keras mode**: Quantize the model and evaluate it\n",
"\n",
Expand Down Expand Up @@ -46,7 +46,7 @@
"outputs": [],
"source": [
"!pip install -q tensorflow\n",
"!pip install -q model-compression-toolkit\n",
"!pip install -q mct-nightly\n",
"!pip install -q torch\n",
"!pip install -q torchvision\n",
"!pip install -q timm\n",
Expand All @@ -63,8 +63,10 @@
"execution_count": null,
"outputs": [],
"source": [
"from typing import Dict, Optional\n",
"from time import time\n",
"import torch\n",
"import tensorflow as tf\n",
"from timm.utils import AverageMeter\n",
"from effdet.config import get_efficientdet_config\n",
"from effdet import create_dataset, create_loader, create_evaluator\n",
Expand All @@ -91,7 +93,7 @@
"execution_count": null,
"outputs": [],
"source": [
"!git clone -b add_ported_effdet_keras_tutorial https://github.com/sony/model_optimization.git local_mct"
"!git clone https://github.com/sony/model_optimization.git local_mct"
],
"metadata": {
"collapsed": false
Expand All @@ -105,8 +107,7 @@
"source": [
"import sys\n",
"sys.path.insert(0,\"/content/local_mct\")\n",
"from tutorials.resources.efficientdet import EfficientDetKeras, TorchWrapper\n",
"from tutorials.resources.utils import load_state_dict"
"from tutorials.resources.efficientdet import EfficientDetKeras"
],
"metadata": {
"collapsed": false
Expand All @@ -116,9 +117,9 @@
{
"cell_type": "markdown",
"source": [
"## Init dataset\n",
"## Initialize dataset\n",
"\n",
"### Load COCO evaluation set"
"### Load the COCO evaluation set"
],
"metadata": {
"collapsed": false
Expand All @@ -145,7 +146,7 @@
{
"cell_type": "markdown",
"source": [
"### Init data loader and evaluation functions\n",
"### Initialize the data loader and evaluation functions\n",
"\n",
"These functions were adapted from the [efficientdet-pytorch](https://github.com/rwightman/efficientdet-pytorch) repository."
],
Expand All @@ -159,13 +160,62 @@
"execution_count": null,
"outputs": [],
"source": [
"class TorchWrapper(torch.nn.Module):\n",
" \"\"\"\n",
" A class to wrap the EfficientDet Keras model in a torch.nn.Module\n",
" so it can be evaluated with timm's evaluation code\n",
" \"\"\"\n",
" def __init__(self, keras_model: tf.keras.Model):\n",
" super(TorchWrapper, self).__init__()\n",
" self.model = keras_model\n",
"\n",
" @property\n",
" def config(self):\n",
" # a property used by the evaluation code\n",
" return self.model.config\n",
"\n",
" def forward(self, x: torch.Tensor,\n",
" img_info: Optional[Dict[str, torch.Tensor]] = None):\n",
" \"\"\"\n",
" mimics the forward inputs of the EfficientDet PyTorch model.\n",
" Args:\n",
" x: inputs images\n",
" img_info: input image info for scaling the outputs\n",
"\n",
" Returns:\n",
" A torch.Tensor of shape [Batch, Boxes, 6], the same as\n",
" the PyTorch model\n",
"\n",
" \"\"\"\n",
" device = x.device\n",
" keras_input = x.detach().cpu().numpy().transpose((0, 2, 3, 1))\n",
" outputs = self.model(keras_input)\n",
"\n",
" outs = [torch.Tensor(o.numpy()).to(device) for o in outputs]\n",
" # reorder boxes (y, x, y2, x2) to (x, y, x2, y2)\n",
" outs[0] = outs[0][:, :, [1, 0, 3, 2]]\n",
" # scale boxes to original image size\n",
" outs[0] = outs[0] * img_info['img_scale'].view((-1, 1, 1))\n",
" return torch.cat([outs[0], outs[1].unsqueeze(2),\n",
" outs[2].unsqueeze(2) + 1], 2)\n",
"\n",
"\n",
"def get_coco_dataloader(batch_size=16, split='val', config=None):\n",
" \"\"\"\n",
" Get the torch data-loader and evaluation object\n",
" Get the torch data-loader and evaluation object\n",
" Args:\n",
" batch_size: batch size for data loader\n",
" split: dataset split\n",
" config: model config\n",
"\n",
" Returns:\n",
" The DataLoader and evaluation object for calculating accuracy\n",
"\n",
" \"\"\"\n",
" root = '/content/coco'\n",
"\n",
" args = dict(interpolation='bilinear', mean=None, std=None, fill_color=None)\n",
" args = dict(interpolation='bilinear', mean=None,\n",
" std=None, fill_color=None)\n",
" dataset = create_dataset('coco', root, split)\n",
" input_config = resolve_input_config(args, config)\n",
" loader = create_loader(\n",
Expand All @@ -185,7 +235,19 @@
" return loader, evaluator\n",
"\n",
"\n",
"def acc_eval(_model, batch_size=16, config=None):\n",
"def acc_eval(_model: tf.keras.Model, batch_size=16, config=None):\n",
" \"\"\"\n",
" This function takes a Keras model, wraps it in a Torch model and runs evaluation\n",
" Args:\n",
" _model: Keras model\n",
" batch_size: batch size of the data loader\n",
" config: model config\n",
"\n",
" Returns:\n",
"\n",
" \"\"\"\n",
" # wrap Keras model in a Torch model so it can run in timm's evaluation code\n",
" _model = TorchWrapper(_model)\n",
" # EValuate input model\n",
" val_loader, evaluator = get_coco_dataloader(batch_size=batch_size, config=config)\n",
"\n",
Expand Down Expand Up @@ -234,14 +296,7 @@
"model_name = 'tf_efficientdet_lite0'\n",
"config = get_efficientdet_config(model_name)\n",
"\n",
"model = EfficientDetKeras(config,\n",
" pretrained_backbone=False\n",
" ).get_model([*config.image_size] + [3])\n",
"\n",
"state_dict = torch.hub.load_state_dict_from_url(config.url, progress=False,\n",
" map_location='cpu')\n",
"state_dict_numpy = {k: v.numpy() for k, v in state_dict.items()}\n",
"load_state_dict(model, state_dict_numpy)\n",
"model = EfficientDetKeras(config, pretrained_backbone=False).get_model([*config.image_size] + [3])\n",
"\n",
"model.save('/content/model.keras')"
],
Expand All @@ -255,7 +310,7 @@
"source": [
"### Evaluate Keras model\n",
"\n",
"Wrap model in a Torch Module, so it can be evaluated with timm's evaluation code. We evaluate the model to verify the conversion succeeded and to compare it to the quantized model evaluation. The \"TorchWrapper\" object is a PyTorch module that serves as an API between the timm's Torch evaluation framework and the Keras model."
"We evaluate the model to verify the conversion to a Keras model succeeded. The result will be compared to the quantized model evaluation."
],
"metadata": {
"collapsed": false
Expand All @@ -267,9 +322,7 @@
"execution_count": null,
"outputs": [],
"source": [
"wrapped_model = TorchWrapper(model)\n",
"\n",
"float_map = acc_eval(wrapped_model, batch_size=64, config=config)"
"float_map = acc_eval(model, batch_size=64, config=config)"
],
"metadata": {
"collapsed": false
Expand All @@ -279,7 +332,11 @@
{
"cell_type": "markdown",
"source": [
"## Quantized Keras model\n",
"## Quantize Keras model\n",
"\n",
"In this section, the Keras model will be quantized by the MCT, with the following parameters:\n",
"- **Target Platform**: IMX500-v1\n",
"- **Mixed-Precision** weights compression so the model will fit the IMX500 memory size\n",
"\n",
"The quantized model is saved as \"quant_model.keras\"."
],
Expand All @@ -297,18 +354,48 @@
"\n",
"\n",
"def get_representative_dataset(n_iter):\n",
" \"\"\"\n",
" This function creates a representative dataset generator\n",
" Args:\n",
" n_iter: number of iterations for MCT to calibrate on\n",
"\n",
" Returns:\n",
" A representative dataset generator\n",
"\n",
" \"\"\"\n",
"\n",
" def representative_dataset():\n",
" \"\"\"\n",
" Creates a representative dataset generator from a PyTorch data loader, The generator yields numpy\n",
" arrays of batches of shape: [Batch, H, W ,C]\n",
" Returns:\n",
" A representative dataset generator\n",
"\n",
" \"\"\"\n",
" ds_iter = iter(loader)\n",
" for _ in range(n_iter):\n",
" t = next(ds_iter)[0]\n",
" yield [t.detach().cpu().numpy().transpose((0, 2, 3, 1))]\n",
" # Convert the Torch tensor from the data loader to a numpy array and transpose to the\n",
" # right shape: [B, C, H, W] -> [B, H, W, C]\n",
" tf_shaped_tensor = t.detach().cpu().numpy().transpose((0, 2, 3, 1))\n",
" yield [tf_shaped_tensor]\n",
"\n",
" return representative_dataset\n",
"\n",
"\n",
"quant_model, _ = mct.ptq.keras_post_training_quantization_experimental(model,\n",
" get_representative_dataset(20))\n",
"# Set IMX500-v1 TPC\n",
"tpc = mct.get_target_platform_capabilities(\"tensorflow\", 'imx500', target_platform_version='v1')\n",
"# set weights memory size, so the quantized model will fit the IMX500 memory\n",
"kpi = mct.KPI(weights_memory=2674291)\n",
"# set MixedPrecision configuration for compressing the weights\n",
"mp_config = mct.core.MixedPrecisionQuantizationConfigV2(use_grad_based_weights=False)\n",
"core_config = mct.core.CoreConfig(mixed_precision_config=mp_config)\n",
"quant_model, _ = mct.ptq.keras_post_training_quantization_experimental(\n",
" model,\n",
" get_representative_dataset(20),\n",
" target_kpi=kpi,\n",
" core_config=core_config,\n",
" target_platform_capabilities=tpc)\n",
"quant_model.save('/content/quant_model.keras')"
],
"metadata": {
Expand All @@ -333,16 +420,36 @@
"execution_count": null,
"outputs": [],
"source": [
"wrapped_model = TorchWrapper(quant_model)\n",
"\n",
"quant_map = acc_eval(wrapped_model, batch_size=64, config=config)\n",
"quant_map = acc_eval(quant_model, batch_size=64, config=config)\n",
"\n",
"print(f' ===>> Float model mAP = {100*float_map:2.3f}, Quantized model mAP = {100*quant_map:2.3f}')"
],
"metadata": {
"collapsed": false
},
"id": "6f93b9b932fb39cc"
},
{
"cell_type": "markdown",
"source": [
"Copyright 2023 Sony Semiconductor Israel, Inc. All rights reserved.\n",
"\n",
"Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"you may not use this file except in compliance with the License.\n",
"You may obtain a copy of the License at\n",
"\n",
" http://www.apache.org/licenses/LICENSE-2.0\n",
"\n",
"Unless required by applicable law or agreed to in writing, software\n",
"distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"See the License for the specific language governing permissions and\n",
"limitations under the License."
],
"metadata": {
"collapsed": false
},
"id": "d36d177779d29347"
}
],
"metadata": {
Expand Down
2 changes: 1 addition & 1 deletion tutorials/resources/efficientdet/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@
# limitations under the License.
# ==============================================================================

from tutorials.resources.efficientdet.effdet_keras import EfficientDetKeras, TorchWrapper
from tutorials.resources.efficientdet.effdet_keras import EfficientDetKeras
Loading

0 comments on commit 74f4b3f

Please sign in to comment.