Fashion_MNIST_Competition.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/martinezlc99/coe_training_LCM/blob/master/Fashion_MNIST_Competition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHwxGemYV_FT"
      },
      "source": [
        "# Fashion MNIST Competition!\n",
        "**Author**: W. Tod Newman\n",
        "\n",
        "**Updates**: New Version\n",
        "\n",
        "## Problem\n",
        "\n",
        "Classify images from the \"Fashion MNIST\" data set.   Optimize the test accuracy.\n",
        "\n",
        "## Metrics\n",
        "\n",
        "This competition is evaluated on the mean Dice coefficient. The Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. The formula is given by:![alt text](https://user-images.githubusercontent.com/26015273/41822460-2ca0a90a-77f0-11e8-9c71-7e88fa6b5c61.gif)\n",
        "\n",
        "\n",
        "The double sum is over the observations `i`, whose number is `N`, and the categories `c`, whose number is `C`. The term `1_{y_i \\in C_c}` is the indicator function of the `i`th observation belonging to the `c`th category. The `p_{model}[y_i \\in C_c]` is the probability predicted by the model for the `i`th observation to belong to the `c`th category. When there are more than two categories, the neural network outputs a vector of `C` probabilities, each giving the probability that the network input should be classified as belonging to the respective category. When the number of categories is just two, the neural network outputs a single probability `\\hat{y}_i`, with the other one being `1` minus the output. This is why the binary cross entropy looks a bit different from categorical cross entropy, despite being a special case of it.\n",
        "\n",
        "## Dataset\n",
        "\n",
        "This dataset is the Fashion MNIST dataset\n",
        "\n",
        "Recently, the researchers at Zalando, an e-commerce company, introduced Fashion MNIST as a drop-in replacement for the original MNIST dataset. Like MNIST, Fashion MNIST consists of a training set consisting of 60,000 examples belonging to 10 different classes and a test set of 10,000 examples. Each training example is a gray-scale image, 28x28 in size. The authors of the work further claim that the Fashion MNIST should actually replace MNIST dataset for benchmarking of new Machine Learning or Computer Vision models.\n",
        "\n",
        "Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255. The training and test data sets have 785 columns. The first column consists of the class labels (see above), and represents the article of clothing. The rest of the columns contain the pixel-values of the associated image.\n",
        "\n",
        "The Labels are:  \n",
        "0 T-shirt/top\n",
        "1 Trouser\n",
        "2 Pullover\n",
        "3 Dress\n",
        "4 Coat\n",
        "5 Sandal\n",
        "6 Shirt\n",
        "7 Sneaker\n",
        "8 Bag\n",
        "9 Ankle boot \n",
        "\n",
        "## Objective\n",
        "\n",
        "In this competition, you can try different variations of the CNN model given as a reference, you may evaluate techniques to squeeze more performance out of a CNN, or you might even try a completely different model, neural network or otherwise.  You will note that there are tips/tricks/techniques documented in many locations on the internet that could be useful.\n",
        "\n",
        "## Rules and Timeline\n",
        "\n",
        "The primary measure for the competition will be the accuracy of prediction on the test data.  Ties will be broken by Precision accuracy first, then Recall Accuracy if needed.\n",
        "\n",
        "The results will be revealed at the end of the last day of class.  Please submit your Metrics blocks (Starts with SUBMIT... and ends with END SUBMISSION) to instructors (wtnewman@raytheon.com) before lunch.\n",
        "\n",
        "A prize will be given to the top finisher(s) based on the judgement of the instructor and the availability of prizes.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V0A2_Mo0KxWc",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from tensorflow import keras\n",
        "from keras.datasets import fashion_mnist\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Flatten,  Conv2D, MaxPooling2D, Activation, BatchNormalization\n",
        "from keras import backend as K\n",
        "from keras.callbacks import EarlyStopping\n",
        "from keras.losses import SparseCategoricalCrossentropy\n",
        "from sklearn import metrics\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.model_selection import train_test_split\n",
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "from keras.callbacks import TensorBoard,  ModelCheckpoint\n",
        "from keras.layers.advanced_activations import LeakyReLU\n",
        "import os\n",
        "%matplotlib inline\n",
        "os.environ['TF_CPP_MIN_LOG_LEVEL'] = \"2\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3y6VjlzZLKPB"
      },
      "source": [
        "## Set Up Variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LtL1H333LHGz",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "epochs = 100                 # Number of Training Epochs\n",
        "num_classes = 10            # This is the number of classes in the Fashion MNIST dataset\n",
        "batch_size = 128       # This parameter can be adjusted\n",
        "img_rows, img_cols = 28, 28 # Pixel sizes of the Images in the Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4SW52B7dT9H5"
      },
      "source": [
        "## Gather and Process Fashion MNIST data\n",
        "\n",
        "1. First, collect the data from Keras (our goal is someday that our organizational data is this easy to get!)\n",
        "2. Then split into train and test sets.\n",
        "3. Next we need to process the data into the proper shape for the CNN\n",
        "4. Then scale the floats to land between 0 and 1.  Often times we use sklearn's MinMaxScaler for this, but in this case we're going for simplicity.\n",
        "5. Next take the y_train and y_test labels and encode them one-hot.  This will enable the CNN to function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GZENio2YLPUy",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Grab the data from the keras repository\n",
        "\n",
        "mnist_data = fashion_mnist.load_data()\n",
        "x = mnist_data[0][0]\n",
        "y = mnist_data[0][1]\n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=41)\n",
        "\n",
        "# Process the date into the right tensor shape.  This is a good practice, but\n",
        "# usually tensorflow uses channels last (the 'else' here)\n",
        "\n",
        "if K.image_data_format() == \"channels first\":\n",
        "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
        "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
        "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "    \n",
        "#\n",
        "#  Cast to a 32 bit float and then scale so the value is a float between 0 and 1\n",
        "    \n",
        "x_train = x_train.astype(\"float32\")\n",
        "x_test = x_test.astype(\"float32\")\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "#\n",
        "# Convert Class Vector to Binary Class Matrices (one-hot encoding).\n",
        "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
        "print(y_test.shape)\n",
        "\n",
        "#\n",
        "# Function to decode one-hot encoding later on when we want to evaluate performance.\n",
        "def decode_one_hot(y):\n",
        "    y_classes = [np.argmax(yi, axis=None, out=None) for yi in y]\n",
        "    return y_classes\n",
        "\n",
        "'''\n",
        "\n",
        "Below we're experimenting with the Keras ImageDataGenerator.  From my experience, if these parameters\n",
        "are set too aggressively, the loss/accuracy will either never improve or it will take too long to improve.\n",
        "Below is an example of a complex data augmentation regime.  This is just for reference.  See my more simple\n",
        "one at the bottom.\n",
        "\n",
        "    \n",
        "datagen = ImageDataGenerator(rotation_range=0.5, \n",
        "                                 zoom_range=0.1,\n",
        "                                 featurewise_center=True,\n",
        "                                 #featurewise_std_normalization=True,\n",
        "                                 width_shift_range=0.1, \n",
        "                                 height_shift_range=0.1, \n",
        "                                 shear_range=0.1,\n",
        "                                 horizontal_flip=True, \n",
        "                                 fill_mode=\"nearest\")\n",
        "'''\n",
        "#\n",
        "#  Set up our Image Augmentation Data Generator\n",
        "#\n",
        "# datagen = ImageDataGenerator(rotation_range=5)\n",
        "# datagen = ImageDataGenerator(rotation_range=0.5, \n",
        "#                                  zoom_range=0.1,\n",
        "#                                  featurewise_center=True,\n",
        "#                                 #  featurewise_std_normalization=True,\n",
        "#                                  width_shift_range=0.1, \n",
        "#                                  height_shift_range=0.1, \n",
        "#                                  shear_range=0.1,\n",
        "#                                  horizontal_flip=True, \n",
        "#                                  fill_mode=\"nearest\")\n",
        "\n",
        "\n",
        "# see https://github.com/yu4u/cutout-random-erasing/blob/master/random_eraser.py\n",
        "def get_random_eraser(p=0.5, s_l=0.02, s_h=0.4, r_1=0.3, r_2=1/0.3, v_l=0, v_h=255, pixel_level=False):\n",
        "    def eraser(input_img):\n",
        "        if input_img.ndim == 3:\n",
        "            img_h, img_w, img_c = input_img.shape\n",
        "        elif input_img.ndim == 2:\n",
        "            img_h, img_w = input_img.shape\n",
        "\n",
        "        p_1 = np.random.rand()\n",
        "\n",
        "        if p_1 > p:\n",
        "            return input_img\n",
        "\n",
        "        while True:\n",
        "            s = np.random.uniform(s_l, s_h) * img_h * img_w\n",
        "            r = np.random.uniform(r_1, r_2)\n",
        "            w = int(np.sqrt(s / r))\n",
        "            h = int(np.sqrt(s * r))\n",
        "            left = np.random.randint(0, img_w)\n",
        "            top = np.random.randint(0, img_h)\n",
        "\n",
        "            if left + w <= img_w and top + h <= img_h:\n",
        "                break\n",
        "\n",
        "        if pixel_level:\n",
        "            if input_img.ndim == 3:\n",
        "                c = np.random.uniform(v_l, v_h, (h, w, img_c))\n",
        "            if input_img.ndim == 2:\n",
        "                c = np.random.uniform(v_l, v_h, (h, w))\n",
        "        else:\n",
        "            c = np.random.uniform(v_l, v_h)\n",
        "\n",
        "        input_img[top:top + h, left:left + w] = c\n",
        "\n",
        "        return input_img\n",
        "\n",
        "    return eraser\n",
        "\n",
        "# datagen = ImageDataGenerator(width_shift_range=0.005,height_shift_range=0.005,horizontal_flip=True)\n",
        "\n",
        "datagen = ImageDataGenerator(\n",
        "    featurewise_center=False,  # set input mean to 0 over the dataset\n",
        "    samplewise_center=False,  # set each sample mean to 0\n",
        "    featurewise_std_normalization=False,  # divide inputs by std of the dataset\n",
        "    samplewise_std_normalization=False,  # divide each input by its std\n",
        "    zca_whitening=False,  # apply ZCA whitening\n",
        "    rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)\n",
        "    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n",
        "    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n",
        "    horizontal_flip=True,  # randomly flip images\n",
        "    vertical_flip=False,  # randomly flip images\n",
        "    preprocessing_function=get_random_eraser(v_l=0.15, v_h=20, pixel_level=True)\n",
        "    )\n",
        "datagen.fit(x_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8FcFnUPDLvnO"
      },
      "source": [
        "## Build the Model\n",
        "\n",
        "* In this example, we define the below block as a Sequential Model. \n",
        "* See the excellent [Keras Documentation](https://keras.io/guides/sequential_model/) on Sequential Models for info.\n",
        "* Many of these parameters can be experimented with.  The documentation will help you understand how much to experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8EWdo8mALXB9",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def _create_model_basic():\n",
        "    #\n",
        "    # This is what is known as a Tensorflow (Keras) Sequential model\n",
        "    # We will talk at some level about each of these layer types in class.\n",
        "    #\n",
        "    model = Sequential()\n",
        "    # model.add(Flatten(input_shape=input_shape))\n",
        "    # model.add(Dense(128, activation='relu'))\n",
        "    # model.add(Dense(10))\n",
        "    model.add(Conv2D(32, kernel_size=(3,3), activation='relu',\n",
        "                    kernel_initializer='he_normal',input_shape=input_shape))\n",
        "    model.add(MaxPooling2D((2, 2)))\n",
        "    model.add(Dropout(0.25))\n",
        "    # Define the first hidden layer.   \n",
        "    # model.add(Dense(units=256, activation='relu'))\n",
        "    # model.add(Dense(units=256))\n",
        "    # model.add(Activation('relu'))\n",
        "    # model.add(Conv2D(64, kernel_size=(3,3)))\n",
        "    model.add(LeakyReLU(alpha=0.05))\n",
        "    # model.add(Dropout(0.1))\n",
        "    # model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "    # model.add(Activation('relu'))\n",
        "    # model.add(Flatten())\n",
        "    # model.add(Dense(256))  \n",
        "    # model.add(Activation('relu'))\n",
        "    # model.add(Dropout(0.4))\n",
        "    # model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "    model.add(Conv2D(128, \n",
        "                    kernel_size=(3, 3), \n",
        "                    activation='relu'))\n",
        "    # model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "    # Add dropouts to the model\n",
        "    # model.add(Dropout(0.4))\n",
        "    # model.add(Conv2D(128, (3, 3), activation='relu'))\n",
        "    # # Add dropouts to the model\n",
        "    model.add(Dropout(0.4))\n",
        "    model.add(Flatten())\n",
        "    model.add(Dense(128, activation='relu'))\n",
        "    # # Add dropouts to the model\n",
        "    model.add(Dropout(0.25))\n",
        "    model.add(Dense(num_classes, activation=\"softmax\"))\n",
        "\n",
        "    return model\n",
        "\n",
        "my_callbacks = [ModelCheckpoint('model_out.hdf5', monitor='accuracy', save_best_only=True, save_freq=1)]\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvIfjap-L8TO"
      },
      "source": [
        "## Fit and Evaluate the Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n5U20dV1L-eP",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Flag to determine whether we use Keras' Image augmentation data generator\n",
        "augmentation = True\n",
        "\n",
        "# create model\n",
        "module = _create_model_basic()\n",
        "# Compile the model so we can fit it. Researching loss functions and optimizers\n",
        "# might be a good thing to do.\n",
        "#\n",
        "model.compile(\n",
        "              loss=keras.losses.categorical_crossentropy, \n",
        "              # loss=SparseCategoricalCrossentropy(from_logits=True),\n",
        "              optimizer=keras.optimizers.RMSprop(),\n",
        "              # optimizer='adam', \n",
        "              metrics=['accuracy'])\n",
        "\n",
        "if not augmentation:\n",
        "    #\n",
        "    # Fit the model.  Once the model is trained we'll evaluate the performance.\n",
        "    print('not using image augmentation')\n",
        "    hist = model.fit(x_train, y_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=epochs,\n",
        "          verbose=1,\n",
        "          validation_data=(x_test, y_test),\n",
        "          callbacks=my_callbacks)\n",
        "else:\n",
        "# fits the model on batches with real-time data augmentation:\n",
        "    hist = model.fit(datagen.flow(x_train, y_train, batch_size=batch_size),\n",
        "                           steps_per_epoch=len(x_train) / batch_size, validation_data=(x_test, y_test),\n",
        "                           epochs=epochs, verbose=1, callbacks=my_callbacks, workers = 2)\n",
        "\n",
        "\n",
        "score = model.evaluate(x_test, y_test)\n",
        "\n",
        "#\n",
        "# Predict on the test data and pass to metrics function\n",
        "yhat = np.argmax(model.predict(x_test), axis=-1)\n",
        "y_dec = decode_one_hot(y_test)\n",
        "\n",
        "print(\"\\nSUBMIT THIS BLOCK for the Competition\\n\")\n",
        "print(metrics.classification_report(y_dec, yhat))\n",
        "print(\"Testing Loss:\", score[0])\n",
        "print(\"Testing Accuracy:\", score[1])\n",
        "print(\"END SUBMISSION BLOCK\\n\")\n",
        "\n",
        "model.summary()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wTXLg2MJMFhy"
      },
      "source": [
        "## Plot the accuracy vs. validation accuracy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P2Z97VRkMIRh",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "epoch_list = list(range(1, len(hist.history['accuracy']) + 1))\n",
        "plt.plot(epoch_list, hist.history['accuracy'], epoch_list, hist.history['val_accuracy'])\n",
        "plt.legend((\"Training Accuracy\", \"Validation Accuracy\"))\n",
        "plt.show()\n",
        "\n",
        "plt.plot(epoch_list, hist.history['loss'], epoch_list, hist.history['val_loss'])\n",
        "plt.legend((\"Training Loss\", \"Validation Loss\"))\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-imOV3gbDji"
      },
      "source": [
        "## Visualization of Performance on the Test Set\n",
        "\n",
        "Here is a visualization of how well our classifier can do inference."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hfsF8TlLwarT",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "from imutils import build_montages\n",
        "from google.colab.patches import cv2_imshow\n",
        "\n",
        "# initialize our list of output images\n",
        "images = []\n",
        "\n",
        "labelNames = [\"top\", \"trouser\", \"pullover\", \"dress\", \"coat\",\n",
        "\t\"sandal\", \"shirt\", \"sneaker\", \"bag\", \"ankle boot\"]\n",
        " \n",
        "# randomly select a few testing fashion items\n",
        "for i in np.random.choice(np.arange(0, len(y_test)), size=(16,)):\n",
        "\t# classify the clothing\n",
        "\tprobs = model.predict(x_test[np.newaxis, i])\n",
        "\tprediction = probs.argmax(axis=1)\n",
        "\tlabel = labelNames[prediction[0]]\n",
        " \n",
        "\t# extract the image from the testData if using \"channels_first\"\n",
        "\t# ordering\n",
        "\tif K.image_data_format() == \"channels_first\":\n",
        "\t\timage = (x_test[i][0] * 255).astype(\"uint8\")\n",
        " \n",
        "\t# otherwise we are using \"channels_last\" ordering\n",
        "\telse:\n",
        "\t\timage = (x_test[i] * 255).astype(\"uint8\")\n",
        "    # initialize the text label color as green (correct)\n",
        "\tcolor = (0, 255, 0)\n",
        " \n",
        "\t# otherwise, the class label prediction is incorrect\n",
        "\tif prediction[0] != np.argmax(y_test[i]):\n",
        "\t\tcolor = (0, 0, 255)\n",
        " \n",
        "\t# merge the channels into one image and resize the image from\n",
        "\t# 28x28 to 96x96 so we can better see it and then draw the\n",
        "\t# predicted label on the image\n",
        "\timage = cv2.merge([image] * 3)\n",
        "\timage = cv2.resize(image, (96, 96), interpolation=cv2.INTER_LINEAR)\n",
        "\tcv2.putText(image, label, (5, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.75,\n",
        "\t\tcolor, 2)\n",
        " \n",
        "\t# add the image to our list of output images\n",
        "\timages.append(image)\n",
        "# construct the montage for the images\n",
        "montage = build_montages(images, (96, 96), (4, 4))[0]\n",
        " \n",
        "# show the output montage\n",
        "cv2_imshow( montage)\n",
        "cv2.waitKey(0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dMDjuN_anxKM",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "Fashion MNIST Competition",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}