notebooks/t003_flux_intro.ipynb

{
 "cells": [
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Tutorial 3: Intro To Flux\n",
    "\n",
    "[Flux White Paper](https://arxiv.org/pdf/1811.01457.pdf)\n",
    "\n",
    "In this tutorial we will cover the following\n",
    " - About Flux, [here](https://github.com/FluxML/Flux.jl)\n",
    " - What is a Flux model, [ModelZoo](https://github.com/FluxML/model-zoo), models as functions, Flux model types\n",
    " - Flux Layer Introduction\n",
    " - Flux Optimizer Introduction\n",
    " - Optimization batches, step size, batch normalization concepts\n",
    " - Different optimizer types, SGD, Flux.ADAM, et cetera\n",
    " - Flux activation functions: tanh, relu, leaky-relu"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Set up\n",
    "As always, we load in Flux, and in addition, we need LinearAlgebra\n",
    "for access to the the `LinearAlgebra.Transpose` namespace/type."
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "using Flux, Flux.Tracker\n",
    "using Flux.Tracker: grad, update!\n",
    "using Random\n",
    "using Test\n",
    "using LinearAlgebra"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Set up the data\n",
    "for the X and y, we want to take X and subsequently predict y"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "X_0 = randn(100,2)\n",
    "X_1 = randn(100,2) .+ 1\n",
    "X = transpose(cat(X_1, X_0, dims = 1))\n",
    "y = cat(ones(100), zeros(100), dims = 1);"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "This will be the data that we use, it's just a simple X/y\n",
    "dataset of N = 200, two dimensions, and one output dimension, y"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Basic Model\n",
    "We will set up a basic neural network, where we will take the two\n",
    "dimension per data point, expand it out to three, then contract, and\n",
    "apply softmax."
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "m = Chain(\n",
    "  Dense(2,3),\n",
    "  Dense(3,1,σ),\n",
    "  x -> reshape(x, :,1)\n",
    ")"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Model Intuition Test\n",
    "Given our model, `m`, and our data, `X`, we can apply our data to the\n",
    "model as if it where a function. What are the dimensions of `m(X)`?"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "n = 42\n",
    "Xp = randn(2,n)\n",
    "y_predicted = m(Xp)\n",
    "#= set y_predicted_size w/ n =#\n",
    "y_predicted_size = (0,0) # Fix me!\n",
    "@test size(y_predicted) == y_predicted_size # some function of n !"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Flux optimizer (an example)\n",
    "Here is an example of using Flux.train! with a loss, model parameters, and Optimization\n",
    "or loss function!"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "loss(X, y) = Flux.mse(m(X), y)\n",
    "ps = Flux.params(m)\n",
    "opt = Flux.ADAM()\n",
    "Flux.train!(loss, ps, [(X,y)], opt)"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "Note, it can be tricky to get the arguments to Flux.train correct\n",
    "what we are looking for, is that for each subsequent round, the (X,y)\n",
    "pair sufficiently matches the signature used in the loss function.\n",
    "\n",
    "Using the following loss function, rewrite the Flux.train function"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "loss(X) = Flux.mse(m(X), y)\n",
    "ps = Flux.params(m)\n",
    "opt = Flux.ADAM()\n",
    "data = [(X,y)]\n",
    "#= set data variable =#\n",
    "data =  [(X,)]\n",
    "Flux.train!(loss, ps, data, opt)"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "using the loss function this way is a little cleaner, as we won't\n",
    "have to pass along y for every batch"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Batch Sizes\n",
    "thus, using the previous formulation of our `data` and `loss` function\n",
    "into Flux.train, we batch by passing in a list of inputs that fit the dimensions\n",
    "of the model\n",
    "[Base.Iterator.zip](https://docs.julialang.org/en/v1/base/iterators/#Base.Iterators.zip)"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "ps = Flux.params(m)\n",
    "opt = Flux.ADAM()\n",
    "N = 200\n",
    "X = randn(MersenneTwister(42), 200, 2)\n",
    "y = cat(ones(100), zeros(100), dims = 1)\n",
    "\n",
    "N_total = size(X, 1)\n",
    "batch_size = 20\n",
    "batch_idx = collect(Base.Iterators.partition(1:N_total, batch_size))\n",
    "X_batches = zip([transpose(X[1:batch_size,:])],[ y[1:batch_size]] ) # Fix this call!\n",
    "Flux.train!(loss, ps, X_batches, opt)\n",
    "@test typeof(first(X_batches)) == Tuple{LinearAlgebra.Transpose{Float64,Array{Float64,2}},Array{Float64,1}}"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Different Optimizers\n",
    "there are a number of different optimizers you can try, for instnace,\n",
    "you can you a simple Descent optimizer [more info here](https://pkg.julialang.org/docs/Flux/QdkVy/0.8.3/training/optimisers/)"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "W = param(rand(MersenneTwister(42),2, 5))\n",
    "b = param(rand(MersenneTwister(42),2))\n",
    "predict(x) = W*x .+ b\n",
    "loss(x, y) = sum((predict(x) .- y).^2)\n",
    "x, y = rand(MersenneTwister(42),5), rand(MersenneTwister(42),2) # Dummy data\n",
    "l = loss(x, y) # ~ 3\n",
    "θ = Params([W, b])\n",
    "grads = Tracker.gradient(() -> loss(x, y), θ);"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Re-write the following example with Flux.Descent"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "η = 0.1 # Learning Rate\n",
    "for p in (W, b)\n",
    "  #= Insert a call to Flux.Tracker here =#\n",
    "end\n",
    "@test W[1,1] == 0.3967483862667399"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "# Re-written example w/ Flux.Descent\n",
    "Using the function Flux.Descent, which takes as an argument the learning rate, η"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "η = 0.1 # Learning Rate\n",
    "#= some function of η =#\n",
    "\n",
    "W = param(rand(MersenneTwister(42),2, 5))\n",
    "b = param(rand(MersenneTwister(42),2))\n",
    "\n",
    "predict(x) = W*x .+ b\n",
    "loss(x, y) = sum((predict(x) .- y).^2)\n",
    "\n",
    "x, y = rand(MersenneTwister(42),5), rand(MersenneTwister(42),2) # Dummy data\n",
    "l = loss(x, y) # ~ 3\n",
    "\n",
    "θ = Params([W, b])\n",
    "grads = Tracker.gradient(() -> loss(x, y), θ)\n",
    "\n",
    "for p in (W, b)\n",
    "  #= Flux.Track.update!(...) use opt here... =#\n",
    "end\n",
    "@test W[1,2] == -0.0984933499933695\n",
    "\n",
    "\n",
    "#= end module =#"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "outputs": [],
   "cell_type": "markdown",
   "source": [
    "*This notebook was generated using [Literate.jl](https://github.com/fredrikekre/Literate.jl).*"
   ],
   "metadata": {}
  }
 ],
 "nbformat_minor": 3,
 "metadata": {
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  },
  "kernelspec": {
   "name": "julia-1.2",
   "display_name": "Julia 1.2.0",
   "language": "julia"
  }
 },
 "nbformat": 4
}