Chapter3-Data-pre-processing-for-clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data pre-processing for clustering\n",
    "\n",
    "Learn practical data preparation methods to ensure the k-means clustering algorithm uncovers well-separated segments\n",
    "\n",
    "### Advantages of k-means clustering\n",
    "- One of the most popular unsupervised learning method\n",
    "- Simple and fast\n",
    "- Works well*\n",
    "* _with certain assumptions about the data_\n",
    "\n",
    "### Key k-means assumptions\n",
    "- Symmetric distribution of variables (not skewed): Las variables tienen distribución simetrica\n",
    "- Variables with same average values: El segundo supuesto es que todas las variables tienen los mismos valores promedio. Esto es clave para garantizar que cada métrica tenga un peso igual en el cálculo de k-means. \n",
    "- Variables with same variance\n",
    "\n",
    "k-means assumes equal mean and equal variance, no es el caso de RFM.\n",
    "\n",
    "### Sequence\n",
    "1. Unskew the data - log transformation\n",
    "2. Standardize to the same average values\n",
    "3. Scale to the same standard deviation\n",
    "4. Store as a separate array to be used for clustering\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Center the data by subtracting average values from each entry\n",
    "data_centered = data - data.mean()\n",
    "\n",
    "# Scale the data by dividing each entry by standard deviation\n",
    "data_scaled = data / data.std()\n",
    "\n",
    "# Normalize the data by applying both centering and scaling\n",
    "data_normalized = (data - data.mean()) / data.std()\n",
    "\n",
    "# Print summary statistics to make sure average is zero and standard deviation is one\n",
    "print(data_normalized.describe().round(2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "             var1    var2    var3\n",
    "    count  100.00  100.00  100.00\n",
    "    mean     0.00   -0.00    0.00\n",
    "    std      1.00    1.00    1.00\n",
    "    min     -1.66   -0.73   -0.36\n",
    "    25%     -0.88   -0.51   -0.36\n",
    "    50%     -0.02   -0.29   -0.33\n",
    "    75%      0.96    0.11   -0.20\n",
    "    max      1.60    5.18    6.26"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a scaler\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler\n",
    "scaler.fit(data)\n",
    "\n",
    "# Scale and center the data\n",
    "data_normalized = scaler.transform(data)\n",
    "\n",
    "# Create a pandas DataFrame\n",
    "data_normalized = pd.DataFrame(data_normalized, index=data.index, columns=data.columns)\n",
    "\n",
    "# Print summary statistics\n",
    "print(data_normalized.describe().round(2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Unskew the data\n",
    "datamart_log = np.log(datamart_rfm)\n",
    "\n",
    "# Initialize a standard scaler and fit it\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(datamart_log)\n",
    "\n",
    "# Scale and center the data\n",
    "datamart_normalized = scaler.transform(datamart_log)\n",
    "\n",
    "# Create a pandas DataFrame\n",
    "datamart_normalized = pd.DataFrame(data=datamart_normalized, index=datamart_rfm.index, columns=datamart_rfm.columns)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}