Random Missing Value Injection into CSV Dataset Algorithm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c8cb7319",
   "metadata": {},
   "source": [
    "Title: Random Missing Value Injection into CSV Dataset\n",
    "\n",
    "Algorithm Methodology:\n",
    "\n",
    "Data Importing: Import necessary Python libraries such as pandas, numpy, random, and os.\n",
    "\n",
    "Function Definition: Define the add_missing_values function which takes as input a CSV file, number of columns where missing values are to be added, the number of missing values to be added, and a list of column names to be excluded from the operation.\n",
    "\n",
    "Data Loading: Load the CSV file into a pandas DataFrame.\n",
    "\n",
    "Initial Missing Values Calculation: Calculate the initial number of missing values in the dataset by using the .isnull().sum() methods in pandas DataFrame. Print the initial number of missing values for each column.\n",
    "\n",
    "Column Selection: From the list of all columns in the DataFrame, exclude the columns provided in the exclude_columns list. After that, randomly select a specified number of columns from the DataFrame where missing values are to be added.\n",
    "\n",
    "Missing Values Addition: Calculate the number of missing values to be added in each selected column. If there are any remaining cells after even distribution, they are assigned to the first few columns. For each selected column, randomly select row indices and add missing values at these indices.\n",
    "\n",
    "Final Missing Values Calculation: Calculate the final number of missing values in the dataset and print the number for each column.\n",
    "\n",
    "Data Saving: Save the DataFrame with added missing values into a new CSV file. Print the name and location of the new file.\n",
    "\n",
    "Function Call: Call the add_missing_values function providing the path of the CSV file, number of columns, number of missing cells to be added, and a list of columns to be excluded.\n",
    "\n",
    "The algorithm provides a way to intentionally introduce missing values into a dataset. This might be useful for testing and comparing different methods of handling missing data in data pre-processing or modeling tasks.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1a4028bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial number of missing values:\n",
      "id                   0\n",
      "dur                  0\n",
      "proto                0\n",
      "service              0\n",
      "state                0\n",
      "spkts                0\n",
      "dpkts                0\n",
      "sbytes               0\n",
      "dbytes               0\n",
      "rate                 0\n",
      "sttl                 0\n",
      "dttl                 0\n",
      "sload                0\n",
      "dload                0\n",
      "sloss                0\n",
      "dloss                0\n",
      "sinpkt               0\n",
      "dinpkt               0\n",
      "sjit                 0\n",
      "djit                 0\n",
      "swin                 0\n",
      "stcpb                0\n",
      "dtcpb                0\n",
      "dwin                 0\n",
      "tcprtt               0\n",
      "synack               0\n",
      "ackdat               0\n",
      "smean                0\n",
      "dmean                0\n",
      "trans_depth          0\n",
      "response_body_len    0\n",
      "ct_srv_src           0\n",
      "ct_state_ttl         0\n",
      "ct_dst_ltm           0\n",
      "ct_src_dport_ltm     0\n",
      "ct_dst_sport_ltm     0\n",
      "ct_dst_src_ltm       0\n",
      "is_ftp_login         0\n",
      "ct_ftp_cmd           0\n",
      "ct_flw_http_mthd     0\n",
      "ct_src_ltm           0\n",
      "ct_srv_dst           0\n",
      "is_sm_ips_ports      0\n",
      "attack_cat           0\n",
      "label                0\n",
      "dtype: int64\n",
      "Final number of missing values:\n",
      "id                    0\n",
      "dur                   0\n",
      "proto                 0\n",
      "service              34\n",
      "state                 0\n",
      "spkts                 0\n",
      "dpkts                 0\n",
      "sbytes                0\n",
      "dbytes                0\n",
      "rate                  0\n",
      "sttl                  0\n",
      "dttl                  0\n",
      "sload                33\n",
      "dload                 0\n",
      "sloss                 0\n",
      "dloss                 0\n",
      "sinpkt                0\n",
      "dinpkt                0\n",
      "sjit                  0\n",
      "djit                  0\n",
      "swin                 33\n",
      "stcpb                 0\n",
      "dtcpb                 0\n",
      "dwin                  0\n",
      "tcprtt                0\n",
      "synack                0\n",
      "ackdat                0\n",
      "smean                 0\n",
      "dmean                 0\n",
      "trans_depth           0\n",
      "response_body_len     0\n",
      "ct_srv_src            0\n",
      "ct_state_ttl          0\n",
      "ct_dst_ltm            0\n",
      "ct_src_dport_ltm      0\n",
      "ct_dst_sport_ltm      0\n",
      "ct_dst_src_ltm        0\n",
      "is_ftp_login          0\n",
      "ct_ftp_cmd            0\n",
      "ct_flw_http_mthd      0\n",
      "ct_src_ltm            0\n",
      "ct_srv_dst            0\n",
      "is_sm_ips_ports       0\n",
      "attack_cat            0\n",
      "label                 0\n",
      "dtype: int64\n",
      "Modified CSV saved as 'D:\\data\\Outlier_dataset\\a part of training and testing set\\new_UNSW_NB15_training-set.csv'\n",
      "Execution times:\n",
      "- Loading CSV: 0.3161 seconds\n",
      "- Column: service\n",
      "  - Random Sampling Time: 0.0010 seconds\n",
      "  - Missing Values Addition Time: 0.0010 seconds\n",
      "- Column: swin\n",
      "  - Random Sampling Time: 0.0000 seconds\n",
      "  - Missing Values Addition Time: 0.0060 seconds\n",
      "- Column: sload\n",
      "  - Random Sampling Time: 0.0000 seconds\n",
      "  - Missing Values Addition Time: 0.0000 seconds\n",
      "- Saving CSV: 1.5133 seconds\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import os\n",
    "import time\n",
    "\n",
    "def add_missing_values(csv_file, num_columns, num_cells, exclude_columns):\n",
    "    # Load the csv file into a pandas DataFrame\n",
    "    start_time = time.time()\n",
    "    df = pd.read_csv(csv_file)\n",
    "    load_time = time.time() - start_time\n",
    "\n",
    "    # Count the initial number of missing values\n",
    "    initial_missing_values = df.isnull().sum()\n",
    "    print(f\"Initial number of missing values:\\n{initial_missing_values}\")\n",
    "\n",
    "    # Get the list of all columns\n",
    "    all_columns = df.columns.tolist()\n",
    "\n",
    "    # Remove the excluded columns from the list of all columns\n",
    "    for col in exclude_columns:\n",
    "        if col in all_columns:\n",
    "            all_columns.remove(col)\n",
    "\n",
    "    # Check if the number of columns to be altered exceeds the total columns\n",
    "    if num_columns > len(all_columns):\n",
    "        print(\"The number of columns to be altered is more than the total number of columns in the file.\")\n",
    "        return\n",
    "\n",
    "    # Randomly select 'num_columns' columns\n",
    "    selected_columns = random.sample(all_columns, num_columns)\n",
    "\n",
    "    # Calculate the number of missing values to be added in each column\n",
    "    cells_per_column = num_cells // num_columns\n",
    "    extra_cells = num_cells % num_columns\n",
    "\n",
    "    # Iterate over each selected column\n",
    "    step_times = []\n",
    "    for i, col in enumerate(selected_columns):\n",
    "        # Distribute the extra cells among the first few columns\n",
    "        extra = 1 if i < extra_cells else 0\n",
    "\n",
    "        # Randomly select 'cells_per_column + extra' row indices\n",
    "        start_time = time.time()\n",
    "        row_indices = random.sample(range(len(df[col])), cells_per_column + extra)\n",
    "        random_sampling_time = time.time() - start_time\n",
    "\n",
    "        # Add missing values at the selected indices\n",
    "        start_time = time.time()\n",
    "        df.loc[row_indices, col] = np.nan\n",
    "        missing_values_addition_time = time.time() - start_time\n",
    "\n",
    "        step_times.append({\n",
    "            'Column': col,\n",
    "            'Random Sampling Time': random_sampling_time,\n",
    "            'Missing Values Addition Time': missing_values_addition_time\n",
    "        })\n",
    "\n",
    "    # Count the final number of missing values\n",
    "    final_missing_values = df.isnull().sum()\n",
    "    print(f\"Final number of missing values:\\n{final_missing_values}\")\n",
    "\n",
    "    # Save the DataFrame to a new csv file\n",
    "    start_time = time.time()\n",
    "    base_dir, file_name = os.path.split(csv_file)\n",
    "    name, extension = os.path.splitext(file_name)\n",
    "    new_file = os.path.join(base_dir, 'new_' + name + extension)\n",
    "    df.to_csv(new_file, index=False)\n",
    "    saving_time = time.time() - start_time\n",
    "\n",
    "    print(f\"Modified CSV saved as '{new_file}'\")\n",
    "    print(\"Execution times:\")\n",
    "    print(f\"- Loading CSV: {load_time:.4f} seconds\")\n",
    "    for step_time in step_times:\n",
    "        print(f\"- Column: {step_time['Column']}\")\n",
    "        print(f\"  - Random Sampling Time: {step_time['Random Sampling Time']:.4f} seconds\")\n",
    "        print(f\"  - Missing Values Addition Time: {step_time['Missing Values Addition Time']:.4f} seconds\")\n",
    "    print(f\"- Saving CSV: {saving_time:.4f} seconds\")\n",
    "\n",
    "# Use the function\n",
    "dataset_file = \"D:\\\\data\\\\Outlier_dataset\\\\a part of training and testing set\\\\UNSW_NB15_training-set.csv\"\n",
    "exclude_columns = ['label'] # replace these with the names of the columns to be excluded; you can leave it empty, but if you want to exclude columns mostly it should be the target\n",
    "add_missing_values(dataset_file, 3, 100, exclude_columns)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "887746d9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}