Projects_in_ML_HW3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNLFkkMX2W9kAt3JlzxS0IK",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/shadowluigi27/Projects-in-ML-F22/blob/main/Projects_in_ML_HW3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Task 1\n",
        "Some websites I used to help with getting my NN working\n",
        "\n",
        "https://www.tensorflow.org/guide/autodiff\n",
        "(provided)\n",
        "\n",
        "https://www.tensorflow.org/api_docs/python/tf/GradientTape\n",
        "(provided)\n",
        "\n",
        "https://scikit-learn.org/stable/modules/neural_networks_supervised.html\n",
        "\n",
        "https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html \n",
        "\n",
        "I wanted to use Skitlearn since I used it in my last HW, and learning the different types of models in one library list would allow me to more effectivly use it"
      ],
      "metadata": {
        "id": "IB6EJSFNWEJW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Task 2"
      ],
      "metadata": {
        "id": "GkpZjml_dLur"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KwXhJqYOMf8c"
      },
      "outputs": [],
      "source": [
        "# import required libraries\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "import math\n",
        "import tensorflow as tf\n",
        "\n",
        "\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.metrics import f1_score\n",
        "from sklearn.preprocessing import Normalizer\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "\n",
        "import keras\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Dataset link: https://archive.ics.uci.edu/ml/datasets/census+income \n",
        "\n",
        "EDA of dataset below\n"
      ],
      "metadata": {
        "id": "09yNdflZbUjV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv('adult.csv')\n",
        "df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 340
        },
        "id": "_Uhb8mVCMuXm",
        "outputId": "a7f1622b-63fb-4c38-9f64-113799b4328a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   age workclass  fnlwgt     education  education.num marital.status  \\\n",
              "0   90         ?   77053       HS-grad              9        Widowed   \n",
              "1   82   Private  132870       HS-grad              9        Widowed   \n",
              "2   66         ?  186061  Some-college             10        Widowed   \n",
              "3   54   Private  140359       7th-8th              4       Divorced   \n",
              "4   41   Private  264663  Some-college             10      Separated   \n",
              "\n",
              "          occupation   relationship   race     sex  capital.gain  \\\n",
              "0                  ?  Not-in-family  White  Female             0   \n",
              "1    Exec-managerial  Not-in-family  White  Female             0   \n",
              "2                  ?      Unmarried  Black  Female             0   \n",
              "3  Machine-op-inspct      Unmarried  White  Female             0   \n",
              "4     Prof-specialty      Own-child  White  Female             0   \n",
              "\n",
              "   capital.loss  hours.per.week native.country income  \n",
              "0          4356              40  United-States  <=50K  \n",
              "1          4356              18  United-States  <=50K  \n",
              "2          4356              40  United-States  <=50K  \n",
              "3          3900              40  United-States  <=50K  \n",
              "4          3900              40  United-States  <=50K  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a9261c79-5011-44ba-abec-9afe7f6147e9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>workclass</th>\n",
              "      <th>fnlwgt</th>\n",
              "      <th>education</th>\n",
              "      <th>education.num</th>\n",
              "      <th>marital.status</th>\n",
              "      <th>occupation</th>\n",
              "      <th>relationship</th>\n",
              "      <th>race</th>\n",
              "      <th>sex</th>\n",
              "      <th>capital.gain</th>\n",
              "      <th>capital.loss</th>\n",
              "      <th>hours.per.week</th>\n",
              "      <th>native.country</th>\n",
              "      <th>income</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>90</td>\n",
              "      <td>?</td>\n",
              "      <td>77053</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Widowed</td>\n",
              "      <td>?</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>4356</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>82</td>\n",
              "      <td>Private</td>\n",
              "      <td>132870</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Widowed</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>4356</td>\n",
              "      <td>18</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>66</td>\n",
              "      <td>?</td>\n",
              "      <td>186061</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Widowed</td>\n",
              "      <td>?</td>\n",
              "      <td>Unmarried</td>\n",
              "      <td>Black</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>4356</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>54</td>\n",
              "      <td>Private</td>\n",
              "      <td>140359</td>\n",
              "      <td>7th-8th</td>\n",
              "      <td>4</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Machine-op-inspct</td>\n",
              "      <td>Unmarried</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>3900</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>41</td>\n",
              "      <td>Private</td>\n",
              "      <td>264663</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Separated</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Own-child</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>3900</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a9261c79-5011-44ba-abec-9afe7f6147e9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a9261c79-5011-44ba-abec-9afe7f6147e9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a9261c79-5011-44ba-abec-9afe7f6147e9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y8L4eHYlPl79",
        "outputId": "862015b4-94cd-4e30-8cf4-41660f6fd728"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(32561, 15)"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.dtypes"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6CyzrR1vTd75",
        "outputId": "2befe087-c9d9-4b7a-e219-3a20c6d9df81"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age                int64\n",
              "workclass         object\n",
              "fnlwgt             int64\n",
              "education         object\n",
              "education.num      int64\n",
              "marital.status    object\n",
              "occupation        object\n",
              "relationship      object\n",
              "race              object\n",
              "sex               object\n",
              "capital.gain       int64\n",
              "capital.loss       int64\n",
              "hours.per.week     int64\n",
              "native.country    object\n",
              "income            object\n",
              "dtype: object"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.isna().sum()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P92b3o6tPowV",
        "outputId": "ce13cea5-b026-493c-f02f-d66472a11a80"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "age               0\n",
              "workclass         0\n",
              "fnlwgt            0\n",
              "education         0\n",
              "education.num     0\n",
              "marital.status    0\n",
              "occupation        0\n",
              "relationship      0\n",
              "race              0\n",
              "sex               0\n",
              "capital.gain      0\n",
              "capital.loss      0\n",
              "hours.per.week    0\n",
              "native.country    0\n",
              "income            0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "no non null values, just data with ?, so we will remove rows with those values\n",
        "\n",
        "From research: https://www.kaggle.com/datasets/uciml/adult-census-income/discussion/32698\n",
        "\n",
        "Fnlweight appears to be different for each state, and just represents # of people who share similar traits."
      ],
      "metadata": {
        "id": "3g7lBKGFPrkc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.columns"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x6yZcaHGXA4O",
        "outputId": "095f6e2f-4378-4553-ef75-1ab74855cc4b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['age', 'workclass', 'fnlwgt', 'education', 'education.num',\n",
              "       'marital.status', 'occupation', 'relationship', 'race', 'sex',\n",
              "       'capital.gain', 'capital.loss', 'hours.per.week', 'native.country',\n",
              "       'income'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#rename columns so we can look at values better\n",
        "df = df.rename(columns={\"education.num\": \"educationNum\", \n",
        "                   \"marital.status\": \"maritalStatus\",\n",
        "                   \"capital.gain\" : \"capitalGain\",\n",
        "                   \"capital.loss\" : \"capitalLoss\",\n",
        "                   \"hours.per.week\" : \"capitalPerWeek\",\n",
        "                   \"native.country\" : \"nativeCountry\"})"
      ],
      "metadata": {
        "id": "evUsb0ssrlip"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "labels = [\"workclass\",\"maritalStatus\",\"occupation\",\"relationship\",\"race\",\"sex\",\"nativeCountry\"]\n",
        "\n",
        "for label in labels:\n",
        "  df = df.loc[df[label] != \"?\"]\n",
        "\n",
        "df.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O9KbmufVZH7b",
        "outputId": "232f3412-03cb-490a-e9de-3fd3a9a77e5e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(30162, 15)"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#df = df.drop('fnlwgt',axis=1)"
      ],
      "metadata": {
        "id": "ZWdyxc8oQtIw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.describe()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "Ucneqo3kPrVf",
        "outputId": "85aaf188-177d-42fd-cd42-94f487e2e5dc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                age        fnlwgt  educationNum   capitalGain   capitalLoss  \\\n",
              "count  30162.000000  3.016200e+04  30162.000000  30162.000000  30162.000000   \n",
              "mean      38.437902  1.897938e+05     10.121312   1092.007858     88.372489   \n",
              "std       13.134665  1.056530e+05      2.549995   7406.346497    404.298370   \n",
              "min       17.000000  1.376900e+04      1.000000      0.000000      0.000000   \n",
              "25%       28.000000  1.176272e+05      9.000000      0.000000      0.000000   \n",
              "50%       37.000000  1.784250e+05     10.000000      0.000000      0.000000   \n",
              "75%       47.000000  2.376285e+05     13.000000      0.000000      0.000000   \n",
              "max       90.000000  1.484705e+06     16.000000  99999.000000   4356.000000   \n",
              "\n",
              "       capitalPerWeek  \n",
              "count    30162.000000  \n",
              "mean        40.931238  \n",
              "std         11.979984  \n",
              "min          1.000000  \n",
              "25%         40.000000  \n",
              "50%         40.000000  \n",
              "75%         45.000000  \n",
              "max         99.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-499ba8aa-4404-40df-8c4d-023c2281182b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>fnlwgt</th>\n",
              "      <th>educationNum</th>\n",
              "      <th>capitalGain</th>\n",
              "      <th>capitalLoss</th>\n",
              "      <th>capitalPerWeek</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>30162.000000</td>\n",
              "      <td>3.016200e+04</td>\n",
              "      <td>30162.000000</td>\n",
              "      <td>30162.000000</td>\n",
              "      <td>30162.000000</td>\n",
              "      <td>30162.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>38.437902</td>\n",
              "      <td>1.897938e+05</td>\n",
              "      <td>10.121312</td>\n",
              "      <td>1092.007858</td>\n",
              "      <td>88.372489</td>\n",
              "      <td>40.931238</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>13.134665</td>\n",
              "      <td>1.056530e+05</td>\n",
              "      <td>2.549995</td>\n",
              "      <td>7406.346497</td>\n",
              "      <td>404.298370</td>\n",
              "      <td>11.979984</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>17.000000</td>\n",
              "      <td>1.376900e+04</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>28.000000</td>\n",
              "      <td>1.176272e+05</td>\n",
              "      <td>9.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>37.000000</td>\n",
              "      <td>1.784250e+05</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>47.000000</td>\n",
              "      <td>2.376285e+05</td>\n",
              "      <td>13.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>45.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>90.000000</td>\n",
              "      <td>1.484705e+06</td>\n",
              "      <td>16.000000</td>\n",
              "      <td>99999.000000</td>\n",
              "      <td>4356.000000</td>\n",
              "      <td>99.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-499ba8aa-4404-40df-8c4d-023c2281182b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-499ba8aa-4404-40df-8c4d-023c2281182b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-499ba8aa-4404-40df-8c4d-023c2281182b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.describe(include=['O'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "-pXoeXNPQ3sK",
        "outputId": "772a1c99-c029-402b-c597-4b90097157a6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       workclass education       maritalStatus      occupation relationship  \\\n",
              "count      30162     30162               30162           30162        30162   \n",
              "unique         7        16                   7              14            6   \n",
              "top      Private   HS-grad  Married-civ-spouse  Prof-specialty      Husband   \n",
              "freq       22286      9840               14065            4038        12463   \n",
              "\n",
              "         race    sex  nativeCountry income  \n",
              "count   30162  30162          30162  30162  \n",
              "unique      5      2             41      2  \n",
              "top     White   Male  United-States  <=50K  \n",
              "freq    25933  20380          27504  22654  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-86b5e656-f9d7-4ad3-96be-96904a87f97a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>workclass</th>\n",
              "      <th>education</th>\n",
              "      <th>maritalStatus</th>\n",
              "      <th>occupation</th>\n",
              "      <th>relationship</th>\n",
              "      <th>race</th>\n",
              "      <th>sex</th>\n",
              "      <th>nativeCountry</th>\n",
              "      <th>income</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "      <td>30162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>7</td>\n",
              "      <td>16</td>\n",
              "      <td>7</td>\n",
              "      <td>14</td>\n",
              "      <td>6</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>41</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>top</th>\n",
              "      <td>Private</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>freq</th>\n",
              "      <td>22286</td>\n",
              "      <td>9840</td>\n",
              "      <td>14065</td>\n",
              "      <td>4038</td>\n",
              "      <td>12463</td>\n",
              "      <td>25933</td>\n",
              "      <td>20380</td>\n",
              "      <td>27504</td>\n",
              "      <td>22654</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-86b5e656-f9d7-4ad3-96be-96904a87f97a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-86b5e656-f9d7-4ad3-96be-96904a87f97a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-86b5e656-f9d7-4ad3-96be-96904a87f97a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Preprocessing Data"
      ],
      "metadata": {
        "id": "781jZpLIpfOY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Upon research, fnlwgt is just how many people fall under those catagories, so we should be able to disregard that in our model, since other feaures determine the income, not how many people fall in this catagory\n",
        "\n",
        "https://www.kaggle.com/datasets/uciml/adult-census-income/discussion/32698\n"
      ],
      "metadata": {
        "id": "bRijGlHdunOU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = df.drop('fnlwgt', axis=1)"
      ],
      "metadata": {
        "id": "-UcfmUfOu7_6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can assign gender and income into a numerical value"
      ],
      "metadata": {
        "id": "6-qehMABRCCw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.sex = df.sex.map({'Male': 0, 'Female': 1 })\n",
        "df.income = df.income.map({'<=50K': 0, '>50K': 1 })"
      ],
      "metadata": {
        "id": "Xgj1ovnERGIw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.income.value_counts(normalize=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p8Eq0RiDeZyo",
        "outputId": "99ba70f8-1bcb-487b-a442-b879f02f5b35"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    0.751078\n",
              "1    0.248922\n",
              "Name: income, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For the rest of the values, we can use one hot encoding."
      ],
      "metadata": {
        "id": "Qqf-GfoZQ_jQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##workclass"
      ],
      "metadata": {
        "id": "BRfIUO9rqzOJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.workclass.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9zMMxGZ8q0hy",
        "outputId": "04147410-5cf2-41cf-d82c-4656a1ca4251"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Private             22286\n",
              "Self-emp-not-inc     2499\n",
              "Local-gov            2067\n",
              "State-gov            1279\n",
              "Self-emp-inc         1074\n",
              "Federal-gov           943\n",
              "Without-pay            14\n",
              "Name: workclass, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "There seems to be a good distrubution of work classes"
      ],
      "metadata": {
        "id": "W6XtkL8Vq9kv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##education"
      ],
      "metadata": {
        "id": "7qPrhOjmrCdz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.education.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1My1bM2_puu4",
        "outputId": "cbf2a546-5d6a-46a3-8097-b5d82e6a35bb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "HS-grad         9840\n",
              "Some-college    6678\n",
              "Bachelors       5044\n",
              "Masters         1627\n",
              "Assoc-voc       1307\n",
              "11th            1048\n",
              "Assoc-acdm      1008\n",
              "10th             820\n",
              "7th-8th          557\n",
              "Prof-school      542\n",
              "9th              455\n",
              "12th             377\n",
              "Doctorate        375\n",
              "5th-6th          288\n",
              "1st-4th          151\n",
              "Preschool         45\n",
              "Name: education, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#16 education values, let us keep the ones that most places use as filters\n",
        "df.loc[~df.education.isin(['HS-grad', 'Bachelors', 'Masters', 'Prof-school','Doctorate']), 'education'] = \"uneducated\""
      ],
      "metadata": {
        "id": "JxTgVCCVpltd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##maritial status"
      ],
      "metadata": {
        "id": "Vuzk1BjMrHAM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.maritalStatus.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TsZOtifSrIWK",
        "outputId": "3782d8f5-09df-4611-e594-18117bf8b649"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Married-civ-spouse       14065\n",
              "Never-married             9726\n",
              "Divorced                  4214\n",
              "Separated                  939\n",
              "Widowed                    827\n",
              "Married-spouse-absent      370\n",
              "Married-AF-spouse           21\n",
              "Name: maritalStatus, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "There seems to be a good distrubution of martial status"
      ],
      "metadata": {
        "id": "pEYDM0_3s7UV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##occupation"
      ],
      "metadata": {
        "id": "hbBP4pi5tBpH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.occupation.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Jw0rDQwEtF08",
        "outputId": "b5ee38d2-14f7-46a9-84df-df0aca0f3a43"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Prof-specialty       4038\n",
              "Craft-repair         4030\n",
              "Exec-managerial      3992\n",
              "Adm-clerical         3721\n",
              "Sales                3584\n",
              "Other-service        3212\n",
              "Machine-op-inspct    1966\n",
              "Transport-moving     1572\n",
              "Handlers-cleaners    1350\n",
              "Farming-fishing       989\n",
              "Tech-support          912\n",
              "Protective-serv       644\n",
              "Priv-house-serv       143\n",
              "Armed-Forces            9\n",
              "Name: occupation, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Although there are many values for occupation, this is important to determine if salary is high"
      ],
      "metadata": {
        "id": "dRQ4pDIctUZx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Relationship,  race, sex have less than 7 values, so it is unesscary to join them more\n",
        "\n",
        "##Native Country"
      ],
      "metadata": {
        "id": "uZnbE4njtcGC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.nativeCountry.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ka_F8UFuttB2",
        "outputId": "25cc6787-b66d-4e04-9a8a-a2075e773f32"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "United-States                 27504\n",
              "Mexico                          610\n",
              "Philippines                     188\n",
              "Germany                         128\n",
              "Puerto-Rico                     109\n",
              "Canada                          107\n",
              "El-Salvador                     100\n",
              "India                           100\n",
              "Cuba                             92\n",
              "England                          86\n",
              "Jamaica                          80\n",
              "South                            71\n",
              "Italy                            68\n",
              "China                            68\n",
              "Dominican-Republic               67\n",
              "Vietnam                          64\n",
              "Guatemala                        63\n",
              "Japan                            59\n",
              "Poland                           56\n",
              "Columbia                         56\n",
              "Iran                             42\n",
              "Taiwan                           42\n",
              "Haiti                            42\n",
              "Portugal                         34\n",
              "Nicaragua                        33\n",
              "Peru                             30\n",
              "Greece                           29\n",
              "Ecuador                          27\n",
              "France                           27\n",
              "Ireland                          24\n",
              "Hong                             19\n",
              "Cambodia                         18\n",
              "Trinadad&Tobago                  18\n",
              "Thailand                         17\n",
              "Laos                             17\n",
              "Yugoslavia                       16\n",
              "Outlying-US(Guam-USVI-etc)       14\n",
              "Hungary                          13\n",
              "Honduras                         12\n",
              "Scotland                         11\n",
              "Holand-Netherlands                1\n",
              "Name: nativeCountry, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#there is a majority of Americans in the dataset, so let us split this based on whether they are in or not in the US\n",
        "df.loc[~df.nativeCountry.isin(['United-States']), 'nativeCountry'] = \"Out-of-States\"\n",
        "df.head()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "id": "FA2K4PPhtyG-",
        "outputId": "14c66918-e1b3-48be-e035-8bf7f72328b3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   age workclass   education  educationNum maritalStatus         occupation  \\\n",
              "1   82   Private     HS-grad             9       Widowed    Exec-managerial   \n",
              "3   54   Private  uneducated             4      Divorced  Machine-op-inspct   \n",
              "4   41   Private  uneducated            10     Separated     Prof-specialty   \n",
              "5   34   Private     HS-grad             9      Divorced      Other-service   \n",
              "6   38   Private  uneducated             6     Separated       Adm-clerical   \n",
              "\n",
              "    relationship   race  sex  capitalGain  capitalLoss  capitalPerWeek  \\\n",
              "1  Not-in-family  White    1            0         4356              18   \n",
              "3      Unmarried  White    1            0         3900              40   \n",
              "4      Own-child  White    1            0         3900              40   \n",
              "5      Unmarried  White    1            0         3770              45   \n",
              "6      Unmarried  White    0            0         3770              40   \n",
              "\n",
              "   nativeCountry  income  \n",
              "1  United-States       0  \n",
              "3  United-States       0  \n",
              "4  United-States       0  \n",
              "5  United-States       0  \n",
              "6  United-States       0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ea305621-57a5-4e4e-a5da-02e7dd0d9318\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>workclass</th>\n",
              "      <th>education</th>\n",
              "      <th>educationNum</th>\n",
              "      <th>maritalStatus</th>\n",
              "      <th>occupation</th>\n",
              "      <th>relationship</th>\n",
              "      <th>race</th>\n",
              "      <th>sex</th>\n",
              "      <th>capitalGain</th>\n",
              "      <th>capitalLoss</th>\n",
              "      <th>capitalPerWeek</th>\n",
              "      <th>nativeCountry</th>\n",
              "      <th>income</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>82</td>\n",
              "      <td>Private</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Widowed</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>4356</td>\n",
              "      <td>18</td>\n",
              "      <td>United-States</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>54</td>\n",
              "      <td>Private</td>\n",
              "      <td>uneducated</td>\n",
              "      <td>4</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Machine-op-inspct</td>\n",
              "      <td>Unmarried</td>\n",
              "      <td>White</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>3900</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>41</td>\n",
              "      <td>Private</td>\n",
              "      <td>uneducated</td>\n",
              "      <td>10</td>\n",
              "      <td>Separated</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Own-child</td>\n",
              "      <td>White</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>3900</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>34</td>\n",
              "      <td>Private</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Other-service</td>\n",
              "      <td>Unmarried</td>\n",
              "      <td>White</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>3770</td>\n",
              "      <td>45</td>\n",
              "      <td>United-States</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>38</td>\n",
              "      <td>Private</td>\n",
              "      <td>uneducated</td>\n",
              "      <td>6</td>\n",
              "      <td>Separated</td>\n",
              "      <td>Adm-clerical</td>\n",
              "      <td>Unmarried</td>\n",
              "      <td>White</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>3770</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ea305621-57a5-4e4e-a5da-02e7dd0d9318')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ea305621-57a5-4e4e-a5da-02e7dd0d9318 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ea305621-57a5-4e4e-a5da-02e7dd0d9318');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Performing 1 hot encoding"
      ],
      "metadata": {
        "id": "8sRumA1DuI3B"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def encode_and_bind(original_dataframe, feature_to_encode):\n",
        "    dummies = pd.get_dummies(original_dataframe[[feature_to_encode]])\n",
        "    res = pd.concat([original_dataframe, dummies], axis=1)\n",
        "    res = res.drop([feature_to_encode], axis=1)\n",
        "    return(res) "
      ],
      "metadata": {
        "id": "cFC5XyLDbemo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "features_to_encode = ['workclass', 'education', 'maritalStatus','occupation','relationship','race','nativeCountry']\n",
        "\n",
        "for feature in features_to_encode:\n",
        "    df = encode_and_bind(df, feature)\n",
        "df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 334
        },
        "id": "xrWCXALqbhEL",
        "outputId": "2ef92f3f-9756-4899-ed0c-c78c1b9ac514"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   age  educationNum  sex  capitalGain  capitalLoss  capitalPerWeek  income  \\\n",
              "1   82             9    1            0         4356              18       0   \n",
              "3   54             4    1            0         3900              40       0   \n",
              "4   41            10    1            0         3900              40       0   \n",
              "5   34             9    1            0         3770              45       0   \n",
              "6   38             6    0            0         3770              40       0   \n",
              "\n",
              "   workclass_Federal-gov  workclass_Local-gov  workclass_Private  ...  \\\n",
              "1                      0                    0                  1  ...   \n",
              "3                      0                    0                  1  ...   \n",
              "4                      0                    0                  1  ...   \n",
              "5                      0                    0                  1  ...   \n",
              "6                      0                    0                  1  ...   \n",
              "\n",
              "   relationship_Own-child  relationship_Unmarried  relationship_Wife  \\\n",
              "1                       0                       0                  0   \n",
              "3                       0                       1                  0   \n",
              "4                       1                       0                  0   \n",
              "5                       0                       1                  0   \n",
              "6                       0                       1                  0   \n",
              "\n",
              "   race_Amer-Indian-Eskimo  race_Asian-Pac-Islander  race_Black  race_Other  \\\n",
              "1                        0                        0           0           0   \n",
              "3                        0                        0           0           0   \n",
              "4                        0                        0           0           0   \n",
              "5                        0                        0           0           0   \n",
              "6                        0                        0           0           0   \n",
              "\n",
              "   race_White  nativeCountry_Out-of-States  nativeCountry_United-States  \n",
              "1           1                            0                            1  \n",
              "3           1                            0                            1  \n",
              "4           1                            0                            1  \n",
              "5           1                            0                            1  \n",
              "6           1                            0                            1  \n",
              "\n",
              "[5 rows x 54 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7ff9a952-5d8e-4f79-b305-3c841e86c30b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>educationNum</th>\n",
              "      <th>sex</th>\n",
              "      <th>capitalGain</th>\n",
              "      <th>capitalLoss</th>\n",
              "      <th>capitalPerWeek</th>\n",
              "      <th>income</th>\n",
              "      <th>workclass_Federal-gov</th>\n",
              "      <th>workclass_Local-gov</th>\n",
              "      <th>workclass_Private</th>\n",
              "      <th>...</th>\n",
              "      <th>relationship_Own-child</th>\n",
              "      <th>relationship_Unmarried</th>\n",
              "      <th>relationship_Wife</th>\n",
              "      <th>race_Amer-Indian-Eskimo</th>\n",
              "      <th>race_Asian-Pac-Islander</th>\n",
              "      <th>race_Black</th>\n",
              "      <th>race_Other</th>\n",
              "      <th>race_White</th>\n",
              "      <th>nativeCountry_Out-of-States</th>\n",
              "      <th>nativeCountry_United-States</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>82</td>\n",
              "      <td>9</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>4356</td>\n",
              "      <td>18</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>54</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>3900</td>\n",
              "      <td>40</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>41</td>\n",
              "      <td>10</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>3900</td>\n",
              "      <td>40</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>34</td>\n",
              "      <td>9</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>3770</td>\n",
              "      <td>45</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>38</td>\n",
              "      <td>6</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>3770</td>\n",
              "      <td>40</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 54 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7ff9a952-5d8e-4f79-b305-3c841e86c30b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-7ff9a952-5d8e-4f79-b305-3c841e86c30b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-7ff9a952-5d8e-4f79-b305-3c841e86c30b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gW1gXbR1cAAT",
        "outputId": "cf8f2386-7baf-4c70-cb9b-639d65d3bc3c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(30162, 54)"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##train test  and Nureal Network creation"
      ],
      "metadata": {
        "id": "b0-enZ2rvlG1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#train test split\n",
        "input = df.drop(['income'], axis=1)\n",
        "target = df.income\n",
        "\n",
        "MMScaler = MinMaxScaler()\n",
        "input = MMScaler.fit_transform(input)\n",
        "\n",
        "x_train,x_test,y_train,y_test = train_test_split(input,target,test_size= .25,random_state = 1)"
      ],
      "metadata": {
        "id": "WHIzjjYgXxbc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will implement a Multi-layered-perceptrion (in our case, 2 layers only) with Sklearn, which uses backpropagation \\"
      ],
      "metadata": {
        "id": "vZpJr2NRwjRH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.neural_network import MLPClassifier\n",
        "\n",
        "NN = MLPClassifier(hidden_layer_sizes = (50), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = True\n",
        "    )\n",
        "NN.fit(x_train,y_train)"
      ],
      "metadata": {
        "id": "al2_IWrgwyo5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6302c412-1d57-4c60-a55e-4290ad98ad52"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Iteration 1, loss = 0.39744453\n",
            "Iteration 2, loss = 0.36187519\n",
            "Iteration 3, loss = 0.35553342\n",
            "Iteration 4, loss = 0.35504572\n",
            "Iteration 5, loss = 0.34802471\n",
            "Iteration 6, loss = 0.34602822\n",
            "Iteration 7, loss = 0.34518929\n",
            "Iteration 8, loss = 0.34044204\n",
            "Iteration 9, loss = 0.34046656\n",
            "Iteration 10, loss = 0.33840083\n",
            "Iteration 11, loss = 0.34059668\n",
            "Iteration 12, loss = 0.34246049\n",
            "Iteration 13, loss = 0.33456596\n",
            "Iteration 14, loss = 0.33490715\n",
            "Iteration 15, loss = 0.33171217\n",
            "Iteration 16, loss = 0.32933607\n",
            "Iteration 17, loss = 0.33369604\n",
            "Iteration 18, loss = 0.32935630\n",
            "Iteration 19, loss = 0.33034579\n",
            "Iteration 20, loss = 0.32618158\n",
            "Iteration 21, loss = 0.32761129\n",
            "Iteration 22, loss = 0.32647170\n",
            "Iteration 23, loss = 0.32464220\n",
            "Iteration 24, loss = 0.32743506\n",
            "Iteration 25, loss = 0.32497949\n",
            "Iteration 26, loss = 0.32859802\n",
            "Iteration 27, loss = 0.32132275\n",
            "Iteration 28, loss = 0.32312169\n",
            "Iteration 29, loss = 0.32230785\n",
            "Iteration 30, loss = 0.32310254\n",
            "Iteration 31, loss = 0.32126494\n",
            "Iteration 32, loss = 0.31996875\n",
            "Iteration 33, loss = 0.32051082\n",
            "Iteration 34, loss = 0.31968170\n",
            "Iteration 35, loss = 0.32161856\n",
            "Iteration 36, loss = 0.31744553\n",
            "Iteration 37, loss = 0.31814251\n",
            "Iteration 38, loss = 0.31784962\n",
            "Iteration 39, loss = 0.31826120\n",
            "Iteration 40, loss = 0.31721746\n",
            "Iteration 41, loss = 0.31836503\n",
            "Iteration 42, loss = 0.31808000\n",
            "Iteration 43, loss = 0.31875806\n",
            "Iteration 44, loss = 0.31980508\n",
            "Iteration 45, loss = 0.31601252\n",
            "Iteration 46, loss = 0.31571070\n",
            "Iteration 47, loss = 0.31679904\n",
            "Iteration 48, loss = 0.31746958\n",
            "Iteration 49, loss = 0.31779506\n",
            "Iteration 50, loss = 0.31790173\n",
            "Iteration 51, loss = 0.31673174\n",
            "Iteration 52, loss = 0.31595053\n",
            "Iteration 53, loss = 0.31283962\n",
            "Iteration 54, loss = 0.31563773\n",
            "Iteration 55, loss = 0.31490755\n",
            "Iteration 56, loss = 0.31692400\n",
            "Iteration 57, loss = 0.31567954\n",
            "Iteration 58, loss = 0.31772147\n",
            "Iteration 59, loss = 0.31181123\n",
            "Iteration 60, loss = 0.31586905\n",
            "Iteration 61, loss = 0.31266248\n",
            "Iteration 62, loss = 0.31261068\n",
            "Iteration 63, loss = 0.31353359\n",
            "Iteration 64, loss = 0.31264404\n",
            "Iteration 65, loss = 0.31354686\n",
            "Iteration 66, loss = 0.31307755\n",
            "Iteration 67, loss = 0.31151606\n",
            "Iteration 68, loss = 0.31139939\n",
            "Iteration 69, loss = 0.31180023\n",
            "Iteration 70, loss = 0.31312100\n",
            "Iteration 71, loss = 0.30874199\n",
            "Iteration 72, loss = 0.30935861\n",
            "Iteration 73, loss = 0.31007922\n",
            "Iteration 74, loss = 0.30976377\n",
            "Iteration 75, loss = 0.30691930\n",
            "Iteration 76, loss = 0.30952200\n",
            "Iteration 77, loss = 0.30838892\n",
            "Iteration 78, loss = 0.30639136\n",
            "Iteration 79, loss = 0.30761322\n",
            "Iteration 80, loss = 0.31102459\n",
            "Iteration 81, loss = 0.30678488\n",
            "Iteration 82, loss = 0.31097771\n",
            "Iteration 83, loss = 0.31827350\n",
            "Iteration 84, loss = 0.31662264\n",
            "Iteration 85, loss = 0.31554284\n",
            "Iteration 86, loss = 0.31431845\n",
            "Iteration 87, loss = 0.31147717\n",
            "Iteration 88, loss = 0.31141548\n",
            "Iteration 89, loss = 0.31061519\n",
            "Training loss did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MLPClassifier(alpha=0.001, hidden_layer_sizes=50, learning_rate_init=0.9,\n",
              "              solver='sgd', verbose=True)"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y_predict = NN.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "accuracy"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cCWHtqlgbMxF",
        "outputId": "46fa829f-4fe1-4d80-d2a6-48a7247ea1b3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8386155682270255"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Task 3\n",
        "\n",
        "My initial hyperparameters are above. I used 50 nodes on the first hidden layer to approximatly match the number of attributes, with the standard learning rate of .9 as a baseline\n",
        "\n",
        "As you can see, the initial accuracy with mostly standard is a respectable 83% accuracy. \n",
        "\n",
        "Below are some variations I thought of either while creating the initial hyperparameters, or after finishing my first implementation of the Nueral Networks\n",
        "\n",
        "(I have turned off the verbose feature as to prevent unesscary printing)"
      ],
      "metadata": {
        "id": "4SrpBtIAdRLy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 1) use Adam instead of SGD\n",
        "NN1 = MLPClassifier(\n",
        "    hidden_layer_sizes = (50), \n",
        "    activation = 'relu',\n",
        "    solver = 'adam',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN1.fit(x_train,y_train)\n",
        "y_predict = NN1.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oco5ez4sLbNN",
        "outputId": "0d747b27-e482-4ba3-ecd1-483bd4e9c9e1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.7479114175838748\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 2) use a different activation function\n",
        "NN2 = MLPClassifier(\n",
        "    hidden_layer_sizes = (50), \n",
        "    activation = 'logistic',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN2.fit(x_train,y_train)\n",
        "y_predict = NN2.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tDYCt7IzLf8L",
        "outputId": "b5623df2-e39f-4fcc-b976-ce7aefea3d78"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8448481633735578\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:696: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
            "  ConvergenceWarning,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 3) Adam + different activation function\n",
        "NN3 = MLPClassifier(\n",
        "    hidden_layer_sizes = (50), \n",
        "    activation = 'logistic',\n",
        "    solver = 'adam',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN3.fit(x_train,y_train)\n",
        "y_predict = NN3.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ANzeuiGALjTE",
        "outputId": "877901be-3667-434e-caf0-95cbe0fade68"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.821111258453786\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 4) 2 layers, 50,25\n",
        "NN4 = MLPClassifier(\n",
        "    hidden_layer_sizes = (50,25), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN4.fit(x_train,y_train)\n",
        "y_predict = NN4.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4cCwZGLEIGyx",
        "outputId": "1485ef4f-a2dd-40d9-bc83-5301cb014135"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.7708526720594086\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 5) 2 layers, learning rate is lower\n",
        "NN5 = MLPClassifier(\n",
        "    hidden_layer_sizes = (50), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .6,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN5.fit(x_train,y_train)\n",
        "y_predict = NN5.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GF1m8UatIQw6",
        "outputId": "91f64986-8c3f-476d-dad0-25ee87bd6684"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8371568757459223\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 6) 2 layers, 50-> 25 nodes\n",
        "NN5 = MLPClassifier(\n",
        "    hidden_layer_sizes = (50,25), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    ) \n",
        "NN5.fit(x_train,y_train)\n",
        "y_predict = NN5.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pOgGz0qdIJJH",
        "outputId": "b9c421c0-b936-4909-ebc0-d2277f315c68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8364938337090572\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 7) 4 layers, each 12 nodes\n",
        "NN7 = MLPClassifier(\n",
        "    hidden_layer_sizes = (12,12,12,12), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN7.fit(x_train,y_train)\n",
        "y_predict = NN7.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "39lHiOymIUjq",
        "outputId": "413d1f6b-e5ab-48ba-a1fa-918ef34d4228"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.7479114175838748\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 8) 4 layers, 12 nodes, alpha is lower (2 + 4)\n",
        "NN8 = MLPClassifier(\n",
        "    hidden_layer_sizes = (12,12,12,12), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .6,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN8.fit(x_train,y_train)\n",
        "y_predict = NN8.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qqiK8KGJImZZ",
        "outputId": "90ff20dd-ca38-4943-cf99-6c828a37b8fb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.7479114175838748\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 9) 4 layers, 50 -> 25 -> 12 -> 6\n",
        "NN9 = MLPClassifier(\n",
        "    hidden_layer_sizes = (50,25,12,6), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN9.fit(x_train,y_train)\n",
        "y_predict = NN9.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BbTLTQTVItHC",
        "outputId": "c70c02bf-2e3a-4535-9ada-213b14661ac1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.7479114175838748\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Variation 10) 2 layers, node count is 25\n",
        "NN10 = MLPClassifier(\n",
        "    hidden_layer_sizes = (25), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN10.fit(x_train,y_train)\n",
        "y_predict = NN10.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0L_6WiP8JGx5",
        "outputId": "5ab2a470-2d19-49f7-8a99-2282a98f4426"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8417981700039783\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Alteration 11) 2 layers, node count is 65\n",
        "NN11 = MLPClassifier(\n",
        "    hidden_layer_sizes = (65), \n",
        "    activation = 'relu',\n",
        "    solver = 'sgd',\n",
        "    alpha = .001,\n",
        "    batch_size = 'auto',\n",
        "    learning_rate_init = .9,\n",
        "    max_iter = 200,\n",
        "    verbose = False\n",
        "    )\n",
        "NN11.fit(x_train,y_train)\n",
        "y_predict = NN11.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mQ8fqxoWJNe4",
        "outputId": "c8c14a08-2c2d-4a00-c05c-26a32089b99b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8431242540777085\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here is a compliation of the accuracy score using Alternatives:\n",
        "\n",
        "------------------------------------------\n",
        "Orginal Accuracy:83.86%\n",
        "\n",
        "-------------------------\n",
        "\n",
        "1) Accuracy: 74.79%\n",
        "\n",
        "2) Accuracy: 84.69%\n",
        "\n",
        "3) Accuracy:76.84%\n",
        "\n",
        "4) Accuracy:77.08%\n",
        "\n",
        "5) Accuracy: 83.715\n",
        "\n",
        "6) Accuracy:83.64%\n",
        "\n",
        "7) Accuracy: 74.79%\n",
        "\n",
        "8) Accuracy:74.79%\n",
        "\n",
        "9) Accuracy: 74.79%\n",
        "\n",
        "10) Accuracy: 84.17%\n",
        "\n",
        "11) Accuracy:84.3%\n",
        "\n",
        "The issue with the models with multiple layers, is that the data is mostly likely overfitting on the training data, and thus is performing worse on the test data.\n",
        "\n",
        "The activation function being a sigmoid instead of relu didn't affect much, but the change from scholastic gradient descent to adam had a huge negative impact on the accuracy of the model\n",
        "\n"
      ],
      "metadata": {
        "id": "qc42Hq4WIZ86"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Task 4 \n",
        "###Using another model \n",
        "\n",
        "I will use linear regression model on the same problem and compare the accuracy on this dataset to the accuracy of our Neural Networks (and the variations  "
      ],
      "metadata": {
        "id": "OBh61qAP5W5U"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "Regress = LogisticRegression(max_iter = 200)\n",
        "\n",
        "Regress.fit(x_train,y_train)\n",
        "\n",
        "y_predict = Regress.predict(x_test)\n",
        "\n",
        "accuracy = accuracy_score(y_test, y_predict)\n",
        "print(accuracy)"
      ],
      "metadata": {
        "id": "7H7dfZspHwnA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9f21406c-cdab-4c70-b1af-26cfb38b87e3"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8460416390399151\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we can see, logistic regression has a very comparable accuracy level to the higher end of Neural networks. The only change made to the regression is that I increased the maximum Iteration to match those of the Neural Networks inital maximum epochs. Without it, the regression does not coverge by the time the 100th iteration is ran.\n",
        "\n",
        "In this case, there isn't as many features that Logistic Regression cannot handle compared to Neural networks, but it does take more iterations to acheive the same level of accuracy, which is a big consideration in datasets with much more information."
      ],
      "metadata": {
        "id": "toXTrD7YHvjg"
      }
    }
  ]
}