week4-pandas-and-kaggle/Kaggle Hack Session Starter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Today, we have our first ever Kaggle Hack Session! We're going to be competing in the Titantic competition. The goal in this competition is to be able to predict who survived and who passed away during this tragedy, given information about the people involved.\n",
    "\n",
    "We know that getting started for one of these competitions can be difficult, so we've provided this starter notebook to help you get up and running. Let's think about what we need to do when approaching any machine learning competition/problem. \n",
    "\n",
    "1) Determine your problem space. Do you have a classification problem, or a regression problem?\n",
    "\n",
    "2) Determine what model you want to use (Always good to start off with simple models).\n",
    "\n",
    "3) Load in and preprocess your dataset. Examine your database to see if there are any NULL or non-numeric values.\n",
    "\n",
    "4) Split up your dataset into training and testing components. \n",
    "\n",
    "5) Create your model. This entails defining your function, your placeholders, the loss function, and the optimizer. \n",
    "\n",
    "6) Train, evaluate, and iterate on your model!\n",
    "\n",
    "7) Once you have a model that you're satisfied with, load in test.csv (the test set for the Titanic competition), compute your predictions, save them to a CSV file, and submit to Kaggle. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Load in Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can download the data from the Kaggle website. The direct link is [here](https://www.kaggle.com/c/titanic/data), but we've already downloaded it for you. It's located in the Data subfolder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Use the Pandas read_csv() function to load in the train.csv\n",
    "titanicTrain = pd.read_csv('Data/train.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Examine Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the head function to see how the first couple rows of the dataframe looks like\n",
    "titanicTrain.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>32.204208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>49.693429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.910400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.454200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>31.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
       "count   891.000000  891.000000  891.000000  714.000000  891.000000   \n",
       "mean    446.000000    0.383838    2.308642   29.699118    0.523008   \n",
       "std     257.353842    0.486592    0.836071   14.526497    1.102743   \n",
       "min       1.000000    0.000000    1.000000    0.420000    0.000000   \n",
       "25%     223.500000    0.000000    2.000000   20.125000    0.000000   \n",
       "50%     446.000000    0.000000    3.000000   28.000000    0.000000   \n",
       "75%     668.500000    1.000000    3.000000   38.000000    1.000000   \n",
       "max     891.000000    1.000000    3.000000   80.000000    8.000000   \n",
       "\n",
       "            Parch        Fare  \n",
       "count  891.000000  891.000000  \n",
       "mean     0.381594   32.204208  \n",
       "std      0.806057   49.693429  \n",
       "min      0.000000    0.000000  \n",
       "25%      0.000000    7.910400  \n",
       "50%      0.000000   14.454200  \n",
       "75%      0.000000   31.000000  \n",
       "max      6.000000  512.329200  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure out what the different column names are\n",
    "titanicTrain.columns.tolist()\n",
    "titanicTrain.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Use other functions such as describe, max, mean, value_counts, etc to learn more about the dataset you're dealing with. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Clean Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is one of the most important parts of any machine learning pipeline. We want to make sure that the inputs we feed into any machine learning model are are valid, non-null, and are numerical values. To get you started with datapreprocessing, we'll show you one example of a column you may want to drop in this dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 12 columns):\n",
      "PassengerId    891 non-null int64\n",
      "Survived       891 non-null int64\n",
      "Pclass         891 non-null int64\n",
      "Name           891 non-null object\n",
      "Sex            891 non-null object\n",
      "Age            714 non-null float64\n",
      "SibSp          891 non-null int64\n",
      "Parch          891 non-null int64\n",
      "Ticket         891 non-null object\n",
      "Fare           891 non-null float64\n",
      "Cabin          204 non-null object\n",
      "Embarked       889 non-null object\n",
      "dtypes: float64(2), int64(5), object(5)\n",
      "memory usage: 83.6+ KB\n"
     ]
    }
   ],
   "source": [
    "# Visualize the data we're working with\n",
    "titanicTrain.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "So, as you can see above, some of the people don't have values for the age and cabin attributes. There are ways we can deal with this (for example, replace the null values with the median of the other values, replace them with 0, etc), but a simple method is to just drop the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Drop the column\n",
    "titanicTrain.drop(['Cabin'], axis = 1, inplace = True)\n",
    "# alternatively, if you don't wish to modify the original data structure, you can re-assign the result of.drop().\n",
    "#titanicTrain_dropped = titanicTrain.drop(['Cabin'], axis = 1) # For axis number (0 for rows and 1 for columns)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Another column that needs processing is the age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Do the preprocessing\n",
    "medianAge = titanicTrain['Age'].median()\n",
    "titanicTrain['Age'].fillna(medianAge, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, try it on your own! The functions you will probably be using are (although you're not limited to just these!):\n",
    "- [drop()](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html)\n",
    "- [fillna](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html)\n",
    "- [get_dummies()](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html)\n",
    "- [dropna()](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Embarked  \n",
       "0      0         A/5 21171   7.2500        S  \n",
       "1      0          PC 17599  71.2833        C  \n",
       "2      0  STON/O2. 3101282   7.9250        S  \n",
       "3      0            113803  53.1000        S  \n",
       "4      0            373450   8.0500        S  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanicTrain.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# TODO Find the other attributes that may give us trouble later on! Once you find these\n",
    "# columns, figure out if you just want to drop the attribute altogether or replace with \n",
    "# median, or something else!\n",
    "\n",
    "# HINT: The name attribute is something you may want to look at. We don't want strings in our ML model!\n",
    "titanicTrain.drop(['Name'], axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that you know a couple ways of dealing with null values and string values, feel free to be creative! The best way to get a more accurate machine learning model is to understand the best ways to visualize and clean your data! This is one of the most important steps in any ML pipeline. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Create Training/Testing Matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "So, now that we've made our final changes to our dataframe, we want to convert it into a matrix of numbers. We want our Y Matrix to be filled with binary labels indicating whether the person survived or not. Our X Matrix should contain all of the features that represent each individual.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 10 columns):\n",
      "PassengerId    891 non-null int64\n",
      "Survived       891 non-null int64\n",
      "Pclass         891 non-null int64\n",
      "Sex            891 non-null object\n",
      "Age            891 non-null float64\n",
      "SibSp          891 non-null int64\n",
      "Parch          891 non-null int64\n",
      "Ticket         891 non-null object\n",
      "Fare           891 non-null float64\n",
      "Embarked       889 non-null object\n",
      "dtypes: float64(2), int64(5), object(3)\n",
      "memory usage: 69.7+ KB\n"
     ]
    }
   ],
   "source": [
    "titanicTrain.info()\n",
    "titanicTrain.head()\n",
    "mapping = {\n",
    "    'female': 1,\n",
    "    'male': 0\n",
    "}\n",
    "titanicTrain['Sex'] = titanicTrain['Sex'].map(mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  Sex   Age  SibSp  Parch            Ticket  \\\n",
       "0            1         0       3    0  22.0      1      0         A/5 21171   \n",
       "1            2         1       1    1  38.0      1      0          PC 17599   \n",
       "2            3         1       3    1  26.0      0      0  STON/O2. 3101282   \n",
       "3            4         1       1    1  35.0      1      0            113803   \n",
       "4            5         0       3    0  35.0      0      0            373450   \n",
       "\n",
       "      Fare Embarked  \n",
       "0   7.2500        S  \n",
       "1  71.2833        C  \n",
       "2   7.9250        S  \n",
       "3  53.1000        S  \n",
       "4   8.0500        S  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanicTrain.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   PassengerId  Survived  Pclass  Sex   Age  SibSp  Parch            Ticket  \\\n",
      "0            1         0       3    0  22.0      1      0         A/5 21171   \n",
      "1            2         1       1    1  38.0      1      0          PC 17599   \n",
      "2            3         1       3    1  26.0      0      0  STON/O2. 3101282   \n",
      "3            4         1       1    1  35.0      1      0            113803   \n",
      "4            5         0       3    0  35.0      0      0            373450   \n",
      "\n",
      "      Fare Embarked  \n",
      "0   7.2500        S  \n",
      "1  71.2833        C  \n",
      "2   7.9250        S  \n",
      "3  53.1000        S  \n",
      "4   8.0500        S  \n",
      "(891, 6)\n",
      "(891, 1)\n",
      "[[False False False False False False]\n",
      " [False False False False False False]\n",
      " [False False False False False False]\n",
      " ..., \n",
      " [False False False False False False]\n",
      " [False False False False False False]\n",
      " [False False False False False False]]\n",
      "(891, 28)\n"
     ]
    }
   ],
   "source": [
    "# Convert to matrices. \n",
    "# TODO Add/Remove columns as you see fit\n",
    "print(titanicTrain.head())\n",
    "X = titanicTrain[['Pclass', 'Age', 'SibSp', 'Sex', 'Parch', 'Fare']].as_matrix()\n",
    "Y = titanicTrain['Survived'].as_matrix()\n",
    "Y = Y.reshape([Y.shape[0], 1]) # Reshaping from (891,) to (891,1)\n",
    "print (X.shape)\n",
    "print (Y.shape)\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "poly = PolynomialFeatures(2)\n",
    "X = poly.fit_transform(X)\n",
    "print(X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "(OPTIONAL) Remember that whenever we have a dataset, it's good practice to seperate the dataset into 2 parts, one that we will use to train the model, and one that we will use to check how our model is doing as a test/validation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(668, 28)\n",
      "(223, 28)\n",
      "(668, 1)\n",
      "[[        nan  0.82737724 -0.56573646 ..., -0.29008832 -0.29704981\n",
      "  -0.19930472]\n",
      " [        nan -1.56610693  0.66386103 ..., -0.29008832 -0.29704981\n",
      "   0.09110119]\n",
      " [        nan  0.82737724 -0.25833709 ..., -0.29008832 -0.29704981\n",
      "  -0.19871319]\n",
      " ..., \n",
      " [        nan  0.82737724 -0.1046374  ...,  1.1701868   0.36825424\n",
      "  -0.17058375]\n",
      " [        nan -1.56610693 -0.25833709 ..., -0.29008832 -0.29704981\n",
      "  -0.15036591]\n",
      " [        nan  0.82737724  0.20276197 ..., -0.29008832 -0.29704981\n",
      "  -0.1988716 ]]\n"
     ]
    }
   ],
   "source": [
    "# Divide into xTrain, yTrain, xTest, and yTest. Take the last 100 examples as test\n",
    "#numExamples = X.shape[0]\n",
    "#numTestExamples = 100\n",
    "\n",
    "#xTrain = X[:numExamples - numTestExamples] # xTrain contains the examples 0-791\n",
    "#yTrain = Y[:numExamples - numTestExamples] # yTrain contains the labels for examples 0-791\n",
    "#xTest = X[numExamples - numTestExamples:] # xTrain contains the examples 792-891\n",
    "#yTest = Y[numExamples - numTestExamples:] # yTrain contains the labels for examples 792-891\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, Y)\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "import numpy as np\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Create Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that we have all of our data loaded in and preprocessed, we can start on creating our model. This component is pretty open ended. You have the freedom to choose whichever model you'd like to create. If you need inspiration, take a look at the code for linear regression and logistic regression in the week2 and week3 folders. A few other reminders:\n",
    "\n",
    "- Think about what types of objects you'll need to create. Placeholders, variables, optimizers, etc\n",
    "- Think back to how we created the linear regression and logistic regression models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 250678.5625\n",
      "loss: 28739.490234375\n",
      "loss: 44614.37109375\n",
      "loss: 38531.62109375\n",
      "loss: 102167.78125\n",
      "loss: 101966.703125\n",
      "loss: 152069.984375\n",
      "loss: 64315.890625\n",
      "loss: 35330.796875\n",
      "loss: 63122.50390625\n",
      "loss: 16132.6201171875\n",
      "loss: 125917.9140625\n",
      "loss: 24268.763671875\n",
      "loss: 82141.109375\n",
      "loss: 46447.57421875\n",
      "loss: 7690.41162109375\n",
      "loss: 63021.59375\n",
      "loss: 250575.421875\n",
      "loss: 235694.4375\n",
      "loss: 118510.859375\n",
      "loss: 8583.3369140625\n",
      "loss: 8934.5361328125\n",
      "loss: 48401.9140625\n",
      "loss: 80947.9140625\n",
      "loss: 19777.6328125\n",
      "loss: 117079.5703125\n",
      "loss: 9368.3857421875\n",
      "loss: 126977.5546875\n",
      "loss: 46854.109375\n",
      "loss: 19823.0546875\n",
      "loss: 38916.54296875\n",
      "loss: 104362.609375\n",
      "loss: 43819.5078125\n",
      "loss: 29292.509765625\n",
      "loss: 82773.0625\n",
      "loss: 96821.3984375\n",
      "loss: 231241.609375\n",
      "loss: 58788.15625\n",
      "loss: 18894.4609375\n",
      "loss: 54967.6484375\n",
      "loss: 97964.7890625\n",
      "loss: 10475.65625\n",
      "loss: 27148.34765625\n",
      "loss: 118247.9296875\n",
      "loss: 226313.296875\n",
      "loss: 26940.796875\n",
      "loss: 223620.125\n",
      "loss: 45561.45703125\n",
      "loss: 206861.75\n",
      "loss: 48157.84375\n",
      "loss: 36913.125\n",
      "loss: 8928.0869140625\n",
      "loss: 51866.40234375\n",
      "loss: 103074.703125\n",
      "loss: 46238.33984375\n",
      "loss: 34007.1875\n",
      "loss: 200766.5625\n",
      "loss: 9973.1357421875\n",
      "loss: 16200.841796875\n",
      "loss: 10957.2578125\n",
      "loss: 50686.69921875\n",
      "loss: 206631.28125\n",
      "loss: 17606.9765625\n",
      "loss: 45384.5703125\n",
      "loss: 211457.53125\n",
      "loss: 16930.12890625\n",
      "loss: 45694.29296875\n",
      "loss: 209486.46875\n",
      "loss: 16328.705078125\n",
      "loss: 46469.88671875\n",
      "loss: 205581.296875\n",
      "loss: 44419.4609375\n",
      "loss: 208269.265625\n",
      "loss: 14647.564453125\n",
      "loss: 16388.40234375\n",
      "loss: 42964.8671875\n",
      "loss: 71500.5234375\n",
      "loss: 48931.6015625\n",
      "loss: 106241.078125\n",
      "loss: 49490.57421875\n",
      "loss: 18712.5859375\n",
      "loss: 99140.1015625\n",
      "loss: 45830.453125\n",
      "loss: 18923.31640625\n",
      "loss: 99637.4140625\n",
      "loss: 47340.515625\n",
      "loss: 19042.294921875\n",
      "loss: 98253.53125\n",
      "loss: 48093.359375\n",
      "loss: 18364.201171875\n",
      "loss: 11573.908203125\n",
      "loss: 15641.412109375\n",
      "loss: 41333.9921875\n",
      "loss: 191603.96875\n",
      "loss: 13521.6572265625\n",
      "loss: 11900.71484375\n",
      "loss: 197615.828125\n",
      "loss: 186272.0625\n",
      "loss: 18176.75\n",
      "loss: 18685.546875\n",
      "0.663677\n"
     ]
    }
   ],
   "source": [
    "# TODO Create your model here\n",
    "lr = .01 # the learning rate\n",
    "batch_size = 128 # the number of examples we will consider per iterations\n",
    "n_epochs = 2500 # the number of iterations we will do\n",
    "# TODO: Create placeholders for X (our features) and Y (our labels)\n",
    "X_hold = tf.placeholder(tf.float32, [None, X.shape[1]])\n",
    "Y_hold = tf.placeholder(tf.float32, [None, Y.shape[1]])\n",
    "\n",
    "\n",
    "w = tf.Variable(tf.truncated_normal(shape = [X.shape[1], Y.shape[1]], stddev=0.01), name = 'w')\n",
    "b = tf.Variable(tf.zeros([1, 1]), name = 'b')\n",
    "\n",
    "logits = tf.matmul(X_hold, w) + b\n",
    "loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits = logits, labels = Y_hold))\n",
    "opt = tf.train.GradientDescentOptimizer(learning_rate = 0.1).minimize(loss)\n",
    "cp = tf.equal(tf.argmax(logits, axis = 1), tf.cast(Y_hold, dtype = tf.int64))\n",
    "acc= tf.reduce_mean(tf.cast(cp, tf.float32))\n",
    "sess = tf.InteractiveSession()\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "for i in range(10000):\n",
    "    if len(y_train.shape) == 1:\n",
    "        y_train = y_train.reshape((y_train.shape[0], 1))\n",
    "    sess.run(opt, feed_dict = {X_hold: X_train, Y_hold: y_train})\n",
    "    if i % 100 == 0:\n",
    "        if len(y_train.shape) == 1:\n",
    "            y_train = y_train.reshape((y_train.shape[0], 1))\n",
    "        l = loss.eval(feed_dict = {X_hold: X_train, Y_hold: y_train})\n",
    "        print(\"loss: {}\".format(l))\n",
    "a = acc.eval(feed_dict = {X_hold: X_test, Y_hold: y_test})\n",
    "print(a)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Train Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that you've created your model by defining your computational graph, you're ready to start training the model. Remember that training model basically means that we want to run our optimizer object over different parts of our training dataset. A few other reminders:\n",
    "- Remember to create a Tensorflow session and initialize all of your variables\n",
    "- Run your optimizer object at every iteration\n",
    "- Keep track of how your model is doing every now and again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "numIterations = 1000 # Adjust this number as you see fit!\n",
    "# TODO Create session and initialize variables\n",
    "for i in range(numIterations):\n",
    "    ...\n",
    "    # TODO Run optimizer object over your data\n",
    "    # TODO Check accuracy every once in a while"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Test Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "By now, you have a trained model and you're almost ready to submit! We want to now see how our model does on data that it has never seen before. We want to compute our predictions for the test set. We will then submit these predictions to Kaggle in order to see how accurate we are. A few reminders:\n",
    "- Remember that preprocessing you did for the training dataset? You'll need to do that same preprocessing for this test set as well. \n",
    "- No need to initialize variables or anything. Everything is already trained! We just want to compute our predictions for this new set of data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "titanicTest = pd.read_csv('Data/test.csv')\n",
    "\n",
    "# TODO Do the same data preprocessing you did for the train set\n",
    "# TODO Compute the predictions for the testing set by evaluating your logits/normalized logits variables\n",
    "# TODO Check that the predictions are the correct dimensionality "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Create Kaggle Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It's very important to be familiar with the exact Kaggle submission format. We basically want to create a CSV file where the first line of the CSV has the column names '' and '' (this will be different from competition to competition). The following lines will be contain the id number for the test as well as the prediction for that example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import csv\n",
    "\n",
    "firstRow = [['id', 'pred']]\n",
    "with open(\"predictions.csv\", \"w\") as f:\n",
    "    writer = csv.writer(f)\n",
    "    writer.writerows(firstRow)\n",
    "    # TODO write the id number and the predictions you got from the last step!\n",
    "    # HINT: Using a for loop might be helpful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once you have the predictions.csv file, you can go ahead and submit to Kaggle! Great job!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}