Kaggle Days Paris -  GBDT workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "211025bc64645c80563e6f2c62e496251a4ed20e"
   },
   "source": [
    "![Kaggle Days Paris](https://kaggledays.com/wp-content/uploads/sites/2/2018/11/46508555_1939772529664297_1579296553191866368_n-1024x536.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c18aa72fa74e5da21a87608d5ac158282eb37755"
   },
   "source": [
    "# Competitive GBDT Specification and Optimization Workshop\n",
    "\n",
    "\n",
    "## Instructors\n",
    "* Luca Massaron [@lmassaron](https://www.linkedin.com/in/lmassaron/) - Data Scientist / Author / Google Developer Expert in Machine Learning \n",
    "* Pietro Marinelli [@pietro-marinelli-0098b427](https://www.linkedin.com/in/pietro-marinelli-0098b427/) - Freelance Data Scientist\n",
    "\n",
    "## About the workshop\n",
    "\n",
    "Gradient Boosting Decision Trees (GBDT) presently represent the state of the art for building predictors for flat table data. However, they seldom perform the best out-of-the-box (using default values) because of the many hyper-parameters to tune. Especially in the most recent GBDT implementations, such as LightGBM, the over-sophistication of hyper-parameters renders finding the optimal settings by hand or simple grid search difficult because of high combinatorial complexity and long running times for experiments. \n",
    "\n",
    "[Random Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (BERGSTRA, James; BENGIO, Yoshua. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 2012, 13.Feb: 281-305.) and [Bayesian Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (SNOEK, Jasper; LAROCHELLE, Hugo; ADAMS, Ryan P. Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems. 2012. p. 2951-2959) are often the answer you'll find from experts.\n",
    "\n",
    "In this workshop we demonstrate how to use different optimization approaches based on [Scikit-Optimize](https://github.com/scikit-optimize/scikit-optimize), a library built on top of NumPy, SciPy and Scikit-Learn, and we present an easy and fast approach to set them ready and usable.\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "You should be aware of the role and importance of hyper-parameter optimization in machine learning.  \n",
    "\n",
    "## Obtaining the Tutorial Material\n",
    "In order to make the workshop easily accessible, we are offering cloud access:\n",
    "* Using [Google Colab](https://colab.research.google.com/github/lmassaron/kaggledays-2019-gbdt/blob/master/Kaggle%20Days%20Paris%20-%20%20GBDT%20workshop.ipynb) \n",
    "* Using [Kaggle Kernels](https://www.kaggle.com/lucamassaron/kaggle-days-paris-gbdt-workshop)\n",
    "\n",
    "We also have a brief exercise that can be found at:\n",
    "* Using [Google Colab](https://colab.research.google.com/github/lmassaron/kaggledays-2019-gbdt/blob/master/Kaggle%20Days%20Paris%20-%20Skopt%20%2B%20CatBoost%20exercise.ipynb)\n",
    "* Using [Kaggle Kernels (with solution)](https://www.kaggle.com/lucamassaron/kaggle-days-paris-skopt-catboost-solution)\n",
    "\n",
    "The solution can be found [here](https://github.com/lmassaron/kaggledays-2019-gbdt/blob/master/Kaggle%20Days%20Paris%20-%20Skopt%20%2B%20CatBoost%20solution.ipynb).\n",
    "\n",
    "All the materials can be cloned from Github at the [kaggledays-2019-gbdt](https://github.com/lmassaron/kaggledays-2019-gbdt) repository. We also have prepared a stand-alone Windows installation using WinPython (just require us for the link).\n",
    "\n",
    "## Local installation notes\n",
    "\n",
    "In order to successfully run this workshop on your local computer, you need a Python3 installation (we suggest installing the most recent [Anaconda](https://www.anaconda.com/download/) distribution) and at least the following packages:\n",
    "\n",
    "* numpy >= 1.15.4\n",
    "* pandas >= 0.23.4\n",
    "* scipy >= 1.1.0\n",
    "* skopt >= 0.5.2\n",
    "* sklearn >= 0.20.2\n",
    "* lightgbm >= 2.2.2\n",
    "* xgboost >= 0.81\n",
    "* catboost >= 0.12.2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART I : Specifying models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "# Gradient Boosting\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "\n",
    "# Scikit-learn\n",
    "from sklearn.metrics import average_precision_score\n",
    "from sklearn.model_selection import StratifiedKFold \n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.metrics import classification_report, confusion_matrix, roc_curve, auc\n",
    "\n",
    "# Graphics\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "# Dataset\n",
    "from sklearn.datasets import load_boston"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Uploading the Boston dataset\n",
    "X, y = load_boston(return_X_y=True)\n",
    "\n",
    "# Transforming the problem into a classification (unbalanced)\n",
    "y_bin = (y > np.percentile(y, 90)).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#CRIM - per capita crime rate by town\n",
    "#ZN - proportion of residential land zoned for lots over 25,000 sq.ft.\n",
    "#INDUS - proportion of non-retail business acres per town.\n",
    "#CHAS - Charles River dummy variable (1 if tract bounds river; 0 otherwise)\n",
    "#NOX - nitric oxides concentration (parts per 10 million)\n",
    "#RM - average number of rooms per dwelling\n",
    "#AGE - proportion of owner-occupied units built prior to 1940\n",
    "#DIS - weighted distances to five Boston employment centres\n",
    "#RAD - index of accessibility to radial highways\n",
    "#TAX - full-value property-tax rate per $10,000\n",
    "#PTRATIO - pupil-teacher ratio by town\n",
    "#B - 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
    "#LSTAT - % lower status of the population\n",
    "#MEDV - Median value of owner-occupied homes in $1000's this is our target variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFSpJREFUeJzt3X+Q3PV93/HnKyAiG3CExIkRHLaE\nrTjgNkAiKC4pxWDHmBhBZuwW6tiiyKOZhri4jccBT5OWpsmYmYwdOrSZKuBGbh1+VAELaEOsUWGw\nOzYgDK4hIlXMD3NIQYeCAsQBjHj3j/3KHOeTdu9u9+70vedjZue731/7fe9nbl/7uc9+97upKiRJ\nB7+fmO0CJEn9YaBLUksY6JLUEga6JLWEgS5JLWGgS1JLGOjqiySPJjl7tuuYTUl+OcnTSV5Kcups\n16P5x0BXV0meTPL+ccsuTfKNffNV9Z6quqfL4yxPUkkOHVCps+33gF+rqiOq6qHxK5vn/uzY55/k\n0CS7ktSYZfckebl5Y9h3u6NZd3aS18csH0lyS5LTxuz/WJLLJjj+FUm29v1Za84w0NUac+CN4h3A\no1222QN8aMz8+cDzE2y3741h3+2CMet2VNURwJHAGcBjwNeTnNus3wB8YoLH/HizTi1loKsvxvbi\nk5yeZGuSF5oe6Reaze5tpnua3uV7k/xEkn+T5Kmmp/rlJD815nE/0azbneQ3xx3n3yXZmOS/J3kB\nuLQ59jeT7EmyM8l1SQ4b83iV5FeTbE/yYpLfTvLOZp8Xmt7uj7Yf9xwnrDXJTyZ5CTgE+E6S7x2g\nqf4bbw7bTwBfnmRzA1AdI1X1W8D1wDVjjvELSd4xpvYTgZ8FbpzKsXRwMNA1CNcC11bV24B3Arc0\ny89qpouaXuc3gUub2/uAE4AjgOsAkpwE/GfgY8Ay4KeA48Yd60JgI7AI+AqwF/hXwNHAe4FzgV8d\nt895wM/T6d1+FljfHON44O8Bl+zneU1Ya1W90vSYAU6uqnfuv2n4KnBWkkVJFgH/CNh0gO17dSvw\nc0kOr6oR4G46PfJ9PgH8r6p6rg/H0hxloKtXX216vXuS7KETtPvzQ+BdSY6uqpeq6lsH2PZjwBeq\n6vGqegm4Cri4GT75CHBHVX2jql4FfgsYf/Ghb1bVV6vq9ar6u6p6sKq+VVWvVdWTwH8B/vG4fa6p\nqheq6lHgEeBrzfH/BvhTYH8faB6o1l69DNwB/FPgYuD2Ztl4/3Fseyf57S6PuwMInTc26AytfBw6\n/1k0tTvc0nIGunp1UVUt2nfjx3u9Y60Ffhp4LMkDST58gG2PBZ4aM/8UcChwTLPu6X0rquoHwO5x\n+z89dibJTye5M8lfNcMwv0untz7Ws2Pu/90E80cwsQPVOhlfptNjPtBwy78c295V9ZtdHvM4Om92\ne5r5W4FlSc4AzgbeCvzPSdapg4yBrr6rqu1VdQmwlM647sYkh/PjvWvo9CzfMWb+7cBrdEJ2JzC8\nb0WStwBLxh9u3Pwf0PmQcGUz5PM5Oj3XfjhQrZPxdTpDSMcA3+iyba9+Gfh2Vf0t/OjNbyOdN42P\nAzc1/+WoxWb7rAC1UJJfAf6sqkab4RnojG2PAq/TGX/+f83yG4HfSPKnzfrfBW6uqteSbAS+leQf\nAluBq+kezkcCLwAvJfkZ4F80j9sP+611Mg9SVZXkgjH3p1RMOjseC3yyua0et8kGOj31BXQ+S1DL\n2UPXIJwHPNqc+XEtcHFVvdz0Gn8H+D/NuPAZwJfonJVxL/AEnfHkTwE0Y9yfAm6i01t/EdgFvHKA\nY38G+GfNtn8I3NzH57XfWierqh5tnt/+XDfuPPQHx6w7tmnbl4AHgL8PnF1VXxv3GPcCfwM8U1UP\nTKVOHVziD1zoYJHkCDpjxCur6onZrkeaa+yha05LckGStzZj8L8HfBd4cnarkuYmA11z3YV0Pozc\nAaykM3zjv5XSBBxykaSWsIcuSS0xo6ctHn300bV8+fKZPKQkHfQefPDB56pqqNt2Mxroy5cvZ+tW\nr94pSZOR5KnuWznkIkmtYaBLUksY6JLUEl7LRVLr/fCHP2RkZISXX57oSsVzx8KFCxkeHmbBggVT\n2t9Al9R6IyMjHHnkkSxfvpypXgxt0KqK3bt3MzIywooVK6b0GA65SGq9l19+mSVLlszZMAdIwpIl\nS6b1X4SBLmlemMthvs90azTQJaklHEOXNO9ccEF/H++OO3rb7q677uKKK65g7969fPKTn+TKK6/s\nax0Gug5oKn/4vf5xS/PJ3r17ufzyy9m8eTPDw8OcdtpprF69mpNOOqlvx3DIRZJmwP3338+73vUu\nTjjhBA477DAuvvhiNm3a1NdjGOiSNAOeeeYZjj/++B/NDw8P88wzz/T1GAa6JM2AiX57ot9n3hjo\nkjQDhoeHefrpp380PzIywrHHHtvXYxjokjQDTjvtNLZv384TTzzBq6++yk033cTq1av7egzPcpE0\n78zGmViHHnoo1113HR/84AfZu3cvl112Ge95z3v6e4y+Ppokab/OP/98zj///IE9vkMuktQSBrok\ntUTXQE/y7iQPj7m9kOTTSRYn2ZxkezM9aiYKliRNrGugV9VfVNUpVXUK8PPAD4DbgCuBLVW1EtjS\nzEuSZslkh1zOBb5XVU8BFwIbmuUbgIv6WZgkaXImG+gXAzc294+pqp0AzXRpPwuTJE1Oz6ctJjkM\nWA1cNZkDJFkHrAN4+9vfPqniJGkgZuH6uZdddhl33nknS5cu5ZFHHunv8RuT6aF/CPh2VT3bzD+b\nZBlAM9010U5Vtb6qVlXVqqGhoelVK0kHqUsvvZS77rproMeYTKBfwhvDLQC3A2ua+2uA/l4HUpJa\n5KyzzmLx4sUDPUZPgZ7krcAHgFvHLP488IEk25t1n+9/eZKkXvU0hl5VPwCWjFu2m85ZL5KkOcBv\nikpSSxjoktQSXm1R0vwzC9fPveSSS7jnnnt47rnnGB4e5uqrr2bt2rV9PYaBLkkz4MYbb+y+0TQ5\n5CJJLWGgS1JLGOiS5oWqmu0SuppujQa6pNZbuHAhu3fvntOhXlXs3r2bhQsXTvkx/FBUUusNDw8z\nMjLC6OjobJdyQAsXLmR4eHjK+xvoklpvwYIFrFixYrbLGDiHXCSpJeyhq++meqnpWfiuh9Qq9tAl\nqSUMdElqCQNdklrCQJekljDQJaklPMtFc8ZUzo7xzBjpDfbQJaklev2R6EVJNiZ5LMm2JO9NsjjJ\n5iTbm+lRgy5WkrR/vfbQrwXuqqqfAU4GtgFXAluqaiWwpZmXJM2SroGe5G3AWcANAFX1alXtAS4E\nNjSbbQAuGlSRkqTueumhnwCMAv81yUNJrk9yOHBMVe0EaKZLJ9o5ybokW5NsnetXOpOkg1kvgX4o\n8HPAH1TVqcDfMonhlapaX1WrqmrV0NDQFMuUJHXTS6CPACNVdV8zv5FOwD+bZBlAM901mBIlSb3o\nGuhV9VfA00ne3Sw6F/hz4HZgTbNsDbBpIBVKknrS6xeLPgV8JclhwOPAP6fzZnBLkrXA94GPDqZE\nSVIvegr0qnoYWDXBqnP7W44kaar8pqgktYSBLkktYaBLUksY6JLUEga6JLWEgS5JLWGgS1JLGOiS\n1BIGuiS1hIEuSS1hoEtSSxjoktQSBroktYSBLkktYaBLUksY6JLUEga6JLWEgS5JLWGgS1JL9PSb\nokmeBF4E9gKvVdWqJIuBm4HlwJPAP6mq5wdTpiSpm8n00N9XVadU1b4fi74S2FJVK4EtzbwkaZZM\nZ8jlQmBDc38DcNH0y5EkTVWvgV7A15I8mGRds+yYqtoJ0EyXTrRjknVJtibZOjo6Ov2KJUkT6mkM\nHTizqnYkWQpsTvJYrweoqvXAeoBVq1bVFGqUJPWgpx56Ve1opruA24DTgWeTLANoprsGVaQkqbuu\ngZ7k8CRH7rsP/CLwCHA7sKbZbA2waVBFSpK662XI5RjgtiT7tv/jqroryQPALUnWAt8HPjq4MiVJ\n3XQN9Kp6HDh5guW7gXMHUZQkafL8pqgktYSBLkktYaBLUksY6JLUEga6JLWEgS5JLWGgS1JLGOiS\n1BIGuiS1hIEuSS1hoEtSSxjoktQSBroktUSvv1ikg9wFF8x2BZIGzR66JLWEgS5JLWGgS1JLGOiS\n1BI9B3qSQ5I8lOTOZn5FkvuSbE9yc5LDBlemJKmbyfTQrwC2jZm/BvhiVa0EngfW9rMwSdLk9BTo\nSYaBXwKub+YDnANsbDbZAFw0iAIlSb3ptYf++8Bngdeb+SXAnqp6rZkfAY6baMck65JsTbJ1dHR0\nWsVKkvava6An+TCwq6oeHLt4gk1rov2ran1VraqqVUNDQ1MsU5LUTS/fFD0TWJ3kfGAh8DY6PfZF\nSQ5teunDwI7BlSlJ6qZroFfVVcBVAEnOBj5TVR9L8j+AjwA3AWuATQOss7Wm8pX8O+7ofx2SDn7T\nOQ/9N4B/neQv6Yyp39CfkiRJUzGpi3NV1T3APc39x4HT+1+SJGkq/KaoJLWEgS5JLWGgS1JLGOiS\n1BIGuiS1hIEuSS1hoEtSSxjoktQSk/pikeaGqVwuQFL72UOXpJYw0CWpJQx0SWoJA12SWsJAl6SW\nMNAlqSUMdElqCQNdklrCQJekljDQJaklugZ6koVJ7k/ynSSPJrm6Wb4iyX1Jtie5Oclhgy9XkrQ/\nvfTQXwHOqaqTgVOA85KcAVwDfLGqVgLPA2sHV6YkqZuugV4dLzWzC5pbAecAG5vlG4CLBlKhJKkn\nPY2hJzkkycPALmAz8D1gT1W91mwyAhy3n33XJdmaZOvo6Gg/apYkTaCnQK+qvVV1CjAMnA6cONFm\n+9l3fVWtqqpVQ0NDU69UknRAkzrLpar2APcAZwCLkuy7nvowsKO/pUmSJqOXs1yGkixq7r8FeD+w\nDbgb+Eiz2Rpg06CKlCR118svFi0DNiQ5hM4bwC1VdWeSPwduSvIfgIeAGwZYpySpi66BXlX/Fzh1\nguWP0xlPlyTNAX5TVJJawkCXpJYw0CWpJQx0SWoJA12SWsJAl6SWMNAlqSUMdElqCQNdklrCQJek\nljDQJaklDHRJagkDXZJaopfL50pz1gUXTH6fO+7ofx3SXGAPXZJawkCXpJYw0CWpJQx0SWqJXn4k\n+vgkdyfZluTRJFc0yxcn2ZxkezM9avDlSpL2p5ce+mvAr1fVicAZwOVJTgKuBLZU1UpgSzMvSZol\nXQO9qnZW1beb+y8C24DjgAuBDc1mG4CLBlWkJKm7SY2hJ1kOnArcBxxTVTuhE/rA0n4XJ0nqXc+B\nnuQI4E+AT1fVC5PYb12SrUm2jo6OTqVGSVIPegr0JAvohPlXqurWZvGzSZY165cBuybat6rWV9Wq\nqlo1NDTUj5olSRPo+tX/JAFuALZV1RfGrLodWAN8vpluGkiFUp9N5XIB4CUDNMZk/4hm6I+nl2u5\nnAl8HPhukoebZZ+jE+S3JFkLfB/46GBKlCT1omugV9U3gOxn9bn9LUeSNFVebbFPpvpvvCT1i1/9\nl6SWMNAlqSVaPeTi2QyS5hN76JLUEga6JLVEq4dcpH7y90s119lDl6SWsIcuDZAfzGsm2UOXpJYw\n0CWpJQx0SWoJA12SWsJAl6SW8CyXCXjlRB20PFl+XrOHLkktYaBLUks45CLNQY6caCrsoUtSS3QN\n9CRfSrIrySNjli1OsjnJ9mZ61GDLlCR100sP/Y+A88YtuxLYUlUrgS3NvCRpFnUN9Kq6F/jrcYsv\nBDY09zcAF/W5LknSJE11DP2YqtoJ0EyX7m/DJOuSbE2ydXR0dIqHkyR1M/APRatqfVWtqqpVQ0ND\ngz6cJM1bUw30Z5MsA2imu/pXkiRpKqYa6LcDa5r7a4BN/SlHkjRVvZy2eCPwTeDdSUaSrAU+D3wg\nyXbgA828JGkWdf2maFVdsp9V5/a5FknSNPhNUUlqCQNdklrCQJekljDQJaklDHRJagkDXZJawkCX\npJYw0CWpJfwJOmm+m6u/dzdX65rD7KFLUksY6JLUEgfNkMtU/vuSpPnEHroktYSBLkktcdAMuUg6\nyM3EuOk8PzPGHroktYQ9dEmT51kKc5I9dElqCQNdklpiWkMuSc4DrgUOAa6vKn8sWtLBpUXDR1Pu\noSc5BPhPwIeAk4BLkpzUr8IkSZMznSGX04G/rKrHq+pV4Cbgwv6UJUmarOkMuRwHPD1mfgT4B+M3\nSrIOWNfMvpTkL6ZxzLngaOC52S5ijrAt3mxW2yOZrSPvl38f+yTTbYt39LLRdAJ9oj+f+rEFVeuB\n9dM4zpySZGtVrZrtOuYC2+LNbI83sz3eMFNtMZ0hlxHg+DHzw8CO6ZUjSZqq6QT6A8DKJCuSHAZc\nDNzen7IkSZM15SGXqnotya8Bf0bntMUvVdWjfats7mrN8FEf2BZvZnu8me3xhhlpi1T92LC3JOkg\n5DdFJaklDHRJagkD/QCSfCnJriSPjFm2OMnmJNub6VGzWeNMSXJ8kruTbEvyaJIrmuXztT0WJrk/\nyXea9ri6Wb4iyX1Ne9zcnDAwLyQ5JMlDSe5s5udzWzyZ5LtJHk6ytVk28NeKgX5gfwScN27ZlcCW\nqloJbGnm54PXgF+vqhOBM4DLm0s9zNf2eAU4p6pOBk4BzktyBnAN8MWmPZ4H1s5ijTPtCmDbmPn5\n3BYA76uqU8acfz7w14qBfgBVdS/w1+MWXwhsaO5vAC6a0aJmSVXtrKpvN/dfpPPCPY752x5VVS81\nswuaWwHnABub5fOmPZIMA78EXN/Mh3naFgcw8NeKgT55x1TVTuiEHLB0luuZcUmWA6cC9zGP26MZ\nYngY2AVsBr4H7Kmq15pNRui86c0Hvw98Fni9mV/C/G0L6Ly5fy3Jg83lT2AGXiv+YpEmJckRwJ8A\nn66qFzIHLyAyU6pqL3BKkkXAbcCJE202s1XNvCQfBnZV1YNJzt63eIJNW98WY5xZVTuSLAU2J3ls\nJg5qD33ynk2yDKCZ7prlemZMkgV0wvwrVXVrs3jetsc+VbUHuIfOZwuLkuzrKM2Xy2GcCaxO8iSd\nq66eQ6fHPh/bAoCq2tFMd9F5sz+dGXitGOiTdzuwprm/Btg0i7XMmGZM9AZgW1V9Ycyq+doeQ03P\nnCRvAd5P53OFu4GPNJvNi/aoqquqariqltO5BMj/rqqPMQ/bAiDJ4UmO3Hcf+EXgEWbgteI3RQ8g\nyY3A2XQuA/os8G+BrwK3AG8Hvg98tKrGf3DaOkl+Afg68F3eGCf9HJ1x9PnYHj9L54OtQ+h0jG6p\nqn+f5AQ6vdTFwEPAr1TVK7NX6cxqhlw+U1Ufnq9t0Tzv25rZQ4E/rqrfSbKEAb9WDHRJagmHXCSp\nJQx0SWoJA12SWsJAl6SWMNAlqSUMdElqCQNdklri/wNZrCGTv5PdRQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1ea41082da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Histogram highlighting the top 10% we use as a target\n",
    "plt.hist(y[y <= np.percentile(y, 90)], bins='auto', alpha=0.7, label='0', color='b')\n",
    "plt.hist(y[y > np.percentile(y, 90)], bins=8, alpha=0.7, label='1', color='r')\n",
    "plt.title(\"Histogram of MEDV\")\n",
    "plt.legend(loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# For convenience, we will create a Pandas dataframe from X\n",
    "train = pd.DataFrame(X)\n",
    "train = train.add_prefix('var_')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(506, 13)\n"
     ]
    }
   ],
   "source": [
    "# Checking about the shape of our training set\n",
    "print(train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Setting a 5-fold stratified cross-validation (note: shuffle=True)\n",
    "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A brief introduction about the key models we will be using during this works.\n",
    "source : BOSCHETTI, Alberto; MASSARON, Luca. Python data science essentials. Packt Publishing Ltd, 3rd ed., 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Tree Boosting\n",
    "\n",
    "Gradient Tree boosting or Gradient Boosting Decision Trees (GBDT) is another improved version of boosting (fitting a sequence of weak learners on reweighted versions of the data). Like AdaBoost, GBDT is based on a gradient descent function. The algorithm has proven to be one of the most proficient ones from the ensemble, though it is characterized by an increased variance of estimates, more sensibility to noise in data (both problems could be attenuated by using sub-sampling), and significant computational costs due to nonparallel operations.\n",
    "\n",
    "Apart from deep learning, gradient boosting is actually the most developed machine learning algorithm. Since Adaboost and the following Gradient Boosting implementation as developed by Jerome Friedman, there appeared various implementations of the algorithms, the most recent ones being XGBoost, LightGBM, and CatBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LightGbm\n",
    "The high-performance [LightGBM](https://github.com/Microsoft/LightGBM) algorithm is capable of being distributed and of fast-handling large amounts of data. It has been developed by a team at Microsoft as an open source project on GitHub (there is also an [academic paper](https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficientgradient-boosting-decision-tree)). \n",
    "\n",
    "LightGBM is based on decision trees, as well as XGBoost, yet it follows a different strategy.\n",
    "Whereas XGBoost uses decision trees to split on a variable and exploring different cuts at that variable (the level-wise tree growth strategy), LightGBM concentrates on a split and goes on splitting from there in order to achieve a better fitting (this is the leaf-wise tree\n",
    "growth strategy). This allows LightGBM to reach first and fast a good fit of the data, and to generate alternative solutions compared to XGBoost (which is good, if you expect to blend, i.e. average, the two solutions together in order to reduce the variance of the estimated). Algorithmically talking, figuring out as a graph the structures of cuts operated by a decision tree, XGBoost peruses a breadth-first search (BFS), whereas LightGBM a depthfirst search (DFS).\n",
    "\n",
    "Tuning LightGBM may appear daunting with more than a [hundred parameters](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst) (also to be found [here](https://lightgbm.readthedocs.io/en/latest/Parameters.html)) to tune."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XGBoost\n",
    "\n",
    "[XGBoost](https://github.com/dmlc/XGBoost) stands for eXtreme Gradient Boosting, an open source project that is not part of Scikit-learn, though recently it has been expanded by a Scikit-Learn wrapper interface that renders using models based on XGBoost more integrated into your data pipeline.\n",
    "\n",
    "The XGBoost algorithm has gained recently gained momentum and popularity in datascience competitions such as Kaggle and the KDD-cup 2015. As the authors (Tianqui Chen, Tong He, and Carlos Guestrin) report on papers they wrote on the algorithm, among 29 challenges held on Kaggle during 2015, 17 winning solutions used XGBoost as a standalone solution or as part of an ensemble of multiple different models.\n",
    "\n",
    "Apart from the successful performances in both accuracy and computational efficiency, XGBoost is also a scalable solution under different points of view. XGBoost represents a new generation of GBM algorithms thanks to important tweaks to the initial tree boost GBM algorithm:\n",
    "\n",
    "* A sparse-aware algorithm; it can leverage sparse matrices, saving both memory (no need for dense matrices) and computation time (zero values are handled in a special way).\n",
    "* Approximate tree learning (weighted quantile sketch), which bears similar results but in much less time than the classical complete explorations of possible branch cuts.\n",
    "* Parallel computing on a single machine (using multi-threading in the phase of the search for the best split) and similarly distributed computations on multiple ones. \n",
    "* Out-of-core computations on a single machine leveraging a data storage solution called Column Block. This arranges data on a disk by columns, thus saving time by pulling data from the disk as the optimization algorithm (which works on column vectors) expects it.\n",
    "* XGBoost can also deal with missing data in an effective way. Other tree ensembles based on standard decision trees require missing data first to be imputed using an off-scale value, such as a negative number, in order to develop an appropriate branching of the tree to deal with missing values.\n",
    "\n",
    "As for as XGBoost's [parameters](https://xgboost.readthedocs.io/en/latest/parameter.html), we have decided to work on a few key ones you will find across competitions and projects.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CatBoost\n",
    "In July 2017, another interesting GBM algorithm was made public by Yandex, the Russian search engine: it is [CatBoost](https://catboost.yandex/), whose name comes from putting together the two words Category and Boosting. In fact, its strongest point is the capability of handling categorical variables, which actually make the most of information in most relational databases, by adopting a mixed strategy of one-hot-encoding and mean encoding (a way to express categorical levels by assigning them an appropriate numeric value for the problem at hand; more on that later).\n",
    "\n",
    "The idea used by CatBoost to encode the categorical variables is not new, but it has been a kind of feature engineering used various times, mostly in data science competitions like at Kaggle’s. Mean encoding, also known as likelihood encoding, impact coding, or target coding, is simply a way to transform your labels into a number based on their association with the target variable. If you have a regression, you could transform labels based on the mean target value typical of that level; if it is a classification, it is simply the probability of classification of your target given that label (probability of your target, conditional on each category value). It may appear as a simple and smart feature engineering trick, but actually, it has side effects, mostly in terms of overfitting because you are taking information from the target into your predictors.\n",
    "\n",
    "CatBoost has quite a few [parameters](https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_parameters-list-docpage/#python-reference_parameters-list), we have delimited our search to the 9 most important ones. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 2000\n",
    "lgb_iter1 = []\n",
    "sklearn_gbm_iter1 = []\n",
    "xgb_gbm_iter1 = []\n",
    "\n",
    "lgb_ap1 = []\n",
    "sklearn_gbm_ap1 = []\n",
    "xgb_gbm_ap1 = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set up the classifier with standard configuration\n",
    "# Later we will more performing parameters with Bayesian Optimization\n",
    "params = {\n",
    "    'learning_rate':  0.06, \n",
    "    'max_depth': 6, \n",
    "    #'lambda_l1': 16.7,\n",
    "    'min_data_in_leaf':5,\n",
    "    'boosting': 'gbdt', \n",
    "    'objective': 'binary', \n",
    "    'metric': 'auc',\n",
    "    'feature_fraction': .9,\n",
    "    'is_training_metric': False, \n",
    "    'seed': 1\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fold  0\n",
      " Best iteration lgb =  12\n",
      " Best iteration sklearn_gbm =  63\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.8092050493366283\n",
      "sklearn_gbn  0.35953654188948314\n",
      "xgboost  0.27641965877259994\n",
      "\n",
      "Fold  1\n",
      " Best iteration lgb =  64\n",
      " Best iteration sklearn_gbm =  71\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.7724999999999999\n",
      "sklearn_gbn  0.37940594059405947\n",
      "xgboost  0.4661716171617162\n",
      "\n",
      "Fold  2\n",
      " Best iteration lgb =  118\n",
      " Best iteration sklearn_gbm =  137\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.9022333222333222\n",
      "sklearn_gbn  0.6422029702970297\n",
      "xgboost  0.8333333333333334\n",
      "\n",
      "Fold  3\n",
      " Best iteration lgb =  5\n",
      " Best iteration sklearn_gbm =  98\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  1.0\n",
      "sklearn_gbn  0.81990099009901\n",
      "xgboost  0.9090909090909091\n",
      "\n",
      "Fold  4\n",
      " Best iteration lgb =  15\n",
      " Best iteration sklearn_gbm =  74\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.9500000000000001\n",
      "sklearn_gbn  0.6396039603960396\n",
      "xgboost  0.730913091309131\n"
     ]
    }
   ],
   "source": [
    "for i, (train_index, test_index) in enumerate(skf.split(train,y_bin)):\n",
    "    \n",
    "    # Create data for this fold\n",
    "    y_train, y_valid = y_bin[train_index], y_bin[test_index]\n",
    "    X_train, X_valid = train.iloc[train_index,:], train.iloc[test_index,:]\n",
    "        \n",
    "    print( \"\\nFold \", i)\n",
    "\n",
    "    # Running models for this fold\n",
    "    \n",
    "    # ->LightGBM\n",
    "    lgb_gbm = lgb.train(params, \n",
    "                          lgb.Dataset(X_train, label=y_train), \n",
    "                          MAX_ROUNDS, \n",
    "                          lgb.Dataset(X_valid, label=y_valid), \n",
    "                          verbose_eval=False, \n",
    "                          #feval= auc, \n",
    "                          early_stopping_rounds=50)\n",
    "    \n",
    "    print( \" Best iteration lgb = \", lgb_gbm.best_iteration)\n",
    "    \n",
    "    # ->Scikit-learn GBM\n",
    "    sklearn_gbm = GradientBoostingClassifier(n_estimators=MAX_ROUNDS, \n",
    "                                    learning_rate = 0.06,\n",
    "                                    max_features=2, \n",
    "                                    max_depth = 6, \n",
    "                                    n_iter_no_change=50, \n",
    "                                    tol=0.01,\n",
    "                                    random_state = 0)\n",
    "    \n",
    "    sklearn_gbm.fit(X_train, y_train)\n",
    "    print( \" Best iteration sklearn_gbm = \", sklearn_gbm.n_estimators_)\n",
    "    \n",
    "    # ->XGBoost\n",
    "    xgb_gbm = xgb.XGBClassifier(max_depth=6, \n",
    "                                n_estimators=MAX_ROUNDS,\n",
    "                                eval_set=[(X_train, y_train), (X_valid, y_valid)],\n",
    "                                learning_rate=0.06,\n",
    "                                early_stopping_rounds=50)\n",
    "\n",
    "    xgb_gbm.fit(X_train, y_train)\n",
    "    \n",
    "    print( \" Best iteration xgboost_gbm = \", xgb_gbm.get_booster().best_iteration)\n",
    "    \n",
    "    # Storing and reporting results of the fold\n",
    "    lgb_iter1 = np.append(lgb_iter1, lgb_gbm.best_iteration)\n",
    "    sklearn_gbm_iter1 = np.append(sklearn_gbm_iter1, sklearn_gbm.n_estimators_)\n",
    "    xgb_gbm_iter1 = np.append(xgb_gbm_iter1, xgb_gbm.get_booster().best_iteration)\n",
    "   \n",
    "    pred = lgb_gbm.predict(X_valid, num_iteration=lgb_gbm.best_iteration)\n",
    "    ap = average_precision_score(y_valid, pred, average='macro', pos_label=1, sample_weight=None)\n",
    "    print('lgb ', ap)\n",
    "    lgb_ap1 = np.append(lgb_ap1, ap)\n",
    "    \n",
    "    pred = sklearn_gbm.predict(X_valid)\n",
    "    ap = average_precision_score(y_valid, pred, average='macro', pos_label=1, sample_weight=None)\n",
    "    print('sklearn_gbn ', ap)\n",
    "    sklearn_gbm_ap1 = np.append(sklearn_gbm_ap1, ap)\n",
    "    \n",
    "    pred  = xgb_gbm.predict(X_valid)\n",
    "    ap = average_precision_score(y_valid, pred, average='macro', pos_label=1, sample_weight=None)\n",
    "    print('xgboost ', ap)\n",
    "    xgb_gbm_ap1 = np.append(xgb_gbm_ap1, ap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lgb_iter1:  42.8\n",
      "sklearn_gbm_iter1:  88.6\n",
      "xgb_gbm_iter1:  1999.0\n",
      "lgb_ap1:  0.8867876743139901\n",
      "sklearn_gbm_ap1:  0.5681300806551244\n",
      "xgb_gbm_ap1:  0.643185721933538\n"
     ]
    }
   ],
   "source": [
    "print('lgb_iter1: ', np.mean(lgb_iter1))\n",
    "print('sklearn_gbm_iter1: ', np.mean(sklearn_gbm_iter1))\n",
    "print('xgb_gbm_iter1: ',np.mean(xgb_gbm_iter1))\n",
    "\n",
    "print('lgb_ap1: ', np.mean(lgb_ap1))\n",
    "print('sklearn_gbm_ap1: ', np.mean(sklearn_gbm_ap1))\n",
    "print('xgb_gbm_ap1: ', np.mean(xgb_gbm_ap1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "poly = PolynomialFeatures(2)\n",
    "poly_train = poly.fit_transform(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "poly_train = pd.DataFrame(poly_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>100</th>\n",
       "      <th>101</th>\n",
       "      <th>102</th>\n",
       "      <th>103</th>\n",
       "      <th>104</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>87616.0</td>\n",
       "      <td>4528.8</td>\n",
       "      <td>117482.40</td>\n",
       "      <td>1474.08</td>\n",
       "      <td>234.09</td>\n",
       "      <td>6072.570</td>\n",
       "      <td>76.194</td>\n",
       "      <td>157529.6100</td>\n",
       "      <td>1976.5620</td>\n",
       "      <td>24.8004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>58564.0</td>\n",
       "      <td>4307.6</td>\n",
       "      <td>96049.80</td>\n",
       "      <td>2211.88</td>\n",
       "      <td>316.84</td>\n",
       "      <td>7064.820</td>\n",
       "      <td>162.692</td>\n",
       "      <td>157529.6100</td>\n",
       "      <td>3627.6660</td>\n",
       "      <td>83.5396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>58564.0</td>\n",
       "      <td>4307.6</td>\n",
       "      <td>95064.86</td>\n",
       "      <td>975.26</td>\n",
       "      <td>316.84</td>\n",
       "      <td>6992.374</td>\n",
       "      <td>71.734</td>\n",
       "      <td>154315.4089</td>\n",
       "      <td>1583.1049</td>\n",
       "      <td>16.2409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>49284.0</td>\n",
       "      <td>4151.4</td>\n",
       "      <td>87607.86</td>\n",
       "      <td>652.68</td>\n",
       "      <td>349.69</td>\n",
       "      <td>7379.581</td>\n",
       "      <td>54.978</td>\n",
       "      <td>155732.8369</td>\n",
       "      <td>1160.2122</td>\n",
       "      <td>8.6436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>49284.0</td>\n",
       "      <td>4151.4</td>\n",
       "      <td>88111.80</td>\n",
       "      <td>1183.26</td>\n",
       "      <td>349.69</td>\n",
       "      <td>7422.030</td>\n",
       "      <td>99.671</td>\n",
       "      <td>157529.6100</td>\n",
       "      <td>2115.4770</td>\n",
       "      <td>28.4089</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 105 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0        1     2     3    4      5      6     7       8    9     ...     \\\n",
       "0  1.0  0.00632  18.0  2.31  0.0  0.538  6.575  65.2  4.0900  1.0   ...      \n",
       "1  1.0  0.02731   0.0  7.07  0.0  0.469  6.421  78.9  4.9671  2.0   ...      \n",
       "2  1.0  0.02729   0.0  7.07  0.0  0.469  7.185  61.1  4.9671  2.0   ...      \n",
       "3  1.0  0.03237   0.0  2.18  0.0  0.458  6.998  45.8  6.0622  3.0   ...      \n",
       "4  1.0  0.06905   0.0  2.18  0.0  0.458  7.147  54.2  6.0622  3.0   ...      \n",
       "\n",
       "       95      96         97       98      99        100      101  \\\n",
       "0  87616.0  4528.8  117482.40  1474.08  234.09  6072.570   76.194   \n",
       "1  58564.0  4307.6   96049.80  2211.88  316.84  7064.820  162.692   \n",
       "2  58564.0  4307.6   95064.86   975.26  316.84  6992.374   71.734   \n",
       "3  49284.0  4151.4   87607.86   652.68  349.69  7379.581   54.978   \n",
       "4  49284.0  4151.4   88111.80  1183.26  349.69  7422.030   99.671   \n",
       "\n",
       "           102        103      104  \n",
       "0  157529.6100  1976.5620  24.8004  \n",
       "1  157529.6100  3627.6660  83.5396  \n",
       "2  154315.4089  1583.1049  16.2409  \n",
       "3  155732.8369  1160.2122   8.6436  \n",
       "4  157529.6100  2115.4770  28.4089  \n",
       "\n",
       "[5 rows x 105 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poly_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "poly_train = poly_train.add_prefix('poly_')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train = pd.concat([train,poly_train], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>var_0</th>\n",
       "      <th>var_1</th>\n",
       "      <th>var_2</th>\n",
       "      <th>var_3</th>\n",
       "      <th>var_4</th>\n",
       "      <th>var_5</th>\n",
       "      <th>var_6</th>\n",
       "      <th>var_7</th>\n",
       "      <th>var_8</th>\n",
       "      <th>var_9</th>\n",
       "      <th>...</th>\n",
       "      <th>poly_95</th>\n",
       "      <th>poly_96</th>\n",
       "      <th>poly_97</th>\n",
       "      <th>poly_98</th>\n",
       "      <th>poly_99</th>\n",
       "      <th>poly_100</th>\n",
       "      <th>poly_101</th>\n",
       "      <th>poly_102</th>\n",
       "      <th>poly_103</th>\n",
       "      <th>poly_104</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>...</td>\n",
       "      <td>87616.0</td>\n",
       "      <td>4528.8</td>\n",
       "      <td>117482.40</td>\n",
       "      <td>1474.08</td>\n",
       "      <td>234.09</td>\n",
       "      <td>6072.570</td>\n",
       "      <td>76.194</td>\n",
       "      <td>157529.6100</td>\n",
       "      <td>1976.5620</td>\n",
       "      <td>24.8004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>...</td>\n",
       "      <td>58564.0</td>\n",
       "      <td>4307.6</td>\n",
       "      <td>96049.80</td>\n",
       "      <td>2211.88</td>\n",
       "      <td>316.84</td>\n",
       "      <td>7064.820</td>\n",
       "      <td>162.692</td>\n",
       "      <td>157529.6100</td>\n",
       "      <td>3627.6660</td>\n",
       "      <td>83.5396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>...</td>\n",
       "      <td>58564.0</td>\n",
       "      <td>4307.6</td>\n",
       "      <td>95064.86</td>\n",
       "      <td>975.26</td>\n",
       "      <td>316.84</td>\n",
       "      <td>6992.374</td>\n",
       "      <td>71.734</td>\n",
       "      <td>154315.4089</td>\n",
       "      <td>1583.1049</td>\n",
       "      <td>16.2409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>...</td>\n",
       "      <td>49284.0</td>\n",
       "      <td>4151.4</td>\n",
       "      <td>87607.86</td>\n",
       "      <td>652.68</td>\n",
       "      <td>349.69</td>\n",
       "      <td>7379.581</td>\n",
       "      <td>54.978</td>\n",
       "      <td>155732.8369</td>\n",
       "      <td>1160.2122</td>\n",
       "      <td>8.6436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>...</td>\n",
       "      <td>49284.0</td>\n",
       "      <td>4151.4</td>\n",
       "      <td>88111.80</td>\n",
       "      <td>1183.26</td>\n",
       "      <td>349.69</td>\n",
       "      <td>7422.030</td>\n",
       "      <td>99.671</td>\n",
       "      <td>157529.6100</td>\n",
       "      <td>2115.4770</td>\n",
       "      <td>28.4089</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 118 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     var_0  var_1  var_2  var_3  var_4  var_5  var_6   var_7  var_8  var_9  \\\n",
       "0  0.00632   18.0   2.31    0.0  0.538  6.575   65.2  4.0900    1.0  296.0   \n",
       "1  0.02731    0.0   7.07    0.0  0.469  6.421   78.9  4.9671    2.0  242.0   \n",
       "2  0.02729    0.0   7.07    0.0  0.469  7.185   61.1  4.9671    2.0  242.0   \n",
       "3  0.03237    0.0   2.18    0.0  0.458  6.998   45.8  6.0622    3.0  222.0   \n",
       "4  0.06905    0.0   2.18    0.0  0.458  7.147   54.2  6.0622    3.0  222.0   \n",
       "\n",
       "     ...     poly_95  poly_96    poly_97  poly_98  poly_99  poly_100  \\\n",
       "0    ...     87616.0   4528.8  117482.40  1474.08   234.09  6072.570   \n",
       "1    ...     58564.0   4307.6   96049.80  2211.88   316.84  7064.820   \n",
       "2    ...     58564.0   4307.6   95064.86   975.26   316.84  6992.374   \n",
       "3    ...     49284.0   4151.4   87607.86   652.68   349.69  7379.581   \n",
       "4    ...     49284.0   4151.4   88111.80  1183.26   349.69  7422.030   \n",
       "\n",
       "   poly_101     poly_102   poly_103  poly_104  \n",
       "0    76.194  157529.6100  1976.5620   24.8004  \n",
       "1   162.692  157529.6100  3627.6660   83.5396  \n",
       "2    71.734  154315.4089  1583.1049   16.2409  \n",
       "3    54.978  155732.8369  1160.2122    8.6436  \n",
       "4    99.671  157529.6100  2115.4770   28.4089  \n",
       "\n",
       "[5 rows x 118 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 2000\n",
    "lgb_iter2 = []\n",
    "sklearn_gbm_iter2 = []\n",
    "xgb_gbm_iter2 = []\n",
    "\n",
    "lgb_ap2 = []\n",
    "sklearn_gbm_ap2 = []\n",
    "xgb_gbm_ap2 = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fold  0\n",
      " Best iteration lgb =  83\n",
      " Best iteration sklearn_gbm =  64\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.7923602912239276\n",
      "sklearn_gbn  0.35953654188948314\n",
      "xgboost  0.3834988540870894\n",
      "\n",
      "Fold  1\n",
      " Best iteration lgb =  120\n",
      " Best iteration sklearn_gbm =  86\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.8226619132501486\n",
      "sklearn_gbn  0.37940594059405947\n",
      "xgboost  0.4661716171617162\n",
      "\n",
      "Fold  2\n",
      " Best iteration lgb =  3\n",
      " Best iteration sklearn_gbm =  121\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.9354365079365079\n",
      "sklearn_gbn  0.4661716171617162\n",
      "xgboost  0.6598019801980199\n",
      "\n",
      "Fold  3\n",
      " Best iteration lgb =  55\n",
      " Best iteration sklearn_gbm =  97\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.9909090909090909\n",
      "sklearn_gbn  0.6598019801980199\n",
      "xgboost  1.0\n",
      "\n",
      "Fold  4\n",
      " Best iteration lgb =  13\n",
      " Best iteration sklearn_gbm =  88\n",
      " Best iteration xgboost_gbm =  1999\n",
      "lgb  0.9572979797979797\n",
      "sklearn_gbn  0.6396039603960396\n",
      "xgboost  0.7297029702970297\n"
     ]
    }
   ],
   "source": [
    "for i, (train_index, test_index) in enumerate(skf.split(train,y_bin)):\n",
    "    \n",
    "    # Create data for this fold\n",
    "    y_train, y_valid = y_bin[train_index], y_bin[test_index]\n",
    "    X_train, X_valid = train.iloc[train_index,:], train.iloc[test_index,:]\n",
    "        \n",
    "    print( \"\\nFold \", i)\n",
    "\n",
    "    # Run model for this fold\n",
    "\n",
    "    lgb_gbm = lgb.train(params, \n",
    "                          lgb.Dataset(X_train, label=y_train), \n",
    "                          MAX_ROUNDS, \n",
    "                          lgb.Dataset(X_valid, label=y_valid), \n",
    "                          verbose_eval=False, \n",
    "                          #feval= auc, \n",
    "                          early_stopping_rounds=50)\n",
    "    \n",
    "    print( \" Best iteration lgb = \", lgb_gbm.best_iteration)\n",
    "    \n",
    "    sklearn_gbm = GradientBoostingClassifier(n_estimators=MAX_ROUNDS, \n",
    "                                    learning_rate = 0.06,\n",
    "                                    max_features=2, \n",
    "                                    max_depth = 6, \n",
    "                                    n_iter_no_change=50, \n",
    "                                    tol=0.01,\n",
    "                                    random_state = 0)\n",
    "    \n",
    "    sklearn_gbm.fit(X_train, y_train)\n",
    "    print( \" Best iteration sklearn_gbm = \", sklearn_gbm.n_estimators_)\n",
    "    \n",
    "    \n",
    "    xgb_gbm = xgb.XGBClassifier(max_depth=6, \n",
    "                                n_estimators=MAX_ROUNDS,\n",
    "                                eval_set=[(X_train, y_train), (X_valid, y_valid)],\n",
    "                                learning_rate=0.06,\n",
    "                                early_stopping_rounds=50)\n",
    "\n",
    "    xgb_gbm.fit(X_train, y_train)\n",
    "    \n",
    "    print( \" Best iteration xgboost_gbm = \", xgb_gbm.get_booster().best_iteration)\n",
    "    \n",
    "    lgb_iter2 = np.append(lgb_iter2, lgb_gbm.best_iteration)\n",
    "    sklearn_gbm_iter2 = np.append(sklearn_gbm_iter2, sklearn_gbm.n_estimators_)\n",
    "    xgb_gbm_iter2 = np.append(xgb_gbm_iter2, xgb_gbm.get_booster().best_iteration)\n",
    "    \n",
    "    pred = lgb_gbm.predict(X_valid, num_iteration=lgb_gbm.best_iteration)\n",
    "    ap = average_precision_score(y_valid, pred, average='macro', pos_label=1, sample_weight=None)\n",
    "    print('lgb ', ap)\n",
    "    lgb_ap2 = np.append(lgb_ap2, ap)\n",
    "    \n",
    "    pred = sklearn_gbm.predict(X_valid)\n",
    "    ap = average_precision_score(y_valid, pred, average='macro', pos_label=1, sample_weight=None)\n",
    "    print('sklearn_gbn ', ap)\n",
    "    sklearn_gbm_ap2 = np.append(sklearn_gbm_ap2, ap)\n",
    "    \n",
    "    pred  = xgb_gbm.predict(X_valid)\n",
    "    ap = average_precision_score(y_valid, pred, average='macro', pos_label=1, sample_weight=None)\n",
    "    print('xgboost ', ap)\n",
    "    xgb_gbm_ap2 = np.append(xgb_gbm_ap2, ap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lgb_iter1:  42.8  lgb_iter2:  54.8\n",
      "sklearn_gbm_iter1:  88.6  sklearn_gbm_iter2:  91.2\n",
      "xgb_gbm_iter1:  1999.0  xgb_gbm_iter2:  1999.0\n",
      "lgb_ap1:  0.8867876743139901  lgb_ap2:  0.8997331566235308\n",
      "sklearn_gbm_ap1:  0.5681300806551244  sklearn_gbm_ap2:  0.5009040080478637\n",
      "xgb_gbm_ap1:  0.643185721933538  xgb_gbm_ap2:  0.647835084348771\n"
     ]
    }
   ],
   "source": [
    "print('lgb_iter1: ', np.mean(lgb_iter1),' lgb_iter2: ', np.mean(lgb_iter2))\n",
    "print('sklearn_gbm_iter1: ', np.mean(sklearn_gbm_iter1), ' sklearn_gbm_iter2: ', np.mean(sklearn_gbm_iter2))\n",
    "print('xgb_gbm_iter1: ',np.mean(xgb_gbm_iter1), ' xgb_gbm_iter2: ',np.mean(xgb_gbm_iter2) )\n",
    "\n",
    "print('lgb_ap1: ', np.mean(lgb_ap1), ' lgb_ap2: ', np.mean(lgb_ap2))\n",
    "print('sklearn_gbm_ap1: ', np.mean(sklearn_gbm_ap1), ' sklearn_gbm_ap2: ', np.mean(sklearn_gbm_ap2))\n",
    "print('xgb_gbm_ap1: ', np.mean(xgb_gbm_ap1), ' xgb_gbm_ap2: ', np.mean(xgb_gbm_ap2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we proceed with optimization, some important questions and insights about model specification.\n",
    "\n",
    "**1. Should we use all methods and mix them up in a competition?**\n",
    "\n",
    "**2. Should we use all methods and mix them up in a work project?**\n",
    "\n",
    "**3. Should we add the polynomial features in a competition?**\n",
    "\n",
    "**4. Should we add the polynomial features in a work project?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "db61f14fd91a1fc33afdb194138c908946021939"
   },
   "source": [
    "# PART II : Optimizing hyper-parameters\n",
    "\n",
    "The second topic of this workshop is also to illustrate how to best optimize the hyperparameters of a gradient boosting model (lightGBM before all, but also XGBoost and CatBoost) in a performing and efficient way. We will also compare the strong and weak points of different tuning approaches, such grid-search, random search and bayesian optimization by Scikit-optimize.\n",
    "\n",
    "Leaving apart grid-search (feasible only when the space of experiments is limited), the usual choice for the practitioner is to apply random search optimization or try some [Bayesian Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (BO) technique, which require a more complex setup. \n",
    "\n",
    "As for as BO, there are quite a few choices (for instance Hyperopt) but we decided for Scikit-Optimize, or skopt, because it is a simple and efficient library to minimize (very) expensive and noisy black-box functions and it works with an API similar to Scikit-learn. It can be found at https://github.com/scikit-optimize/scikit-optimize/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "_uuid": "5abe0cd47e94562c7a1248c6dc946146eb713d50",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/scikit-optimize/scikit-optimize.git\n",
      "  Cloning https://github.com/scikit-optimize/scikit-optimize.git to c:\\users\\utente\\appdata\\local\\temp\\pip-req-build-x3o7ddws\n",
      "Requirement already satisfied (use --upgrade to upgrade): scikit-optimize==0.5.2+48.g692c21d from git+https://github.com/scikit-optimize/scikit-optimize.git in c:\\anaconda3\\lib\\site-packages\n",
      "Requirement already satisfied: pyaml in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (18.11.0)\n",
      "Requirement already satisfied: numpy in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (1.15.4)\n",
      "Requirement already satisfied: scipy>=0.14.0 in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (1.1.0)\n",
      "Requirement already satisfied: scikit-learn>=0.19.1 in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (0.20.2)\n",
      "Requirement already satisfied: PyYAML in c:\\anaconda3\\lib\\site-packages (from pyaml->scikit-optimize==0.5.2+48.g692c21d) (3.12)\n",
      "Building wheels for collected packages: scikit-optimize\n",
      "  Running setup.py bdist_wheel for scikit-optimize: started\n",
      "  Running setup.py bdist_wheel for scikit-optimize: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\utente\\AppData\\Local\\Temp\\pip-ephem-wheel-cache-q2x2hxk0\\wheels\\11\\6f\\86\\2b772172db85ad0b4487d67e325e535ee8e7782b2a1dfcadf5\n",
      "Successfully built scikit-optimize\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 18.0, however version 19.0.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "# Installing the most recent version of skopt directly from Github\n",
    "!pip install git+https://github.com/scikit-optimize/scikit-optimize.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: catboost in c:\\anaconda3\\lib\\site-packages (0.12.2)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in c:\\anaconda3\\lib\\site-packages (from catboost) (1.15.4)\n",
      "Requirement already satisfied, skipping upgrade: six in c:\\anaconda3\\lib\\site-packages (from catboost) (1.11.0)\n",
      "Requirement already satisfied, skipping upgrade: pandas>=0.19.1 in c:\\anaconda3\\lib\\site-packages (from catboost) (0.20.3)\n",
      "Requirement already satisfied, skipping upgrade: enum34 in c:\\anaconda3\\lib\\site-packages (from catboost) (1.1.6)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2 in c:\\anaconda3\\lib\\site-packages (from pandas>=0.19.1->catboost) (2.6.1)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2011k in c:\\anaconda3\\lib\\site-packages (from pandas>=0.19.1->catboost) (2017.2)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 18.0, however version 19.0.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "# Assuring you have the most recent CatBoost release\n",
    "!pip install catboost -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Importing core libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from time import time\n",
    "import pprint\n",
    "import joblib\n",
    "\n",
    "# Suppressing warnings because of skopt verbosity\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Our example dataset\n",
    "from sklearn.datasets import load_boston\n",
    "\n",
    "# Classifiers\n",
    "from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "from catboost import CatBoostClassifier\n",
    "\n",
    "# Hyperparameters distributions\n",
    "from scipy.stats import randint\n",
    "from scipy.stats import uniform\n",
    "\n",
    "# Model selection\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import average_precision_score\n",
    "from sklearn.metrics import make_scorer\n",
    "\n",
    "# Skopt functions\n",
    "from skopt import BayesSearchCV\n",
    "from skopt import gp_minimize # Bayesian optimization using Gaussian Processes\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "from skopt.utils import use_named_args # decorator to convert a list of parameters to named arguments\n",
    "from skopt.callbacks import DeadlineStopper # Stop the optimization before running out of a fixed budget of time.\n",
    "from skopt.callbacks import VerboseCallback # Callback to control the verbosity\n",
    "from skopt.callbacks import DeltaXStopper # Stop the optimization If the last two positions at which the objective has been evaluated are less than delta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "02a9b6a1bf577931da26f5862d1631f0d7bd4478"
   },
   "source": [
    "Optimizing hyper-parameters requires time and resources. In order to speed up the demonstration we will be using a toy dataset, the Boston Houseprice dataset for a classification task, to predicted the top 10% most expensive houses.\n",
    "\n",
    "The dataset presents information collected by the U.S Census Service concerning housing proces and conditions in the area of Boston Mass. Originally found in the [StatLib archive](http://lib.stat.cmu.edu/datasets/boston), the dataset has been used extensively throughout the literature to benchmark machine learning algorithms. The data was originally published by :\n",
    "> Harrison, D. and Rubinfeld, D.L. Hedonic prices and the demand for clean air', J. Environ. Economics & Management, vol.5, 81-102, 1978.\n",
    "\n",
    "The dataset contains 14 variabile relative to 506 house that were sold in the suburbs of Boston. Among the variables, the 14th, MEDV - Median value of owner-occupied homes in $1000's - is commonly used as a target for regression problems. In our example we will use it for classification, after binarizing it at the 90th percentile (also creating an unbalanced classification problem, since the positive cases are just 10 percent of the total). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "_uuid": "ec69b7012e67494e560a75c09459d608f74be88a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Uploading the Boston dataset\n",
    "X, y = load_boston(return_X_y=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "_uuid": "7853f7a2a67c58b41ff98efe77df437ebd57f15f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Transforming the problem into a classification (unbalanced)\n",
    "y_bin = (y > np.percentile(y, 90)).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a6be6080c9429abc5a041f9b6bf825635ddbdf09"
   },
   "source": [
    "# Optimizing Scikit-learn GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GridSearchCV, RandomizedSearchCV (from Scikit-learn) and BayesSearchCV (from Scikit-optimize) all have the same API. A wrapper can just put together optimization, callbacks, best results reporting and time monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "_uuid": "4dd7d7b1699775a7a4e08c553ab3ec93a516cf56",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Reporting util for different optimizers\n",
    "def report_perf(optimizer, X, y, title, callbacks=None):\n",
    "    \"\"\"\n",
    "    A wrapper for measuring time and performances of different optmizers\n",
    "    \n",
    "    optimizer = a sklearn or a skopt optimizer\n",
    "    X = the training set \n",
    "    y = our target\n",
    "    title = a string label for the experiment\n",
    "    \"\"\"\n",
    "    start = time()\n",
    "    if callbacks:\n",
    "        optimizer.fit(X, y, callback=callbacks)\n",
    "    else:\n",
    "        optimizer.fit(X, y)\n",
    "    best_score = optimizer.best_score_\n",
    "    best_score_std = optimizer.cv_results_['std_test_score'][optimizer.best_index_]\n",
    "    best_params = optimizer.best_params_\n",
    "    print((title + \" took %.2f seconds,  candidates checked: %d, best CV score: %.3f \"\n",
    "           +u\"\\u00B1\"+\" %.3f\") % (time() - start, \n",
    "                                  len(optimizer.cv_results_['params']),\n",
    "                                  best_score,\n",
    "                                  best_score_std))    \n",
    "    print('Best parameters:')\n",
    "    pprint.pprint(best_params)\n",
    "    print()\n",
    "    return best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "_uuid": "9600a2f3b8baf43e0d86bf856621b206d0aedc9f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Converting average precision score into a scorer suitable for model selection\n",
    "avg_prec = make_scorer(average_precision_score, greater_is_better=True, needs_proba=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "_uuid": "ab53b3efa1a0412bf7d02544e94143f609a303fc",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Setting a 5-fold stratified cross-validation (note: shuffle=True)\n",
    "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A Scikit-learn GBM classifier\n",
    "clf = GradientBoostingClassifier(n_estimators=20, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grid search exhaustively searches through the hyperparameters and is not feasible in high dimensional space\n",
    "This is a very simple algorithm and suffers from the curse of dimensionality, though it's embarrassingly parallel.\n",
    "\n",
    "Here we use, GridSearchCV, a function from Scikit-learn. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "_uuid": "9746d8b7103c2a0fc1aeb1c04d9135807bf7bef3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GridSearchCV took 83.45 seconds,  candidates checked: 96, best CV score: 0.909 ± 0.072\n",
      "Best parameters:\n",
      "{'learning_rate': 0.01,\n",
      " 'max_features': None,\n",
      " 'min_samples_leaf': 10,\n",
      " 'min_samples_split': 2,\n",
      " 'n_estimators': 500,\n",
      " 'subsample': 0.5}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# GridSearchCV needs a predefined plan of the experiments\n",
    "grid_search = GridSearchCV(clf, \n",
    "                           param_grid={\"learning_rate\": [0.01, 1.0],\n",
    "                                       \"n_estimators\": [10, 500],\n",
    "                                       \"subsample\": [1.0, 0.5],\n",
    "                                       \"min_samples_split\": [2, 10],\n",
    "                                       \"min_samples_leaf\": [1, 10],\n",
    "                                       \"max_features\": ['sqrt', 'log2', None]\n",
    "                                       },\n",
    "                           n_jobs=-1,\n",
    "                           cv=skf,\n",
    "                           scoring=avg_prec,\n",
    "                           iid=False, # just return the average score across folds\n",
    "                           return_train_score=False)\n",
    "\n",
    "best_params = report_perf(grid_search, X, y_bin,'GridSearchCV')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random search, which simply samples the search space randomly, is feasible in high dimensional spaces, and is widely used in practice. The downside of random search, however, is that it doesn’t use information from prior experiments to select the next setting.\n",
    "\n",
    "You simply need to be lucky to catch the right hyper-parameters, or just try more ;-).\n",
    "\n",
    "In fact, the 2×Random Search is the Random Search algorithm when it was allowed to sample two points for each point the other algorithms evaluated. While some authors have claimed that 2×Random Search is highly competitive with Bayesian Optimization methods, a [study by Google](http://delivery.acm.org/10.1145/3100000/3098043/p1487-golovin.pdf) (GOLOVIN, Daniel, et al. Google vizier: A service for black-box optimization. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017. p. 1487-1495) suggests that this is only true when the dimensionality of the problem is sufficiently high (e.g., over 16)\n",
    "\n",
    "RandomizedSearchCV is a function from Scikit-learn, though skopt has it own random optimizer, *[dummy_minimize](https://scikit-optimize.github.io/#skopt.dummy_minimize)*.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "_uuid": "498623fb37dd803dd1edf5920e768e4b67ceef0a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomizedSearchCV took 33.58 seconds,  candidates checked: 40, best CV score: 0.927 ± 0.097\n",
      "Best parameters:\n",
      "{'learning_rate': 0.10237388946089819,\n",
      " 'max_features': 'log2',\n",
      " 'min_samples_leaf': 2,\n",
      " 'min_samples_split': 9,\n",
      " 'n_estimators': 384,\n",
      " 'subsample': 0.8266004099285669}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# RandomizedSearchCV needs the distribution of the experiments to be tested\n",
    "# If you can provide the right distribution, the sampling will lead to faster and better results.\n",
    "\n",
    "random_search = RandomizedSearchCV(clf, \n",
    "                                   param_distributions={\"learning_rate\": uniform(0.01, 1.0),\n",
    "                                                        \"n_estimators\": randint(10, 500),\n",
    "                                                        \"subsample\": uniform(0.5, 0.5),\n",
    "                                                        \"min_samples_split\": randint(2, 10),\n",
    "                                                        \"min_samples_leaf\": randint(1, 10),\n",
    "                                                        \"max_features\": ['sqrt', 'log2', None]\n",
    "                                       },\n",
    "                                   n_iter=40,\n",
    "                                   n_jobs=-1,\n",
    "                                   cv=skf,\n",
    "                                   scoring=avg_prec,\n",
    "                                   iid=False, # just return the average score across folds\n",
    "                                   return_train_score=False,\n",
    "                                   random_state=0)\n",
    "\n",
    "best_params = report_perf(random_search, X, y_bin, 'RandomizedSearchCV')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After examining the classical and most known approaches, it is time to dwelve into Bayesian optimization.\n",
    "\n",
    "Bayesian optimization is behind [Google Cloud Machine Learning Engine](https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-cloud-machine-learning-engine-using-bayesian-optimization) services.\n",
    "\n",
    "The key idea behind Bayesian optimization is that we optimize a proxy function instead than the true objective function (what actually grid search and random search both do). This holds if testing the true objective function is costly (if it is not, then we simply go for random search :-))\n",
    "\n",
    "Bayesian search balances exploration against exploitation. At start it randomly explores, doing so it builds up a surrogate function of the objective. Based on that surrogate function it exploits an initial approximate knowledge of how the predictor works in order to sample more useful examples and minimize the cost function at a global level, not a local one.\n",
    "\n",
    "As the Bayesian part of the title suggests, we use priors in order to make smarter decisions about sampling during optimizing in order to reach a minimization faster by limiting the number of evaluations we need to make.\n",
    "\n",
    "Bayesian Optimization uses an acquisition function to tell us how promising an observation will be.\n",
    "In fact, to rule the tradeoff between exploration and exploitation, the algorithm defines an acquisition function that provides a single measure of how useful it would be to try any given point.\n",
    "\n",
    "From the figure taken from [Skopt API documentation](https://scikit-optimize.github.io/notebooks/bayesian-optimization.html), you can figure out that the surrogate function (the green dotted line, whose error band is represented by the light green area) has somehow approximated the true cost function (the red dotted line):\n",
    "\n",
    "![figure_1](https://scikit-optimize.github.io/notebooks/bayesian-optimization_files/bayesian-optimization_21_0.png)\n",
    "\n",
    "The observations supporting the construction of the surrogate function are not randomly sparse around, because, through an acquisition function (in a gaussian processes it is a function guiding the selection of the next evaluation points), they have been picked as the most useful examples in order to guess how to minimize the cost function.\n",
    "\n",
    "In respect of a random optimization, a bayesian optimization is more of an educated guess, then, first sampling randomly, but then focussing on the most important combination of hyper-parameters in order to figure out, first the surrogate function of the cost function, then the global minimum of the cost function:\n",
    "\n",
    "![figure_2](https://scikit-optimize.github.io/notebooks/bayesian-optimization_files/bayesian-optimization_18_1.png)\n",
    "\n",
    "Gaussian process (GP) is one of the possible ways to build a surrogate function: it consists of a distribution on functions.\n",
    "Originally GPs were developed to help search for gold ([kriging](https://en.wikipedia.org/wiki/Kriging)). Please note that the approach is closely related to the statistical ideas in the optimal design of experiments.\n",
    "In a gaussian process, based on a distribution of functions resembling the true cost function, the alogorithm operates in:\n",
    "\n",
    "* Exploration -> seeking points and areas on the optimization surface with high variance\n",
    "* Exploitation -> seeking points with low mean\n",
    "\n",
    "This is done by a second, specialized function, the acquisition function.\n",
    "\n",
    "Other approaches are 1) ensembles of decision trees 2) Tree of Parzen Estimators (TPE used by [Hyperopt](http://hyperopt.github.io/hyperopt/) \n",
    "another Bayesian optimization package package) \n",
    "\n",
    "Gaussian Processes are just models, and they're much more like k-nearest neighbors and linear regression than may at first be apparent. If you want to understand more of GPs, you can read the post: [https://planspace.org/20181226-gaussian_processes_are_not_so_fancy/](https://planspace.org/20181226-gaussian_processes_are_not_so_fancy)by Aaron Schumacher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "_uuid": "cca6073d5c1d67dd11ab6b8f77800d77ff044cca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BayesSearchCV_GP took 259.84 seconds,  candidates checked: 40, best CV score: 0.928 ± 0.059\n",
      "Best parameters:\n",
      "{'learning_rate': 0.35323427697468307,\n",
      " 'max_features': None,\n",
      " 'min_samples_leaf': 10,\n",
      " 'min_samples_split': 8,\n",
      " 'n_estimators': 284,\n",
      " 'subsample': 1.0}\n",
      "\n",
      "BayesSearchCV_RF took 225.23 seconds,  candidates checked: 40, best CV score: 0.945 ± 0.051\n",
      "Best parameters:\n",
      "{'learning_rate': 0.6049627162690525,\n",
      " 'max_features': 'sqrt',\n",
      " 'min_samples_leaf': 9,\n",
      " 'min_samples_split': 10,\n",
      " 'n_estimators': 485,\n",
      " 'subsample': 0.6024816853254407}\n",
      "\n",
      "BayesSearchCV_ET took 198.74 seconds,  candidates checked: 40, best CV score: 0.933 ± 0.059\n",
      "Best parameters:\n",
      "{'learning_rate': 0.39752021916372565,\n",
      " 'max_features': None,\n",
      " 'min_samples_leaf': 6,\n",
      " 'min_samples_split': 9,\n",
      " 'n_estimators': 78,\n",
      " 'subsample': 0.7687211202072857}\n",
      "\n",
      "BayesSearchCV_GBRT took 136.78 seconds,  candidates checked: 40, best CV score: 0.933 ± 0.068\n",
      "Best parameters:\n",
      "{'learning_rate': 0.42127830500668484,\n",
      " 'max_features': 'log2',\n",
      " 'min_samples_leaf': 4,\n",
      " 'min_samples_split': 8,\n",
      " 'n_estimators': 377,\n",
      " 'subsample': 0.5657650066445118}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# also BayesSearchCV needs to work on the distributions of the experiments but it is less sensible to them\n",
    "\n",
    "search_spaces = {\"learning_rate\": Real(0.01, 1.0),\n",
    "                 \"n_estimators\": Integer(10, 500),\n",
    "                 \"subsample\": Real(0.5, 1.0),\n",
    "                 \"min_samples_split\": Integer(2, 10),\n",
    "                 \"min_samples_leaf\": Integer(1, 10),\n",
    "                 \"max_features\": Categorical(categories=['sqrt', 'log2', None])}\n",
    "\n",
    "for baseEstimator in ['GP', 'RF', 'ET', 'GBRT']:\n",
    "    opt = BayesSearchCV(clf,\n",
    "                        search_spaces,\n",
    "                        scoring=avg_prec,\n",
    "                        cv=skf,\n",
    "                        n_iter=40,\n",
    "                        n_jobs=-1,\n",
    "                        return_train_score=False,\n",
    "                        optimizer_kwargs={'base_estimator': baseEstimator},\n",
    "                        random_state=4)\n",
    "    \n",
    "    best_params = report_perf(opt, X, y_bin,'BayesSearchCV_'+baseEstimator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching more complex spaces\n",
    "If you have multiple models to optimize, you can leverage the *Pipeline* command in order to search different search spaces based on different models. That's requires to access to the hyper-parameters accordingly to *Pipeline* specifications, anyway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BayesSearchCV_GP took 167.03 seconds,  candidates checked: 40, best CV score: 0.924 ± 0.068\n",
      "Best parameters:\n",
      "{'model': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
      "              learning_rate=0.18398436757471331, loss='deviance',\n",
      "              max_depth=3, max_features='sqrt', max_leaf_nodes=None,\n",
      "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
      "              min_samples_leaf=10, min_samples_split=2,\n",
      "              min_weight_fraction_leaf=0.0, n_estimators=500,\n",
      "              n_iter_no_change=None, presort='auto', random_state=0,\n",
      "              subsample=0.7456226682789747, tol=0.0001,\n",
      "              validation_fraction=0.1, verbose=0, warm_start=False),\n",
      " 'model__learning_rate': 0.18398436757471331,\n",
      " 'model__max_features': 'sqrt',\n",
      " 'model__min_samples_leaf': 10,\n",
      " 'model__min_samples_split': 2,\n",
      " 'model__n_estimators': 500,\n",
      " 'model__subsample': 0.7456226682789747}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Initialize a pipeline with a model\n",
    "pipe = Pipeline([('model', GradientBoostingClassifier(n_estimators=20, random_state=0))])\n",
    "\n",
    "# Define search space for GBM;\n",
    "search_space_GBM = {\"model\": Categorical([GradientBoostingClassifier(n_estimators=20, random_state=0)]),\n",
    "                    \"model__learning_rate\": Real(0.01, 1.0),\n",
    "                    \"model__n_estimators\": Integer(10, 500),\n",
    "                    \"model__subsample\": Real(0.5, 1.0),\n",
    "                    \"model__min_samples_split\": Integer(2, 10),\n",
    "                    \"model__min_samples_leaf\": Integer(1, 10),\n",
    "                    \"model__max_features\": Categorical(categories=['sqrt', 'log2', None])}\n",
    "\n",
    "# Define search space for RF\n",
    "search_space_RF  = {\"model\": Categorical([RandomForestClassifier(n_estimators=20, random_state=0)]),\n",
    "                    \"model__n_estimators\": Integer(10, 200),\n",
    "                    \"model__min_samples_split\": Integer(2, 10),\n",
    "                    \"model__min_samples_leaf\": Integer(1, 10),\n",
    "                    \"model__max_features\": Categorical(categories=['sqrt', 'log2', None])}\n",
    "\n",
    "opt = BayesSearchCV(pipe,\n",
    "                        search_spaces=[(search_space_GBM, 20), (search_space_RF, 20)],\n",
    "                        scoring=avg_prec,\n",
    "                        cv=skf,\n",
    "                        n_jobs=-1,\n",
    "                        return_train_score=False,\n",
    "                        optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                        random_state=4)\n",
    "    \n",
    "best_params = report_perf(opt, X, y_bin,'BayesSearchCV_GP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Controlling the time cost of Bayesian optimization\n",
    "\n",
    "Running a single LightGBM model could take long time and in a Kaggle competition time is often a luxury. \n",
    "\n",
    "*DeadlineStopper* and *DeltaXStopper* are skopt callbacks that control the total time spent and the improvement of a BayesSearchCV (in our implementation to be called with *report_perf*, using the parameter *callbacks=[]*). \n",
    "\n",
    "Anyway, sometimes it is easier to control manually the optimization steps, hence the usage of low-level optimizers. \n",
    "\n",
    "We start defining a custom callback, using a different approach to search spaces (a list instead of a dictionary), and to manually create our objective function to be minimized.\n",
    "\n",
    "In our custom callback, we print the last evaluation point (so you know what's happening) and the best score and parameters foudn up so far. We also record the list of explored points (*x0*) and their relative results (*y0*). This will help us to reprise the learning at a later time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "_uuid": "4218521e57192fa23c0ab536653f3d07d04e2402",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counter = 0\n",
    "def onstep(res):\n",
    "    global counter\n",
    "    x0 = res.x_iters   # List of input points\n",
    "    y0 = res.func_vals # Evaluation of input points\n",
    "    print('Last eval: ', x0[-1], \n",
    "          ' - Score ', y0[-1])\n",
    "    print('Current iter: ', counter, \n",
    "          ' - Score ', res.fun, \n",
    "          ' - Args: ', res.x)\n",
    "    joblib.dump((x0, y0), 'checkpoint.pkl') # Saving a checkpoint to disk\n",
    "    counter += 1\n",
    "\n",
    "# Our search space\n",
    "dimensions = [Real(0.01, 1.0, name=\"learning_rate\"),\n",
    "              Integer(10, 500, name=\"n_estimators\"),\n",
    "              Real(0.5, 1.0, name=\"subsample\"),\n",
    "              Integer(2, 10, name=\"min_samples_split\"),\n",
    "              Integer(1, 10, name=\"min_samples_leaf\"),\n",
    "              Categorical(categories=['sqrt', 'log2', None], name=\"max_features\")]\n",
    "\n",
    "# The objective function to be minimized\n",
    "def make_objective(model, X, y, space, cv, scoring):\n",
    "    # This decorator converts your objective function with named arguments into one that\n",
    "    # accepts a list as argument, while doing the conversion automatically.\n",
    "    @use_named_args(space) \n",
    "    def objective(**params):\n",
    "        model.set_params(**params)\n",
    "        return -np.mean(cross_val_score(model, \n",
    "                                        X, y, \n",
    "                                        cv=cv, \n",
    "                                        n_jobs=-1,\n",
    "                                        scoring=scoring))\n",
    "\n",
    "    return objective\n",
    "\n",
    "objective = make_objective(clf,\n",
    "                           X, y_bin,\n",
    "                           space=dimensions,\n",
    "                           cv=skf,\n",
    "                           scoring=avg_prec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are different low-level optimizers that can be used for the purpose:\n",
    "* **gp_minimize** Bayesian optimization using Gaussian Processes.\n",
    "* **forest_minimize** Sequential optimisation using decision trees\n",
    "* **gbrt_minimize** Sequential optimization using gradient boosted trees\n",
    "* **dummy_minimize** Random search by uniform sampling within the given bounds (a replacement for Scikit-learn's RandomSearch)\n",
    "\n",
    "Each optimizer has its own parameters, so they cannot be just automatically switched, though they share most of the key parameters.\n",
    "\n",
    "Here we encounter also a new parameter, **acq_func**, useful for defining how the acquisition function should behave, that is, if to take as minimum the lower confidence bound, the minimum expected value or probability (the suggested default is usually a good choice)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "_uuid": "88361c5f2f425f8cbf334d8b494692405e8c0b86"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [0.7049433514242192, 368, 0.8187395776032731, 8, 9, None]  - Score  -0.9053516521792384\n",
      "Current iter:  0  - Score  -0.9053516521792384  - Args:  [0.7049433514242192, 368, 0.8187395776032731, 8, 9, None]\n",
      "Last eval:  [0.24229127648570348, 16, 0.7320268705932458, 8, 2, 'sqrt']  - Score  -0.8892368868276309\n",
      "Current iter:  1  - Score  -0.9053516521792384  - Args:  [0.7049433514242192, 368, 0.8187395776032731, 8, 9, None]\n",
      "Last eval:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']  - Score  -0.9234068462086364\n",
      "Current iter:  2  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.010263121534902437, 442, 0.972884851277688, 9, 2, 'sqrt']  - Score  -0.9049683094683093\n",
      "Current iter:  3  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.8875384524348883, 205, 0.9771110531385809, 8, 8, None]  - Score  -0.8977127929278959\n",
      "Current iter:  4  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.9408844987095382, 278, 0.7733463688856329, 7, 2, None]  - Score  -0.7844031794538096\n",
      "Current iter:  5  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.3343579995412422, 367, 0.5956932474515344, 8, 9, 'log2']  - Score  -0.9143859278549682\n",
      "Current iter:  6  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.7879021133849456, 252, 0.75143818812381, 8, 5, 'log2']  - Score  -0.5935980587554553\n",
      "Current iter:  7  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.8164027897144066, 346, 0.6236194997341288, 6, 7, 'sqrt']  - Score  -0.8692463647463649\n",
      "Current iter:  8  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.7856253596534273, 448, 0.7012180692236435, 9, 8, 'sqrt']  - Score  -0.753150122881838\n",
      "Current iter:  9  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n"
     ]
    }
   ],
   "source": [
    "gp_round = gp_minimize(func=objective,\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Defining what to minimize \n",
    "                       n_calls=10,\n",
    "                       callback=[onstep],\n",
    "                       random_state=22)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "_uuid": "de1369d11ea6a3bc79328bc2f2e14dd2621f4686"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [0.7856253596534273, 448, 0.7012180692236435, 9, 8, 'sqrt']  - Score  -0.753150122881838\n",
      "Current iter:  10  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']  - Score  -0.9433474858474857\n",
      "Current iter:  11  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.3045592604790276, 38, 0.6363281472900566, 6, 8, 'log2']  - Score  -0.9160759254036765\n",
      "Current iter:  12  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.39885694813982153, 420, 0.6686980802086342, 7, 4, None]  - Score  -0.8992387417077818\n",
      "Current iter:  13  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.14894727260851873, 436, 0.7368040226368553, 8, 6, None]  - Score  -0.9172835775335775\n",
      "Current iter:  14  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.7234263281786577, 295, 0.7686866147245054, 8, 2, 'log2']  - Score  -0.9011155635306421\n",
      "Current iter:  15  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.1944690198934924, 371, 0.6082751772121859, 3, 4, 'sqrt']  - Score  -0.9163157411434923\n",
      "Current iter:  16  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.23009817436907185, 199, 0.9512992377647025, 6, 7, None]  - Score  -0.901292475498358\n",
      "Current iter:  17  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.10828754685538415, 485, 0.8265700178989689, 3, 4, None]  - Score  -0.9187638287638288\n",
      "Current iter:  18  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.6117523620283132, 169, 0.5192127132363674, 7, 10, 'log2']  - Score  -0.8649301045063078\n",
      "Current iter:  19  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.6387082848675283, 498, 0.7909251647192672, 5, 5, 'log2']  - Score  -0.7693444638581505\n",
      "Current iter:  20  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n"
     ]
    }
   ],
   "source": [
    "x0, y0 = joblib.load('checkpoint.pkl')\n",
    "\n",
    "gp_round = gp_minimize(func=objective,\n",
    "                       x0=x0,              # already examined values for x\n",
    "                       y0=y0,              # observed values for x0\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Expected Improvement.\n",
    "                       n_calls=10,\n",
    "                       callback=[onstep],\n",
    "                       random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "_uuid": "2826f2d0f47b973052a81643498617f3b6f1cbd1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2'] -0.9433474858474857\n"
     ]
    }
   ],
   "source": [
    "best_parameters = gp_round.x\n",
    "best_result = gp_round.fun\n",
    "print(best_parameters, best_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In conclusion, just keep in mind a few points from the workshop:\n",
    "\n",
    "* Bayesian Optimization has its own hyper-parameters (therefore use defaults, [unless you know what you doing](https://i0.kym-cdn.com/entries/icons/original/000/008/342/ihave.jpg))\n",
    "\n",
    "* Experiments are run sequentially (skopt can leverage some parallelism, though), having multiple cores is helpful for your learning algorithm,but Bayesian Optimization will always be slower than Random Search. Use it only when needed.\n",
    "\n",
    "* Packages are not all that friendly (hence the workshop :-)) but you can reuse some simple wrappers re-adaptable to being used in different Kaggle competitions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1a82e07c7940ff0587db89b8ceba55be99564fea"
   },
   "source": [
    "# A Practical Example: Optimizing LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "_uuid": "2d1d9fbdb869a040c93f1ffacf59c14f3e473264"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LightGBM took 300.93 seconds,  candidates checked: 18, best CV score: 0.888 ± 0.077\n",
      "Best parameters:\n",
      "{'colsample_bytree': 0.4162224985830699,\n",
      " 'learning_rate': 0.2587825932038943,\n",
      " 'max_bin': 100000,\n",
      " 'max_depth': 251,\n",
      " 'min_child_samples': 0,\n",
      " 'min_child_weight': 5,\n",
      " 'n_estimators': 8169,\n",
      " 'num_leaves': 225,\n",
      " 'reg_alpha': 3.4016259449541346e-07,\n",
      " 'reg_lambda': 0.0003593663142644772,\n",
      " 'scale_pos_weight': 0.26016146142834107,\n",
      " 'subsample': 0.9169335498052059,\n",
      " 'subsample_for_bin': 281437,\n",
      " 'subsample_freq': 3}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "clf = lgb.LGBMClassifier(boosting_type='gbdt',\n",
    "                         class_weight='balanced',\n",
    "                         objective='binary',\n",
    "                         n_jobs=1, \n",
    "                         verbose=0)\n",
    "\n",
    "search_spaces = {\n",
    "        'learning_rate': Real(0.01, 1.0, 'log-uniform'),\n",
    "        'num_leaves': Integer(2, 500),\n",
    "        'max_depth': Integer(0, 500),\n",
    "        'min_child_samples': Integer(0, 200), # minimal number of data in one leaf\n",
    "        'max_bin': Integer(100, 100000), # max number of bins that feature values will be bucketed\n",
    "        'subsample': Real(0.01, 1.0, 'uniform'),\n",
    "        'subsample_freq': Integer(0, 10), # bagging fraction\n",
    "        'colsample_bytree': Real(0.01, 1.0, 'uniform'), # enabler of bagging fraction\n",
    "        'min_child_weight': Integer(0, 10), # minimal number of data in one leaf.\n",
    "        'subsample_for_bin': Integer(100000, 500000), # number of data that sampled for histogram bins\n",
    "        'reg_lambda': Real(1e-9, 1000, 'log-uniform'), # L2 regularization\n",
    "        'reg_alpha': Real(1e-9, 1.0, 'log-uniform'), # L1 regularization\n",
    "        'scale_pos_weight': Real(1e-6, 500, 'log-uniform'), \n",
    "        'n_estimators': Integer(10, 10000)        \n",
    "        }\n",
    "\n",
    "opt = BayesSearchCV(clf,\n",
    "                    search_spaces,\n",
    "                    scoring=avg_prec,\n",
    "                    cv=skf,\n",
    "                    n_iter=40,\n",
    "                    n_jobs=-1,\n",
    "                    return_train_score=False,\n",
    "                    refit=True,\n",
    "                    optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                    random_state=22)\n",
    "    \n",
    "best_params = report_perf(opt, X, y_bin,'LightGBM', \n",
    "                          callbacks=[DeltaXStopper(0.001), \n",
    "                                     DeadlineStopper(60*5)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "_uuid": "5e4e99b9b5c3d22ae5afdf8e2437c66e3c84cefc",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counter = 0\n",
    "\n",
    "clf = lgb.LGBMClassifier(boosting_type='gbdt',\n",
    "                         class_weight='balanced',\n",
    "                         objective='binary',\n",
    "                         n_jobs=1, \n",
    "                         verbose=0)\n",
    "\n",
    "dimensions = [Real(0.01, 1.0, 'log-uniform', name='learning_rate'),\n",
    "              Integer(2, 500, name='num_leaves'),\n",
    "              Integer(0, 500, name='max_depth'),\n",
    "              Integer(0, 200, name='min_child_samples'),\n",
    "              Integer(100, 100000, name='max_bin'),\n",
    "              Real(0.01, 1.0, 'uniform', name='subsample'),\n",
    "              Integer(0, 10, name='subsample_freq'),\n",
    "              Real(0.01, 1.0, 'uniform', name='colsample_bytree'),\n",
    "              Integer(0, 10, name='min_child_weight'),\n",
    "              Integer(100000, 500000, name='subsample_for_bin'),\n",
    "              Real(1e-9, 1000, 'log-uniform', name='reg_lambda'),\n",
    "              Real(1e-9, 1.0, 'log-uniform', name='reg_alpha'),\n",
    "              Real(1e-6, 500, 'log-uniform', name='scale_pos_weight'),\n",
    "              Integer(10, 10000, name='n_estimators')]\n",
    "\n",
    "objective = make_objective(clf,\n",
    "                           X, y_bin,\n",
    "                           space=dimensions,\n",
    "                           cv=skf,\n",
    "                           scoring=avg_prec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "_uuid": "6de6368797702d3944fcb35082984be4910cfbbe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]  - Score  -0.1007765482430596\n",
      "Current iter:  0  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.2211233200687724, 348, 173, 186, 26332, 0.7532551005821186, 3, 0.8527816404253235, 2, 416305, 178.4645772067005, 1.095045483795558e-05, 0.0022184775182806722, 3581]  - Score  -0.1007765482430596\n",
      "Current iter:  1  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.02436190508232831, 52, 241, 123, 95130, 0.3552688118981681, 3, 0.45204083607316786, 10, 171831, 5.159278367758748e-08, 5.643816697371209e-09, 0.05759462438024217, 3109]  - Score  -0.1007765482430596\n",
      "Current iter:  2  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.036765413368033246, 461, 217, 84, 98624, 0.7956342983145053, 3, 0.307335685328769, 5, 349302, 0.07650111834271556, 2.1489283637488493e-07, 0.0033482236099689746, 6676]  - Score  -0.1007765482430596\n",
      "Current iter:  3  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]  - Score  -0.5184708747168858\n",
      "Current iter:  4  - Score  -0.5184708747168858  - Args:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]\n",
      "Last eval:  [0.47548110586714587, 267, 20, 19, 79910, 0.4083645994626534, 7, 0.5069042541174733, 4, 414327, 260.63557428396865, 2.0969660888077091e-07, 0.03329930701856852, 8682]  - Score  -0.1007765482430596\n",
      "Current iter:  5  - Score  -0.5184708747168858  - Args:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]\n",
      "Last eval:  [0.08293216060373823, 57, 400, 100, 19798, 0.8773143587445786, 3, 0.24456885606236203, 4, 254361, 6.267403019545165e-07, 3.8195699908270306e-08, 0.0029281270257610842, 8418]  - Score  -0.4932644929680439\n",
      "Current iter:  6  - Score  -0.5184708747168858  - Args:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]\n",
      "Last eval:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]  - Score  -0.6843172927234125\n",
      "Current iter:  7  - Score  -0.6843172927234125  - Args:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]\n",
      "Last eval:  [0.42324568665282264, 203, 293, 37, 81251, 0.5850389357102426, 4, 0.8001005930590471, 5, 267367, 1.6977743026226538e-09, 0.0014254878711280096, 0.01761769861494498, 2214]  - Score  -0.3623421719518635\n",
      "Current iter:  8  - Score  -0.6843172927234125  - Args:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]\n",
      "Last eval:  [0.12620776965554556, 262, 244, 53, 52776, 0.9846210751689275, 6, 0.6332613654964749, 9, 485008, 0.010500201470771384, 3.0484544036415625e-08, 0.0025495207496935173, 9446]  - Score  -0.1007765482430596\n",
      "Current iter:  9  - Score  -0.6843172927234125  - Args:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]\n"
     ]
    }
   ],
   "source": [
    "gp_round = gp_minimize(func=objective,\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge',\n",
    "                       n_calls=10, # Minimum is 10 calls\n",
    "                       callback=[onstep],\n",
    "                       random_state=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "_uuid": "4ba09b481009f9e87378aae6c4f17024878d31e1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.12324384123200562, 414, 117, 41, 95838, 0.7684912331585715, 2, 0.47468749760756623, 5, 168945, 3.8096250370376167e-06, 2.625825905482537e-07, 51.89197450218198, 5038] -0.8572449837163276\n"
     ]
    }
   ],
   "source": [
    "x0, y0 = joblib.load('checkpoint.pkl')\n",
    "\n",
    "gp_round = gp_minimize(func=objective,\n",
    "                       x0=x0,              # already examined values for x\n",
    "                       y0=y0,              # observed values for x0\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Expected Improvement.\n",
    "                       n_calls=10,\n",
    "                       #callback=[onstep],\n",
    "                       random_state=3)\n",
    "\n",
    "best_parameters = gp_round.x\n",
    "best_result = gp_round.fun\n",
    "print(best_parameters, best_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "561467430472a5562fa25d04d2e3c4f311800ee6"
   },
   "source": [
    "# Practical example: Optimizing XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "_uuid": "c899ead400eb74ce87a9176aca1a47d1fcbf5946",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = xgb.XGBClassifier(\n",
    "        n_jobs = 1,\n",
    "        objective = 'binary:logistic',\n",
    "        silent=1,\n",
    "        tree_method='approx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "_uuid": "71b250ac2647634c4528c4c3a55de66b66127144",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "search_spaces = {'learning_rate': Real(0.01, 1.0, 'log-uniform'),\n",
    "                 'min_child_weight': Integer(0, 10),\n",
    "                 'max_depth': Integer(0, 50),\n",
    "                 'max_delta_step': Integer(0, 20), # Maximum delta step we allow each leaf output\n",
    "                 'subsample': Real(0.01, 1.0, 'uniform'),\n",
    "                 'colsample_bytree': Real(0.01, 1.0, 'uniform'), # subsample ratio of columns by tree\n",
    "                 'colsample_bylevel': Real(0.01, 1.0, 'uniform'), # subsample ratio by level in trees\n",
    "                 'reg_lambda': Real(1e-9, 1000, 'log-uniform'), # L2 regularization\n",
    "                 'reg_alpha': Real(1e-9, 1.0, 'log-uniform'), # L1 regularization\n",
    "                 'gamma': Real(1e-9, 0.5, 'log-uniform'), # Minimum loss reduction for partition\n",
    "                 'n_estimators': Integer(50, 100),\n",
    "                 'scale_pos_weight': Real(1e-6, 500, 'log-uniform')}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "_uuid": "5ef9b0986d0677d866304c50a3c503b0bc4a4f40"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "XGBoost took 289.50 seconds,  candidates checked: 38, best CV score: 0.904 ± 0.089\n",
      "Best parameters:\n",
      "{'colsample_bylevel': 0.01,\n",
      " 'colsample_bytree': 1.0,\n",
      " 'gamma': 1e-09,\n",
      " 'learning_rate': 1.0,\n",
      " 'max_delta_step': 0,\n",
      " 'max_depth': 33,\n",
      " 'min_child_weight': 10,\n",
      " 'n_estimators': 71,\n",
      " 'reg_alpha': 1.0,\n",
      " 'reg_lambda': 2.4728747872370573,\n",
      " 'scale_pos_weight': 499.99999999999994,\n",
      " 'subsample': 0.9929849363221644}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "opt = BayesSearchCV(clf,\n",
    "                    search_spaces,\n",
    "                    scoring=avg_prec,\n",
    "                    cv=skf,\n",
    "                    n_iter=40,\n",
    "                    n_jobs=-1,\n",
    "                    return_train_score=False,\n",
    "                    refit=True,\n",
    "                    optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                    random_state=22)\n",
    "    \n",
    "best_params = report_perf(opt, X, y_bin,'XGBoost',                           \n",
    "                          callbacks=[DeltaXStopper(0.001), \n",
    "                                     DeadlineStopper(60*5)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f8d79cc09922ea88e960afdd9fadc021dc71ce1c"
   },
   "source": [
    "# Practical Example: Optimizing CatBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "_uuid": "24c3a9bdb05e58c83d3ffd51be6163f43d26f1aa",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = CatBoostClassifier(loss_function='Logloss',\n",
    "                         verbose = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "_uuid": "277816facc46abcc9581dd6a485cd519130656b2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "search_spaces = {'iterations': Integer(10, 100),\n",
    "                 'depth': Integer(1, 8),\n",
    "                 'learning_rate': Real(0.01, 1.0, 'log-uniform'),\n",
    "                 'random_strength': Real(1e-9, 10, 'log-uniform'), # randomness for scoring splits\n",
    "                 'bagging_temperature': Real(0.0, 1.0), # settings of the Bayesian bootstrap\n",
    "                 'border_count': Integer(1, 255), # splits for numerical features\n",
    "                 'l2_leaf_reg': Integer(2, 30), # L2 regularization\n",
    "                 'scale_pos_weight':Real(0.01, 10.0, 'uniform')}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "_uuid": "816a4893a523ec3d9a155d6a095e0c023666b5d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CatBoost took 265.12 seconds,  candidates checked: 14, best CV score: 0.933 ± 0.078\n",
      "Best parameters:\n",
      "{'bagging_temperature': 1.0,\n",
      " 'border_count': 141,\n",
      " 'depth': 8,\n",
      " 'iterations': 75,\n",
      " 'l2_leaf_reg': 5,\n",
      " 'learning_rate': 0.8946167607229063,\n",
      " 'random_strength': 7.458051733842692e-05,\n",
      " 'scale_pos_weight': 1.0223472923161119}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "opt = BayesSearchCV(clf,\n",
    "                    search_spaces,\n",
    "                    scoring=avg_prec,\n",
    "                    cv=skf,\n",
    "                    n_iter=40,\n",
    "                    n_jobs=1,  # use just 1 job with CatBoost in order to avoid segmentation fault\n",
    "                    return_train_score=False,\n",
    "                    refit=True,\n",
    "                    optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                    random_state=22)\n",
    "\n",
    "best_params = report_perf(opt, X, y_bin,'CatBoost', \n",
    "                          callbacks=[DeltaXStopper(0.001), \n",
    "                                     DeadlineStopper(60*5)])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}