005-data-sci-tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| &nbsp; | &nbsp; | &nbsp; |\n",
    "|--------|--------|--------|\n",
    "| [Return to Index Notebook](./index.ipynb) | [View on GitHub](https://github.com/hpcgarage/prymer) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/hpcgarage/prymer/blob/master/005-data-sci-tools.ipynb) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| &nbsp; | &nbsp; |\n",
    "|---|---|\n",
    "| ![Cover of Vanderplas's \"Whirlwind Tour of Python\" book](vanderplas-cover-small.jpg) | *This notebook contains an excerpt from the [Whirlwind Tour of Python](http://www.oreilly.com/programming/free/a-whirlwind-tour-of-python.csp) by Jake VanderPlas; the content is available [on GitHub](https://github.com/jakevdp/WhirlwindTourOfPython). <br /><br /> The text and code are released under the [CC0](https://github.com/jakevdp/WhirlwindTourOfPython/blob/master/LICENSE) license; see also the companion project, the [Python Data Science Handbook](https://github.com/jakevdp/PythonDataScienceHandbook).* |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **We will cover two of the packages mentioned below, Numpy/Scipy and pandas, in much greater depth later in the semester. For now, you won't need to learn these packages for the first part of our course.**\n",
    "\n",
    "# A Preview of Data Science Tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is part of a [Python primer (Prymer)](https://github.com/hpcgarage/prymer).\n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/hpcgarage/prymer/blob/master/005-data-sci-tools.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to spring from here and go farther in using Python for scientific computing or data science, there are a few packages that will make your life much easier. This section will introduce and preview several of the more important ones, and give you an idea of the types of applications they are designed for.\n",
    "\n",
    "The most important packages of the data science stack are: [Numpy/Scipy](https://www.scipy.org) (for numerical computing), [pandas](https://pandas.pydata.org) (for managing data frames), [matplotlib](https://matplotlib.org) (for graphing), and [scikit-learn](https://scikit-learn.org/stable/) (for machine learning). Let's take a brief look at each of these in turn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NumPy: Numerical Python\n",
    "\n",
    "NumPy provides an efficient way to store and manipulate multi-dimensional dense arrays in Python.\n",
    "The important features of NumPy are:\n",
    "\n",
    "- It provides an ``ndarray`` structure, which allows efficient storage and manipulation of vectors, matrices, and higher-dimensional datasets.\n",
    "- It provides a readable and efficient syntax for operating on this data, from simple element-wise arithmetic to more complicated linear algebraic operations.\n",
    "\n",
    "In the simplest case, NumPy arrays look a lot like Python lists.\n",
    "For example, here is an array containing the range of numbers 1 to 9 (compare this with Python's built-in ``range()``):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "x = np.arange(1, 10)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy's arrays offer both efficient storage of data, as well as efficient element-wise operations on the data.\n",
    "For example, to square each element of the array, we can apply the \"``**``\" operator to the array directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare this with the much more verbose Python-style list comprehension for the same result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[val ** 2 for val in range(1, 10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, Numpy array operations can be much, much faster!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_list = list(range(10000000))\n",
    "input_np = np.array(input_list)\n",
    "\n",
    "t_py = %timeit -o [v**2 for v in input_list]\n",
    "t_np = %timeit -o (input_np ** 2)\n",
    "\n",
    "print(\"\\n==> Speedup ~ {:.1f}x\".format(t_py.average / t_np.average))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike Python lists (which are limited to one dimension, unless you use nested lists), NumPy arrays can be multi-dimensional.\n",
    "For example, here we will reshape our ``x`` array into a 3x3 array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = x.reshape((3, 3))\n",
    "M"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A two-dimensional array is one representation of a matrix, and NumPy knows how to efficiently do typical matrix operations. For example, you can compute the transpose using ``.T``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or a matrix-vector product using ``np.dot``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(M, [5, 6, 7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M.dot([5, 6, 7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Syntatic sugar variant\n",
    "M @ [5, 6, 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and even more sophisticated operations like eigenvalue decomposition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.eigvals(M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such linear algebraic manipulation underpins much of modern data analysis, particularly when it comes to the fields of machine learning and data mining.\n",
    "\n",
    "For more information on NumPy, see [Resources for Further Learning](16-Further-Resources.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas: Labeled Column-oriented Data\n",
    "\n",
    "Pandas is a much newer package than NumPy, and is in fact built on top of it.\n",
    "What Pandas provides is a labeled interface to multi-dimensional data, in the form of a DataFrame object that will feel very familiar to users of R and related languages.\n",
    "DataFrames in Pandas look something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame({'label': ['A', 'B', 'C', 'A', 'B', 'C'],\n",
    "                   'value': [1, 2, 3, 4, 5, 6]})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Pandas interface allows you to do things like select columns by name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['label']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply string operations across string entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['label'].str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply aggregates across numerical entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['value'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, perhaps most importantly, do efficient database-style joins and groupings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('label').sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here in one line we have computed the sum of all objects sharing the same label, something that is much more verbose (and much less efficient) using tools provided in Numpy and core Python.\n",
    "\n",
    "For more information on using Pandas, see [Resources for Further Learning](16-Further-Resources.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matplotlib MatLab-style scientific visualization\n",
    "\n",
    "Matplotlib is currently the most popular scientific visualization packages in Python.\n",
    "Even proponents admit that its interface is sometimes overly verbose, but it is a powerful library for creating a large range of plots.\n",
    "\n",
    "To use Matplotlib, we can start by enabling the notebook mode (for use in the Jupyter notebook) and then importing the package as ``plt``\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this if using Jupyter notebook\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')  # make graphs in the style of R's ggplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create some data (as NumPy arrays, of course) and plot the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10)  # range of values from 0 to 10\n",
    "y = np.sin(x)           # sine of these values\n",
    "plt.plot(x, y);         # plot as a line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run this code live, you will see an interactive plot that lets you pan, zoom, and scroll to explore the data.\n",
    "\n",
    "This is the simplest example of a Matplotlib plot; for ideas on the wide range of plot types available, see [Matplotlib's online gallery](http://matplotlib.org/gallery.html) as well as other references listed in [Resources for Further Learning](16-Further-Resources.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SciPy: Scientific Python\n",
    "\n",
    "SciPy is a collection of scientific functionality that is built on NumPy.\n",
    "The package began as a set of Python wrappers to well-known Fortran libraries for numerical computing, and has grown from there.\n",
    "The package is arranged as a set of submodules, each implementing some class of numerical algorithms.\n",
    "Here is an incomplete sample of some of the more important ones for data science:\n",
    "\n",
    "- ``scipy.fftpack``: Fast Fourier transforms\n",
    "- ``scipy.integrate``: Numerical integration\n",
    "- ``scipy.interpolate``: Numerical interpolation\n",
    "- ``scipy.linalg``: Linear algebra routines\n",
    "- ``scipy.optimize``: Numerical optimization of functions\n",
    "- ``scipy.sparse``: Sparse matrix storage and linear algebra\n",
    "- ``scipy.stats``: Statistical analysis routines\n",
    "\n",
    "For example, let's take a look at interpolating a smooth curve between some data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import interpolate\n",
    "\n",
    "# choose eight points between 0 and 10\n",
    "x = np.linspace(0, 10, 8)\n",
    "y = np.sin(x)\n",
    "\n",
    "# create a cubic interpolation function\n",
    "func = interpolate.interp1d(x, y, kind='cubic')\n",
    "\n",
    "# interpolate on a grid of 1,000 points\n",
    "x_interp = np.linspace(0, 10, 1000)\n",
    "y_interp = func(x_interp)\n",
    "\n",
    "# plot the results\n",
    "plt.figure()  # new figure\n",
    "plt.plot(x, y, 'o')\n",
    "plt.plot(x_interp, y_interp);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we see is a smooth interpolation between the points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Data Science Packages\n",
    "\n",
    "There are many other data science packages, including general tools like [Scikit-Learn](http://scikit-learn.org) for machine learning, [Scikit-Image](http://scikit-image.org) for image analysis, and [Statsmodels](http://statsmodels.sourceforge.net/) for statistical modeling, as well as more domain-specific packages like [AstroPy](http://astropy.org) for astronomy and astrophysics, [NiPy](http://nipy.org/) for neuro-imaging, and many, many more.\n",
    "\n",
    "No matter what type of scientific, numerical, or statistical problem you are facing, it's likely there is a Python package out there that can help you solve it."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}