2.Text Analysis Using nltk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Analysis Using nltk.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.text import Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['two', 'plus', 'two', 'is', 'four']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_string = \"Two plus two is four, minus one that's three — quick maths. Every day man's on the block. Smoke trees. See your girl in the park, that girl is an uckers. When the thing went quack quack quack, your men were ducking! Hold tight Asznee, my brother. He's got a pumpy. Hold tight my man, my guy. He's got a frisbee. I trap, trap, trap on the phone. Moving that cornflakes, rice crispies. Hold tight my girl Whitney.\"\n",
    "tokens = word_tokenize(my_string)\n",
    "tokens = [word.lower() for word in tokens]\n",
    "tokens[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Text: two plus two is four , minus one...>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = Text(tokens)\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method of converting raw strings to NLTK `Text` instances can be used when reading text from a file. For instance:\n",
    "```python\n",
    "f = open('my-file.txt','rU') # Opening a file with the mode 'U' or 'rU' will open a file for reading in universal newline mode. All three line ending conventions will be translated to a \"\\n\"\n",
    "raw = f.read()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 1 of 1 matches:\n",
      " girl in the park , that girl is an uckers . when the thing went quack quack q\n"
     ]
    }
   ],
   "source": [
    "t.concordance('uckers') # concordance() is a method of the Text class of NLTK. It finds words and displays a context window. Word matching is not case-sensitive.\n",
    "# concordance() is defined as follows: concordance(self, word, width=79, lines=25). Note default values for optional params."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hold tight; quack quack\n"
     ]
    }
   ],
   "source": [
    "t.collocations() # def collocations(self, num=20, window_size=2). num is the max no. of collocations to print."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.count('quack')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.index('two')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "guy\n"
     ]
    }
   ],
   "source": [
    "t.similar('brother') # similar(self, word, num=20). Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t.dispersion_plot(['man', 'thing', 'quack']) # Reveals patterns in word positions. Each stripe represents an instance of a word, and each row represents the entire text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEeCAYAAACHXhKxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8XHW5x/HPk6RJmzZdaSFQaO3CUrrQJlUooCLoRcQFRARZxWtdLsriRa7LveJ2VUTwioIUEAQUBUShRQWVtbImpXQB2kJLhbaU7lu6pMlz/zhn2mlIkzMzmTmTOd/36zWvZCZ5zu9JO3Oec36/3/kdc3dERCS5yuJOQERE4qVCICKScCoEIiIJp0IgIpJwKgQiIgmnQiAiknAqBCIiCadCICKScCoEIiIJVxF3AlHss88+Pnz48Kxit27dSq9evbJuW/GKV7ziu2t8Y2Pjancf3OkvunvRP+rq6jxbDQ0NWccqXvGKV3x3jgcaPMI+Vl1DIiIJp0IgIpJwKgQiIgmnQiAiknAqBCIiCadCICKScCVfCJpbdAc2EZGOdIsLyrKxvmkHF9z6HK+t2kjjZMfM4k5JRKQolewZQb9ePVi2fitrt7by8pub4k5HRKRolWwhMDOOGRVcWT1z0eqYsxERKV4lWwgA3n3wPgA8vmhVzJmIiBSvki4ER48KCsGzS9ayrbkl5mxERIpTSReCffpUMbxfBdt3ttK4dF3c6YiIFKWSLgQA4/etBOAJjROIiLSr5AvBhH2rAJj5isYJRETaU/KF4LDBlVRWlDF/+UbWbtkRdzoiIkWn5AtBVbkxefgA3OGfr6h7SESkrZIvBADHjg6uJ3hC00hFRN4mEYXgmHAa6cxFqwnu3iYiIimJKARjavsyqHclyzdsY/HqLXGnIyJSVBJRCMrKjClpZwUiIrJbIgoBwLFhIdD1BCIie0pMIThmdFAInl68huaW1pizEREpHokpBPv378XIwb3ZvH0ns19fH3c6IiJFI2+FwMx+ZWZvmdm8tNcGmtnfzGxR+HVAvtpvz+5ppOoeEhFJyecZwa3AiW1e+y/gH+4+GvhH+Lxgdk8j1fUEIiIpeSsE7v44sLbNyx8Ffh1+/2vgY/lqvz1HjhxERZnxwhsb2LituZBNi4gUrUKPEezr7isAwq9DCtl4n6oKJh7Un5ZW56lX1xSyaRGRomX5vNLWzIYDM9x9bPh8vbv3T/v5Ondvd5zAzKYCUwFqa2vrpk+fnlUOTU1NVFdX73p+14ub+f38zZw4sprPTuqbcXyu7Ste8YpXfKHi6+vrG929vtNfdPe8PYDhwLy05wuA2vD7WmBBlO3U1dV5thoaGvZ43rh0rQ+7fIa/98ePZBWfa/uKV7ziFV+oeKDBI+xjC901dD9wXvj9ecB9BW6f8Qf0o6ZnBUtWb+H1tU2Fbl5EpOjkc/roncBTwCFm9oaZfQb4IfB+M1sEvD98XlAV5WVMGTkIgJlallpEhIp8bdjdz9zLj47PV5tRHTN6MA/OX8nMRas5850HxZ2OiEisEnNlcbrUukP/fHU1La1allpEki2RhWDYoGqGDujF+qZm5i/fEHc6IiKxSmQhMDMtNyEiEkpkIQA4dnRqWWotNyEiyZbYQjBl5CDMoHHpOpp27Iw7HRGR2CS2EPSvrmT8Af1obnGeWdJ2SSQRkeRIbCGA3Ter0e0rRSTJkl0IRgUDxioEIpJkiS4Ek4b1p7qynAUrN/HWxm1xpyMiEotEF4KqinLe9Y6BgJabEJHkSnQhgGC5CdD1BCKSXIkvBKnrCWa+sjq1VLaISKIkvhCMHtKHfftWsWrTdhas3BR3OiIiBZf4QmBmHD1K00hFJLkSXwgA3q1xAhFJMBUC2HVG8MySNWzf2RJzNiIihaVCAAyuqeLQ/WrY1txK42vr4k5HRKSgVAhCu1Yj1fUEIpIwKgSh1PUEGjAWkaRRIQi9c/hAKsvLmLd8A+u27Ig7HRGRglEhCPWqLGfyOwbgHtzLWEQkKVQI0mg1UhFJIhWCNLtvX6nlJkQkOVQI0oyp7cvA3pUsW7+VJau3xJ2OiEhBqBCkKSszpowcBGhZahFJDhWCNtK7h0REkkCFoI3U9QRPv7qGnS2tMWcjIpJ/KgRtHNC/FyMG92bT9p288Mb6uNMREck7FYJ2HDtK3UMikhwqBO3QchMikiQqBO04csRAysuM519fz5ZmjROISGmLpRCY2SVmNt/M5pnZnWbWM4489qamZw8mHtifllZn/ltad0hESlvBC4GZHQB8Gah397FAOXBGofPozDHhNNIXVqoQiEhpi6trqALoZWYVQDWwPKY89urYcJxgzsrtMWciIpJfFseaOmZ2EfB9YCvwkLuf1c7vTAWmAtTW1tZNnz49q7aampqorq7OOK6l1Tn/vrdo2uncdPJgBvQqL2j7ile84hWfa3x9fX2ju9d3+ovuXtAHMAB4GBgM9AD+BJzdUUxdXZ1nq6GhIevYM6c95cMun+F/f/HNWNpXvOIVr/hcAA0eYb8cR9fQCcASd1/l7s3AvcCUGPLo1LgD+gEw540NMWciIpI/cRSCfwFHmlm1mRlwPPBSDHl0atzQoBDMXaZCICKlq+CFwN2fAe4BZgFzwxymFTqPKMYf0B8Izghc9ycQkRJVEUej7v4t4FtxtJ2JAwf2ok8PY/Xm7by5cRu1/XrFnZKISJfTlcUdMDNGDuwBaJxAREqXCkEnRg4ICsFcFQIRKVEqBJ1IFYI5GjAWkRKlQtCJEWEhmLdMA8YiUppUCDoxuLqMgb0rWbtlB8vWb407HRGRLqdC0Akz23VhmcYJRKQUqRBEMD68sEzjBCJSilQIItAZgYiUMhWCCMYPTV1hvF4DxiJSclQIIti3bxWDa6rYuG0n/1rbFHc6IiJdSoUggj0GjDVOICIlRoUgIo0TiEipUiGIaNfMIRUCESkxKgQRpc4I5i3bQGurBoxFpHSoEEQ0pG9P9uvbk03bd/Lami1xpyMi0mVUCDKgO5aJSClSIcjAeN3DWERKkApBBnadEagQiEgJUSHIQGrAeP7yDbRowFhESkTGhcDMBpjZ+HwkU+wG9anigP692LKjhSWrN8edjohIl4hUCMzsUTPra2YDgReAW8zs6vymVpzGaZxAREpM1DOCfu6+ETgVuMXd64AT8pdW8RqnC8tEpMRELQQVZlYLnA7MyGM+RW+8ppCKSImJWgi+DTwIvOLuz5nZCGBR/tIqXukDxjtbWmPORkQkd1ELwQp3H+/uXwRw98VAIscI+ldXctDAarY1t/LKKg0Yi0j3F7UQXBvxtUTQSqQiUkoqOvqhmR0FTAEGm9mlaT/qC5TnM7FiNm5oPx6Yu4K5yzbwifoD405HRCQnHRYCoBLoE/5eTdrrG4HT8pVUsdNSEyJSSjosBO7+GPCYmd3q7ksLlFPROzwsBC+u2EhzSys9ynWBtoh0X52dEaRUmdk0YHh6jLu/Lx9JFbt+vXrwjn16s2T1Fhau3MTh+/eLOyURkaxFLQR3A78EbgJacm3UzPqH2xoLOHCBuz+V63YLadwB/Viyegtz39igQiAi3VrUQrDT3a/vwnb/D/iru59mZpVAdRduuyDGD+3H/S8sZ86yDZwRdzIiIjmI2rk93cy+aGa1ZjYw9cimQTPrC7wbuBnA3Xe4+/psthWnsZpCKiIlIuoZwXnh18vSXnNgRBZtjgBWESxcNwFoBC5y9251/8fD9++LGbz85ka272yhqiKxs2lFpJsz98Kuq29m9cDTwNHu/oyZ/R+w0d3/u83vTQWmAtTW1tZNnz49q/aampqors6+56mj+C//dRXLNrVw5QmDGDmgR8HbV7ziFa/4jtTX1ze6e32nv+junT6Ac9t7RIltZ1v7Aa+lPT8WeKCjmLq6Os9WQ0ND1rGdxV/8u+d92OUz/I6nX4ulfcUrXvGK7wjQ4BH2y1HHCCanPY4FrgA+Er0u7VF43gReN7NDwpeOB17MZltx01ITIlIKIo0RuPuX0p+bWT/g9hza/RLwm3DG0GLg0zlsKzbjdW8CESkBUQeL22oCRmfbqLvPBjrvtypyY/bvS5nBwpWb2NbcQs8eGjAWke4nUiEws+kEs4QgWGzuMOCufCXVXVRXVjBqSB8WrtzMSys2MvGgAXGnJCKSsahnBFelfb8TWOrub+Qhn25n3AH9WbhyM/OWbVAhEJFuKdJgsQeLz71MsALpAGBHPpPqTjROICLdXaRCYGanA88CnyC4b/EzZpbYZajTjdM9jEWkm4vaNfQNYLK7vwVgZoOBvwP35Cux7mJMbV/Ky4yFKzexdUcLvSo1YCwi3UvU6wjKUkUgtCaD2JLWs0c5B+9bQ6vDiyt0ViAi3U/UnflfzexBMzvfzM4HHgD+nL+0uhfdsUxEurMOC4GZjTKzo939MuAGYDwwAXgKmFaA/LqFsUN1hbGIdF+dnRH8FNgE4O73uvul7n4JwdnAT/OdXHex64xAA8Yi0g11VgiGu/ucti+6ewPBbSsFOLS2hh7lxqurNrNl+8640xERyUhnhaBnBz/r1ZWJdGdVFeUcsl8N7jB/+ca40xERyUhnheA5M/ts2xfN7DMEN5SR0LgD+gMw541ud7M1EUm4zq4juBj4o5mdxe4dfz1QCZySz8S6m/FD+3Hns7qwTES6nw4LgbuvBKaY2XHA2PDlB9z94bxn1s3o3gQi0l1FvR/BI8Ajec6lWzt43xoqy8tYvHoLG7c107dn+7euFBEpNro6uItUVpRxWG0NAPPUPSQi3YgKQRdKLUCnQiAi3YkKQRcav2vmkAqBiHQfKgRdSEtSi0h3pELQhUYP6UNVRRlL1zSxoak57nRERCJRIehCFeVlHL5/X0BnBSLSfagQdLHxQ8NxgmW6wlhEugcVgi42VheWiUg3o0LQxXQzexHpblQIutjIwX3o1aOcZeu3snbLjrjTERHplApBFysvM8YeoAFjEek+VAjyILUk9VwtSS0i3YAKQR5onEBEuhMVgjzQFcYi0p2oEOTBOwb1pk9VBSs2bGPdtpa40xER6ZAKQR6UldmuK4wXr9NSEyJS3GIrBGZWbmbPm9mMuHLIp9Q4wavrdsaciYhIx+I8I7gIeCnG9vNqXLjUxKtrdUYgIsUt0q0qu5qZDQU+BHwfuDSOHPJtfLjUxMtrdvCThxZkvZ0VKzbx6Jrs46u2bmPSJMfMst6GiJQ2c/fCN2p2D/ADoAb4T3c/uZ3fmQpMBaitra2bPn16Vm01NTVRXV2dda7Zxrs7F0xfxcbtrVm33VX+fWINHxzVO6vYuP79FK94xeceX19f3+ju9Z39XsHPCMzsZOAtd280s/fu7ffcfRowDaC+vt7r6uqyaq+xsZFsY3ONv2XwOu5+fA77779/1u0vX7486/i1W3Zw65OvcesLmzm+/nCOGjko423E+e+neMUrPrf4qOLoGjoa+IiZnQT0BPqa2R3ufnYMueRV3bABMKYPdXWjs95GY+PGnOLXrVnFfQu28B+/ncX9Fx7N0AHZH12ISGkq+GCxu3/N3Ye6+3DgDODhUiwCxeKscX14z8GDWbtlB1Nva2TrDl3XICJ70nUEJa7cjJ+dMZHhg6p5ccVGLrvnBeIYFxKR4hVrIXD3R9sbKJau1a+6BzeeW0/vynJmzFnB9Y+9GndKIlJEdEaQEKP3reGnZ0wE4McPLuCRl9+KOSMRKRYqBAny/jH7cun7D8Ydvnzn87y6anPcKYlIEVAhSJgLjxvFB8fux6btO/nsbQ1s3KYrn0WSToUgYcrKjKs+MYFD96th8aotXPy72bS0avBYJMlUCBKod1UF086pp391Dx5++S2u/lv2S1iISPenQpBQBw2q5hefmkR5mfGLR15lxpzlcackIjFRIUiwo0ftw9dPOgyAy+6ew4vLN8ackYjEQYUg4S44ejgfnzSUrc0tfPa2BtZu2RF3SiJSYCoECWdmfP+UsUw4sD/L1m/li79ppLkl/hVTRaRwVAiEnj3KueHsOgbXVPH04rV8/4GSvV+QiLRDhUAA2K9fT355dh2V5WXc+uRr3PXc63GnJCIFokIgu9QNG8D3PjYWgG/+aR6NS9fFnJGIFIIKgezh9MkHct5Rw9jR0srn72hkzVYtWy1S6mK5Z7EUt2+ePIYFKzfx9OK1fOPhNdzy4tNZb2vTpk3UNCY3vrc38ZMxzfTr1SPrbYjkmwqBvE2P8jKuO6uOj/5iJq+v3cqqxWty2+CqZMdf/Lvnuem8yZSXWW55iOSJCoG0a2DvSh66+D38/h/PcPDog7PezsJFCxMbv7W5hYvvbOSRBau46qEFXH7ioVnnIZJPKgSyV70qyxk3pIq6UftkvY2qDUsTHf+Vo/rz3SfWc/2jrzKmti8fnrB/1tsSyRcNFovk0bghVXzzQ+EyHve8wLxlG2LOSOTtVAhE8uz8KcP5RN1QtjW38rnbG1mzeXvcKYnsQYVAJM/MjO+dMpYjwmU8vvCbWVrGQ4qKCoFIAVRVlHPDOXUMqani2SVr+e6MF+NOSWQXFQKRAtm3b09uOCdYxuO2p5byu2f/FXdKIoAKgUhBTTxoAN87JVjG47/vm0fj0rUxZySiQiBScKfXH8j5U4bT3OJ87vZZrNiwNe6UJOFUCERi8I0PHcaUkYNYvXk7n7u9kW3NWtNJ4qNCIBKDHuVl/PxTkxg6oBdz3tjA1++di7vHnZYklAqBSEwG9q7kxnPr6dWjnHufX8bNM5fEnZIklAqBSIwOq+3LT06fAMD//vklnli0KuaMJIlUCERidtK4Wr70vlG0Olz42+dZumZL3ClJwqgQiBSBS044mBMOG8KGrc189rYGNm/fGXdKkiAFLwRmdqCZPWJmL5nZfDO7qNA5iBSbsjLjmk8ewaghfVi4cjNfuWs2ra0aPJbCiOOMYCfwFXc/DDgS+A8zGxNDHiJFpaZnD6adU0dNzwoenL+Snz28KO6UJCEKXgjcfYW7zwq/3wS8BBxQ6DxEitGIwX249syJlBn89O+LeGbZtrhTkgSI9cY0ZjYcmAg8E2ceIsXkvYcM4asnHsoP//IyVz65nmue+XPW2/JWx+5VfFzxNZXGz/uvZsrI7G9uVAgW10UsZtYHeAz4vrvf287PpwJTAWpra+umT5+eVTtNTU1UV1dnnafiFR9HvLtz0/ObePDVJjRS0L316WH86IRB7Ncn8+PuXN9/9fX1je5e3+kvunvBH0AP4EHg0ii/X1dX59lqaGjIOlbxio87/ulnn/PtzS1ZP55SfGzxW3fs9FN/+jcfdvkM/8DVj/nmbc0Ff/8ADR5hH1vwriEzM+Bm4CV3v7rQ7Yt0JxVlRmVF9kN5PRQfa/xF7+rHFf/cwoKVm7j0rtlcf1YdZWWW9fbyJY5ZQ0cD5wDvM7PZ4eOkGPIQEcmr3j3KuPHc+l0zwa59+JW4U2pXHLOGZrq7uft4dz8ifGQ/GiMiUsRGDu7Dz86YiBlc8/eFPDT/zbhTehtdWSwikmfHHTqEr/7boQBc8vvZLFy5KeaM9qRCICJSAJ9/zwg+PGF/tuxoYeptDWxoao47pV1UCERECsDMuPLj4zl8/768tqaJC++cxc6W1rjTAlQIREQKpldlOTecU8fA3pU8sWg1Vz64IO6UABUCEZGCGjqgmuvOmkRFmTHt8cX86fllcaekQiAiUmhHjhjEtz4crLV5+R/mMPeNDbHmo0IgIhKDs48cxhmTD2T7zlam3t7Aqk3bY8tFhUBEJAZmxrc/ejh1wwawYsM2vnBHIzt2xjN4rEIgIhKTqopyrj97Evv17UnD0nVcMX1+LHmoEIiIxGhITU9uOKeOyooyfvvMv7jj6aUFz0GFQEQkZhMO7M8PTx0HwBX3z+fZJWsL2r4KgYhIETh10lA+c8w72NnqfOGORpat31qwtlUIRESKxNc+eCjHjNqHNVt28LnbG9i+szC3JVIhEBEpEhXlZfz8UxM5aGA185Zt5LqGDambeeWVCoGISBHpX13JjefWU11ZzszXt3HjE4vz3qYKgYhIkTlkvxquPv0IBvYso374wLy3V/BbVYqISOdOHLsfNVv2YdJBA/Lels4IRESKVM8c7pecCRUCEZGEUyEQEUk4FQIRkYRTIRARSTgVAhGRhFMhEBFJOBUCEZGEs0KsY5ErM1sFZLtI9z7A6hyaV7ziFa/47ho/zN0Hd/pb7l7SD6BB8YpXvOKTGB/1oa4hEZGEUyEQEUm4JBSCaYpXvOIVn9D4SLrFYLGIiORPEs4IRESkAyoEIiIJpxvTRGBmtcBad98edy75ZmZVbf/O9l4TKVVd8RkwswHAaKBn6jV3f7zrsuxaiTkjMLP9cgi/HXjZzK6K2NbRZtY7/P5sM7vazIbl0H5GzCyXe9s9FfG19to9taNH1ATM7Dttnpeb2W+ixufCzN4R5bU85zDBzC4MHxOyiM/5/WdmB5jZFDN7d+qRaR7ZMLMyMzu9EG11IOvPAICZ/TvwOPAg8O3w6xXZJGJmA8xsfDaxmUjSGcHNwIeyCXT3E8zMgDERQ64HJoQf4q+Gbd8GvKezQDPbF/hfYH93/6CZjQGOcvebM0j5GTObDdwC/MUjzAgIC+UBQC8zmwhY+KO+QHXEdj/cwc8cuDfidg4ys6+5+w/MrAq4G5jVWZCZTQ/baT8B949EaPsPwKQ2r90D1EWIxcwGEXzojw5zmQl8x93XRIy/CPgsu/+t7jCzae5+bZT4UNbvvzCHHwGfBF4EWsKXnWDnFiW+Cvg4MJy0fYy7f2dvMWm/02pmFwJ3RWlrL+0fTPBvsK+7jw13pB9x9+91EtcVnwGAi4DJwNPufpyZHUpQEKLm/yjwEYJ/u9nAKjN7zN0vzSCHjGjWUB6Y2Sx3n2Rm/wMsc/ebU69FiP0LwQ78G+4+wcwqgOfdfVwG7RtwAnAB8E7g98Ct7r6wg5jzgPOBeqAh7UebwthIO3EzKwNOc/dcPsgG/AaYCxxHUMyuiRCX2tGdCuwH3BE+PxN4zd2/3kHsocDhwJXAZWk/6gtc5u6HR8z9bwQ7zFTbZwHvdfcTIsbPISj8W8LnvYGn3D3yUWEu778wfgEwPtvuQDP7K7ABaGR3IcHdfxIx/r+BrQTv2y1p8Wsjxj9G8H94g7tPDF+b5+5jO4nrqs/Ac+4+OTwYe5e7bzez2e5+RMT45919YnhmcaC7f8vM5mTyHshYIS5fTtoDeAz4GrCQYIdUDsyNGPtc+PX5tNdm55DLccAyYH2Y11Gd/P7Hu+DvfzzLuElpj3cRHA39IvVaLu13lhPwUYICvCb8mnr8DJiSQduN7bwWeZkAguLXM+15z6jvna54/4XxfwH65PD/Py/H98+Sdh6LM4jP6TOU62cA+CPQn+DM8HHgPuDPGb4HaoGHgMnha3NyyamzR5K6hgrpk8CngM+4+5tmdhDw44ixW8LuBQcwsyMJjq4iC+PPBs4F3gS+BNwPHEHQzbLXPm93/4OZfYjg6Dh9oKvT0/o0fzOz/yTzI7q2R4zrCLrjfkLw7/G+iO0PNrMR7r4YdvXxd7jwlrvfB9xnZke5e+T+4HY8YmZnsLtr4zTggQzibyHo2vtj+PxjBF07mcjl/QfQBMw2s38Au84K3P3LEeOfNLNx7j43gzZ3cfdcx2RWm9lIdn+GTgNWZBD/DzO7GkiNizxG0L0X6XPo7qeE315hZo8A/YC/ZtD+dwjGFWa6+3NmNgJYlEF8xtQ1VGTMbBJwLTAWmEewAzvN3edksI2FBAPcv3L3ZW1+drm7/6iD2F8S9IceB9xEsCN71t0/k0H7S9p52d19RNRt5MLMTiS4InNx+NJw4HPu/mCE2MEEffTD2bN/+4KIbW8CegOt4Utl7C6G7u59I2xjEnAMQR/14+7+fJS2u0rYRfI27v7riPEvEsyYWUxQSCwIj9a1YWbn7qX92yLGjyD4/59CcDCxBDjL3SOtYGxmfyD47KX+3nOACe4eecJDd6NC0IXCHaADq9z9XTlspwI4hOADtMDdmzOMnwx8HRjGnjuzTj+Iqb7ItK99gHvd/QMZ/RE5MLP/Ba509/Xh8wHAV9z9mxlsowo4NHz6skfs7zazJ4EneHv/9h+itp2LcMbUE8CTHo4TZBA7092PCYtR+gc7tSPutAh1hXCG0gDg2PClx4H1GeyI0wfGewLHA7Pc/bSI8VUEBzDDgYHARoK/P9JZbXv9+Zn08efKzG6hnUkPUQ9GsqGuoS7UBae0Ke9k9xHpJDOLfDQUugP4T4KjmtZOfretreHXJjPbn6DPPNLfZWbvc/eHbS9TRT3iYBvwQU8b2HX3dWZ2EtBhIdhbu8DI8N8wSvvV7n55xDz3lkcuc8hfIxjc/lm4Q3+C4Kzgvs4C3f2Y8GtNpjmnM7PRwA8IuuXS/4aoZ3QfA/6dYOaTEZyd3khwptspd/9Sm3z6hduI6j6CMbFZwPIM4lK2mtkx7j4zbP9odn8uCmFG2vc9gVPI7u+ITIWgyJjZ7cBIgoHS9Kl7mRSCVe4+PcsUZphZf4I+5Vlh2zdFjH0P8DDBNFInPBJN+xq1EJRb2gU8ZtYLqIoQ1xXTV2eY2Unu/ucIv/s24UyPi4ChBP+HRxLMQY80vuHuvwJ+FU5lPJ2goE8Fctq5Z+gW4FvANQRdhJ9m91TKKD4DHOm7Zz79iODfIJMpsOmaCAprVEPd/cQs2wL4PHBbWIAg6F5qt7ssH9qefZrZncDf89mmuoaKjJm9BIzxHP5jzOx4gqPKtoN9UXfEqe1UEcxgyXSw+ivsLgCE328gmFEzO0L8VwnmUadOkS8A7nf3KzPJIxNp3SlG0Me/HWgmw24VM5vL7jnkR6TmkLv7JyPG30RwJL6S4GxgJkG3yM4M/6SsmVmju9eZ2VwPpy2b2RPufmxnseHvziWY7bItfN6TYCZPpCnQtuf1IOXAYcBd7v5fEeOnAddmO1htZqn5+n3Cr5vJ4P3b1czsEOABdx+VrzZ0RlB85hFM+ctklkNbnyboH+/B7q6hyEfkZjaFtMHSLLqm6gjmYt9PsCP9EPAc8Hkzu7uzHbq7XxnuTI4P478bZaA3Lf9+BEe0kWd95Nqdkmabu28zs9SyBC+HH+SoBhHF+RTCAAAH6ElEQVTs/NYDa4HVhSwCoW3h9SCLLLi4axkwJIP4XGc+pV/BvxNY6u5vdBYUvmec4H37aTPLarCa4L2b/v79FBm8f3PV5qDECWb+5dRd2WmbOiMoDmlHQTUE0zyfZc+j+ShXxaa2NTfq0Vc7se12TWUwdRAze5BgLvbm8HkfgqtzTyE4qop6hXZWcpn1Ec7YaWsDwc6o0x1yuPP7NHAxQXfQOqCHu58UMf3Udg4D/g24BCh396GZxOcinGzwEsFc+O8SXFR3pbs/k8E2cpr5ZMEV9pPDp8+6+1sRYjpcRiODwepY379x0BlB8biK4EPzI4IjqJTUa5l42szGuPuLWeRRT45dU8BBwI60580EN9Heamadzt4Jr524lqBLoJLgCHlLBrNeRrr7x9Oef9uCqzyjuI7gArZUt8I44AVgkJl93t0f6ig41znkZnYywWybdxPMvHmYoIuokJxgcHYYwVklBIO9ka9sdfdZRFgWpD0WrDX0Y+BRgvf/tWZ2mbvf00mbkXb0EeT0/s3WXg5Cdgn/TfNChaBIuPtjAGbWI/V9SjhYmoljgPPC6ayZnhp3RdfUbwmKUWqmy4eBOy1YLiFKcfo5cAbBxW/1BBfGZdI/msusj9cILsSaH8aOIViu4LsEXWt7LQRhd8ocD5cyaPv/GNGpBBcT/Z+7Lw+3m+mBQK5+Q/A3zyXzWWdd4RsEYwxvwa5rO/5OcFReCLm+f7OVfkHl26b/Ev2Cyoypa6hImNkXgC8CI4BX035UA/zT3c/OYFvtniJ3dMTUlV1T4fbq2N01MNPdGzoJSY9tcPd6S1tfxcyedPcpEeOPIOgW2mPWh0e4KK+jOeRR5pJbsErq19z9X1FybSf+bWsCWb7XmXl7DjNTU1Hj0LZrMyywL2Tb3ZllDlm/f7ug7V4E+4JjCD6TTwDXpwbf80FnBMXjtwRrvPwASJ8dsckjLraVkuUpcld2TeHujQQXZWWjycwqCZY5uJLg7KR3BvEvESweN5Kgn3sDwd8U5ersBWZ2PfC78PkngYXhDKooF/bVAvPN7Fn2XF6jw0KafiBgwcJzKTXAPyO025W+Fc5eymnWWQ7+EvbT3xk+/ySQ1XTebOX4/s3VrwkugvtZ+PxMgunjeVueW2cEsociOSIdBrxF0D99CcGR/XXu/krE+L+y+4KijFa/bHM0ZgTTN68DthFcbLa5k/hn2XP1UgN+5J1caR7OdBpAFxwI5MrM7iCYdTaftFlnnscrW9u0/2XgdYKxktRg8x87jiodZvaCu0/o7LUubVOFQKBru6biZhGWHM5j27EX0lzlMuusi9r/HsEY0SzgV8CDOU5e6FbM7Fbgl+7+dPj8XQRdm1/MW5sJ+veVDhTZEWlqzaY9eMQlDrK5oMjM7nL309Pmordtu8MdeYkV0huBa7KcddZVORjwAYKpuPUEq7ne7O6vdhhYAiy4qPQQIDXOdBBBd2crmV0PEb1NFQIpNhYso53SE/gEMNDd/6eTuPQLijJa/dLMat19hZndxdu7dq509w77Z4upkOYq3BGNJFi1M5sLsroqjwkEheBE4BGC5Tr+5u5fLWQehdZV10Nk1KYKgXQHUWaydMUHqBS6dnKVzayzLm7/ywRr+6wmWOfqT+7enLra2d1HFiKPJNGsISk6bS6sKSPoGuh0CYhcdlRFNmsnVoXa4XdgH+DUtnl4cD/jk2PKqaTpjECKTnhFbuqNuZPgIq+rvIN7LndBmyXTtSOSKRUCKTrW/uqlu7j71QVPSqSEqWtIilEdwYJj9xEUgw8T3OXq9TiTEilVOiOQomNmDxGs/rgpfF4D3O253WxERPaiLO4ERNrRdvXHHQT3RxCRPFDXkBSj24Fnw7X9nWAd+F93HCIi2VLXkBSlcApp6taIGd/YRESiUyEQEUk4jRGIiCScCoGISMKpEEjimNk3zGy+mc0xs9nhMr/5autRM6vP1/ZFuoJmDUmimNlRwMnAJHffbmb7AJUxpyUSK50RSNLUAqvdfTuAu6929+Vm9j9m9pyZzTOzaeF6+Kkj+mvM7HEze8nMJpvZvWa2KLyBCmY23MxeNrNfh2cZ95hZdduGzewDZvaUmc0ys7vNrE/4+g/N7MUw9qoC/luIACoEkjwPAQea2UIzu87M3hO+/nN3nxze2awXwVlDyg53fzfwS4JlL/4DGAucn3bvhEOAaeFy1RsJVjLdJTzz+CZwQrjMdQNwqZkNJLhO4vAw9nt5+JtFOqRCIIkS3nO4DpgKrAJ+b2bnA8eZ2TPhzW3eBxyeFnZ/+HUuMN/dV4RnFIuBA8Ofve7uqeWq7yC453G6I4ExwD/NbDbBevvDCIrGNuAmMzsVaOqyP1YkIo0RSOK4ewvwKPBouOP/HDAeqHf3183sCoI7o6VsD7+2pn2fep76DLW9IKftcyO4u9aZbfMxs3cCxxPcp/dCgkIkUjA6I5BEMbNDzGx02ktHAAvC71eH/fanZbHpg8KBaIAzgZltfv40cLSZjQrzqDazg8P2+rn7n4GLw3xECkpnBJI0fYBrzaw/wU1vXiHoJlpP0PXzGvBcFtt9CTjPzG4AFgHXp//Q3VeFXVB3mllV+PI3gU3AfWbWk+Cs4ZIs2hbJiZaYEMmRmQ0HZoQDzSLdjrqGREQSTmcEIiIJpzMCEZGEUyEQEUk4FQIRkYRTIRARSTgVAhGRhFMhEBFJuP8HeQ05OTj4tO4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t.plot(20) # plots 20 most common tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FreqDist({'!': 1,\n",
       "          \"'s\": 4,\n",
       "          ',': 8,\n",
       "          '.': 11,\n",
       "          'a': 2,\n",
       "          'an': 1,\n",
       "          'asznee': 1,\n",
       "          'block': 1,\n",
       "          'brother': 1,\n",
       "          'cornflakes': 1,\n",
       "          'crispies': 1,\n",
       "          'day': 1,\n",
       "          'ducking': 1,\n",
       "          'every': 1,\n",
       "          'four': 1,\n",
       "          'frisbee': 1,\n",
       "          'girl': 3,\n",
       "          'got': 2,\n",
       "          'guy': 1,\n",
       "          'he': 2,\n",
       "          'hold': 3,\n",
       "          'i': 1,\n",
       "          'in': 1,\n",
       "          'is': 2,\n",
       "          'man': 2,\n",
       "          'maths': 1,\n",
       "          'men': 1,\n",
       "          'minus': 1,\n",
       "          'moving': 1,\n",
       "          'my': 4,\n",
       "          'on': 2,\n",
       "          'one': 1,\n",
       "          'park': 1,\n",
       "          'phone': 1,\n",
       "          'plus': 1,\n",
       "          'pumpy': 1,\n",
       "          'quack': 3,\n",
       "          'quick': 1,\n",
       "          'rice': 1,\n",
       "          'see': 1,\n",
       "          'smoke': 1,\n",
       "          'that': 3,\n",
       "          'the': 4,\n",
       "          'thing': 1,\n",
       "          'three': 1,\n",
       "          'tight': 3,\n",
       "          'trap': 3,\n",
       "          'trees': 1,\n",
       "          'two': 2,\n",
       "          'uckers': 1,\n",
       "          'went': 1,\n",
       "          'were': 1,\n",
       "          'when': 1,\n",
       "          'whitney': 1,\n",
       "          'your': 2,\n",
       "          '—': 1})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.vocab()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another thing that might be useful in analysis is finding common contexts. Our text is too small so we will use a bigger one.\n",
    "\n",
    "NLTK comes with several interesting **corpora**, which are large collections of text. You can check out what kinds of corpora are found in `nltk.corpus` in Section 1 [here](http://www.nltk.org/book/ch02.html).\n",
    "\n",
    "`reuters` is a corpus of news documents. More specifically, `reuters` is a *corpus reader* for the Reuters corpus which provides us with methods to access the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "in_1986 and_and by_. begins_1 paid_1986 in_, early_. or_, /_shipment\n",
      "of_. last_. for_shipment for_to on_3 on_12 on_19 last_when in_.\n",
      "in_1987 -_.\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import reuters\n",
    "text = Text(reuters.words()) # .words() is one method corpus readers provide for reading data from a corpus. We will learn more about these methods in Chapter 2.\n",
    "text.common_contexts(['August', 'June']) # It seems that .common_contexts() takes 2 words which are used similarly and displays where they are used similarly. It also seems that '_' indicates where the words would be in the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will further explore the Reuters corpus as well as several others in later chapters."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}