5_Variables_Gradients_Placeholders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Variables, Gradients, Placeholders\n",
    "\n",
    "So far, all we have done could've been implemented in Numpy. The real power of TensorFlow is in being able to take a gradient with respect to (almost) any variables you create. The function can be a linear regression, a neural network, or an ensemble of neural networks, as long as the functions in the computation graph are differentiable. You can for example solve [partial differential equations](https://www.tensorflow.org/tutorials/pdes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variables vs Constants\n",
    "What's the difference between variables and constants? As the name suggests, variables are mutable, whereas constants are not. To see this, let's look at a simple exmaple.\n",
    "\n",
    "Consider a situation where you would like to update the values of a matrix. Using [`tf.scatter_update`](https://www.tensorflow.org/api_docs/python/tf/scatter_update), this can be written as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'ScatterUpdate' Op requires that input 'ref' be a mutable tensor (e.g.: a tf.Variable)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-1b5c435ffdcb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m## This throws an error ##\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0msparse_update\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscatter_update\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconst\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindices\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py\u001b[0m in \u001b[0;36mscatter_update\u001b[0;34m(ref, indices, updates, use_locking, name)\u001b[0m\n\u001b[1;32m    595\u001b[0m   result = _op_def_lib.apply_op(\"ScatterUpdate\", ref=ref, indices=indices,\n\u001b[1;32m    596\u001b[0m                                 \u001b[0mupdates\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mupdates\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muse_locking\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_locking\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 597\u001b[0;31m                                 name=name)\n\u001b[0m\u001b[1;32m    598\u001b[0m   \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    599\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\u001b[0m in \u001b[0;36mapply_op\u001b[0;34m(self, op_type_name, name, **keywords)\u001b[0m\n\u001b[1;32m    619\u001b[0m             raise TypeError(\n\u001b[1;32m    620\u001b[0m                 (\"'%s' Op requires that input '%s' be a mutable tensor \" +\n\u001b[0;32m--> 621\u001b[0;31m                 \"(e.g.: a tf.Variable)\") % (op_type_name, input_name))\n\u001b[0m\u001b[1;32m    622\u001b[0m           \u001b[0minput_types\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtypes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    623\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'ScatterUpdate' Op requires that input 'ref' be a mutable tensor (e.g.: a tf.Variable)"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "const   = tf.constant([1,2,13,97,5,6])\n",
    "indices = tf.constant([2,3])\n",
    "values  = tf.constant([3,4])\n",
    "\n",
    "## This throws an error ## \n",
    "sparse_update = tf.scatter_update(const, indices, values) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! What's wrong here? As the error suggets, `const_mat` is not a mutable tensor. Kindly, the error suggests using a `tf.Variable`. Let's use that instead: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "var     = tf.Variable([1,2,13,97,5,6])\n",
    "indices = tf.constant([2,3])\n",
    "values  = tf.constant([3,4])\n",
    "\n",
    "updated_var = tf.scatter_update(var, indices, values) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! It works now. As you have just seen, you need to use `tf.Variables` whenever you have to mutate the tensors. This includes using gradients to update the values e.g. in neural networks. We will see `tf.Variables` quite a bit in the coming sections. Now, let's evaluate the updated values and see if it's what we wanted. Let's initialize a session and run the tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "FailedPreconditionError",
     "evalue": "Attempting to use uninitialized value Variable\n\t [[Node: ScatterUpdate_1 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@Variable\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/cpu:0\"](Variable, Const_3, Const_4)]]\n\nCaused by op 'ScatterUpdate_1', defined at:\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/__main__.py\", line 3, in <module>\n    app.launch_new_instance()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n    app.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelapp.py\", line 478, in start\n    self.io_loop.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/ioloop.py\", line 888, in start\n    handler_func(fd_obj, events)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 233, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/ipkernel.py\", line 208, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/zmqshell.py\", line 537, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2728, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2850, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2910, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-2-6985616cf3c8>\", line 5, in <module>\n    updated_var = tf.scatter_update(var, indices, values)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py\", line 597, in scatter_update\n    name=name)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 767, in apply_op\n    op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 2630, in create_op\n    original_op=self._default_original_op, op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1204, in __init__\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\n\nFailedPreconditionError (see above for traceback): Attempting to use uninitialized value Variable\n\t [[Node: ScatterUpdate_1 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@Variable\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/cpu:0\"](Variable, Const_3, Const_4)]]\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFailedPreconditionError\u001b[0m                   Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1326\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1327\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1328\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1305\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1306\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1307\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/contextlib.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, type, value, traceback)\u001b[0m\n\u001b[1;32m     87\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m                 \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py\u001b[0m in \u001b[0;36mraise_exception_on_not_ok_status\u001b[0;34m()\u001b[0m\n\u001b[1;32m    465\u001b[0m           \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpywrap_tensorflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_Message\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstatus\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 466\u001b[0;31m           pywrap_tensorflow.TF_GetCode(status))\n\u001b[0m\u001b[1;32m    467\u001b[0m   \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFailedPreconditionError\u001b[0m: Attempting to use uninitialized value Variable\n\t [[Node: ScatterUpdate_1 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@Variable\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/cpu:0\"](Variable, Const_3, Const_4)]]",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mFailedPreconditionError\u001b[0m                   Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-808564385b21>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0;31m## This throws an error ##\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mupdated_var\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1122\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1123\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1124\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1125\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1126\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1319\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1320\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1321\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1322\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1323\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1338\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1339\u001b[0m           \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1340\u001b[0;31m       \u001b[0;32mraise\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnode_def\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1341\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1342\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFailedPreconditionError\u001b[0m: Attempting to use uninitialized value Variable\n\t [[Node: ScatterUpdate_1 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@Variable\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/cpu:0\"](Variable, Const_3, Const_4)]]\n\nCaused by op 'ScatterUpdate_1', defined at:\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/__main__.py\", line 3, in <module>\n    app.launch_new_instance()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n    app.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelapp.py\", line 478, in start\n    self.io_loop.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/ioloop.py\", line 888, in start\n    handler_func(fd_obj, events)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 233, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/ipkernel.py\", line 208, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/zmqshell.py\", line 537, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2728, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2850, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2910, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-2-6985616cf3c8>\", line 5, in <module>\n    updated_var = tf.scatter_update(var, indices, values)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py\", line 597, in scatter_update\n    name=name)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 767, in apply_op\n    op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 2630, in create_op\n    original_op=self._default_original_op, op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1204, in __init__\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\n\nFailedPreconditionError (see above for traceback): Attempting to use uninitialized value Variable\n\t [[Node: ScatterUpdate_1 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@Variable\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/cpu:0\"](Variable, Const_3, Const_4)]]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    ## This throws an error ## \n",
    "    print(sess.run(updated_var))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another error! This is because, whenever you have variables, you have to `initialize` them. To initialize all variables, you can run `tf.global_variables_initializer()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3 4 5 6]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(updated_var))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! We have an updated vector that we wanted :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Gradients\n",
    "\n",
    "Once you build a computation graph, you ca use `tf.gradients` to calculate gradients of most values. Let's start with a simple example.\n",
    "\n",
    "Let  $f(a) = 2a^2+a$. What is $f'(8)$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.constant(8.0)\n",
    "fa = 2 * (a ** 2) + a\n",
    "gfa = tf.gradients(fa, a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33.0]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(gfa))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function can be very complicated. TensorFlow can easily do derivatives we can't do by hand.\n",
    "\n",
    "Let $g(a) = sin(a)^{42} + tanh(a)-e^{a^{-\\pi}}$. What is $g'(8)$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "a = tf.constant(8.0)\n",
    "ga = tf.sin(a) ** 42 + tf.tanh(a) - tf.exp(a ** (-math.pi))\n",
    "gga = tf.gradients(ga, a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-3.9404514]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(gga))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also take derivatives with respect to specific variables. This means that TensorFlow can operate chain rules. This is crucial for e.g. back propagation in training neural networks.\n",
    "\n",
    "Let $b = 2a$. What is the derivative of $a+b$ w.r.t $a,b$, evaluated at $a=1$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.constant(1.)\n",
    "b = 2 * a\n",
    "gc = tf.gradients(a+b, [a, b])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(gc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $b = 2a + 3$, $c = sigmoid(b)$. What is the derivative of $c$ w.r.t $a,b$, evaluated at $a=1$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.constant(1.0)\n",
    "b = 2 * a + 3\n",
    "c = tf.sigmoid(b)\n",
    "gc = tf.gradients(c, [a, b])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.013296066, 0.0066480329]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(gc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Placeholders\n",
    "`tf.placeholder` creates a \"placeholder\" that will be fed with some data during the execution, namely, inside the session. Let's look at a simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "InvalidArgumentError",
     "evalue": "You must feed a value for placeholder tensor 'Placeholder' with dtype float and shape [3]\n\t [[Node: Placeholder = Placeholder[dtype=DT_FLOAT, shape=[3], _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\n\nCaused by op 'Placeholder', defined at:\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/__main__.py\", line 3, in <module>\n    app.launch_new_instance()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n    app.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelapp.py\", line 478, in start\n    self.io_loop.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/ioloop.py\", line 888, in start\n    handler_func(fd_obj, events)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 233, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/ipkernel.py\", line 208, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/zmqshell.py\", line 537, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2728, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2850, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2910, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-13-c7fd00d89770>\", line 2, in <module>\n    a = tf.placeholder(tf.float32, shape=[3])\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py\", line 1548, in placeholder\n    return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py\", line 2094, in _placeholder\n    name=name)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 767, in apply_op\n    op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 2630, in create_op\n    original_op=self._default_original_op, op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1204, in __init__\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\n\nInvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder' with dtype float and shape [3]\n\t [[Node: Placeholder = Placeholder[dtype=DT_FLOAT, shape=[3], _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1326\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1327\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1328\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1305\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1306\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1307\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/contextlib.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, type, value, traceback)\u001b[0m\n\u001b[1;32m     87\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m                 \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py\u001b[0m in \u001b[0;36mraise_exception_on_not_ok_status\u001b[0;34m()\u001b[0m\n\u001b[1;32m    465\u001b[0m           \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpywrap_tensorflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_Message\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstatus\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 466\u001b[0;31m           pywrap_tensorflow.TF_GetCode(status))\n\u001b[0m\u001b[1;32m    467\u001b[0m   \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mInvalidArgumentError\u001b[0m: You must feed a value for placeholder tensor 'Placeholder' with dtype float and shape [3]\n\t [[Node: Placeholder = Placeholder[dtype=DT_FLOAT, shape=[3], _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-c7fd00d89770>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m     \u001b[0;31m## This throws an error ##\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1122\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1123\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1124\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1125\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1126\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1319\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1320\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1321\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1322\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1323\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1338\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1339\u001b[0m           \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1340\u001b[0;31m       \u001b[0;32mraise\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnode_def\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1341\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1342\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mInvalidArgumentError\u001b[0m: You must feed a value for placeholder tensor 'Placeholder' with dtype float and shape [3]\n\t [[Node: Placeholder = Placeholder[dtype=DT_FLOAT, shape=[3], _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\n\nCaused by op 'Placeholder', defined at:\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/__main__.py\", line 3, in <module>\n    app.launch_new_instance()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n    app.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelapp.py\", line 478, in start\n    self.io_loop.start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/ioloop.py\", line 888, in start\n    handler_func(fd_obj, events)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 233, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/ipkernel.py\", line 208, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/ipykernel/zmqshell.py\", line 537, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2728, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2850, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2910, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-13-c7fd00d89770>\", line 2, in <module>\n    a = tf.placeholder(tf.float32, shape=[3])\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py\", line 1548, in placeholder\n    return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py\", line 2094, in _placeholder\n    name=name)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 767, in apply_op\n    op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 2630, in create_op\n    original_op=self._default_original_op, op_def=op_def)\n  File \"/Users/kojin/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1204, in __init__\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\n\nInvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder' with dtype float and shape [3]\n\t [[Node: Placeholder = Placeholder[dtype=DT_FLOAT, shape=[3], _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\n"
     ]
    }
   ],
   "source": [
    "# create a placeholder for a vector of 3 elements, type tf.float32\n",
    "a = tf.placeholder(tf.float32, shape=[3])\n",
    "b = tf.constant([5, 5, 5], tf.float32)\n",
    "# use the placeholder as you would a constant or a variable\n",
    "c = a + b\n",
    "with tf.Session() as sess:\n",
    "    ## This throws an error ## \n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! What's wrong with this code? Well, we defined a placeholder for something. So we need feed that something during the execution. We do using the `feed_dict` parameter which takes in a dictionary, where the keys are the names of all the placeholders defined in the graph and the values are the values we would like to feed in. In this case,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 6.  7.  8.]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(c, feed_dict={a: [1, 2, 3]})) # use feed_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass in a matrix in a similar way, by specifing the shape. Note that shapes can be `None`. This means that we don't specify the dimension of that axis. This is useful when the values we will feed in can change shapes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 6.  7.  8.]]\n",
      "[[  6.   7.   8.]\n",
      " [  9.  10.  11.]\n",
      " [ 12.  13.  14.]]\n"
     ]
    }
   ],
   "source": [
    "a = tf.placeholder(tf.float32, shape=[None,3]) # create a placeholder for a matrix\n",
    "b = tf.constant([5, 5, 5], tf.float32)\n",
    "c = a + b\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(c, feed_dict={a: [[1, 2, 3]]})) # shape=[1,3] (not [3,])\n",
    "    print(sess.run(c, feed_dict={a: [[1, 2, 3],[4, 5, 6],[7, 8, 9]]})) # shape=[3,3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Placeholders are useful because, for example when training neural networks, we are passing in different batches to a same network. In such a case, it is conveninent to define the data tensor as a `tf.placeholder`. As a final example, you can use a Python `for` to feed different values to placeholders. This will be used a lot in minibatch settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "2.0\n",
      "4.0\n",
      "8.0\n",
      "16.0\n",
      "32.0\n",
      "64.0\n",
      "128.0\n",
      "256.0\n",
      "512.0\n"
     ]
    }
   ],
   "source": [
    "a = tf.placeholder(tf.float32)\n",
    "b = 2 ** a\n",
    "with tf.Session() as sess:\n",
    "    for a_val in range(10):\n",
    "        print(sess.run(b, {a: a_val}))\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $g(a) = sin(a)^{282} + tanh(282+a)-e^{a^{-\\pi}}$. Plot $g'(a)$ using what we learned about `gradients`, `placeholders`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "gga_nums = []\n",
    "\n",
    "def gga(a):\n",
    "    ga = tf.sin(a) ** 282 + tf.tanh(282+a) - tf.exp(a ** (-math.pi))\n",
    "    return tf.gradients(ga, a)\n",
    "a = tf.placeholder(tf.float32)\n",
    "gga_a = gga(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "gga_nums = []\n",
    "with tf.Session() as sess:\n",
    "    for i in np.arange(1,15,0.1):\n",
    "        gga_nums.append(sess.run(gga_a,feed_dict={a: i}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXuQJVd937+/7r6Pee1Ls7ta7a60K2kRDwFC3giwHWwZsARFIXAlFSmuhGCInArExGVIUFFJhYqdchISjCuEIGwQtgmKIVZQZAMWhAJXwkMrkMRKQg9WQtqH9jHSzO7M3Effe0/+6D7dp++981jt9O3T93w/VVsz97F7z/ac85tvf3+/8zuilAIhhJDxxyt6AIQQQkYDAz4hhDgCAz4hhDgCAz4hhDgCAz4hhDgCAz4hhDgCAz4hhDgCAz4hhDgCAz4hhDhCUPQATGZnZ9W+ffuKHgYhhJSK+++//4xSavta77Mq4O/btw+HDh0qehiEEFIqRORn63kfLR1CCHEEBnxCCHEEBnxCCHEEBnxCCHEEBnxCCHEEBnxCCHEEBnxCCHEE5wL+uWaIrzxwrOhhEELIyHEu4H/18HP4wJ0P4LmFZtFDIYSQkeJcwG+GXQBAu9MreCSEEDJanAv4OtB3lSp4JIQQMlo2JOCLyGdF5JSIHDae2yYi94rIE/HXrRvxWRdKSwf8HhU+IcQtNkrh3wHgxr7nPgzgm0qpAwC+GT8unEThM96TVTjbDPGtn5wqehiEbCgbEvCVUt8B8Hzf0zcB+Hz8/ecBvGMjPutCCeNI36HCJ6tw1w+P4Tc+fx/ONcOih0LIhpGnh79TKXUCAOKvO4a9SURuFZFDInLo9OnTOQ4nQit8xnuyGkvtDpRicp+MF4UnbZVStyulDiqlDm7fvmb//gum3WXSlqwNk/tkHMkz4J8UkV0AEH+1whBtM2lL1gHvBMk4kmfAvxvAu+Lv3wXgKzl+1rpJFD4XMlkFKnwyjmxUWeYXAXwXwFUiclRE3gPg9wG8WUSeAPDm+HHh6IXMpC1ZDS0Mej0GfDI+bMiZtkqpW1Z46Y0b8e9vJLxVJ+shtf4Y8Mn4UHjSdtQwaUvWAy0dMo44F/DDLpO2ZG1o6ZBxxLmAz522ZD1Q4ZNxxOGAz4hPViat5mLAJ+ODcwG/RYVP1gGT+2QccS7gh0zaknVAS4eMI84F/DaTtmQd0NIh62FusYXvPJ5/D7CNYiwC/oPPzuNDX3oQp86tfWwhk7ZkPSSWDhU+WYU773sW777jPnRKElDGIuA/d7aJL91/FKfPtdZ8L5O2ZD2kHj4DPlmZs80Q3Z5CpyTzZCwC/kwt2jB8rtlZ871hN/rBlOQXMikIbtAj66HZjs7ILsud4FgE/Ol6FPAX1xHwmYwj64FVOmQ9NMIo4Jcl1zMWAX+mXgEALLZWD/hKqVS5UeKTVaDCJ+uhEZZLGIxFwJ9OLJ3Vj6NrG0G+y3VMVoEePlkPjdjSKYswGIuAPxNbOufWUPihEeWZtCWrwW6ZZD00aemMnlrgoeLLmklb83xSOjpkJXpG1UVZlBspBnr4BSAimK4FayZtzYBflqw6GT2m9UdLh6wGLZ2CmKlX1kzamgG/QxOfrEDLvBMsyUImxaAtnbIIg7EJ+NO14DyTtuX4AZHRE5rzpCQLOW/Cbm9NQeUitHQKYroenKeHTxOfDIfW3yCf+ZsjeNsf/k3Rw7COJOCXZJ6MTcDftJ6A32XSlqwNk/uDHJ9v4PjC2r2qXEN7+LR0Rsx0LTgvD5/KjawEk7aDNMMe2p0eFNdNQq+n0vM1SnJdxifg19cO+KY3y6QtWYk2k7YD6MAWct0kNDvd5PuyxJOxCfgz9QrONcNVFQgVPlkPmSodKnwAQCv2qtv0uBK0nQOUJ56MTcCfrgUIuyqzWPsxX+swaUtWgMJgkEThr7K+XEMnbIHyCIOxCfib6mu3SGbSlqyHNssyB2h1qPD7aYZU+IWRtEhexcc31QmTcWQlQlo6AzTjrpBtKvyERrt8AnJ8An4tbpG8DoXvCUpzQg0ZPZkqnZIot7zRls5qlqlr0NIpkKRj5iq7bbU6maj4XMhkRViHP0hi6TDgJzRo6RRH0hN/FUsnCfjVoDS/kcnoYdJ2kFaoyzIZ8DVmlU5Z4snYBPxN8alX60naTlS90vyAyOhpMWk7gLZymLRNadLSKY70XNu1LZ164JfmB0RGT5tJ2wFo6QxCD79AtKWzWpVOu9tDxRf4njBpS1aEls4gLVbpDJCxdEoyT8Ym4FcDD7XAW93S6fRQ9T0EvnAhkxXRQc2T8ii3POn1VGLl0NJJySRtSzJPgqIHsJHM1INVk7Zht4dq4MEX4UImKxJ2e/C96E6wLMotT8wgT4WfkvHwSzJPxirgR4egrK7wK74XLWQGfLIC7W50JwiUR7nlibZzAAZ8E1bpFMxMvbJm0rYaMOCT1cnOk6JHUzwtoyskLZ0U1uEXzFo98VuxpePR0iGr0IrvBD0pz0LOkyYV/lDMgM/2yAUws8apV6GRtC2L55Y3zbCLY/ONoodhFe1ODzXeCSaYCp8br1KaYReTVR9AeYRB7gFfRJ4WkR+LyAMicijPz1rrXNs2Ff4Af/Ldp3Hjx7/DhWyg5wmTthFm/xz20klptLuYisvBy7J8RqXwr1dKXaOUOpjnh8zUgjV76VR9DwGVW8LJsy2ca3XwHM8rTWh3uqj6kTBg0rbPw2fAT2iE3WT/T1mEwZhZOhUstjornnoVdpm07Wc5rjQ4Tlsngcn9LJkqnbJI2RHQCHuYqsWWTknmySgCvgLw1yJyv4jcmucHTdcD9FQ2mWKiFzItnZRGO7LA6OOnhF2VzpOSKLc8aVLhD6XZNhR+SeLJKOrwf0EpdVxEdgC4V0R+opT6jn4x/iVwKwBceumlF/RBScfMZgeT1cH/mq6+YNI2RSv8Yy8w4Gu09ed7tHSArMJnrielEXaxtzYBgEnbBKXU8fjrKQB3Abiu7/XblVIHlVIHt2/ffkGfNbPGMYdm0pYLOSKxdBYY8DWtTNK26NEUj5mopcJPaYRm0rYcEyXXgC8iUyIyo78H8KsADuf1eTNrHHMYdnuoxUlbNk+LWI4tnaNU+Amp9VcebzZPdNLW94QB36BpVOmUJZ7kbensBHCXiOjP+u9Kqa/l9WEzSU/84ZU6eiG3u73S/EbOm8TSoYefoKt0mLSN0Ap/ph5kzgpwHbNKpyzCINeAr5Q6AuDVeX6GSdIieSVLJ/bwuz3FhRyjE9zH5xtQSiH+5ew0mf0aJfFm80Q3CZupB5kD3l0m7PbQ6SlMVVmWWRgXTVUBAKcXW0Nf1wqfSduUpVYXItH2+eeX2kUPxwqYtM2ik7YztQrLMmO0UGJZZoHMTtdQDbwVK07Mcruy/IDyptHuYM/WqNLg+Dw3XwF9dfgUBmh1onbRk1WfHn5MM7ZC6xW/VPNkrAK+5wn2bJkYmoBUSsUnXjFpq1FKYTns4sCOGQDAsfnlgkdkB5k6fM4TtDpd1AIvyn8x4ANIFf6EDvgluSxjFfABYPfWCRx9YTBw6VvRWuDB4606gEi5KQUc2DENgJU6GlPhl6W+Ok9acTM5XfBA0g6iE1UfvpRnnoxdwN+zdbjC18qk6kcnXlHhA0tx+equzXVMVn1aOkjvBPU8ocKPkra1wEfVp8LX9Ct8tkcuiD1bJzG31E7qyzVh/AOpBh58Jm0BpCWZk9UAu7dM0NJBeidYDTx4HtBjfIsUfsVDhQo/oWF4+GU6N2EMA75OQGZVvlYmlVi50dJJVcpkzcclWyZYi4++O8ESJePypBX2UA981KjwE3Sp6kTVL9V+jbEN+M++MDzga2+Wlo6p8H3s3jpBSwfZecKkbUSr00WtwqStyUDStiTCYAwD/iSAwQRkuxv9gHTAB8pTO5sXy7GHP1GJLJ3nh1hhrmFaOkzaRjBpO4i2dCYqfqnKvMcu4G+frqHqewOVOu1O7OHHlg5Qnv4XeZFR+FtYiw8MJvep8HXAj5K23GkboRV+vVquA5XGLuB7nsSlmf0KXys3ge/HCt9x9bZs7BbUTaBcV/hhJmlbnoWcJ1GVDpO2JtrDr1f8aJ6UJJaMXcAHhpdmpsrNTxS+64tZH34yUQ0QxL8Ew5KUl+VFK+PhUxQA0TWpV2KF31WlsS/yxLR0ytSCY2wD/rEBSyfrzQK0dJZasaVT8VHxoqnQcVzB9c8T10UBkN1pC/CYQyCydAJPkqq/ssSSMQ34kziz2E5+CwPZW3UmbSPMskyt8MsycfNCB/yaPsTc7csBICrLrFU81OKAz1OvorUzUYkap3klSu6PacCPEpDmRqJWUocvScAvi++WF8vtDnxPUPU9VBJLx+3FrNVrhQo/IUnaaoXPxC2aYRf1ahTwy5TcH+uAb9bim710koBfkh9SXiy3u5is+BARBIml4/Y1YZXOIEnS1qelo2m0swq/LJdkTAN+VItvtklm0naQ5VYXE7FKSS2dkszcnMhsvCrRrXpeKKXi1gpR0hagwgeApXYXk1rhe+VJ7o9lwN8e98V/+sxS8lyY3KoLPCp8AFFZpi7H1OrN9SqdzMYrKvzMnTEtnZTnl9rYFh+45HteaebJWAZ8zxO8avdm/PCZF5LnzFv1gAEfQFSWqW9LA48KH8jOEyr8NPfFKp0sc4stXDRdAwD4JSrfHcuADwDX7d+Gh44uJJU6w8oymbRNb0up8COyuR6KAn28Ya3CpK3J3GI7OVKV7ZEt4Lr929DpKfwoVvn9PVIALual9hAPvyQTNy8ywoCWTrKjtBZ49PBjmmEX51odzE5HAb9Mh92PbcD/ucu2whPge089D8Boj+x5TNrGNNodTFUjDz+p0qGlA8BM2hY8oILRlk7dVPiOWzrPL7UBILV0uNO2eGbqFbziks34wVNzABCfZxslbJm0jchaOmytAAyem+D6HGl1BhW+63s15hbjgG9YOlT4FnDd/m340TPzaHW60Tml8YRl0jaikbF02FoBiISBSDRHyrSQ82Jo0tZxS2duqQUAuMiwdKjwLeC6/dvQ6vTwjUdO4d5HTmJ2JroF85i0BQAstTuJwg/YXwgAEmEgIjzsHkbS1thp23I94CcKP4onQYmEQVD0APLkb+3bBgD4rTt/hHrg4c/e+1oAVPhA1EeoGfYwWe2vw3d7Mbc6vSSw+SVKxuVFU1s6FSZtNQMKnztt7WDbVBUvvXgGVd/D5959HV5z6VYAYNIWRuO0ZLegQIRVOmG3lzQJ8zyBUtFuU1fRCr8eMGmrmVtsoxp4mI43LUa5nnJck7FW+ADwX/7+tQAUrtwxkzzHpG32tCtNxfMQlmTi5kXLyPWYwkCXrbpGa4jCd/3UqzOLbcxOVSHx/ChTk72xD/hX7pgeeI6WTnqylbZ0gKgW33WFf+yFBnZurgOIeqQAUa5n7BfKCnCn7SBzS+kuWwClKt8da0tnJZi0Ha7wA0+cr9I5cmYRl89GIkHPE5dvetKAz522mueX2ol/D0StFcoiHp0M+InCd1jN6oA/YVo6voewJBM3D5ZaHZw828Ll26cAGJaOw8KgFaaWjl43rgf8ucW0cRqAUp197GTA97iQkx5Dg5aOu4v5qbi76uWzccCn9ZfutA2icxOqgYeWw3NEKYUziy3MGpaOL+VpsudkwOdCjmrwgX5Lx3Paw//p6UUAwOXbY0tHeBRmK+xCJN2JXfM9hB13r8dSu4tWp5fssgUioVSWWOJkwGfS1lT4pqUjTls6R04vQQS47KLoAB12VdXHG3pJRUo18NDudtf4W+PL3KKuwTeStlT4dpMk40ryQ8qD5aGWjue0pXPkzBL2bJ1A3Ti6DnBc4cfn2Woqvue0h39G77I1k7aelGaHupMBP2kj4LB9ocsyJ/qqdFxunnbkdFqhAzBpC0R1+HojGhArfIcDvlb4s1NZhV8Wt8DJgM+k7Qobr3zP2fbISik8dWYpqdABjDr8kizmPGiGveSOB4gCvsuiIG2NnFX4ZbkLdDLgM2kbBfyq7yU9dAC3N16dPNvCcrubJGwBM2lb1KiKZ0Dh+57TzdPm4oBvlmWWqauqkwF/paRtM+xibrHlRO+URruTsXOAuLWCox7+kbhC54pZU+HzTrAV9lCr9Fk6js4RADiz2MJ0Lcjc9UTtkQsc1HmQ+45xEbkRwCcA+AD+SCn1+3l/5lqslLR99+fuw3ePzGFTPcBLds7g2su24pq9W3DF9mlcdtFk5odcdszDTzSBL86qt5/qGnxD4bt+J6iUwtxSO5O0rfoe2h2Xq3Syu2wBtkdOEBEfwCcBvBnAUQD3icjdSqlH8vzctVgpafuzuSVcvXsTrtm7BY+eOIc7/u/TiZoRAXZvmcD+2SlcvKmO2ZkaZqdrmJ2uYutkFVO1ADP1AFO1ANPVAFM1PzlUxDZ6vWghDwZ8D0utTkGjKpYjpxcxWfWxc1M2GQe4W831sb9+DA88O48P3XBV8lw18JKEv4vMLbUyNfhAutNWKZWUr9pK3gr/OgBPKqWOAICI3AngJgCFBvyVFP5CI8RbXrkL/+ptLwcQ+ZePP7eII2cW8dSZpeTP4yfPYW6xvWYpVr3ioRb4qPgeqr6gEkSeefLYjxpSRc8JPIlOWfIkalXsicDTX730ezGe122N9XtFBAIAAgii1wSIv0ZK5OsPP4cjp5fwxpfuyIy3skKVzuFjC/j246czzw2zvYbFxf6nhr9n7X9r6JUeNoYX+Xnffvw09s9OZRbsSgr/iZPn8ONjC1hsdca2YuXpuSX82feewS3XXYp/+stXJM9XAw8LjeHz/v/99AweOX52VEMcKUoBzy+38eiJc/i5y7ZmXvMTYRD11dH84Knn8djJc4BS6MVttnsqmqPD1s+r9mzBdfu35fnfyD3g7wbwrPH4KIDXmm8QkVsB3AoAl156ac7DidA/IDNgtzs9LLW72DJRSZ6rBT5euWczXrln88C/0espLDRCnFlsYb4RYrHVwVL8Z7HVTb5vdXoIuz20469hV6HdzT631O4i7PTQUyr+E/0yUvHXSD0g87qKn+9/r/5eAYCKgpsyJpkC8Oo9W/CJm6/BW67elfk/Bb4MVOmcOtvELZ/5Hs417VV1/aJqmMYaprz6n/nHb7g88zip5uoL+O/9k0P42dzy+Q6zdPzatbvxu++4OnPtqkPq8BcaIf7tPY/gy/cfHfUQR0rgCbbP1PCml2WFklnN5SfuQQ/v/twPsNRev/31T37pitIH/GFrL7N6lFK3A7gdAA4ePDiSe+dhym2hEQIANk9Whv6dfjxPsHWqiq19t3dlJtp4lf0RfPR/P4JWp4d7f/sNuOyiqYG/s1HBdtidcNG3x/6QO8FuT+HoCw38o5/fh/f/ypWoBt7Q/3PZEZHkgA+TYUnbv/fp7+KJU4t4//VX4r1/e39yBz1uTFeDof+3YY7BE6cWsdTu4nffcTVuvPpiCJDcuUv8tZ/qCCzgvAP+UQB7jcd7ABzP+TPXxB+yg3KhEZVbbZ5YX8AfRyqeZA5A+cYjJ/GXPz6BD/7qS3Bg58wqf3M8GVaHf+pcE92ewoGd05kGWq7Qv9O20+3hJ8+dw/uvvxIfNLx+lxh2gt4Dz84DAH7xylmr5knev1LuA3BARPaLSBXAzQDuzvkz12SYpaMV/pbJ8VHs50u/wv93X30UV+2cwa1vuGKVvzW+DEvaHp9vAgAu2TxRyJiKpl/hN+Pg77JQGla++8Az89gyWUn6MtlCrgFfKdUB8H4AXwfwKIA/V0o9nOdnrgcvTnSaC3l+OQ74Dk/cip9N2h6fb+CXrtqeHHzhGqn1lz53fL4BANi1pV7EkAqn1tdaQTfhq1fcnCPA8K6qDx6dx6v3bCncluwn9zp8pdRfAfirvD/nfPEl2/BIB3yXlUrgZVsrtDu9zC5L1xh2q35iIQr4l2xxWOEbAb8Z6oA/PntUzhd93rGeJ0utDh4/eQ43vOLiIoc1FGdXc3//i/nE0nE44ButFTrdHnpqNIkkWxmWjDs+38R0LcCmupvzpOr3WToM+AO9uR46uoCeAq65dEuRwxqKs6u5/6T5hUYIEWDG0YUMxEccxotZ77h11c4BhldznVhoYNdmN+0cIJoj3Z5KrkkjDvgTDgf8/nny4NEoYfvqPQz41tBv6Swst7GpXkl+eC4SGH2928nh1c5OkaFdVY/PN7HLUTsHSAWAFgbNMPra35fJJfqtvweemcdlF01mGqzZgrOr2fezp9TMN0Kn/XsgqtLRW8T1bXs1cHghDynfPbHQwG5HE7ZAGvD1HWAjZNI2PSgnevzAs/O4Zq996h5wOeD3HVowvxw67d8DUR0+AIRdhVZIS6dfubU6XZxZbGOXoyWZQDof9B0gPXxjv4ZSmFts4bmzTbxy9+DufBtwdjX3e/hU+EiavXV6veTcUqctnfi/ru8En1uIavBd9vBr8RxpdxnwNWYLjqVWdD1s3c/j7GruD/hnG6G1P6RRUfFThd+kwh+owz8273ZJJoCkN74O9E0mbRHEyqCXsULtXDe51+HbSv85lPPLbWyecPZyADDbRvesn7ijoP9M2xPzVPj60Hu94SrdeOVuwDdbcHTiuVL17Sz+cDbCBX56aIHufLllwm2Fn1o6ilU6MJNxccB3fNMVkJ6BrM9E1q0VXFb4pqWjRaStQsnOUY0AM2l7rtVBT7m96QowLZ1eUoXhcsDvT9oem29i21TVaTWrA/5SfAiKVvhOzxOjDl/fGVcs3bBo56hGgOnhn22wrQKQepGdrqnw3Q1u/U2xXN90BaSWznJLK/zokPNxbYm8HjxjnoR6wyIDvl2YAZ99dCJ0T5BOr5cEfFtvTUfBgKUz33S6JBMwLZ1I4TfbXafveADjxKueQksrfEvXjZ2jGgFm0nY+7oXPKh29i1KhFR9UbatSGQX9Sdvjjm+6AoCp+FAU7eE3wq7T/j2QtXSo8C3FTNomrZEd9/AD04vUlo7TOyijr72ewnK7g3PNDnY6b+lkPfxm2HO6rQKQbcFhe3WbnaMaAabCTw4/cdzSSRW+UZZpqVIZBWbSVivaYcf+uUQt8OCJUZYZdp1O2AKpFdrrpT2GbF03do5qBATeYMDf5HjATz18tlYAzDNtwaqlGBHBZDVIdpQ2wy4VvgzmvujhW4aXSdq2Ua94ziefdJWOqfBdrtIx++G3Ql1+6O710ExWfTRCbel0UXf8mpiH3be7euOVnaHVzlGNALMOf36Zm66AtA6/01WJoq1YumNwFJiWju3e7CiZqpkKnx5+Ok/SpnIM+JaRSdo22CkTyDZPa3W6qAaedWdyjhKzDl9bXK5bOkC0q1aXZTbCrtOtkYE0ud/tqdTDt3Se2DmqEeBJesThQiN03r8H0iqdMN545XpwMw+nbnEjWsJUzU9bK4Ssw89YOpbfGTu7os3TnRaWQ+crdIC0SkdbOq4HfLNbJstUUyaqAZYY8BNM6y/s9uBJerdsG3aOagRkkraNNi0dDO60tdWHHBW6W0BXcSOayVTVx3LLqMN3PeD3KXxb++gADgf8gaSt47tsAaCSVOnElo7jC1lE4EmfpUOFj4mqn9lp67qH3988zVb/HnA54MdJ22bYRavTw6a62xtqAEPhd+OkrcVKZVT4nmQUPj18YKoaYLndQdjtodtTziv8tA5fWX9nbO/IcsaPk7bnmtGtqeuN04A04Id64lqsVEaFTu7zfICUyThp2+DxhgCyh92HVPh2opO2Z5vcZaupJO2Ro41XDG5pV9UWu4cmTFYCtDo9LMU+PgO+0UuHHr6deF6k3HQv/E11BvzA3HgV2q1URoUvwjr8PqZqUYCfW4y6zNLSMRW+snrd2DuynPFFK/xIpWxy/DxbwGie1qPC12hhQA8/Re+sfX4pCvhU+GnStkWFbye+L+gpKnyT9BBzevganbRtd3oQsXdDzSiZik+90gF/our2PEnPTQA9fFvRZZn08FN8z6zS6aFKNRu30Y66ZVZ9t1tNaLTCn9MK3/F54vuppRNV6dg7R9wN+Dpp24gtHSp8iAgqviRVOrR0AN9L6/B5PSJShd8CANTZPA1AVJZJhW8pvk7aNkMEnji/eUQTeJ6h8HlNkqRtp+v8RjTNZK3Pw3dc4Scno8UnXtns4TubqUwVftQ4jbfqEYEvyZm2Nm8gGRVJ0lYpKvwYfcxhUqVDhQ8gPRrU5nXjdMDvqWjjFXfZplR8L+mlwzYCadK202XA12hLJ/HwHZ8n/a0VbD3tCnDZ0jGStkzYpgSepN0yLVYqo0LPk2hfgttKVtNflul6Hb6IQCSydMKu3evG3pHlTKTw4174TNgmVHwv2TJPDz+2dLSHz+sBwFD4i3HS1vGAD6TCgDttLUXfhs0vh9x0ZRD4kmyZ5yaj7EJmwI+oVzyIAGebHYhw9zGQWn/caWspOuC/sNymwjcIPEnOK7V54o6K6NyEqA6fVToRIoLJ+FrUA58FD0ir/pxV+CLyb0TkmIg8EP95a16f9WLQAZ/HG2ap+F5yXikDflyHr1Sy8YpETNaiu2LXE7Ya3arF9n74eXsZH1dKfSznz3hR6FIqpcAqHYPAFywmlo69E3dUpJZOl1VLBro00/WErcYzFD532lqI56U/FCr8lMDzktOMbFYqo0JEEoXPX4Apk1Wt8BnwgcgxKEML7bxH9n4ReUhEPisiW3P+rPMiMAM+PfyECpO2Gcx++LweKVrhM+BHeCJoxtVtY+vhi8g3ROTwkD83AfgUgCsAXAPgBID/tMK/cauIHBKRQ6dPn76Q4ZwXWYVPS0cTeB6WqPAT0jp8lmWapAGf1wSIcj1lKGe+oEinlHrTet4nIp8BcM8K/8btAG4HgIMHD6oLGc/5QIU/nMBPD3dnkjLqk6J3UDLgpyQevuNtFTS+CJrxITljq/BXQ0R2GQ/fCeBwXp/1YvCFHv4wzMnKJGW/pcProdGbr1xvnKbxfRl/hb8G/0FErgGgADwN4Ddz/KzzxqPCH4p550OFr73ZHpQC6/ANdMdM11sja3zDw7d53eQW8JVS/yCvf3sjMAPbDMsyEzIK32KlMip8L13IvB4pukqHZZkRnidolCD3Ze/IckYrfN+TxI8k6UHmAKt0gEi5sUyOo3FoAAAKBklEQVR1ECZts/giaHbGvEqnzGiFv6kecGu4QeClU4IBLhIGeucxFX4KN15l8T1Bo806fGvx4iDPhG0W85BumyfuqPAlTcbxjieFG6+yeCJoJXX49gpIZ1e0nyh8BnyTrKXj7PRI8L3oBDCA18NkqsaNVya+ZwoDe+eJvSPLmcTS4aarDLR0spjVXLweKRMVKnwTfWQqQA/fSjwq/KHo21GRbCWTq5h357R0UrTCp4cf4ZdEGNg7spwJGPCHEsTqpBZ4TGYjq/C5ES2FVTpZzI2cVPgWkiZtaemYVOIAZ/PmkVFiLmSbvdlRs2OmDpHoK4lacGhsXjvORjsmbYeTKHzeqgMoz636qNm7bRLf/uD12LttouihWEFZ5gkDPssyM+gqHZtVyijJWDr08DNcetFk0UOwBs+4E7R57dg7spzxWaUzlIqXeviElg5ZH6bCr1g8T+wdWc7s3FRDxRdcPjtd9FCsIlH4Fk/aUeJ7DPhkbfySKHxn5e2uzRN4+KM3MrD1YVbpkL5bdV4TsgIZhc+dtnbCBTxIUqXDawMgOslIQw+frIRvVLfZXM7MVU0ypAqfwQ1Ik7Yidis3Uix6ntg+RxjwSYYKPfwM2pvlRjSyGnqe2L5u7B4dGTm6l47NiadRom/VecdDVsNPFL7d68bu0ZGRo6t02EYgwiuJciPFUpZ5YvfoyMipcONVhlTh83qQldHLxfZ1Y/foyMhJLB0GOAAM+GR9+CVZN3aPjoycxNKhZw0gvVXn9SCroYU9PXxSKvSEtV2pjIrkVp3Xg6wCq3RIKQloYWTwhNeDrA3r8EkpocLPknj4bBdNViFV+HbPE65qkiH18Dk1ACZtyfpIWytQ4ZMSEbA9coay1FeTYvFK0oPK7tGRkcPWClmo8Ml60JYOq3RIqdi7dRLvu/4KXH/VjqKHYgU+yzLJOvBLcha0s/3wyXA8T/ChG15a9DCswaPCJ+sg6aVj+Tyxe3SEFIwWbOwtRFajLArf7tERUjBJHb7lC5kUS1mS+3aPjpCCYR0+WQ9snkbIGOBzpy1ZBx6rdAgpP0zakvWQJm258YqQ0lKWplikWAImbQkpPzzikKwH7rQlZAygpUPWQ3InSIVPSHlJkraswyer4PEQc0LKDy0dsh7Kkuu5oNGJyN8VkYdFpCciB/teu01EnhSRx0TkhgsbJiHFcO1lW/A7b34JDu7bWvRQiMX4JVH4F9pL5zCAXwPwafNJEXk5gJsBvALAJQC+ISIvUUp1L/DzCBkptcDHP3vjgaKHQSynLLmeCxqdUupRpdRjQ166CcCdSqmWUuopAE8CuO5CPosQQmzF9fbIuwE8azw+Gj83gIjcKiKHROTQ6dOncxoOIYTkh1+Sssw1LR0R+QaAi4e89BGl1FdW+mtDnlPD3qiUuh3A7QBw8ODBoe8hhBCbCUpyiPmaAV8p9aYX8e8eBbDXeLwHwPEX8e8QQoj1vPbybfjNN1yOV1yyueihrEpe9x93A7hZRGoish/AAQA/yOmzCCGkUGbqFdz21pdZb+lcaFnmO0XkKIDXA/hLEfk6ACilHgbw5wAeAfA1AO9jhQ4hhBTLBZVlKqXuAnDXCq/9HoDfu5B/nxBCyMZh9/0HIYSQDYMBnxBCHIEBnxBCHIEBnxBCHIEBnxBCHIEBnxBCHEGUsqebgYicBvCzF/nXZwGc2cDh5A3Hmy8cb36UaayAG+O9TCm1fa03WRXwLwQROaSUOrj2O+2A480Xjjc/yjRWgOM1oaVDCCGOwIBPCCGOME4B//aiB3CecLz5wvHmR5nGCnC8CWPj4RNCCFmdcVL4hBBCVmEsAr6I3Cgij4nIkyLy4aLH04+I7BWRb4nIoyLysIh8IH5+m4jcKyJPxF+3Fj1WjYj4IvIjEbknfrxfRL4fj/V/iEi16DFqRGSLiHxZRH4SX+PXW35tfzueB4dF5IsiUrfp+orIZ0XklIgcNp4bej0l4g/jtfeQiFxryXj/YzwfHhKRu0Rki/HabfF4HxORG2wYr/HaB0VEichs/HhDr2/pA76I+AA+CeAtAF4O4BYReXmxoxqgA+B3lFIvA/A6AO+Lx/hhAN9USh0A8M34sS18AMCjxuN/D+Dj8VhfAPCeQkY1nE8A+JpS6qUAXo1o3FZeWxHZDeC3ABxUSl0NwAdwM+y6vncAuLHvuZWu51sQHXB0AMCtAD41ojGa3IHB8d4L4Gql1KsAPA7gNgCI193NAF4R/53/GseQUXIHBscLEdkL4M0AnjGe3tjrq5Qq9R9Eh6983Xh8G4Dbih7XGmP+SvyDfQzArvi5XQAeK3ps8Vj2IFrUvwLgHkRnFJ8BEAy75gWPdROApxDno4znbb22uwE8C2AbovMo7gFwg23XF8A+AIfXup4APg3glmHvK3K8fa+9E8AX4u8z8QHA1wG83obxAvgyIsHyNIDZPK5v6RU+0gWkORo/ZyUisg/AawB8H8BOpdQJAIi/7ihuZBn+AMC/ANCLH18EYF4p1Ykf23SNLwdwGsDnYgvqj0RkCpZeW6XUMQAfQ6TiTgBYAHA/7L2+mpWuZxnW328A+Gr8vZXjFZG3AzimlHqw76UNHe84BPxhx8RbWXokItMA/ieAf66UOlv0eIYhIm8DcEopdb/59JC32nKNAwDXAviUUuo1AJZgiX0zjNj7vgnAfgCXAJhCdNvejy3Xdy1snhsQkY8gslS/oJ8a8rZCxysikwA+AuBfD3t5yHMverzjEPCPAthrPN4D4HhBY1kREakgCvZfUEr9Rfz0SRHZFb++C8CposZn8AsA3i4iTwO4E5Gt8wcAtoiIPhLTpmt8FMBRpdT348dfRvQLwMZrCwBvAvCUUuq0UioE8BcAfh72Xl/NStfT2vUnIu8C8DYAv65iPwR2jvcKRALgwXjd7QHwQxG5GBs83nEI+PcBOBBXOVQRJWTuLnhMGUREAPwxgEeVUv/ZeOluAO+Kv38XIm+/UJRStyml9iil9iG6lv9HKfXrAL4F4O/Eb7NirACglHoOwLMiclX81BsBPAILr23MMwBeJyKT8bzQ47Xy+hqsdD3vBvAP42qS1wFY0NZPkYjIjQD+JYC3K6WWjZfuBnCziNREZD+iZOgPihijRin1Y6XUDqXUvnjdHQVwbTy3N/b6jjpZkVMC5K2IMvE/BfCRosczZHy/iOg27CEAD8R/3orIG/8mgCfir9uKHmvfuH8ZwD3x95cjWhhPAvgSgFrR4zPGeQ2AQ/H1/V8Attp8bQF8FMBPABwG8KcAajZdXwBfRJRfCOPg856Vriciy+GT8dr7MaLqIxvG+yQi71uvt/9mvP8j8XgfA/AWG8bb9/rTSJO2G3p9udOWEEIcYRwsHUIIIeuAAZ8QQhyBAZ8QQhyBAZ8QQhyBAZ8QQhyBAZ8QQhyBAZ8QQhyBAZ8QQhzh/wOWV1+XnT0y2wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x124f5d7f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(gga_nums)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow_workshop]",
   "language": "python",
   "name": "conda-env-tensorflow_workshop-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}