HW8_ImageAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "CNJF9PVslw5Y",
   "metadata": {
    "id": "CNJF9PVslw5Y"
   },
   "source": [
    "# Homework 8 - Image Analysis\n",
    "\n",
    "This notebook provides some skeleton code to get you started on the homework. Add in your own code and markdown cells to answer the homework questions. If you want to submit the notebook as a PDF, make sure your code and markdowns are clear and concise to make grading easy for the TAs.\n",
    "\n",
    "This notebook can be opened in Colab\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/zlisto/social_media_analytics/blob/main/HW8_ImageAnalysis.ipynb)\n",
    "\n",
    "\n",
    "Before starting, select \"Runtime->Factory reset runtime\" to start with your directories and environment in the base state.\n",
    "\n",
    "If you want to save changes to the notebook, select \"File->Save a copy in Drive\" from the top menu in Colab. This will save the notebook in your Google Drive.\n",
    "\n",
    "For all plots, make sure your axes have nice labels with easy to read fontsizes, otherwise points will be deducted.\n",
    "\n",
    "The OpenAI API calls to do this homework cost approximately <insert cost>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "utVwd-MCuE_d",
   "metadata": {
    "id": "utVwd-MCuE_d"
   },
   "source": [
    "# Clones, Installs, and Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "nzFWTbuarc62",
   "metadata": {
    "id": "nzFWTbuarc62"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "sv6-tXg99Gen",
   "metadata": {
    "id": "sv6-tXg99Gen"
   },
   "source": [
    "# Problem 1. (31 points) Basic Image Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2uo6_BQXuddW",
   "metadata": {
    "id": "2uo6_BQXuddW"
   },
   "source": [
    "## (5 points) 1. Load Image\n",
    "\n",
    "Load the image with file name `\"data/image_compressed_leomessi/3291736544082520189_427553890_small.jpeg\"` and display it. Convert the image to an appropriate `image_url` for Vision (GPT-4V) and add it to a list `image_urls`. Print out the list `image_urls` so we know you did it right. Also, display the image so you can see how cool it looks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "CcOEFpCMi9fS",
   "metadata": {
    "id": "CcOEFpCMi9fS"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9_pak_rVvGtk",
   "metadata": {
    "id": "9_pak_rVvGtk"
   },
   "source": [
    "## (5 points) 2. Recommendations to Enhance Image\n",
    "\n",
    "Use Vision to provide recommendations to make the image more engaging for Instagram.  Tell it who the target audience is for this image.  Call the recommendation a string `recommendation`.  Print out `recommendation` using `tr.fill`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2IE16t6i-U_",
   "metadata": {
    "id": "d2IE16t6i-U_"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fX5CXW3yuEBj",
   "metadata": {
    "id": "fX5CXW3yuEBj"
   },
   "source": [
    "## (4 points) 3. Describe Image\n",
    "\n",
    "Use Vision to describe the original image and call this `image_description`.  Print out `image_description` with `tr.fill`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94Ih7nhjoPTC",
   "metadata": {
    "id": "94Ih7nhjoPTC"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "SFploxbiLXlR",
   "metadata": {
    "id": "SFploxbiLXlR"
   },
   "source": [
    "## (5 points) 4. Enhanced Image\n",
    "\n",
    "Create a `prompt` that combines `image_description` with the recommendations from the previous part and have DALL-E 3 generate the enhanced image.  Display your enhanced image using the `display_image_url` function so the TAs can see it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ym4_PYprjAXe",
   "metadata": {
    "id": "ym4_PYprjAXe"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "QsApFxtRx9aB",
   "metadata": {
    "id": "QsApFxtRx9aB"
   },
   "source": [
    "## (6 points) 5. Generate Caption for Image\n",
    "Go to the Instagram account for leomessi and copy a caption for one of his images and create a string called `caption_example` that equals this caption.  Then feed the generated image to Vision and have it generate an Instagram caption for it using the `caption_example` as a style and tone guide.  Print out the generated caption using `tr.fill`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "PCrx6ij0jBpu",
   "metadata": {
    "id": "PCrx6ij0jBpu"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "XN7QSQRsxJpu",
   "metadata": {
    "id": "XN7QSQRsxJpu"
   },
   "source": [
    "## (6 points) 6. Compare Images\n",
    "\n",
    "Give Vision the original image and the generated image as inputs and ask it to explain which image will be more engaging on Instagram for an audience of soccer fans.  Print out the explanation using `tr.fill` so it is easy to read.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "G4TCZMA9jDB7",
   "metadata": {
    "id": "G4TCZMA9jDB7"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "qsq390FDxYAF",
   "metadata": {
    "id": "qsq390FDxYAF"
   },
   "source": [
    "# Problem 2. (30 points) Cluster Analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "_OVrRX1sFU8E",
   "metadata": {
    "id": "_OVrRX1sFU8E"
   },
   "source": [
    "## (3 points) 1. Load Image Data\n",
    "\n",
    "Load the file `data/leomessi_embedding.csv` into a dataframe `df`.  Print out the head of the dataframe.  You should see the image path and the UMAP coordinates for each image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ppxgYBcvjFUT",
   "metadata": {
    "id": "ppxgYBcvjFUT"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "PxDC6v648yUW",
   "metadata": {
    "id": "PxDC6v648yUW"
   },
   "source": [
    "## (8 points) 2. Cluster Images\n",
    "\n",
    "Apply K-means clustering to the UMAP embedding with `k=4` clusters, `n_init='auto'` and `random_state = 0`.  Add these labels to `df` in a column called `cluster`.  Make a `countplot` of the `cluster` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "mNJ4Q2P5jGMa",
   "metadata": {
    "id": "mNJ4Q2P5jGMa"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "-Q08Rb4ICrqu",
   "metadata": {
    "id": "-Q08Rb4ICrqu"
   },
   "source": [
    "## (8 points) 3. Analyze Clusters\n",
    "\n",
    "We will now use Vision to analyze each cluster.\n",
    "Do the following for each cluster in `df`.\n",
    "Sample 4 random images from each cluster and give them to Vision with `instructions` telling it to describe the common theme in the cluster.  Display each sampled image and print out the cluster description (use the `tr.fill` function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "856U98BnElLK",
   "metadata": {
    "id": "856U98BnElLK"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "O9wjPR_i6ckx",
   "metadata": {
    "id": "O9wjPR_i6ckx"
   },
   "source": [
    "## (6 points) 4. New Content Ideas\n",
    "\n",
    "Take a sample of 4 images from one of the clusters and give them to Vision and ask it to generate three ideas for new content this account could post in a similar style.  Ask it to return the answer as an HTML string and dislay the response with the `display` and `HTML` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "yvBlvwEyjgbJ",
   "metadata": {
    "id": "yvBlvwEyjgbJ"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8YhvEuK84zBH",
   "metadata": {
    "id": "8YhvEuK84zBH"
   },
   "source": [
    "## (5 points) 5. Generate Image From Idea\n",
    "\n",
    "Take one of the image ideas from the previous problem and generate an image from it using DALL-E 3.  Display the image using the `display_image_url` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "DInkjyfzo9jn",
   "metadata": {
    "id": "DInkjyfzo9jn"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b6ECdt2Qk4x5",
   "metadata": {
    "id": "b6ECdt2Qk4x5"
   },
   "source": [
    "# Problem 3 (18 points) Tweeting a Reaction to an Image\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8NK2R5MGtSNV",
   "metadata": {
    "id": "8NK2R5MGtSNV"
   },
   "source": [
    "## (5 points) 1. Load Image\n",
    "\n",
    "Load the image with file name `\"data/image_compressed_cristiano/1603695572069767264_173560420_small.jpeg\"` and display it. Convert the image to an appropriate `image_url` for Vision and add it to a list `image_urls`. Print out the list `image_urls` so we know you did it right.  Also display the image so you can see how cool it looks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "qtON4cBi9NXO",
   "metadata": {
    "id": "qtON4cBi9NXO"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "YJ_O9jMWtZtS",
   "metadata": {
    "id": "YJ_O9jMWtZtS"
   },
   "source": [
    "## (7 points) 2. Describe Twitter User\n",
    "\n",
    "Write a `user_description` of a Twitter user who will see this image and react to it.  Be creative :).  Then give Vision `instructions` that tell it to provide more details about the tweeting style of this user.  Call this output string `tweet_style`.  Print out `tweet_style` using `tr.fill`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "_yjPZ_BJ9N6N",
   "metadata": {
    "id": "_yjPZ_BJ9N6N"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "WFzhZjkytev-",
   "metadata": {
    "id": "WFzhZjkytev-"
   },
   "source": [
    "## (6 points) 3. Tweet Reaction to Image\n",
    "\n",
    "Use Vision to write a tweet in the style of the user from the previous part to the image in `image_urls` from part 1.  Print out your instructions prompt and the tweet reaction using `tr.fill`.  Also display the orignal image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "DUwDiQwOo7Dp",
   "metadata": {
    "id": "DUwDiQwOo7Dp"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "xAM0opBOxk2d",
   "metadata": {
    "id": "xAM0opBOxk2d"
   },
   "source": [
    "# Problem 4. (21 points) Generate Text Narration of a Video"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "h0AD-xSD5Vv1",
   "metadata": {
    "id": "h0AD-xSD5Vv1"
   },
   "source": [
    "## (6 points) 1. Load Frames From Video\n",
    "\n",
    "Load the video located at `\"data/videos/leomessi_Argentina.mp4\"` and convert it into a list of images called `base64Frames`.  Print out the length of `base64Frames`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wLDTk6xC9Paw",
   "metadata": {
    "id": "wLDTk6xC9Paw"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "thHYHmBT5bRQ",
   "metadata": {
    "id": "thHYHmBT5bRQ"
   },
   "source": [
    "## (3 points) 2. Sample Frames\n",
    "\n",
    "Sample every 60 frames from the video and call this array `base64Frames_samples`.  Print out the length of  `base64Frames_samples`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "RKXJETqdo5iW",
   "metadata": {
    "id": "RKXJETqdo5iW"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "jXLKOY-k7H4F",
   "metadata": {
    "id": "jXLKOY-k7H4F"
   },
   "source": [
    "## (3 points) 3. Narration Length\n",
    "\n",
    "A human speaks 200 words per minute.  Using this, calculate `nwords_max`, the maximum number of words in the narration for the video.  Print out the duration of the video in seconds and `nwords_max`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pf5_qnP29RKh",
   "metadata": {
    "id": "pf5_qnP29RKh"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "S6p4xkQN5ouL",
   "metadata": {
    "id": "S6p4xkQN5ouL"
   },
   "source": [
    "## (6 points) 4. Create Text of Narration for Video\n",
    "\n",
    "Give Vision  `base64Frames_samples` and some `instructions` to generate the narration text for the video.  You can choose the style and focus of the narrative, but it should relate to the video content.\n",
    "\n",
    " Make sure the narration has fewer than `nwords_max` words.  Print out the narration (use `tr.fill`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "NtI-wgPL9R08",
   "metadata": {
    "id": "NtI-wgPL9R08"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "dN7q-iQp7c8L",
   "metadata": {
    "id": "dN7q-iQp7c8L"
   },
   "source": [
    "## (3 points) 5. Word Count of Narration\n",
    "\n",
    "Print out the number of words in the narration.  Make sure it is less than `nwords_max`. If it does not, generate another narration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "QOpwdWr6o2F6",
   "metadata": {
    "id": "QOpwdWr6o2F6"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fKyHGbTp3mcb",
   "metadata": {
    "id": "fKyHGbTp3mcb"
   },
   "source": [
    "## (0 points) 6. Add Narration to Video\n",
    "\n",
    "If you like, you can generate the audio for the narration and add it to the video using a video editing software.  If you think the video is really compelling, email it to the Professor and if he agrees, he will highlight it during lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "T7VPcuwU9csO",
   "metadata": {
    "id": "T7VPcuwU9csO"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}