gpu_example_deepface_youtube.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9e63d6ef",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a436ec7e",
   "metadata": {},
   "source": [
    "<p><img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/5/50/Oracle_logo.svg/2560px-Oracle_logo.svg.png\" width=\"200\" align = \"left\"></p>\n",
    "\n",
    "# **<h1 align =\"middle\"><b> Oracle CloudWorld - Las Vegas</b></h1>**\n",
    "\n",
    "### **<h1 align =\"middle\"><b> Use case 1. Person Detection in Video</b></h1>**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4b983ba",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05935c8a",
   "metadata": {},
   "source": [
    "## **Prerequisites to run notebook**\n",
    "- Perform all steps in the \"1_prep_ocw_las_vegas_v1.ipynb\" file\n",
    "- Download folder from Github, containing 1) notebook, profile image example, VGG weights (vgg_face_weights.h5), score.py, and runtime.yaml\n",
    "- Set up Custom Networking (......)\n",
    "- Dynamic Group has been created and OCI Data Science policies are in order. Documentation can be found [here](https://docs.oracle.com/en-us/iaas/data-science/using/policies.htm)\n",
    "- A correct config file with private key are stored in the /home/datascience/.oci directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e5aeadd",
   "metadata": {},
   "source": [
    "# **| 1. Import libraries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "226689b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd \n",
    "from deepface import DeepFace\n",
    "import uuid\n",
    "import glob      \n",
    "import ocifs\n",
    "import base64\n",
    "import io\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import fsspec\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "from ads.model.framework.tensorflow_model import TensorFlowModel\n",
    "from ads.common.model_metadata import UseCaseType\n",
    "from ads.common.model_artifact import ModelArtifact\n",
    "from ads.common.model_export_util import prepare_generic_model\n",
    "from pytube import YouTube\n",
    "import uuid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc21052",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25ec7e0b",
   "metadata": {},
   "source": [
    "# **| 2. End to End Script**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7425e133",
   "metadata": {},
   "source": [
    "### **There are several functions in the end to end script. They are splitted in:**\n",
    "- **Function 1:** Function 1 fetches the YouTube URL, downloads the YouTube Video and stores it locally\n",
    "- **Function 2:** Function 2 fetches the encoded profile image, decodes the profile images, and stores it locally\n",
    "- **Prep for Predict:** Prep for Predict prepares for the predict function in deleting all previously ran files (e.g., images)\n",
    "- **Predict:** Predict calls Function 1 and Function 2. Following, the function splits the video into images and runs DeepFace algorithm on the split images. DeepFace compares each splitted image with the profile image and reviews whether the same person is in both the profile image and the split image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4c9cb649",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd \n",
    "from deepface import DeepFace\n",
    "import uuid\n",
    "import glob      \n",
    "import ocifs\n",
    "import base64\n",
    "import io\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import fsspec\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "from ads.model.framework.tensorflow_model import TensorFlowModel\n",
    "from ads.common.model_metadata import UseCaseType\n",
    "from ads.common.model_artifact import ModelArtifact\n",
    "from ads.common.model_export_util import prepare_generic_model\n",
    "from pytube import YouTube\n",
    "\n",
    "##########################################################################################################################################\n",
    "######################################################## Function 1          #############################################################\n",
    "##########################################################################################################################################\n",
    "\n",
    "def input_youtube_video(input_url):\n",
    "    \n",
    "    #delete previous videos\n",
    "    !rm -r /home/datascience/youtube_videos\n",
    "\n",
    "    #create a local directory to store the video\n",
    "    path_input_locally = \"/home/datascience/youtube_videos/\" \n",
    "\n",
    "    try:       \n",
    "        if not os.path.exists(path_input_locally):         \n",
    "            os.makedirs(path_input_locally)    \n",
    "\n",
    "    except OSError: \n",
    "        print ('Error: Creating directory for youtube video locally')\n",
    "        \n",
    "\n",
    "    #download file from youtube\n",
    "    yt = YouTube(input_url)\n",
    "\n",
    "    #store in local folder\n",
    "    stream = yt.streams.get_by_itag(22)\n",
    "    file_name_random = str(uuid.uuid4())\n",
    "    file_location_local = stream.download(output_path=path_input_locally, filename  = file_name_random + \".mp4\")\n",
    "    \n",
    "    print(\"Youtube download completed and stored in \" + str(file_location_local))\n",
    "    \n",
    "    return file_location_local\n",
    "\n",
    "##########################################################################################################################################\n",
    "######################################################## Function 2          #############################################################\n",
    "##########################################################################################################################################\n",
    "\n",
    "def input_profile_image(profile_image_as_bytes):\n",
    "    \n",
    "    #create a local directory to store the image\n",
    "    path_input_locally_image = \"/home/datascience/profile_image/\" \n",
    "\n",
    "    try:       \n",
    "        if not os.path.exists(path_input_locally_image):         \n",
    "            os.makedirs(path_input_locally_image)    \n",
    "\n",
    "    except OSError: \n",
    "        print ('Error: Creating directory for profile image locally')\n",
    "    \n",
    "    ##### decoding of profile image\n",
    "    img_bytes_p = io.BytesIO(base64.b64decode(profile_image_as_bytes.encode('utf-8')))\n",
    "    profile_image = Image.open(img_bytes_p).resize((224, 224)) \n",
    "    \n",
    "    #save image locally  \n",
    "    profile_image_loc = path_input_locally_image + \"pf_image.jpg\"\n",
    "    profile_image = profile_image.save(profile_image_loc)\n",
    "    \n",
    "    return profile_image_loc\n",
    "\n",
    "\n",
    "##########################################################################################################################################\n",
    "######################################################## Prep for Predict    #############################################################\n",
    "##########################################################################################################################################\n",
    "\n",
    "# Delete images if there are images in the local folder already\n",
    "path_split_images = \"/home/datascience/split_images\"\n",
    "files = glob.glob('/home/datascience/split_images/*.jpg')\n",
    "\n",
    "for f in files:\n",
    "    os.remove(f)\n",
    "\n",
    " #create a local folder to the images\n",
    "path_split_images = \"/home/datascience/split_images\"\n",
    "\n",
    "try:       \n",
    "    # creating a folder named split_images \n",
    "    if not os.path.exists(path_split_images):         \n",
    "        os.makedirs(path_split_images)    \n",
    "\n",
    "except OSError: \n",
    "    print ('Error: Creating directory of data for split images')\n",
    "\n",
    "\n",
    "##########################################################################################################################################\n",
    "######################################################## Predict             #############################################################\n",
    "##########################################################################################################################################\n",
    "\n",
    "def predict(input_youtube_url, input_pf):\n",
    "    \n",
    "#     #fetch variables from data payload\n",
    "#     profile_image_as_bytes = data['data']['pf_image']\n",
    "#     input_url = data['data']['input_url']\n",
    "    \n",
    "    ######\n",
    "    ###### Function 1    \n",
    "    file_location_local = input_youtube_video(input_youtube_url)\n",
    "    \n",
    "    ######\n",
    "    ###### Function 2\n",
    "    #profile_image_loc = input_profile_image(profile_image_as_bytes) \n",
    "    #save image\n",
    "    im1 = input_pf.save(\"./inputimage.jpg\")\n",
    "    \n",
    "        \n",
    "    print(\"Fetching video from \" + file_location_local)\n",
    "    #print(\"Fetching profile image from \" + profile_image_loc)\n",
    "    \n",
    "    \n",
    "    # Read the video from specified path \n",
    "    cam = cv2.VideoCapture(file_location_local)\n",
    "    \n",
    "    #get fps of original video\n",
    "    fps = cam.get(cv2.CAP_PROP_FPS)\n",
    "    print(\"**************************************************************** Original fps in video is \" + str(fps))\n",
    "    \n",
    "    #define list of frames to analyze. \n",
    "    list_of_frames = list(range(1, 18001, int(fps)))  #starts at frame 1, ends at frame 601 (which is 20 seconds at FPS = 30 and 10 seconds at FPS = 60) with 30 frames in between (= 1 second). So, takes 20 (= 21 seconds) frames from the video\n",
    "\n",
    "    #loop through the video and cut into images\n",
    "    currentframe = 0\n",
    "\n",
    "    while(True):\n",
    "        \n",
    "        for frame in list_of_frames:\n",
    "            cam.set(cv2.CAP_PROP_POS_FRAMES, frame)\n",
    "            print(\"Analyze frame number \" + str(frame))\n",
    "\n",
    "            # reading from frame \n",
    "            ret,frame = cam.read()\n",
    "\n",
    "            if ret:\n",
    "                if currentframe < 10:   \n",
    "                    name = path_split_images + '/frame000' + str(currentframe) + '.jpg'           \n",
    "\n",
    "                elif currentframe >= 10 and currentframe < 100:   \n",
    "                    name = path_split_images + '/frame00' + str(currentframe) + '.jpg'          \n",
    "\n",
    "                elif currentframe >= 100 and currentframe < 1000:   \n",
    "                    name = path_split_images + '/frame0' + str(currentframe) + '.jpg'   \n",
    "\n",
    "                else:\n",
    "                    name = path_split_images + '/frame' + str(currentframe) + '.jpg'      \n",
    "\n",
    "                print ('Creating...' + name) \n",
    "\n",
    "                # writing the extracted images \n",
    "                cv2.imwrite(name, frame) \n",
    "\n",
    "                # increasing counter\n",
    "                currentframe += 1\n",
    "            \n",
    "        else: \n",
    "            break\n",
    "\n",
    "    cam.release()    \n",
    "    \n",
    "    #apply DeepFace to the images\n",
    "    try:\n",
    "        dfs = DeepFace.find(img_path = \"./inputimage.jpg\", db_path = \"/home/datascience/split_images\", enforce_detection=False)  #first input is the profile image, second is the folder containing the split images\n",
    "        \n",
    "    except:\n",
    "        pass #if no face is found in any of the images\n",
    "    \n",
    "    #get the dataframe of the results\n",
    "    output_df = dfs[0]\n",
    "    \n",
    "    ########## calculations\n",
    "    seconds_in_screen = output_df.shape[0]  # = total frames detected = frames per second as we are looping through each frame\n",
    "    total_seconds_video_analyzed = len(list_of_frames)\n",
    "    \n",
    "    print(\"**************************************************************** Total seconds analyzed of entire video \" + str(len(list_of_frames)) + \" seconds\")\n",
    "    print(\"**************************************************************** This person was \" + str(seconds_in_screen) + \" seconds in screen\")\n",
    "\n",
    "    #delete pickle file and deleting the video afterwards\n",
    "    !rm -r /home/datascience/split_images/representations_vgg_face.pkl\n",
    "    \n",
    "    in_screen = str()\n",
    "    \n",
    "    if seconds_in_screen > 0:\n",
    "        in_screen = \"Person is found in video\"\n",
    "    else:\n",
    "        in_screen = \"Person is not found in video\"\n",
    "        \n",
    "    \n",
    "    return in_screen, total_seconds_video_analyzed, seconds_in_screen\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0eee11a",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd62bc31",
   "metadata": {},
   "source": [
    "# **| 3. Testing the Script**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77d373b1",
   "metadata": {},
   "source": [
    "## **| 3.1 Input for Test**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "be912f9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ############\n",
    "# ############ Input 1 = input_url\n",
    "# ############ \n",
    "# input_url = \"https://www.youtube.com/shorts/ugwUcdtygok\" \n",
    "# #other example: https://www.youtube.com/shorts/Y-PBRyEz4xY\n",
    "\n",
    "# ############\n",
    "# ############ Input 2 = Encoded Profile image\n",
    "# ############ \n",
    "\n",
    "# #get a dummy example. In the video of Jimmy Car, we'll use a Jimmy Car Image to analyze the video.\n",
    "# file_path=\"./jimmycar.jpg\" \n",
    "\n",
    "# #open and encode the dummy image\n",
    "# with open(file_path, \"rb\") as image2string:\n",
    "#     converted_string = base64.b64encode(image2string.read()).decode('ascii')\n",
    "               \n",
    "# #add payload to full string\n",
    "# payload1 = json.dumps(converted_string)\n",
    "# pf_image_encoded = json.loads(payload1)\n",
    "\n",
    "# ############\n",
    "# ############ Input 1 and 2 into one payload\n",
    "# ############ \n",
    "# data = {'data':{'pf_image': pf_image_encoded, 'input_url': input_url}}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea59021",
   "metadata": {},
   "source": [
    "## **| 3.2 Run Prediction in Gradio**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9ccfbc5",
   "metadata": {},
   "source": [
    "## examples\n",
    "- Harvard lecture, 10 min video: https://www.youtube.com/watch?v=p7iwXvBnbIE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0337f693",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finding representations:  82%|████████▏ | 233/283 [00:48<00:10,  4.71it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local URL:  http://127.0.0.1:7860\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finding representations: 100%|█████████▉| 282/283 [00:59<00:00,  4.12it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on public URL: https://34cd8298b288e7c816.gradio.live\n",
      "\n",
      "This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Finding representations: 100%|██████████| 283/283 [00:59<00:00,  4.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Representations stored in /home/datascience/split_images/representations_vgg_face.pkl file.Please delete this file when you add new identities in your database.\n",
      "find function lasts  60.730356216430664  seconds\n",
      "**************************************************************** Total seconds analyzed of entire video 783 seconds\n",
      "**************************************************************** This person was 111 seconds in screen\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://34cd8298b288e7c816.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gradio as gr\n",
    "\n",
    "with gr.Blocks() as demo:\n",
    "\n",
    "    gr.Markdown(\"Detecting a Person in a (YouTube) video\")\n",
    "        \n",
    "    with gr.Tab(\"YouTube Video\"):\n",
    "        with gr.Row():\n",
    "            #input\n",
    "            input_youtube_url = gr.Text(label=\"YouTube URL\", info=\"Please add the full YouTube URL\")\n",
    "            input_pf = gr.Image(type='pil', label=\"Profile Image\", info=\"Please add Profile Image here\")\n",
    "        \n",
    "        #trigger\n",
    "        button_1 = gr.Button(\"Cross-check Profile Image with the YouTube Video\")\n",
    "        \n",
    "        ##output\n",
    "        in_screen = gr.Text(label='Was person in screen?')\n",
    "        total_seconds_video_analyzed = gr.Text(label='Total duration video in seconds')\n",
    "        seconds_in_screen  = gr.Text(label='Total seconds person in profile image was in the video')\n",
    "\n",
    "        \n",
    "        ## buttons\n",
    "        button_1.click(predict, inputs=[input_youtube_url, input_pf], outputs=[in_screen, total_seconds_video_analyzed, seconds_in_screen])\n",
    "\n",
    "demo.launch(share=True, debug=True) #width=800, height=1100, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9479d4ab",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e65c3c2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e38a38a9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c58a93d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8bd4706e",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea8ece3b",
   "metadata": {},
   "source": [
    "# **The End**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow28_p38_gpu_v1]",
   "language": "python",
   "name": "conda-env-tensorflow28_p38_gpu_v1-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}