Project_2_description.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# W200 Introduction to Data Science Programming, UC Berkeley MIDS\n",
    "# Project 2\n",
    "\n",
    "The final project will comprise an Exploratory Data Analysis of a dataset of your choosing using the numpy and pandas tools.\n",
    "\n",
    "- Push your proposal to your team's GitHub repository and update the link in gradescope by 11:59PM PST the day before class, **week 12** \n",
    "- Your group will give a 10 minute presentation during your final class, **week 14**\n",
    "- Push your report to your team's GitHub repository and update the link in gradescope by 11:59PM PST the day **after** class, **week 14**.\n",
    "\n",
    "Some past showcase projects are here: https://github.com/UCB-INFO-PYTHON/Project2_Showcase\n",
    "\n",
    "## Instructions\n",
    "\n",
    "#### Team\n",
    "- Form a team of 2 to 3 students. By exception, a team of 4 students can be created with approval of your instructor and needs to be large in scope. A team of 5 students or more is not allowed. Record your team [here](https://docs.google.com/spreadsheets/d/1BK1E7BxYBo5eVU2Vq7N6vOaKiWkZ8tRKdSukUWAp4KM/edit?usp=sharing).\n",
    "- Create a new GitHub repository for your team in the UC-Berkeley-I-School organization. Name this folder as Project2 plus each last name of the students in the group like: **Project2_Foster_Kleemann_Llop**; please make this repo PRIVATE and invite the Instructor team / the rest of your team members.\n",
    "- You can choose your team members from among students in your section, or in exceptional cases from among students in other sections. (Please email **both** section instructors to get an OK)\n",
    "\n",
    "#### Data Analysis\n",
    "- This project is relatively unguided. You will come up with your own interesting questions that you can answer using the variables in your dataset, then perform an analysis based on those questions.\n",
    "- You can analyze either an *instructor-approved dataset of your finding* or one that we have pre-approved. You may join several datasets together. For example, you might combine two datasets to answer questions like, \"Is felony crime higher in NYC after 2 days of rain?\"\n",
    "- This project emphasizes the exploratory and descriptive techniques covered in class. Although some of you have a background in statistics, we ask that you avoid any statistical inference or other advanced techniques. In particular, this means that you should confine your analysis to the sample of data you have, and avoid making statements about the population that the sample comes from.\n",
    "\n",
    "#### Proposal (10%)\n",
    "With your team members, prepare a 1 to 2 page proposal about the questions that you intend to ask of the data. \n",
    "\n",
    "The proposal should describe:\n",
    "\n",
    "- Names of team members,\n",
    "- Name of your team's GitHub repository,\n",
    "- A primary dataset you intend to analyze,\n",
    "- Initial plots, figures, or tables,\n",
    "- Some of the variables (column names) you intend to explore and what kind of insights you expect to glean,\n",
    "- Supplemental datasets, if any, to complement your primary dataset - this means links, columns that you'll join on, etc.,\n",
    "- What you plan to cover in the final report and how you plan to organize it.\n",
    "\n",
    "#### In-Class Presentation (20%)\n",
    "The final class will include a 10 minute presentation per group with a 5 minute question/answer period. The presentation should include:\n",
    "- Your overall question\n",
    "- The steps your group took to analyze the question\n",
    "- Any assumptions you made in the analysis\n",
    "- The key is to aim for clarity and telling a story with the data\n",
    "- Organize your argument clearly\n",
    "- Guide the listeners through the evidence in the data\n",
    "- Include any key figures/plots/charts or graphs\n",
    "- You do not need to show any code but it might be handy to have the code ready in case there are some questions on it\n",
    "\n",
    "#### Report (70%)\n",
    "\n",
    "Grading Breakdown:\n",
    "- 10% Questions\n",
    "- 20% Data Cleaning / Sanity Checks\n",
    "- 20% Compelling Text and Data Stories\n",
    "- 20% Compelling Figures\n",
    "\n",
    "Your report should be 8-10 pages (including appropriately sized figures) and describe what you found out from the data. This should focus on telling stories and explaining the narrative of the exploration and challenges associated with that. The report should not include any code - rather, all code should be included in a sub-folder in either plain python files or in jupyter notebooks. If your report contents are over 8-10 pages, please edit to include the most important material in the first 8-10 pages and any extra material in the Appendix. \n",
    "\n",
    "For the report, any figure/plot/chart/graph should be annotated with descriptions about why they are included.\n",
    "\n",
    "Note: Using GitHub to share jupyter notebooks has proven problematic in the past. Be sure to create copies of notebooks when you make an edit.  As your code becomes increasingly stable, you may find it easier to just put it into a python file and import it into a jupyter notebook as needed.\n",
    "\n",
    "Points for the report:\n",
    "- Your analysis is a written argument.\n",
    "- The key is to aim for clarity and exposition.\n",
    "- Organize your argument clearly\n",
    "- Guide the reader through the evidence in the data.\n",
    "- Proofread!\n",
    "- Every figure/plot/chart/graph in the report should be mentioned in your writing, explain what it means\n",
    "- If you don’t have something to say about your figure/plot/chart/graph, don’t display it at all.\n",
    "- No output dumps\n",
    "- Document decisions\n",
    "  - If you decide that observations should be removed, state which ones.\n",
    "  - If values are suspicious, but you leave them in, state that too.\n",
    "  - If you transform a variable, for example, by taking the logarithm, state that.\n",
    "  - Your justification can often be very brief (just a sentence), but make sure the reader can follow your logic.\n",
    "- Characterize relationships between variables\n",
    "  - Keep in mind the purpose of the analysis.  \n",
    "  - For example, if you’re interested in explaining the price of a house, look to see what kind of relationship that variable has with the other variables.\n",
    "- Use only descriptive statistics\n",
    "  - Descriptive statistics summarize a particular sample of data.\n",
    "  - In the field of inferential statistics (which you’ll learn in w203) you’ll see how to create a model that represents the population (or process) from which the sample came from, and to make assertions about that population.\n",
    "  - Since we haven’t taught you any inference, please don’t use it.\n",
    "  - Beware of the word ‘significant’\n",
    "    - This has a technical meaning, implying that you’ve performed a statistical test\n",
    "\n",
    "\n",
    "## Instructor Pre-Approved Datasets\n",
    "- [Political Ad Archive](https://web.archive.org/web/20230608114140/http://politicaladarchive.org/data/)\n",
    "- [Global warming data (Berkeley Earth) - use the *time series section* ](http://berkeleyearth.org/data/)\n",
    "- [US Gov't Web Logs](https://analytics.usa.gov/)\n",
    "- [Parks graffitti report (St. Paul)](https://information.stpaul.gov/City-Infrastructure/Parks-Graffiti-Report-Dataset/gcu2-spkd) \n",
    "- [The National UFO Reporting Center Online Database](http://www.nuforc.org) will be challenging but fun since the reports are not uniform and are split across multiple tables that you may have to scrape.\n",
    "\n",
    "## Other Datasets\n",
    "**REMEMBER NONE OF THESE HAVE BEEN APPROVED, YOU HAVE TO TALK TO YOUR INSTRUCTOR TO GET THEM APPROVED.**\n",
    "\n",
    "- [Historical Weather Data](https://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/global-historical-climatology-network-ghcn)\n",
    "  - This can be a bit difficult to figure out what to download. [In this discussion](https://news.ycombinator.com/item?id=10465342) there are some pointers on how to get at it. This dataset is pre-approved as a **supplemental** dataset, but see your instructor if you would like this to be the primary dataset.\n",
    "- [Data is Plural](http://tinyletter.com/data-is-plural/archive)\n",
    "  - The pre-approved datasets come primarily from here. Pretty extensive list.\n",
    "- [Awesome Public Datasets](https://github.com/caesar0301/awesome-public-datasets)\n",
    "  - Awesome list that's worth saving for future reference. Be sure to actually download and explore the data before proposing one to your instructor. There are some awesome datasets here like the NYC taxi dataset.\n",
    "- [Data.gov](http://data.gov/)\n",
    "  - All government data, which is good!\n",
    "- [Reddit/r/Datasets](https://www.reddit.com/r/datasets)\n",
    "  - Community-generated and definitely some potential for good datasets.\n",
    "  \n",
    "#### To get a dataset approved ...\n",
    "1. Send your instructor a description of the data, including a link to the exact dataset or a place from where it can be downloaded with one click without registering (or anything like that). \n",
    "    - The data needs to be in a raw text file format like json or csv (or at least very easy to convert to that format).\n",
    "    - Report the size of the dataset or the subset you plan on looking at. This is **not** a big data project.\n",
    "2. Show that you've done some preliminary research and that there will be enough interesting questions to ask of the data. This should include column names and information about any missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}