ProjectProposalGroup096-Fa21.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COGS 108 - Final Project Proposal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Names\n",
    "\n",
    "- Joonsung Park\n",
    "- James Chung\n",
    "- Richard Gross\n",
    "- Madison Hambly\n",
    "- Colin Lintereur"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Research Question"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do the most popular topics from previous COGS 108 final projects vary per quarter from 2019 to 2021? Also what other trends can we find in past submissions, like changes in word count, and number of graphs used?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background and Prior Work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While looking through the previous COGS 108 final projects for our project review, we noticed that the main COGS108 repository contains a lot of final project submission from previous quarters [1]. During the process for looking through the previous submissions, we thought that there might have been an increase of projects related to certain topics, like COVID. Because of this, we thought it would be interesting to use those submissions as our dataset and to quantify the popularity of the topics of those previous projects and how popular topics for projects changed over time. In order to be able to use the previous submissions, we looked into how we could parse the notebook files so that we could easily extract whatever we needed from them since the structure of notebook files is strange [2]. Understanding this methodology would be essential to deciding the approach that we will take to acquire and eventually wrangle the topic data that we seek to analyze. \n",
    "\n",
    "References (include links):\n",
    "- 1) https://github.com/orgs/COGS108/repositories?page=1\n",
    "- 2) https://towardsdatascience.com/notebook-meta-analysis-jupyter-as-a-zero-infrastructure-alternative-to-experiment-trackers-69e7343d1343?gi=b945c70dda05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We predict that projects related to public health and politics will be more popular and we think that popular topics do not vary too much per quarter. Due to most of the submissions between 2019 and 2021 being submitted during COVID and periods of social unrest, we suspect that it would have been a very popular to run data analysis on topics related to these events. We also expect that whatever topics were popular in one quarter, were still popular in subsequent quarters since quarters are a relatively short amount of time and it would be unlikely that popular project topics would change drastically during that time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our initial dataset will be the final project submissions of previous groups from each quarter's repos. As our dataset consists of Jupyter notebooks, we will be working with semi-structured data in the form of JSON files (as Jupyter notebooks are JSONs). Our preliminary thoughts on collecting data on these notebooks is to parse either the title or research question to categorize each notebook into certain topics. If parsing through the notebook as a JSON does not seem effective, we may use webscraping instead if necessary. Our dataset would include roughly 400 projects over 6 quarters (2019-2021). Each observation will have their own `group` number, and because we want to examine the trends in topic popularity between quarters, we will also use a `quarter` variable ranging from `sp19, fa19,...,sp21` (this will also help us differentiate between duplicate group numbers between quarters). The main data we will be collecting though is the parsing of the notebooks as mentioned previously, categorizing each project under a `topic` variable. The main idea is that we will be trying to keep general topics so as to count as many projects as possible that have similar projects and answer our research question based on that. We are not necessarily using data directly in the notebooks but rather parsing them for the variables we want to investigate, so we will be constructing a new dataset that has our relevant information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ethics & Privacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One area of our topic that concerns the privacy of others is that previous project submissions include the names of the group members that made that project. To account for this, these names will be ignored and removed from our analysis to maintain the privacy of those individuals. One way our analysis may be biased is that our topic categories may not properly capture the full scope of the project and will be influenced by our own opinions since there is no way to objectively categorize these projects. To combat this, our team will work together to make sure that we all agree about our decisions in how we delineate project topics. This way, it will be less likely that one person's bias will affect the entire project. Another bias that we will encounter is the issue of certain group submissions not existing in the COGS 108 repositories. For whatever reason this is, we cannot fully categorize every previous project submission."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Team Expectations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Each member is expected to give their full effort in completing the tasks that were assigned to them and finishing the project\n",
    "- Each member is expected to commuincate transparently about any successes or issues encountered\n",
    "- Each member is expected to respect each other"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project Timeline Proposal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Meeting Date  | Meeting Time| Completed Before Meeting  | Discuss at Meeting |\n",
    "|---|---|---|---|\n",
    "| 10/27  |  12 PM | Obtain and clean our dataset | Discuss anomalies that we find in the data and how to move forward | \n",
    "| 11/3  |  12 PM | Wrangle data | Discuss ideal dataset | \n",
    "| 11/10  | 12 PM  | Submit first checkpoint | Discuss approaches to categorizing project topics |\n",
    "| 11/17  | 12 PM  | Start categorizing previous project topics | Better methods for categorizing topics |\n",
    "| 11/22  | 12 PM  | Finish categorizing topics and submit second checkpoint | Discuss starting out analysis on our data |\n",
    "| 12/1  | 12 PM  | Complete analysis; Draft results/conclusion| Discuss/edit full project |\n",
    "| 12/10  | Before 11:59 PM  | Final project and video | Turn in Final Project & Group Project Surveys |"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ac59ebe37160ed0dfa835113d9b8498d9f09ceb179beaac4002f036b9467c963"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}