hw_7_starter.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hw_7_starter.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hannahkates/nyu-python-public-policy/blob/master/hw_7_starter.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7fo4kU_UjgyY",
        "colab_type": "text"
      },
      "source": [
        "# **NYU Wagner - Python Coding for Public Policy**\n",
        "# Starter notebook for HW 7: Final Assignment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VmBy78gEXc7R",
        "colab_type": "text"
      },
      "source": [
        "## **Data Sources**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Co-yzOuAXliG",
        "colab_type": "text"
      },
      "source": [
        "311 Complaints Data\n",
        "- File path on google drive: '/content/drive/My Drive/Data for Python/cleaned_311_data_revised.csv'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X874QH7mdyCT",
        "colab_type": "text"
      },
      "source": [
        "Community District Profiles\n",
        "- [Data Dictionary](https://communityprofiles.planning.nyc.gov/data/cd_profile_data_dictionary.csv)\n",
        "- [Indicators data](https://planninglabs.carto.com/api/v2/sql?format=csv&q=SELECT%20cartodb_id,the_geom,the_geom_webmercator,acres,acs_tooltip,acs_tooltip_2,acs_tooltip_3,area_sqmi,borocd,cb_email,cb_website,cd_full_title,cd_short_title,cd_son_fy2018,cd_tot_bldgs,cd_tot_resunits,count_hosp_clinic,count_libraries,count_parks,count_public_schools,crime_count,crime_count_boro,crime_count_nyc,crime_per_1000,crime_per_1000_boro,crime_per_1000_nyc,female_10_14,female_15_19,female_20_24,female_25_29,female_30_34,female_35_39,female_40_44,female_45_49,female_5_9,female_50_54,female_55_59,female_60_64,female_65_69,female_70_74,female_75_79,female_80_84,female_85_over,female_under_5,fp_100_area,fp_100_bldg,fp_100_cost_burden,fp_100_cost_burden_value,fp_100_mhhi,fp_100_mortg_value,fp_100_openspace,fp_100_ownerocc,fp_100_ownerocc_value,fp_100_permortg,fp_100_pop,fp_100_rent_burden,fp_100_rent_burden_value,fp_100_resunits,fp_500_area,fp_500_bldg,fp_500_cost_burden,fp_500_cost_burden_value,fp_500_mhhi,fp_500_mortg_value,fp_500_openspace,fp_500_ownerocc,fp_500_ownerocc_value,fp_500_permortg,fp_500_pop,fp_500_rent_burden,fp_500_rent_burden_value,fp_500_resunits,lep_rate,lep_rate_boro,lep_rate_nyc,lot_area_commercial_office,lot_area_industrial_manufacturing,lot_area_mixed_use,lot_area_open_space,lot_area_other_no_data,lot_area_parking,lot_area_public_facility_institution,lot_area_res_1_2_family_bldg,lot_area_res_multifamily_elevator,lot_area_res_multifamily_walkup,lot_area_transportation_utility,lot_area_vacant,lots_commercial_office,lots_industrial_manufacturing,lots_mixed_use,lots_open_space,lots_other_no_data,lots_parking,lots_public_facility_institution,lots_res_1_2_family_bldg,lots_res_multifamily_elevator,lots_res_multifamily_walkup,lots_total,lots_transportation_utility,lots_vacant,male_10_14,male_15_19,male_20_24,male_25_29,male_30_34,male_35_39,male_40_44,male_45_49,male_5_9,male_50_54,male_55_59,male_60_64,male_65_69,male_70_74,male_75_79,male_80_84,male_85_over,male_under_5,mean_commute,mean_commute_boro,mean_commute_nyc,moe_bach_deg,moe_bach_deg_boro,moe_bach_deg_nyc,moe_foreign_born,moe_hh_rent_burd,moe_hh_rent_burd_boro,moe_hh_rent_burd_nyc,moe_lep_rate,moe_lep_rate_boro,moe_lep_rate_nyc,moe_mean_commute,moe_mean_commute_boro,moe_mean_commute_nyc,moe_over65_rate,moe_over65_rate_boro,moe_over65_rate_nyc,moe_poverty_rate,moe_under18_rate,moe_under18_rate_boro,moe_under18_rate_nyc,moe_unemnployment,moe_unemployment_boro,moe_unemployment_cd,neighborhoods,over65_rate,over65_rate_boro,over65_rate_nyc,pct_asian_nh,pct_bach_deg,pct_bach_deg_boro,pct_bach_deg_nyc,pct_black_nh,pct_clean_strts,pct_clean_strts_boro,pct_clean_strts_nyc,pct_foreign_born,pct_hh_rent_burd,pct_hh_rent_burd_boro,pct_hh_rent_burd_nyc,pct_hispanic,pct_other_nh,pct_served_parks,pct_white_nh,pop_2000,pop_2010,pop_acs,pop_change_00_10,poverty_rate,poverty_rate_boro,poverty_rate_nyc,puma,shared_puma,shared_puma_cd,son_issue_1,son_issue_2,son_issue_3,total_lot_area,under18_rate,under18_rate_boro,under18_rate_nyc,unemployment_boro,unemployment_cd,unemployment_nyc%20FROM%20community_district_profiles&filename=indicators.csv)\n",
        "- [Floodplain indicators data](https://planninglabs.carto.com/api/v2/sql?format=csv&q=SELECT%20*%20FROM%20cd_floodplains_v201902&filename=cd_floodplains_v201902.csv) (not covered in the data dictionary but I'm happy to help you track down the right fields to use)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LvGH8N9CLcj4",
        "colab_type": "text"
      },
      "source": [
        "## **Problem Statement:** \n",
        "311 data is an incredible resource, but one major concern with 311 complaints is that there can be bias in the data. The fact that one community district has a higher number of 311 noise complaints, doesn't necessarily mean it is actually louder than another community district. It simply means more people submitted 311 complaints. Some neighborhoods may have residents that are more likely to submit complaints than others for various reasons (ex: different expectations, more awareness about 311, more free time, more sense of empowerment or hope that the complaint will be resolved, etc). This makes it difficult to get a clear picture of conditions across the city, and it can help to compare datasets against each other.\n",
        "\n",
        "[Community District Profiles](https://communityprofiles.planning.nyc.gov/) is an application that provides objective statistics about community districts. In this assignment, you will chose an indicator published on Community District Profiles and compare it against 311 complaint data. \n",
        "\n",
        "**Some ideas for inspiration:**\n",
        "- How does the number of buildings in the floodplain correlate with the number of 311 flooding complaints?\n",
        "- How does the street cleanliness score correlate with 311 complaints about garbage and cleanliness?\n",
        "- How does the percent of residents living in rent burden correlate with a number of 311 housing complaints (no heat/hot water or unsanitary conditions)?\n",
        "\n",
        "**Note:** This is a more open ended assignment where you can dig into policy concerns that are of greatest interest to you. If the results of your analysis are inconclusive, that is okay, as long as you are using proper python coding techniques. You will be graded on approach and methodology, rather than the outcome of the analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xmKsEI-BkoXZ",
        "colab_type": "text"
      },
      "source": [
        "**Step 1**. Select an indicator from the Community District Profiles data that you want to analyze for this assignment. Create a dataframe that only captures the borocd and selected indicator column."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kc3DTsF7dWoq",
        "colab_type": "text"
      },
      "source": [
        "**Step 2**. Review the complaint types in the 311 complaint data. Choose the complaint type(s) you hypothesize should correlate with the indicator you chose in Step 2. Create a new dataframe that only captures the 311 complaints you want to analyze."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9CzRENAYg0bD",
        "colab_type": "text"
      },
      "source": [
        "**Step 3**. Write out the hypothesis for your analysis. What relationship are you expecting to find between the variables and why?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v_seR13meplF",
        "colab_type": "text"
      },
      "source": [
        "**Step 4**. Get the count of 311 complaints per community district (borocd) using the filtered dataframe you created in Step 2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FT7E3RjqfP3M",
        "colab_type": "text"
      },
      "source": [
        "**Step 5**. Join the indicator dataframe (from Step 1) with 311 counts dataframe (from Step 4) using the borocd value."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rupwOfaZfn0M",
        "colab_type": "text"
      },
      "source": [
        "**Step 6**. Produce a scatter plot of the indicator values vs. 311 complaint counts. The should be one point per community district."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5NQyNsUf07W",
        "colab_type": "text"
      },
      "source": [
        "**Step 7**. Conduct an OLS linear regression using the indicator values vs. 311 complaint counts and interpret the results."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y7sPO6qagdZ1",
        "colab_type": "text"
      },
      "source": [
        "**Step 8**. Use the linear regression results to add a line of best fit to your scatterplot"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WI6S7LpBgkIl",
        "colab_type": "text"
      },
      "source": [
        "**Step 9**. Write a paragraph explaining the results of your analysis? Have you found a meaningful relationship?"
      ]
    }
  ]
}