M1ExploreDataSet-lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Survey Dataset Exploration Lab**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   Load the dataset that will used thru the capstone project.\n",
    "-   Explore the dataset.\n",
    "-   Get familier with the data types.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the required libraries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is available on the IBM Cloud at the below url.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset_url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/LargeData/m1_survey_data.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data available at dataset_url into a dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_survey = pd.read_csv(dataset_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the data set\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a good idea to print the top 5 rows of the dataset to get a feel of how the dataset will look.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the top 5 rows and columns from your dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>22.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>New Zealand</td>\n",
       "      <td>No</td>\n",
       "      <td>Some college/university study without earning ...</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Bisexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Somewhat more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Cour...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Australia</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>29.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>Hispanic or Latino/Latina;Multiracial</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Respondent                      MainBranch Hobbyist  \\\n",
       "0           4  I am a developer by profession       No   \n",
       "1           9  I am a developer by profession      Yes   \n",
       "2          13  I am a developer by profession      Yes   \n",
       "3          16  I am a developer by profession      Yes   \n",
       "4          17  I am a developer by profession      Yes   \n",
       "\n",
       "                                         OpenSourcer  \\\n",
       "0                                              Never   \n",
       "1                         Once a month or more often   \n",
       "2  Less than once a month but more than once per ...   \n",
       "3                                              Never   \n",
       "4  Less than once a month but more than once per ...   \n",
       "\n",
       "                                          OpenSource          Employment  \\\n",
       "0  The quality of OSS and closed source software ...  Employed full-time   \n",
       "1  The quality of OSS and closed source software ...  Employed full-time   \n",
       "2  OSS is, on average, of HIGHER quality than pro...  Employed full-time   \n",
       "3  The quality of OSS and closed source software ...  Employed full-time   \n",
       "4  The quality of OSS and closed source software ...  Employed full-time   \n",
       "\n",
       "          Country Student                                            EdLevel  \\\n",
       "0   United States      No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "1     New Zealand      No  Some college/university study without earning ...   \n",
       "2   United States      No        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "3  United Kingdom      No        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "4       Australia      No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "\n",
       "                                      UndergradMajor  ...  \\\n",
       "0  Computer science, computer engineering, or sof...  ...   \n",
       "1  Computer science, computer engineering, or sof...  ...   \n",
       "2  Computer science, computer engineering, or sof...  ...   \n",
       "3                                                NaN  ...   \n",
       "4  Computer science, computer engineering, or sof...  ...   \n",
       "\n",
       "                              WelcomeChange  \\\n",
       "0   Just as welcome now as I felt last year   \n",
       "1   Just as welcome now as I felt last year   \n",
       "2  Somewhat more welcome now than last year   \n",
       "3   Just as welcome now as I felt last year   \n",
       "4   Just as welcome now as I felt last year   \n",
       "\n",
       "                                        SONewContent   Age Gender Trans  \\\n",
       "0  Tech articles written by other developers;Indu...  22.0    Man    No   \n",
       "1                                                NaN  23.0    Man    No   \n",
       "2  Tech articles written by other developers;Cour...  28.0    Man    No   \n",
       "3  Tech articles written by other developers;Indu...  26.0    Man    No   \n",
       "4  Tech articles written by other developers;Indu...  29.0    Man    No   \n",
       "\n",
       "                 Sexuality                              Ethnicity Dependents  \\\n",
       "0  Straight / Heterosexual           White or of European descent         No   \n",
       "1                 Bisexual           White or of European descent         No   \n",
       "2  Straight / Heterosexual           White or of European descent        Yes   \n",
       "3  Straight / Heterosexual           White or of European descent         No   \n",
       "4  Straight / Heterosexual  Hispanic or Latino/Latina;Multiracial         No   \n",
       "\n",
       "            SurveyLength                  SurveyEase  \n",
       "0  Appropriate in length                        Easy  \n",
       "1  Appropriate in length  Neither easy nor difficult  \n",
       "2  Appropriate in length                        Easy  \n",
       "3  Appropriate in length  Neither easy nor difficult  \n",
       "4  Appropriate in length                        Easy  \n",
       "\n",
       "[5 rows x 85 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_survey.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find out the number of rows and columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by exploring the numbers of rows and columns of data in the dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the number of rows in the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of rows in the dataframe is 11552.\n"
     ]
    }
   ],
   "source": [
    "print('The number of rows in the dataframe is ' + str(df_survey.shape[0]) +'.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the number of columns in the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of columns in the dataframe is 85.\n"
     ]
    }
   ],
   "source": [
    "print('The number of columns in the dataframe is ' + str(df_survey.shape[1]) +'.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify the data types of each column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explore the dataset and identify the data types of each column.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the datatype of all columns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Respondent       int64\n",
      "MainBranch      object\n",
      "Hobbyist        object\n",
      "OpenSourcer     object\n",
      "OpenSource      object\n",
      "                 ...  \n",
      "Sexuality       object\n",
      "Ethnicity       object\n",
      "Dependents      object\n",
      "SurveyLength    object\n",
      "SurveyEase      object\n",
      "Length: 85, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(df_survey.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the mean age of the survey participants.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30.77239449133718\n"
     ]
    }
   ],
   "source": [
    "print(df_survey['Age'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is the result of a world wide survey. Print how many unique countries are there in the Country column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "United States        3173\n",
      "India                 911\n",
      "United Kingdom        841\n",
      "Germany               715\n",
      "Canada                442\n",
      "                     ... \n",
      "Liechtenstein           1\n",
      "Rwanda                  1\n",
      "Brunei Darussalam       1\n",
      "Cape Verde              1\n",
      "Iraq                    1\n",
      "Name: Country, Length: 135, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df_survey['Country'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ramesh Sannareddy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rav Ahuja\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By        | Change Description                 |\n",
    "| ----------------- | ------- | ----------------- | ---------------------------------- |\n",
    "| 2020-10-17        | 0.1     | Ramesh Sannareddy | Created initial version of the lab |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Copyright © 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}