02_getting_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting data\n",
    "Getting nice data and making it computationallly usable often takes the lion's share of computational work. As a rule of thumb supplemental tables are less accessible than data provided by organizations, and organizations with long-term efforts provide data in the nicest way. One excellent source of data for biology is the National Libary of Medicine. Besides providing online versions of their tools such as Pubmed, you can also find most of their information on a server that allows you to download the underlying data. The web addres of this goldmine is: ftp://ftp.ncbi.nlm.nih.gov\n",
    "\n",
    "For this tutorial we will \n",
    "- Get the names of the genes with the highest number of human GeneRIF statements (which is similar to our publication where we used PubMed). GeneRIF statements provide short paragraph on the function of genes. \n",
    "- We will further write a function that will display all GeneRIF statements for a gene of your interest.\n",
    "\n",
    "We will need two files:\n",
    "- A table containing the names of human genes: ftp://ftp.ncbi.nlm.nih.gov//gene/DATA/GENE_INFO/Mammalia/Homo_sapiens.gene_info.gz\n",
    "- A table containing all GeneRIF statements: ftp://ftp.ncbi.nlm.nih.gov//gene/GeneRIF/generifs_basic.gz\n",
    "\n",
    "Download and unzip those two files to a folder on your computer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting the data into this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start pandas you will need to execute the following line of code. You can do this by clicking on the code and then pressing Control together with Enter, or by clicking on \"Cell\" in the top naviation bar and then clicking on \"Run Cells\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations. If you executed the above code correctly, the square brackets on the left should contain a number."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The GeneRIF statements and the information on human genes come as tables, but we do not know their precise format. While this could be a problem, pandas contains a function called .read_table, which makes educated guesses. Oftentimes, this works without further tweaking from your side.\n",
    "\n",
    "<font color=\"red\">You will need to adjust the following path according to the name you gave to the folder, and its location. If using Windows you will also need to change the ~/Desktop to the location of the desktop within the file system of your computer. Tip: right click one of the files in the folder and inspect properties to see path.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_gene_info = pd.read_table('~/Desktop/data_sources/Homo_sapiens.gene_info')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To peak at the first rows of gene_info, use the function .head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_gene_info.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"green\">What did we just see?</font>\n",
    "- The content of the file Homo_sapiens.gene_info now is stored in <code>human_gene_info</code>\n",
    "- More generally <code>x_left_of_equal_sign = whatever_commmand_right_of_equal sign</code>, will first execute the code right to the equal sign and then hand it over to the thing on the left hand side.\n",
    "- Here the thing on the left hand side of the equal sign is a variable, though you could give it an arbitrary name, it is useful to give it a name that is well readable, such as <code>human_gene_info</code>\n",
    "- Functions do function-specific stuff\n",
    "- While the data is outside of this notebook you can not apply functions that come with the loaded data itself, instead you have to resort to a function of pandas itself to the data into the notebook. Only after the data has been loaded, you can apply a function that comes with the data (these other functions, such as <code>.head()</code>, will essentially have been stuck to the data in <code>human_gene_info</code> by pandas and its <code>read_table function</code>). Though you may later encounter exceptions, this is a general guide on using functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try to only see the last few rows. The function to of doing so starts with \"tai\". To see the complete name of the function, start typing and then press the tabulator key. This will show the available functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you display the last rows? If so, you might want to manually compare the first and the last rows. You will likely see that the last rows have fwer content, and that the GeneID is much higher."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now load the table containing GeneRIFs. Don't be afraid by the warning that will likely occur. It indicates that parts of the data use text characters in a column that largely consists of numbers. Knowing about such unexpected observations is helpful when working with larger data. Luckily this won't affect this specific tutorial, for which no futher action will be required to resolve this suspicious observation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_rif = pd.read_table('/Users/tstoeger/Desktop/data_sources/generifs_basic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gene_rif.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Think which data you will need\n",
    "Carrying large amount of data throughout an analysis can become messsy. Further, fewer data means that you will need to worry about fewer unexpected surprises that come with the data.\n",
    "\n",
    "- Gene IDs are Entrez Gene IDs which are unique identifiers (and change between orthologs in distinct species). Yet they are not very human readable. Symbol and description instead contain human-readable gene names.\n",
    "- The GeneRIF data does not contain human-readable gene names. This is good since the same gene could have multiple names and as the same genes name could have historically referrred to distinct genes. Instead GeneRIF contains the unambiguous Entrez Gene IDs. \n",
    "- Since we want to later read all GeneRIF statements of a gene, we will need to keep the GeneRIF text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"green\">To select multiple columns you will need the square brackets twice. This is part of the language used to interact with pandas</font>:\n",
    "- The outer ones signal to human_gene_info that something should be selected\n",
    "- The inner ones list all columns that should be chosen.\n",
    "\n",
    "<font color=\"green\">To select columns by their name, you must use the ' sign.</font>\n",
    "- The ' sign will signal to pandas that GeneID, Symbol, and description must be understood as text.\n",
    "- If the ' sign was absent, GeneID would be interpreted as a placeholder, whose name could vary independent of its content (similar to human_gene_info). Since we have not defined such a variable, you would hence encounter an error if skipping the ' sign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_human_gene_info = human_gene_info[\n",
    "    ['GeneID', 'Symbol', 'description']\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_human_gene_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_gene_rif = gene_rif[\n",
    "    ['Gene ID', 'GeneRIF text']\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_gene_rif.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have two tables, stored in the variables cleaned_human_gene_info, and cleaned_gene_rif, which we could combine to create a table containing both columns together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"green\"><b>Attention: the name of the column referring to the gene names is different. </b> While a human will immediately noteice the similarity between GeneID and Gene ID, computers often don't like ambiguity (or worse: yield misleading results if it would affect subsequent analysis).</font> \n",
    "\n",
    "We will hence ensure that the name is consistent. We will use the .rename functions which uses the {} brackets to convey information about the former and new name. Presently it is best to think of those brackets as parts of the language used for pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_gene_rif = cleaned_gene_rif.rename(\n",
    "    columns={'Gene ID': 'GeneID'}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_gene_rif.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Avoid duplicates\n",
    "The original datasets contained additional columns. Sometimes this these could contain additional implicit information. For instance two distinct publications could yield the same statement on an individual gene. In that case the same statement would be duplicated. Whether or not avoid such duplicates, will depend upon the scope of the analysis, and there often won't be a \"right\" or \"wrong\" for such situations. For the current tutorial, we are not interested, whether or not a statement occurs in multiple paper, so we .drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_gene_rif = cleaned_gene_rif.drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_human_gene_info = cleaned_human_gene_info.drop_duplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now merge the datasets\n",
    "To merge datasets by their shared column, use pandas' merge function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_statements = pd.merge(cleaned_human_gene_info, cleaned_gene_rif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_statements.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations again. Now we have a table containing human readable gene names and the GeneRIF statements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Genes with the most GeneRIF statements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the gene occuring in the most statements, .count_values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# attention here only level of square brackets is necessary\n",
    "# btw, # signals a comment which should not be executed\n",
    "human_statements['Symbol'].value_counts()        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not really surprising, no, TP53 is the gene with the most statements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's perhaps visually explore the above table. We see that many genes that occur in GeneRIF only have one or two enries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np     # for mathematics\n",
    "import seaborn as sns    # for visualization\n",
    "\n",
    "sns.distplot(     # draws a histogram\n",
    "    np.log10(       # log10 transforms data\n",
    "        human_statements['Symbol'].value_counts()   # counts occurences of every 'Symbol'\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write a function to get all gene RIFs for your favorite gene"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_favorite_gene = 'HSP90AA1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The first square brackets are used indicate the selction of data\n",
    "# The second square brackets create a mask for all rows where 'Symbol' matches\n",
    "#    the content of the variable my_favorite_gene. Note that double equation\n",
    "#    marks are necessary\n",
    "summary_for_my_favorite_gene = human_statements[\n",
    "    human_statements['Symbol']==my_favorite_gene\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_for_my_favorite_gene.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# To display the GeneRIF text completely, you will need to write a loop \n",
    "#   which will process individual entries within the column GeneRIF text;\n",
    "#   note that loop will execute the commands which are written with below\n",
    "#   it, if the commands are indentated (using tab or space) on beginning of line\n",
    "\n",
    "for individual_gene_rif in summary_for_my_favorite_gene['GeneRIF text']:\n",
    "    print(individual_gene_rif)   # print to notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can make your own functions by using def. Can you decipher the following logic?\n",
    "def print_gene_rif_of_my_favorite_gene(my_favorite_gene):\n",
    "    summary_for_my_favorite_gene = human_statements[\n",
    "        human_statements['Symbol']==my_favorite_gene\n",
    "    ]\n",
    "    \n",
    "    print('------------------------------------------')\n",
    "    print(my_favorite_gene, 'contains', summary_for_my_favorite_gene.shape[0], 'Gene RIF statements:')\n",
    "    \n",
    "    for individual_gene_rif in summary_for_my_favorite_gene['GeneRIF text']:\n",
    "        print(individual_gene_rif)   # print to notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_gene_rif_of_my_favorite_gene('AKT1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print_gene_rif_of_my_favorite_gene('AKT2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print_gene_rif_of_my_favorite_gene('AKT3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Suggested work prior the course on November 26th\n",
    "- Choose Mus musculus or Drosophila melanogaster or Caenorhabditis elegans and identify the most studied genes.\n",
    "- Do the same for Sus scrofa or Bos taurus or Oryza sativa. Does looking at their GeneRIFs inform about different or similar biology?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}