RepoLibrarian.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"font-size:250%\">Curating Repositories</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook was initially designed as active component to the [repoLibrarian](repoLibrarian.py) module (which in turn resulted from the needs of the [historical](RepoAnalysis_Historical.ipynb) part of the [RepoAnalysis](RepoAnalysis.ipynb) notebook). As many of the management queries and optimization measure were done on the fly in those notebooks, this notebook only acts as usage example (and test ground) for the [repoLibrarian](repoLibrarian.py) module and showcases some additional optimization measures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to use the repoLibrarian module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%aimport repoLibrarian"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where are repos currently saved?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/mnt/brick/crm20/repos/'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repoLibrarian.getReposFolder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use some local folder instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./repos/'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repoLibrarian.setReposFolder('./repos/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which repos are already saved there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['MarioLizana/RadioControlSED.git',\n",
       " 'json-iterator/java.git',\n",
       " 'craigslist206/huffman.git',\n",
       " 'pleonex/ChatRMI.git',\n",
       " 'pleonex/CocoKiller.git',\n",
       " 'pleonex/NiKate-Origins.git',\n",
       " 'pleonex/locaviewer.git',\n",
       " 'miken22/304-Project.git',\n",
       " 'NeebalLearningPvtLtd/InventoryManagementSystem.git',\n",
       " 'alibaba/arthas.git',\n",
       " 'zzjove/MOMA.git',\n",
       " 'tainarareis/Urutau.git',\n",
       " 'italopaiva/EuVou.git',\n",
       " 'ShutUpPaulo/TecProg_2016-01.git',\n",
       " 'andrevctr12/PAA_HUFFMAN.git',\n",
       " 'Elena-Zhao/Weibao.git',\n",
       " 'Elena-Zhao/MOMA.git',\n",
       " 'Elena-Zhao/Database-Auto-troubleshooting.git',\n",
       " 'Elena-Zhao/Guimi.git',\n",
       " 'Elena-Zhao/Mini-Chatter.git',\n",
       " 'alstonlo/Bongo-Cat-Attacc.git',\n",
       " 'dataspy/surprise-theory.git',\n",
       " 'ieeeugrsb/ieeextreme8.git',\n",
       " 'bptlab/Unicorn.git',\n",
       " 'bptlab/correlation-analysis.git',\n",
       " 'bptlab/scylla.git',\n",
       " 'bptlab/cepta.git',\n",
       " 'allantsai123/COSC310project.git',\n",
       " 'JTReed/Porygon2.git',\n",
       " 'JTReed/Porygon.git',\n",
       " 'shengnwen/WeiBaoSSE.git',\n",
       " 'lucasBrilhante/campus-party-mobile.git',\n",
       " 'lucasBrilhante/das-framework-teste.git',\n",
       " 'mairdl/jabref.git',\n",
       " 'tyler124/Monitordroid.git',\n",
       " 'YashArora0606/SignInSystem.git',\n",
       " 'GourdErwa/jasper_sample.git',\n",
       " 'igorribeiroduarte/formulasTesouroDireto.git']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(repoLibrarian.knownRepos())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about certain repos?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print(repoLibrarian.hasRepo('bptlab', 'scylla'))\n",
    "print(repoLibrarian.hasRepo('bptlab', 'fcm2cpn'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But I want that certain repo!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloned repo \"bptlab/fcm2cpn\"\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "user = 'bptlab'\n",
    "project = 'fcm2cpn'\n",
    "repoLibrarian.downloadRepo(user, project)\n",
    "print(repoLibrarian.hasRepo('bptlab', 'fcm2cpn'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If I don't want it anymore ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted repo \"bptlab/fcm2cpn\"\n"
     ]
    }
   ],
   "source": [
    "repoLibrarian.deleteRepo('bptlab', 'fcm2cpn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if don't want want to check if the repo exists and just want a handle?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloned repo \"bptlab/fcm2cpn\"\n",
      "<git.repo.base.Repo '/mnt/brick/home/lbein/jupyterNotebook/repos/bptlab/fcm2cpn.git'>\n",
      "<git.repo.base.Repo '/mnt/brick/home/lbein/jupyterNotebook/repos/bptlab/scylla.git'>\n"
     ]
    }
   ],
   "source": [
    "print(repoLibrarian.getRepo('bptlab', 'fcm2cpn'))\n",
    "print(repoLibrarian.getRepo('bptlab', 'scylla'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section includes some of the optimization measures that have been applied over time to the module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choose the gitPython database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gitpython provides two git object databases: GitDB and GitCmdObjectDb. According to [GitPython documentation](https://gitpython.readthedocs.io/en/stable/tutorial.html#object-databases) and also personal tests, GitDB is \"2 to 5 times slower when extracting large quantities small of objects from densely packed repositories\". As this is exactly what we want to do, GitCmdObjectDb is chosen (as opposed to the default GitDB)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Faster way to check if repo is a java repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way of checking if a repo is java by iterating over all files of the head commit is quite expensive for some repositories. <br>\n",
    "`git ls-tree` can be used to list all files for the head commit and seems promising:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./repos/'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repoLibrarian.setReposFolder('./repos/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "repo = repoLibrarian.getRepo('alibaba', 'arthas')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The approach is to list all files currently in HEAD and check if any of them ends with `.java`. This way no gitPython wrappers need to be created or traversed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(filter(lambda x: x.endswith('.java'), repo.git.ls_tree('--full-tree', '--name-only', '-r', 'HEAD').split('\\n')), None) != None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "155 ms ± 9.07 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit \n",
    "commit = list(repo.iter_commits())[0]\n",
    "any(repoLibrarian.isJavaFile(obj) for obj in commit.tree.traverse())\n",
    "repoLibrarian.isJavaRepo('alibaba', 'arthas')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52.3 ms ± 189 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit  next(filter(lambda x: x.endswith('.java'), repo.git.ls_tree('--full-tree', '--name-only', '-r', 'HEAD').split('\\n')), None) != None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Timeit implies that the time is approx half as long or even shorter, so the optimization is realized."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}