Text+mining+on+Non-Commercial+Use+Collection+of+OA+Subset+of+PubMed+.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text mining on Non-Commercial Use Collection of OA Subset of PubMed "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project consists of two parts: 1- Downloading the Non-Commercial Use Collection of OA files and 2- Running the code\n",
    "    \n",
    "1-Downloading the Non-Commercial Use Collection of OA files:\n",
    "    \n",
    "First, using FTP client software (Like Filezila) we get access to PMC OA Subset bulk XML/txt article package from the \"oa_bulk\" directory by entering ftp://ftp.ncbi.nlm.nih.gov/pub/pmc as host address, no password needed as it is an open access directory.\n",
    "After connecting, in the directory section, we select “Oa_bulk”; inside “oa_bulk” there are two sets of files, TXT and XML, from which we select XML ones which include:\n",
    "\n",
    "non_comm_use.A-B.xml.tar.gz\n",
    "\n",
    "non_comm_use.C-H.xml.tar.gz\n",
    "\n",
    "non_comm_use.I-N.xml.tar.gz\n",
    "\n",
    "non_comm_use.O-Z.xml.tar.gz\n",
    "\n",
       "comm_use.A-B.xml.tar.gz\n",
    "\n",
    "comm_use.C-H.xml.tar.gz\n",
    "\n",
    "comm_use.I-N.xml.tar.gz\n",
    "\n",
    "comm_use.O-Z.xml.tar.gz\n",
    "\n",
    "2-Running the code\n",
    "\n",
    "After downloading the files we extract them in the folder D:\\uncomm1\\. It is possible to change the directory to the desired one.\n",
    "For running the code correctly, we need to install python 3.6 with the following dependencies: \n",
    "Spacy, HTMLParser, codecs, fnmatch, GeoText and nltk\n",
    "\n",
    "The code will create a TSV (Tab separated file), Extract.tsv in the path which is indicated in the code, in our case it is set to: \"D:\\Pubmed\\extracted.tsv”.  It is possible to change it to any other desired path. \n",
    "\n",
    "After running the code successfully, the output file is \"Extract.tsv\" which is a table of records which contains the following fields:\n",
    "File_ID, Journal ID, issn No., Publisher Name, Article Title, Abstract, Date, Countibuter(s), affiliation(s), Countries Involved\n",
    "\n",
    "Each record of the file contains information extracted from one Article. Hence, the number of records is equal to the number of articles in the corpus.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import xml.etree.ElementTree as ET\n",
    "import spacy\n",
    "import os, sys,re\n",
    "from html.parser import HTMLParser\n",
    "import codecs\n",
    "import fnmatch\n",
    "nlp = spacy.load('en')\n",
    "from geotext import GeoText\n",
    "import nltk\n",
    "\n",
    "# List of country names + iso codes \n",
    "\n",
    "countries = [\n",
    "    (\"Afghanistan\", \"AF\", \"AFG\", \"004\", \"Afghanistan\"),\n",
    "    (\"Åland Islands\", \"AX\", \"ALA\", \"248\", \"Åland Islands\"),\n",
    "    (\"Albania\", \"AL\", \"ALB\", \"008\", \"Albania\"),\n",
    "    (\"Algeria\", \"DZ\", \"DZA\", \"012\", \"Algeria\"),\n",
    "    (\"American Samoa\", \"AS\", \"ASM\", \"016\", \"American Samoa\"),\n",
    "    (\"Andorra\", \"AD\", \"AND\", \"020\", \"Andorra\"),\n",
    "    (\"Angola\", \"AO\", \"AGO\", \"024\", \"Angola\"),\n",
    "    (\"Anguilla\", \"AI\", \"AIA\", \"660\", \"Anguilla\"),\n",
    "    (\"Antarctica\", \"AQ\", \"ATA\", \"010\", \"Antarctica\"),\n",
    "    (\"Antigua and Barbuda\", \"AG\", \"ATG\", \"028\",\n",
    "            \"Antigua and Barbuda\"),\n",
    "    (\"Argentina\", \"AR\", \"ARG\", \"032\", \"Argentina\"),\n",
    "    (\"Armenia\", \"AM\", \"ARM\", \"051\", \"Armenia\"),\n",
    "    (\"Aruba\", \"AW\", \"ABW\", \"533\", \"Aruba\"),\n",
    "    (\"Australia\", \"A\", \"AUS\", \"036\", \"Australia\"),\n",
    "    (\"Austria\", \"AT\", \"AUT\", \"040\", \"Austria\"),\n",
    "    (\"Azerbaijan\", \"AZ\", \"AZE\", \"031\", \"Azerbaijan\"),\n",
    "    (\"Bahamas\", \"BS\", \"BHS\", \"044\", \"Bahamas\"),\n",
    "    (\"Bahrain\", \"BH\", \"BHR\", \"048\", \"Bahrain\"),\n",
    "    (\"Bangladesh\", \"BD\", \"BGD\", \"050\", \"Bangladesh\"),\n",
    "    (\"Barbados\", \"BB\", \"BRB\", \"052\", \"Barbados\"),\n",
    "    (\"Belarus\", \"BY\", \"BLR\", \"112\", \"Belarus\"),\n",
    "    (\"Belgium\", \"BE\", \"BEL\", \"056\", \"Belgium\"),\n",
    "    (\"Belize\", \"BZ\", \"BLZ\", \"084\", \"Belize\"),\n",
    "    (\"Benin\", \"BJ\", \"BEN\", \"204\", \"Benin\"),\n",
    "    (\"Bermuda\", \"BM\", \"BM\", \"060\", \"Bermuda\"),\n",
    "    (\"Bhutan\", \"BT\", \"BTN\", \"064\", \"Bhutan\"),\n",
    "    (\"Bolivia, Plurinational State of\", \"BO\", \"BOL\", \"068\",\n",
    "            \"Bolivia, Plurinational State of\"),\n",
    "    (\"Bolivia\", \"BO\", \"BOL\", \"068\",\n",
    "            \"Bolivia, Plurinational State of\"),\n",
    "    (\"Bonaire, Sint Eustatius and Saba\", \"BQ\", \"BES\", \"535\",\n",
    "            \"Bonaire, Sint Eustatius and Saba\"),\n",
    "    (\"Bosnia and Herzegovina\", \"BA\", \"BIH\", \"070\",\n",
    "            \"Bosnia and Herzegovina\"),\n",
    "    (\"Bosnia\", \"BA\", \"BIH\", \"070\",\n",
    "            \"Bosnia and Herzegovina\"),\n",
    "    (\"Botswana\", \"BW\", \"BWA\", \"072\", \"Botswana\"),\n",
    "    (\"Bouvet Island\", \"BV\", \"BVT\", \"074\", \"Bouvet Island\"),\n",
    "    (\"Brazil\", \"BR\", \"BRA\", \"076\", \"Brazil\"),\n",
    "    (\"British Indian Ocean Territory\", \"IO\", \"IOT\", \"086\",\n",
    "            \"British Indian Ocean Territory\"),\n",
    "    (\"Brunei Darussalam\", \"BN\", \"BRN\", \"096\",\n",
    "            \"Brunei Darussalam\"),\n",
    "    (\"Bulgaria\", \"BG\", \"BGR\", \"100\", \"Bulgaria\"),\n",
    "    (\"Burkina Faso\", \"BF\", \"BFA\", \"854\", \"Burkina Faso\"),\n",
    "    (\"Burundi\", \"BI\", \"BDI\", \"108\", \"Burundi\"),\n",
    "    (\"Cambodia\", \"KH\", \"KHM\", \"116\", \"Cambodia\"),\n",
    "    (\"Cameroon\", \"CM\", \"CMR\", \"120\", \"Cameroon\"),\n",
    "    (\"Canada\", \"CA\", \"CAN\", \"124\", \"Canada\"),\n",
    "    (\"Cabo Verde\", \"CV\", \"CPV\", \"132\", \"Cabo Verde\"),\n",
    "    (\"Cayman Islands\", \"KY\", \"CYM\", \"136\", \"Cayman Islands\"),\n",
    "    (\"Central African Republic\", \"CF\", \"CAF\", \"140\",\n",
    "            \"Central African Republic\"),\n",
    "    (\"Central Africa\", \"CF\", \"CAF\", \"140\",\n",
    "            \"Central African Republic\"),\n",
    "    (\"Chad\", \"TD\", \"TCD\", \"148\", \"Chad\"),\n",
    "    (\"Chile\", \"CL\", \"CHL\", \"152\", \"Chile\"),\n",
    "    (\"China\", \"CN\", \"CHN\", \"156\", \"China\"),\n",
    "    (\"Christmas Island\", \"CX\", \"CXR\", \"162\", \"Christmas Island\"),\n",
    "    (\"Cocos (Keeling) Islands\", \"CC\", \"CCK\", \"166\",\n",
    "            \"Cocos (Keeling) Islands\"),\n",
    "    (\"Cocos\", \"CC\", \"CCK\", \"166\",\n",
    "            \"Cocos (Keeling) Islands\"),\n",
    "    (\"Colombia\", \"CO\", \"COL\", \"170\", \"Colombia\"),\n",
    "    (\"Comoros\", \"KM\", \"COM\", \"174\", \"Comoros\"),\n",
    "    (\"Congo\", \"CG\", \"COG\", \"178\", \"Congo\"),\n",
    "    (\"Congo, Democratic Republic of the\", \"CD\", \"COD\", \"180\",\n",
    "            \"Congo, Democratic Republic of the\"),\n",
    "    (\"Cook Islands\", \"CK\", \"COK\", \"184\", \"Cook Islands\"),\n",
    "    (\"Costa Rica\", \"CR\", \"CRI\", \"188\", \"Costa Rica\"),\n",
    "    (\"Côte d'Ivoire\", \"CI\", \"CIV\", \"384\", \"Côte d'Ivoire\"),\n",
    "    (\"Croatia\", \"HR\", \"HRV\", \"191\", \"Croatia\"),\n",
    "    (\"Cuba\", \"C\", \"CUB\", \"192\", \"Cuba\"),\n",
    "    (\"Curaçao\", \"CW\", \"CUW\", \"531\", \"Curaçao\"),\n",
    "    (\"Cyprus\", \"CY\", \"CYP\", \"196\", \"Cyprus\"),\n",
    "    (\"Czechia\", \"CZ\", \"CZE\", \"203\", \"Czechia\"),\n",
    "    (\"Denmark\", \"DK\", \"DNK\", \"208\", \"Denmark\"),\n",
    "    (\"Djibouti\", \"DJ\", \"DJI\", \"262\", \"Djibouti\"),\n",
    "    (\"Dominica\", \"DM\", \"DMA\", \"212\", \"Dominica\"),\n",
    "    (\"Dominican Republic\", \"DO\", \"DOM\", \"214\", \"Dominican Republic\"),\n",
    "    (\"Dominican\", \"DO\", \"DOM\", \"214\", \"Dominican Republic\"),\n",
    "    (\"Ecuador\", \"EC\", \"EC\", \"218\", \"Ecuador\"),\n",
    "    (\"Egypt\", \"EG\", \"EGY\", \"818\", \"Egypt\"),\n",
    "    (\"El Salvador\", \"SV\", \"SLV\", \"222\", \"El Salvador\"),\n",
    "    (\"Equatorial Guinea\", \"GQ\", \"GNQ\", \"226\", \"Equatorial Guinea\"),\n",
    "    (\"Eritrea\", \"ER\", \"ERI\", \"232\", \"Eritrea\"),\n",
    "    (\"Estonia\", \"EE\", \"EST\", \"233\", \"Estonia\"),\n",
    "    (\"Ethiopia\", \"ET\", \"ETH\", \"231\", \"Ethiopia\"),\n",
    "    (\"Falkland Islands (Malvinas)\", \"FK\", \"FLK\", \"238\",\n",
    "            \"Falkland Islands (Malvinas)\"),\n",
    "    (\"Faroe Islands\", \"FO\", \"FRO\", \"234\", \"Faroe Islands\"),\n",
    "    (\"Fiji\", \"FJ\", \"FJI\", \"242\", \"Fiji\"),\n",
    "    (\"Finland\", \"FI\", \"FIN\", \"246\", \"Finland\"),\n",
    "    (\"France\", \"FR\", \"FRA\", \"250\", \"France\"),\n",
    "    (\"French Guiana\", \"GF\", \"GUF\", \"254\", \"French Guiana\"),\n",
    "    (\"French Polynesia\", \"PF\", \"PYF\", \"258\", \"French Polynesia\"),\n",
    "    (\"French Southern Territories\", \"TF\", \"ATF\", \"260\",\n",
    "            \"French Southern Territories\"),\n",
    "    (\"Gabon\", \"GA\", \"GAB\", \"266\", \"Gabon\"),\n",
    "    (\"Gambia\", \"GM\", \"GMB\", \"270\", \"Gambia\"),\n",
    "    (\"Georgian\", \"GE\", \"GEO\", \"268\", \"Georgia\"),\n",
    "    (\"Germany\", \"DE\", \"DEU\", \"276\", \"Germany\"),\n",
    "    (\"Ghana\", \"GH\", \"GHA\", \"288\", \"Ghana\"),\n",
    "    (\"Gibraltar\", \"GI\", \"GIB\", \"292\", \"Gibraltar\"),\n",
    "    (\"Greece\", \"GR\", \"GRC\", \"300\", \"Greece\"),\n",
    "    (\"Greenland\", \"GL\", \"GRL\", \"304\", \"Greenland\"),\n",
    "    (\"Grenada\", \"GD\", \"GRD\", \"308\", \"Grenada\"),\n",
    "    (\"Guadeloupe\", \"GP\", \"GLP\", \"312\", \"Guadeloupe\"),\n",
    "    (\"Guam\", \"G\", \"GUM\", \"316\", \"Guam\"),\n",
    "    (\"Guatemala\", \"GT\", \"GTM\", \"320\", \"Guatemala\"),\n",
    "    (\"Guernsey\", \"GG\", \"GGY\", \"831\", \"Guernsey\"),\n",
    "    (\"Guinea\", \"GN\", \"GIN\", \"324\", \"Guinea\"),\n",
    "    (\"Guinea-Bissa\", \"GW\", \"GNB\", \"624\", \"Guinea-Bissa\"),\n",
    "    (\"Guyana\", \"GY\", \"GUY\", \"328\", \"Guyana\"),\n",
    "    (\"Haiti\", \"HT\", \"HTI\", \"332\", \"Haiti\"),\n",
    "    (\"Heard Island and McDonald Islands\", \"HM\", \"HMD\", \"334\",\n",
    "            \"Heard Island and McDonald Islands\"),\n",
    "    (\"Holy See\", \"VA\", \"VAT\", \"336\", \"Holy See\"),\n",
    "    (\"Honduras\", \"HN\", \"HND\", \"340\", \"Honduras\"),\n",
    "    (\"Hong Kong\", \"HK\", \"HKG\", \"344\", \"Hong Kong\"),\n",
    "    (\"Hungary\", \"H\", \"HUN\", \"348\", \"Hungary\"),\n",
    "    (\"Iceland\", \"IS\", \"ISL\", \"352\", \"Iceland\"),\n",
    "    (\"India\", \"IN\", \"IND\", \"356\", \"India\"),\n",
    "    (\"Indonesia\", \"ID\", \"IDN\", \"360\", \"Indonesia\"),\n",
    "    (\"Iran, Islamic Republic of\", \"IR\", \"IRN\", \"364\",\n",
    "            \"Iran, Islamic Republic of\"),\n",
    "    (\"Iran\", \"IR\", \"IRN\", \"364\",\n",
    "            \"Iran, Islamic Republic of\"),\n",
    "    (\"Iraq\", \"IQ\", \"IRQ\", \"368\", \"Iraq\"),\n",
    "    (\"Ireland\", \"IE\", \"IRL\", \"372\", \"Ireland\"),\n",
    "    (\"Isle of Man\", \"IM\", \"IMN\", \"833\", \"Isle of Man\"),\n",
    "    (\"Israel\", \"IL\", \"ISR\", \"376\", \"Israel\"),\n",
    "    (\"Italy\", \"IT\", \"ITA\", \"380\", \"Italy\"),\n",
    "    (\"Jamaica\", \"JM\", \"JAM\", \"388\", \"Jamaica\"),\n",
    "    (\"Japan\", \"JP\", \"JPN\", \"392\", \"Japan\"),\n",
    "    (\"Jersey\", \"JE\", \"JEY\", \"832\", \"Jersey\"),\n",
    "    (\"Jordan\", \"JO\", \"JOR\", \"400\", \"Jordan\"),\n",
    "    (\"Kazakhstan\", \"KZ\", \"KAZ\", \"398\", \"Kazakhstan\"),\n",
    "    (\"Kenya\", \"KE\", \"KEN\", \"404\", \"Kenya\"),\n",
    "    (\"Kiribati\", \"KI\", \"KIR\", \"296\", \"Kiribati\"),\n",
    "    (\"Korea, Democratic People's Republic of\", \"KP\", \"PRK\", \"408\",\n",
    "            \"Korea, Democratic People's Republic of\"),\n",
    "    (\"Korea, Republic of\", \"KR\", \"KOR\", \"410\", \"Korea, Republic of\"),\n",
    "    (\"Korea\", \"KR\", \"KOR\", \"410\", \"Korea, Republic of\"),\n",
    "    (\"Kosovo\", \"XK\", \"XKX\", \"000\", \"Kosovo\"),\n",
    "    (\"Kuwait\", \"KW\", \"KWT\", \"414\", \"Kuwait\"),\n",
    "    (\"Kyrgyzstan\", \"KG\", \"KGZ\", \"417\", \"Kyrgyzstan\"),\n",
    "    (\"Lao People's Democratic Republic\", \"LA\", \"LAO\", \"418\",\n",
    "            \"Lao People's Democratic Republic\"),\n",
    "    (\"Latvia\", \"LV\", \"LVA\", \"428\", \"Latvia\"),\n",
    "    (\"Lebanon\", \"LB\", \"LBN\", \"422\", \"Lebanon\"),\n",
    "    (\"Lesotho\", \"LS\", \"LSO\", \"426\", \"Lesotho\"),\n",
    "    (\"Liberia\", \"LR\", \"LBR\", \"430\", \"Liberia\"),\n",
    "    (\"Libya\", \"LY\", \"LBY\", \"434\", \"Libya\"),\n",
    "    (\"Liechtenstein\", \"LI\", \"LIE\", \"438\", \"Liechtenstein\"),\n",
    "    (\"Lithuania\", \"LT\", \"LT\", \"440\", \"Lithuania\"),\n",
    "    (\"Luxembourg\", \"L\", \"LUX\", \"442\", \"Luxembourg\"),\n",
    "    (\"Macao\", \"MO\", \"MAC\", \"446\", \"Macao\"),\n",
    "    (\"Macedonia, the former Yugoslav Republic of\", \"MK\", \"MKD\", \"807\",\n",
    "            \"Macedonia, the former Yugoslav Republic of\"),\n",
    "    (\"Macedonia\", \"MK\", \"MKD\", \"807\",\n",
    "            \"Macedonia, the former Yugoslav Republic of\"),\n",
    "    (\"Madagascar\", \"MG\", \"MDG\", \"450\", \"Madagascar\"),\n",
    "    (\"Malawi\", \"MW\", \"MWI\", \"454\", \"Malawi\"),\n",
    "    (\"Malaysia\", \"MY\", \"MYS\", \"458\", \"Malaysia\"),\n",
    "    (\"Maldives\", \"MV\", \"MDV\", \"462\", \"Maldives\"),\n",
    " \n",
    "    (\"Malta\", \"MT\", \"MLT\", \"470\", \"Malta\"),\n",
    "    (\"Marshall Islands\", \"MH\", \"MHL\", \"584\", \"Marshall Islands\"),\n",
    "    (\"Martinique\", \"MQ\", \"MTQ\", \"474\", \"Martinique\"),\n",
    "    (\"Mauritania\", \"MR\", \"MRT\", \"478\", \"Mauritania\"),\n",
    "    (\"Mauritius\", \"M\", \"MUS\", \"480\", \"Mauritius\"),\n",
    "    (\"Mayotte\", \"YT\", \"MYT\", \"175\", \"Mayotte\"),\n",
    "    (\"Mexico\", \"MX\", \"MEX\", \"484\", \"Mexico\"),\n",
    "    (\"Micronesia, Federated States of\", \"FM\", \"FSM\", \"583\",\n",
    "            \"Micronesia, Federated States of\"),\n",
    "    (\"Micronesia\", \"FM\", \"FSM\", \"583\",\n",
    "            \"Micronesia, Federated States of\"),\n",
    "    \n",
    "    (\"Moldova, Republic of\", \"MD\", \"MDA\", \"498\",\n",
    "            \"Moldova, Republic of\"),\n",
    "    \n",
    "    (\"Moldova\", \"MD\", \"MDA\", \"498\",\n",
    "            \"Moldova, Republic of\"),\n",
    "    (\"Monaco\", \"MC\", \"MCO\", \"492\", \"Monaco\"),\n",
    "    (\"Mongolia\", \"MN\", \"MNG\", \"496\", \"Mongolia\"),\n",
    "    (\"Montenegro\", \"ME\", \"MNE\", \"499\", \"Montenegro\"),\n",
    "    (\"Montserrat\", \"MS\", \"MSR\", \"500\", \"Montserrat\"),\n",
    "    (\"Morocco\", \"MA\", \"MAR\", \"504\", \"Morocco\"),\n",
    "    (\"Mozambique\", \"MZ\", \"MOZ\", \"508\", \"Mozambique\"),\n",
    "    (\"Myanmar\", \"MM\", \"MMR\", \"104\", \"Myanmar\"),\n",
    "    (\"Namibia\", \"NA\", \"NAM\", \"516\", \"Namibia\"),\n",
    "    (\"Naur\", \"NR\", \"NR\", \"520\", \"Naur\"),\n",
    "    (\"Nepal\", \"NP\", \"NPL\", \"524\", \"Nepal\"),\n",
    "    (\"Netherlands\", \"NL\", \"NLD\", \"528\", \"Netherlands\"),\n",
    "    (\"New Caledonia\", \"NC\", \"NCL\", \"540\", \"New Caledonia\"),\n",
    "    (\"New Zealand\", \"NZ\", \"NZL\", \"554\", \"New Zealand\"),\n",
    "    (\"Nicaragua\", \"NI\", \"NIC\", \"558\", \"Nicaragua\"),\n",
    "    (\"Niger\", \"NE\", \"NER\", \"562\", \"Niger\"),\n",
    "    (\"Nigeria\", \"NG\", \"NGA\", \"566\", \"Nigeria\"),\n",
    "    (\"Niue\", \"N\", \"NIU\", \"570\", \"Niue\"),\n",
    "    (\"Norfolk Island\", \"NF\", \"NFK\", \"574\", \"Norfolk Island\"),\n",
    "    (\"Northern Mariana Islands\", \"MP\", \"MNP\", \"580\",\n",
    "            \"Northern Mariana Islands\"),\n",
    "    (\"Norway\", \"NO\", \"NOR\", \"578\", \"Norway\"),\n",
    "    (\"Oman\", \"OM\", \"OMN\", \"512\", \"Oman\"),\n",
    "    (\"Pakistan\", \"PK\", \"PAK\", \"586\", \"Pakistan\"),\n",
    "    (\"Pala\", \"PW\", \"PLW\", \"585\", \"Pala\"),\n",
    "    (\"Palestine, State of\", \"PS\", \"PSE\", \"275\",\n",
    "            \"Palestine\"),\n",
    "    (\"Panama\", \"PA\", \"PAN\", \"591\", \"Panama\"),\n",
    "    (\"Papua New Guinea\", \"PG\", \"PNG\", \"598\",\n",
    "            \"Papua New Guinea\"),\n",
    "    (\"Paraguay\", \"PY\", \"PRY\", \"600\", \"Paraguay\"),\n",
    "    (\"Per\", \"PE\", \"PER\", \"604\", \"Per\"),\n",
    "    (\"Philippines\", \"PH\", \"PHL\", \"608\", \"Philippines\"),\n",
    "    (\"Pitcairn\", \"PN\", \"PCN\", \"612\", \"Pitcairn\"),\n",
    "    (\"Poland\", \"PL\", \"POL\", \"616\", \"Poland\"),\n",
    "    (\"Portugal\", \"PT\", \"PRT\", \"620\", \"Portugal\"),\n",
    "    (\"Puerto Rico\", \"PR\", \"PRI\", \"630\", \"Puerto Rico\"),\n",
    "    (\"Qatar\", \"QA\", \"QAT\", \"634\", \"Qatar\"),\n",
    "    (\"Réunion\", \"RE\", \"RE\", \"638\", \"Réunion\"),\n",
    "    (\"Romania\", \"RO\", \"ROU\", \"642\", \"Romania\"),\n",
    "    (\"Russian Federation\", \"R\", \"RUS\", \"643\",\n",
    "            \"Russian Federation\"),\n",
    "    (\"Russia\", \"R\", \"RUS\", \"643\",\n",
    "            \"Russian Federation\"),\n",
    "    (\"Rwanda\", \"RW\", \"RWA\", \"646\", \"Rwanda\"),\n",
    "    (\"Saint Barthélemy\", \"BL\", \"BLM\", \"652\",\n",
    "            \"Saint Barthélemy\"),\n",
    "    (\"Saint Helena, Ascension and Tristan da Cunha\",\n",
    "            \"SH\", \"SHN\", \"654\",\n",
    "            \"Saint Helena, Ascension and Tristan da Cunha\"),\n",
    "    (\"Saint Kitts and Nevis\", \"KN\", \"KNA\", \"659\",\n",
    "            \"Saint Kitts and Nevis\"),\n",
    "    (\"Saint Lucia\", \"LC\", \"LCA\", \"662\", \"Saint Lucia\"),\n",
    "    (\"Saint Martin (French part)\", \"MF\", \"MAF\", \"663\",\n",
    "            \"Saint Martin (French part)\"),\n",
    "    (\"Saint Pierre and Miquelon\", \"PM\", \"SPM\", \"666\",\n",
    "            \"Saint Pierre and Miquelon\"),\n",
    "    (\"Saint Vincent and the Grenadines\", \"VC\", \"VCT\", \"670\",\n",
    "            \"Saint Vincent and the Grenadines\"),\n",
    "    (\"Samoa\", \"WS\", \"WSM\", \"882\", \"Samoa\"),\n",
    "    (\"San Marino\", \"SM\", \"SMR\", \"674\", \"San Marino\"),\n",
    "    (\"Sao Tome and Principe\", \"ST\", \"STP\", \"678\",\n",
    "            \"Sao Tome and Principe\"),\n",
    "    (\"Saudi Arabia\", \"SA\", \"SAU\", \"682\", \"Saudi Arabia\"),\n",
    "    (\"Senegal\", \"SN\", \"SEN\", \"686\", \"Senegal\"),\n",
    "    (\"Serbia\", \"RS\", \"SRB\", \"688\", \"Serbia\"),\n",
    "    (\"Seychelles\", \"SC\", \"SYC\", \"690\", \"Seychelles\"),\n",
    "    (\"Sierra Leone\", \"SL\", \"SLE\", \"694\", \"Sierra Leone\"),\n",
    "    (\"Singapore\", \"SG\", \"SGP\", \"702\", \"Singapore\"),\n",
    "    (\"Sint Maarten (Dutch part)\", \"SX\", \"SXM\", \"534\",\n",
    "            \"Sint Maarten (Dutch part)\"),\n",
    "    (\"Slovakia\", \"SK\", \"SVK\", \"703\", \"Slovakia\"),\n",
    "    (\"Slovak Republic\", \"SK\", \"SVK\", \"703\", \"Slovakia\"),\n",
    "    (\"Slovak\", \"SK\", \"SVK\", \"703\", \"Slovakia\"),\n",
    "    \n",
    "    (\"Slovenia\", \"SI\", \"SVN\", \"705\", \"Slovenia\"),\n",
    "    (\"Solomon Islands\", \"SB\", \"SLB\", \"090\", \"Solomon Islands\"),\n",
    "    (\"Somalia\", \"SO\", \"SOM\", \"706\", \"Somalia\"),\n",
    "    (\"South Africa\", \"ZA\", \"ZAF\", \"710\", \"South Africa\"),\n",
    "    (\"South Georgia and the South Sandwich Islands\",\n",
    "            \"GS\", \"SGS\", \"239\",\n",
    "            \"South Georgia and the South Sandwich Islands\",),\n",
    "    (\"South Sudan\", \"SS\", \"SSD\", \"728\", \"South Sudan\"),\n",
    "    (\"Spain\", \"ES\", \"ESP\", \"724\", \"Spain\"),\n",
    "    (\"Sri Lanka\", \"LK\", \"LKA\", \"144\", \"Sri Lanka\"),\n",
    "    (\"Sudan\", \"SD\", \"SDN\", \"729\", \"Sudan\"),\n",
    "    (\"Suriname\", \"SR\", \"SUR\", \"740\", \"Suriname\"),\n",
    "    (\"Svalbard and Jan Mayen\", \"SJ\", \"SJM\", \"744\",\n",
    "            \"Svalbard and Jan Mayen\"),\n",
    "    (\"Eswatini\", \"SZ\", \"SWZ\", \"748\", \"Eswatini\"),\n",
    "    (\"Sweden\", \"SE\", \"SWE\", \"752\", \"Sweden\"),\n",
    "    (\"Switzerland\", \"CH\", \"CHE\", \"756\", \"Switzerland\"),\n",
    "    (\"Syrian Arab Republic\", \"SY\", \"SYR\", \"760\",\n",
    "            \"Syrian Arab Republic\"),\n",
    "    (\"Syrian\", \"SY\", \"SYR\", \"760\",\n",
    "            \"Syrian Arab Republic\"),\n",
    "\n",
    "    (\"Taiwan, Province of China\", \"TW\", \"TWN\", \"158\",\n",
    "            \"Taiwan\"),\n",
    "    (\"Taiwan\", \"TW\", \"TWN\", \"158\",\n",
    "            \"Taiwan\"),\n",
    "    \n",
    "    (\"Tajikistan\", \"TJ\", \"TJK\", \"762\", \"Tajikistan\"),\n",
    "    (\"Tanzania, United Republic of\", \"TZ\", \"TZA\", \"834\",\n",
    "            \"Tanzania, United Republic of\"),\n",
    "    (\"Thailand\", \"TH\", \"THA\", \"764\", \"Thailand\"),\n",
    "    (\"Timor-Leste\", \"TL\", \"TLS\", \"626\", \"Timor-Leste\"),\n",
    "    (\"Togo\", \"TG\", \"TGO\", \"768\", \"Togo\"),\n",
    "    (\"Tokela\", \"TK\", \"TKL\", \"772\", \"Tokela\"),\n",
    "    (\"Tonga\", \"TO\", \"TON\", \"776\", \"Tonga\"),\n",
    "    (\"Trinidad and Tobago\", \"TT\", \"TTO\", \"780\",\n",
    "            \"Trinidad and Tobago\"),\n",
    "    (\"Tunisia\", \"TN\", \"TUN\", \"788\", \"Tunisia\"),\n",
    "    (\"Turkey\", \"TR\", \"TUR\", \"792\", \"Turkey\"),\n",
    "    (\"Turkmenistan\", \"TM\", \"TKM\", \"795\", \"Turkmenistan\"),\n",
    "    (\"Turks and Caicos Islands\", \"TC\", \"TCA\", \"796\",\n",
    "            \"Turks and Caicos Islands\"),\n",
    "    (\"Tuval\", \"TV\", \"TUV\", \"798\", \"Tuval\"),\n",
    "    (\"Uganda\", \"UG\", \"UGA\", \"800\", \"Uganda\"),\n",
    "    (\"Ukraine\", \"UA\", \"UKR\", \"804\", \"Ukraine\"),\n",
    "    (\"United Arab Emirates\", \"AE\", \"ARE\", \"784\",\n",
    "            \"United Arab Emirates\"),\n",
    "    (\"United Kingdom of Great Britain and Northern Ireland\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    (\"ENGLAND\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    (\"Scotland\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    (\"United Kingdom\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    (\"Wales\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    (\"Northern Ireland\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    (\"Great Britain\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    (\"UK\",\n",
    "            \"GB\", \"GBR\", \"826\",\n",
    "            \"United Kingdom of Great Britain and Northern Ireland\"),\n",
    "    \n",
    "    (\"United States of America\", \"US\", \"USA\", \"840\",\n",
    "            \"United States of America\"),\n",
    "    (\"United States\", \"US\", \"USA\", \"840\",\n",
    "            \"United States of America\"),\n",
    "    (\"USA\", \"US\", \"USA\", \"840\",\n",
    "            \"United States of America\"),\n",
    "    (\"United States Minor Outlying Islands\", \"UM\", \"UMI\", \"581\",\n",
    "            \"United States Minor Outlying Islands\"),\n",
    "    (\"Uruguay\", \"UY\", \"URY\", \"858\", \"Uruguay\"),\n",
    "    (\"Uzbekistan\", \"UZ\", \"UZB\", \"860\", \"Uzbekistan\"),\n",
    "    (\"Vanuat\", \"V\", \"VUT\", \"548\", \"Vanuat\"),\n",
    "    (\"Venezuela, Bolivarian Republic of\", \"VE\", \"VEN\", \"862\",\n",
    "            \"Venezuela, Bolivarian Republic of\"),\n",
    "    (\"Venezuela\", \"VE\", \"VEN\", \"862\",\n",
    "            \"Venezuela, Bolivarian Republic of\"),\n",
    "    (\"Viet Nam\", \"VN\", \"VNM\", \"704\", \"Viet Nam\"),\n",
    "    (\"Virgin Islands, British\", \"VG\", \"VGB\", \"092\",\n",
    "            \"Virgin Islands, British\"),\n",
    "    (\"Virgin Islands, U.S.\", \"VI\", \"VIR\", \"850\",\n",
    "            \"Virgin Islands, U.S.\"),\n",
    "    (\"Wallis and Futuna\", \"WF\", \"WLF\", \"876\", \"Wallis and Futuna\"),\n",
    "    (\"Western Sahara\", \"EH\", \"ESH\", \"732\", \"Western Sahara\"),\n",
    "    (\"Yemen\", \"YE\", \"YEM\", \"887\", \"Yemen\"),\n",
    "    (\"Zambia\", \"ZM\", \"ZMB\", \"894\", \"Zambia\"),\n",
    "    (\"Zimbabwe\", \"ZW\", \"ZWE\", \"716\", \"Zimbabwe\")]\n",
    "\n",
    "# U.S. State  List\n",
    "\n",
    "states=[\"Alabama\",\"Alaska\",\"Arizona\",\"Arkansas\",\"California\",\"Colorado\",\n",
    "  \"Connecticut\",\"Delaware\",\"Florida\",\"Georgia\",\"Hawaii\",\"Idaho\",\"Illinois\",\n",
    "  \"Indiana\",\"Iowa\",\"Kansas\",\"Kentucky\",\"Louisiana\",\"Maine\",\"Maryland\",\n",
    "  \"Massachusetts\",\"Michigan\",\"Minnesota\",\"Mississippi\",\"Missouri\",\"Montana\",\n",
    "  \"Nebraska\",\"Nevada\",\"New Hampshire\",\"New Jersey\",\"New Mexico\",\"New York\",\n",
    "  \"North Carolina\",\"North Dakota\",\"Ohio\",\"Oklahoma\",\"Oregon\",\"Pennsylvania\",\n",
    "  \"Rhode Island\",\"South Carolina\",\"South Dakota\",\"Tennessee\",\"Texas\",\"Utah\",\n",
    "  \"Vermont\",\"Virginia\",\"Washington\",\"West Virginia\",\"Wisconsin\",\"Wyoming\"]\n",
    "\n",
    "#U.S. State Postal Abbreviations List\n",
    "\n",
    "states1 = [\"AL\", \"AK\", \"AZ\", \"AR\", \"CA\", \"CO\", \"CT\", \"DC\", \"DE\", \"FL\", \"GA\", \n",
    "          \"HI\", \"ID\", \"IL\", \"IN\", \"IA\", \"KS\", \"KY\", \"LA\", \"ME\", \"MD\", \n",
    "          \"MA\", \"MI\", \"MN\", \"MS\", \"MO\", \"MT\", \"NE\", \"NV\", \"NH\", \"NJ\", \n",
    "          \"NM\", \"NY\", \"NC\", \"ND\", \"OH\", \"OK\", \"OR\", \"PA\", \"RI\", \"SC\", \n",
    "          \"SD\", \"TN\", \"TX\", \"UT\", \"VT\", \"VA\", \"WA\", \"WV\", \"WI\", \"WY\"]\n",
    "\n",
    "# Striping messy tags\n",
    "\n",
    "def strip_tags(html):\n",
    "    s =  html.replace(\"</italic>\",\" \")\n",
    "    s =  html.replace (\"<country>\", \" \")\n",
    "    s =  html.replace (\"</country>\", \" \")\n",
    "    s =  html.replace (\"country\", \" \")\n",
    "    s =  s.replace(\"<italic>\",\" \")\n",
    "    s =  s.replace(\"<sub>\",\" \")\n",
    "    s =  s.replace(\"</sub>\",\" \")\n",
    "    s =  s.replace(\"<p>\",\" \")\n",
    "    s =  s.replace(\"</p>\",\" \")\n",
    "    s =  s.replace(\"<sup>\",\" \")\n",
    "    s =  s.replace(\"</sup>\",\" \")\n",
    "    s =  s.replace(\"<title>\",\" \")\n",
    "    s =  s.replace(\"</title>\",\" \")\n",
    "    s =  s.replace(\"<bold>\",\" \")\n",
    "    s =  s.replace(\"</bold>\",\" \")\n",
    "    s =  s.replace(\"<article-title>\",\" \")\n",
    "    s =  s.replace(\"</alt-title>\",\" \")\n",
    "    s =  s.replace(\"</article-title>\",\" \")\n",
    "    s =  s.replace(\"<alt-title>\",\" \")\n",
    "    s=   s.replace(\"/\", \" \")\n",
    "    s=   s.replace(\"<\", \" \")\n",
    "    s=   s.replace(\">\", \" \")\n",
    "    s=   s.replace(\"sec\", \" \")\n",
    "    s=   s.replace(\"bold\", \" \")\n",
    "    s=   s.replace(\"aff id=\", \" \")\n",
    "    s=   s.replace(\"label\", \" \")\n",
    "    s=   s.replace(\"subtitle\", \" \")\n",
    "    s=   s.replace(\"p id=\", \" \")\n",
    "    s=   s.replace(\"aff\", \" \")\n",
    " \n",
    "    return s \n",
    "\n",
    "# Striping numbers and other useless characters for further processing\n",
    "\n",
    "def onlywords(text1):\n",
    "   text1=re.sub(r'[0-9]+', ', ', text1)\n",
    "   pattern = r\"[a-zA-Z,]+\"\n",
    "   result = re.findall(pattern,text1)\n",
    "   division=\" \"\n",
    "   strg=division.join(result)\n",
    "   return strg\n",
    "\n",
    "\n",
    "# Function for extracting countries from afflitions\n",
    "\n",
    "def  count_ex(texts):\n",
    "           #print ( \"text=\",texts)\n",
    "           texts=onlywords(texts)\n",
    "           for cn in countries:\n",
    "                if texts.endswith(cn[0]):\n",
    "                    #print(cn[0])\n",
    "                    country_final=cn[2]\n",
    "                    if country_final==\"GEO\" :\n",
    "                         if texts.find(\"Atlanta\")!=-1 or  texts.find(\"Columbus\")!=-1 :country_final=\"USA\"\n",
    "\n",
    "                    return country_final\n",
    "           doc = nlp(texts)\n",
    "           country_final=\"\"\n",
    "           if texts.find(\",USA\")!=-1 or texts.find(\" USA\")!=-1:\n",
    "             country_final=\"USA\"\n",
    "             return country_final\n",
    "\n",
    "\n",
    "           for st in states1:\n",
    "               st_name=\",\"+\" \"+st+\" \"\n",
    "               if texts.find(st_name)!=-1 or texts.endswith(\",\"+\" \"+st):\n",
    "                 country_final=\"USA\"\n",
    "                 return country_final\n",
    "\n",
    "            \n",
    "           if  country_final==\"\":\n",
    "            for entity in doc.ents:\n",
    "                #print(entity.text, \"=\", entity.label_)\n",
    "                if  entity.label_==\"GPE\":\n",
    "                     #print(entity.text, \"=\", entity.label_)\n",
    "                     count_name=entity.text\n",
    "                     for cn in countries:\n",
    "                          if count_name==cn[0]:\n",
    "                              #print (\"***\"+ cn[2])\n",
    "                              country_final=cn[2]\n",
    "                              if country_final==\"GEO\" :\n",
    "                                    # Exception for Georgia, since Georgia is both a name of the country Georgia and one of the US states, one more test should be carried out.\n",
    "                                  if texts.find(\"Atlanta\")!=-1 or  texts.find(\"Columbus\")!=-1  or texts.find(\"Savannah\")!=-1  or texts.find(\"Athens\")!=-1 :country_final=\"USA\"\n",
    "                \n",
    "                              \n",
    "            if country_final==\"\":\n",
    "              #print(\"phase3\")\n",
    "              for entity in doc.ents:\n",
    "                if  entity.label_==\"GPE\":\n",
    "                  for st in states:\n",
    "                     if entity.text==st:\n",
    "                         country_final=\"USA\"\n",
    "                         #print (\"*** phase3 state!!  USA\")\n",
    "                         \n",
    "            if country_final==\"\":\n",
    "              #print(\"phase2\")\n",
    "              for entity in doc.ents:\n",
    "                if  entity.label_==\"GPE\":\n",
    "                 #print (entity.text)\n",
    "                 #print ( GeoText(entity.text).country_mentions  )\n",
    "                 inputc=entity.text.replace(\" \",\",\")\n",
    "                 mycountry=GeoText(inputc).country_mentions\n",
    " \n",
    "                 for key,values in  mycountry.items():\n",
    "                     for cn in countries:\n",
    "                          if key==cn[0] or key==cn[1]:\n",
    "                              #print (\"pahe 2 \"+cn[2])\n",
    "                              country_final=cn[2]\n",
    "\n",
    "            if country_final==\"\":\n",
    "              #print(\"phase4\")\n",
    "              for entity in doc.ents:\n",
    "                if  entity.label_==\"ORG\":\n",
    "                 inputc=entity.text\n",
    "                 mycountry=GeoText(inputc).country_mentions\n",
    " \n",
    "                 for key,values in  mycountry.items():\n",
    "                     for cn in countries:\n",
    "                          if key==cn[0] or key==cn[1]:\n",
    "                              #print (\"*** phase4 \"+cn[2])\n",
    "                              country_final=cn[2]                            \n",
    "\n",
    "            #print (country_final)                 \n",
    "            return country_final\n",
    "        \n",
    "# Counting the records number\n",
    "rec_no=0                              \n",
    "\n",
    "# Opening the output file extracted.tsv\n",
    "\n",
    "tableoutput =[\"Journal ID \",\"issn No. \",\" Publisher Name\",\" Location\",\"Article Title\",\"\",\"Abstract \",\" Date\",\"Countibuter(s) \",\" affiliation(s)\" ,\"\", \"Countries Involved\"]\n",
    "fw=codecs.open(,\"w\",encoding='utf-8')\n",
    "fw.write(\"file_id\"+\"\\t\")\n",
    "for x in tableoutput:\n",
    "     fw.write(x+\"\\t\")\n",
    "     x=\"\"\n",
    "fw.write(\"\\n\")\n",
    "\n",
    "# Accessing the xml files\n",
    "path = \"D:\\\\uncomm1\\\\\" # path for the articles folders\n",
    "\n",
    "for rootc, dirs, files in os.walk(path):\n",
    "     for file in fnmatch.filter(files,\"*.nxml\"):\n",
    "        #print(file)\n",
    "        fileWithPath=str(os.path.join(rootc, file))\n",
    "        #print(fileWithPath)\n",
    "        utf8_file= codecs.open(fileWithPath, 'r' ,\"utf-8\")\n",
    "        tree = ET.parse(utf8_file)\n",
    "        root = tree.getroot()\n",
    "\n",
    "        fw.write(file+\"\\t\")\n",
    "        \n",
    "# Extracting  journal-id, issn, publisher-name and publisher-loc from XML header\n",
    "        for x in tableoutput:\n",
    "            x=\"\"\n",
    "     \n",
    "        i=-1\n",
    "        for child in root[0][0]:\n",
    "            i=i+1\n",
    "            if child.tag==\"journal-id\":\n",
    "                    tableoutput[0]= tableoutput[0]+ \" \" + child.text.rstrip('\\r\\n\\t')\n",
    "            if child.tag==\"issn\":\n",
    "                    tableoutput[1]=child.text.rstrip('\\r\\n\\t')\n",
    "            for child in root[0][0][i]:\n",
    "                      #print (child.tag, child.text.rstrip('\\r\\n\\t'))\n",
    "                      if child.tag==\"publisher-name\":\n",
    "                            tableoutput[2]=tableoutput[2]+child.text.rstrip('\\r\\n\\t')\n",
    "                      if child.tag==\"publisher-loc\":\n",
    "                          tableoutput[3]= tableoutput[3]+child.text.rstrip('\\r\\n\\t')\n",
    "# End of extracting  journal-id ....\n",
    "\n",
    "# Extracting artcile title\n",
    "        title=\"\"\n",
    "        i=-1\n",
    "        for child in root[0][1]:\n",
    "             i=i+1\n",
    "             if child.tag==\"title-group\":\n",
    "                 titleindex=i\n",
    "                 #print (\"titleindex:\",titleindex)\n",
    "                 break\n",
    "        for child in root[0][1][titleindex]:\n",
    "                   title=title + \" \"+ str(ET.tostring(child), \"utf-8\")\n",
    "                   #title=re.sub(r'\\W+\\s', '', title)\n",
    "                   \n",
    "        tableoutput[4]=title\n",
    "# End of extracting artcile title\n",
    "    \n",
    "# Extracting the article abstract\n",
    " \n",
    "        i=-1\n",
    "        abstracti=0\n",
    "        for child in root[0][1]:\n",
    "             i=i+1\n",
    "             \n",
    "             if child.tag==\"abstract\":\n",
    "                 abstracti=i\n",
    "                 #print (\"abstract:\",abstracti)\n",
    "                 break\n",
    "        abstract=\"\"\n",
    "        if abstracti!=0 :\n",
    "             for child in root[0][1][abstracti]:\n",
    "                   abstract=abstract + \" \"+str(ET.tostring(child), \"utf-8\")\n",
    "                   #abstract=re.sub(r'\\W+\\s', '', abstract)\n",
    "\n",
    "        tableoutput[6]= (abstract)\n",
    "# End of extracting the article abstract\n",
    "        \n",
    "# Extracting the date\n",
    " \n",
    "        i=-1\n",
    "        for child in root[0][1]:\n",
    "             i=i+1\n",
    "             \n",
    "             if child.tag==\"pub-date\":\n",
    "                 pubi=i\n",
    "                 #print (\"pubi:\",pubi)\n",
    "                 break\n",
    "        pubdate=\"\";\n",
    "        for child in root[0][1][pubi]:\n",
    "                   #print (child.tag, child.text)\n",
    "                   pubdate=child.text.rstrip('\\r\\n\\t')\n",
    "                   if pubdate==None:\n",
    "                       pubdate=\"NA\"\n",
    "        #print (pubdate)\n",
    "        tableoutput[7]=pubdate\n",
    "# End of extracting the date\n",
    "\n",
    "# Extracting  the artcile contributers\n",
    "        i=-1\n",
    "        cgi=-1\n",
    "        for child in root[0][1]:\n",
    "             i=i+1\n",
    "        \n",
    "             if child.tag==\"contrib-group\":\n",
    "                 cgi=i\n",
    "                 #print (\"cgi:\",cgi)           \n",
    "        i=-1\n",
    "        all_contrib=[]\n",
    "        if cgi!=-1: \n",
    "         for x in root[0][1][cgi]:\n",
    "            i=i+1\n",
    "            for child in root[0][1][cgi][i]:\n",
    "                if child.tag==\"name\":\n",
    "                    for child in root[0][1][cgi][i][0]:\n",
    "                        #print (child.tag, child.text)\n",
    "                        if child.tag==\"surname\":\n",
    "                            surname=child.text\n",
    "                        if child.tag==\"given-names\":\n",
    "                           if child.text!=None:\n",
    "                               all_contrib.append(surname+\" \"+child.text)\n",
    "        tableoutput[8]=str(all_contrib)\n",
    "        country_final=\"\"\n",
    "# End of extracting  the artcile contributers  \n",
    "\n",
    "# Extracting the artcicle affiliations and involved_countries\n",
    "        all_aff=\"\"\n",
    "        involved_countries=\"\"\n",
    " \n",
    "        for el in root[0][1][cgi].iterfind('aff'):\n",
    "            aff=str(ET.tostring(el), \"UTF-8\")\n",
    "            aff=strip_tags(aff)\n",
    " \n",
    "            all_aff=all_aff+\"\"+aff\n",
    "            involved_countries=involved_countries+\" \"+  str( count_ex(str(aff)))\n",
    "        tableoutput[9]=all_aff\n",
    "        tableoutput[11]=involved_countries\n",
    "        tableoutput[10]=\"\"\n",
    "# End of extracting the artcicle affiliations and involved_countries\n",
    "\n",
    "# Extracting the affiliations and involved_countries when affilition is in xml root\n",
    "        if all_aff==\"\":\n",
    "  \n",
    "         involved_countries=\"\"\n",
    " \n",
    "         for el in root[0][1].iterfind('aff'):\n",
    "             aff=str(ET.tostring(el), \"UTF-8\")\n",
    "            \n",
    "             aff=strip_tags(aff)\n",
    "   \n",
    "             all_aff=all_aff+\".\"+aff\n",
    "             involved_countries=involved_countries+\" \"+  str( count_ex(str(aff)))\n",
    "    \n",
    "         tableoutput[9]=all_aff\n",
    "         tableoutput[11]=involved_countries\n",
    "# End of extracting the affiliations\n",
    "\n",
    "# Extracting country from contributers attributes(not affiliations)\n",
    "        if tableoutput[11]==\"\":\n",
    "         involved_countries=\"\"\n",
    "         contrib1=root[0][1][cgi].iterfind('contrib')\n",
    "         for el in contrib1:\n",
    "            cont=str(ET.tostring(el), \"utf-8\")\n",
    "            cont=strip_tags(cont)\n",
    "            doc=nlp(cont)\n",
    "            for entity in doc.ents:\n",
    "                #print(entity.text, \"=\", entity.label_)\n",
    "                if  entity.label_==\"GPE\":\n",
    "                     #print(entity.text, \"=\", entity.label_)\n",
    "                     count_name=entity.text\n",
    "                     for cn in countries:\n",
    "                          if count_name==cn[0]:\n",
    "                              #print (\"*** from names \"+ cn[2])\n",
    "                              country_final=cn[2]\n",
    "            involved_countries=involved_countries+\" \"+  country_final\n",
    "         tableoutput[11]=involved_countries\n",
    "\n",
    "#Writing the list to file\n",
    "        \n",
    "        for x in range(0, 12):\n",
    "                  tableoutput[x]=tableoutput[x].replace(\"\\\\n\",\" \")\n",
    "                  tableoutput[x]=tableoutput[x].replace(\"\\n\",\" \")\n",
    "                  tableoutput[x]=strip_tags(tableoutput[x])\n",
    "                  fw.write(tableoutput[x]+\"\\t\")\n",
    "                \n",
    "        fw.write(\"\\n\")\n",
    "        rec_no=rec_no+1\n",
    "        #print(rec_no)\n",
    "                      \n",
    "fw.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}