prog_alchemy_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 程序 `alchemy_data`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个程序的主要目的是将提取之后的特征数据进行必要的重组，使之成为可以被 PyTorch Geometric (pyg) 读取的作为训练、验证、测试的数据。\n",
    "\n",
    "但这些返回的数据事实上在实际代入模型前还会作简单的预处理。因此这个程序作的是粗处理。但实际代入模型的数据的信息量不会比现在粗处理的信息量更大。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "RDKit WARNING: [20:20:42] Enabling RDKit 2019.09.1 jupyter extensions\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "\n",
    "import torch\n",
    "from sdf_reader import *\n",
    "from alchemy_data import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_1409364 = sdf_to_dict(\"raw-sdf/dev/sdf/atom_11/1409364.sdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 函数 `atom_dat_reader`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该函数读入提取后特征的原子部分，返回 pyg 的 Data 类型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data(edge_attr=[289, 7], edge_index=[2, 289], x=[17, 13], y=[])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "atom_data = atom_dat_reader(dict_1409364[0], 0)\n",
    "atom_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述打印出来的意义是矩阵的维度。\n",
    "\n",
    "- `x` 表示的是原子特征 (顶点特征)\n",
    "\n",
    "- `edge_attr` 表示原子之间相互的特征 (边特征)\n",
    "\n",
    "- `edge_index` 表示的是每个原子间相互特征，其对应的两个原子的序号 (边对应的两个顶点序号)\n",
    "\n",
    "- `y` 表示数据集所提供的目标值 (target)；在这里我们不考虑这个变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "取出其中的矩阵的方法也比较简单："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "        [ 2.8442, 12.6574,  1.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "        [ 4.5064,  9.3202,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "        ...,\n",
       "        [ 7.9830,  0.1253,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "        [ 7.9976,  0.1250,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "atom_data.edge_attr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于顶点特征 `x` 其有 17 行代表分子中有 17 个原子；而其 13 列是不随分子的改变而改变的，意义是\n",
    "\n",
    "- 0：核电荷数\n",
    "\n",
    "- 1:8：原子的 one-hot 编码\n",
    "\n",
    "- 8：原子是否带有电荷\n",
    "\n",
    "- 9：原子是否是芳香性的\n",
    "\n",
    "- 10:13：原子的杂化方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于边而言也是类似的，289=17×17 表示边的数量，而 7 列不随分子的改变而改变，意义是\n",
    "\n",
    "- 0：原子间距离\n",
    "\n",
    "- 1：原子间核排斥能\n",
    "\n",
    "- 2:7：键类型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 函数 `orbital_dat_reader`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该函数读入提取后特征的轨道部分，返回 pyg 的 Data 类型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data(atom_idx=[65], edge_attr=[4225, 8], edge_index=[2, 4225], x=[65, 13], y=[])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orbital_data = orbital_dat_reader(dict_1409364[1], 0)\n",
    "orbital_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里额外存了 `atom_idx`，它代表的是轨道中心所对应的原子的序号。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "顶点特征是也是 13 列，意义是\n",
    "\n",
    "- 0：轨道对应原子的核电荷\n",
    "\n",
    "- 1:8：原子的 one-hot 编码\n",
    "\n",
    "- 8：轨道作为 STO-3G 基组的指数系数 $\\zeta$\n",
    "\n",
    "- 9：轨道是否是价层轨道\n",
    "\n",
    "- 10:13：轨道的空间取向 ($x, y, z$ 方向)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "边特征 8 列，意义是\n",
    "\n",
    "- 0：轨道对应原子间的距离\n",
    "\n",
    "- 1：重叠积分矩阵\n",
    "\n",
    "- 2：动能积分矩阵\n",
    "\n",
    "- 3：核势能积分矩阵\n",
    "\n",
    "- 4:7：偶极积分矩阵\n",
    "\n",
    "- 7：密度初猜 (通过 $F=T+V$ 并作对角化得到)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 类 `AlchemyData`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "类 `AlchemyData` 是从硬盘读取整理过后的数据集，并返回数据的类型；它会实际地用于模型的训练与测试中。\n",
    "\n",
    "但若硬盘没有整理后的数据集，`AlchemyData` 在实例化过程中会先生成这个数据集。生成数据集的过程我们这里不作讨论。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AlchemyData(1000)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dev_atom_dataset = AlchemyData(mode='dev', net_type=\"atom\", train_csv_path=\"./raw/train.csv\")\n",
    "dev_atom_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述程序表明原子数据的训练集 (`\"atom\"` 与 `\"dev\"`) 共有 1000 个分子。事实上，我们方才计算的分子就是第 963 号训练集数据集分子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "963\n"
     ]
    }
   ],
   "source": [
    "for idx, i in enumerate(dev_atom_dataset):\n",
    "    try:\n",
    "        if np.allclose(i.x, atom_data.x):\n",
    "            print(idx)\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除此之外，由于我们引入了目标值文件 `\"./raw/train.csv\"`，因此我们还能看到该分子的 12 个性质的目标值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.8340, -0.0377, -1.3812, -2.3111, -0.6381, -2.3111,  1.0454, -2.3111,\n",
       "         -2.3111, -2.1717,  1.4114,  0.8701]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dev_atom_dataset[963].y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你看到的下述文件夹下的文件，就是处理完后的训练、5-fold 验证、测试集的数据所在硬盘上的文件了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['orbital-test.pt',\n",
       " 'atom-valid_04.pt',\n",
       " 'orbital-dev.pt',\n",
       " 'orbital-valid_03.pt',\n",
       " 'orbital-valid_01.pt',\n",
       " 'atom-test.pt',\n",
       " 'orbital-valid_02.pt',\n",
       " 'atom-dev.pt',\n",
       " 'atom-valid_00.pt',\n",
       " 'atom-valid_03.pt',\n",
       " 'orbital-valid_04.pt',\n",
       " 'atom-valid_01.pt',\n",
       " 'atom-valid_02.pt',\n",
       " 'orbital-valid_00.pt']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(\"processed/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中，对于每个分子，其原子图的数据与轨道图的数据分开存在两个文件中。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}