Day7_2_Transformer_advance_pb.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Day7_2_Transformer_advance_pb.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"jIXN43x5sLQ9","colab_type":"text"},"source":["#Transformer Advance PB\n"]},{"cell_type":"markdown","metadata":{"id":"RGkIxr2bSAPi","colab_type":"text"},"source":["## Serving Models   \n","\n","**TensorFlow Serving은 생산 환경을 위해 설계된 기계 학습 모델을위한 유연하고 고성능의 서비스 시스템 적용**\n","\n","**TensorFlow Serving을 사용하면 새로운 알고리즘과 실험을 쉽게 배포 할 수 있으며 동일한 서버 아키텍처와 API를 유지 가능**"]},{"cell_type":"markdown","metadata":{"id":"adp1mzeESgsK","colab_type":"text"},"source":["## 구글 프로토콜 버퍼\n","\n","**프로토토콜 버퍼는 구글에서 개발하고 오픈소스로 공개한, 직렬화 데이타 구조 (Serialized Data Structure)**\n","\n","**C++,C#, Go, Java, Python, Object C, Javascript, Ruby 등 다양한 언어를 지원**\n","\n","**직렬화 속도가 빠르고 직렬화된 파일의 크기도 작음**"]},{"cell_type":"code","metadata":{"id":"J3RD1Snr41F5","colab_type":"code","outputId":"185dd09f-6f80-4552-c23c-793ba6a80457","executionInfo":{"status":"ok","timestamp":1563574213736,"user_tz":-540,"elapsed":48933,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":649}},"source":["!pip install --upgrade tensorflow-gpu==1.13.0rc1"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting tensorflow-gpu==1.13.0rc1\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ac/2d/6b9b115b0d507b885a0766717c324fb71fce6cb6380e67fe24db1ab699e7/tensorflow_gpu-1.13.0rc1-cp36-cp36m-manylinux1_x86_64.whl (345.2MB)\n","\u001b[K     |████████████████████████████████| 345.2MB 93kB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (0.8.0)\n","Requirement already satisfied, skipping upgrade: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (1.12.0)\n","Requirement already satisfied, skipping upgrade: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (0.2.2)\n","Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (1.1.0)\n","Requirement already satisfied, skipping upgrade: absl-py>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (0.7.1)\n","Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (1.16.4)\n","Requirement already satisfied, skipping upgrade: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (3.7.1)\n","Collecting tensorboard<1.13.0,>=1.12.0 (from tensorflow-gpu==1.13.0rc1)\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/07/53/8d32ce9471c18f8d99028b7cef2e5b39ea8765bd7ef250ca05b490880971/tensorboard-1.12.2-py3-none-any.whl (3.0MB)\n","\u001b[K     |████████████████████████████████| 3.1MB 34.4MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (1.15.0)\n","Requirement already satisfied, skipping upgrade: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (1.1.0)\n","Requirement already satisfied, skipping upgrade: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (0.33.4)\n","Requirement already satisfied, skipping upgrade: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==1.13.0rc1) (1.0.8)\n","Collecting tensorflow-estimator<1.14.0rc0,>=1.13.0rc0 (from tensorflow-gpu==1.13.0rc1)\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bb/48/13f49fc3fa0fdf916aa1419013bb8f2ad09674c275b4046d5ee669a46873/tensorflow_estimator-1.13.0-py2.py3-none-any.whl (367kB)\n","\u001b[K     |████████████████████████████████| 368kB 56.3MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow-gpu==1.13.0rc1) (41.0.1)\n","Requirement already satisfied, skipping upgrade: werkzeug>=0.11.10 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow-gpu==1.13.0rc1) (0.15.5)\n","Requirement already satisfied, skipping upgrade: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow-gpu==1.13.0rc1) (3.1.1)\n","Requirement already satisfied, skipping upgrade: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow-gpu==1.13.0rc1) (2.8.0)\n","Collecting mock>=2.0.0 (from tensorflow-estimator<1.14.0rc0,>=1.13.0rc0->tensorflow-gpu==1.13.0rc1)\n","  Downloading https://files.pythonhosted.org/packages/05/d2/f94e68be6b17f46d2c353564da56e6fb89ef09faeeff3313a046cb810ca9/mock-3.0.5-py2.py3-none-any.whl\n","\u001b[31mERROR: tensorflow 1.14.0 has requirement tensorboard<1.15.0,>=1.14.0, but you'll have tensorboard 1.12.2 which is incompatible.\u001b[0m\n","\u001b[31mERROR: tensorflow 1.14.0 has requirement tensorflow-estimator<1.15.0rc0,>=1.14.0rc0, but you'll have tensorflow-estimator 1.13.0 which is incompatible.\u001b[0m\n","Installing collected packages: tensorboard, mock, tensorflow-estimator, tensorflow-gpu\n","  Found existing installation: tensorboard 1.14.0\n","    Uninstalling tensorboard-1.14.0:\n","      Successfully uninstalled tensorboard-1.14.0\n","  Found existing installation: tensorflow-estimator 1.14.0\n","    Uninstalling tensorflow-estimator-1.14.0:\n","      Successfully uninstalled tensorflow-estimator-1.14.0\n","Successfully installed mock-3.0.5 tensorboard-1.12.2 tensorflow-estimator-1.13.0 tensorflow-gpu-1.13.0rc1\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"6Aq3HlVnsKyg","colab_type":"code","outputId":"ea136745-7427-4500-c08a-f1af6c713f7e","executionInfo":{"status":"ok","timestamp":1563576416612,"user_tz":-540,"elapsed":2251796,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":122}},"source":["from google.colab import auth\n","auth.authenticate_user()\n","\n","from google.colab import drive\n","drive.mount('/content/gdrive')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n","\n","Enter your authorization code:\n","··········\n","Mounted at /content/gdrive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"0hPkzfSAzKRq","colab_type":"code","outputId":"16b97e8a-6c60-408d-80f2-b772ccf05da1","executionInfo":{"status":"ok","timestamp":1563576423499,"user_tz":-540,"elapsed":2889,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":136}},"source":["!ls -al"],"execution_count":0,"outputs":[{"output_type":"stream","text":["total 32\n","drwxr-xr-x 1 root root 4096 Jul 19 22:46 .\n","drwxr-xr-x 1 root root 4096 Jul 19 22:08 ..\n","-rw-r--r-- 1 root root 2487 Jul 19 22:46 adc.json\n","drwxr-xr-x 1 root root 4096 Jul 19 22:46 .config\n","drwx------ 3 root root 4096 Jul 19 22:46 gdrive\n","drwxr-xr-x 1 root root 4096 Jul  3 16:14 sample_data\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"lwkpWReDzjRy","colab_type":"code","colab":{}},"source":["DATA_IN_PATH = './data_in/'\n","DATA_OUT_PATH = './data_out/'"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"PK0UpN9aznZk","colab_type":"code","outputId":"f5a6c95e-f4cb-4a20-90ce-35a1ceefd1ff","executionInfo":{"status":"ok","timestamp":1563576425448,"user_tz":-540,"elapsed":3438,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["!rm -rf data_in;rm -rf data_out;ls;"],"execution_count":0,"outputs":[{"output_type":"stream","text":["adc.json  gdrive  sample_data\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"X192d5qczo9D","colab_type":"code","colab":{}},"source":["import os\n","\n","if not os.path.exists(DATA_IN_PATH):\n","    os.makedirs(DATA_IN_PATH)\n","    \n","if not os.path.exists(DATA_OUT_PATH):\n","    os.makedirs(DATA_OUT_PATH)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"D2FPrgkGzrJ7","colab_type":"code","outputId":"705d9ce7-1e58-4849-993e-363e2f553eb9","executionInfo":{"status":"ok","timestamp":1563576427183,"user_tz":-540,"elapsed":3171,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["!ls -al '/content/gdrive/My Drive/Colab Notebooks/Data/'"],"execution_count":0,"outputs":[{"output_type":"stream","text":["total 800183\n","-rw------- 1 root root    889842 Jun 29 03:10 ChatBotData.csv\n","-rw------- 1 root root   1413786 Jul  9 05:57 data_configs.json\n","-rw------- 1 root root 117931806 Jul  9 03:58 test.csv.zip\n","-rw------- 1 root root  18766496 Jul  9 03:58 test_id.npy\n","-rw------- 1 root root 290878832 Jul  9 03:59 test_q1.npy\n","-rw------- 1 root root 290878832 Jul  9 03:59 test_q2.npy\n","-rw------- 1 root root  22202741 Jul  9 03:58 train.csv.zip\n","-rw------- 1 root root   2388336 Jul  9 03:58 train_label.npy\n","-rw------- 1 root root  37017352 Jul  9 03:58 train_q1.npy\n","-rw------- 1 root root  37017352 Jul  9 03:58 train_q2.npy\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"trqD66gXztAJ","colab_type":"code","outputId":"9a58c724-7761-4939-9fad-58b98e48b825","executionInfo":{"status":"ok","timestamp":1563576434491,"user_tz":-540,"elapsed":3336,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":153}},"source":["!ls -al '/content/gdrive/My Drive/Colab Notebooks/Seq2seq/'"],"execution_count":0,"outputs":[{"output_type":"stream","text":["total 48\n","-rw------- 1 root root  1451 Jul 18 03:52 configs.py\n","-rw------- 1 root root 11523 Jul 12 21:55 data.py\n","-rw------- 1 root root  6820 Jul  5 22:09 model_att.py\n","-rw------- 1 root root  4657 Jun 29 03:42 model.py\n","-rw------- 1 root root  9275 Jul 18 03:02 model_tra_adv_pb.py\n","-rw------- 1 root root  9123 Jul 12 21:35 model_tra_adv.py\n","drwx------ 2 root root  4096 May 19 19:57 __pycache__\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"yeJYcqdlzz5S","colab_type":"code","colab":{}},"source":["#!wget -P '/content/gdrive/My Drive/Colab Notebooks/Seq2seq/' https://raw.githubusercontent.com/changwookjun/learningspoons/master/Seq2seq/model_tra_adv_pb.py"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"c6wHDNUkNIQ4","colab_type":"code","outputId":"b1962b48-db15-4ce8-8ab8-a1964e9ca33a","executionInfo":{"status":"ok","timestamp":1563421962865,"user_tz":-540,"elapsed":1667,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["#!wget -P '/content/gdrive/My Drive/Colab Notebooks/Seq2seq/' https://raw.githubusercontent.com/changwookjun/learningspoons/master/Seq2seq/configs.py  "],"execution_count":0,"outputs":[{"output_type":"stream","text":["--2019-07-18 03:52:41--  https://raw.githubusercontent.com/changwookjun/learningspoons/master/Seq2seq/configs.py\n","Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n","Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 1451 (1.4K) [text/plain]\n","Saving to: ‘/content/gdrive/My Drive/Colab Notebooks/Seq2seq/configs.py’\n","\n","\rconfigs.py            0%[                    ]       0  --.-KB/s               \rconfigs.py          100%[===================>]   1.42K  --.-KB/s    in 0s      \n","\n","2019-07-18 03:52:41 (116 MB/s) - ‘/content/gdrive/My Drive/Colab Notebooks/Seq2seq/configs.py’ saved [1451/1451]\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"_V7fT2EeHH9u","colab_type":"code","colab":{}},"source":["#!wget -P '/content/gdrive/My Drive/Colab Notebooks/Seq2seq/' https://raw.githubusercontent.com/changwookjun/learningspoons/master/Seq2seq/data.py  "],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"y9KszrHwz45J","colab_type":"code","outputId":"18d51fcc-c933-425e-eca4-ba662fc06983","executionInfo":{"status":"ok","timestamp":1563576466275,"user_tz":-540,"elapsed":3300,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["!cp '/content/gdrive/My Drive/Colab Notebooks/Data/ChatBotData.csv' ./data_in/ChatBotData.csv;ls -al ./data_in/"],"execution_count":0,"outputs":[{"output_type":"stream","text":["total 884\n","drwxr-xr-x 2 root root   4096 Jul 19 22:47 .\n","drwxr-xr-x 1 root root   4096 Jul 19 22:47 ..\n","-rw------- 1 root root 889842 Jul 19 22:47 ChatBotData.csv\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"TPdKU10G0GAv","colab_type":"code","outputId":"8ff01ccd-304c-4482-ad3d-fd217c7596ae","executionInfo":{"status":"ok","timestamp":1563576466279,"user_tz":-540,"elapsed":2954,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["import pandas as pd\n","\n","train_data = pd.read_csv(DATA_IN_PATH + 'ChatBotData.csv')\n","train_data.head()"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Q</th>\n","      <th>A</th>\n","      <th>label</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>12시 땡!</td>\n","      <td>하루가 또 가네요.</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1지망 학교 떨어졌어</td>\n","      <td>위로해 드립니다.</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3박4일 놀러가고 싶다</td>\n","      <td>여행은 언제나 좋죠.</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3박4일 정도 놀러가고 싶다</td>\n","      <td>여행은 언제나 좋죠.</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>PPL 심하네</td>\n","      <td>눈살이 찌푸려지죠.</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                 Q            A  label\n","0           12시 땡!   하루가 또 가네요.      0\n","1      1지망 학교 떨어졌어    위로해 드립니다.      0\n","2     3박4일 놀러가고 싶다  여행은 언제나 좋죠.      0\n","3  3박4일 정도 놀러가고 싶다  여행은 언제나 좋죠.      0\n","4          PPL 심하네   눈살이 찌푸려지죠.      0"]},"metadata":{"tags":[]},"execution_count":10}]},{"cell_type":"code","metadata":{"id":"TWh0uvVN0Ldx","colab_type":"code","outputId":"87bf9656-9ece-4ee3-cb25-2e77cb26084d","executionInfo":{"status":"ok","timestamp":1563576471588,"user_tz":-540,"elapsed":5400,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":173}},"source":["!pip install konlpy"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting konlpy\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/3d/4e983cd98d87b50b2ab0387d73fa946f745aa8164e8888a714d5129f9765/konlpy-0.5.1-py2.py3-none-any.whl (19.4MB)\n","\u001b[K     |████████████████████████████████| 19.4MB 3.4MB/s \n","\u001b[?25hCollecting JPype1>=0.5.7 (from konlpy)\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/07/09/e19ce27d41d4f66d73ac5b6c6a188c51b506f56c7bfbe6c1491db2d15995/JPype1-0.7.0-cp36-cp36m-manylinux2010_x86_64.whl (2.7MB)\n","\u001b[K     |████████████████████████████████| 2.7MB 58.3MB/s \n","\u001b[?25hInstalling collected packages: JPype1, konlpy\n","Successfully installed JPype1-0.7.0 konlpy-0.5.1\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"yjX9IVkM0M66","colab_type":"code","outputId":"335b0a4d-06d9-4b24-82d7-c30635c6bd1f","executionInfo":{"status":"ok","timestamp":1563576475090,"user_tz":-540,"elapsed":8650,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["!pip install tqdm"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (4.28.1)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"NNvvdABn0lwE","colab_type":"code","colab":{}},"source":["import tensorflow as tf\n","import numpy as np\n","from tqdm import tqdm\n","from konlpy.tag import Okt\n","import re\n","import os\n","import sys"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"atR1xxs20Ucy","colab_type":"code","colab":{}},"source":["sys.path.insert(0, '/content/gdrive/My Drive/Colab Notebooks/Seq2seq/')\n","\n","import data\n","import configs as conf\n","import model_tra_adv_pb as ml"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"jb97zqMT0d8B","colab_type":"code","outputId":"4c81801d-97c5-433c-8b46-ae6a422fc3d1","executionInfo":{"status":"ok","timestamp":1563576508334,"user_tz":-540,"elapsed":36582,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":221}},"source":["char2idx,  idx2char, vocabulary_length = data.load_vocabulary()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/jpype/_core.py:210: UserWarning: \n","-------------------------------------------------------------------------------\n","Deprecated: convertStrings was not specified when starting the JVM. The default\n","behavior in JPype will be False starting in JPype 0.8. The recommended setting\n","for new code is convertStrings=False.  The legacy value of True was assumed for\n","this session. If you are a user of an application that reported this warning,\n","please file a ticket with the developer.\n","-------------------------------------------------------------------------------\n","\n","  \"\"\")\n","100%|██████████| 11823/11823 [00:18<00:00, 644.92it/s]\n","100%|██████████| 11823/11823 [00:11<00:00, 1053.97it/s]\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"n7gyaitq0xKh","colab_type":"code","outputId":"005b759f-d7c4-4afb-ae3c-aba6d49f67d2","executionInfo":{"status":"ok","timestamp":1563576508335,"user_tz":-540,"elapsed":36099,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["print(vocabulary_length)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["12657\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"_KKl15SE8wnB","colab_type":"code","colab":{}},"source":["train_input, train_label, eval_input, eval_label = data.load_data()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"7NpjPfwk7g2-","colab_type":"code","outputId":"a217e4df-cb9a-4583-a0f2-c442067bf881","executionInfo":{"status":"ok","timestamp":1563576529576,"user_tz":-540,"elapsed":55537,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["train_input_enc, train_input_enc_length = data.enc_processing(train_input, char2idx)\n","train_output_dec, train_output_dec_length = data.dec_input_processing(train_label, char2idx)\n","train_target_dec = data.dec_target_processing(train_label, char2idx)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["100%|██████████| 7921/7921 [00:05<00:00, 1333.00it/s]\n","100%|██████████| 7921/7921 [00:07<00:00, 1062.10it/s]\n","100%|██████████| 7921/7921 [00:07<00:00, 1056.45it/s]\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"Nf--G6tw8r0y","colab_type":"code","outputId":"4bf8bfa0-6931-4e51-82e3-818c5d84a9d0","executionInfo":{"status":"ok","timestamp":1563576540034,"user_tz":-540,"elapsed":65750,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["eval_input_enc, eval_input_enc_length = data.enc_processing(eval_input,char2idx)\n","eval_output_dec, eval_output_dec_length = data.dec_input_processing(eval_label, char2idx)\n","eval_target_dec = data.dec_target_processing(eval_label, char2idx)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["100%|██████████| 3902/3902 [00:02<00:00, 1337.58it/s]\n","100%|██████████| 3902/3902 [00:03<00:00, 1067.20it/s]\n","100%|██████████| 3902/3902 [00:03<00:00, 1074.68it/s]\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"DtfddW2D8yYq","colab_type":"code","colab":{}},"source":["check_point_path = os.path.join(os.getcwd(), conf.DEFINES.check_point_path)\n","os.makedirs(check_point_path, exist_ok=True)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"3Ts6koLx80UC","colab_type":"code","outputId":"1a08bf0d-f7a2-405b-fbe8-393a5ba6a173","executionInfo":{"status":"ok","timestamp":1563576563245,"user_tz":-540,"elapsed":696,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":173}},"source":["# 에스티메이터 구성한다.\n","transformer = tf.estimator.Estimator(\n","    model_fn=ml.Model,  # 모델 등록한다.\n","    model_dir=conf.DEFINES.check_point_path,  # 체크포인트 위치 등록한다.\n","    params={  # 모델 쪽으로 파라메터 전달한다.\n","        'embedding_size': conf.DEFINES.embedding_size,\n","        'model_hidden_size': conf.DEFINES.hidden_size,  \n","        'ffn_hidden_size': 512,\n","        'attention_head_size': 4,\n","        'learning_rate': conf.DEFINES.learning_rate,  \n","        'vocabulary_length': vocabulary_length,  \n","        'embedding_size': conf.DEFINES.embedding_size,  \n","        'layer_size': conf.DEFINES.layer_size,\n","        'max_sequence_length': conf.DEFINES.max_sequence_length,\n","    })"],"execution_count":0,"outputs":[{"output_type":"stream","text":["INFO:tensorflow:Using default config.\n","INFO:tensorflow:Using config: {'_model_dir': './data_out/check_point', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n","graph_options {\n","  rewrite_options {\n","    meta_optimizer_iterations: ONE\n","  }\n","}\n",", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f97a6ca4898>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"80YdjjNM85lx","colab_type":"code","outputId":"856e889c-20e9-4acd-8a89-3c12eaa7dd4d","executionInfo":{"status":"ok","timestamp":1563577300589,"user_tz":-540,"elapsed":737857,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":1000}},"source":["transformer.train(input_fn=lambda: data.train_input_fn(train_input_enc, train_output_dec, train_target_dec, conf.DEFINES.batch_size), steps=conf.DEFINES.train_steps)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Colocations handled automatically by placer.\n","WARNING:tensorflow:From /content/gdrive/My Drive/Colab Notebooks/Seq2seq/data.py:195: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.\n","INFO:tensorflow:Calling model_fn.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.cast instead.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.cast instead.\n","INFO:tensorflow:Done calling model_fn.\n","INFO:tensorflow:Create CheckpointSaverHook.\n","INFO:tensorflow:Graph was finalized.\n","INFO:tensorflow:Running local_init_op.\n","INFO:tensorflow:Done running local_init_op.\n","INFO:tensorflow:Saving checkpoints for 0 into ./data_out/check_point/model.ckpt.\n","INFO:tensorflow:loss = 9.4974575, step = 0\n","INFO:tensorflow:global_step/sec: 17.821\n","INFO:tensorflow:loss = 1.6963165, step = 100 (5.615 sec)\n","INFO:tensorflow:global_step/sec: 28.5388\n","INFO:tensorflow:loss = 1.4375715, step = 200 (3.505 sec)\n","INFO:tensorflow:global_step/sec: 28.3939\n","INFO:tensorflow:loss = 1.4559299, step = 300 (3.521 sec)\n","INFO:tensorflow:global_step/sec: 28.3356\n","INFO:tensorflow:loss = 1.2474421, step = 400 (3.527 sec)\n","INFO:tensorflow:global_step/sec: 28.4716\n","INFO:tensorflow:loss = 1.0696027, step = 500 (3.514 sec)\n","INFO:tensorflow:global_step/sec: 28.4888\n","INFO:tensorflow:loss = 1.1616455, step = 600 (3.510 sec)\n","INFO:tensorflow:global_step/sec: 28.1919\n","INFO:tensorflow:loss = 1.1175017, step = 700 (3.545 sec)\n","INFO:tensorflow:global_step/sec: 28.2695\n","INFO:tensorflow:loss = 1.0528407, step = 800 (3.540 sec)\n","INFO:tensorflow:global_step/sec: 28.2565\n","INFO:tensorflow:loss = 0.88931334, step = 900 (3.539 sec)\n","INFO:tensorflow:global_step/sec: 28.0513\n","INFO:tensorflow:loss = 1.0424408, step = 1000 (3.563 sec)\n","INFO:tensorflow:global_step/sec: 28.195\n","INFO:tensorflow:loss = 0.85154295, step = 1100 (3.547 sec)\n","INFO:tensorflow:global_step/sec: 28.0681\n","INFO:tensorflow:loss = 0.7903445, step = 1200 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 27.8705\n","INFO:tensorflow:loss = 0.6890667, step = 1300 (3.588 sec)\n","INFO:tensorflow:global_step/sec: 27.9243\n","INFO:tensorflow:loss = 0.61611706, step = 1400 (3.579 sec)\n","INFO:tensorflow:global_step/sec: 27.9879\n","INFO:tensorflow:loss = 0.60878503, step = 1500 (3.574 sec)\n","INFO:tensorflow:global_step/sec: 27.746\n","INFO:tensorflow:loss = 0.5898498, step = 1600 (3.603 sec)\n","INFO:tensorflow:global_step/sec: 27.8774\n","INFO:tensorflow:loss = 0.51034003, step = 1700 (3.588 sec)\n","INFO:tensorflow:global_step/sec: 27.938\n","INFO:tensorflow:loss = 0.51811105, step = 1800 (3.580 sec)\n","INFO:tensorflow:global_step/sec: 27.7092\n","INFO:tensorflow:loss = 0.50355697, step = 1900 (3.610 sec)\n","INFO:tensorflow:global_step/sec: 27.7917\n","INFO:tensorflow:loss = 0.44240487, step = 2000 (3.597 sec)\n","INFO:tensorflow:global_step/sec: 27.8704\n","INFO:tensorflow:loss = 0.4637135, step = 2100 (3.585 sec)\n","INFO:tensorflow:global_step/sec: 27.7253\n","INFO:tensorflow:loss = 0.4381715, step = 2200 (3.607 sec)\n","INFO:tensorflow:global_step/sec: 27.9302\n","INFO:tensorflow:loss = 0.44015983, step = 2300 (3.580 sec)\n","INFO:tensorflow:global_step/sec: 27.9356\n","INFO:tensorflow:loss = 0.41218337, step = 2400 (3.583 sec)\n","INFO:tensorflow:global_step/sec: 27.802\n","INFO:tensorflow:loss = 0.40567127, step = 2500 (3.597 sec)\n","INFO:tensorflow:global_step/sec: 28.0675\n","INFO:tensorflow:loss = 0.4176835, step = 2600 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 28.0631\n","INFO:tensorflow:loss = 0.41847542, step = 2700 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 27.9798\n","INFO:tensorflow:loss = 0.37743744, step = 2800 (3.575 sec)\n","INFO:tensorflow:global_step/sec: 28.0414\n","INFO:tensorflow:loss = 0.39572898, step = 2900 (3.564 sec)\n","INFO:tensorflow:global_step/sec: 27.8499\n","INFO:tensorflow:loss = 0.36525443, step = 3000 (3.592 sec)\n","INFO:tensorflow:global_step/sec: 28.0518\n","INFO:tensorflow:loss = 0.35434738, step = 3100 (3.566 sec)\n","INFO:tensorflow:global_step/sec: 28.1205\n","INFO:tensorflow:loss = 0.3870132, step = 3200 (3.556 sec)\n","INFO:tensorflow:global_step/sec: 27.9818\n","INFO:tensorflow:loss = 0.37365955, step = 3300 (3.571 sec)\n","INFO:tensorflow:global_step/sec: 28.1665\n","INFO:tensorflow:loss = 0.3457445, step = 3400 (3.553 sec)\n","INFO:tensorflow:global_step/sec: 28.0416\n","INFO:tensorflow:loss = 0.31738797, step = 3500 (3.566 sec)\n","INFO:tensorflow:global_step/sec: 27.8975\n","INFO:tensorflow:loss = 0.31482413, step = 3600 (3.584 sec)\n","INFO:tensorflow:global_step/sec: 28.0077\n","INFO:tensorflow:loss = 0.33106902, step = 3700 (3.571 sec)\n","INFO:tensorflow:global_step/sec: 28.007\n","INFO:tensorflow:loss = 0.30619323, step = 3800 (3.570 sec)\n","INFO:tensorflow:global_step/sec: 27.9074\n","INFO:tensorflow:loss = 0.27024662, step = 3900 (3.584 sec)\n","INFO:tensorflow:global_step/sec: 28.1732\n","INFO:tensorflow:loss = 0.25226527, step = 4000 (3.549 sec)\n","INFO:tensorflow:global_step/sec: 28.0383\n","INFO:tensorflow:loss = 0.21234268, step = 4100 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 28.0985\n","INFO:tensorflow:loss = 0.2501368, step = 4200 (3.560 sec)\n","INFO:tensorflow:global_step/sec: 28.0531\n","INFO:tensorflow:loss = 0.217422, step = 4300 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 28.196\n","INFO:tensorflow:loss = 0.18426773, step = 4400 (3.549 sec)\n","INFO:tensorflow:global_step/sec: 28.0734\n","INFO:tensorflow:loss = 0.16114734, step = 4500 (3.560 sec)\n","INFO:tensorflow:global_step/sec: 28.1086\n","INFO:tensorflow:loss = 0.12806816, step = 4600 (3.558 sec)\n","INFO:tensorflow:global_step/sec: 28.1819\n","INFO:tensorflow:loss = 0.17001164, step = 4700 (3.551 sec)\n","INFO:tensorflow:global_step/sec: 28.0385\n","INFO:tensorflow:loss = 0.11126522, step = 4800 (3.563 sec)\n","INFO:tensorflow:global_step/sec: 28.1445\n","INFO:tensorflow:loss = 0.12371929, step = 4900 (3.556 sec)\n","INFO:tensorflow:global_step/sec: 27.9039\n","INFO:tensorflow:loss = 0.06968487, step = 5000 (3.581 sec)\n","INFO:tensorflow:global_step/sec: 28.1177\n","INFO:tensorflow:loss = 0.062086392, step = 5100 (3.556 sec)\n","INFO:tensorflow:global_step/sec: 28.1495\n","INFO:tensorflow:loss = 0.0971788, step = 5200 (3.552 sec)\n","INFO:tensorflow:global_step/sec: 27.8989\n","INFO:tensorflow:loss = 0.07236557, step = 5300 (3.584 sec)\n","INFO:tensorflow:global_step/sec: 27.9387\n","INFO:tensorflow:loss = 0.04531852, step = 5400 (3.589 sec)\n","INFO:tensorflow:global_step/sec: 27.9341\n","INFO:tensorflow:loss = 0.030099254, step = 5500 (3.571 sec)\n","INFO:tensorflow:global_step/sec: 27.7562\n","INFO:tensorflow:loss = 0.027105357, step = 5600 (3.605 sec)\n","INFO:tensorflow:global_step/sec: 27.9282\n","INFO:tensorflow:loss = 0.025342729, step = 5700 (3.578 sec)\n","INFO:tensorflow:global_step/sec: 27.7858\n","INFO:tensorflow:loss = 0.014485734, step = 5800 (3.602 sec)\n","INFO:tensorflow:global_step/sec: 27.8939\n","INFO:tensorflow:loss = 0.019010825, step = 5900 (3.581 sec)\n","INFO:tensorflow:global_step/sec: 28.084\n","INFO:tensorflow:loss = 0.013900392, step = 6000 (3.564 sec)\n","INFO:tensorflow:global_step/sec: 28.0798\n","INFO:tensorflow:loss = 0.0076343734, step = 6100 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 28.0017\n","INFO:tensorflow:loss = 0.0073129777, step = 6200 (3.567 sec)\n","INFO:tensorflow:global_step/sec: 28.121\n","INFO:tensorflow:loss = 0.017224042, step = 6300 (3.558 sec)\n","INFO:tensorflow:global_step/sec: 27.9692\n","INFO:tensorflow:loss = 0.015572787, step = 6400 (3.577 sec)\n","INFO:tensorflow:global_step/sec: 27.5311\n","INFO:tensorflow:loss = 0.024754526, step = 6500 (3.631 sec)\n","INFO:tensorflow:global_step/sec: 27.6055\n","INFO:tensorflow:loss = 0.044960074, step = 6600 (3.622 sec)\n","INFO:tensorflow:global_step/sec: 27.474\n","INFO:tensorflow:loss = 0.034458283, step = 6700 (3.640 sec)\n","INFO:tensorflow:global_step/sec: 28.1422\n","INFO:tensorflow:loss = 0.050663523, step = 6800 (3.552 sec)\n","INFO:tensorflow:global_step/sec: 27.9937\n","INFO:tensorflow:loss = 0.043037403, step = 6900 (3.574 sec)\n","INFO:tensorflow:global_step/sec: 27.9754\n","INFO:tensorflow:loss = 0.01059984, step = 7000 (3.572 sec)\n","INFO:tensorflow:global_step/sec: 27.8723\n","INFO:tensorflow:loss = 0.0065184683, step = 7100 (3.592 sec)\n","INFO:tensorflow:global_step/sec: 28.0498\n","INFO:tensorflow:loss = 0.0046898113, step = 7200 (3.564 sec)\n","INFO:tensorflow:global_step/sec: 28.1061\n","INFO:tensorflow:loss = 0.005297203, step = 7300 (3.555 sec)\n","INFO:tensorflow:global_step/sec: 27.9223\n","INFO:tensorflow:loss = 0.0027374933, step = 7400 (3.585 sec)\n","INFO:tensorflow:global_step/sec: 28.0509\n","INFO:tensorflow:loss = 0.0011431613, step = 7500 (3.563 sec)\n","INFO:tensorflow:global_step/sec: 28.0674\n","INFO:tensorflow:loss = 0.007068994, step = 7600 (3.563 sec)\n","INFO:tensorflow:global_step/sec: 27.9666\n","INFO:tensorflow:loss = 0.0038425943, step = 7700 (3.574 sec)\n","INFO:tensorflow:global_step/sec: 28.2106\n","INFO:tensorflow:loss = 0.0058923597, step = 7800 (3.544 sec)\n","INFO:tensorflow:global_step/sec: 28.1279\n","INFO:tensorflow:loss = 0.00071693654, step = 7900 (3.555 sec)\n","INFO:tensorflow:global_step/sec: 27.7353\n","INFO:tensorflow:loss = 0.0036866488, step = 8000 (3.609 sec)\n","INFO:tensorflow:global_step/sec: 28.0094\n","INFO:tensorflow:loss = 0.0033681896, step = 8100 (3.570 sec)\n","INFO:tensorflow:global_step/sec: 28.0118\n","INFO:tensorflow:loss = 0.0032900656, step = 8200 (3.570 sec)\n","INFO:tensorflow:global_step/sec: 28.0582\n","INFO:tensorflow:loss = 0.0018172106, step = 8300 (3.561 sec)\n","INFO:tensorflow:global_step/sec: 28.098\n","INFO:tensorflow:loss = 0.0021799076, step = 8400 (3.561 sec)\n","INFO:tensorflow:global_step/sec: 28.0871\n","INFO:tensorflow:loss = 0.003227266, step = 8500 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 27.8943\n","INFO:tensorflow:loss = 0.00034689813, step = 8600 (3.585 sec)\n","INFO:tensorflow:global_step/sec: 28.0653\n","INFO:tensorflow:loss = 0.0027198356, step = 8700 (3.559 sec)\n","INFO:tensorflow:global_step/sec: 28.103\n","INFO:tensorflow:loss = 0.0004900593, step = 8800 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 28.0556\n","INFO:tensorflow:loss = 0.0005935285, step = 8900 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 27.9379\n","INFO:tensorflow:loss = 0.0005045766, step = 9000 (3.579 sec)\n","INFO:tensorflow:global_step/sec: 28.0746\n","INFO:tensorflow:loss = 0.0021529081, step = 9100 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 27.8719\n","INFO:tensorflow:loss = 0.09166429, step = 9200 (3.588 sec)\n","INFO:tensorflow:global_step/sec: 28.0932\n","INFO:tensorflow:loss = 0.10093908, step = 9300 (3.560 sec)\n","INFO:tensorflow:global_step/sec: 28.0285\n","INFO:tensorflow:loss = 0.07484934, step = 9400 (3.568 sec)\n","INFO:tensorflow:global_step/sec: 28.1059\n","INFO:tensorflow:loss = 0.031215692, step = 9500 (3.554 sec)\n","INFO:tensorflow:global_step/sec: 28.0319\n","INFO:tensorflow:loss = 0.0069583748, step = 9600 (3.570 sec)\n","INFO:tensorflow:global_step/sec: 28.052\n","INFO:tensorflow:loss = 0.0045569073, step = 9700 (3.566 sec)\n","INFO:tensorflow:global_step/sec: 28.018\n","INFO:tensorflow:loss = 0.0023479136, step = 9800 (3.569 sec)\n","INFO:tensorflow:global_step/sec: 28.1286\n","INFO:tensorflow:loss = 0.003438604, step = 9900 (3.553 sec)\n","INFO:tensorflow:global_step/sec: 27.9768\n","INFO:tensorflow:loss = 0.00094129285, step = 10000 (3.577 sec)\n","INFO:tensorflow:global_step/sec: 28.009\n","INFO:tensorflow:loss = 0.0004952661, step = 10100 (3.569 sec)\n","INFO:tensorflow:global_step/sec: 27.8141\n","INFO:tensorflow:loss = 0.00063685904, step = 10200 (3.597 sec)\n","INFO:tensorflow:global_step/sec: 27.9377\n","INFO:tensorflow:loss = 0.0036731036, step = 10300 (3.578 sec)\n","INFO:tensorflow:global_step/sec: 27.8817\n","INFO:tensorflow:loss = 0.0018529781, step = 10400 (3.586 sec)\n","INFO:tensorflow:global_step/sec: 28.0673\n","INFO:tensorflow:loss = 0.00052499096, step = 10500 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 28.0649\n","INFO:tensorflow:loss = 0.00041927528, step = 10600 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 28.0877\n","INFO:tensorflow:loss = 0.0026503056, step = 10700 (3.557 sec)\n","INFO:tensorflow:global_step/sec: 28.0447\n","INFO:tensorflow:loss = 0.00028714017, step = 10800 (3.569 sec)\n","INFO:tensorflow:global_step/sec: 28.1428\n","INFO:tensorflow:loss = 0.0029764874, step = 10900 (3.553 sec)\n","INFO:tensorflow:global_step/sec: 27.8967\n","INFO:tensorflow:loss = 0.004842891, step = 11000 (3.584 sec)\n","INFO:tensorflow:global_step/sec: 28.1244\n","INFO:tensorflow:loss = 0.00254086, step = 11100 (3.553 sec)\n","INFO:tensorflow:global_step/sec: 27.9702\n","INFO:tensorflow:loss = 0.00025225463, step = 11200 (3.577 sec)\n","INFO:tensorflow:global_step/sec: 28.0838\n","INFO:tensorflow:loss = 0.0025484534, step = 11300 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 28.0621\n","INFO:tensorflow:loss = 0.005296864, step = 11400 (3.561 sec)\n","INFO:tensorflow:global_step/sec: 28.0775\n","INFO:tensorflow:loss = 0.002067845, step = 11500 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 28.0105\n","INFO:tensorflow:loss = 0.0016212574, step = 11600 (3.569 sec)\n","INFO:tensorflow:global_step/sec: 28.0301\n","INFO:tensorflow:loss = 0.004889341, step = 11700 (3.568 sec)\n","INFO:tensorflow:global_step/sec: 27.9053\n","INFO:tensorflow:loss = 0.0002120891, step = 11800 (3.581 sec)\n","INFO:tensorflow:global_step/sec: 28.1568\n","INFO:tensorflow:loss = 0.0027419236, step = 11900 (3.554 sec)\n","INFO:tensorflow:global_step/sec: 28.0543\n","INFO:tensorflow:loss = 0.00018571258, step = 12000 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 28.0141\n","INFO:tensorflow:loss = 0.0018635266, step = 12100 (3.572 sec)\n","INFO:tensorflow:global_step/sec: 28.0317\n","INFO:tensorflow:loss = 0.003987743, step = 12200 (3.564 sec)\n","INFO:tensorflow:global_step/sec: 27.9794\n","INFO:tensorflow:loss = 0.0001609573, step = 12300 (3.575 sec)\n","INFO:tensorflow:global_step/sec: 27.9593\n","INFO:tensorflow:loss = 0.0023673498, step = 12400 (3.577 sec)\n","INFO:tensorflow:global_step/sec: 28.1058\n","INFO:tensorflow:loss = 0.00029689184, step = 12500 (3.559 sec)\n","INFO:tensorflow:global_step/sec: 27.7068\n","INFO:tensorflow:loss = 0.00019880717, step = 12600 (3.609 sec)\n","INFO:tensorflow:global_step/sec: 28.091\n","INFO:tensorflow:loss = 0.00028801823, step = 12700 (3.559 sec)\n","INFO:tensorflow:global_step/sec: 28.0727\n","INFO:tensorflow:loss = 0.18649216, step = 12800 (3.559 sec)\n","INFO:tensorflow:global_step/sec: 27.9986\n","INFO:tensorflow:loss = 0.116449945, step = 12900 (3.573 sec)\n","INFO:tensorflow:global_step/sec: 28.2142\n","INFO:tensorflow:loss = 0.10706282, step = 13000 (3.543 sec)\n","INFO:tensorflow:global_step/sec: 28.1381\n","INFO:tensorflow:loss = 0.021906076, step = 13100 (3.557 sec)\n","INFO:tensorflow:global_step/sec: 28.1217\n","INFO:tensorflow:loss = 0.005219022, step = 13200 (3.555 sec)\n","INFO:tensorflow:global_step/sec: 28.0702\n","INFO:tensorflow:loss = 0.0018132684, step = 13300 (3.562 sec)\n","INFO:tensorflow:global_step/sec: 27.9519\n","INFO:tensorflow:loss = 0.0011531715, step = 13400 (3.576 sec)\n","INFO:tensorflow:global_step/sec: 28.0592\n","INFO:tensorflow:loss = 0.0022143899, step = 13500 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 28.0537\n","INFO:tensorflow:loss = 0.00067349797, step = 13600 (3.563 sec)\n","INFO:tensorflow:global_step/sec: 28.0019\n","INFO:tensorflow:loss = 0.0015057714, step = 13700 (3.571 sec)\n","INFO:tensorflow:global_step/sec: 27.9188\n","INFO:tensorflow:loss = 0.000413737, step = 13800 (3.582 sec)\n","INFO:tensorflow:global_step/sec: 27.8806\n","INFO:tensorflow:loss = 0.0021993623, step = 13900 (3.587 sec)\n","INFO:tensorflow:global_step/sec: 28.2491\n","INFO:tensorflow:loss = 0.00040951956, step = 14000 (3.539 sec)\n","INFO:tensorflow:global_step/sec: 27.6856\n","INFO:tensorflow:loss = 0.0044508902, step = 14100 (3.612 sec)\n","INFO:tensorflow:global_step/sec: 27.6131\n","INFO:tensorflow:loss = 0.00027964255, step = 14200 (3.621 sec)\n","INFO:tensorflow:global_step/sec: 27.9844\n","INFO:tensorflow:loss = 0.00031713152, step = 14300 (3.575 sec)\n","INFO:tensorflow:global_step/sec: 27.9587\n","INFO:tensorflow:loss = 0.0013921999, step = 14400 (3.578 sec)\n","INFO:tensorflow:global_step/sec: 28.1837\n","INFO:tensorflow:loss = 0.0053445376, step = 14500 (3.548 sec)\n","INFO:tensorflow:global_step/sec: 27.9669\n","INFO:tensorflow:loss = 0.002486572, step = 14600 (3.575 sec)\n","INFO:tensorflow:global_step/sec: 27.9136\n","INFO:tensorflow:loss = 0.00030423497, step = 14700 (3.583 sec)\n","INFO:tensorflow:global_step/sec: 27.8976\n","INFO:tensorflow:loss = 0.00021040447, step = 14800 (3.585 sec)\n","INFO:tensorflow:global_step/sec: 27.7408\n","INFO:tensorflow:loss = 0.00023312155, step = 14900 (3.604 sec)\n","INFO:tensorflow:global_step/sec: 28.1268\n","INFO:tensorflow:loss = 0.0037990203, step = 15000 (3.554 sec)\n","INFO:tensorflow:global_step/sec: 28.0512\n","INFO:tensorflow:loss = 0.00019907657, step = 15100 (3.566 sec)\n","INFO:tensorflow:global_step/sec: 27.4498\n","INFO:tensorflow:loss = 0.0013559007, step = 15200 (3.640 sec)\n","INFO:tensorflow:global_step/sec: 27.5518\n","INFO:tensorflow:loss = 0.0044151978, step = 15300 (3.633 sec)\n","INFO:tensorflow:global_step/sec: 27.5777\n","INFO:tensorflow:loss = 0.00012734656, step = 15400 (3.623 sec)\n","INFO:tensorflow:global_step/sec: 27.7872\n","INFO:tensorflow:loss = 0.00067408924, step = 15500 (3.603 sec)\n","INFO:tensorflow:global_step/sec: 28.0844\n","INFO:tensorflow:loss = 0.00015901307, step = 15600 (3.560 sec)\n","INFO:tensorflow:global_step/sec: 28.1887\n","INFO:tensorflow:loss = 0.0004432255, step = 15700 (3.544 sec)\n","INFO:tensorflow:global_step/sec: 27.7831\n","INFO:tensorflow:loss = 0.001657006, step = 15800 (3.599 sec)\n","INFO:tensorflow:global_step/sec: 28.0929\n","INFO:tensorflow:loss = 0.00139208, step = 15900 (3.560 sec)\n","INFO:tensorflow:global_step/sec: 27.8162\n","INFO:tensorflow:loss = 0.000140249, step = 16000 (3.597 sec)\n","INFO:tensorflow:global_step/sec: 28.1674\n","INFO:tensorflow:loss = 0.0021187798, step = 16100 (3.548 sec)\n","INFO:tensorflow:global_step/sec: 28.1057\n","INFO:tensorflow:loss = 0.0008473465, step = 16200 (3.560 sec)\n","INFO:tensorflow:global_step/sec: 28.1082\n","INFO:tensorflow:loss = 0.0018989997, step = 16300 (3.558 sec)\n","INFO:tensorflow:global_step/sec: 27.9793\n","INFO:tensorflow:loss = 0.000109811575, step = 16400 (3.572 sec)\n","INFO:tensorflow:global_step/sec: 28.1436\n","INFO:tensorflow:loss = 0.00020414051, step = 16500 (3.557 sec)\n","INFO:tensorflow:global_step/sec: 28.0792\n","INFO:tensorflow:loss = 0.00011615559, step = 16600 (3.560 sec)\n","INFO:tensorflow:Saving checkpoints for 16667 into ./data_out/check_point/model.ckpt.\n","INFO:tensorflow:global_step/sec: 23.8521\n","INFO:tensorflow:loss = 0.00015542276, step = 16700 (4.193 sec)\n","INFO:tensorflow:global_step/sec: 27.9702\n","INFO:tensorflow:loss = 0.035540532, step = 16800 (3.578 sec)\n","INFO:tensorflow:global_step/sec: 27.8052\n","INFO:tensorflow:loss = 0.19824506, step = 16900 (3.595 sec)\n","INFO:tensorflow:global_step/sec: 27.978\n","INFO:tensorflow:loss = 0.06606908, step = 17000 (3.575 sec)\n","INFO:tensorflow:global_step/sec: 27.7458\n","INFO:tensorflow:loss = 0.031764187, step = 17100 (3.603 sec)\n","INFO:tensorflow:global_step/sec: 27.6464\n","INFO:tensorflow:loss = 0.013176837, step = 17200 (3.617 sec)\n","INFO:tensorflow:global_step/sec: 28.0289\n","INFO:tensorflow:loss = 0.004442256, step = 17300 (3.569 sec)\n","INFO:tensorflow:global_step/sec: 27.9949\n","INFO:tensorflow:loss = 0.0025784955, step = 17400 (3.571 sec)\n","INFO:tensorflow:global_step/sec: 27.8883\n","INFO:tensorflow:loss = 0.0020020006, step = 17500 (3.587 sec)\n","INFO:tensorflow:global_step/sec: 28.0839\n","INFO:tensorflow:loss = 0.002186113, step = 17600 (3.561 sec)\n","INFO:tensorflow:global_step/sec: 28.0094\n","INFO:tensorflow:loss = 0.0025978156, step = 17700 (3.570 sec)\n","INFO:tensorflow:global_step/sec: 27.9207\n","INFO:tensorflow:loss = 0.002166746, step = 17800 (3.578 sec)\n","INFO:tensorflow:global_step/sec: 28.023\n","INFO:tensorflow:loss = 0.0019490208, step = 17900 (3.571 sec)\n","INFO:tensorflow:global_step/sec: 28.084\n","INFO:tensorflow:loss = 0.0003049469, step = 18000 (3.558 sec)\n","INFO:tensorflow:global_step/sec: 27.9163\n","INFO:tensorflow:loss = 0.00024816225, step = 18100 (3.585 sec)\n","INFO:tensorflow:global_step/sec: 28.043\n","INFO:tensorflow:loss = 0.001540813, step = 18200 (3.566 sec)\n","INFO:tensorflow:global_step/sec: 27.9993\n","INFO:tensorflow:loss = 0.0005002279, step = 18300 (3.569 sec)\n","INFO:tensorflow:global_step/sec: 27.9362\n","INFO:tensorflow:loss = 0.00022122583, step = 18400 (3.580 sec)\n","INFO:tensorflow:global_step/sec: 28.0457\n","INFO:tensorflow:loss = 0.00019747211, step = 18500 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 28.072\n","INFO:tensorflow:loss = 0.00022141101, step = 18600 (3.566 sec)\n","INFO:tensorflow:global_step/sec: 27.92\n","INFO:tensorflow:loss = 0.00021543188, step = 18700 (3.580 sec)\n","INFO:tensorflow:global_step/sec: 28.0476\n","INFO:tensorflow:loss = 0.0001855416, step = 18800 (3.563 sec)\n","INFO:tensorflow:global_step/sec: 27.8698\n","INFO:tensorflow:loss = 0.0013062665, step = 18900 (3.591 sec)\n","INFO:tensorflow:global_step/sec: 27.8908\n","INFO:tensorflow:loss = 0.0022689884, step = 19000 (3.585 sec)\n","INFO:tensorflow:global_step/sec: 28.0283\n","INFO:tensorflow:loss = 0.0006525615, step = 19100 (3.566 sec)\n","INFO:tensorflow:global_step/sec: 28.0265\n","INFO:tensorflow:loss = 0.0003510913, step = 19200 (3.570 sec)\n","INFO:tensorflow:global_step/sec: 27.5407\n","INFO:tensorflow:loss = 0.000152169, step = 19300 (3.632 sec)\n","INFO:tensorflow:global_step/sec: 28.0894\n","INFO:tensorflow:loss = 0.0017254351, step = 19400 (3.560 sec)\n","INFO:tensorflow:global_step/sec: 27.9949\n","INFO:tensorflow:loss = 0.0001533677, step = 19500 (3.568 sec)\n","INFO:tensorflow:global_step/sec: 27.8633\n","INFO:tensorflow:loss = 0.0007345945, step = 19600 (3.593 sec)\n","INFO:tensorflow:global_step/sec: 28.1193\n","INFO:tensorflow:loss = 0.00012979527, step = 19700 (3.556 sec)\n","INFO:tensorflow:global_step/sec: 28.0428\n","INFO:tensorflow:loss = 0.0001414489, step = 19800 (3.565 sec)\n","INFO:tensorflow:global_step/sec: 27.8919\n","INFO:tensorflow:loss = 0.0016584921, step = 19900 (3.584 sec)\n","INFO:tensorflow:Saving checkpoints for 20000 into ./data_out/check_point/model.ckpt.\n","INFO:tensorflow:Loss for final step: 0.0017208112.\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow_estimator.python.estimator.estimator.Estimator at 0x7f97a6ca41d0>"]},"metadata":{"tags":[]},"execution_count":22}]},{"cell_type":"code","metadata":{"id":"SZZlFKPt9KVF","colab_type":"code","outputId":"a67fcf6a-f23f-44ff-9398-50b79614b8bc","executionInfo":{"status":"ok","timestamp":1563577306085,"user_tz":-540,"elapsed":743184,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":258}},"source":["eval_result = transformer.evaluate(input_fn=lambda: data.eval_input_fn(eval_input_enc, eval_output_dec, eval_target_dec, conf.DEFINES.batch_size))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n","INFO:tensorflow:Done calling model_fn.\n","INFO:tensorflow:Starting evaluation at 2019-07-19T23:01:41Z\n","INFO:tensorflow:Graph was finalized.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use standard file APIs to check for files with this prefix.\n","INFO:tensorflow:Restoring parameters from ./data_out/check_point/model.ckpt-20000\n","INFO:tensorflow:Running local_init_op.\n","INFO:tensorflow:Done running local_init_op.\n","INFO:tensorflow:Finished evaluation at 2019-07-19-23:01:45\n","INFO:tensorflow:Saving dict for global step 20000: accuracy = 0.86347514, global_step = 20000, loss = 1.9410063\n","INFO:tensorflow:Saving 'checkpoint_path' summary for global step 20000: ./data_out/check_point/model.ckpt-20000\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Ipu6D9eo9Tac","colab_type":"code","outputId":"89ecbf1d-5369-4162-b005-6c4791fd9df0","executionInfo":{"status":"ok","timestamp":1563577306086,"user_tz":-540,"elapsed":741163,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["print('\\nEVAL set accuracy: {accuracy:0.3f}\\n'.format(**eval_result))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["\n","EVAL set accuracy: 0.863\n","\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"rPK0sg1bBsW2","colab_type":"code","outputId":"dc1c1246-6df6-42f2-db80-bd74b1c9229c","executionInfo":{"status":"ok","timestamp":1563577308630,"user_tz":-540,"elapsed":743035,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":289}},"source":["!ls -al ./data_out/check_point"],"execution_count":0,"outputs":[{"output_type":"stream","text":["total 287432\n","drwxr-xr-x 3 root root     4096 Jul 19 23:01 .\n","drwxr-xr-x 3 root root     4096 Jul 19 22:49 ..\n","-rw-r--r-- 1 root root      179 Jul 19 23:01 checkpoint\n","drwxr-xr-x 2 root root     4096 Jul 19 23:01 eval\n","-rw-r--r-- 1 root root 50305252 Jul 19 23:01 events.out.tfevents.1563576570.27319d2b6228\n","-rw-r--r-- 1 root root 43809803 Jul 19 22:49 graph.pbtxt\n","-rw-r--r-- 1 root root 53885788 Jul 19 22:49 model.ckpt-0.data-00000-of-00001\n","-rw-r--r-- 1 root root    10728 Jul 19 22:49 model.ckpt-0.index\n","-rw-r--r-- 1 root root 12830536 Jul 19 22:49 model.ckpt-0.meta\n","-rw-r--r-- 1 root root 53885788 Jul 19 22:59 model.ckpt-16667.data-00000-of-00001\n","-rw-r--r-- 1 root root    10728 Jul 19 22:59 model.ckpt-16667.index\n","-rw-r--r-- 1 root root 12830536 Jul 19 22:59 model.ckpt-16667.meta\n","-rw-r--r-- 1 root root 53885788 Jul 19 23:01 model.ckpt-20000.data-00000-of-00001\n","-rw-r--r-- 1 root root    10728 Jul 19 23:01 model.ckpt-20000.index\n","-rw-r--r-- 1 root root 12830536 Jul 19 23:01 model.ckpt-20000.meta\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Ck97oF0i70cT","colab_type":"code","colab":{}},"source":["def serving_input_receiver_fn():\n","    receiver_tensor = {\n","        'input': tf.placeholder(dtype=tf.int32, shape=[None, conf.DEFINES.max_sequence_length]),\n","        'output': tf.placeholder(dtype=tf.int32, shape=[None, conf.DEFINES.max_sequence_length])    \n","    }\n","    features = {\n","        key: tensor for key, tensor in receiver_tensor.items()\n","    }\n","    return tf.estimator.export.ServingInputReceiver(features, receiver_tensor) # serving_input_receiver_fn의 반환 유형"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"5Sv6_L2RCTdC","colab_type":"code","outputId":"e7104e17-44ef-437c-e6fc-780651ddcd54","executionInfo":{"status":"ok","timestamp":1563577421485,"user_tz":-540,"elapsed":26976,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":275}},"source":["save_model_path = transformer.export_savedmodel( export_dir_base=conf.DEFINES.check_point_path, serving_input_receiver_fn=serving_input_receiver_fn)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["INFO:tensorflow:Calling model_fn.\n","INFO:tensorflow:Done calling model_fn.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n","INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n","INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n","INFO:tensorflow:Signatures INCLUDED in export for Predict: ['indexs', 'serving_default']\n","INFO:tensorflow:Signatures INCLUDED in export for Train: None\n","INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n","INFO:tensorflow:Restoring parameters from ./data_out/check_point/model.ckpt-20000\n","INFO:tensorflow:Assets added to graph.\n","INFO:tensorflow:No assets to write.\n","INFO:tensorflow:SavedModel written to: ./data_out/check_point/temp-b'1563577394'/saved_model.pb\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"rkCDmW8IC7r9","colab_type":"code","outputId":"4676ee7e-45a1-4a3d-945f-ca3ad79a6541","executionInfo":{"status":"ok","timestamp":1563577439161,"user_tz":-540,"elapsed":733,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["print(save_model_path)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["b'./data_out/check_point/1563577394'\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"wCER7vByEN0S","colab_type":"code","outputId":"d582452a-7bb6-444d-fc4d-bbfeeea9d62f","executionInfo":{"status":"ok","timestamp":1563577445845,"user_tz":-540,"elapsed":2601,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":323}},"source":["!ls -al './data_out/check_point/'"],"execution_count":0,"outputs":[{"output_type":"stream","text":["total 287440\n","drwxr-xr-x 5 root root     4096 Jul 19 23:03  .\n","drwxr-xr-x 3 root root     4096 Jul 19 22:49  ..\n","drwxr-xr-x 3 root root     4096 Jul 19 23:03  1563577394\n","-rw-r--r-- 1 root root      179 Jul 19 23:01  checkpoint\n","drwxr-xr-x 2 root root     4096 Jul 19 23:01  eval\n","-rw-r--r-- 1 root root 50305252 Jul 19 23:01  events.out.tfevents.1563576570.27319d2b6228\n","-rw-r--r-- 1 root root 43809803 Jul 19 22:49  graph.pbtxt\n","-rw-r--r-- 1 root root 53885788 Jul 19 22:49  model.ckpt-0.data-00000-of-00001\n","-rw-r--r-- 1 root root    10728 Jul 19 22:49  model.ckpt-0.index\n","-rw-r--r-- 1 root root 12830536 Jul 19 22:49  model.ckpt-0.meta\n","-rw-r--r-- 1 root root 53885788 Jul 19 22:59  model.ckpt-16667.data-00000-of-00001\n","-rw-r--r-- 1 root root    10728 Jul 19 22:59  model.ckpt-16667.index\n","-rw-r--r-- 1 root root 12830536 Jul 19 22:59  model.ckpt-16667.meta\n","-rw-r--r-- 1 root root 53885788 Jul 19 23:01  model.ckpt-20000.data-00000-of-00001\n","-rw-r--r-- 1 root root    10728 Jul 19 23:01  model.ckpt-20000.index\n","-rw-r--r-- 1 root root 12830536 Jul 19 23:01  model.ckpt-20000.meta\n","drwxr-xr-x 2 root root     4096 Jul 19 23:02 \"temp-b'1563577354'\"\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Rc8fHWmAEbeU","colab_type":"code","outputId":"1a1c5fd4-d2ba-453f-8eed-c5e2149c99b7","executionInfo":{"status":"ok","timestamp":1563577458358,"user_tz":-540,"elapsed":2560,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["!ls -al '/content/data_out/check_point/1563577394'"],"execution_count":0,"outputs":[{"output_type":"stream","text":["total 9276\n","drwxr-xr-x 3 root root    4096 Jul 19 23:03 .\n","drwxr-xr-x 5 root root    4096 Jul 19 23:03 ..\n","-rw-r--r-- 1 root root 9483975 Jul 19 23:03 saved_model.pb\n","drwxr-xr-x 2 root root    4096 Jul 19 23:03 variables\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"frvq-oDDFfmw","colab_type":"code","outputId":"2e463eb5-b50d-4638-82f7-db29406ef9fc","executionInfo":{"status":"ok","timestamp":1563577467487,"user_tz":-540,"elapsed":695,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["predic_input_enc, predic_input_enc_length = data.enc_processing([\"남자친구 승진 선물로 뭐가 좋을까?\"], char2idx)\n","\n","predic_output_dec, predic_output_decLength = data.dec_input_processing([\"\"], char2idx)       \n","\n","predic_target_dec = data.dec_target_processing([\"\"], char2idx)      "],"execution_count":0,"outputs":[{"output_type":"stream","text":["100%|██████████| 1/1 [00:00<00:00, 347.10it/s]\n","100%|██████████| 1/1 [00:00<00:00, 411.81it/s]\n","100%|██████████| 1/1 [00:00<00:00, 1763.79it/s]\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"7VFgQzT4G8az","colab_type":"code","outputId":"7b5d7409-0769-47d5-8544-6bd49a2bb1bc","executionInfo":{"status":"ok","timestamp":1563577495633,"user_tz":-540,"elapsed":9663,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":224}},"source":["predictor_fn = tf.contrib.predictor.from_saved_model(export_dir=\"/content/data_out/check_point/1563577394\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["\n","WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n","For more information, please see:\n","  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n","  * https://github.com/tensorflow/addons\n","If you depend on functionality not listed there, please file an issue.\n","\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/predictor/saved_model_predictor.py:153: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.\n","INFO:tensorflow:Restoring parameters from /content/data_out/check_point/1563577394/variables/variables\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"-bXTEUTtRYNZ","colab_type":"code","colab":{}},"source":["predictions = predictor_fn({'input':predic_input_enc, 'output':predic_target_dec})"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"3L6W2KrUHfcI","colab_type":"code","outputId":"c37781cd-00d8-48c6-8b55-bb29f119542d","executionInfo":{"status":"ok","timestamp":1563577539790,"user_tz":-540,"elapsed":717,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["print(predictions)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["{'output': array([[ 7608,  8455,  4673,  6371,  7304,  6118,  3486,     2,    81,\n","         2018,     2,  6118,  3486,     2,     2,     2,     2,     2,\n","            2,  2352, 12202,  3687,     2,     2,     2]])}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"FFWXE8JLH_ZI","colab_type":"code","colab":{}},"source":["def pred_next_string(value, dictionary):\n","    sentence_string = []\n","    is_finished = False\n","\n","    value = value['output']\n","    for v in value:\n","        sentence_string = [dictionary[index] for index in v]\n","\n","    answer = \"\"\n","\n","    for word in sentence_string:\n","        if word == '<END>':\n","            is_finished = True\n","            break\n","\n","        if word != '<PAD>' and word != '<END>':\n","            answer += word\n","            answer += \" \"\n","\n","    return answer, is_finished"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Mc2C0-A9G8dy","colab_type":"code","outputId":"44b57cc6-577f-4449-d8c6-5716dfd54ded","executionInfo":{"status":"ok","timestamp":1563577557563,"user_tz":-540,"elapsed":702,"user":{"displayName":"ChangWook Jun","photoUrl":"https://lh6.googleusercontent.com/-NHLAhAbNfPg/AAAAAAAAAAI/AAAAAAAAAEc/NDLQ8z7-qns/s64/photo.jpg","userId":"00685987924881157185"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["answer = pred_next_string(predictions, idx2char)\n","print(answer)\n"],"execution_count":0,"outputs":[{"output_type":"stream","text":["('평소 에 필요했던 게 좋을 것 같아요 ', True)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"AxTbQweUHYUt","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}