3_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ZapZap Awards!\n",
    "\n",
    "### Bem vindo ao ZapZap Awards! Use as setinhas ← ↑ → ↓ pra navegar pelas paginas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn as sk\n",
    "from collections import defaultdict\n",
    "import random\n",
    "import requests\n",
    "import json, string\n",
    "import settings\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline\n",
    "pd.set_option('display.max_colwidth', -1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(f'{settings.GROUP_NAME}_messages.csv')\n",
    "df.datetime = df.datetime.str.replace(' ', 'T')\n",
    "df.loc[df.text.isnull(), 'text'] = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Normalizing dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import string\n",
    "def sanitize(text):\n",
    "    text = text.lower()\n",
    "    text = re.sub(r'\\bkkk+', 'kkkk', text)\n",
    "    text = re.sub(r'\\b(h[aeui]*){3,}', 'hahaha', text)\n",
    "    return re.sub('[{}]'.format(string.punctuation), '', text)\n",
    "\n",
    "def shortify_links(text):\n",
    "    return re.sub(r'(https?://.*?/.{10})(.*?)( |$)', r'\\1[...]', text)\n",
    "\n",
    "print(sanitize('Todo dia, sièges. hahahahahaha haehaehaehea kkkkkkkk VAMO Q VAMO! 😀')) # TODO: enrich examples\n",
    "df['stext'] = df.text.apply(sanitize)\n",
    "df['short_links_text'] = df.text.apply(shortify_links)\n",
    "df['datetime'] = pd.to_datetime(df.datetime)\n",
    "df['date'] = df.datetime.dt.date\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center> Estatisticas por Pessoa </center>\n",
    "\n",
    "Antes de comecarmos com os awards, vamos levantar umas estatisticas interessantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zapzap.stats import Stats\n",
    "stats = Stats(df, STOP_WORDS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Horas favoritas\n",
    "\n",
    "Aqui temos o numero de mensagens q cada um mandou ao longo do ano, ao longo das 24h de um dia\n",
    "\n",
    "Tem gente q nao mandou uma unica mensagem as 5 da manha (no meu caso, foi shifted, para as 7)!\n",
    "\n",
    "(Aperte ↓ pra entrar nesse topico) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.favorite_hours()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Palavras caracteristicas\n",
    "\n",
    "Quais as palavras que voce falou e que ninguem mais usa?\n",
    "\n",
    "Omiti os nomes das pessoas nesse ae. Quero ver se vcs conseguem adivinhar quem eh quem (pelo menos os seus proprios)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "stats.characteristic_person()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center> Estatisticas do Surfe </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Palavras favoritas\n",
    "\n",
    "As palavras mais usadas no surfe, por mes.\n",
    "\n",
    "Primeiro, filtradas por palavras [comuns](https://raw.githubusercontent.com/stopwords-iso/stopwords-pt/master/stopwords-pt.txt)\n",
    "\n",
    "Depois, sem filtros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "stats.group_most_common_words(print_word_count=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Agora, sem mais delongas..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "# Single Awards\n",
    "\n",
    "Vamos a eles... os awards individuais! Ô familia, quarta feira abencoada...\n",
    "\n",
    "## Troféu Metalinguistico - SAIU!\n",
    "Quem falou mais as palavras 'surf' e 'night'\n",
    "## Troféu Early riser - SAIU\n",
    "Quem manda a primeira mensagem do dia (>6am)\n",
    "## Troféu Wanderley - SAIU\n",
    "Quem grita mais? Quem usou mais caps lock em 2017\n",
    "## Troféu Machado de Assis\n",
    "Quem tem o maior vocabulario? distinct words / words\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Troféu Pagerank\n",
    "O mais citado\n",
    "## Troféu Feliz\n",
    "Quem ri mais?\n",
    "## Troféu Fanfarrao\n",
    "Quem mudou mais o subject do grupo\n",
    "## Troféu Audiófilo\n",
    "Quem mandou mais audio\n",
    "## Troféu AlfiNET\n",
    "Quem mandou mais imagens/videos\n",
    "## Troféu Vacuo\n",
    "Quem mandou a ultima mensagem da conversa (uma conversa) = 3h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Segundo round\n",
    "\n",
    "## Troféu NDP (sponsor Dibob)\n",
    "Autoexplicativo\n",
    "## Troféu Popstar\n",
    "Quem foi mais chamado pelo nome próprio/apelidos\n",
    "## Troféu Atention seeker\n",
    "Mais chamou alguém por nome proprio/apelidos\n",
    "## Troféu Amizade \n",
    "Quem mais xingou os amiguinhos\n",
    "## Troféu Chulo (sponsor Chulé)\n",
    "Quem mais falou palavrão\n",
    "## Trofeu Mais Que Mil Palavras\n",
    "Quem mais usou emoji"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zapzap.awards import Awards\n",
    "awards = Awards(df, group_name='surf na night')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Troféu Metalinguistico 🏆\n",
    "\n",
    "Quem falou mais as palavras 'surf', 'night' e variantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.root.setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "awards.metalinguistic()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Com 19 citacoes, o ouro vai pro **BODE**! Prata pra mim e Bronze pro Chule! \n",
    "\n",
    "Eis ai um verdadeiro surfista noturno, um grande abracador da causa, sempre esta na atividade, estronda, farpacao... Um abraco mano debo! Comecando os trabalhos!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Troféu Early riser 🏆\n",
    "Quem mandou mais vezes a primeira mensagem do dia (>6am)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "awards.early_riser()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Mas que momento! Temos claramente a familia caprina vencendo no quesito quem cedo madruga. Mano Jay vai levar uma medalha de bronze pra casa tambem!\n",
    "\n",
    "Ceia, uma criatura naturalmente noturna, conseguiu empatar com o nosso trio ternura do sumico. Luke, David e o lendario Guilha.\n",
    "\n",
    "Gostaria de enfatizar que ganhei do Law e do Kim. Q minha fama de dorminhoco seja revista!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Troféu Wanderley 🏆\n",
    "Quem grita mais? Quem usou mais caps lock em 2017"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "awards.shouts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Cara! Achei esse resultado mt curioso! \n",
    "\n",
    "Kimzola ganhou em primeiro lugar DISPARADO! \n",
    "\n",
    "Em segundo lugar o mano CEIA, quem diria cara. Ele fala pouco, mas fala gritando. \n",
    "\n",
    "E o Dibob, provavelmente o favorito, conseguiu chegar no top 3.\n",
    "\n",
    "Parabens aos envolvidos!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "# To be developed...\n",
    "<img src=\"https://gcn.com/~/media/GIG/GCN/Redesign/Generic/AgileDevelopment.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "############################################################################################### APELIDAS #######\n",
    "\n",
    "#display(df[df.stext.str.contains(r'\\b(?:{})'.format(r'fre'), regex=True)])\n",
    "df.stext.str.extract(r'.*\\b({})'.format(r'fr.*?\\b'), expand=False).dropna().unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df.stext.str.contains('.*raf', regex=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df[df.stext.str.match(r'.*\\b({})'.format(r'.*o cao\\b'))].head(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Single Stats\n",
    "\n",
    "## Favorite Hours\n",
    "Normalizado por total de mensagens\n",
    "## Palavras caracteristicas\n",
    "As palavras que so vc fala\n",
    "\n",
    "\n",
    "# Surf Stats\n",
    "\n",
    "## Palavras favoritas\n",
    "\n",
    "## Mensagens por hora/weekday\n",
    "\n",
    "\n",
    "# Surf Awards\n",
    "\n",
    "## Dia Mar Sinistro\n",
    "Dia com maior numero de mensagens\n",
    "## Combo\n",
    "Dias consecutivos com mais de n mensagens\n",
    "## AntiCombo\n",
    "Dias consecutivos com 0 mensagens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Surf Stats\n",
    "\n",
    "## Palavras favoritas\n",
    "\n",
    "## Mensagens por hora/weekday"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<style>\n",
    " .chat {\n",
    "     display: flex;\n",
    "     flex-direction: column;\n",
    " }\n",
    " .box-right {\n",
    "     display: flex;\n",
    "     justify-content: flex-end;\n",
    "     \n",
    " }\n",
    " .box-left {\n",
    "     display: flex;\n",
    "     justify-content: flex-start;\n",
    " }\n",
    " .box-right-content {\n",
    "    min-height: 30px;\n",
    "    max-width: 40%;\n",
    "    color: white;\n",
    "    background: #ff974b;\n",
    "    margin-right: 7px;\n",
    "    padding: 10px;\n",
    "    box-shadow: 1px 1px 3px 0px #CCC;\n",
    "    border-radius: 10px;\n",
    "    margin-bottom: 5px;\n",
    " }\n",
    " .box-left-content {\n",
    "    min-height: 30px;\n",
    "    max-width: 40%;\n",
    "    background-color: #EDEDED;\n",
    "    color: #4A4A4A;\n",
    "    margin-right: 7px;\n",
    "    padding: 10px;\n",
    "    box-shadow: 1px 1px 3px 0px #CCC;\n",
    "    border-radius: 10px;\n",
    "    margin-bottom: 5px;\n",
    " }\n",
    " .chat-text {\n",
    "    margin:0;\n",
    " }\n",
    " .chat-time {\n",
    "    opacity: 0.6;\n",
    "    font-size: 0.917em;\n",
    "    line-height: 1.167em;\n",
    "    margin:0 !important;\n",
    " }\n",
    " .box-info {     \n",
    "     display: flex;\n",
    "     justify-content: center;\n",
    " }\n",
    " .box-info-content {     \n",
    "    background-color: #333;\n",
    "    opacity: 0.5;\n",
    "    padding: 5px;\n",
    "    min-width: 125px;\n",
    "    box-shadow: 1px 1px 3px 0px #CCC;\n",
    "    border-radius: 10px;\n",
    "    color: #FFF;\n",
    "    text-align: center;\n",
    "         font-size: 0.917em;\n",
    "\n",
    "\n",
    " }\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def owner_msg(text, time):\n",
    "    return f'''<div class=\"box-right\">\n",
    "                 <div class=\"box-right-content\"> <p class=\"chat-text\">{text}</p> <p class=\"chat-time\">{time}</p></div>\n",
    "               </div>'''\n",
    "def other_msg(author, text, time):\n",
    "    return f'''<div class=\"box-left\">\n",
    "                 <div class=\"box-left-content\"> <p class=\"chat-text\">{author}: {text}</p> <p class=\"chat-time\">{time}</p></div>\n",
    "               </div>'''\n",
    "def info_msg(text):\n",
    "    return f'''<div class=\"box-info\">\n",
    "                     <div class=\"box-info-content\"> {text} </div>\n",
    "               </div>'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_chat(df, owner):\n",
    "    html = ''\n",
    "    html += info_msg('A very nice chat!')\n",
    "    for idx, author, text, datetime in df[['name', 'text', 'datetime']][:50].to_records():\n",
    "        html += other_msg(author, text, datetime) if author != owner else owner_msg(text, datetime)\n",
    "    from IPython.display import display, HTML\n",
    "    display(HTML(html))\n",
    "\n",
    "display_chat(df, df.name[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}