Skip to content

Latest commit

 

History

History
405 lines (325 loc) · 21 KB

README.md

File metadata and controls

405 lines (325 loc) · 21 KB
Raw

TelegramBot-OpenAI-API

🤖 Powered by ChatKeke 🚀

  • A simple-to-use, quick-to-deploy Python-based Telegram bot for OpenAI API
  • 🎙 Transcribed voice messages over Whisper API
    • (auto-transcriptions, translations, and other messages to the bot over TG's voice messages)
  • ☁️ Real-time weather info, weather alerts, and geolocation data via OpenWeatherMap, WeatherAPI and U.S. NWS (weather.gov)
  • 🗺 Geolocation and map lookups via MapTiler API
    • (with weather forecasts around the world in all OpenAI API supported languages)
  • 🧭 Navigation instructions via Openrouteservice API
  • 📊 Daily token usage tracking & rate limiting for API usage / cost management
  • 🔍 Perplexity API models alongside OpenAI models
    • Useful for fact-checking and supplementing OpenAI's cutoff dates
  • 📚 Built-in Elasticsearch RAG steps
    • Increase knowledge rate with your own documents
    • Generate extra insights with the Q&A pair creator
  • 📅 Holiday notifications via Python's holidays module
    • Localized to supported countries, or add your own special reminders
  • 📈 Fetching stock prices via Alpha Vantage API & Yahoo! Finance
    • Real-time access to financial & stock market data
  • 📰 RSS feeds from all available sources
    • Fetch news and more via RSS
  • 🦆 DuckDuckGo searches as context-augmented function calls
    • New! Sub-agentic browsing for enhanced, precise searches!
  • 🌐 Web browsing (page dumps w/ links) as context-augmented function calls
    • With optional domain/IP allow/disallow lists for safety
  • 🧮 Calculator function call module for precise calculations
    • No more fumbling with AI arithmetic hallucinations!
  • 🐳 Dockerized for safety and ease of deplyoment
    • For those who love their Dockers, ready to roll within minutes!

General minimum requirements:

🐧 Installing without Docker on Linux

Prerequisites

  • Tested & working on Python 3.10.12 to 3.12.2
  • Install required Python packages with pip install -r requirements.txt (tested and working with the versions listed in requirements.txt
  • pydub usually requires ffmpeg to be installed separately. Note that neither pydub nor ffmpeg are practically not required if you are not utilizing the voice message/WhisperAPI functionality, but if you are, suggested install (Debian/Ubuntu Linux): sudo apt-get install ffmpeg
  • NOTE: DuckDuckGo searches require lynx to be installed on your system; it needs to be run as a subprocess. (Install on Debian/Ubuntu Linux with: sudo apt-get install lynx)
  1. Clone the repository with:
git clone https://github.com/FlyingFathead/TelegramBot-OpenAI-API/ &&
cd TelegramBot-OpenAI-API/
  1. Install the required packages:
pip install -r requirements.txt
  1. (Recommended) install the optional packages:
  • On Ubuntu/Debian tree Linux systems:
sudo apt-get install -y ffmpeg lynx
  1. Set up your Telegram bot token:
  • Either set your Telegram Bot API token as TELEGRAM_BOT_TOKEN environment variable, or put it into a text file named bot_token.txt inside the config/ directory (= config/bot_token.txt)
  1. Set up your OpenAI API token:
  • Either as OPENAI_API_KEY environment variable or put into a text file named api_token.txt inside the main program directory
  1. Other modules:
  • If you wish to use the OpenWeatherMap API and the MapTiler API for i.e. localized weather data retrieval, set the OPENWEATHERMAP_API_KEY and the MAPTILER_API_KEY environment variables accordingly. You can get the API keys from OpenWeather and MapTiler
  • Additional weather info (moon phases, weather warnings etc) are fetched from WeatherAPI, set the WEATHERAPI_KEY environment variable to use it.
  • If you wish to use the Openrouteservice API for driving instructions, set the OPENROUTESERVICE_API_KEY environment variable from Openrouteservice
  • If you wish to use Perplexity API's supplementary fact-checking with their online models, register at Perplexity.ai, buy some API credits and set your Perplexity API key to environment variable: PERPLEXITY_API_KEY
  1. Further adjustments:
  • Adjust your configuration and settings by editing config/config.ini to your liking
  1. Run:
  • Run the program with: python src/main.py

🐳 Installing Dockerized

Prerequisites

  1. Docker must be installed on your machine.

  2. Telegram Bot API Key and OpenAI API Key:

    • You will need a valid Telegram Bot API key. You can get one by creating a bot with BotFather.
    • You will also need an OpenAI API key. If you don't have one, you can generate it from the OpenAI API page.

Step 1: Clone the Repository

First, clone the repository from GitHub:

git clone https://github.com/FlyingFathead/TelegramBot-OpenAI-API.git
cd TelegramBot-OpenAI-API

Step 2: Run the Setup Script

This project includes a setup script that will guide you through entering your API keys and generating a .env file.

Run the script:

./docker_setup.sh

Follow the instructions provided by the script. It will ask for your OpenAI API key and Telegram Bot API key, validate them, and create a .env file with your credentials.

Step 3: Build the Docker Image

Once your .env file has been created, you need to build the Docker image.

You can run the featured docker_deploy.sh to build the Docker image:

sudo ./docker_deploy.sh

Or, you can build it manually:

sudo docker build -t telegrambot-openai-api .

This will build the image locally based on the Dockerfile in the repository.

Step 4: Run the Docker Container

After the image is successfully built, you can start the bot in a Docker container.

Run the container with the following command:

sudo docker run --env-file .env --name telegrambot-openai-api -d telegrambot-openai-api
  • The -d flag runs the container in detached mode (in the background).
  • The --env-file .env flag injects your API keys into the container.

Step 5: Check the Running Container

You can check if the container is running by using:

sudo docker ps

This will list all running containers. If your bot is running correctly, it should appear in the list.

Step 6: Stopping the Container

If you need to stop the bot, you can do so by running:

sudo docker stop <container_id>

Replace <container_id> with the actual container ID, which you can obtain from the docker ps output.

Additional Steps (Optional)

  • Logs: If you need to view the bot’s logs to troubleshoot any issues, you can use:

    sudo docker logs <container_id>

  • Restart the Container: If you stop the container and want to start it again, you can either run the docker run command again or restart the existing container with:

    sudo docker start <container_id>

Updating the Bot

If the repository receives updates and you want to apply them, follow these steps:

  1. Pull the latest changes from GitHub:

    git pull origin main

  2. Rebuild the Docker image:

    sudo docker build -t telegrambot-openai-api .

  3. Stop the currently running container:

    sudo docker stop <container_id>

  4. Start a new container using the updated image:

    sudo docker run --env-file .env --name telegrambot-openai-api -d telegrambot-openai-api

There is also a docker_deploy.sh script included that aims to make the rebuilding and deploying less of a hassle.

You should now have the TelegramBot-OpenAI-API running in a Docker container, fully connected to both Telegram and OpenAI. Enjoy your bot!

If you run into any issues, consult the logs or reach out on the repository's Issues page.

Updating your config.ini

  • Use the configmerger.py to update old configuration files into a newer version's config.ini. You can do this by saving a copy of your existing config to i.e. a file named myconfig.txt and including in it the lines you want to keep for the newer version.

  • After that, just run i.e. python src/configmerger.py myconfig.txt and all your existing config lines will be migrated to the new one. Works in most cases, but remember to be careful and double-check any migration issues with i.e. diff!

(Example) How to merge and update your existing configuration:

python3 src/configmerger.py myconfig.txt

Changelog

  • v0.75055 - fixes to the html sanitizer (for Telegram's API; better handling of malformed html), using BeautifulSoup4+lxml for parsing now
  • v0.75054 - small fixes and more error catching in calc_module.py
  • v0.75053 - only include eligible territories in U.S. NWS queries
    • list of queried / eligible territories can be set in config.ini under the NWS section
  • v0.75052 - include the details from U.S. National Weather Service on alerts
  • v0.75051 - updated config.ini for configuring NWS weather forecasts & alerts
    • suggested method is to supplement via NWS the additional weather data you need
    • leaving U.S. NWS's weather alerts on in config.ini is highly recommended, even if you have other fetching methods enabled (i.e. OpenWeatherMap), rather be safe than sorry
  • v0.7505 - U.S. NWS (National Weather Service, weather.gov) added as a weather data source
    • for additional information; especially weather alerts
    • all data will be combined from OpenWeatherMap and U.S. NWS sources by default
  • v0.7504 - fixed usage logs and charts directory mapping
  • v0.7503 - improved message formatting & error catching
  • v0.7502 - added docker_setup.sh for easier Docker-based deployment
  • v0.7501 - Dockerfile and better error catching when receiving 401 Unauthorized
  • v0.75 Major refactoring (5. Oct 2024) 👀💦🌀
    • entire project has been tidied up and a lot of bugs fixed while at it
    • python src/main.py to start the bot from here on out
    • massive list of new features, such as:
      • improved logging
      • checking for Elasticsearch on startup; multi-step exception catching
      • Elasticsearch verbosity added for clarity
      • Elasticsearch can now be configured via config.ini
      • enhanced logging (chat + bot data into separate unified logging)
      • chatlogs now have a source for them whenever called via function calls, so that any external data pulls are now logged more clearly
      • overall this update makes the bot very much more easily deployable
  • v0.7431 - potential handling of API keys as textfiles moved under config/ by default
  • v0.743 - config loading changes & streamlining
    • tidying up; all configurations are now under config/
    • imported the new logic for bot token reading from my whisper-transcriber-telegram-bot
    • bot_token.py changed and updated accordingly
    • config_paths.py now has the project-wide configuration for configuration file paths (config.ini, etc...)
    • move any existing bot_token.txt (if used) to config/
    • use configmerger.py to update with your custom configs
    • (more WIP on project restructuring front)
  • v0.742 - Finnish name day RAG step fetch
  • v0.741 - changed to most current Perplexity API model (llama-3.1-sonar-small-128k-online) due to deprecations and updates in their models
  • v0.74 - sub-agentic browsing with DuckDuckGo search engine searches is here!
    • Enabled from config.ini via EnableAgenticBrowsing = True
    • It allows the AI helper to follow links for further info and return relevant results
    • Additional bugfixes to edge cases and their output parsing
  • v0.7373 - small fixes to api_get_duckduckgo_search.py; agentic link following WIP
  • v0.7372 - Further parsing logic changes (markdown_to_html in modules.py)
  • v0.7371 - Improved parsing for markdown/html
  • v0.737 - Changes to Perplexity API inclusion (main model now includes it in the context for better applicability and coherence.)
  • v0.736 - Calculator module calc_module.py added for precision in calculation requests.
  • v0.735 - Lynx website browsing with allow/disallow lists for domains allowed/disallowed to be viewed
  • v0.734 - Now hosting over 100 RSS feeds by default (when RAG triggered)
    • DuckDuckGo searches added as a function call
  • v0.733 - RSS parsing logic streamlined into RAG context
  • v0.7321 - RSS parsing improvements
  • v0.732 - added ElasticSearch RAG function calls to RSS feeds
    • (for i.e. news sources etc, see rss_parser.py)
  • v0.73101 - modularized Perplexity API calls further into a separate handler
    • (see: perplexity_handler.py)
  • v0.731 - added Yahoo! Finance as an API function call for stock price searches (requires the yfinance pip package)
  • v0.730 - added Alpha Vantage API function calling to fetch stock prices in real time (requires Alpha Vantage API key)
  • v0.729 - switched to gpt-4o-mini in default configurations instead of gpt-3.5-turbo (newer, cheaper, better)
  • v0.728 - more edge case handling when fetching multi-API weather data
  • v0.727 - fixed WeatherAPI & weather fetching edge case API glitches (returning odd values, etc)
  • v0.726 - switched to OpenAI API from langdetect when using i.e. Perplexity API for information queries
    • (detect_language function in api_perplexity_search.py)
    • this is for better accuracy in multilingual environments
    • => less responses in wrong language when a translation path is needed for the end-user
    • much more accurate than langdetect and can be further leveraged with model selection
  • v0.7251 - small robustness improvements & fixes to the api_key.py module
  • v0.725 - additional Perplexity API call + translation adjustments
  • v0.724 - splitting logic for lengthier Perplexity API responses
  • v0.723 - added jitter, modified retry logic in Perplexity API+translation requests
  • v0.7201 - added Perplexity API model configuration to config.ini
  • v0.72 - improved error catching + messaging with Perplexity's API
  • v0.71 - holiday mentions via Python's holidays module (can be extended)
  • v0.708 - improved astronomy data combining via WeatherAPI
  • v0.707 - code cleanup + enhancements to Perplexity API handling
  • v0.706 - further weather fetching options; additional country-based data fetching
  • v0.705 - improved weather data combining; small tweaks
  • v0.703 - Language translations and tweaks to WeatherAPI data fetching
  • v0.70 - WeatherAPI support added, to enable, get an API key from weatherapi.com
  • v0.61 - improved handling of weather and time/data data globally
  • v0.60 - url info pre-parsing and additional info fetching for media sources, i.e. with yt-dlp
  • v0.59 - custom function calling via Elasticsearch RAG (if enabled)
  • v0.58.4 - more parsing & formatting fixes
  • v0.58.3 - parsing, formatting & chunking adjustments
  • v0.58.2 - improved formatting in pplx API calls
  • v0.58.1 - improved markdown parsing in translated Perplexity API calls
  • v0.58 - chunking, parsing and other small fixes
  • v0.57.5 - changes made to Perplexity API handling; new sonar-online models
  • v0.57.1 - improved fallbacks on external API calls like Perplexity API
  • v0.57 - improved error catching & failsafe fallbacks
  • v0.56 - Added Elasticsearch support for RAG - use the ElasticsearchEnabled flag in new config.ini (set to True or False to enable or disable)
  • v0.55.4 - API call tracking, extra wait times if needed for external API calls
  • v0.55.3 - reply activity, better chunking logic
  • v0.55 - better reply animation handling
  • v0.52 - more accurate weather data globally via the OpenWeatherMap API & Maptiler API
  • v0.51 - "smart" chunking of external requests to mitigate timeout-related issues
  • v0.50.3 - langdetect & handling fixes
  • v0.50.2 - typing animation on replies, adjustments on processing logic
  • v0.50.1 - langdetect on auto-assessing translation requirements
  • v0.50 - Custom function call: Perplexity API fact-checking
  • v0.49.1 - Modularity adjustments
  • v0.48 - Openrouteservice API implementation
  • v0.47 - more token counting & polling logic fixes
  • v0.46.2 - fixes to token count & polling logic
  • v0.46 - rewrote the polling logic on daily token count resets
  • v0.45 - /usagechart feature added for utilization charts (requires matplotlib)
  • v0.44 - API function calling, OpenWeatherMap API weather searches and MapTiler API geolookup
  • v0.43.2 - Fixed a small bug in daily token caps
  • v0.43.1 - Better error catching
  • v0.43 - New admin commands: /setsystemmessage <message> (valid until bot restart) and /resetsystemmessage (reset from config.ini)
  • v0.42 - /reset command added for bot reset. Set ResetCommandEnabled and AdminOnlyReset flags in config.ini accordingly.
  • v0.41 - modularized text message handling to text_message_handler.py and voice message handling to voice_message_handler.py
  • v0.40 - session timeout management for compacting chat history (see config.ini => SessionTimeoutMinutes, MaxRetainedMessages)
  • v0.39.5 - small fixes to OpenAI API payload implementation
  • v0.39.4 - modularized log_message & rotate_log_file (log file handling) => modules.py
  • v0.39.3 - modularized check_global_rate_limit => modules.py
  • v0.39.2 - text style parsing and WhisperAPI STT pre-processing for the model improved
  • v0.39 - better parsing for codeblocks, html and other markups, modularized more; see modules.py
  • v0.38 - keep better record of daily token usage, streamlined (note: you will need to clear out your existing token_usage.json, the file structure has changed from the previous version)
  • v0.37 - better enforcing of voice msg limits
  • v0.36 - bot command fixes and adjustments
  • v0.35 - modularized bot commands to bot_commands.py, fixed configmerger.py version
  • v0.34 - added configmerger.py to ease updating the bot (merge old configuration flags with new versions)
  • v0.33 - more performance fixes and added+unified async functionalities
  • v0.32 - Daily token counter reset polling & small bugfixes
  • v0.31 - Context memory token counter adjusted & fixed to be more precise
  • v0.30 - Whisper API interaction fine adjustments & small fixes
  • v0.29 - WhisperAPI transcriptions via voice messages added
  • WhisperAPI voice messages use the same OpenAI API token as the regular text chat version
    • see the config.ini to turn the option on or off
    • WIP for additional transcription features
  • v0.28 - customizable /start greeting in config.ini
  • v0.27 - added /usage command to track token usage (for bot owner only, 0 to disable in config.ini)
  • v0.26 - added separate chat logging and a global limiter functionality for requests/min (see config.ini)
  • v0.25 - daily token usage limit functionality
    • added a functionality to set daily token usage limits (for bot cost control), see config.ini
    • modularized extra utils (startup msg etc) into utils.py
  • v0.24 - bug fixes & rate limit pre-alpha
  • v0.23 - option to log to file added, see new logging options in config.ini
  • v0.22 - escape_markdown moved into a separate .py file, it was unused anyway
  • v0.21 - Comprehensive Refactoring and Introduction of Object-Oriented Design
    • Implemented object-oriented programming principles by encapsulating bot functionality within the TelegramBot class.
    • Refined code structure for improved readability, maintainability, and scalability.
  • v0.20 - modularization, step 1 (key & token reading: api_key.py, bot_token.py)
  • v0.19 - timeout error fixes, retry handling; Timeout value added to config.ini
  • v0.18 - model temperature can now be set in config.ini
  • v0.17 - time & date stamping for better temporal awareness
  • v0.16 - /help & /about
  • v0.15 - chat history context memory (trim with MAX_TOKENS)
  • v0.14 - bug fixes
  • v0.13 - parsing/regex for url title+address markdowns
  • v0.12 - more HTML regex parsing from the API markdown
  • v0.11 - switched to HTML parsing
  • v0.10 - MarkdownV2 tryouts (code blocks + bold is mostly working)
  • v0.09 - using MarkdownV2
  • v0.08 - markdown for bot's responses
  • v0.07 - log incoming and outgoing messages
  • v0.06 - API system message fixed
  • v0.05 - retry, max retries, retry delay
  • v0.04 - chat history trimming

Contribute

  • All contributions appreciated! Feel free to also post any bugs and other issues on the repo's "Issues" page.
  • Don't forget to star it if you like it. :-)

About