-brown?logo=linux&style=for-the-badge){kind=icon}
An app to download novels from online sources and generate e-books.
Discord: https://discord.gg/wMECG2Q
Telegram: https://t.me/epub_smelter_bot
- Lightnovel Crawler
This application uses Calibre to convert ebooks.
Install it from https://calibre-ebook.com/download
Without it, you will only get output in epub, text, and web formats.
For macOS, you need to manually add the path to Calibre.
Before starting lncrawl, use the command:
$ export PATH="$PATH:/Applications/calibre.app/Contents/MacOS"If you used a folder other than Applications during installation, replace /Applications/ with your path to Calibre.
⏬ Windows: lncrawl.exe ~ 25MB
In Windows 8, 10 or later versions, it might say that
lncrawl.exeis not safe to dowload or execute. You should bypass/ignore this security check to execute this program.
⏬ Linux: lncrawl ~ 30MB
It is recommended to install via pip if you are on Linux
⏬ To get older versions visit the Releases page
📦 A python package named lightnovel-crawler is available at pypi.
Make sure you have installed Python v3.8 or higher and have pip enabled. Visit these links to install python with pip in Windows, Linux and MacOS. Feel free to ask on the Discord server if you are stuck.
To install this app or to update installed one via pip, just run:
$ pip install -U lightnovel-crawlerIn some cases you have to use python3 -m pip or pip3 or python -m pip. And you do not need --user option, if you are running from root.
Next, open your terminal and enter:
$ lncrawl
To view extra logs, use:
lncrawl -lll
If you want to get the cutting-edge (sometimes unstable) from the dev branch, you can get it by:
The master branch contains the latest stable code. If you can not wait for it to be released in the PyPi, you can get it like this:
$ pip install -U git+https://github.com/dipu-bd/lightnovel-crawler.git#egg=lightnovel-crawler
The dev branch contains cutting-edge, sometimes unstable changes. To install it:
$ pip install -U https://github.com/dipu-bd/lightnovel-crawler/tarball/refs/heads/dev#egg=lightnovel-crawler
Docker is a convenient way to run it anywhere.
- First clone the project.
$ git clone https://github.com/dipu-bd/lightnovel-crawler
- Build docker:
$ cd lightnovel-crawler
$ docker build -t lncrawl -f ./scripts/Dockerfile .
- Run commands using docker:
$ mkdir ~/Lightnovels
$ docker run -v ~/Lightnovels:/home/appuser/app/Lightnovels -it lncrawl
You can setup alias to the above command in your terminal's profile to run using single a single-word command.
Please read before proceeding:
- It is not guaranteed that the app will run smoothly in all devices.
- It may take a long time to install depending on your mobile processor.
- It is recommended to use the bots on either Discord or Telegram if you are on mobile.
📱 Using Termux, you can run this app in your android phones too. Follow this instructions:
- Install Termux from playstore.
- Open the app and run these commands one by one:
pkg upgradepkg install python libxml2 libxslt libjpeg-turbo rust python-lxml python-grpciopip install -U pip wheel setuptoolspip install lightnovel-crawlertermux-setup-storagecd ~/storage/downloadslncrawl
- You can navigate up using Vol UP + W and down using Vol UP + S.
When there is a new update available, you can install it just by running pip install -U lightnovel-crawler. You will not have to run all the above commands again.
PyDroid
You can also use PyDroid in Android phones. Check this discussion for a custom script to run the app: dipu-bd#1137
Join our server: https://discord.gg/7A5Hktx
Or, visit this link to install discord bot to your own server: https://discordapp.com/oauth2/authorize?client_id=537526751170002946&permissions=51264&scope=bot
Visit this link to get started with the telegram bot: https://t.me/epub_smelter_bot
Send !help to open the bot help message.
Simply fill out the environment variables and you get a running instance.
- First clone the repository:
$ git clone https://github.com/dipu-bd/lightnovel-crawler- Open command prompt inside of the project folder and install requirements:
$ pip install -r requirements.txt- Run the program (use python v3.8 or higher):
$ python lncrawlThere are two chatbots available at this moment: Telegram and Discord. To run your own bot server, follow these instructions:
- Clone this repository
$ git clone https://github.com/dipu-bd/lightnovel-crawler-
Install calibre for pdf, mobi etc. formats.
-
Install requirements
$ pip3 install -r requirements.txt-
Copy
.env.examplefile to.envfile. Edit this file and give your API credentials here. -
To run the discord bot:
$ python3 lncrawl --bot discord --shard-id 0 --shard-count 1- To run the telegram bot
$ python3 lncrawl --bot telegramThere is a start.sh script to run a bot in ubuntu servers. It will basically execute the python3 lncrawl and send the task to run in background. I use it to run my discord bot in the server.
$ lncrawl -h
================================================================================
╭╮╱╱╱╱╱╱╭╮╱╭╮╱╱╱╱╱╱╱╱╱╱╱╱╭╮╱╭━━━╮╱╱╱╱╱╱╱╱╱╭╮
┃┃╱╱╱╱╱╱┃┃╭╯╰╮╱╱╱╱╱╱╱╱╱╱╱┃┃╱┃╭━╮┃╱╱╱╱╱╱╱╱╱┃┃
┃┃╱╱╭┳━━┫╰┻╮╭╋━╮╭━━┳╮╭┳━━┫┃╱┃┃╱╰╋━┳━━┳╮╭╮╭┫┃╭━━┳━╮
┃┃╱╭╋┫╭╮┃╭╮┃┃┃╭╮┫╭╮┃╰╯┃┃━┫┃╱┃┃╱╭┫╭┫╭╮┃╰╯╰╯┃┃┃┃━┫╭╯
┃╰━╯┃┃╰╯┃┃┃┃╰┫┃┃┃╰╯┣╮╭┫┃━┫╰╮┃╰━╯┃┃┃╭╮┣╮╭╮╭┫╰┫┃━┫┃
╰━━━┻┻━╮┣╯╰┻━┻╯╰┻━━╯╰╯╰━━┻━╯╰━━━┻╯╰╯╰╯╰╯╰╯╰━┻━━┻╯
╱╱╱╱╱╭━╯┃ v3.7.2
╱╱╱╱╱╰━━╯ 🔗 https://github.com/dipu-bd/lightnovel-crawler
--------------------------------------------------------------------------------
usage: lncrawl [options...]
lightnovel-crawler [options...]
options:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-l Set log levels. (-l = warn, -ll = info, -lll = debug).
--log-file [FILE] To store application logs to a file.
--list-sources Display a list of available sources.
--crawler [FILES ...]
Load additional crawler files.
-s URL, --source URL Profile page url of the novel.
-q STR, --query STR Novel query followed by list of source sites.
-x [REGEX], --sources [REGEX]
Filter out the sources to search for novels.
--login USER PASSWD User name/email address and password for login.
--format E [E ...] Define which formats to output. Default: all.
--add-source-url Add source url at the end of each chapter.
--single Put everything in a single book.
--multi Build separate books by volumes.
-o PATH, --output PATH
Path where the downloads to be stored.
--filename NAME Set the output file name
--filename-only Skip appending chapter range with file name
-f, --force Force replace any existing folder.
-i, --ignore Ignore any existing folder (do not replace).
--all Download all chapters.
--first [COUNT] Download first few chapters (default: 10).
--last [COUNT] Download last few chapters (default: 10).
--page START [STOP. ...]
The start and final chapter urls.
--range FROM TO., --index FROM TO., --chapter FROM TO.
The start and final chapter indexes.
--volumes [N ...] The list of volume numbers to download.
--chapters [URL ...] A list of specific chapter urls.
--proxy-file FILE Proxies as SCHEME://HOST:PORT@USER:PASSWORD format in
each line. All except HOST are optional
--auto-proxy Use some free proxies from https://free-proxy-
list.net/
--bot {console,telegram,discord,lookup}
Select a bot. Default: console.
--shard-id [SHARD_ID]
Discord bot shard id (default: 0)
--shard-count [SHARD_COUNT]
Discord bot shard counts (default: 1)
--selenium-grid URL Selenium Grid URL for Chrome Webdriver
--suppress Suppress all input prompts and use defaults.
--ignore-images Ignore images in chapters when downloading.
--close-directly Do not prompt to close at the end for windows
platforms.
--resume [NAME/URL] Resume download of a novel containing in
/home/runner/work/lightnovel-crawler/lightnovel-
crawler/Lightnovels
ENV [chatbots only] Pass query string at the end of all
options. It will be use instead of .env file. Sample:
"BOT=discord&DISCORD_TOKEN=***&LOG_LEVEL=DEBUG"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~
--------------------------------------------------------------------------------
🔗 https://github.com/dipu-bd/lightnovel-crawler/issues
================================================================================Open your console and type lncrawl --version first to check if you have installed it properly.
Here are some example usage of the app:
-
To start an interactive session:
lncrawl -
To download using an url:
lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/ -
To search novels:
lncrawl -q "Strongest Sword God" -
To search novels from selected sources:
lncrawl -q "Strongest Sword God" --sources -
To download all chapters:
lncrawl --all -
To download first 25 chapters:
lncrawl --first 25 -
To download all between two chapters:
lncrawl --range 10 30 -
To download all between two chapter links:
lncrawl -s https://novelfull.com/release-that-witch.html --chapters https://novelfull.com/release-that-witch/chapter-6-training-part-i.html https://novelfull.com/release-that-witch/chapter-8-months-of-the-demons-part-1.html -
To download a specific volumes:
lncrawl --volumes 2 3 -
To define output path:
lncrawl -o "D:\Lightnovels\reincarnation-of-the-strongest-sword-god" -
To delete the output folder if exists:
lncrawl -f -
To ignore the output folder if exists:
lncrawl -i -
To resume download where is has been left previously:
lncrawl -i -
To specify output formats:
lncrawl --format epub pdf mobi -
To display list of supported sources:
lncrawl --list-sources -
If you provide an option in the argument, it will skip it in the interactive session. If you want to disable all interactive prompts, pass
--suppressat the end. -
You can stack up options like this:
lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/ -o "D:\Lightnovels\reincarnation-of-the-strongest-sword-god" --last 50 -i --format pdf --suppress
Visit the discussions page for more information. You can also post your query there too.
Login to www.wuxiaworld.com
Follow this guide to know how to login: dipu-bd#1360
You are very welcome to contribute in this project. You can:
- create new issues pointing out the bugs.
- solve existing issues.
- add your own sources.
- add new output formats.
- create new bots.
- Use
lncrawl --bot lookupfirst to auto-generate your crawler from an existing template. - Check inside the
sources/_examples. Read all the comments of all the files. And pick the one you like. - You can find plenty examples in the
sourcesfolder. Try to check the latest ones - Put your source file inside the language folder.
The
enfolder has too many files, therefore it is grouped using the first letter of the domain name. - Before making commit format files using
blakeformatter, and usescripts/lint.shorscripts/lint.batto check linting issues.
- Create a new bot file using
bots/_sample.pyas template. - Import bot to
bots/__init__.pyfile.
Request new one by creating a new issue.
We are supporting 406 sources and 353 crawlers.
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://arnovel.me/ | 23 |   |
| 🔍 | https://kolnovel.com/ | 1 | |
| https://rewayat.club/ | 8 | |
| Source URL | Version | Contributors | |
|---|---|---|---|
| https://cclawtranslations.home.blog/ | 71 |  |
|
| https://domentranslations.wordpress.com/ | 71 | |
|
| 🔍🔑 | https://novelasligeras.net/ | 3 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍🖼️ | https://anime-sama.fr/ | 4 |  |
| 🔍 | https://chireads.com/ | 10 | |
| 🔍 | https://lightnovelfr.com/ | 9 |  |
| 🤖🔍 | https://lnmtlfr.com/ | 9 | |
| https://noveldeglace.com/ | 2 | ||
| https://xiaowaz.fr/ | 3 | |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://blnovels.net/ | 6 | |
| 🔍 | https://centralnovel.com/ | 9 | |
| https://www.ceunovel.com/ | 1 | |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍🖼️ | https://bestmanga.club/ | 75 |  |
| https://bookhamster.ru/ | 3 |  |
|
| https://ifreedom.su/ | 3 | |
|
| https://jaomix.ru/ | 9 |  |
|
| 🔍 | https://litnet.com/ | 6 | |
| https://renovels.org/ | 2 | |
|
| 🔑 | https://tl.rulate.ru/ | 7 |  |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🤖🔍 | https://docln.net/ | 11 | |
| 🤖🔍 | https://ln.hako.vn/ | 11 | |
| 🤖🔍 | https://truyenfull.vn/ | 5 | |
| 🤖🔍 | https://truyentr.info/ | 5 | |
- JSON
- EPUB
- TEXT
- WEB
- DOCX
- MOBI
- RTF
- TXT
- AZW3
- FB2
- LIT
- LRF
- OEB
- PDB
- RB
- SNB
- TCR
 |