Skip to content

A Denoising Transformer-based Detector-Purificator-Corrector Framework for Spelling Error Correction

License

Notifications You must be signed in to change notification settings

kafan1986/DPCSpell

 
 

Repository files navigation

DPCSpell

A Transformer-based Detector-Purificator-Corrector Framework for Spelling Error Correction of Bangla and Resource Scarce Indic Languages

dpcspell

Running Test

Operating System Requirement Remark
Ubuntu 16.04.7 LTS requirements_ubuntu.txt ✔️ Successful
Ubuntu 18.04.6 LTS (Google Colab) requirements_colab.txt -
Windows 10 requirements_windows.txt -

Get Started

git clone https://github.com/mehedihasanbijoy/DPCSpell.git

or manually download and extract the github repository of DPCSpell.


Environment Setup

Create A Virtual Environment

conda env create -f environment.yml (for Ubuntu 16.04.7 LTS)
or
conda env create -f environment.yml (for Ubuntu 18.04.6 LTS)
or
conda env create -f environment.yml (for Windows 10)

Activate the Environment

conda activate DPCSpell

Prepare SEC Corpora

gdown https://drive.google.com/drive/folders/1vfCAqqXy0ZTL8cPKR-K5q5coBnNv2Zxf?usp=sharing -O ./Dataset --folder

or manually download the folder from here and keep the extracted files into ./Dataset/


Training and Evaluation of DPCSpell

Detector Network

python detector.py --CORPUS "./Dataset/corpus.csv" --HID_DIM 128 --ENC_LAYERS 5 --DEC_LAYERS 5 --ENC_HEADS 8 --DEC_HEADS 8 --ENC_PF_DIM 256 --DEC_PF_DIM 256 --ENC_DROPOUT 0.1 --DEC_DROPOUT 0.1 --CLIP 1 --LEARNING_RATE 0.0005 --N_EPOCHS 100

Purificator Network

python purificator.py --HID_DIM 128 --ENC_LAYERS 5 --DEC_LAYERS 5 --ENC_HEADS 8 --DEC_HEADS 8 --ENC_PF_DIM 256 --DEC_PF_DIM 256 --ENC_DROPOUT 0.1 --DEC_DROPOUT 0.1 --CLIP 1 --LEARNING_RATE 0.0005 --N_EPOCHS 100 

Corrector Network

python corrector.py --HID_DIM 128 --ENC_LAYERS 5 --DEC_LAYERS 5 --ENC_HEADS 8 --DEC_HEADS 8 --ENC_PF_DIM 256 --DEC_PF_DIM 256 --ENC_DROPOUT 0.1 --DEC_DROPOUT 0.1 --CLIP 1 --LEARNING_RATE 0.0005 --N_EPOCHS 100 

Benchmarking Bangla SEC Task

benchmark

About

A Denoising Transformer-based Detector-Purificator-Corrector Framework for Spelling Error Correction

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%