Skip to content

Latest commit

 

History

History
41 lines (32 loc) · 1.77 KB

README.md

File metadata and controls

41 lines (32 loc) · 1.77 KB
Raw

IL-NER

  • This annotated corpora has been developed under the Bhashini project funded by Ministry of Electronics and Information Technology (MeitY), Government of India. We thank MeitY for funding this work.

  • This dataset is licensed under Creative Commons Attribution 4.0 (CC-BY-4.0) license. The details of the dataset are given below. This dataset was developed by three partnering institutes, IIIT Hyderabad, CDAC Noida, and IIIT Bhubaneshwar.

Language Train Test Dev
Hindi 11076 1389 1389
Urdu 8720 1096 1094
Odia 12109 1519 1517
Telugu 2993 384 384

  • The NER models have been developed by fine-tuning the XLM-Roberta-Base model on the annotated datasets. NER is modeled as a token classification task where a Softmax classifier is applied on the pooled layer of XLM-Roberta-base.

    • The following hyperparameters are used during training:
      • learning_rate: 5e-05
      • train_batch_size: 4
      • eval_batch_size: 4
      • optimizer: Adam
      • lr_scheduler_type: linear
      • num_epochs: 10.0

  • The models have been benchmarked on Hi-NER, Naamapadam datasets and Dev, Test datasets in this annotated data.

  • Package Versions

    • Transformers 4.38.2
    • Pytorch 1.9.1
    • Datasets 2.14.6
    • Tokenizers 0.15.0

  • To use this dataset, cite the paper as

    @misc{bahad2024finetuning,
      title={Fine-tuning Pre-trained Named Entity Recognition Models For Indian Languages}, 
      author={Sankalp Bahad and Pruthwik Mishra and Karunesh Arora and Rakesh Chandra Balabantaray and Dipti Misra Sharma and Parameswari Krishnamurthy},
      year={2024},
      eprint={2405.04829},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}