Spaces:
Runtime error
Runtime error
Lohith9923
commited on
Commit
β’
35e6f3b
1
Parent(s):
cd6b7b8
Update README.md
Browse files
README.md
CHANGED
@@ -1,12 +1,92 @@
|
|
1 |
-
|
2 |
-
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# English to Hindi Text Translation using Transformers
|
2 |
+
|
3 |
+
This project showcases a simple text translation model that translates English text to Hindi using the Hugging Face Transformers library. The model utilizes pre-trained sequence-to-sequence architecture for accurate and efficient translation.
|
4 |
+
|
5 |
+
## Table of Contents
|
6 |
+
|
7 |
+
- [Project Overview](#project-overview)
|
8 |
+
- [Installation](#installation)
|
9 |
+
- [Usage](#usage)
|
10 |
+
- [Model Training and Dataset](#model-training-and-dataset)
|
11 |
+
- [Model Testing and Deployment](#model-testing-and-deployment)
|
12 |
+
- [User Interface](#user-interface)
|
13 |
+
- [Challenges Faced](#challenges-faced)
|
14 |
+
- [Contributions](#contributions)
|
15 |
+
|
16 |
+
## Project Overview
|
17 |
+
|
18 |
+
Text translation is an essential task in natural language processing, and this project aims to provide a practical example of building and deploying a translation model. The project covers the following aspects:
|
19 |
+
|
20 |
+
- Data preprocessing: Tokenization and dataset preparation.
|
21 |
+
- Model training: Training a sequence-to-sequence model for English-to-Hindi translation.
|
22 |
+
- Model testing: Translating text using the trained model.
|
23 |
+
- User interface: Creating a user-friendly interface for text translation.
|
24 |
+
|
25 |
+
## Installation
|
26 |
+
|
27 |
+
To run this project, you'll need the following dependencies:
|
28 |
+
|
29 |
+
- Python 3.x
|
30 |
+
- TensorFlow
|
31 |
+
- Hugging Face Transformers
|
32 |
+
- Datasets library
|
33 |
+
- Gradio
|
34 |
+
|
35 |
+
You can install the required libraries using the following shell command:
|
36 |
+
|
37 |
+
```shell
|
38 |
+
pip install datasets transformers[sentencepiece] tensorflow gradio -q
|
39 |
+
```
|
40 |
+
|
41 |
+
## Usage
|
42 |
+
Download the folder from here and the run the following command
|
43 |
+
|
44 |
+
```shell
|
45 |
+
python3 app.py
|
46 |
+
```
|
47 |
+
After running this command
|
48 |
+
## Model Training and Dataset
|
49 |
+
For training the text translation model.
|
50 |
+
You can checkout the pre-trained model from [here](https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fhuggingface.co%2FHelsinki-NLP%2Fopus-mt-en-hi) and Dataset from [here](https://huggingface.co/datasets/cfilt/iitb-english-hindi/viewer/cfilt--iitb-english-hindi).
|
51 |
+
- First Download the pre-trained model using **transformers** library in python.
|
52 |
+
- Load the Dataset **cfilt/iitb-english-hindi** using **Datasets** library in python.
|
53 |
+
- Initialized the model, tokenizer, and preprocessing function.
|
54 |
+
- Tokenized the dataset and prepared the training and validation data.
|
55 |
+
- Compiled the model with the optimizer(**Adam**) with required parameters.
|
56 |
+
- Trained the model for the desired number of epochs.
|
57 |
+
|
58 |
+
## Model Testing and Deployment
|
59 |
+
To test the trained model and deploy a user interface:
|
60 |
+
|
61 |
+
- Saved the trained model at a preferred location.
|
62 |
+
- Loaded the model from the location and tokenizer for testing.
|
63 |
+
- Translated sample input text using the model.
|
64 |
+
- Deployed a Gradio interface for user-friendly translation.
|
65 |
+
|
66 |
+
## User Interface
|
67 |
+
|
68 |
+
The Gradio interface provides an interactive way to translate English text to Hindi. To use the interface:
|
69 |
+
|
70 |
+
- Run the project and navigate to the specified URL.
|
71 |
+
- Enter English text in the input box.
|
72 |
+
- Checkout the translated Hindi text in the output box.
|
73 |
+
|
74 |
+
## Challenges Faced
|
75 |
+
|
76 |
+
- Surfed through lot of resources in google and other platforms for best dataset for my project.
|
77 |
+
- Spent a lot of time gathering the correct resources for understanding about transformers, LLM's and gradio.
|
78 |
+
|
79 |
+
## Contributions
|
80 |
+
Contributions to this project are welcome! Here are some ways you can contribute:
|
81 |
+
|
82 |
+
- Improve the model's translation quality and performance.
|
83 |
+
- Enhance the user interface for a better user experience.
|
84 |
+
- Add support for more languages and translation directions.
|
85 |
+
|
86 |
+
To contribute, follow these steps:
|
87 |
+
|
88 |
+
- Fork this repository.
|
89 |
+
- Create a new branch for your feature or bug fix.
|
90 |
+
- Commit your changes and push them to your fork.
|
91 |
+
- Open a pull request with a detailed description of your changes.
|
92 |
+
|