add new model

README.md

@@ -4,28 +4,28 @@ language:
 license: apache-2.0
 ---
 
-# Randeng-TransformerXL-1.1B-Paraphrasing
+# Randeng-Transformer-1.1B-Denoise
 
 - Github: [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM)
 - Docs: [Fengshenbang-Docs](https://fengshenbang-doc.readthedocs.io/)
 
 ## 简介 Brief Introduction
 
-基于Transformer-XL结构实现中文句子改写。
+以语法纠错任务为微调目标的中文Transformer-XL。
 
-Paraphrase Chinese sentences based on Transformer-XL.
+Chinese Transformer-XL with a denoising task as the fine-tuning objective.
 
 ## 模型分类 Model Taxonomy
 
 |  需求 Demand  | 任务 Task       | 系列 Series      | 模型 Model    | 参数 Parameter | 额外 Extra |
 |  :----:  | :----:  | :----:  | :----:  | :----:  | :----:  |
-| 通用 General | 自然语言转换 NLT | 燃灯 Randeng | Transformer |      1.1B      |     中文-改写 Chinese-Paraphrasing    |
+| 通用 General | 自然语言转换 NLT | 燃灯 Randeng | Transformer |      1.1B      |     中文-去噪 Chinese-Denoise    |
 
 ## 模型信息 Model Information
 
-使用1.1B参数的Transformer-XL在悟道语料库(280G版本)上进行预训练，然后在同义句标注数据集上进行微调。
+我们先使用Transformer-XL的模型结构在悟道语料库(180G版本)上进行预训练，然后在我们自主构建的语法错误数据集上进行微调。其中，去噪任务是从包括 **随机插入/交换/删除/替换/句子重排** 的具有噪声的输入中重建一个流畅和干净的文本。
 
-The 1.1B Transformer-XL model is pre-trained on the Wudao corpus (with 280G samples), and then fine-tuned on the annotated similar-sentence pair dataset.
+We first pre-trained Transformer-XL on the Wudo corpus (180G version), and then fine-tuned it on a denoised dataset (developed by us). The denoise task is to reconstruct a fluent and clean text from a noisy input which includes **random insertion/swap/deletion/replacement/sentence reordering**.
 
 ## 使用 Usage
 

config.json

@@ -0,0 +1,15 @@
+{
+    "num_layers":32,
+    "vocab_size":50048,
+    "hidden_size":1600,
+    "num_attention_heads":25,
+    "embedding_dropout_prob":0.1,
+    "attention_dropout_prob":0.1,
+    "output_dropout_prob":0.1,
+    "max_sequence_length":512,
+    "max_memory_length":512,
+    "checkpoint_activations":false,
+    "checkpoint_num_layers":1,
+    "parallel_output":true,
+    "relative_encoding":true
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40bb2c508f87756a715bcbf417dcec56c08c2186b431ba109abf61e5fdf0c45c
+size 2291627287

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ea6f4164152bc58d23e24e48f7bf4187aad72a32e97ec4b3acc832fe183cbc2
+size 1021864