commit

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

added_tokens.json

@@ -0,0 +1,40 @@
+{
+  "\t\t": 50294,
+  "\t\t\t": 50293,
+  "\t\t\t\t": 50292,
+  "\t\t\t\t\t": 50291,
+  "\t\t\t\t\t\t": 50290,
+  "\t\t\t\t\t\t\t": 50289,
+  "\t\t\t\t\t\t\t\t": 50288,
+  "\t\t\t\t\t\t\t\t\t": 50287,
+  "  ": 50286,
+  "   ": 50285,
+  "    ": 50284,
+  "     ": 50283,
+  "      ": 50282,
+  "       ": 50281,
+  "        ": 50280,
+  "         ": 50279,
+  "          ": 50278,
+  "           ": 50277,
+  "            ": 50276,
+  "             ": 50275,
+  "              ": 50274,
+  "               ": 50273,
+  "                ": 50272,
+  "                 ": 50271,
+  "                  ": 50270,
+  "                   ": 50269,
+  "                    ": 50268,
+  "                     ": 50267,
+  "                      ": 50266,
+  "                       ": 50265,
+  "                        ": 50264,
+  "                         ": 50263,
+  "                          ": 50262,
+  "                           ": 50261,
+  "                            ": 50260,
+  "                             ": 50259,
+  "                              ": 50258,
+  "                               ": 50257
+}

config.json

@@ -0,0 +1,29 @@
+{
+  "_name_or_path": "/data/avner/results/spec_decoding/job-ft-af0609ec-33f0-446a-9603-b142ac39a7d8/checkpoints/checkpoint-9500",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 2048,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 32,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.37.2",
+  "use_cache": false,
+  "vocab_size": 51200
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.37.2"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b00519c33eb6bdbccf18128258932d5aca36439b5022f4a12ed6bc12555ef234
+size 4915876192

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenization_codegen25.py

@@ -0,0 +1,248 @@
+# Copyright (c) 2023, salesforce.com, inc.
+# All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+# For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/Apache-2.0
+"""Tokenization classes for CodeGen2.5."""
+
+from typing import List, Optional
+
+from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer
+from transformers.utils import logging
+
+try:
+    import tiktoken
+except ModuleNotFoundError as e:
+    raise ModuleNotFoundError("CodeGen2.5 requires the installation of tiktoken. Please install it via `pip install tiktoken`.") from e
+
+
+logger = logging.get_logger(__name__)
+
+MAX_MODEL_INPUT_SIZES = {
+    "Salesforce/codegen25-7b-multi": 2048,
+    "Salesforce/codegen25-7b-mono": 2048,
+    "Salesforce/codegen25-7b-instruct": 2048,
+}
+
+
+def tiktoken_tokenizer(base="gpt2", pad_token=None, add_special=True):
+    if not add_special:
+        return tiktoken.get_encoding(base)
+
+    def include_whitespace(n_min=2, n_max=20):
+        whitespaces = [" " * n for n in reversed(range(n_min, n_max))]
+        return whitespaces
+
+    def include_tabs(n_min=2, n_max=20):
+        tabs = ["\t" * n for n in reversed(range(n_min, n_max))]
+        return tabs
+
+    def include_fim_tokens():
+        fim_tokens = [
+            "<fim_prefix>",
+            "<fim_middle>",
+            "<fim_suffix>",
+            "<fim_pad>",
+            "<filename>",
+            "<gh_stars>",
+            "<issue_start>",
+            "<issue_comment>",
+            "<issue_closed>",
+            "<jupyter_start>",
+            "<jupyter_text>",
+            "<jupyter_code>",
+            "<jupyter_output>",
+            "<empty_output>",
+            "<commit_before>",
+            "<commit_msg>",
+            "<commit_after>",
+            "<reponame>"
+        ]
+        return fim_tokens
+
+    def include_codegen2_tokens():
+        tokens = []
+        tokens += [f"<dummy_{i}>" for i in range(4)]
+        tokens.append("<sep>")  # 50317
+        tokens.append("<eom>")  # 50318
+        tokens += [f"<mask_{i}>" for i in reversed(range(1, 51199-50318+1))]
+        return tokens
+
+    add_whitespaces = include_whitespace(n_min=2, n_max=32)
+    add_tabs = include_tabs(n_min=2, n_max=10)
+    fim_tokens = include_fim_tokens()
+    codegen2_tokens = include_codegen2_tokens()
+
+    tokenizer = tiktoken.get_encoding(base)
+
+    idx = tokenizer.n_vocab
+
+    bpe_ranks = tokenizer._mergeable_ranks
+
+    for wsp in add_whitespaces:
+        bpe_ranks[bytes(wsp, 'ascii')] = idx
+        idx += 1
+    for t in add_tabs:
+        bpe_ranks[bytes(t, 'ascii')] = idx
+        idx += 1
+
+    special_tokens = dict()
+
+    for sp in fim_tokens:
+        special_tokens[sp] = idx
+        idx += 1
+    for sp in codegen2_tokens:
+        special_tokens[sp] = idx
+        idx += 1
+
+    if pad_token and pad_token not in tokenizer._special_tokens and pad_token not in special_tokens:
+        special_tokens[pad_token] = idx
+        idx += 1
+    # In production, load the arguments directly instead of accessing private attributes
+    # See openai_public.py for examples of arguments for specific encodings
+    enc = tiktoken.Encoding(
+        # If you're changing the set of special tokens, make sure to use a different name
+        # It should be clear from the name what behaviour to expect.
+        name=base.replace("base", "im"),
+        pat_str=tokenizer._pat_str,
+        mergeable_ranks=bpe_ranks,
+        special_tokens={
+            **tokenizer._special_tokens,
+            **special_tokens
+        }
+    )
+    return enc
+
+
+class CodeGen25Tokenizer(PreTrainedTokenizer):
+    """
+    Construct a CodeGen2.5 tokenizer. Based on byte-level Byte-Pair-Encoding.
+    Args:
+        vocab_file (`str`):
+            Path to the vocabulary file.
+    """
+    max_model_input_sizes = MAX_MODEL_INPUT_SIZES
+    model_input_names = ["input_ids", "attention_mask"]
+
+    def __init__(
+            self,
+            pad_token=None,
+            eos_token="<|endoftext|>",
+            add_eos_token=False,
+            add_special_tokens=True,
+            **kwargs,
+    ):
+        pad_token_added = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token
+        eos_token_added = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token
+        self.encoder = tiktoken_tokenizer(base="gpt2", pad_token=pad_token, add_special=add_special_tokens)
+        super().__init__(
+            pad_token=pad_token_added,
+            eos_token=eos_token_added,
+            add_eos_token=add_eos_token,
+            **kwargs,
+        )
+        self.add_eos_token = add_eos_token
+
+    @property
+    def vocab_size(self):
+        """Returns vocab size"""
+        return self.encoder.n_vocab
+
+    def get_vocab(self):
+        """Returns vocab as a dict"""
+        vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
+        return vocab
+
+    def _tokenize(self, text, **kwargs):
+        """Returns a tokenized string."""
+        return self.encoder.encode(text, allowed_special="all")
+
+    def _convert_token_to_id(self, token):
+        """Converts a token (str) in an id using the vocab."""
+        if isinstance(token, str):
+            return self.encoder.encode_single_token(token)
+        else:
+            return token
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        try:
+            token = self.encoder.decode_single_token_bytes(index).decode("utf-8")
+        except Exception:
+            token = ""
+        return token
+
+    def _decode(self, token_ids: List[int], skip_special_tokens: bool = False, **kwargs):
+        if skip_special_tokens:
+            token_ids = [t for t in token_ids if t not in self.all_special_ids]
+        return self.encoder.decode(token_ids)
+
+    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None) -> List[int]:
+        """Build model inputs from a sequence by appending eos_token_id."""
+        eos_token_id = [self.eos_token_id] if self.add_eos_token else []
+
+        output = token_ids_0 + eos_token_id
+
+        if token_ids_1 is not None:
+            output = output + token_ids_1 + eos_token_id
+
+        return output
+
+    def get_special_tokens_mask(
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None,
+            already_has_special_tokens: bool = False
+    ) -> List[int]:
+        """
+        Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer `prepare_for_model` method.
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
+                Whether the token list is already formatted with special tokens for the model.
+        Returns:
+            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
+            )
+
+        eos_token_id = [1] if self.add_eos_token else []
+
+        if token_ids_1 is None:
+            return ([0] * len(token_ids_0)) + eos_token_id
+        return ([0] * len(token_ids_0)) + eos_token_id + ([0] * len(token_ids_1)) + eos_token_id
+
+    def create_token_type_ids_from_sequences(
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT
+        sequence pair mask has the following format:
+        ```
+        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
+        | first sequence    | second sequence |
+        ```
+        if token_ids_1 is None, only returns the first portion of the mask (0s).
+        Args:
+            token_ids_0 (`List[int]`):
+                List of ids.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+        Returns:
+            `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
+        """
+        eos_token_id = [self.eos_token_id] if self.add_eos_token else []
+
+        output = [0] * len(token_ids_0 + eos_token_id)
+
+        if token_ids_1 is not None:
+            output += [1] * len(token_ids_1 + eos_token_id)
+
+        return output
+
+    # has no vocab file
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None):
+        return ()

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": true,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}