--- description: Setting up and runnning H2O LLM Studio requires the following minimal prerequisites. This page lists out the speed and performance metrics of H2O LLM Studio based on different hardware setups. --- # H2O LLM Studio performance Setting up and runnning H2O LLM Studio requires the following minimal [prerequisites](set-up-llm-studio.md#prerequisites). This page lists out the speed and performance metrics of H2O LLM Studio based on different hardware setups. The following metrics were measured. - **Hardware setup:** The type and number of computing devices used to train the model. - **LLM backbone:** The underlying architecture of the language model. For more information, see [LLM backbone](concepts.md#llm-backbone). - **Quantization:** A technique used to reduce the size and memory requirements of the model. For more information, see [Quantization](concepts.md#quantization). - **Train**: The amount of time it took to train the model in hours and minutes. - **Validation:** The amount of time it took to validate the mode in hours and minutes. | Hardware setup | LLM backbone | Quantization | Train (hh:mm:ss)| Validation (hh:mm:ss) | |---|---|---|---|---| | 8xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 11:35 | 3:32 | | 4xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 21:13 | 06:35 | | 2xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 37:04 | 12:21 | | 1xA10G | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 1:25:29 | 15:50 | | 8xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 14:26 | 06:13 | | 4xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 26:55 | 11:59 | | 2xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 48:24 | 23:37 | | 1xA10G | h2oai/h2ogpt-4096-llama2-7b | nf4 | 1:26:59 | 42:17 | | 8xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 4xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 2xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 1xA10G | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | OOM | OOM | | 8xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 25:07 | 10:58 | | 4xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 48:43 | 21:25 | | 2xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 1:30:45 | 42:06 | | 1xA10G | h2oai/h2ogpt-4096-llama2-13b | nf4 | 2:44:36 | 1:14:20 | | 8xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | | 4xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | | 2xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | | 1xA10G | h2oai/h2ogpt-4096-llama2-70b | nf4 | OOM | OOM | |---|---|---|---|---| | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 7:04 | 3:55 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 13:14 | 7:23 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | bfloat16 | 23:36 | 13:25 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 9:44 | 6:30 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 18:34 | 12:16 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-7b | nf4 | 34:06 | 21:51 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 11:46 | 5:56 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 21:54 | 11:17 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | bfloat16 | 39:10 | 18:55 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 16:51 | 10:35 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 32:05 | 21:00 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-13b | nf4 | 59:11 | 36:53 | | 4xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 1:13:33 | 46:02 | | 2xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 2:20:44 | 1:33:42 | | 1xA100 80GB | h2oai/h2ogpt-4096-llama2-70b | nf4 | 4:23:57 | 2:44:51 | :::info The runtimes were gathered using the default parameters.
Expand to see the default parameters ``` architecture: backbone_dtype: int4 force_embedding_gradients: false gradient_checkpointing: true intermediate_dropout: 0.0 pretrained: true pretrained_weights: '' augmentation: random_parent_probability: 0.0 skip_parent_probability: 0.0 token_mask_probability: 0.0 dataset: add_eos_token_to_answer: true add_eos_token_to_prompt: true add_eos_token_to_system: true answer_column: output chatbot_author: H2O.ai chatbot_name: h2oGPT data_sample: 1.0 data_sample_choice: - Train - Validation limit_chained_samples: false mask_prompt_labels: true parent_id_column: None personalize: false prompt_column: - instruction system_column: None text_answer_separator: <|answer|> text_prompt_start: <|prompt|> text_system_start: <|system|> train_dataframe: /data/user/oasst/train_full.pq validation_dataframe: None validation_size: 0.01 validation_strategy: automatic environment: compile_model: false find_unused_parameters: false gpus: - '0' - '1' - '2' - '3' - '4' - '5' - '6' - '7' huggingface_branch: main mixed_precision: true number_of_workers: 8 seed: -1 trust_remote_code: true use_fsdp: false experiment_name: default-8-a10g llm_backbone: h2oai/h2ogpt-4096-llama2-7b logging: logger: None neptune_project: '' output_directory: /output/... prediction: batch_size_inference: 0 do_sample: false max_length_inference: 256 metric: BLEU metric_gpt_model: gpt-3.5-turbo-0301 metric_gpt_template: general min_length_inference: 2 num_beams: 1 num_history: 4 repetition_penalty: 1.2 stop_tokens: '' temperature: 0.3 top_k: 0 top_p: 1.0 problem_type: text_causal_language_modeling tokenizer: add_prompt_answer_tokens: false max_length: 512 max_length_answer: 256 max_length_prompt: 256 padding_quantile: 1.0 use_fast: true training: batch_size: 2 differential_learning_rate: 1.0e-05 differential_learning_rate_layers: [] drop_last_batch: true epochs: 1 evaluate_before_training: false evaluation_epochs: 1.0 grad_accumulation: 1 gradient_clip: 0.0 learning_rate: 0.0001 lora: true lora_alpha: 16 lora_dropout: 0.05 lora_r: 4 lora_target_modules: '' loss_function: TokenAveragedCrossEntropy optimizer: AdamW save_best_checkpoint: false schedule: Cosine train_validation_data: false warmup_epochs: 0.0 weight_decay: 0.0 ```
:::