| --- | |
| library_name: custom | |
| tags: | |
| - robotics | |
| - diffusion | |
| - mixture-of-experts | |
| - multi-modal | |
| license: mit | |
| datasets: | |
| - CALVIN | |
| language: | |
| - en | |
| pipeline_tag: robotics | |
| --- | |
| # MoDE (Mixture 1of Diffusion Experts) Model | |
| This model implements a Mixture of Diffusion Experts architecture for robotic manipulation, combining transformer-based processing with expert routing and diffusion-based action prediction. | |
| ## Model Architecture | |
| - Base Architecture: MoDE with custom Mixture of Experts Transformer | |
| - Vision Encoder: {getattr(model_instance, 'resnet_type', 'ResNet')} with FiLM conditioning | |
| - EMA: Enabled | |
| - Action Window Size: {model_instance.act_window_size} | |
| - Sampling Steps: {model_instance.num_sampling_steps} | |
| - Sampler Type: {model_instance.sampler_type} | |
| ## Input/Output Specifications | |
| - RGB Static Camera: (B, T, 3, H, W) tensor | |
| - RGB Gripper Camera: (B, T, 3, H, W) tensor | |
| - Language Instructions: Text strings | |
| - Output: (B, T, 7) tensor representing 7-DoF actions | |
| ## Usage Example | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| import torch | |
| weights_path = hf_hub_download(repo_id="{repo_name}", filename="model_cleaned.safetensors") | |
| model.load_pretrained_parameters(weights_path) | |
| obs = { | |
| "rgb_obs": { | |
| "rgb_static": static_image, | |
| "rgb_gripper": gripper_image | |
| } | |
| } | |
| goal = {"lang_text": "pick up the blue cube"} | |
| action = model.step(obs, goal) | |
| ``` | |
| ## Training Configuration | |
| - Optimizer: AdamW | |
| - Learning Rate: {config.optimizer.learning_rate} | |
| - Weight Decay: {config.optimizer.transformer_weight_decay} | |