HUGS on AWS with NVIDIA GPUs

The Hugging Face Generative AI Services, also known as HUGS, can be deployed in Amazon Web Services (AWS) via the AWS Marketplace offering.

This collaboration brings Hugging Face’s extensive library of pre-trained models and their Text Generation Inference (TGI) solution to AWS customers, enabling seamless integration of state-of-the-art Large Language Models (LLMs) within the AWS infrastructure.

HUGS provides access to a hand-picked and manually benchmarked collection of the most performant and latest open LLMs hosted in the Hugging Face Hub to TGI-optimized container applications, allowing users to deploy third-party Kubernetes applications on AWS or on-premises environments.

With HUGS, developers can easily find, subscribe to, and deploy Hugging Face models using AWS infrastructure, leveraging the power of NVIDIA GPUs on optimized, zero-configuration TGI containers.

Subscribe to HUGS on AWS Marketplace

Go to HUGS AWS Marketplace listing
Subscribe to the product in AWS Marketplace by following the instructions on the page. At the time of writing (December 2024), the steps are to:
1. Click Continue to Subscribe, then go to the next page.
2. Click Continue to Configuration, then go to the next page.
3. Select the fulfillment option e.g. HUGS v2 for NVIDIA GPUs and AWS Inferentia2, and the software version e.g. 0.2.0.
Then click Continue to Launch. You successfully subscribe to HUGS. You can now follow the steps below to deploy your preferred HUGS container and model using Amazon EKS, with the provided container URIs.

To know whether you are subscribed or not, you can either see if a blue modal appears on top of the product page with a text saying “You have access to this product”, meaning that either you or someone else from your organization has already requested access for your account; otherwise, you can go to the AWS Marketplace service in the AWS Console and search for “HUGS (HUgging Face Generative AI Services)” is listed among your subscribed products.

Deploy HUGS on Amazon EKS

This example showcases how to create a Kubernetes Cluster on Amazon EKS, how to create a node group with the necessary permissions and compute requirements, and how to deploy HUGS on Amazon EKS using a Helm template.

This example assumes that you have an AWS Account, that you have installed and setup the AWS CLI, and that you are logged in into your account with the necessary permissions to subscribe to offerings in the AWS Marketplace, and create and manage IAM permissions and resources such as Elastic Kubernetes Service (EKS), Elastic Container Service (ECS), and EC2.

Requirements

Before proceeding, you need to have installed both kubectl, eksctl, and helm, to interact with the Kubernetes Cluster, to create, configure and delete resources on Amazon EKS, and to interact with the Helm templates, respectively.

To install both kubectl and eksctl you can follow the instructions at Amazon EKS Documentation - Set up kubectl and eksctl. Whilst for helm you can follow the instructions at Helm Documentation - Installing Helm.

Finally, for convenience the following environment variables will be set:

export REGION="us-east-1"
export NAMESPACE="default"
export CLUSTER_NAME="hugs-cluster"
export NODE_GROUP_NAME="hugs-node-group"
export SERVICE_ACCOUNT_NAME="hugs-service-account"
export DEPLOYMENT_NAME="hugs"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

If your AWS Account has multiple profiles remember to set the AWS_PROFILE environment variable to your profile so that the EKS commands use the correct profile.

Setup Amazon EKS

To create the EKS Cluster, Node Group, and add the necessary IAM permissions, you should run eksctl with the provided configuration file eks-cluster.yaml that looks like:

# Specifies the API version and kind of the configuration
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

# Defines the basic cluster metadata
metadata:
  name: $CLUSTER_NAME # Cluster name, used in various AWS resource names
  region: $REGION # AWS region where the cluster will be created
  version: "1.30" # Kubernetes version to use for the cluster

# IAM configuration for the cluster
iam:
  withOIDC: true # Enables IAM roles for service accounts (IRSA) using OIDC
  serviceAccounts:
    # Configures a service account for marketplace metering
    - metadata:
        name: $SERVICE_ACCOUNT_NAME
        namespace: $NAMESPACE
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage
    # Configures a service account for the AWS Load Balancer Controller (just required
    # if ingress is enabled within the HUGS Helm Template)
    - metadata:
        name: aws-load-balancer-controller
        namespace: $NAMESPACE
      attachPolicyARNs:
        - arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSLoadBalancerControllerIAMPolicy
      roleName: AmazonEKSLoadBalancerControllerRole

# Defines the managed node group for the cluster
managedNodeGroups:
  - name: $NODE_GROUP_NAME
    instanceType: g5.xlarge # GPU-enabled instance type, required by HUGS
    minSize: 1
    maxSize: 2 # Set to a greater number if you want to enable auto-scaling
    desiredCapacity: 1 # Fixed size node group, can be adjusted for scaling

# Specifies the EKS add-ons to be installed (default ones)
# All the addons below will be installed within the `kube-system` namespace
addons:
  - name: vpc-cni # Amazon VPC CNI plugin for Kubernetes
  - name: coredns # CoreDNS for Kubernetes DNS services
  - name: kube-proxy # kube-proxy for Kubernetes network proxy

# Configures CloudWatch logging for the cluster
cloudWatch:
  clusterLogging:
    enableTypes: ["*"] # Enables all types of control plane logging

Alternatively, you can download the file from the huggingface/hugs-helm-chart GitHub repository as follows:

curl -O https://raw.githubusercontent.com/huggingface/hugs-helm-chart/main/aws/eks-cluster.yaml

Optionally, before creating the cluster you may need to create the IAM Policy required to enable the AWS LoadBalancer Controller i.e. AWSLoadBalancerControllerIAMPolicy, so as to deploy the ingress which requires the AWS LoadBalancer Controller to be enabled, as per Install AWS Load Balancer Controller with Helm - Step 1: Create IAM Role using eksctl.

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/docs/install/iam_policy.json
aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

Note that the policy just needs to be created once per account, meaning that if it’s already created then you can reuse the existing one. Otherwise, you can alternatively create the policy under a different name, but along this example the policy is assumed to be named AWSLoadBalancerControllerIAMPolicy.

Alternatively, if you decided to skip the AWS LoadBalancer creation above, then you should remove the iam.serviceAccounts for the aws-load-balancer-controller before moving on to the next step.

Then you need to run the following command, that will replace the values set in the environment variables above within the eks-cluster.yaml file provided in huggingface/hugs-helm-chart. To replace the environment variable values within the eks-cluster.yaml file above, envsubst will be used, which may not be available for Windows users.

envsubst < eks-cluster.yaml > eks-cluster.yaml

Once the eks-cluster.yaml file has been updated, then you can just run the following command:

eksctl create cluster --config-file eks-cluster.yaml

If you want to enable or use the AWS LoadBalancer, you will need to deploy that separately before deploying HUGS, in order to enable the ingress.

helm repo add eks https://aws.github.io/eks-charts
helm repo update eks
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
    --namespace $NAMESPACE \
    --set clusterName=$CLUSTER_NAME \
    --set serviceAccount.create=false \
    --set serviceAccount.name=aws-load-balancer-controller

If you decide to deploy the AWS LoadBalancer Controller and enable the ingress within the HUGS deployment, then you will need to wait until the ALB controller is running before deploying HUGS, to do so you can use the following kubectl command:

kubectl wait --namespace $NAMESPACE \
    --for=condition=ready pod \
    --selector=app.kubernetes.io/name=aws-load-balancer-controller \
    --timeout=90s

Deploy HUGS with Helm on Amazon EKS

Finally, you can install the Helm template to deploy the HUGS container with the selected model, e.g. meta-llama/Llama-3.1-8B-Instruct; and you need to set the container URI provided by the AWS Marketplace for your account, either via the --set option when running helm install (as shown below), or modifying its value within the eks-values.yaml file provided in huggingface/hugs-helm-chart.

curl -O https://raw.githubusercontent.com/huggingface/hugs-helm-chart/main/aws/eks-values.yaml

helm repo add hugs https://raw.githubusercontent.com/huggingface/hugs-helm-chart/main/charts/hugs
helm repo update hugs

helm install $DEPLOYMENT_NAME hugs/hugs \
    -f eks-values.yaml \
    --set image.registry="XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com" \
    --set image.repository="hugging-face" \
    --set image.name="nvidia-meta-llama-meta-llama-3.1-8b-instruct" \
    --set image.tag="0.1.0" \
    --set serviceAccountName=$SERVICE_ACCOUNT_NAME \
    --set nodeSelector."eks\.amazonaws\.com/nodegroup"=$NODE_GROUP_NAME

The command above assumes that you followed the steps within this example, if you changed the node group name, the service account name, the container, or the number of accelerators; you should manually modify or create a new values.yaml file out of eks-values.yaml with your custom settings.

Inference on HUGS

To run the inference over the deployed HUGS service, you can either:

Forward the port via port-forwarding to a local port as e.g. 8080 (so that you can send requests to the service via localhost) with the command:
```
kubectl port-forward service/$DEPLOYMENT_NAME 8080:80
```
Use the external IP or hostname of the ingress, if ingress.enabled: true, that can be retrieved with the following command:
```
kubectl get ingress $DEPLOYMENT_NAME -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```

Then you can send requests to the Messages API via either localhost, the ingress IP or the ingress hostname, from outside the running pod.

In the inference examples in the guide below, the host is assumed to be localhost which is the case when deploying HUGS via Kubernetes with port-forwarding. If you have deployed HUGS on Kubernetes using an ingress under a specific IP, host, and/or with SSL (HTTPS), note that you should update the localhost references below with your host or IP.

Refer to Run Inference on HUGS to see how to run inference on HUGS.

Uninstall HUGS

Since HUGS was installed via helm install you can simply uninstall it as:

helm uninstall $DEPLOYMENT_NAME

And helm uninstall will remove all the workloads created, in this case being the Deployment, the Service, the Ingress, and the Horizontal Pod Autoscaler (HPA); alternatively, if you also installed the AWS LoadBalancer Controller, you can also uninstall it as follows:

helm uninstall aws-load-balancer-controller

Alternatively, once you are done using Amazon EKS Cluster, you can safely delete it to avoid incurring in unnecessary costs as:

eksctl delete cluster --name=$CLUSTER_NAME --region=$REGION

< > Update on GitHub

Hugging Face Generative AI Services (HUGS)