In the world of AI, models and datasets often reside in a single cloud bucket, while the GPUs needed for development, training, or serving are increasingly scattered across different cloud providers. This geographical and organizational split creates a significant challenge: cross-cloud data transfer fees. Every time your data needs to move from its storage location to your compute infrastructure on another cloud, you incur costly egress charges.
Fortunately, a powerful solution has emerged to eliminate this barrier. Working in close collaboration, SkyPilot and Hugging Face have introduced a seamless integration that keeps your models and datasets on the Hugging Face Hub while SkyPilot orchestrates your compute tasks on any available GPU cluster. This innovative approach allows you to leverage diverse compute resources without the burden of data egress costs, making multi-cloud AI development more efficient and economical than ever before.
Bridging the Divide: Hugging Face Storage as a SkyPilot Backend
SkyPilot tasks have long supported reading from and writing to various cloud object stores like S3, GCS, Azure, and R2. Now, Hugging Face Storage has been added as a first-class SkyPilot backend, accessible via the familiar hf:// scheme. This single URL scheme simplifies the entire AI lifecycle, from pulling base models and datasets to saving checkpoints and publishing finished models.
Integrating Hugging Face resources into your SkyPilot jobs is straightforward and intuitive. You can mount Hugging Face Buckets or any Hub repository directly into your SkyPilot tasks, enabling read-write access for checkpoints or read-only access for models and datasets. This means most teams already using the Hugging Face Hub can adopt this solution without any migration steps or the need to create new storage accounts.
Here’s a glimpse of how you can specify Hugging Face mounts in your SkyPilot configuration:
/checkpoints: source: hf://buckets/my-org/qwen-sft store: hf mode: MOUNT(A Hugging Face Bucket for read-write checkpoints)/base-model: source: hf://Qwen/Qwen3.5-4B store: hf mode: MOUNT(A model repo, mounted read-only)/data: source: hf://datasets/my-org/my-dataset@main store: hf mode: MOUNT(A dataset repo, pinned to a specific revision)
Smart Data Access: The Power of Lazy Mounting
The magic behind this integration lies in Hugging Face’s hf-mount FUSE backend when using mode: MOUNT. This technology makes a Hugging Face Bucket or repository appear as a local path, just like other SkyPilot FUSE mounts. Crucially, data fetching happens at the filesystem layer, meaning only the bytes your code explicitly reads are pulled across the network.
This “lazy read” mechanism ensures that processes can begin working with large files almost immediately, without waiting for an entire dataset or model to download first. Your GPU stays busy, training on data as it streams in, instead of sitting idle (and racking up bills) during a full data copy. Furthermore, hf-mount maintains an on-disk cache, so repeat reads for the same data stay local, significantly boosting performance after the initial access.
For authentication, SkyPilot leverages the HF_TOKEN you likely already possess. Simply set this token in your environment and pass it to your SkyPilot run with --secret HF_TOKEN. This single token works universally across AWS, GCP, Azure, Nebius, Lambda, or your own Kubernetes clusters, eliminating the complexity of managing multiple cloud-specific bucket keys.
Unleashing Multi-Cloud Freedom: Zero Egress, More Choice
In today’s AI landscape, securing sufficient GPU capacity often requires utilizing resources from multiple vendors—hyperscalers, neoclouds, or even on-prem racks. SkyPilot is engineered precisely for this scenario, allowing you to schedule a single job spec across 20+ clouds, Kubernetes, and on-prem environments, landing it wherever reserved capacity is available.
However, traditional object storage has been a bottleneck, as it is inherently regional and cloud-specific. Feeding a GPU on a different vendor’s network typically means either maintaining redundant data copies in multiple buckets or incurring substantial egress fees. These charges, often around $0.09/GB out of clouds like AWS, can quickly accumulate, especially when pulling large models onto inference nodes or iterating datasets over many epochs.
Hugging Face Storage fundamentally alters this cost dynamic by offering zero egress or CDN fees on the read side. With storage itself priced competitively at $12-18/TB/month (compared to AWS S3 at roughly $23/TB plus egress), the same Hugging Face Bucket becomes freely accessible from any of your compute clusters, regardless of their location. While writing back to the bucket still incurs your compute cloud’s egress charges, for most AI workflows, reads significantly dominate, thus providing immense cost savings and operational flexibility.
This zero-egress policy means you are no longer forced to “pin” each AI run to the specific cloud vendor that happens to host your data. Instead, you can truly optimize your resource utilization by launching jobs wherever the most cost-effective or available GPU capacity resides, maximizing the value of your reserved compute infrastructure.
A Real-World Example: Fine-Tuning Qwen with Zero Egress
To demonstrate this powerful integration, we conducted a benchmark involving a small fine-tuning task. We used Qwen/Qwen3.5-4B on the HuggingFaceH4/Multilingual-Thinking dataset, leveraging TRL’s SFTTrainer. The base model was mounted read-only from its Hub repo, while checkpoints were written directly to a Hugging Face Bucket.
The same SkyPilot YAML configuration was launched across AWS, GCP, and Lambda, simply by changing the --infra flag. SkyPilot intelligently placed each job where GPUs were available, with all three instances seamlessly reading from and writing to the same Hugging Face Bucket without incurring any egress fees for data reads.
A significant benefit for checkpointing is that Hugging Face Buckets are built on Xet, which employs content-defined chunking and deduplication. This means only new or changed chunks of data are actually stored when you save a checkpoint, dramatically reducing storage consumption and upload times for incremental updates. The deduplication is entirely automatic, saving you space without extra effort.
Get Started Today
Embrace the freedom of running your AI workloads anywhere without the burden of prohibitive data egress costs. Getting started is remarkably simple, allowing you to quickly unlock multi-cloud capabilities for your AI projects.
- First, install the necessary SkyPilot components:
pip install "skypilot[huggingface]" - Next, authenticate with Hugging Face:
hf auth loginor set your token as an environment variable:export HF_TOKEN=<your-token> - Finally, add an
hf://mount to any SkyPilot task configuration and launch your job. Note thatMOUNTmode requires a base image withglibc 2.34+and/dev/fuse.
This robust integration is a testament to open-source collaboration, with contributions from Nikhil Jha, the Hugging Face team, and the SkyPilot team. The entire stack—SkyPilot, Hugging Face’s hf-mount, and the huggingface_hub client—is fully open source, ensuring transparency and continuous improvement for the AI community.
Source: Hugging Face Blog
