Build Your Own Object Storage with Oxide and MinIO

What object storage is, and why it kept growing

Object storage is one of the most commonly used services in cloud-native applications, which treats every file as a unique key accessible via HTTPS. Since AWS launched S3 in 2006, it has become the backbone of data lakes, machine learning training sets, container registries, database backups, logs, document repositories, media archives, and cold storage. Storage tiers further classify data by retrieval latency and business criticality.

The reasons for object storage’s adoption are simple. First, it accepts any file format, making it nearly universal. It also requires minimal management as you manage data via UI, SDK, or API, regardless of scale, from gigabytes to exabytes. Finally, storage and compute decouple and scale independently.

Object Storage market adoption rate is impressive. The Global object storage market is roughly at $8.4 billion in 2025 and growing at around 16% CAGR (Compound Annual Growth Rate) , touching $34 billion by 2034. AI workloads drive most of that growth, whether it’s about landing raw data, curating the datasets across medallion layers, running inference workloads, or simply training or fine-tuning the model on the collected data in object storage.

The pain of object storage in the public cloud

While demand for object storage climbs, it brings major challenges.

The first bottleneck organizations are facing is the rising cost of public cloud, and object storage plays a big role in that. Egress and lock-in are the biggest pain points. AWS charges around $0.09/GB for the first 10 TB, which reduces to $0.07/GB at the 150 TB tier. This means that for any organization, moving 1 PB out of S3 in a single month costs around $54,000 in egress costs, which climbs towards $90,000 when including cross-region transfers, NAT gateway charges, and per-request fees. Egress costs represent meaningful percentage of cloud spend, which makes every gigabyte today raises the cost of moving off tomorrow, making it perfect example of vendor lock-in.

Sovereignty is the second challenge, and it’s moving faster than most teams have planned for. The EU Cloud Sovereignty Framework landed in October 2025 with explicit penalties against US legislation like the CLOUD Act. The EU Data Act took effect in September 2025, adding mandatory data portability rules, including two-month termination windows and 30-day export deadlines for cloud providers. For regulated businesses serving European or UK customers, storing object data in a US hyperscaler region is now an active compliance problem.

Organizations are asking two questions.

• How do we use object storage as cost-effectively as possible?
• How do we satisfy residency requirements?

Both points to the same answer. Stop renting object storage and start owning it.

Oxide is the substrate

Enterprises are looking for sovereign, scalable, and cost-efficient object storage without rewriting their existing application code. Oxide provides the substrate to enable object storage workloads by providing compute, storage, network, and observability in an API-driven approach. Object storage sits on top of that, just as it would for Postgres, Kafka, or Kubernetes would.

For object storage workloads, the following Oxide primitives come into play:

• Cloud API: Provision instances, disks, VPCs, firewall rules through one consistent interface.
• Anti-affinity groups: Ensure distributed services are not co-located on the same compute sled, preventing a single sled failure into a service failure.
• Floating IPs: Provide a stable customer-facing object storage endpoint that does not change when the underlying infrastructure scales.
• Flexible disk types: Support replication at the application layer or infrastructure layer, depending on architecture needs.

Why MinIO first?

The primitives above provide a solid foundation to deploy an S3-compatible object store on Oxide. For the purposes of this build, I chose MinIO as the first implementation to evaluate, primarily because it is a well-known S3-compatible object store with a relatively simple deployment model. Other S3-compatible object stores could be deployed on the same Oxide foundation, and the underlying infrastructure is intentionally designed to remain object store agnostic. MinIO runs as a single Go binary managed by systemd. It handles erasure coding at the application layer, and multi-site replication. It’s S3 API compatibility lets existing S3 application code run against it without changes. More than half of Fortune 500 companies use MinIO’s object storage to power data infrastructure, spanning industries like technology, finance, and manufacturing. MinIO moved the community edition into maintenance mode in December 2025 but is still under AGPLv3. AIStor is the current commercial platform of MinIO where all active development, features enhancements and enterprise investments are focused now.

Architecture

This architecture deploys a distributed MinIO cluster on Oxide using Terraform, with the following components:

• 4 MinIO instances, each with 4 local disks for a total of 16 disks. This provides EC:4 erasure coding, 75% storage efficiency, and fault tolerance of one instance.
• 2 HAProxy load balancers in active/standby.
• 1 floating IP fronting the load balancers that provides a single S3 endpoint that fails over autonomously when the active load balancer dies.

Everything stays inside a single security boundary within the silo.

Here is the process flow:

Client makes a request to S3 endpoint provided by the Floating IP –> HAProxy receives the request and sends it to a healthy MinIO instance. –> MinIO serves the request its local disks, replying along the same path.

The full build is open-sourced github.com/build your own object storage as a Terraform module. Given a working Oxide silo, a project, and an Ubuntu image, it produces the cluster end to end — the four MinIO nodes, the load balancer pair, the Floating IP, and all the bootstrap glue.

Reference architecture diagram

Architecture Decisions

Let’s discuss the architecture decisions we made for this setup.

Local disks for data, distributed disks for boot

MinIO erasure-codes data at the application layer. Stacking that on top of Oxide’s 3x Crucible-replicated distributed disks would push raw overhead to roughly 4x for no extra durability. With EC:4 (12 data shards plus 4 parity), every 1 TB of usable data takes 1.33 TB of physical disk. Local disks skip the Crucible replication and hold the capacity overhead at 1.33x. We can keep boot disk on distributed storage.

Encryption at the application layer

Encryption sits one layer above the disk choice. MinIO supports server-side encryption natively: SSE-KMS via the Key Encryption Service (KES), SSE-S3 with an internal KMS, and SSE-C with caller-provided keys. For regulated workloads, SSE-KMS with KES fronting Vault (or another customer-managed KMS) is the standard pattern in which keys stay with the customer, never with the storage.

Anti-affinity with policy “fail”

The Oxide policy doesn’t allow two MinIO nodes on the same sled. Two nodes co-located on one sled would turn a single sled failure into a cluster failure, which exceeds EC:4’s tolerance. This approach will tolerate four simultaneous drive failures, or one full node, or two partial node failures.

Active/standby load balancers, not active/active

Floating IPs in Oxide attach to one instance at a time today. A small watcher script polls the peer’s TCP port and fires an idempotent failover through the Oxide API when the active load balancer dies. End-to-end failover measured around 11 seconds.

TLS at the load balancer

Single certificate to rotate. Underlay traffic stays inside the VPC, on the Oxide-provided private network. The handful of customers whose compliance regime requires end-to-end TLS can re-encrypt at the second hop. Default stays simple.

GitHub Repository

Clone it, fill in a terraform.tfvars, and run terraform apply. You’ll have an S3 endpoint in under 20 minutes.

The repo contains:

• The Terraform module that builds the full cluster on Oxide.
• Idempotent bootstrap scripts that handle disk format, MinIO install, load balancer configuration, and the failover watcher.
• An architecture FAQ that captures the “why” behind every design call.
• Raw lab guide to setup everything from scratch step by step manually.

github.com/build your own object storage

Watch the build

Conclusion

A decade of on-prem storage went on wrestling the hardware, the SAN, the firmware, and the appliance vendor. With Oxide as the underlying infrastructure, the complete hardware and software stack is co-designed to provide a simple API-driven user experience. The customer picks the storage engine that fits the workload, points the API at it, and gets on with the actual job.

The Cloud Computer is the platform. The store is the choice.