How to setup Distributed MinIO Cluster on Kubernetes
MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storage service. Designed to be Kubernetes Native.
MinIO runs on bare metal, network attached storage and every public cloud.
Distributed MinIO Kubernetes Architecture
Distributed MinIO provides protection against multiple node/drive failures and bit rot using erasure code. As the minimum disks required for distributed MinIO is 4 (same as minimum disks required for erasure coding), erasure code automatically kicks in as you launch distributed MinIO. With the highest level of redundancy, you may lose up to half (N/2) of the total drives and still be able to recover the data.
A distributed MinIO setup with m servers and n disks will have your data safe as long as m/2 servers or m*n/2 or more disks are online.
Note: MinIO creates erasure-coding sets of 4 to 16 drives per set. The number of drives you provide in total must be a multiple of one of those numbers.
The architecture of MinIO in Distributed Mode on Kubernetes consists of the StatefulSet deployment kind. The following lists the service types and persistent volumes used.
MinIO Services
Services are used to expose the app to other apps or users within the cluster or outside.
1. HeadLess Service for MinIO StatefulSet
2. LoadBalancer for exposing MinIO to external world.
Persistent Volumes
1. Dynamic
Deployment:
MinIO is Kubernetes native and containerized. This makes it very easy to deploy and test. Furthermore, it can be setup without much admin work.
The following steps direct how to setup a distributed MinIO environment on Kubernetes on AWS EKS but it can be replicated for other public clouds like GKE, Azure, etc.
The deployment comprises 4 servers of MinIO with 10Gi of ssd dynamically attached to each server.
Prerequisites:
- Kubernetes 1.5+ with Beta APIs enabled to run MinIO in distributed mode.
- LoadBalancer support
- PV provisioner support in the underlying infrastructure.
Deployment steps:
1. Copy the K8s manifest/deployment yaml file (minio_dynamic_pv.yml) to Bastion Host on AWS or from where you can execute kubectl commands.
2. kubectl apply -f minio-distributed.yml
3. kubectl get po (List running pods and check if minio-x are visible)
4. kubectl get svc
List the services running and extract the Load Balancer endpoint.
5. Paste this URL in browser and access the MinIO login.
accessKey : minio
secretKey: minio123
6. In the dashboard create a bucket clicking “+”
7. Upload an object in that bucket.
8. Verify the uploaded files show in the dashboard
Enjoy your Distributed MinIO Cluster with High Availability and Fault Tolerance !!
Source Code: fazpeerbaksh/minio: MinIO setup on Kubernetes (github.com)
