Introducing pvc-autoresizer - Kintone Engineering Blog

By Akihiro Ikezoe.

Today, we are excited to announce pvc-autoresizer, which is new, Kubernetes-native, open-source software to provide the ability to expand the size of Persistent Volume Claims (PVCs) on Kubernetes cluster automatically.

pvc-autoresizer can virtually be used with all CSI drivers that support VolumeExpansion.

github.com

Why pvc-autoresizer?

To create persistent storage in Kubernetes, we can use Kubernetes resources called Persistent Volume (PV) and Persistent Volume Claim.

If you have CSI drivers which support Dynamic Provisioning, a PV will be created dynamically when a PVC is created. On the other hand, there is no dynamic Volume Expansion feature in Kubernetes.

As the number of volumes increases, extending them one by one can become a burden.

Prerequisite

pvc-autoresizer can be used the PVCs that meets the following conditions:

Provisioned by CSI drivers which support VolumeExpansion.
Filesystem volume mode.

For example:

TopoLVM is a new local storage for Kubernetes. If you're interested in TopoLVM, you can read the following article.

blog.kintone.io

How it works

To expand a PVC, pvc-autoresizer works as follows:

Get the target PVC information from kube-apiserver.
Get StorageClass related to the PVC from kube-apiserver.
Get filesystem usage metrics from Prometheus that scrapes the information from kubelet.
Expand PVC storage request size if PVC has less than the specified amount of free filesystem capacity.

How to use

Learn how to use pvc-autoresizer in a Kubernetes cluster where your target CSI driver is deployed.

Setup Prometheus

pvc-autoresizer collects PV filesystem usage from Prometheus. You have to setup Prometheus in your Kubernetes cluster.

And to collect kubelet metrics, you have to add the following scrape config to Prometheus:

scrape_configs:
  - job_name: 'kubernetes-nodes'
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
      - source_labels: [__name__]
        action: drop
        regex: kubelet_runtime_operations_duration_seconds.*

Prometheus Operator is one of the best solutions. It will automatically deploy Prometheus and configure scraping for you.

Deploy pvc-autoresizer

Deploy pvc-autoresizer as shown in the following manifest:

neco-apps/pvc-autoresizer/base at master · cybozu-go/neco-apps · GitHub

Note that you should set --prometheus-url args according to your environment.

spec:
  template:
    spec:
      containers:
      - name: manager
        args:
        - --prometheus-url=http://prometheus.monitoring.svc:9090

Enable auto-resizing

You can specify whether or not to enable automatic resizing for each StorageClass. If you want to enable it, add resize.topolvm.io/enabled: "true" annotation to the StorageClass.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: my-provisioner
  annotations:
    resize.topolvm.io/enabled: "true" 
provisioner: topolvm.cybozu.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Prepare a PVC and a Pod

Prepare a PVC you want to auto-resize and a Pod to use it.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-pvc
  annotations:
    resize.topolvm.io/threshold: 20%
    resize.topolvm.io/increase: 20Gi
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 30Gi
    limits:
      storage: 100Gi
  storageClassName: my-provisioner

spec.resources.limits.storage must be specified. It acts as safeguard. pvc-autoresizer increases PVC's spec.resources.requests.storage up to spec.resources.limits.storage.

In addition, there are two optional annotations.

resize.topolvm.io/threshold is the threshold of free space. If the free space of the target PV is smaller than the specified value, it will be resized. The default value is 10%.

resize.topolvm.io/increase is the amount of increased size. The request size of the target PVC is increased by a specified percentage of the request size. The default value is 10%.

Resizing!

Enter into the target pod:

kubectl exec -it example-pod bash

Make sure current volume usage:

root@example-pod:/# df -h /test1
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/topolvm/8ad1c617-e572-4d0d-b4e8-d66e5a572df9 1014M   34M  981M   4% /test1

Create a file that take up 90% of the volume:

fallocate -l 900M /test1/test.txt

Make sure current volume usage again:

root@example-pod:/# df -h /test1
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/topolvm/8ad1c617-e572-4d0d-b4e8-d66e5a572df9 1014M  934M   81M  93% /test1

After a few minutes, the volume will be resized to 2GiB:

root@example-pod:/# df -h /test1
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/topolvm/8ad1c617-e572-4d0d-b4e8-d66e5a572df9  2.0G  935M  1.1G  46% /test1