#Ephemeral storage kubernetes how to#
#Ephemeral storage kubernetes code#

I understand this last question is hard to answer. I'm trying to find any documentation that is pointing towards that KubeletDisk is still a Preview feature but I'm unable to do so.Īt the old API spec that you shared we can see a note about it: īut in the new version: I can't see any notice about this.ĭo you have any pointers? Also when should we expect KubeletDisk to be out of preview? AgentPoolsClient #CreateOrUpdate: Failure sending request: StatusCode=400 - Original Error: Code='PreviewFeatureNotRegistered' Message='Preview feature Microsoft.ContainerService/KubeletDisk not registered.' │ Error : creating Node Pool : ( Agent Pool Name 'standard3 ' / Managed Cluster Name 'aks-dev-we-aks1 ' / Resource Group 'rg-dev-we-aks ') : containerservice. Right now, root-dir is just on the OS disk which is why you see the mismatch (I think) I'm a bit curious about what happens to non-ephemeral things that normally live in /var/lib/kubelet.

#Ephemeral storage kubernetes code#

Kubelet has -root-dir and -log-dir, after digging through the code seems like root-dir maps to the desired cadvisor FsStats: (and similar) Your node can have as many other filesystems, not used for Kubernetes, as you like. The first filesystem does not hold any image layers or writeable layers. The kubelet to place container image layers and writeable layers is on this second filesystem. In this configuration, the directory where you tell You also use a separate filesystem, backed by a different logical storage device. The kubelet also writes node-level container logs into the first filesystem, and treats these similarly to ephemeral local storage. In the abstract this is relatively easy to debug: use kubectl exec to get a shell in the pod, and then use normal Unix commands like du to find where the space is going. Something inside your process is using a lot of local disk space. You can use this filesystem for other data (for example: system logs not related to Kubernetes) it can even be the root filesystem. 'Ephemeral storage' here refers to space being used in the container filesystem that's not in a volume. You have a filesystem on the node that you're using for ephemeral data that comes from running Pods: logs, and emptyDir volumes. This bit from the k8s docs explains the implications of splitting the two (emphasis mine): So we end up with /var/lib/kubelet + /var/log. Size of cluster (how many worker nodes are in the cluster?) 13ĪKS seems to currently uses the first configuration, where root dir contains both 'ephemeral storage' and other files.Kubernetes version (use kubectl version): 1.12.6.I'm not certain how where the ephemeral storage is mounted on a node or how much is allocated to it, but presumably just deploy an AKS cluster with a temp disk larger than 32 GiB

#Ephemeral storage kubernetes how to#

How to reproduce it (as minimally and precisely as possible): Since this class of VM has 128GiB of temp disk, I would expect the ephemeral storage to be mounted on that scratch disk and have roughly that much capacity