One of the most important features of virtualization and the reason why virtualization is so cool is the ability to overcommit resources. In terms of RAM it means that we can configure virtual machines with more RAM than we have physical RAM installed in a host. When configured we rely on ESX server to automatically manage the resources in a way that unneeded memory is taken away (reclaimed) from a VM and instead allocated to another VM in need.
In this article I am not going into memory reclamation techniques details and I will assume you are aware of terms such as ballooning.
There are three types of swapping that can take place on ESXi host and here’s a short description of each of them. Usually 2 and 3 are confused so here is how they works and differ.
1. VMX swapping
Let’s start with VMX swapping as it is the simplest one to describe. ESX host allocates memory not only for VM’s operating system use but also for different components such as virtual machine monitor and virtual devices. In an overcommited cluster this memory can be swapped in order to allow more physical memory for VMs that need it. According to VMware this feature is able to reduce memory usage from about 50MB per VM to 10MB without noticeably impacting VM’s performance.
It is enabled by default and VMX swap files are kept in VM folder (the location can be changed with sched.swap.vmxSwapDir parameter). VMX swapping can also be disabled on a VM with sched.swap.vmxSwapEnabled = FALSE.
2. Virtual machine swapping
This is what the operating system installed on a virtual machine will do when it realizes that the memory is low – it will swap. Please note that it has nothing to do with host’s physical or virtual memory. For example a machine with 4GB of RAM configured can have this RAM allocated by the host in host’s physical RAM or into swap files (vide: point 3) – for this VM it does not matter and it will treat the whole 4GB as if it was physical memory simply because it has no idea that this RAM is virtualized.
When the VM wants to use more than 4GB of memory, it will use swap techniques depending on the operating system, in Windows the VM will swap to the pagefile, in Linux it will use swap partition, etc. Therefore the swapped data will go directly to the VM’s virtual disk, i.e. to the vmdk files on the storage (when you think about it, it is logical as pagefile/swap partition are a part of VM’s virtual drives).
This is exactly what happens when the ballooning driver is in use. When it inflates it makes the VM operating system allocate to it VM physical memory and if the VM does not have free RAM at the moment it will start swapping (ballooning driver will make sure it receives unswappable, real RAM). And this is ok as the VM operating system knows best which data can be swapped from RAM to disk.
3. Host swapping
When you realize that your host is swapping a lot, it means that the cluster is highly overcommited and all other memory reclamation and save techniques (ballooning, page sharing and memory compression) were not enough. The host is in Hard state (1%-2% memory free) or in Low state (1% or less). This condition will severly impact performance and should be avoided at all costs. Another problem with host swapping is that you have no control over what is swapped – inlike VM swapping, the host does not know which part of data in the memory is important for the VMs.
So where this time the memory is swapped? When you open a datastore you will notice a .vswp file.
.vswp file in a VM directory
This is true only for a powered on VM – at power on the host will create a file and will remove it when the machine is powered off. Lokk at its size. The size it equal to VM configured memory minus reservation. This is because for the host must assure there is RAM available for the VM when it is switched on (regardless if it is a real, physical RAM or virtual memory swapped on disk). The reservation will assure that the given amount of physical RAM is available but what about the rest. The host cannot inform the machine that there is simply no RAM for it, see ya later. Hence the vswp file.
Look at this example. I have a turned off VM configured with 4GB of RAM but on the datastore there is only 1.8GB free. What happens when we try to start the VM?
No space for the swap file = no fun
So what can we do now? Well, we can free up some space on the datastore of course but if it is not possible this is what can be done instead. By default the swap file is kept in the VM directory. This can be changed – if the host is standalone, go to Configuration > Software > Virtual Machine Swapfile Location. If it a part of the cluster have a look in cluster settings first to change it to „Store the swapfile in the datastore specified by the host” and then go to the host configuration mentioned below:
Swap file location in cluster settings
Host settings - swapfile location
I will choose the datastore with enough space and try to start the VM and, sure enought, it starts fine.
The location of swap files can also be modified on the VM level. Just open VM settings, go to Options tab and choose the option you like:
Virtual machine settings - swapfile location
Another reason why you may want to change the swap files location is performance – you can put swap files on a diffrerent storage tier, on a slower/ cheaper if you’re sure there will be almost no swapping (to save on better-performance storage) or, if the cluster is overcommited and there will be swapping – use a faster tier to improve performance.