ESX

...now browsing by category

 

A virtual machine gone bad – how to shut down an unresponsive VM

piątek, Sierpień 30th, 2013

Some time ago I encountered a problem with a VM that didn’t want to shut down. First, the problem was the VM was the vCenter server. And I mapped a CD via vCenter to itself (mind: from a local disk, not from a datastore). After I started copying data from the CD to the VM it became frozen. As VM monitoring was configured and the machine stopped sending heartbeats, VM monitorin decided it was time to shut down and power on the VM. The shutdown was stuck at 95%. Now, I know there are many stories about hanged VM all over the Internet but stay with me.

A usual stuff (service mgmt-vmware restart) did clear the status of the poweroff in progress but the VM was still running as I was going to learn. The VM appeared as inactive and the display name changed to unknown. I removed the machine from inventory and tried to register it again but with the same effect. apparently it was still running on the host. I managed to change the name of vmx file and register it correctly but of course it did not power on – the virtual disk files were locked.

In this case find the virtual machine world id:

vm-support -x

and kill the machine:

vm-support -X VM_world_id

Answer „no” to screenshot question and „yes” to the other two. It’s gonna take some time. After that the machine could be registered correctly and I could power it on.

Extending virtual disk over 1TB (on ESX 3.5)

piątek, Grudzień 14th, 2012

Yes, some people still use ESX 3.5. So please stop laughing and check this out: this is a gentle and informative message you get when you try to extend the virtual disk size (let’s say from 1.4 to 1.5 TB):

Surprise!

Surprise!

Say what? But fine, Google is your friend and here it is – VMware KB: Cannot extend a vmdk over 1TB. A quick quote:

The only way to increase the size of a vmdk beyond 1023 GB in ESX 3.5 using vmkfstools.

Well, thanks a lot VMware! Not cool, bros, not cool…

CPU Identification Utility by VMware – check your CPU

środa, Listopad 14th, 2012

Some time ago I described a HP tool to check CPU compatibility with Hyper-V. Here’s a similar tool from VMware.

CPU Identification Utility is a tool designed by VMware to help you with finding out what features your your server’s CPU support. It can be very useful when troubleshooting CPU compatibility for VMotion and identifying if the CPU provides support for 64-bit guests.

Here’s the link to download the iso. Boot your server with it and it will check your CPU for VMotion compatibility presenting you with a similar screen:

CPUID results

CPUID results

Upgrade to vSphere 5.1 – Part 6 – Upgrading datastores

poniedziałek, Listopad 12th, 2012

vSphere 5 introduced new file system version: VMFS-5. It brings a lot of improvements for scalability and performance (details can be found here) so if you haven’t done it already this is how you can upgrade your VMFS-3 datastores to VMFS-5.

The operation is non-disruptive for VMs hosted on the datastore but cannot be rolled back. To upgrade a datastore successfully, all hosts with access to this datastore must be ESXi 5.0 or higher.

Go to Hosts and Clusters inventory view, select a host that can access the datastore to be upgraded. Go to the Configuration tab and under Hardware select Storage. This will show all datastores the selected host has access to:

VMFS datasores

VMFS datasores

Select the VMFS-3 datastore. Below you will see a link Upgrade to VMFS-5. Click it.

Upgrade link

Upgrade link

A warning will appear confirming that all your hosts connected to the datastore must be in version 5. Click Ok.

All hosts must be ESXi 5 or higher

All hosts must be ESXi 5 or higher

That’s it. The datastore will be upgraded and you will be able to use new features such as GPT and uniformed block size.

What about block size and partition table on upgraded datastores?

When creating a VMFS-3 datastore you had an option to choose a block size for the new partition:

Supported block sizes

Supported block sizes

This is no longer an option as VMFS-5 uses only one, uniformed block size – 1 MB. However, when a VMFS-3 datastore is upgraded to VMFS-5, it will retain all partition characteristics with one exception: MBR will be converted (seamlessly) to GPT at the moment the datastore’s size exceeds 2TB.

The rest of the partition features will remain the same so VMware suggest temporarily moving the VMs with Storage VMotion to another datastore and recreating the empty one with VMFS-5 from the scratch.

Types of swapping on ESX hosts

piątek, Listopad 9th, 2012

One of the most important features of virtualization and the reason why virtualization is so cool is the ability to overcommit resources. In terms of RAM it means that we can configure virtual machines with more RAM than we have physical RAM installed in a host. When configured we rely on ESX server to automatically manage the resources in a way that unneeded memory is taken away (reclaimed) from a VM and instead allocated to another VM in need.

In this article I am not going into memory reclamation techniques details and I will assume you are aware of terms such as ballooning.

There are three types of swapping that can take place on ESXi host and here’s a short description of each of them. Usually 2 and 3 are confused so here is how they works and differ.

1. VMX swapping

Let’s start with VMX swapping as it is the simplest one to describe. ESX host allocates memory not only for VM’s operating system use but also for different components such as virtual machine monitor and virtual devices. In an overcommited cluster this memory can be swapped in order to allow more physical memory for VMs that need it. According to VMware this feature is able to reduce memory usage from about 50MB per VM to 10MB without noticeably impacting VM’s performance.

It is enabled by default and VMX swap files are kept in VM folder (the location can be changed with sched.swap.vmxSwapDir parameter). VMX swapping can also be disabled on a VM with sched.swap.vmxSwapEnabled = FALSE.

2. Virtual machine swapping

This is what the operating system installed on a virtual machine will do when it realizes that the memory is low – it will swap. Please note that it has nothing to do with host’s physical or virtual memory. For example a machine with 4GB of RAM configured can have this RAM allocated by the host in host’s physical RAM or into swap files (vide: point 3) – for this VM it does not matter and it will treat the whole 4GB as if it was physical memory simply because it has no idea that this RAM is virtualized.

When the VM wants to use more than 4GB of memory, it will use swap techniques depending on the operating system, in Windows the VM will swap to the pagefile, in Linux it will use swap partition, etc. Therefore the swapped data will go directly to the VM’s virtual disk, i.e. to the vmdk files on the storage (when you think about it, it is logical as pagefile/swap partition are a part of VM’s virtual drives).

This is exactly what happens when the ballooning driver is in use. When it inflates it makes the VM operating system allocate to it VM physical memory and if the VM does not have free RAM at the moment it will start swapping (ballooning driver will make sure it receives unswappable, real RAM). And this is ok as the VM operating system knows best which data can be swapped from RAM to disk.

3. Host swapping

When you realize that your host is swapping a lot, it means that the cluster is highly overcommited and all other memory reclamation and save techniques (ballooning, page sharing and memory compression) were not enough. The host is in Hard state (1%-2% memory free) or in Low state (1% or less). This condition will severly impact performance and should be avoided at all costs. Another problem with host swapping is that you have no control over what is swapped – inlike VM swapping, the host does not know which part of data in the memory is important for the VMs.

So where this time the memory is swapped? When you open a datastore you will notice a .vswp file.

.vswp file in a VM directory

.vswp file in a VM directory

This is true only for a powered on VM – at power on the host will create a file and will remove it when the machine is powered off. Lokk at its size. The size it equal to VM configured memory minus reservation. This is because for the host must assure there is RAM available for the VM when it is switched on (regardless if it is a real, physical RAM or virtual memory swapped on disk). The reservation will assure that the given amount of physical RAM is available but what about the rest. The host cannot inform the machine that there is simply no RAM for it, see ya later. Hence the vswp file.

Look at this example. I have a turned off VM configured with 4GB of RAM but on the datastore there is only 1.8GB free. What happens when we try to start the VM?

No space for the swap file = no fun

No space for the swap file = no fun

So what can we do now? Well, we can free up some space on the datastore of course but if it is not possible this is what can be done instead. By default the swap file is kept in the VM directory. This can be changed – if the host is standalone, go to Configuration > Software > Virtual Machine Swapfile Location. If it a part of the cluster have a look in cluster settings first to change it to „Store the swapfile in the datastore specified by the host” and then go to the host configuration mentioned below:

Swap file location in cluster settings

Swap file location in cluster settings

 

Host settings - swapfile location

Host settings - swapfile location

I will choose the datastore with enough space and try to start the VM and, sure enought, it starts fine.
The location of swap files can also be modified on the VM level. Just open VM settings, go to Options tab and choose the option you like:

 

Virtual machine settings - swapfile location

Virtual machine settings - swapfile location

Another reason why you may want to change the swap files location is performance – you can put swap files on a diffrerent storage tier, on a slower/ cheaper if you’re sure there will be almost no swapping (to save on better-performance storage) or, if the cluster is overcommited and there will be swapping – use a faster tier to improve performance.

Inconsistent iso file – be careful what you click!

poniedziałek, Listopad 5th, 2012

Recently I’ve discovered that after a power outage some files on my storage, including iso files, have been damaged. This is a message that appears when I try to map the iso to a virtual CD-ROM on a VM running on ESXi 5.1:

The operation on file failed

The operation on file failed

Be careful though what you click. If you select Cancel „to end this session” your VM will stop with a similar error message:

VMware ESX cannot synchronize with the disk before cancelling. Disk /vmfs/volumes/5017d683-32e5bd3d-1654-001cc494e6fc/ISO/VMware-VIMSetup-all-5.1.0-799735.iso may be inconsistent.

The option to follow is Continue „to forward the error to the guest operating system”. It will eventually allow you to disconnect your iso and the VM will keep running.

You can disconnect the iso

You can disconnect the iso

VMware PVSCSI overview

środa, Październik 24th, 2012

VMware Paravirtualized SCSI (VPSCSI) has been with vSphere since version 4.0. With versions 4.1 and 5.0 its limitations were generally removed so that this type of adapter can now be broadly used.

VMware Paravirtualized SCSI adapter goes hand in hand with vmxnet3 paravirtualized NIC and offers better performance for I/O intensive disks. Our old friends – BusLogic and LsiLogic adapters emulates physical devices they are based on. That means that most modern operating systems will have correct drivers shipped with them and will work with these SCSI controllers out-of-box. VPSCSI on the other hand requires drivers, has some limitations (or used to have as we will see) but works on guest systems in much more efficient manner providing you with better performance using fewer CPU resources.

When introduced in vSphere 4 VPSCSI presented some problems. This is a short lists of things you need to consider before you deploy a VM with VPSCSI on vSphere 4.0:

  • Not suitable for VMs with VMware Fault Tolerance,
  • Worse results on low I/O intensive disks (lower than 2000 IOPS),
  • Severe problem on Windows Server 2008 R2 and Windows 7 virtual machines (see this excellent post from Michael Webster for more details and where to find fixes).
  • And last but not least – it is not possible to use the VPSCSI adapter for boot disks.

So as one can see VPSCSI adapter was designed with the following in mind: “Use LSI Logic adapter for your boot and OS disk and introduce VPSCSI with read/write intensive disks for solutions such as MS SQL Server or Exchange”.  Fortunately all the problems stated above has been already fixed and in vSphere 5.0 we can enjoy VMware Paravirtualized SCSI under all circumstances.

For details check this excellent article by Scott Lowe on PVSCSI limitatuons and his update on fixes introduced in vSphere 4.1.

More resources:

VMware KB: Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters

Definite guide to iSCSI with FreeNAS on ESX(i) – Part 3 – Configuring iSCSI on ESX 3.5 (standard vswitch)

piątek, Październik 19th, 2012

This is Part 3 of the definite guide to iSCSI with FreeNAS on ESX(i). Make sure to check out the other parts as well.

1. Introduction
2. FreeNAS installation and configuration
3. Configuring iSCSI on ESX 3.5 (standard vswitch)
4. Configuring iSCSI on ESXi 5.1 (standard vswitch)
5. Configuring iSCSI on a distributed vswitch
6. Migrating iSCSI from a standard to a distributed vswitch

 

3. Configuring iSCSI on ESX 3.5 (standard vswitch)

Make sure your ESX server has got at least two NICs – if you’re doing this in the nested, virtual environment and you have created a virtual ESX with only one NIC, shut it down, add a new network card and start the server again. We will use one for management and the second for iSCSI configuration. You will probably want to use more NICs anyway (for VM traffic, VMotion, redundancy, etc.) but for this demonstration we will use only two NICs – vmnic0 (management) and vmnic1 (iSCSI).

After you have installed ESX 3.5, go to the Configuration tab and under Hardware click on Networking.  All you will see is one virtual switch (vSwitch0). First of all we will create a new vswitch for iSCSI traffic. In the upper-right corner click on ‘Add Networking…’.

Select VMkernel and click Next.

Select VMkernel

Select VMkernel

Select ‘Create a virtual switch’ (this option is available if you have at least one unassigned NIC) and select an uplink NIC to be used with your new vswitch. Click Next.

Adding an uplink tpo the vSwitch

Adding an uplink tpo the vSwitch

Now you will create a new portgroup on your new vswitch. Give it a meaningfull name (label) like ‘iSCSI’ for example. Configure VLAN (if necessary, if not – leave this field empty).

Setting IP configuration for VMkernel port

Setting IP configuration for VMkernel port

Click Next and then Finish. A new virtual switch called vSwitch1 is created.

New virtual switch

New virtual switch

Under Hardware click on Storage Adapters. And select iSCSI Software Adapter. Click on Properties.

Software iSCSI initiator

Software iSCSI initiator

The software initiator is disabled by default so click on Configure and enable it.

Enabling software adapter

Enabling software adapter

You will see the following message:

Service Console needs to be able to communicate with iSCSI storage too

Service Console needs to be able to communicate with iSCSI storage too

Here’s the thing – target’s discovery is done by the Service Console. That means that both ports: VMkernel (the one we created) and Service Console (already created on vSwitch0) must be able to communicate with iSCSI target (on the FreeNAS server). If you look on the screenshot above you will see that both Service Console and VMketnel port for IP storage are in the same subnet (as well as FreeNAS server of course) so they both will be able to communicate with the target. However, if your Service Console is in the different network (and it should be if it is a production environment in order to introduce traffic separation), you will need to create a second Service Console – on vSwitch1 – and assign it an IP address allowing it to discover the FreeNAS iSCSI target. In our case it is not necessary.

Let’s continue with the software initiator’s confiuration. Click on the Dynamic Discovery tab and click Add… Put the IP address and you configured on FreeNAS Target.

IP  and port of FreeNAS server iSCSI service (remember the portal in Part 2?)

IP and port of FreeNAS server iSCSI service (remember the portal in Part 2?)

You will be asked to rescan the HBAs. Click on Yes.

Rescan the host's HBAs

Rescan the host's HBAs

If all goes well, you will see the target device:

LUN presented on FreeNAS to ESX host

LUN presented on FreeNAS to ESX host

Now you can go on and create a new storage for ESX server on the LUN (VMFS or RDM).

Definite guide to iSCSI with FreeNAS on ESX(i) – Part 1 – Introduction

piątek, Październik 12th, 2012

This is Part 1 of the definite guide to iSCSI with FreeNAS on ESX(i). Make sure to check out the other parts as well.

1. Introduction
2. FreeNAS installation and configuration
3. Configuring iSCSI on ESX 3.5 (standard vswitch)
4. Configuring iSCSI on ESXi 5.1 (standard vswitch)
5. Configuring iSCSI on a distributed vswitch
6. Migrating iSCSI from a standard to a distributed vswitch

 

1. Introduction

Since many search queries on my blog lead to this article where I struggled with FreeNAS configuration on nested ESXi 5.0, I decided to write a guide how to configure FreeNAS to work with ESX and ESXi, both in „regular” and „nested” environment, on standard and distributed switches. By „nested” I mean ESX servers installed as virtual machines on a physical ESX(i) or on VMware Workstation.

iSCSI is configured on ESX and ESXi in slightly different manners so they will be covered separately. I would also like to show you how to configure iSCSI connection on a distributed virtual switch and how to migrate to a distributed vswitch a working connection from a standard vswitch.

When writing this article I assume that you’ve got:

  • basic understading of VI / vSphere netowrking concepts
  • working knowledge of vSphere client

Have fun reading this guide and please laeve me a comment or contact me directly if something’s not clear or if you have additional questions/issues/doubts.

Distributed virtual switch and the .dvsData folder

poniedziałek, Październik 8th, 2012

If you use a distributed virtual switch you can notice that on some of your datastores there is a folder called .dvsData. Why is it created and what it is used for? Here you will find a description of this folder’s role.

As you can see below, a virtual machine called Workstation 1 is connected to the „Virtual machines internal” vdportgroup which in turn is on the vds called simply „ds”.

Workstation 1 connected to the distributed portgroup on the distributed virtual switch

Workstation 1 connected to the distributed portgroup on the distributed virtual switch

Virtual switches view

Virtual switches view

The VM is stored on the VMFS I datastore:

Datastore view

Datastore view

where one can also see a folder called .dvsData. Let’s have a look inside:

A look inside .dvsData folder

A look inside .dvsData folder

In the .dvsData folder there is a sub-folder which name suits the UUID of the distributed switch:

Checking vds's UUID with the esxcli command

Checking vds's UUID with the esxcli command

In this subfolder you will find one more files and their numbers will suit the number of ports the virtual machine(s) is connected to:

File listing

File listing

The file name's matches the port's number

That means that .dvsData will appear on the datastore where your VM’s configuration file (vmx) is stored. If you open the configuration file you will find the same information:

VM's configuration file networking part

VM's configuration file networking part

This information is synchronized (from vmx to .dvsData folder) by the host every 5 minutes. So now you know what this folder is, why subfolder and files’ names are so strange, etc. But what’s the purpose of the .dvsData folder and its content?

It is used by HA. Imagine the situation when your host fails and HA starts a VM on another physical server. It needs to know which port is should connect the machine to. I did a very simple test – I removed the file called 264 from the datastore and I stopped the ESXi server the VM Workstation 1 was running on. A few seconds later HA detected a possible failure of the host. However this was the result on the VM:

Oops, the failover failed...

Oops, the failover failed...

Leson learnt: don’t touch the folder .dvsData unless you really know what you are doing.

PS. As you see all operations in this note were done using the new Web-Client. More on this client soon.