How VMware Virtual Machine vMotion works?

Or, How Virtual Machine vMotion works in backend in VMware?

Or, Understanding technical background processing of Virtual Machine’s vMotion.

Or, Sequence of Virtual Machine vMotion operation in VMware.

Virtual Machine vMotion: vMotion allows Virtual Machine to be migrated from one ESXi host to another ESXi host without causing any downtime to the Virtual Machine.

If you are performing a normal vMotion of a Virtual Machine, vMotion do not copy or migrate all contents of the Virtual Machine to destination host (e.g. datastore), it only copies the memory state(CPU and RAM information) of the Virtual Machine and migrate it to elected (manual or automatic) ESXi host using Pre-Copy Mechanism.

Pre-Copy Mechanism: Virtual Machine’s vMotion uses Pre-copy mechanism for migrating memory state of the virtual machine from one to another ESXi host which prevents any downtime or connectivity breakdown of the Virtual Machine which is being migrated.

Example: If a VM is having 16GB RAM capacity and its being migrated to another host, First It will initiate migration of free memory (e.g. 10GB, Please Note: - VMs use memory what needed by them up to maximum allocated capacity, if you have given 16GB memory, means not that the VM will always reserve 16GB RAM capacity for it) to the destination host. Now, the VM dirties 6GB Memory, it will again attempt migration of remaining 6GB memory and if still some dirties memory is remaining the same process will be initiated again till the time all memory state is migrated.

Once all the memory state is migrated successfully to destination ESXI host, the vMotion operation is completed successfully.

IMP Note: For Device state and CPU migration, Checkpoint mechanism works. Once all the Memory state is migrated to destination host, it free up the VM’s resources from source host.

Sequence of VM vMotion operation:

1. Pre-Copy Memory state from Source to Destination.

2. Quiese VM on Source Machine.

3. Transfer Device state from source to destination.

4. Resume VM on Destination.

5. Copy remainder of memory from source to destination.

6. Free VM resources from Destination.

Friends, I have written it based on my best possible understanding of this VM vMotion concept. Please write me about your feedback, suggestion or if you feel anything should be corrected…

Happy Learning. Cheers…

How to decrease drive capacity of a Windows OS based Virtual Machine in VMware?

Or, Can we decrease drive capacity of Virtual Machine having windows operating system in VMware?

If you are looking for a simple way of decreasing the drive space of a virtual machine as you do for increasing drive space, it’s not there in VMware. So, the answer is NO, we can’t decrease the drive space of a Virtual Machine in VMware but there is a way out of doing it and that is V2V migration of the Virtual Machine.

Possibility: For decreasing the drive capacity of a virtual machine you may perform V2V migration of the virtual machine by using VMware vSphere Standalone Converter (use o latest one). If you don’t want your exiting production to be impacted, you may choose to perform V2V migration of the virtual machine in powered on state.

When you reached to the “Options” page of the converter tool, you are required to follow below steps to decrease the drive capacity of a Virtual Machine (Please refer below screenshots with highlighted steps to be taken):

To Know about How to do V2V or P2V conversion of a virtual machine, please see another article:

How to do V2V or P2V migration in VMware?

IMP Points To Note:

1. Once the Migration is completed, you can check your destination VM’s drive capacity it should be appearing as you have defined the value (in GB) as per last screenshot.

2. In some cases, you may need to re-join your newly migrated VM in domain to make it fully functional.  It better you delete the old computer account and then re-join this machine in domain to create a new computer account for this new VM instance.

3. You may need to re-activate the license of the Windows machine after this activity.

Cheers, You may write me back for any suggestion or feedback..

What is difference between HA (High Availability) and FT (Fault Tolerance) in VMware?

Or, How HA and FT works in VMware?

Or, What is HA (High Availability) in VMware?

Or, What is FT (Fault Tolerance) in VMware?

HA (High Availability) in VMware:  VMware HA provides high availability on VMware Instances. In-case any of the ESXi host in a cluster will go down the Virtual Machines running on this particular ESXi host will be migrated to another available ESXi host.

But in case of any sudden hardware failure, which does not give enough time for VMs to be migrated to another available host, the VMs become unresponsive and in some cases it requires a reboot to be up and running...  

FT (Fault Tolerance) in VMware: FT provides very efficient high availability of Virtual Machines by keeping one copy of virtual machine continuously replicated with all live contents changes. In-case one virtual machine will be down due to any reason like Host Sudden failures etc.. Another virtual machine running on some other available ESXi host in the same cluster will automatically come up in fractions of seconds (1 RTO approx.).

This feature is very useful when you are running very critical business application on Virtual environment e.g. Exchange Server.

Understanding Virtual Machine Snapshots in VMware!

Or, Virtual Machine Snapshot in VMware – How it works?

VMware Virtual Machine snapshot is very useful tool to take instant temporary backup of a virtual machine anytime. It preserves the state of a virtual machine at any specific point of time. If you are performing any activity which may cause some changes on application or OS level, it’s better to have a snapshot of the VM before proceeding with the activity.

In-case somethings went wrong, you can revert the Virtual Machine’s snapshot to the last recent well stable state in few minutes (depending on the VM size).

A VM snapshot preserves states of following files while performing VM snapshot:

1. Configuration State

2. Memory State

3. Disk State

What is difference between ESX and ESXi servers in VMware?

Or, Understanding technical difference between ESX and ESXi server.

ESX Server: ESX stands for Elastic Sky X. ESX server is VMware hypervisor platform which runs using console operating system. In ESX, VMkernel is the virtualization kernel which is managed by a console operating system. This console operating system is known as Service Console.

Any management agents or third party agents are installed on Service Console in ESX servers. In ESXi server environment, any host processes runs on Service Console.

Whereas,

ESXi Server: ESXi stands for Elastic Sky X integrated.  This is also as VMware hypervisor platform from which Service Console concept has been removed. ESXi Server uses DCUI (Direct Console User Interface) in place of Service Console.

Any management agents, third party agents, hardware agents or Host processes of ESXi servers directly run through VMkernel.

For Understanding VMware terminologies or technicality behind concepts of Service Console, DCUI (Direct Console User Interface), and VMkernel, please refer my another article given below:

Explain Service Console, DCUI (Direct Console User Interface) and VMkernel in VMware

What is Difference between Service Console, (DCUI) Direct Console User Interface, and VMkernel?

Or, Explain Service Console, DCUI (Direct Console User Interface) and VMkernel in VMware.

Or, Understanding VMware terminologies or technicality behind concepts of Service Console, DCUI (Direct Console User Interface), and VMkernel.

Service Console: Service console conceptually is a command line interface for the modified version of Linux which is ESX Server. This Service console provides almost all kind of manageability, troubleshooting, and monitoring interface for ESX servers.

Direct Console User Interface: DCUI allows users to have direct access of the ESXi hosts for numerous management and administrative tasks. Tasks like IP configurations, Troubleshooting, Logs collection or management network configurations can be performed on DCUI console.

VMkernel: VMkernel is responsible for handling TCP/IP stack traffic for some ESXi services like vSphere vMotion, Host Management, NFS and ISCSI.

VMkernel is used for management network in VMware networking architecture. 

VMware vSpehere Update Manager Installation Error "The vCenter Server Entered is Not Reachable".

Or, VMware vSpehere Update Manager Installation Failed With Error "The vCenter Server Entered is Not Reachable".

Resolution:
So Friends, This was one of the very silly issue with VMware vSphere Update Manager Installation which I shorted out by just changing the default port number 80 to some other available port number on the vCenter server.

Error message looks like below:

What is Ready Percentage in VMware ESXTOP Command?

What is Ready Percentage?

If multiple idle physical CPUs are not available when the virtual machine wants to run, the virtual machine remains in a special wait state. The time a virtual machine spends in this wait state is called ready time.

For better performance of a VM, this value should be below 5.5 approx...

VMware Virtual Machine's resources allocation common Best Practices !

While allocating RAM or CPU to a Virtual Machine In VMware, we always thinks that how much memory or CPU should we allocate to a Virtual machine.

The answer is, Allocate minimum as much as you can according your application requirement running on the Virtual Machine. Because, over allocating resources to a virtual machine slow down the performance instead of improving it.

Recommendations:
Do not over allocate Resources (CPU/RAM) )to a Virtual Machine. Provide only required resources to a VM. Start by giving minimum number of vCPUs to the VM and keep on increasing them gradually until the VM perform as intended.

Example: I am taking an Example of a Virtual Machine with over allocated CPU cores.

Sometime you can observe that your ESXi host is hosting only one VM but the CPU usage of the Host is reaching to 100%. It happens, when a VM is oversubscribed in terms of CPU.

Another symptom is, if you will run ESXTOP command on the particular ESXi host, the affected VM’s Ready Percentage would be going up to 1000 or more which should be below 5.5 approx..

To check the ready percentage on ESXi host, SSH your ESXi host with root credentials > Run ESXTOP command and see the value highlighted in below screenshot:

If it is too high, please consider decreasing the CPU capacity and then monitor the performance of the VM. For more insight on this, you can refer the VMware KB: http://kb.vmware.com/kb/1010184

To know more about Ready Percentage, see my another article:
What is Ready Percentage? 

Cheers..

How to choose right number of Virtual Sockets and Core per Socket for a Virtual machine in VMware?

Or, What are the factors to consider while allocating CPU cores to a Virtual Machine?
Or, CPU Core allocation for a Virtual Machine in VMware vSphere.

Factors that should be considers while allocating CPU cores to a Virtual Machine:

On Virtual Machine:

1. Number of virtual sockets

2. Number of cores per socket

On ESXi Host:

1. Processor Sockets

2. Processor Cores per Socket

3. Logical Processors

Descriptions:

The truth is, you can choose to have any number of Processor Socket and Core per Socket while allocating vCPUs to any Virtual Machine. That’s the real virtualization concept. No matter how much processor socket your ESXi host have, or how much processor cores are mentioned per socket.

The only thing is, Number of CPU cores should match your requirement.

Example: If you want to allocate 6 vCPU (Cores) to a virtual machine, you can do it one of way shown below (both are correct):

Another thing that we should focus on is total number of Logical Processors (per host).

Logical Processor (host): This shows that, how much maximum cores can be allocated to a virtual machine if you want to keep one ESXi reserved for any specific virtual machine. If the total numbers of Logical Processors are 40, you can allocate 40 cores to a VM.

Please Note: It does not mean that you can’t allocate more than 40 Cores to a VM. But the good practice is to not overload the ESXi host. If you will overload it, its going to reduce the performance of the VM.

Overall allocation can go beyond 40 cores combining all/multiple VMs running on a ESXi host but the allocation for a single VM should not exceed the maximum capacity of it’s current host. In short, VMs can share their resources between VMs as per the usage and requirements, but Hosts can’t share resources with another Host in the cluster.

Example: lets understand what’s good and what’s not with the help of below graph.

Hope it will make your understanding clear about CPU allocation in VMware virtualization. For any feedback, suggestion or query, please write me back in comment box... above explanation is based my understanding of resources allocation. It may be changed or updated.

Cheers..!!

The greatest confusion between VMware vSphere performance report of a VM’s CPU/RAM usage and the actual usage trend of CPU/RAM inside the VM’s Guest OS.

Or, Why CPU and RAM usage percentage of Guest OS is different than it appears in VMware vSphere performance monitoring tab?

Answer: Please read this full article to understand.

Or, Can VMware vSphere give you actual CPU/Memory usage trend of Guest OS according to inside utilization status?

Answer: No

Or, Is VMware vSphere capable of Monitoring RAM/CPU usage of Guest OS (e.g. MS Windows)?

Answer: No

Friends, it has been quite long since I have been looking for some crystal clear and authentic information about these confusions, and finally I got all my answers with from VMware Technical Support... So, I thought to share this information with you all so that at least you can get some idea behind the scene... However, I would recommend you to please get in touch with VMware Technical Support if you still need some clarifications.

The conclusion of this post is to clarify that the VMware vSphere does not have any intelligence to monitor resources usage inside the Virtual Machine (guest OS).

My Observation: When I checked my Virtual Machine(Windows Server 2012) from inside the windows, it was utilizing 95% of memory but when I checked the Virtual Machine performance graph from vSphere client, it was showing memory usage only  10% to 20%. Isn’t it shocking?

Concepts behind the performance monitoring in VMware vSphere:

VMware performance statistics (esxtop and performance chart) show Memory and CPU details from vCenter and ESXi perspective. These stats are for monitoring the usage of resources for VMs on the ESXi hosts, Parameters such as Memory usage, Consumed and active all show the amount of memory used, consumer and currently in use as seen by ESXi host kernel ( vm kernel ) for each VM .

VMware does not intrude into the guest OS performance stack and do not have any parameters which shows the amount of CPU, Memory and Disk usage seen within the guest OS. This because each OS has its own method and architecture for managing its resources.

For instance, There are times when the Guest OS (windows perfmon, etc) will show lower CPU usage than VMware reports.  The guest doesn’t know anything about the CPU used to virtualize the hardware resources it is requesting. ESXi does, and accurately attributes that load. 

There are also times when the Guest OS will report a higher usage of CPUs than the hypervisor will. Windows doesn’t know that part of the time, the CPUs are being stolen away from it and given to other guest machines. As far as it knows, it is using all the CPU it can on 3 of the 4 CPUs- so that must equal 75% load. It doesn’t know that part of the time, it has no access to the CPU’s as they are being used elsewhere, so the real time CPU usage is only about 50%.

Key take away from this is that we need use both VMware performance stats as well as guest OS specific monitoring tools like perfmon, task manager etc . . you cannot use the ESXi host’s view of the CPU/Memory load to tell if a system wants more than the resources allocated to it – only how much of the resources allocated to it are being used from ESXi stand point.

There are different parameters to understand the memory and CPU usage of a Virtual Machine in vSphere client, like Memory Usage, Active Memory, and Consumed Memory. To know more about these parameters, please read my article.

Understanding Memory Usage, Active Memory, and Consumed Memory in VMware vSphere.

What is difference between Usage, Active and Consumed parameters of performance monitoring in VMware vSphere ?

Or, Understanding Memory Usage, Active Memory, and Consumed Memory in VMware vSphere.

Memory Usage -
This parameter shows how much % of memory or CPU is being used by the Specific Virtual Machine against of the ESXi host on which this Virtual Machine is existing at that particular time.

For Example: If your virtual machine is assigned with 10GB RAM and the ESXi host which is hosting this VM is having 100GB RAM, and the RAM usage in vSphere client for this VM is showing as 5%, it means Your virtual machine is utilizing 5% memory of the current ESXi host not 5% of total assigned memory to this virtual machine (10 GB).

Active Memory:
it shows amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages. Because, VMs does not use all the allocated memory if it’s not needed by them actually. It only use the memory what it need.

Consumed Memory:
Amount of guest physical memory consumed by the virtual machine for guest memory. Consumed memory does not include overhead memory. It includes shared memory and memory that might be reserved, but not actually used. Use this metric for charge-back purposes.

I have written it as per my best understanding till now. If you have any query or suggestion, please write me back.. 

Explanation for these parameters are different for ESXi Hosts. The above explanation is valid only for Virtual Machines.

Explained: EccSinglebitErrors and EccMultibitErrors in CISCO UCS Blade Servers

However there are lots of architectural differences in memory errors EccSinglebitErrors and EccMultibitErrors, the most logical difference is as below:

EccSinglebitErrors: These errors are also known as correctable errors which means if you will swap the  memory encountered with this error with another healthy memory in the same blade server, the erroneous memory will become healthy. But if the error counts are huge in number, then it may not work sometime by swapping it.
In this scenario, you have to replace the memory module with new one.

EccMultibitErrors: These errors are also known as non-correctable or in-correctable  errors which means there is no way to solve this error at your end. Even a Single Multibit error is indication of it's hardware fault and it will be resolved only after replacement of DIMM(as per my working experience on such issues with CISCO Tech support).

It looks like below:

How to avoid unexpected VMs reboot?

Or, Common causes of unexpected reboot of Virtual Machines.

Or, VMware Event: HA rebooted the Virtual Machine

Or, Why to disable “VM Monitoring” in VMware vSphere Cluster setting?

Imp Note: Hey Guys, I have seen sometime that Virtual Machines are getting rebooted automatically and when you check Windows event logs, it will show you that the last reboot of the server was unexpected.

Definitely, there are multiple factors which can cause Unexpected Server reboot but I would suggest checking few Settings on your VMware Cluster as well. One of the factor that I have seen is, If you have enabled VM Monitoring Setting of the VMware Cluster, this will reboot your VM servers whenever VMware tool heartbeats are not received within a set time. 

Likewise, another component of VM Monitoring is Application Monitoring, which also cause VMs to be rebooted unexpectedly if it not receives the VMware tool heartbeats within a set time.

Its good to go ahead and disable the "VM Monitoring" option in VMware cluster.

To get it done, please follow the steps given below:

Login to vSphere Clientwith admin privilege > Right Click on your Cluster > Go to Edit Settings > Go to VM Monitoring> Under VM Monitoring drop-down menu, Select Disabled, as highlighted in below screenshot:

Reference KB (VMware): http://kb.vmware.com/kb/1027734

Cheers, its done now.

How To Integrate vSphere 6.0 With AD ?

Or, How To Integrate vSphere 6.0 With Active Directory?

1. Make sure your vCenter Server is joined in your domain; it should not be part of Workgroup.

2.  If it is not already joined in domain, please join it in domain.

3. Open CMD with Administrative privilege:

    Go to Search> Type CMD > Right Click on it and Say Run as Administrator

4. Locate to directory “C:\Program Files\VMware\vCenter Server\VMware Identity Services\scripts” By typing on CMD:

cd C:\Program Files\VMware\vCenter Server\VMware Identity Services\scripts

      5. Type below command and hit Enter:

sso-add-native-ad-idp.cmd vmware.com

6. Wait for this command to be completed.

You can verify it now in vCenter vSphere console if the domain name is being highlighted in access rights management section or not.

It should look like below reference screenshot(your domain name should be appearing there):

How to create upgrade baseline image in VMware vSphere 6.0 ?

Or, How To Create Upgrade Baseline Image In VMware vSphere For ESXi Servers 6.0?
Or, How To Create Baseline Image For ESXi Servers Upgrade?

Imp Note: Login to your vSphere Client from the machine where you already have Update Manager plugin installed. If not sure about this, please login to vCenter Server and access vSphere client from there.

Steps To Follow For Creating Baseline:

1. Go to Your VMware Cluster> Update manager > Admin View

2. Go to Baseline and Groups > Click on Create

3. Give Friendly Baseline Name and Descriptions > Select “Host Upgrade” > Click on Next

4. Select your appropriate ESXi Image (in my case, I have only one :) ) > Click Next > Finish

5. Now you are done with creating ESXi Upgrade Baseline Image for upgrading your ESXi servers.  You may verify the Imported ESXi Image under below highlighted tab for your assurance.

Cheers,...

If you have any query or suggestions please write me in the comment box..

How To Import ESXi Images in vSphere For Upgrading ESXi Hosts Using Update Manager?

Or, How To Import ESXi Image in vSphere For Creating Upgrade Baseline Images?

Imp Note: Login to your vSphere Client from the machine where you already have Update Manager plugin installed. If not sure about this, please login to vCenter Server and access vSphere client from there.

Lets Start with Importing ESXi server .iso Image now.

Steps To Follow:

1. Go to Update Manager > Admin View

2. Go to  ESXi Images > Import ESXi Images

3. Got to Browse >  Select appropriate ESXi .iso Image > Click on Next

4. Wait for ESXi Image to be be uploaded successfully > Click on Next

5. Give Baseline Image Name > Write a short Description(Not Necessary) > Click on Finish

6. That’s it, Your ESXi image has been imported successfully. It should appear similar to the below reference in screenshot.

Cheers!!!! You are done with Importing/Preparing ESXi Image in vSphere 6.0.

If you have any query/suggestion, please write me on comment box..

How to Upgrade ESXi Hosts Using VMware vSphere Update Manager?

Or, How to use VMware Update Manager to upgrade ESXi Servers?

Imp Note: : I assume that you already have a baseline image created with latest ESXi package. If not, please create a baseline image first before proceeding with the steps given below.

Steps:

1. Login to your vSphere client from a machine where you have already installed the update manager plugins. If not sure about it, login to vSphere client from vCenter Server.

2. Enter ESXi host in maintenance mode

3. Click on “Update Manager” as shown in below screenshot.

4.  Click on “Attach” > Select your ESXi upgrade base line under “Upgrade Baselines” option > Click on “Attach” to attach the base line image to ESXi host. Reference screenshot is given below.

5. After attaching the baseline image, the update manager windows should like below highlighted screenshot.

You can see that the ESXi host, for which you have selected the baseline image to upgrade, is appearing as unknown in right side “Host Compliance” section. It should either appear compliant or non-compliant.

6. Click on “Scan” > Check mark on “Upgrades” on the pop-up window > Click on “Scan” again as highlighted in below screenshot.

7. Now the “scan entity” process will be initiated to check the compliance status of the selected ESXi host. Please wait for this scan to be completed. You can monitor the “Recent Tasks” section on vSphere client for scan completion percentage as shown in below screenshot.

8. Once the scan is completed, your selected ESXi host will be appearing as “Non-Compliant” as highlighted in below screenshot. Please verify it and proceed to next step.

9. Now follow the Remediation process to initiate the ESXi server upgrade as shown in below screenshots (Please refer the steps highlighted in numbers 1,2,3 etc..)

10. Remediate task is going to take approx 20-40 minutes (can deffer), Once the Remediate task is completed, you can see the host status. Its now appearing as Compliant and the build version of ESXi host has also been changed.

Finally, you are done with upgrading your ESXi host using VMware update manager. You may verify your ESXi host health status before bringing it back to production cluster.

Cheers guys, please write me if you have any feedback or suggestion about this post..

How to configure SMTP server settings in VMware vSphere 6.0

Or, How to configure SMTP server settings in VMware vCenter Server 6.0?

Or, How to change default SMTP port number in VMware vSphere 6.0?

Configuration Steps:

1. Login to vSphere Web-Client with administrative rights

2. Go to Host and Cluster

3. Go to vCenter Server > Manage > Settings > Advanced Settings (refer the pointers 1, 2, 3, 4 in below screenshot)

4. Search for “smtp” in the search box on right side (refer the pointer 5 )

5. Change the values as per your need by editing them.

That’s All, you are done with configuring SMTP settings on VMware vSphere 6.0.

Unable to un-install/remove corrupted VMware tool.

Issue: Unable to install/unistall/remove VMware tool. Its asks for locating to c:\users\usersprofile\xxxx\temp\.msi setup file.

Cause: VMware tool may be corrupted or the stale entries of VMware tool in registry creating problem..

My Environment: Windows Server 2008 R2 X64

Solution(Steps):  Lets try force removal of VMware tool and then try to re-install it.

1. Map the VMware tool installation setup from guest>install/upgrade vmware tool option

2. Login to the machine which is having problem.

3. Open CMD(Run as Administrator)

4. Locate the Installation drive letter(in my case its D:\)

5. Run the command Setup64.exe /C

6. Say Ok

7. Remap the VMware tool installation file from (guest>install/upgrade vmware tool)

8. On the same session of CMD run the command Setup64.exe

9. It will initiate normal installation process of VMware tool.

10. Wait for VMware tool to be installed successfully.

11. Reboot your machine and you are done with installing VMware tool.

In my case it worked like awesome.. but before you do it, please verify your environment suitable for this scenario.

Let me know if you have any query or suggestion on this post.

Cheers, have healthy VM running... :)