Using VisualEsxtop to troubleshoot performance issues in vSphere

What is VisualEsxtop?

VisualEsxtop is a new performance monitoring tool that was recently posted on the VMware Labs Flings project. On Flings, apps and tools built by VMware engineers for fun are available for download. The intent is to make VMware Administrators’ lives easier in their daily work.

VisualEsxtop is an enhanced version of resxtop and esxtop. VisualEsxtop can connect to VMware vCenter Server or ESX hosts, and display ESX server stats with a better user interface and more advanced features.

How to install VisualEsxtop ?

1. Download visualEsxtop.zip from http://labs.vmware.com/flings/visualesxtop
2. Unzip visualEsxtop.zip to folder
3. Make sure Java 1.6 is in the PATH.
   1. On windows, to verify if Java is in the path,
      Click on Start > run > TYPE cmd and press ENTER > TYPE java and press Enter.
      If java is not in path, you will notice error like this –
      
   2. If JDK 1.6 or later is already installed on your machine but not in the path, here is how you add it (Instructions for Windows 7, other versions might be slightly different)
      -Go to Control Panel > System and Security > System >
      -Click Advanced system settings
      -Click on Environmental Variables
      -Click on New
      -Under Edit User Variable type the following and click OK
      Variable name: path
      Variable value: The path to the JDK 1.6 binary folder (C:\Program Files (x86)\Java\jdk1.6.0_14\bin\  for example)
   3. Then, open cmd again (Start > Run> Type cmd) and type “java”. This should successfully return usage options for java command.

What can you do with VisualEsxtop?

• Real-time Performance monitoring of individual ESX(i) hosts or vCenter Server. The default interval (5 seconds) is modifiable. Type Ctrl+N and change to the new value
• Multiple sessions to different hosts or same host at the same time. This comes in very handy when you are comparing stats between hosts or between multiple views/fields.
• Flexible counter selection and filtering. This is in my opinion the best feature of this tool. You can filter results to get specific outputs. The examples in the next section will show you how to.
• Save data to a batch file. You can now pick and choose relevant tabs and fields and also chose intervals, number of snapshots for the output. Type Ctrl+S to get the save option
• Load batch output and replay them. Type Ctrl+B to load a saved csv file.
• Line chart for selected performance counters
• Embedded tooltip for counter description
• Color coding for important counters

Running the tool:

1. Run visualEsxtop.sh (Linux) or visualEsxtop.bat (Windows) from the extracted files.  (Note: Type export JAVA_OPTS=-Xmx2048m  if loading large amounts of data)
2. On the VMWARE VisualEsxtop window, select File > Connect to Live server.
3. Choose the IP address of the host or vCenter and the credentials to connect.

Examples of using the tool:

Example 1:

The example below is listing only the devices that have DAVG value of above 20ms. The filter used is DAVG/cmd under Disk World tab.  Typically we do not want DAVG (device driver level  latency) to be over 20ms for lengthy period. Note that by using the filter, it is very easy to list only the devices that currently have high latency values.

Example 2:

The example below is listing the vmkernels and virtual machines that are currently running on vmnic0. The filter used is TEAM-PNIC  under Network tab. You can also sort by %DRPTX , %DRPRX to filter for any devices reporting packet drops. Note that the vmnic number will only show up if the uplink ports are not in an etherchannel binding.

Tips on working with Charts:

• To build a new chart, under Chart tab click twice on Object Types to start to select fields
• To add a field to the chart, expand the related object and click twice on the field
• To remove a field from the current chart, click twice on the field from the bottom left window pane
• The chart allows you to add any fields from any objects at the same time. You have to be the judge regarding what fields are relevant. For example, listing DAVG and Physical CPU Core Util% in the same chart may not provide much value.

Chart view (tab) screenshot:

Where do I get tips on Troubleshooting performance on ESXi ?

Details about different fields in esxtop tool can be found in this communities blog post: Interpreting esxtop Statistics.

There are many great articles and tips on performance troubleshooting in the VMware Knowledgebase. Here are a couple that I recommend to give you a start –

• Troubleshooting ESX/ESXi virtual machine performance issues (2001003)
• Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205)