IE8 based Painting Issues

I then launched vSphere client only to be unable to see any of the objects in the left hand window in the client.

vSphere Painting Issues

This VM is running on ESX 3.5U4 on VM Hardware 4.   The quick fix is to upgrade to VM Hardware 7 which entails all the updates to vSphere 4 & VM tools updates.

The other fix is due to the following two bugs in Hardware version 4.

1. Completely uninstall the View Agent
2. Reboot
3. Reinstall the View Agent (Make sure that the Video Driver version is [...].0032)

If this doesn’t do it then you are probably having an issue with VRAM.   The fix is to adjust the pool inside of View for this machine and set the resolution and # of monitors so they come out to a number divisible by 64.  (Kudos to my Support Wizard for finding this one.)

The magic formula is

((#of monitors * Width of Resolution) * (# of monitors * Height of Resolution) * 4 )/1024 == Multiple of 64

Keep in mind if you have less monitors than you set the pool to, PCoIP handles this gracefully and it doesn’t cause issues.

Our conversation usually goes something like this:

The hot new badness just came out and we need to order more hardware.

Awesome.   So how much does this puppy have in it?  RAM?  CPUs?   Slots for HBAs & NICs?

Did you know the new motherboard comes with 4 NICs now so our standard config can go from 4U to 2U and gobs of RAM with 6 core CPUs now.

Awesome!   *pause*  You know with that much RAM I can put 100 Win7 VDI systems on there.   Umm.. What about when it goes down?

Oh.. Hrm.   That wouldn’t be so good.   ….

That being said we generally end up breaking it down to a couple of factors.

1. What is the current capacity configuration we run with today?
2. What is our current pain points in CPU, Memory, Network or Storage?
3. Is there any new architecture changes coming that will impact this design?   Is there a new switch fabric that needs to be plugged into?   Is there changes to storage that need to be addressed?
4. How much does this new hardware configuration cost?
5. How will this change affect DRS’s Chaos Theory?  The more hosts, the more DRS can do for you due to Chaos theory.
6. What is our Risk level for number of eggs in a single basket?

The point is most corporation’s environments aren’t starting from scratch.  In my case we have a known configuration today to use as a baseline and adjust the environment and design every hardware order to make it better.

In our most recent order we had this discussion all over again.  This time we had some architectural changes needed to prevent some false positive HA events from happening in a 2 time a year strange events.   So we are going to a 3 switch connectivity solution to enable network beacon for NIC teamed connections.  We started with the following information:

• Baseline:  HP DL585 G5, 4 sockets w/ quad cores, 128G of RAM, 3 Dual 1G NICs, 2 Emulex LPe11000 HBAs
• Cluster: 10 Host Clusters with ~30 per Host in Servers and ~65 Workstations per Host in View
• Pain Points:  CPU starvation, Licensing Issues with 10 Host sized clusters
• Risk Level:  Politically we are getting pretty touchy about more than 30 Servers going down in a single blow even if HA works on bringing them up in under 15 mins automatically.

We compared 3 different models of newer, faster, badder and more wicked hardware from HP since the DL585 G5s are not really on the manufacturing line anymore.   So we looked at the BL495cG6, DL585 G6, DL385 G6 and DL580 G6.

DL585 G6:

• Pros
  • Proven and comfortable AMD based stable platform with a good price/performance cost.
  • Gain more CPU resources with the additional 2 cores per socket.  6 core systems.
  • Can build 5 Host clusters to address licensing issues.   Issues with HA support for the density involved.
• Cons
  • Same Risk Level as before.

  Push

  • Same architectural solution today with maybe another NIC card to enable the NIC Beaconing

DL580 G5:

• Pros
  • Fastest individual cores out there.   Lots of good press about the Intel.
  • Should get better CPU resources with higher performing CPUs.
  • Can build 5 Host clusters to address licensing issues.   Issues with HA support for the density involved.
• Cons
  • Significant premium in cost for speed.   See easily a 25% premium for a 10% faster performance.
  • Same Risk Level as before.
• Push
  • Same architectural solution today with maybe another NIC card to enable the NIC Beaconing.

DL385 G6:

• Pros
  • Lowers the risk level without lowering performance
  • Best price/performance cost for 6 core systems
  • Has enough slots to move to the newer network layout to enable NIC Beaconing
  • Gain more CPU resources with the additional 2 cores per socket.  6 core systems.
  • Put 64G of RAM into them and build 5 host clusters for licensing problematic applications.
• Cons
  • More physical hosts to deal with (cabling, power, rack space, cooling, management)

BL495c G6:

• Pros
  • Blades reduce the amount of cabling
  • Gain more CPU resources with the additional 2 cores per socket.  6 core systems
• Cons
  • Firmware Management is an issue
  • Increases our Risk Level with more eggs in the same basket unless we get multiple chassis to spread the blades across
  • New solution from the ground up running ESX on blades
  • Not ready to support Flex10 and because of this we have limited NIC capabilities to fit our requirements

We decided to go with the DL385 G6s based on these criteria.  We will dedicate a specific 5 Host cluster for problem children applications with licensing issues.   The RAM size of the hosts will limit the number of VMs we can end up putting in a cluster which addresses the Risk Level of number of VMs per Host.  We are still way ahead of the game using VMware so having to have a couple more physicals for all these improvements is not an issue.

In your company or solution something else may be more appropriate.  The key in an ongoing improvement mentality is have things you can measure and then criteria on what to change along with why.   There is no one size fits all answer which is why VMware works so well for so many different folks.   We don’t have to change how we do things to gain a lot of flexibility in the Datacenter while not changing how we ultimately end up managing these systems.

Per the site manifest:

This is our place to share cool tools created by VMware engineers.  There is a wide range of tools here for you, including one for automating tasks, getting ESX performance graphs, a rich Internet application framework and much more. These tools are offered under Technical Preview or relevant Open Source License.

They are calling each app/tool/API a fling.   This is a pretty smart naming.   I have seen several start up companies employees start making some cool code that never gets to see the light of day.   They are just flings of interest to help a specific problem.   They don’t always become full fledged products.

Here’s the current list:

Apache Pivot

Like most modern development platforms, Pivot provides a comprehensive set of foundation classes that together comprise a “framework”. These classes form the building blocks upon which more complex and sophisticated applications can be built.

Dynamo RIO

DynamoRIO exports an interface for building dynamic tools for a wide variety of uses: program analysis and understanding, profiling, instrumentation, optimization, translation, etc. Unlike many dynamic tool systems, DynamoRIO is not limited to insertion of callouts/trampolines and allows arbitrary modifications to application instructions via a powerful IA-32/AMD64 instruction manipulation library. DynamoRIO provides efficient, transparent, and comprehensive manipulation of unmoOndified applications running on stock operating systems (Windows or Linux) and commodity IA-32 and AMD64.

esxplot

Esxplot is a GUI based tool that lets you explore the data collected by esxtop in batch mode. The program loads files of this data and presents it as a hierarchical tree where the values are selectable in the left panel of the tool, graphs of the selected metrics are plotted in the right panel.

Onyx

Onyx is a standalone application that serves as a proxy between the vSphere Client and the vCenter Server. It monitors the network communication between them and translates it into an executable PowerShell code. Later this code could be modified and saved into a reusable function or script.

SVGA Sonar

VGA Sonar is a demo application for SVGADevTap. SVGADevTap is a user-level library that communicates with the VMware SVGA guest driver to provide low-latency notifications of changes to the screen.

vApprun

The vApprun tool implements the same vApp/OVF feature set as the vSphere 4 release. Thus, Workstation/Fusion can be used as a development environment for advanced OVF packages, and it can be used to evaluate and test OVF packages on your desktops and laptops.

vCMA

VMware vCenter Mobile Access (vCMA) – vCMA allows you to monitor and manage VMware Infrastructure from your mobile phone with an interface that is optimized for such devices.

VGC

VMware Guest Console allows you to manage the Guest OSes from the VMware layer.

VI Java

vSphere Java API is a set of Java libraries that sits on top of existing vSphere SDK Web Services interfaces. It provides full managed object model and run-time type checking, resulting dramatic productivity boost. With the new Web Services engine in 2.0, it also performs much faster than engines like Apache AXIS up to 15 times.

Virtual USB Analyzer

The Virtual USB Analyzer is a free and open source tool for visualizing logs of USB packets, from hardware or software USB sniffer tools. As far as we know, it’s the world’s first tool to provide a graphical visualization along with raw hex dumps and high-level protocol analysis.

If you want to see what is possible with a companies products, these are some of the tools to go look at.  http://labs.vmware.com

Update 18 Feb 10 @ 8:28pm:

@vmwarecares Network outage here caused by small plane crash in Palo Alto

http://www.cnn.com/2010/US/02/18/texas.plane.crash/index.html?hpt=C1

VMware Support back and up and running by 9pm CST.

VMware aims to build from the ground up the best cloud computing solution for sale as possible.   That is taking into account that cloud computing definition today is as about as vague as a real cloud in the sky.  Today that cloud is fluffy and in 5 mins that cloud is shaped like a rabbit.   As such they have built a pretty strong infrastructure level for customers with vSphere, vCenter and various add-on tools.   They have picked up SpringSource to offer ultimately a platform for services and understanding of how the JVM interacts more closely with the hypervisor.   Now they are getting into the services space with Email/Calendaring.

• IAAS -> vSphere/vCenter
• PAAS -> SpringSource
• SAAS -> Zimbra

Each of these areas is really focused on a different customer base at the end of the day.  Sure you can say IT and that’s like saying your customer base for is for the TV viewing audience.    It is too vague and there is better & a more definable end customer grouping.

• IAAS -> Server/Storage/PC/Hardware Teams – Ground Level System Admins
• PAAS -> Development Teams making solutions up – Architects/Developers
• SAAS -> Back Office management/utilities – Often more visible by the CxOs.

So where are they going next and what areas are missing for the full suite for all the different customer bases they are aiming for?

“As the VMware environment administrator, how can I make better use of what I have available to me?”

For my environment I run into a variety of political reasons going from..

• “I am going to need that extra 2 CPUs someday in the future so I can’t give them up now.”
• “The vendor docs say I really do need 8 CPUs and 128G of RAM for my 3 users even though 126G is unused.”
• “Someone on your team said I really do need that 8G of RAM so I won’t give it up”
• “Oh come on.. what’s another 2G of RAM”
• “I gave up my budget for a physical to do this as a virtual even though I’m still spending less in the grand scheme.  Gimme more resources.”

to the begging

• “Pleaseeee.  I think it’ll help my issues.  It might even make me look better to my co-workers.”

I have two distinct use cases that really showcase that this kind of capability can be a hard item to use.

Case #1:  The poorly written VBscript

Back in the early Windows 3.1 days when VB was a novel concept, some developers made this ground breaking app that would pull data from a remote system, massage the data a bit and put it into a centralized Btrieve database.   Well this script that they wrote goes to sleep for a minute after the remote system’s queue it checks is empty.  This script sleep function checks the clock to see if a minute has passed.  It constantly checks the clock which consumes 100% of the CPU all the time.   This wasn’t much of an issue when each one of these systems was on its own old PC system.  We virtualized them since 16 XP workstations in the datacenter is a management headache.   Now that’s 16 high power, multiple generation newer cores being used 100% all day long for no good reason.

We, VMware Admins, have discovered that on the old PCs these systems would easily take 5-10 mins to work through their work queues.   On the newest hardware we have with these as VMs, it takes under 15 seconds to do the same work.   So for 60 seconds it is doing nothing except checking the hardware clock.

Solution #1:  CPU limits good

We implemented a CPU limiting resource pool for these VBscript VMs.   They are still running mega fast in comparison to where they were a year ago.   Now they are using no more than 8 cores worth at any given time.  A big improvement until the app developers decide if they are going to replace all that code with sleep 60 or recode the entire app.

Case #2:  vCenter SQL Server Memory Limits

Due to a feature in vCenter 4.0U1 and ESX 3.5 Hosts, when I increased the RAM on my vCenter dedicated SQL Server from 4G to 8G, a Memory limit was set of 4G.   When I would go onto the SQL instance, SQL Server.exe would only be using about 3600 Megs yet all 8G was consumed/used.   This screamed to me an issue with the OS instance.   After close to 10 days of head beating and not understanding why my brand new vCenter 4.0U1 system was running so poorly, a co-worker with a fresh set of eyes noticed this setting on the SQL Server instance.

Solution #2:  Memory limits bad

This is obvious.  We disabled the limit and the SQL Server performance went through the roof instantly. We simply couldn’t tell easily that the driver was using 4G of RAM as it wasn’t a process.  Nobody noticed the ballooning happening.

At the end of the day there’s pros and cons to having this level of capabilities.  This is why I like ESX and the general approach of VMware.   Give you everything we can in terms of options, configurations and rope to hang yourself and two of your friends.   We will attempt to automate this and hide this as much as we can.   The Vendor will never know all the situations we, people in the field, are going to run into so let’s give us all the options they can.  Use that rope with caution.

Along with that View 4 is finally released.   I’ve been playing with some beta bits for a while now and the PCoIP is pretty impressive catchup with ICA protocol.   I’m looking forward to the mass quantity of comparisons that are going to come out now between ICA & PCoIP.

I am connecting from a laptop via View client to a Virtual Workstation running XP that then I launch VI Client on it and go to the console of my Virtual Workstation.  

I went to the IO DRS Tech Preview and got the same excitement I’ve had from previous years where you know your seeing something innovative.  Several of the other sessions I hit were really partner style presentations that did not say much.   So a good 25/75 day for sessions which is pretty good.  

Now that the Self Service Labs were finally working properly I gave a shot at the vCenter Orchestrator product offering.  The Lab was responsive and well documented.  It was pretty nice and really hinted at the power this system can offer for DataCenter Automation.  The theory is this is free with vSphere 4 so I’m going to have to really look into that and find out.

During my open times during the day I had some good meetings with some VMware employees to discuss some of the vStorage & vCloud directions, HP folks around OpenView and Virtualization tools, AMD & Intel on their functionality futures and  Hitachi around their multipathing technology for VMware (still no roadmap).    

The Party was fun.   Foreigner still knows how to rock and I can actually climb rock walls.   The nice thing about the party this year is it was right at the Moscone Center.

A very productive and long day.

To start off RIM & VMware have been working together for 2 years and it is officially supported on VMware.   Together RIM & VMware have done many numerous and successful engagements running BES on VMware.  The interesting thing is RIM runs their own BES on VMware for over 3 years now. 

Today BES best practice is no more than 1k users per server and they are not very multi-core friendly.   It is not cluster aware or have any HA built in.   The new 5.0 version of BES is coming with some HA availability via replication at the application layer.   One thing that has been seen in various engagements is if you put the BES servers on the same VMware Hosts as virtualized Exchange, there are noticable performance improvements. 

The support options for BES do clearly state that they support on VMware ESX.  

One of the big reasons to virtualize BES is that since it can not use multi-cores effectively the big 32 core boxes today are only able to use a fraction.  By virtualizing BES can get significant consolidation.   Then when doing the virtualization BES gets all the advantages of running virtual such as Test/Dev deployments and server consolidation and HA etc.   Things that are well known and talked about already.  

BES encourages template use to do rapid deployments.   The gotcha is just what your company policies and rules are and can potentially save quite a bit of time.   This presentation is really trying to show how to use VMware/Virtualization with BES for change management improvements, server maintenance, HA, component failures and other base vSphere technologies.   VMware is looking towards using Fault Tolerance for their own BES servers. 

BES is often not considered Tier 1 for DR events.   Even though email is often the biggest thing needed to start working after a DR event to start communications.   The reason is generally been seen due to the complexity and cost of DR. 

The performance testing with the Alliance Team from VMware has been successfully done numerous times for the past couple of years.   They have done testing at both RIM & VMware offices.   The main goal of these efforts was to generate white papers and a reference architectures that are known to work.   The testing was to use Exchange LoadGen & PERK load driver (BES testing driver).  Part of this is how to scale outwith more VMs  as the scale up is known. 

The hardware was 8 cpus, Intel E5450 3Ghz, 16 G RAM and FAS3020 Netapp on vSphere 4 & BES 4.1.6.  The 2k user test with 2 Exchange systems the results were 23% CPU utilization on 2 vCPU BES VMs.   Latency numbers was under 10 ms.   Nothing majorly wrong seen in the testing metrics.   Going from ESX 3.5 to vSphere 4 was a 10-15% CPU reduction in the same workload tests.   Adding in Hardware Assist for Memory saw what looks like another 3-5% reducting in CPU usage.   In their high load testing when doing VMotion there is a small hiccup of about 10% increase in CPU utilization during the cut over period of the VMotion.   This is well within the capacity available on the host and in the Guest OS. 

Their recommendation is to do no more than 2k users on a 2vCPU VM.  If you need more then add more VMs.   Scales and performs well in this scale out architecture.   Be sure you give the storage the number of spindles needed.   The standard statement when talking about virtualization management.  

 The presenter then went into a couple of reference architecture designs.  Small Business & Enterprise with a couple different varieties. 

BES @ VMware.   3 physical locations, 6,500 Exchange users.   1k of them have 5G mailboxes and the default for the rest are 2G.   BES has become pretty common.   They run Exchange 2007 & Windows 2003 for AD & the Guest OS.   Looks fairly straight forward. 

4 prod BES VMS, 1 STandby BES VM, 1 Attachment BES VM and 1 BES dedicated Database VM.   Done on 7 physical servers and 40 additional VM workloads on this cluster.