Cool Apps to play with by VMware Engineers

Came across VMware Labs website today.   A nice website for VMware to show case the quality work that its employees are developing to improve the general vSphere environment.  I have used or looked at 1/2 of these and was pleasantly surprised to discover some new tools.

Per the site manifest:

This is our place to share cool tools created by VMware engineers.  There is a wide range of tools here for you, including one for automating tasks, getting ESX performance graphs, a rich Internet application framework and much more. These tools are offered under Technical Preview or relevant Open Source License.

They are calling each app/tool/API a fling.   This is a pretty smart naming.   I have seen several start up companies employees start making some cool code that never gets to see the light of day.   They are just flings of interest to help a specific problem.   They don’t always become full fledged products.

Here’s the current list:

Apache Pivot

Like most modern development platforms, Pivot provides a comprehensive set of foundation classes that together comprise a “framework”. These classes form the building blocks upon which more complex and sophisticated applications can be built.

Dynamo RIO

DynamoRIO exports an interface for building dynamic tools for a wide variety of uses: program analysis and understanding, profiling, instrumentation, optimization, translation, etc. Unlike many dynamic tool systems, DynamoRIO is not limited to insertion of callouts/trampolines and allows arbitrary modifications to application instructions via a powerful IA-32/AMD64 instruction manipulation library. DynamoRIO provides efficient, transparent, and comprehensive manipulation of unmoOndified applications running on stock operating systems (Windows or Linux) and commodity IA-32 and AMD64.

esxplot

Esxplot is a GUI based tool that lets you explore the data collected by esxtop in batch mode. The program loads files of this data and presents it as a hierarchical tree where the values are selectable in the left panel of the tool, graphs of the selected metrics are plotted in the right panel.

Onyx

Onyx is a standalone application that serves as a proxy between the vSphere Client and the vCenter Server. It monitors the network communication between them and translates it into an executable PowerShell code. Later this code could be modified and saved into a reusable function or script.

SVGA Sonar

VGA Sonar is a demo application for SVGADevTap. SVGADevTap is a user-level library that communicates with the VMware SVGA guest driver to provide low-latency notifications of changes to the screen.

vApprun

The vApprun tool implements the same vApp/OVF feature set as the vSphere 4 release. Thus, Workstation/Fusion can be used as a development environment for advanced OVF packages, and it can be used to evaluate and test OVF packages on your desktops and laptops.

vCMA

VMware vCenter Mobile Access (vCMA) – vCMA allows you to monitor and manage VMware Infrastructure from your mobile phone with an interface that is optimized for such devices.

VGC

VMware Guest Console allows you to manage the Guest OSes from the VMware layer.

VI Java

vSphere Java API is a set of Java libraries that sits on top of existing vSphere SDK Web Services interfaces. It provides full managed object model and run-time type checking, resulting dramatic productivity boost. With the new Web Services engine in 2.0, it also performs much faster than engines like Apache AXIS up to 15 times.

Virtual USB Analyzer

The Virtual USB Analyzer is a free and open source tool for visualizing logs of USB packets, from hardware or software USB sniffer tools. As far as we know, it’s the world’s first tool to provide a graphical visualization along with raw hex dumps and high-level protocol analysis.

If you want to see what is possible with a companies products, these are some of the tools to go look at.  http://labs.vmware.com

90 days to no more ESX 3.5 General Support

For everyone out, ESX 3.5 has 90 more days until General Support ends.

http://www.vmware.com/support/policies/lifecycle/vi/eos.html

VMware Support offline for most of day due to power outage

On the 17th of Feb some form of Power Failure occurred at one of VMware’s Palo Alto locations.   From what I understand this primarily affected the support systems and as such the phones were down for most of the day.

Update 18 Feb 10 @ 8:28pm:

@vmwarecares Network outage here caused by small plane crash in Palo Alto

http://www.cnn.com/2010/US/02/18/texas.plane.crash/index.html?hpt=C1

VMware Support back and up and running by 9pm CST.

Upgrading Power Management Firmware on HP Blades

When you update your Onboard Administrator on HP c-Chassis to 2.5x, the OA will start to check if the Power Management version is new enough and if it is not it will give you a Major Error with details of : C01668472

To update this you need to either have a running OS or you can do the following steps using tools you have already downloaded most likely.

bbFirmware CD 1.60
PowerManagement 3.4D Linux install

1. Boot off bbFirmware CD via the iLO Virtual Media
2. At the boot: prompt type in console and hit enter.  Don’t hit the default.
3. Wait for a console shell prompt to come up.  This may take a couple minutes.
4. Mount a Folder (which will show up as a USB device to the OS) via the iLO to the directory you have put the PowerManagement update in.   It is smart to minimize how many files are in that directory.
5. Type in mkdir /mnt/power
6. Type in mount /dev/sda /mnt/power
7. Type in cd /mnt/power
8. Type in ./CP011627.scexe
9. Once this is complete go ahead and reboot the box via the iLO.

That should address this issue.

Updating Opensolaris to latest development build

To be able to update from a post Opensolaris 2008.11 release to the latest development build use the following steps:

# pkg set-authority -O http://dev.opensolaris.org/dev development
# pkg refresh
# pkg image-update -v

I have finally found the correct website for the most up to date instructions:

http://pkg.opensolaris.org/dev/en/index.shtml

Cloud Computing Solution Provider – VMware

The recent Zimbra acquisition by VMware threw me a bit for a loop initially.  Then I started chewing on it and read the good post by Rodney Haywood.   Very shortly afterwords I had a classic Homer Duh moment.

VMware aims to build from the ground up the best cloud computing solution for sale as possible.   That is taking into account that cloud computing definition today is as about as vague as a real cloud in the sky.  Today that cloud is fluffy and in 5 mins that cloud is shaped like a rabbit.   As such they have built a pretty strong infrastructure level for customers with vSphere, vCenter and various add-on tools.   They have picked up SpringSource to offer ultimately a platform for services and understanding of how the JVM interacts more closely with the hypervisor.   Now they are getting into the services space with Email/Calendaring.

• IAAS -> vSphere/vCenter
• PAAS -> SpringSource
• SAAS -> Zimbra

Each of these areas is really focused on a different customer base at the end of the day.  Sure you can say IT and that’s like saying your customer base for is for the TV viewing audience.    It is too vague and there is better & a more definable end customer grouping.

• IAAS -> Server/Storage/PC/Hardware Teams – Ground Level System Admins
• PAAS -> Development Teams making solutions up – Architects/Developers
• SAAS -> Back Office management/utilities – Often more visible by the CxOs.

So where are they going next and what areas are missing for the full suite for all the different customer bases they are aiming for?

Ideal Software Licensing Model – Requirements Collection

I’m looking for some feedback and thoughts from the community to help define a reasonable Licensing Model that takes Physical & Virtual into account.  From my view as a client I don’t think this is all that complicated at the end of the day.

More discussions with some vendors around licensing and I’m finding more and more that the following two axioms are defining these discussions:

• Vendors want to get paid for their software (obviously the most they can be).   They are not stupid in most cases.
• Clients want to pay for what they use (obviously the least they have to).   They are not stupid in most cases.

The challenges come from the fact that Vendors don’t get the following generally:

• A VM in VMware is limited in processing to the vCPUs it has.
• A vCPU is limited to what a given core is individually capable of.
• More clients might be willing to use your software if I didn’t need to pay for 12 cores of power when I only need 2 today.
• VMotion of a VM does not mean I’m suddenly gaining more cores of processing.

Clients get upset cause of the following items:

• When a Vendor assumes I’m an idiot and can pull the wool over my eyes.   This a good relationship does not make.
• A Vendor goes and says a Virtual does less than a physical, then charges me more if it is virtual.
• A Vendor requires me to license this big physical box and I only want a couple cores worth or less than # of cores in physical box.
• I want to use your software and because I’m running it as a virtual you want to charge me more.  I can’t even buy smaller physicals to use your software within my software budget (smallest thing I can buy within reason today is an 8 core system and I only need 2 cores worth).
• A Vendor limits me to some physical box even though the OS/Software will be on a virtual machine.   (Who cares what physical box is on it as long as I pay for the CPU MHz I’m using?  Your software doesn’t.  Only your legal does.)
• If I buy a lot of your software you can cut me deals since I’m spending a lot of money with you and then I’ll be interested in licensing models by physical cores or just a volume level discount.  I’d rather not start there if I can avoid it.
• We’ve seen what happens to good tech when licensing models can’t take tech into account.   See the Mainframe and Computer Associates licensing stubbornness in the 80s contribute significantly to the rise of the distributed computing space.  We don’t want to deal with that migration if we can avoid it.

So there’s some of the requirements I have come up with.   What other requirements/gotchas can you think of that have got you in dealing with vendors?   Anything different when dealing with Solaris or AIX or HP/UX virtualization?

Limits have their limits

I’ve been chewing on this post by Duncan at Yellow Bricks for the past month and a half.  It covers some complicated issues that one has to deal with in a enterprise size environment with many assumptions on what gets you into this mess in the first place.   The best thing to do is downscale and upscale as needed based on good performance monitoring and bottleneck research.  Thankfully I’ve managed to make good relationships with most teams where I work that this has become the standard operating procedure though sometimes we just can’t.   At the end of the day the issue boils down to the simple goal:

“As the VMware environment administrator, how can I make better use of what I have available to me?”

For my environment I run into a variety of political reasons going from..

• “I am going to need that extra 2 CPUs someday in the future so I can’t give them up now.”
• “The vendor docs say I really do need 8 CPUs and 128G of RAM for my 3 users even though 126G is unused.”
• “Someone on your team said I really do need that 8G of RAM so I won’t give it up”
• “Oh come on.. what’s another 2G of RAM”
• “I gave up my budget for a physical to do this as a virtual even though I’m still spending less in the grand scheme.  Gimme more resources.”

to the begging

• “Pleaseeee.  I think it’ll help my issues.  It might even make me look better to my co-workers.”

I have two distinct use cases that really showcase that this kind of capability can be a hard item to use.

Case #1:  The poorly written VBscript

Back in the early Windows 3.1 days when VB was a novel concept, some developers made this ground breaking app that would pull data from a remote system, massage the data a bit and put it into a centralized Btrieve database.   Well this script that they wrote goes to sleep for a minute after the remote system’s queue it checks is empty.  This script sleep function checks the clock to see if a minute has passed.  It constantly checks the clock which consumes 100% of the CPU all the time.   This wasn’t much of an issue when each one of these systems was on its own old PC system.  We virtualized them since 16 XP workstations in the datacenter is a management headache.   Now that’s 16 high power, multiple generation newer cores being used 100% all day long for no good reason.

We, VMware Admins, have discovered that on the old PCs these systems would easily take 5-10 mins to work through their work queues.   On the newest hardware we have with these as VMs, it takes under 15 seconds to do the same work.   So for 60 seconds it is doing nothing except checking the hardware clock.

Solution #1:  CPU limits good

We implemented a CPU limiting resource pool for these VBscript VMs.   They are still running mega fast in comparison to where they were a year ago.   Now they are using no more than 8 cores worth at any given time.  A big improvement until the app developers decide if they are going to replace all that code with sleep 60 or recode the entire app.

Case #2:  vCenter SQL Server Memory Limits

Due to a feature in vCenter 4.0U1 and ESX 3.5 Hosts, when I increased the RAM on my vCenter dedicated SQL Server from 4G to 8G, a Memory limit was set of 4G.   When I would go onto the SQL instance, SQL Server.exe would only be using about 3600 Megs yet all 8G was consumed/used.   This screamed to me an issue with the OS instance.   After close to 10 days of head beating and not understanding why my brand new vCenter 4.0U1 system was running so poorly, a co-worker with a fresh set of eyes noticed this setting on the SQL Server instance.

Solution #2:  Memory limits bad

This is obvious.  We disabled the limit and the SQL Server performance went through the roof instantly. We simply couldn’t tell easily that the driver was using 4G of RAM as it wasn’t a process.  Nobody noticed the ballooning happening.

At the end of the day there’s pros and cons to having this level of capabilities.  This is why I like ESX and the general approach of VMware.   Give you everything we can in terms of options, configurations and rope to hang yourself and two of your friends.   We will attempt to automate this and hide this as much as we can.   The Vendor will never know all the situations we, people in the field, are going to run into so let’s give us all the options they can.  Use that rope with caution.

New vmware.com HomePage Layout & View 4 is released

New vmware.com HomePage is now live.   I had an “anonymous internet tipster” give the heads up last night.   Looks good and a bit more sleek in fitting with the branding of the new vmware logo.

Along with that View 4 is finally released.   I’ve been playing with some beta bits for a while now and the PCoIP is pretty impressive catchup with ICA protocol.   I’m looking forward to the mass quantity of comparisons that are going to come out now between ICA & PCoIP.

Running 20008 Active Directory with Bind

One of the fun things about running stuff at home, once you have something working remotely well you sure don’t want to change it.   As such I have setup Bind 9 with DHCP and it works pretty good with a bunch of scripts I have to handle various dynamic adding and removing from my running system.  Every system in this house works off of this base, very stable system.  This works pretty well and there in comes the issue.     I am attempting to bring up an Active Directory Domain for testing a variety of products and as is typical MS, as long as you give all your money to MS, everything works great.   The minute you step out of that paradigm the documentation and functionality tends to falter a bit.

In order to setup Windows 2008 Active Directory domain, the system basically expects you to use MS DNS/DHCP services.   I have no desire to tear down a perfectly functional environment and make it work now with MS DNS.   After a significant amount of digging I found some good webpages on how to setup Bind to work with AD.   None of them worked right though.

• Bind and Active Directory – didn’t work.   I simply couldn’t get the Windows box to talk to bind for some reason
• Bind DNS and Active Directory R2 – I kept getting these errors.   No luck down this path.
• DNS Bind Delegation – This isn’t ideal as now I have to change a significant amount of configuration for DNS searching to work right.  Not the route I want to go down.

So plan D at this point was reading some of the docs on Active Directory and DNS on technet and then turning up named logging and watching for what the future DC was asking for.   Create the domain without passing the network tests and then just run dcdiag /test:DNS a couple dozen times added in each entry as you need to.  Add this into /etc/named.conf and then run service named restart.

logging {

category “default” { “debug”; };
category “general” { “debug”; };
category “database” { “debug”; };
category “security” { “debug”; };
category “config” { “debug”; };
category “resolver” { “debug”; };
category “xfer-in” { “debug”; };
category “xfer-out” { “debug”; };
category “notify” { “debug”; };
category “client” { “debug”; };
category “unmatched” { “debug”; };
category “network” { “debug”; };
category “update” { “debug”; };
category “queries” { “debug”; };
category “dispatch” { “debug”; };
category “dnssec” { “debug”; };
category “lame-servers” { “debug”; };
channel “debug” {
file “/tmp/namedebug” versions 2 size 50m;
print-time yes;
print-category yes;
};

After watching this log file and trying to promote a machine to a DC a couple dozen times and testing this, I found the following DNS entries in your zone are needed.

If my DC is going to have a DNS name of DC.home.here.org with an IP of 192.168.1.4 and the domain is HOME, then these are the entries needed.

home.here.org     A      192.168.1.4
DC.home.here.org      A      192.168.1.4
UID #1 – the subkey in HKLM\Software\Microsoft\Cryptography\AutoEnrollment\AEDirectoryCache
UID #2 – I don’t know where this comes from.  I think it is something Domain related.  It isn’t in the registry.
(These UID based DNS entries might not be needed – not sure)

$ORIGIN _msdcs.home.here.org.
(UID #1 – might not be needed)  CNAME DC.home.here.org.

$ORIGIN _tcp.Default-First-Site-Name._sites.dc._msdcs.home.here.org.
_kerberos       SRV 0 0 88 DC.home.here.org.
_ldap           SRV 0 0 389 DC.home.here.org.

$ORIGIN _tcp.dc._msdcs.home.here.org.
_kerberos       SRV 0 0 88 DC.home.here.org.
_ldap           SRV 0 0 389 DC.home.here.org.

$ORIGIN _msdcs.home.here.org.
_ldap._tcp.(UID#2 – might not be needed).domains SRV 0 0 389 DC.home.here.org.
gc          A   192.168.1.4

$ORIGIN gc._msdcs.home.here.org.
_ldap._tcp.Default-First-Site-Name._sites SRV 0 0 389 DC.home.here.org.
_ldap._tcp      SRV 0 0 389 DC.home.here.org.

$ORIGIN _msdcs.home.here.org.
_ldap._tcp.pdc      SRV 0 0 389 DC.home.here.org.

$ORIGIN _tcp.Default-First-Site-Name._sites.home.here.org.
_gc         SRV 0 0 3268 DC.home.here.org.
_kerberos       SRV 0 0 88 DC.home.here.org.
_ldap           SRV 0 0 389 DC.home.here.org.

$ORIGIN _tcp.home.here.org.
_kerberos       SRV 0 0 88 DC.home.here.org.
_kpasswd        SRV 0 0 464 DC.home.here.org.
_ldap           SRV 0 0 389 DC.home.here.org.

$ORIGIN home.here.org.
_kerberos._udp      SRV 0 0 88 DC
DC         A   192.168.1.4

Enter all these and then try creating your domain again.   This got me up and running.   It would be nice in the spirit of openness to give me an option from dcdiag to dump all the DNS entries the system is looking for and testing.

• What do you see?

  appspeed bailout Changes CIFS citrix cloud datacenter DirecTV dtrace ESX fishworks Get-View Get-VM hardware hp hyper-v ilo joke keynote licensing LiveMigration news nfs NFSv4 OpenSolaris performance Powershell root server Server Virtualization silly sun Tivo vdi VI view virtual iron Virtualization VI Toolkit VMotion VMware vmworld VMworld2008 vSphere waste

• Recent Posts

  • Cool Apps to play with by VMware Engineers
  • 90 days to no more ESX 3.5 General Support
  • VMware Support offline for most of day due to power outage
  • Upgrading Power Management Firmware on HP Blades
  • Updating Opensolaris to latest development build

• Recent Comments

  • DoubleCloud » Vote for vSphere Java API at VMware Labs on Cool Apps to play with by VMware Engineers
  • VMware ’s next steps towards the cloud « UP2V on TA3461 – IO DRS: Tech Preview for VM Performance Isolation
  • VMware PEX 2010: Tech Preview of DRS I/O Resource Shares | VM /ETC on TA3461 – IO DRS: Tech Preview for VM Performance Isolation
  • VMware PEX 2010: a short comment on DRS IO « VirtuallyNil on TA3461 – IO DRS: Tech Preview for VM Performance Isolation
  • Rodos on Cloud Computing Solution Provider – VMware

• Virtualization

  • VMTN Blog

• Administrata

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org