It’s Just Another Layer

For a large organization that has more than 8 VMware Hosts (nodes) one should only make 8 node clusters for the time being according to the vSphere 4 Config Max doc.

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

If you look on page 7 it says “Configurations exceeding 40 VMs per host are limited to cluster size no greater than 8 nodes“.  

What that means is to be supported you have the following chart:

8 Nodes  x 100 VMs / Node  == 800 VMs per cluster max

9 Nodes x 40 VMs / Node     == 360 VMs per cluster max

20 Nodes x 40 VMs/ Node    == 800 VMs per cluster max

32 Nodes x 40 VMs / Node   == 1,280 VMs per clustre max

Now I might not be too smart here some days though I think the math is pretty obvious.  Either I do 4 clusters of 8 nodes to give me 3,200 VMs on 32 boxes or I stick them all into one nice big cluster and get 1,280 VMs out of those 32 boxes.   Easy math to me.  

The question I have out of this is, why do you document it this way?  That sure is confusing and I could have easily missed that.

vSphere 4 is now GA and available for download if you are evaluating it.   However if you have Support & Service contract for in-line free upgrade of your licenses, your out of luck (or at least this morning when I tried 3 different ways).

If I went in and said I wanted to evaluate it, then I could download everything just fine.

If I get an email saying vSphere 4 is ready for download I should be able to download it.  Am I asking too much?

Just recently I had an interesting conversation with one of my clients.  In this internal consult with her, she comes to me and says of our VMware Solutions that there is a

“potential issue with 4 vCPUs and scheduling”

I looked at her and was a bit confused.   I told her

“No.. there isn’t a scheduling issue for 4 vCPUs in our environment.”

The response was

“Well in our last consult effort, the solution was changing the system down to 2 vCPUs instead of using 4 vCPUs.”

Ah ha.   I understand now.  The light bulb suddenly came on.

One of the big benefits in Server Virtualization is the idea that we can make some rather “drastic” (by legacy mentality) changes to a Machine/OS instance that you couldn’t or didn’t want to do in a physical system.

1. Ask yourself, When was the last time I went and pulled out Processors from a physical system in load testing?   I’ll bet you a beer at VMworld that you haven’t done that in years if ever.
2. Now when’s the last time you changed a Virtual Machine from a 4 vCPU to a 2 vCPU system to try to improve performance of a given OS instance.   I’ll bet a beer again that you have done this in the last 60 days.

It is so easy for us to make those changes we don’t think of the appearance of how it looks to our end clients.   As in my recent discussion, they assumed that this was due to VMware ESX not being able to schedule multiple CPUs well.  This may have been due to poor wording on my part or a leap of logic by my internal clients.   In reality it is more likely, after I’ve gone back and started digging into some of the metrics collected at that time, that the application itself doesn’t work as efficiently with 4 vCPUs as it does with 2 vCPUs.

Also it is very easy and a lazy / simple approach for us as IT folks to just assume that since I’ve got two sockets quad core in a physical that the load testing that we’ve done is going to be the fastest application performance we can get.   The reality of the situation is that unless we do some tests covering changing the hardware around we really have no idea that if I got a single core, single socket my application may run (WAY) faster.   We just don’t know.

If your serious about IT and doing things Right and Correct for your company and yourself, you have to be honest with what your looking at (metrics and performance) and what it really means to yourself (flexibility) and to your clients (appearance).   I know I’ve had a fantastic ephinany and that will help me be a better IT person now.

It’s official. There’s about a zillion blog postings and news articles coming out about the next generation of ESX.

In watching the press conference yesterday the one thing that really hit me is that this is the next game changer. Cloud computing has been stuck for years in lock-in approaches. Amazon EC2, Google App Engine, Microsoft Azure are development environments that you really need to develop to for your apps to work. The designs and setup have to be extremely customized to function. This is the main reason cloud computing hasn’t taken off full scale. No cross vendor solutions. No ability to take my application out of the box deploy to the cloud system and it just works. No way for me to honestly develop in house and then move it easily to the cloud.

vSphere 4 (or Cloud Infrastructure) is the first Cloud Computing solution that I have seen that doesn’t lock you into a specific vendor. I can run Microsoft Azure on top of vSphere 4 (woah!). It can run on my infrastructure of commodity parts (yeah.. that white box I have can be a cloud computing solution). I can set it up, test it, run it in my basement, then deploy it up to a cloud provider with more bang and power than I have.  I can develop on linux, suse, freebsd, windows, or solaris and have the whole thing packaged up as a deliverable tool. No middleman. This is a powerful concept. This is the game changer.

The reason Microsoft OSes took off in the early ’90s is they were simply the easiest and most accessible development environment out there.  Today Linux/Java/Web is taking most of that development energy by storm.  It costs me nothing to develop solutions on those products.   If I can setup a development environment of my own without having to pay some thousands of dollars just to get started with the tools, I can make the next facebook/twitter/ebay.  I don’t have to be a corporation to develop a solution.

VMware gets that and Paul Maritz was a key component of that understanding at Microsoft. Welcome to the next great thing.

Good friend of mine calls me and hollers on 1 Apr that his websites aren’t working.  I look and find out that the DNS servers at register.com aren’t responding.   After a call to Register.com they say “not our issue and is something happening to the net”.

Guess it is their issue.

My friend can’t wait for this to end to move his stuff to a different Registrar.

Sun to be bought by IBM for $7 billion. SGI bought by Rackable for $25 million.

The SGI purchase I can understand as Rackable wants to be a one stop shop for their customers.  SGI has been on a downward spiral for years.   Who needs to buy a super expensive custom OS/Hardware to do graphics processing or HPC when I can get commodity hardware and do just as well if not better?

The Sun purchase doesn’t make much sense to me.   IBM has no need/want for a new OS and it sure can’t be for the sales force/contacts that Sun has.   I can think two things that it “might” want.   One is to expand their server hardware line to AMD though why?   IBM is well known for their “better than Intel” line of Intel x86 chips and Sun is of the same quality generation in AMD terms.   The other thing I can think of is that IBM wants to take full control of the Java Stack.  They have a heavy investment in WebSphere and is one of their more profitable product lines so why not take control all the way down if you can?

My good friend has found another VI Toolkit call that is amazingly slow compared to pulling useful data from it. I’ve done performance testing before of Get-VM against Get-View to get to the useful data needed day to day.   In that test using Get-View was 18x faster than using Get-VM.

This past week he was able to dig through the API/mob to find disk data for individual VMs for an internal tracking and reporting system we have.   This time he found a way to get the majority of useful data from Get-Harddisk a different way.   Rather significant change in speed with another couple hours of work.

This report he generates goes through about 1/2 of our VMs to pull VMDK information.

Get-HardDisk => 35 hours

New method => 2 mins 15 seconds

Yes.  That’s right.   976 times faster.   Do the math yourself.

Microsoft has renamed the Parent Partition in Hyper-V.  It is now called the Management Operating System.

Nice that Microsoft folks are starting to admit in passing that their is a heavy dependency on this MOS.  Giving it a real name is a starting point.

Recently at the request of our vendor (HP), I went and had HP Systems Insight Manager updated to 5.3 about 3 weeks.  Now this isn’t “bleeding edge” and so I didn’t feel it was a big deal to jump up to it already.   Well over the past 3 weeks I’ve been getting this new alert coming from our SIM system and I just figured “Oh.. something new” and didn’t fret about it too much.

Event Name: (SNMP) Remote Insight/ Integrated LightsOut Interface Error (9006)

URL: https://HPSIM:2381/

Event originator: HPSIM

Event Severity: Major

Event received: 12-Mar-2009, 17:31:02

Event description: Remote Insight/ Integrated Lights-Out Interface Error. The host OS has detected an error in the Remote Insight/ Integrated Lights-Out interface. The firmware is not responding.

As you can tell from my lazy comments, this became a pretty big deal as iLO was completely toasty on these boxes where SIM was generating this alert.  I contacted HP and it was a pretty entertaining exchange.

HP Support: Are you running HP SIM 5.3?

Me: Yes

HP Support: Do you have iLO version 1.60 or 1.61?

Me: Beats me.  The iLOs are not responding so I have zero way to tell.

HP Support: I think I know your problem.

Me: <Silence for a minute.>  Well don’t leave me hanging here.

HP Support: I’m reading an email alert that came through while we were talking.   There’s a hotfix for HP SIM 5.3 and a recommendation to go up to firmware 1.70 for the iLO.

Me: Oh great.  *sigh*

So now I get to physically pull power cords to fix iLOs everywhere.  (Yes this means an OS reboot too.)

* Hey HP.  Put a button onto the servers to “reboot the iLO”.

Today’s big news:

Cisco has come out with a single stop full solution rack for CPU, Disk & Network in one using all the best of virtualization technology of Storage, Server & Networking.   Tight VMware integration, all Cisco hardware & lots of virtualization technology at 10G.

Over the past several years I’ve kinda figured that Cisco has lost their way.  Floundering a bit.  Now I know someone at Cisco has a brain.   VMware has blazed the path in recent history into the Enterprise Datacenter.  The gotcha  has been trying to integrate with networking and storage.   This is in theory resolved by this solution of a fully integrated delivered product line.

So how does HP & IBM & Dell correspond with this?