Chinwag Reloaded with Alan Renouf (@alanrenouf)


Alan Renouf blogs at Despite the name, Alan or Al to his friends is most certainly not French. But rumour has it his family go all the way back to 1066 and the Battle of Hastings. I first met Alan via the London VMUG when he used to do workshops in the morning about all things PowerCLI/PowerShell related. Imagine that? Those folks were being “taught” by the guy who would go on to the be Product Manager! Just goes to show the hidden value of VMUGs, you often meet the rising stars of the future there…

I caught up with Alan at the VMware CorpHQ Campus in Palo Alto, CA. I happened to be in town doing one of my monthly catch-ups with the EVO:RAIL team, so it was great to catch-up with Al over lunch. As you might suspect our discussions were very PowerCLI orientated, but we did range around other subjects to including:

  • What have you learned about products now that you manage one? Much jive between your experience as user?
  • PowerActions
  • What’s new in 5.8
  • Why have you stopped blogging (joke!) but serious no blogs between June til Sept – is that life as PM


The links above Alan’ss photo take you to the feeds – iTunes for Video/Audio versions of the blog as well as a generic RSS feed location as well. Please subscribe! The Twitter icon will take you to Scott Lowe’s twitter. Please follow Scott because he only 18K+ followers and could do with the boost! [JOKE!]

Click here to listen to the directly on this site…. or alternatively watch the youtube video below:


Posted by on December 19, 2014 in Chinwag

Comments Off

London VMUG: Its not the X-Factor, It’s the V-Factor!

This year I helped spearhead an initiative to promote more people speaking from the vCommunity at VMUGs in general. VMUGs have always had a fair slice of “vendor” speakers, which I guess I must admit I’m part of – now that I work for VMware. But I still passionately believe that User Group meetings should be about Users, not the vendors – who let’s remember are largely there to pimp their warez. If you haven’t heard of the “Feed4ward” program – toddle off to the landing zone to learn more.

There’s now another initiative called vFactor which is the name adopted by the London VMUG – to describe a more carrot than stick approach to garnering your contributions. So team there have put together vFactor – which award prizes for the best session delivered by a member of the community. The prizes are so jaw-droppingly excellent – I’m considering resigning my position at VMware, to throw my hat in the ring [JOKE!]

  • 1st Prize: MacBook Air
  • 2nd Prize: iPad Air
  • 3rd Prize: iPad Mini
  • 4th Prize: Amazon voucher
  • 5th Prize: Amazon voucher

So basically – Come and present a 10min talk at London VMUG on 22nd Jan 2015 and have the chance to win Apple shiny good or Amazon vouchers – you can submit abstract & the rules here now –

The VMUG committee pick 5 submissions from the entries to present at the event and the closing date is 19th December 2014 – everyone who presents will win something, and the audience will vote for their favourite to determine what loot you go home with from the above list…

You may be no Duncan Laverick, or Mike Dennemann. But if you regularly tell your colleagues what your project is and how its evolving you can speak at a VMUG…

Finally, is it me or is the prospect of Duncan Laverick – one the most horrifying thought experiments ever to be born?



Posted by on December 12, 2014 in VMUG

Comments Off

UK VMUG raises £400 for Blood Bikers


Last months UK VMUG was a roaring success in more ways than one. At the event we managed to raise £400 for the “Blood Bikers” charity (National Association of Blood Bikes). Their volunteers are (mainly retired) bikers who ferry plasma, blood, milk, and other medical supplies around the country from one hospital to another from the hours of 7pm at night, round the clock til 7am. It’s a great charity – because of the great outcomes – but also saves our NHS thousands (perhaps millions?) a year in providing service for free. The money was raised in a raffle at the vCurry and the event itself – and was won by none other than Chris Wahl.

The breakdown of the money raised is a little bit funkier than the straight £400. We actually raised £325.10. But the runners-up in the vCurry Quiz (the somewhat risque named “Team vCOC”) donated their prize of £45 in Amazon vouchers to the cause. It’s proved difficult to change this donation in back into cash. So I’ve “bought” the vouchers, and added the cash to the total. My wife and I figured we would probably spend the money on Amazon at Christmas anyway. That brought the total £380.10. So we then made a personal donation to the pot of £19.90 to bring the grand total of £400. Phew!

Anyway, in the interest of transparency – below is record of the transactions – including a letter from National Association of Blood Bikes

Read the rest of this entry »


Posted by on December 12, 2014 in Announcements

Comments Off

Chinwag Reloaded with Scott Lowe (@scott_lowe)


Scott Lowe blogs at We’ve known each for a while – I think we met first at VMworld in Las Vegas. Ever since I’ve been bumping into him at VMworlds, and once unexpectedly at Heathrow Airport. In this episode of the chinwag we talk about the following topics

  • Closing Comments in blogs – Does signal an end of era where bloggers developed an active dialog, and now were broadcasters?
  • Intel Rack-Scale
  • Vagrant. What is it? Yet another scripting language to master or something really game shifting?
  • Open Source  – where do you feel the movement is going. Is it still really for ISPs and uber companies with deep pockets and full time development staff?


The links above Scott’s photo take you to the feeds – iTunes for Video/Audio versions of the blog as well as a generic RSS feed location as well. Please subscribe! The Twitter icon will take you to Scott Lowe’s twitter. Please follow Scott because he only 18K+ followers and could do with the boost! [JOKE!]

Click here to listen to the Podcast on this site…. or alternatively watch the youtube video below:


Posted by on December 4, 2014 in Chinwag

Comments Off

EVO:RAIL and NetApp – And then there was 9…

Screen Shot 2014-12-03 at 09.37.39

I’m pleased to say that NetApp have become the 9th Qualified EVO:RAIL Partner (QEP) to join the group. I think this great news for customers and for those folks who are from a storage perspective aligned to NetApp is their preferred vendor. This means I will have to have good old sweep through my powerpoint decks and do some logo adding!

Like with a lot of the other EVO:RAIL partners, NetApp have agreed to add-value and differentiate themselves in the market place. So you might have seen that Dell and SuperMicro have partnered with Nexenta to provide their software on top of EVO:RAIL. Whereas HP have a bundle which includes their StorVirtual VSA. Whilst EMC have yet to release their EVO:RAIL, Chad Sakac on his virtualgeek blog has indicated that they are likely to include Recover Point and Avarmar Backup functionality. In fact in the EVO:RAIL team were working on a new version that enable our partners to intregrate this added value into their primary ‘configuration” workflow. More about that in good time, when the new version of the EVO:RAIL engine is released.

The way I’m seeing it is every EVO:RAIL is the same (RAM, Disk capacity, CPU, Networking) but every EVO:RAIL is different – depending on how the QEP choses to add-value or differentiate themselves in the marketplace…

In the case of NetApp they will be bundling along side the EVO:RAIL their own storage. This is physically a separate 2U NTAP FAS appliance which will be bundled with a NetAPP branded EVO:RAIL appliance that is based on vSphere and Virtual SAN – total footprint of both appliances in a rack is 4U. The NTAP FAS unit will ideally connect into the same TOR switch that the NTAP branded EVO:RAIL appliance will connect into. NTAP and the channel may also decide/choose to bundle in a TOR switch. EVO:RAIL will expose a link and launch capability that NTAP can leverage (as well as other QEPs) from within the EVO:RAIL DCM Engine to rapidly deploy/configure/integrate the NTAP FAS unit with the NTAP branded EVO:RAIL appliance.

So to say in a nutshell – if folks choose to source their EVO:RAIL from NetApp they will have the choice and flexiblity to use both VSAN and NTAP FAS unit. I think Chuck Hollis ( has it right on the money. For me its all about the same, but different viewpoint. From a hardware perspective the EVO:RAIL offers the same specification from partner-to-partner. How they chose to take that to their customers is really their decision. They know their customers and their requirements, and it isn’t for VMware to go about dictating to those partners what secret sauce they choose to bundle. So find this concept an odd one, but I don’t. I guess in life folks like to go about making strict catagories – Converged is this. Hyper-converged is that. Allowing for little or no innovation for something that might be something in between. I guess the desire to make very descrete catagories is an attempt to “make life simple”. After all there is so much going on in the world of IT now, who wouldn’t want to put everything in neat little boxes. But I actually think its more interesting to have those assumptions questions. So will folks find the NetApp flavour attractive. Perhaps to early to tell. But what I do believe is more choice and variety is better news for customers to have that choice, than none at all…

More Info Here:


Posted by on December 3, 2014 in EVO:RAIL

Comments Off

Mike’s Music – 2014 Round-Up

It’s been absolute age since I wrote a blogpost about my 1st love – music. I’ve been busy with work (but heck, I always am) and in my mind I was waiting for something “big” to jump out at me and reflecting pure musical brilliance on Jools Holland’s Later. But then it occurred me I’ve been picking up gems of musical brilliance all year – but not written them about them. As the year draws to a close I’m looking back over the year and what I’ve been listening to. And I thought I would rave about them here, in the hope that others might be similarly inspired.


Read the rest of this entry »


Posted by on November 25, 2014 in Mike's Music

Comments Off

Chinwag Reloaded with Chad Sakac (@sakacc)

It’s back! I’m pleased to announce that I’m getting back into the Disc Jockey chair – and The Chingwag is back! With a bang. I’m so stoked to say my first chinwaggie is Chad Sakac of EMC. This is the first time I’ve had him on the show. And its all part of my “Rolling Thunder” Chinwag Relaunch – I will be interviewing the TOP people in our world for the next couple weeks. In the medium term I will reaching out to folks in the community who I know online or meet at VMUGs. So this new version of the podcast will not be a vRockstar only thing.

I’ve building up a library of Chinwags with folks since mid-October – this was recorded just before VMworld EU – 2014


Chad Sakac blogs at’ve known each for a while – met first at VMworld in Las Vegas. That was back when I was getting into SRM, and EMC and NetApp both helped me out with access to storage for the SRM 4.0 book. In this chinwag Chad gives his take on

  • Converged Vs Hyper-Converged
  • EMC’s plans for EVO:RAIL
  • EMC XtremIO upgrading


Click here to listen to the Podcast on this site…. or alternatively watch the youtube video below:


Posted by on November 21, 2014 in Chinwag

Comments Off

EVO:RAIL – Anatomy of a JSON Configuration File

One common question that comes up around EVO:RAIL is where all those pre-defined settings in the appliance UI come from, where are they stored and how can you modify them? In case you haven’t seen the EVO:RAIL UI (and if you haven’t what stone have you been living under! :-) ) The “Configuration” front-end and engine of EVO:RAIL is a service (vmware-marvin) that runs inside the vCenter Server Appliance which initially comes up on node01 of the first appliance itself:

Screen Shot 2014-10-24 at 13.30.01

As you can see from the single page there are actually quite a number of settings gathered before the configuration is initialized, built, configured and finalized. These include:

  • ESXi host naming convention and vCenter FQDN
  • vSphere ESXi host networking (IP Ranges/Subnet Mask/Default Gateway/VLAN IDs) for vMotion, Virtual SAN and ESXi hosts
  • Virtual Machine Network (VLAN ID and associated network names)
  • Passwords (for ESXi and vCenter) and optional Active Directory configuration
  • Global Settings such as: Time Zone, NTP, DNS, Syslog/Log Insight IP, Proxy Server settings including hostname, port, username and password


Where do these settings come from? Well, they are held in a JSON file on the vCenter Server Appliance that runs on the EVO:RAIL in the following directory:

/usr /lib /vmware-marvin /marvind /webapps /ROOT /WEB-INF /classes/

The file is called default-config-static.json

There is a sample JSON file in our User Guide appendix – for ease of use I’ve reproduced it at the end of this blog post.

The reason the file is named “static” is that it uses a static IP configuration for all components that require an IP address. When you order an EVO:RAIL, you could supply the VLAN/IP/hostname data to a Qualified EVO:RAIL Partner (QEP) for customization at the factory. Then, when it arrives onsite, you would just plumb it into the network and click the “Just Go!” button. Personally, I always check “Customize Me!” unless I know I’m just repeating the same process over and over again for validating purposes. For example I’m working a lot in the HOL where I repeatedly build the same EVO:RAIL appliance for testing purposes.

Screen Shot 2014-10-24 at 13.44.35

Incidentally, if you do click “Just Go!”, then all you will be prompted is to set the ESXi hosts and vCenter Server passwords. From here you can change the default password that is held in the JSON file to something that suits your needs.

Screen Shot 2014-10-24 at 13.48.41

So what if the settings in the JSON file don’t suit your needs, and you want to change them without endlessly retyping? Perhaps you would like to send an EVO:RAIL out to a remote location, and send the person who’s plugging it in for the first time, settings that are valid for their location. Simple – send them a custom JSON file with the correct values…

When confronted by the EVO:RAIL Welcome screen they would use “Customize Me!” and then click the link to “UPLOAD CONFIGURATION FILE”:

Screen Shot 2014-10-24 at 13.52.24

EVO:RAIL will then inspect and validate the JSON file to check it for errors – so if there are bum entries in it you will receive a warning. (Note that EVO:RAIL does not validate the syntax of the JSON file, so syntax errors will prevent the file from being read.)

For instance in the example below I deliberately created a JSON with insufficient IP addresses in the pool for the Virtual SAN pool of IP addresses, as well you can see there is a bum IP range from the JSON file:

Screen Shot 2014-10-24 at 14.02.50

Screen Shot 2014-10-24 at 14.03.36

Note: 10.20.x.y and 10.10.x.y are not invalid IP ranges.

If no errors are found in the JSON file, then it should pass with flying colors – and lead you to the Build Appliance button.

Screen Shot 2014-10-24 at 14.07.22

One thing you’ll notice about my ranges is that they begin x.y.z.101 to x.y.z.116 to include 16 IP addresses even though I only need 4 IP addresses for the 4 servers in my EVO:RAIL appliance. That’s deliberate on my part – so if I ever need to add an additional EVO:RAIL appliance, it can simply be detected on the network and added. To add an appliance all I need to supply is the ESXi and vCenter passwords in the tiny workflow. By creating a pool of IP addresses in the JSON file, it is very easy to add more EVO:RAIL appliances. When a new appliance becomes present on the network, the EVO:RAIL Management UI will discover it, and allow the administrator to add it to the existing cluster.


And if there are free IP addresses in the pool, all that’s required is the passwords. These are indicated by the green ticks in the Add New EVO:RAIL appliance UI.


If the network pool of IP address is not big enough or depleated – you get blue (i) information badges, and you’ll need to specify additional bundles of IP ranges to add the appliance. This is mildly taxing, and I think its just neater to specify the full range of IP (16) for each type of pool upfront. It’s just a slicker experience.

Screen Shot 2014-10-24 at 16.31.29

If the pool is depleted of IP addresses because the range was set too small its not the end of the world – it just means you need to fill in more IP data before you can add the EVO:RAIL appliance. In the example below where the IP pool was kept deliberately small (just 4-IPs) there’s a blue (i) info alert to indicate the operator needs to add more IP addresses before continuing.


  • You can use JSON files to automate and customize the build out of EVO:RAIL without the need to manually type IP configurations (not that it takes much time to do that!)
  • The EVO:RAIL Configuration engine validates your IP setting
  • It’s nice to have a pool of IP addresses large enough to make adding a second appliance a trivial process.
  • However, there are limits – the incorrect IP address, or bad VLAN values could still pass a validation test by the operator inputting incorrect settings or by the person who creates the JSON file. After all, EVO:RAIL has no idea if you have mistyped the default gateway IP address…
  • Finally, it is possible to leave the password field in the JSON file blank – which means no password is ever stored in clear-text and the person doing the configuration file would have to type in a valid password.


Click here to see a sample JSON file:

Read the rest of this entry »


Posted by on November 10, 2014 in EVO:RAIL

Comments Off

EVO:RAIL – Getting the pre-RTFM in place…

One of things we are really proud of within the EVO:RAIL team is how easy, simple and quick it is to setup the first EVO:RAIL and then using our auto-discovery method – scale out the environment by quickly adding subsequent EVO:RAIL appliances. As ever the experience is highly dependent on the network pre-requisites being in place, and so as long as you RTFM, you should be good-to-go.

In case you don’t know, EVO:RAIL implements a version of “Zero Network Configuration” which is a widely recognized RFC standard. VMware’s implementation of this is called VMware Loudmouth (it was originally to be called LoudSpeaker) and it’s a service that runs on both EVO:RAIL’s built-in vCenter Server Appliance and on the 4-server nodes that run VMware ESXi in the 2U enclosure. Below you can see the Loudmouth status running on the first ESXi server node, and on the vCenter Server Appliance:

Screen Shot 2014-10-23 at 12.18.52

The User Guide for EVO:RAIL outlines the network requirements for the appliance, but its worth restating them here – and also flagging up the consequences if you don’t meet these pre-requisites. Firstly, EVO:RAIL requires a physical switch (which isn’t part of the offering, but most of the Qualified EVO:RAIL Partners (QEPs) would be happy to sell you one if you don’t have one already!) that supports 10Gbps networking. Both RJ-45 or SFP+ connections are supported, and some QEPs have two EVO:RAIL offerings to support these, although some just support one type at present.

IPv6 support is required on all the ports on the switch, and you need at least 8 ports free to plug-in each of the 2x10Gps nics (vmnic0/1) that the server in the EVO:RAIL uses. You may actually want to have a separate switch for the single 1Gps nic that each server node has for BMC/ILO/DRAC style access. These could be plugged into the 10Gbps switch if you have spare ports, or you might prefer to plug them into a spare 1Gbps switch if you have ports free. After all those 10Gps ports are precious… The User Guide has a good diagram that illustrates this – and of course, it is supported to have vmnic0 plugged into one switch, and vmnic1 plugged into another – to provide switch redundancy.

Screen Shot 2014-10-23 at 11.55.03

Whilst EVO:RAIL doesn’t require VLANs as such, I think it is the case that the vast majority of people will want to use them. Personally I would strongly recommend that if the switch is brand new or hasn’t been used by vSphere before you ensure that the VLAN configuration is in place BEFORE you even rack-and-stack the EVO:RAIL. This will lead to a more out-of-the-box experience.

In total you will need at least 3 VLANs to cover the vMotion, Virtual SAN and Virtual Machine traffic. Of course, you can have as many VLANs to separate your various VMs as you need. By default management traffic is not tagged by the EVO:RAIL appliance, and will reside on the default VLAN of the physical switch (this is normally VLAN0 or VLAN1 depending on the vendor). You’ll notice that in the EVO:RAIL UI there isn’t an option to set VLAN Tagging for the VMware ESXi hosts:

Screen Shot 2014-10-23 at 12.30.55

The EVO:RAIL could have set a VLAN ID, but of course doing that would have disconnected you from the appliance, and potentially it would mean a re-IP of the local management machine, and even a re-patch to the right VLAN. So by default the EVO:RAIL uses the default network of VLAN0/1 on the switch. If you want/need to change the management network – you could use the Web Client or vSphere Desktop Client to enable tagging for the Management VMkernel portgroup and the management virtual machine network. For me this is similar to the days when I used the “Ultimate Deployment Appliance” to install ESXi across the network using a PXE situation. In that setup I needed to use a native VLAN, as you don’t get VLAN tagging support in a PXE boot. For me it was easiest to use VLAN0/1 to do the install, and make my ESXi use that as the management network, rather than go through a reconfiguration afterwards.

Of course you do get the option to set the VLAN Tagging ID for vMotion, Virtual SAN and of course, for your VM Networks:

Screen Shot 2014-10-23 at 12.31.54 Screen Shot 2014-10-23 at 12.32.53 Screen Shot 2014-10-23 at 12.34.12

Now, here’s the important part – both the Virtual SAN and Management VLAN will need multicast enabled for both IPv4 and IPv6. This has always been a requirement for Virtual SAN, and the EVO:RAIL requires it for the auto-discovery process to work correctly. You could enable multicast for IPv4 and IPv6 on ALL the ports of the physical switch (this would be desirable if the customer isn’t using VLANs at all), or alternatively enable it just on the two networks that require it (Management and Virtual SAN). If you are using multiple switches ISL multicast traffic for IPv4 and IPv6 must be able to communicate between the switches.

To allow multicast traffic to pass through, you have two options for either all EVO:RAIL ports on your TOR switch or for the Virtual SAN and management VLANs (if you have VLANs configured):

1) Enable IGMP Snooping on your TOR switches AND enable IGMP Querier. By default, most switches enable IGMP Snooping, but disable IGMP Querier


2) Disable IGMP Snooping on your switches. This option may lead to additional multicast traffic on your network.

In my experience folks often over-react to the idea of multicast. I’m not sure why that is, given that this traffic is used to transmit a relatively small amount of network traffic in the form of metadata. Perhaps old-timers like me are thinking back to the days when technologies like Symantec Ghost or multi-cast video had the potential to flatten a network with excessive multicast traffic, and bring it to its knees. But we are not talking about that sort of volume of traffic in the case of Virtual SAN or EVO:RAIL. In case you don’t know (and I didn’t before joining the EVO:RAIL team) IGMP Snooping software examines IGMP protocol messages within a VLAN to discover which interfaces are connected to hosts or other devices interested in receiving this traffic. Using the interface information, IGMP Snooping can reduce bandwidth consumption in a multi-access LAN environment to avoid flooding an entire VLAN. IGMP Snooping tracks ports that are attached to multicast-capable routers to help manage IGMP membership report forwarding. It also responds to topology change notifications. For IPv6, MLD (Multicast Listener Discovery) is essentially the same as IGMP (Internet Group Management Protocol) in IPv4.

Note: Text in italics are direct quotes from the EVO:RAIL user guide!

So the 60million dollar question. What happens if you choose to totally ignore these recommendations and pre-requisites? Well firstly, it strikes me as obvious that configuring EVO:RAIL to speak to VLANs that don’t even exist is going to result in all manner of problems. You really have two options – either do the VLAN work on the physical switch first (my personal recommendation) or do not use VLANs at all and have a flat network. What’s not a good idea is to setup EVO:RAIL without any VLAN configuration, and then reconfigure the physical switch to use them after the fact. That would mean vSphere networking on each of the physical hosts would need reconfiguring to be an area of the VLAN ID for tagging purposes on the portgroup.

The above seems pretty straightforward to me – and I’m really sorry if it sounds like teaching Grandma how to suck eggs. The multi-cast requirements are less obvious as most folks will be as new to EVO:RAIL as I am (this is my 9th week!). What happens if the multicast requirements aren’t met on either the Management or the Virtual SAN networks? Firstly on the EVO:RAIL Management network, if multicast is not enabled, then the servers that make up the EVO:RAIL will not be discovered and you will see an error message at around 2% of the configuration process.

As for Virtual SAN the EVO:RAIL configuration will continue and will complete, but you will find Virtual SAN will show that individual disk groups have been created for each appliance node, rather than one single disk group containing all the storage (HDD/SSD) for all the nodes in the cluster. It’s perhaps best to use the vSphere Web Client which is available from the EVO:RAIL Management UI to inspect the status of Virtual SAN configuration, and from there you can:

  1. Navigate to the Home tab, select Hosts & Clusters
  2. Expand the Marvin-Datacenter and Marvin-Virtual-SAN-Cluster to show you now have four hosts provided by the EVO:RAIL Appliance
  3. With the Marvin-Virtual-SAN-Cluster selected, click the Manage Tab, and under the Settings column, select General under Virtual SAN.
  4. Confirm that all 4 hosts are marked as “Eligible” and the network status is “Normal“. If multicast has not been enabled you would see the status “Misconfiguration Detected”



Note: This screen grab is taken from the new version of the HOL I’ve been working on.

As more realistic view from an actual EVO:RAIL appliance would look like this:


If you’re seeing “Misconfiguration Detected” most likely you will find that the root of the problem is multicast related. If you are troubleshooting multicast issues with Virtual SAN, a good place to start is this blog post on the vSphere blog written by my colleague, Joe Cook:


  • Get your switch configured first with the right settings. This is true of almost any hyper-converged environment.
  • If you get the 2/3% an error during the EVO:RAIL configuration process – check your multicast settings for the management network (VLAN0 or VLAN1 depending on vendor). Then click try again.
  • After the build of the first EVO:RAIL appliance, check your Virtual SAN settings to make sure there are no unpleasant messages and that all the server nodes are in the same disk group.


Posted by on November 4, 2014 in EVO:RAIL

Comments Off

Back To Basics: Viewing and Modifying vSphere HA Settings

 All the settings for vSphere HA can be found under the properties of the cluster >> Manage and the Edit button.

Screen Shot 2014-09-17 at 14.50.46.png

Host Monitoring

The host monitoring portion of the Edit Cluster Settings options control whether vSphere HA is turn on or off. As indicated earlier it is possible to turn off Host Monitoring. This controls whether the vSphere host share the network heartbeat that is used to calculate if the host is alive or dead. This can check can be temporarily turned off if you know that some network maintenance (such as physical switch or router upgrade) is likely to cause the network to be down for a period of time. The virtual machine options control what happens by default if there is a failure or isolation event. Two settings are available here, VM Restart Priority and Host Isolation Response. Restart Priority allows for four options – disabled, low, medium, high. By default is medium is selected, and all VMs would have the same restart priority of medium. It’s then possible under the VM Overrides options to add individual VMs, and indicate that some VMs have a low priority, or high priority – are started after or before any VMs with a medium priority. Alternatively, VMs can be excluded from the restart process altogether by using the VM Over-rides to disabled. This can be useful if you have non-critical VMs are that are not needed be available – and frees up resources for the more critical VMs. The Isolation Response controls what happens if a host becomes disconnected from the cluster due to some network outage or configuration error. In this case the isolated host may well be able to communicate to the router, but not the other hosts. Alternatively using what are called “datastore heartbeats” vSphere can work out that the host maybe disconnected from the network, but still connected to shared cluster resources. In such a case the host could still be running, and the VMs are unaffected. In this case the default policy would be to “Leave Powered On”. The alternatively, is assume a failure has occurred and either power of and restart, or shutdown the guest operating system, and restart on to the remaining hosts.

Screen Shot 2014-09-17 at 14.52.04.png

Admission Control Policy

Screen Shot 2014-09-17 at 15.05.33.png

The Admission Control Policy controls how resources are reserved to the clusters – and whether VMs are powered on or not based on the remaining resources left in the cluster. One policy allows for the definition of capcity by Static Number of Hosts. The spinner allows the SysAdmin to indicate how many host they feel they could comfortably lose but still maintain good quality of service. This spinner can now be taken as high 31. This is because the maximum number of vSphere host in a cluster is 32, which would allow logically for 31 failures leaving just one 1 host left over. As you might imagine its highly unlikely that one remaining node could take over from the lost of 31 servers. However, its more reasonable to suggest that in 32 node cluster that is 50% loaded, that a much higher number of physical servers could fail than the default of just 1.

By default the “slot” size is calculated based on the total maximum reservations used for CPU/Memory. A reservation is expressed on a VM or resource pool as guarantee of the given resources – rather than it being allocated on a on-demand basis. The idea of basing the “slot” size on these values is to try to guarantee that VMs are able to have their reservations allocated during power on. In some case this dynamically calculated “slot” size isn’t appropriate for customers – as it can be skewed by mix of very large and very small VMs. That can result in an either very large slot sizes which quickly reduces the number of VMs that can be powered on, or very small slot sizes which are quickly consummed by series of very large VMs. For this reason it is possible to modify the static number of hosts policy, by specifying a Fixed Slot Size expressed in CPU in Mhz and Memory in MB. Additionally, the Calculate button can be used to see which VMs could potentially require more than one slot. This can be used to verify if the fixed slot size is appropriately set. Once calculated the View link will show a list of VMs requiring more than one slot.

As alternatively to using a static number of hosts together with a slot size, vSphere provides the option to manage admission control by reserving a percentage of cluster resources. As you might gather this involves reserving an amount of CPU or Memory as proportion of the overall amount provided by the cluster. This can have a very similar effect to using a static number of hosts. For instance on three node cluster, if 33% was reserved for failover, this would be similar (but not the same as) to indicating +1 redundancy. This method dispenses with slot sizes altogether, and has proved to be a popular reconfiguration of vSphere Clustering.

SysAdmin are able to configure Dedicated Failover Hosts – in these case the specified hosts do not take a VM load, and held in reserve ready for vSphere host failure. Whilst this guarantees that the resource will be available. Many customers find this an expensive option and would prefer to allow their hosts to take some kind of load, but manage the overall load, with a reservation of resources.

Finally, Admission Control can be turned of by using Do not reserve capacity. This keeps vSphere HA running but doesn’t impose any restrictions on the whether a VM can be failover or power on manually. Occasionally.

VM Monitoring

VM Monitoring is sub-component of VMware HA, and is an optional feature. It can be used to inspect the state of virtual machines, and based on the result reboot them if they appear to have become unresponsive. The default is VM Monitoring is disabled, and some customer prefer this because they are anxious about vSphere ‘getting it wrong’ and unnecessarily rebooting VMs. This is because VM Monitoring inspects the VMware Tools “Heartbeat Service” and uses a successive lack of responses to determine if a VM is stalled or not. Significant work has been undertaken by VMware to lessen this concern – so in conjunction with the heartbeat, VM Monitoring now inspect IO activity. The assumption is that if the heartbeat returns no answer AND no disk IO activity is taking place, there’s good likelihood that the VM has halted with either a kernel panic in Linux or Blue Screen of Death (BSOD) in Windows.

When VM Monitoring is enabled it comes with two options – VM Monitoring Only and VM and Application Monitoring. First monitors the VM heartbeat and restarts if no response is given within a specific time. VM and Application Monitoring checks for heartbeat signals from both VMware Tools as well as Applications/Services running within the guest operating system. This is called VMware AppHA and requires a virtual appliance to be configured, leverages VMware’s Hyperic software inside the guest operating system, and offers support to range of applications/services running in Enterprize environments. For simplicity we will cover “VM Monitoring” here, and cover VMware AppHA separately.

Screen Shot 2014-10-22 at 16.33.36.png

Monitoring Sensitivity comes with two options a preset value which allows you to indicate a Preset level of sensitivity. With high sensitivity this sets a sensitive policy that would only reset the VM if three have been three failures in 60mins, and checks counts a failure as no response within 30secs. As you move the slider bar from right to left VM Monitoring become increasingly conservative, and restarts are less likely to occur

High (2) – No response in 30sec, 3 failures per 60min

Medium (1) – No response in 60sec, 3 failures in 24hrs

Low (0)- No response in 2mins, 3 failures in 7days

If these settings are to aggressive or too conservative – then Custom setting allows the administrator control over these tolerances.

Datastore Heartbeating

Along side using network heartbeats to evaluate the availability of a vSphere host, vSphere HA can also validate storage heartbeats to validate the connectivity. The assumption being if both network and storage heartbeats are both unavailable, then its highly liked the host has suffered a catastrophic failure. This can be regarded as increasing the condition required to initiate the restart of the VMs to another host, and another method of reducing the occurence of split-brain. Datastore Heartbeating requires two or more datastores accessible to all the hosts in the cluster, and is a mandatory feature. Therefore if the hosts do not share datastores or if HA is enabled before the storage configuration has completed, this is likely to generate a warning on the vSphere hosts.

Screen Shot 2014-10-22 at 17.00.28.png

By default the vSphere HA Automatically select(s) datastores accessible to the host(s). In some case this selection may not reflect your preference. For instance if you work in a volatile lab environment where storage is temporarily mounted, VMs created, then destroyed and then unmounted. You may perfer to instead use datastores which know will always be mounted to the cluster. For this reason it’s possible to Use datastores from the specified list or else Use datastores from the specified list and complement automatically if needed. This last option feels like a good compromise between control, whilst at the same time protecting the environment from situations where the datastore maybe reconfigured or become unavailable.

In this case the policy was change to ensure that the “Software” and “Current-Templates” datastore locations were selected as the datastore heartbeat preference.

Screen Shot 2014-10-22 at 17.14.19.png

Advanced Options

Advanced Options allows the administrator to supplement the vSphere HA configuration with additional parameters that control is functionality. A complete list of these options are available in VMwareKB Article 2033250. Typically, these settings are modified in environments that present unique requirements or demands generally around networking. These settings have been recently updated with the March, 2014 release of VMware Virtual SAN. vSphere HA can now leverage aspects of the Virtual SAN networking requirements as part of its internal logic.


Posted by on November 3, 2014 in BackToBasics

Comments Off