Chinwag Reloaded with Amy Manley (@wyrdgirl)


This week’s chinwaggie is Amy Manley (@wyrdgirl) she blogs at and has particular interest in orchestration and automation. I spoke to on a weekend back in Feb, but just getting round to releasing our chat this week – I like to have a bit of buffer in terms of interviews in case I get busy with stuff related to my day job with the EVO:RAIL team.



Posted by on March 18, 2015 in Chinwag

Comments Off

EVO:RAIL – Under The Covers: Daemons in the Machine

This title is a bit of a pun on “Ghost in the Machine”, typically used to describe some undocumented hidden magic at the heart of things – think of Arthur C. Clarke’s famous pronouncement!. Of course, there’s no ghost in the EVO:RAIL machine, but there are certainly daemons. By which I mean the services running in the background that make it all happen. In the spirit of demystification I want to explain where they are, what they are, what they do (roughly!) and how to test if they are functioning – and what to do if they’re not!

Note: Quite why Linux calls these things “daemons” and Windows calls such built-in processes “services” has always been a mystery to me. But for the “We only run Windows” folks out there – two things.

A.) No you don’t – Linux is all over the place, you just can’t see it…

B.) daemons = services, services = daemons.

Try to not get too worried by that tautology, it is mere linguistics. Anything else is in the realms of Harry Potter. :-)

VMware Loudmouth

There are two core daemons with the EVO:RAIL; one is called “Loudmouth” and the other has the original project name of EVO:RAIL – aka “MARVIN”. The Loudmouth service runs on BOTH the EVO:RAIL node and the vCenter Server Appliance. The “Loudmouth” service running on ESXi is installed at the factory using a VIB (Virtual Infrastructure Bundle) file, and in the vCenter Server Appliance using a .RPM file. These are pre-installed at the factory by your preferred Qualified EVO:RAIL Partner. They should start automatically and you shouldn’t have to do anything to it. However, if you did want to validate that VMware Loudmouth had started and was working properly you would:

On The ESXi Host run the command:

/etc/init.d/loudmouth status

On The vCenter Server Appliance run the command:

/etc/init.d/vmware-loudmouth status

You can use the flags “restart”, “stop” and “start” as well.


VMware Loudmouth is an implementation of Zero Network Configuration or ZeroConf (think of it like Apple Bonjour for vCenter/ESXi). It’s primarily used in three areas:

  • The initial discovery and configuration of the 1st
  • The adding of subsequent EVO:RAIL appliances to add more compute and storage capacity – call it auto-discovery and auto scale-out if you wish!
  • Replacement of failed node with a brand new node, typically caused by a motherboard failure, or CPU failure – where wholesale replacing of a node is quicker and easier than replacing individual components.


In contrast the VMware MARVIN daemon only resides on the vCenter Server Appliance. It’s essentially a dedicated Apache Tomcat instance whose sole purpose is to assist the administrator in both configuring the EVO:RAIL appliance for the first time, and also providing the Management UI once the configuration has completed.


Note: This is a screen grab of the EVO:RAIL Management UI as provided by the VMware MARVIN service. The VMware MARVIN service listens on a SSL enabled port number of 7443. For instance to get to the EVO:RAIL UI you would use

As you might suspect, confirming the status of VMware MARVIN means connecting to the vCenter Server Appliance and running the following command:

/etc/init.d/vmware-marvin status

Again there are very few reasons to ever restart this service – and of course a reboot of vCenter would restart it as well. I can give you just one case where I have had to restart it. Occasionally, I like to modify the default JSON file that provides the default IP Pools, Hostnames and so on in the main Configuration pages. The JSON file is located at: /usr /lib /vmware-marvin /marvind /webapps /ROOT /WEB-INF /classes/ and is called default-config-static.json. The file JSON file is read when the vmware-marvin service starts for the first time. So any edits to it require a restart of the service in order for changes be picked up and used.


EVO:RAIL is magic, but it’s not magic spells that makes it work but daemons. Fortunately, these daemons are good ones that are not your enemy, they’re more like Prof. Dumbledore than Lord Voldermort. They are pre-installed by the QEP at the factory, and you should never have to restart them, or check their status. But you can. Simple. Now, I might just disappear in a puff of my own logic….


Posted by on March 18, 2015 in EVO:RAIL

Comments Off

EVO:RAIL – Under The Covers – Networking (Part1)

This is a topic I’ve written about previously in a blogpost called “Getting the RTFM In place”. Typically enterprise products are generally easy to setup so long as you have got your pre-requisites in place – fail to do that and your mileage will vary significantly. You’d be surprised (or perhaps not!) how frequently things go awry in life simple because the basics haven’t even been addressed. So despite having written about this before I want to touch on this topic again – this time adding steps the guy charged with the setup routine can follow. My apologies in advance if anyone thinks I’m teaching Grandma how to suck eggs. To be honest most of this is well documented in our EVO:RAIL User Guide. But heck, who downloads and reads an 8-page PDF before holding their nose and jumping in with both feet?

So the first thing you need to confirm is – do you have a 10GbE switch? EVO:RAIL requires a 10GbE switch primarily to ensure VSAN performs at optimum performance under ALL conditions (for instance a rebuild caused by a hardware failure…). But also because EVO:RAIL requires IPv4/IPv6 Multicast for the auto-discovery process carried out by the VMware Loudmouth service/daemon. In case you don’t know, VMware Loudmouth is an implementation of the “Zero Network Configuration” which is used for three core processes:

  • 1st appliance build,
  • Adding an additional appliance
  • Adding a replacement node (where the original node has experienced a hardware failure).

This 10GbE would need at least 8 free ports (RJ-45 or SFP+) to plug in the 2x10Gps ports on each one of the 4-nodes in the 2U chassis (For the mathematically challenged that’s 2×4=8). Optionally, you would need to find a further 4-ports for the BMC (DRAC/ILO/RAC). Of course, you need a spare port for communicating to the EVO:RAIL Configuration UI (this is the VMware-Marvin daemon that runs inside the vCenter Server Appliance which is on the same management network as the ESXi hosts.)

Screen Shot 2015-03-10 at 12.52.13

Note: This screen grab comes from our new “New User Guide” available here:

The first EVO:RAIL you setup is likely to be your first ever experience of the appliance. True, you can get a simulated experience using the VMware Hands-on-Lab, but that does not perfectly match the real deal. As ever, simulation is never the same as the real experience (no sniggering at the back there, boy!). ESXi is pre-installed at the factory. So you might ask what kind of TCP-IP configuration is in place by default. The answer is nothing. The default management network (vmk0 on Standard Switch vSwitch0) is set to use a DHCP. If there is no DHCP on the default (V)LAN for the management network the ESXi host will default to using a “Link Local” address for IPv4 based on the 169.254.y.z format. It should also receive an IPv6 “Link Local” address at the same time.

Q. Do you need to set a static IP address of any type for the ESXi host before progressing to the EVO:RAIL Configuration UI?

A. No.

In fact doing so is likely to complicate matters, and could indeed cause problems. The EVO:RAIL Configuration will automagically create a new “Management Network” using a static IP address you provide the EVO:RAIL configuration UI. At the same time it renames the old “Management Network” to be called the “Marvin Management” network.


Note: This screengrab could be more realistic – its taken from the HOL, and shows a 192.168.14 address for “Marvin Management”. In the ‘real’ physical world this address likely to be either 169.254.x.y OR an IP Address from a DHCP Server 

The main thing I would say is that if people are experiencing network problems it’s very tempting to “temporarily” assign a manual IP to the each of the EVO:RAIL nodes to carry out basic ping tests. This isn’t a good idea because

a.) It can create more problems than it resolves especially if the issue resides at a physical level – and

b.) The EVO:RAIL uses auto-IP and these auto-IP address should be used to the same diagnostic tests anyway. So what kind of validation of the network could be done before you even start to touch the EVO:RAIL Configuration UI?

Test1. Ping between EVO:RAIL Nodes:

Well, first you could get a BMC connection going to each of the EVO:RAIL Nodes. This would at least show you the ESXi “Direct Console User Interface” (DCUI). What you should see is that the yet to be configured ESXi host would have an auto-IP address for IPv4/IPv6. If a DHCP server is present on the network this is likely to be a valid IP address for that network – the IPv6 address is likely to be auto-IP assigned address unless you’re so ahead of the curve that you have IPv6 DHCP scopes listening on your network.


By default SSH is NOT enabled in VMware ESXi if you want to do tests such as a ping using these auto-IP addresses. So you have two options. You can use the DCUI with a BMC connection and use Alt+F1 to start an interactive command prompt. Then login as ‘root’ with the EVO:RAIL’s default password of “Passw0rd!”. To do that you need to enable the ESXi Shell


Alternatively, you could give your workstation a valid IP address and use the vSphere Client to connect to each of the nodes, and temporarily enable SSH. Remember SSH can be temporarily enabled using the DCUI – I would recommend turning off SSH once you’re done, as this is best practice generally – plus it also creates unhelpful health alarms in vCenter and in the EVO:RAIL Management UI. This is why I prefer to do this via the DCUI Console.

From there you could carry out basic tests such as pinging each of the remaining 4-nodes in the chassis. This at least tests basic connectivity and ensures that the four nodes can communicate with each other.


  • If this fails it’s likely one of the nodes isn’t plugged in properly at a physical level…
  • Or that one of the physical ports is exclusively set for a particular VLAN and isn’t available to the default management (V)LAN.

Test2. Ping from vCenter to each of the EVO:RAIL Nodes:

Another test you could do is to validate if EVO:RAIL’s built-in vCenter can communicate to the EVO:RAIL nodes. Unlike the EVO:RAIL nodes in the appliance, the vCenter Server Appliance defaults to using an static IPv4 and dynamic IPv6 IP address assigned auto-IP. There is no need to re-IP the vCSA from its default IP address of The EVO:RAIL Configuration engine can automatically re-IP the vCenter (if required).

Given that the nodes in the EVO:RAIL are unlikely to have an IPv4 address that can reside in the same range – the best way to do a ping test here is to use the automatically assigned IPv6 address to vCSA. This can be done by first using the vSphere Client to access node01, and then using a remote console to get a window on the vCenter Server Appliance.

You can log into the vCenter using ‘root’ and the default password set by the QEP at the factory that is ‘Passw0rd!’. Using the “ip” command you can retrieve the IPv6 IP address, and then use this to ping the vCSA from one of the EVO:RAIL nodes.


This should reproduce replies from each of the EVO:RAIL nodes. Once again, this should result in valid responses. In EVO:RAIL each of the four nodes resides on the same network as the vCenter Appliance by default. If not then the configuration and validation would fail. Again if there is a failed response this is likely to indicate:

  • One of the nodes isn’t plugged in properly at a physical level…
  • One of the physical ports is exclusively set for a particular VLAN and isn’t available to the default management (V)LAN.

Test3: Multicast on the Management Network

One of the requirements for EVO:RAIL is support for IPv4 and IPv6 on the Management Network. You can run IPv6 ping tests using a multicast IP address to see what responses come back on the network. This test might not work if support for this functionality has been turned off by whoever manages the switch. It’s worth mentioning that ALL nodes with a valid IPv6 Multicast address could respond, so you would see replies not just from the EVO:RAIL nodes, the vCSA and Log Insight, but other IPv6 enabled devices too such as a router, switch IP address and others.

In this case ping -6 –I vmk0 ff02::1 is used on node01 in the EVO:RAIL.


In the screen grab b6e=node01, b72=node02, b76=node03, b7a=node04, 14d7=vCenter. There is a response from 1b64 and I believe this my router

  • If this fails it’s likely one of the nodes isn’t plugged in properly at a physical level…
  • One of the physical ports is exclusively set for a particular VLAN and isn’t available to the default (V)LAN.
  • Multicast for IPv4/6 isn’t enabled on this network
  • …or this test doesn’t work because the functionality required for it on the physical switch has been turned off…

Another way to test for multicast functionality is using the tcpdump utility on an ESXi host. This allows you to listen for VMware Loudmouth traffic. As you might remember “VMware Loudmouth” is an implementation of Zero Network Configuration where systems can communicate to each other without the need for IP address that assigned to that subnet (such as 192, 172, 10) and instead use multicast network as the way to communicate. VMware Loudmouth listens on the UDP Port of 5353. The exact command is:

tcpdump-uw –i vmk0 -s0 -t -c 20 udp port 5353

Note: -i (interface), -s0 (entire packet), -t ( (no timestamp) –c (frames to capture)


Note: This is difficult image to grab. But click it – it will get bigger and be easier to see…!


In this article I’ve focused on the multicast requirement for management traffic, but of course, VSAN has similar requirements too. If after EVO:RAIL has completed its work, and you have issues with VSAN – you might be interested in a blogpost from the VSAN team that shows how to do diagnostic checks for VSAN multicast networking:

If all is right in the world the EVO:RAIL appliance should take 15 minutes or less to setup and configure. However, like any technology, it has pre-requisites especially around the networking. What I’ve tried to do in the blogpost is peel back the curtain to expose those pre-requisites and offer up some practical steps that can be used to validate that networking. I suspect over time that EVO:RAIL will do even more validation of pre-requisites than it already does today. I also suspect that customers who are keen to use EVO:RAIL across their environment will iron out networking wrinkles in the PoC phases such that when they need to setup a new location the physical switch is already pre-configured with these defaults to ensure that 15 minute experience. What I would also love to see is our QEP helping customers with this – so alongside bundling the appliance with their own value-add, they would also bundle the networking with EVO:RAIL as well. Many QEP are doing that right now, for instance if you visit the SuperMicro page for EVO:RAIL they have links to their switches as well. Of course, we’d always want to leave it the customer where they source those switches from…


Posted by on March 10, 2015 in EVO:RAIL

Comments Off

New Year: New Product Page: New Guides

Well, it’s March the 4th I’m not sure whether it still counts as the “new” year – are we properly into the year its lost its new-ness? Anyway, the new March has ushered in a new product landing page for EVO:RAIL. It’s been redesigned by our new PMM (that’s Product Marketing Manager for those who have abbreviations). And I think its clearer to navigate around and there more things on the main LZ that should stop people going round and round in circles looking for stuff.

We have a new look and improved data sheet (for people who get off on data sheets), of more interest to use technically minded are two new docs on the site called “Network User Guide” and “Setup and Network Configuration Cards”. These have been written by my colleague, Judy Snow who is based in Palo Alto. Judy is force to be reckoned with as her attention and puts my slapdash blogging to shame – riddled as it is with spelling and grammar errors – and the odd split infinitive…

Anyway, you will find the new docs in the “Resource” tab…

Screen Shot 2015-03-04 at 11.05.07

The Setup and Network Configuration Cards are essentially checklist – The first is very high-level and covers the hardware and software requirements for EVO:RAIL. I imagine many of our SEs and Qualified EVO:RAIL SE will use these with PoC engagements – as way of confirming that everything is in place before arriving on site. The second check list (within the same doc) is a Table that allows you to collect all the host, IP and settings data that is used to complete the configuration of EVO:RAIL. You’d be surprised how often when you ask folks what IP address network pool for management – how someone has to go off to consult a spreadsheet that was last edited in 2007 (shocking, wrong on some many levels – but it happens).

The other document is even of greater interest is called the Network User Guide. It incorporates the Setup and Network Configuration Cards (for ease of printing) but then goes on to discuss the networking requirements. This covers scenarios and requirement such as:

  • Multiple Switch and ISL configuration
  • IPv4/6 Multicast settings
  • VLAN including configuring EVO:RAIL to use VLAN Tagging for the management network

For the most part so long as you get your networking ducks in a row, your experience will be fine. And the requirements are well-known and well documented. Of course, our industry is riddled with folks who never RTFM. As you might recall that was the old site name of my blog. In the next blog about EVO:RAIL I will be investigating these prerequisites and talk about diagnostic tests you can do in case you encounter issues by not configuring the network correctly.

Finally (some blowing of my own trumpet here), the resource page also links to the all-new EVO:RAIL Hands-on-Lab which debuted at PEX – which now shows adding a 2nd appliance, some patch mangement and how to replace a failed node inside the EVO:RAIL.


Posted by on March 4, 2015 in EVO:RAIL

Comments Off

Vote! Vote! Vote! Top vBlog 2015

Screen Shot 2015-02-18 at 15.21.57

Here’s a quick little post to remind you to vote for you top bloggers for 2015 – and of course, I’m hoping I will make the cut again this year. This years Top vBlog is sponsored by Infinio – and voting opened today and will be open until the end for Feb. The Top vBlog voting is always interesting as it makes me look back on my previous years output. Last year I carried on with my “Back To Basics” series – I’m about half-way through the content with more to come. By far the most popular post on has been the post about the configuration of vCenter after setting up Windows/Server Appliance – that’s had 71K page views alone, with its companion video pushing toward 2.5K views. It just goes to show that core vSphere knowledge is still a must, and there’s still a demand for that type of help online. I will have some exciting news around the “Back To Basics” series to announce shortly.

Last year I re-introduced my “Chinwag” podcast after about 18 month gap. This time around I’ve aimed for less aggressive cadence of releases. When I was independent I had more free time for the podcast and I was doing a weekly show. After joining VMware I had to manage my time more efficiently, and in the end the podcast was just put on the back-burner whilst I made the transition from independent to vendor aligned. August this year, will be third year at VMware, and my feet are firmly under the table. So I felt it was time for me to bring back the podcast albeit on a bi-weekly (or there abouts) schedule. The chinwag has started reloaded with some big names – but its my intention for the chinwag podcast to keep its roots in being a QA session with customers and folks from the community rather than it being a plaform for recognised vRockstars. I just needed some vRockstar love to get its back in front of people’s eyeballs…

August, 2014 saw me switch to into the all-new EVO:RAIL product. It’s my first time at VMware with brand new technology and working with all the different types of roles you get within a software company – the Exec, Product Manager, Marketing Manager, Development Team and Logistics and Documentation – and of course, our Qualified EVO:RAIL Partners. I’m 6 months into the role and loving every moment of it – even more so because all of its different and new experience for me. So what I’m trying to do with EVO:RAIL is get beyond marketing, technical content out there via my blog. That’s been quite task as I’ve been simultaneously learning the product whilst learning to deliver on the new role.

I remain committed to both promoting and contributing to the VMware User Group (VMUG) program. Last year I tried to attend and contribute to every VMUG in the Untied Kingdom. I didn’t quite make it to everyone as sometimes business and customers must come first – I think I was at 99% of the events. VMUG is very close to my heart because like so many others, it was the first public platform that stood upon and presented at. In effort to spread the benefits of the VMUG program I helped launch our feed4ward initiative, that offers an informal mentoring service to VMUG community members who are considering on getting up on the stage. I’ve used the blog to promote VMUGs, Feed4ward as well as other benefits and programs such as the all-new EVALExperience.

To vote for my blog and my contribution to the VMware Community this year – please visit the voting site here:


Posted by on March 3, 2015 in Announcements

Comments Off

Post Adverts – BitDefender

One of my sponsors, BitDefender recently made a request to use “in blogpost” adverts on my site. I’ve chosen to accept their offer – so from now until further notice you will find at the end of each post, and advert like the one above. I did a couple of experiments, and even I found an advert being present in the middle of the post, very distracting – especially when I was blogging about partners recent developments around EVO:RAIL. I’ve taken up BitDefender’s offer – but you will find their ad appears at the end of blogposts – and only when you click the full title of the blogpost – usually from some sort of google referral. I hope my readers are okay with this and don’t find the ads too intrusive. Let me know via twitter if you find them too much.

Cheers, Mike


Posted by on February 27, 2015 in Announcements

Comments Off

Chinwag Reloaded with John Troyer (@jtroyer)


I’m always interested in meeting new comers to our community, and this guy is no exception. His name is John Troyer, and you should really check him out – because I think he’s really on track to become a big name in the future.

Okay, joking aside – of course, everyone knows who John Troyer is – he’s been at the epicentre of all things VMware since forever. Last year, John decided to go it alone and leave VMware to start his own venture – it’s called @TechReckoning. John also runs the @Geek_Whisperers podcast along with his co-hosts Matthew Brender @mjbrender and Amy Lewis @CommsNinja

I caught up with John at his home on coastal California in January. I was over in the Bay Area for the first for my two visits per quarter to keep in touch with colleagues on the EVO:RAIL Team. The light was fading a little as John filled me in on life post-VMware – why he felt the need to move along, and his new project is all about.



Posted by on February 24, 2015 in Chinwag

Comments Off

Sponsor Update: Welcome to SolarWinds

This is just a quick blogpost to welcome my new sponsor – SolarWinds. In case you don’t know SolarWinds produces a number of management and monitoring technologies, across whole host of different areas. In terms of VMware vSphere they have a technology called “Virtualization Manager” which produces performance and diagnostics – as well as spotting rogue VMs, orphaned VMs, Wasted Licenses and Idle Licenses.



Posted by on February 23, 2015 in Announcements

Comments Off

EVALExperience Update

Screen Shot 2015-02-18 at 16.06.49

A few short weeks ago VMware and the VMUG announced the EVALExperience. In case you don’t know this is something myself and others in the community have been crying out for, for sometime. It’s essentially a beyond 60-eval subscription where you can get your sticky paws on VMware software for 365-day period. Those who subscribed early might have been caught out by the subscription containing a small number of CPU sockets of VMware ESXI hosts. Initially, the licenses only allowed for two hosts – of course, that creates issues of technologies where 3 VMware ESXi hosts are a minimum. I’m pleased to say this was noticed very quickly, and fixed even quicker-er (is that word?). So if you have the old keys – you need update your licenses via the subscription portal in order to fulfil the full allocation.



Posted by on February 18, 2015 in VMUG

Comments Off

HP ConvergedSystem 200-HC EVO:RAIL Appliance – Available Now!

One of the things I was so pleased about at VMworld Barcelona last year – was when HP and VMware announced that HP was joining the EVO:RAIL program. One of the most consistent messages I was getting from customers was “do you have HP model”. HP is a very dominant player in EMEA, and globally really – and there are many loyal customers to HP that really wouldn’t like to see anything but HP badge in their racks. Anyway, HP have announced this week the GA of their EVO:RAIL. There are one or two things to clear up around product names and offerings.

HP have opted to use the product name “HP ConvergedSystem 200-HC” for two quite distinct offerings. The EVO:RAIL offering and separate and different “ConvergedSystem 200-HC StorVirtual VSA” offering which doesn’t use EVO:RAIL and VMware VSAN, and instead uses the HP StorVirtual Virtual Storage Appliance. The form-factor is the same look/feel; – a 2U chassis with 4-nodes, but the specifications (CPU/Memory/Disk) are different. The “HP ConvergedSystem 200-HC EVO:RAIL” Appliance product page is located here:is located at this page here:


From the front!!!


From behind!!!

Like many of our other Qualified EVO:RAIL Partners (QEP), HP offer their EVO:RAIL in two flavours based on either RJ-45 or SFP+ connectivity – their part numbers are L3W30A and L3W31A respectively. The tech specs are the following:

Four HP ProLiant SL210t Gen8 Servers, each with:
• Two Intel® E5-2620 v2 six-core CPUs
• 192 GB memory
• One SAS 300 GB 10k rpm drive ESXi boot device
• Three SAS 1.2 TB 10k rpm drive for the VMware Virtual SAN data store
• One 400 GB MLC enterprise-grade SSD for read/write cache
• One H220 host bus adapter (HBA) pass-through controller
• Two 10GbE NIC ports, configured for SFP+ connections
• One 1GbE IPMI port for remote (out-of-band) management


Posted by on February 13, 2015 in EVO:RAIL

Comments Off