I Don’t Believe I.T. iPhoto Experience – Because Delete Doesn’t mean Delete…

22 Oct

One of the things I didn’t get across in my previous post about “I Don’t Believe IT” was those capital letters. It’s a bit of bad pun – “I don’t believe Information Technology”. Basically, this series is homage to my every increasing “Grumpy Old Man” syndrome about technology. One of the slightly depressing things about being IT is the ludicrious opitimism that abounds the area of technology. It’s like people will think that Technology will always ride into town and save the day. I don’t really see it that way.

Don’t get me wrong I’m internal optimist by prediclition – but what agreeves me is the blind faith people put into technology. It seems people are all too willing to forget that we are monkey’s with monkey brains, and human flaws are often revealed in flawed technology and flawed business processes.

So anyway, this weeks “I Don’t Believe IT” concerns our friend (or enemy) Apple Mac iPhoto. I’m lazy you see and tend to use the default apps that ship with the Mac. Although somewhere between Mountain Lion and Mavericks – iPhoto stopped being free to new uses, and now you have to pay for the darn thing. Here’s the thing – when take a photo in iPhoto and send it to trash – it doesn’t actually delete it.

Screen Shot 2014-10-06 at 14.38.08

I’ve noticed that if you select an “event” in and select File, Reveal in Finder, and Original – you’ll find that the files are still cuffing there!

Screen Shot 2014-10-10 at 08.23.43

Why? WTF. If I send something to the trash, it should be deleted or least be sent to the trash can. I’ve been remiss in trying to work out WHY this happens or how to actually removed these orphaned and unwanted image files (some being anywhere from 1MG-5MB depending on the format used on my iPhone).

Things came to ahead this weekend, when I found my SSD drive was almost full. So I decided to google for iPhoto – as I thought that might be good place to try and free up some precious space. It turns out iPhoto has its own “empty the trash” option – that I’d never heard of before. It’s not suprising as its not in the main File/Edit/Photos menu bar, but under the iPhoto menu itself.

Screen Shot 2014-10-06 at 14.44.10

I wasn’t disappointed. I had 4,500 orphaned files. Emptying the very special iPhoto Trash freed up 5GB of space.

Of course, there will be those who will tell me that iPhoto a PoS, and I should be using something else. Like Windows for instance. But blow me, I assumed that when I delete files they actually deleted. It sound more like the “Trash” is more like a “Remove from Inventory” like you get in the vSphere Client(s), rather than a “Delete from Disk”.

Comments Off

Posted in IDBI


VSAN vINCEPTION: Failed to join the host in VSAN Cluster (Return Code 1)

20 Oct

As you might know vINCEPTION is my term for what many others called “Nested” virtualization. It’s the peculiar moment when you realise VMware software can eat itself – by being able to run VMware ESXi in a Virtual Machine running on top of either VMware Fusion, Workstation or even VMware ESXi itself. I’ve been experimenting with running a nested version of VSAN in my home lab, with the prime reason of wanting to be able to run my own private version of EVO:RAIL in a nested environment.

As you probably/hopefully know by now EVO:RAIL is physcial 2U appliance housing 4 independent server nodes. The EVO:RAIL is delivered by partners in a highly controlled process. So it’s not like I could just slap the binaries that make up EVO:RAIL (that I have private access too from our buildweb) on to my existing homelab servers and expect it all to work. The EVO:RAIL team have worked very hard with our Qualified Partners to ensure consistency of experience – it’s the partnership between a software vendor and hardware vendors that delivers the complete package.

Nonetheless we can, and do have EVO:RAIL running in a nested environment (with some internal tweaks) and it’s sterling bit of work by one of our developers Wit Riewrangboonya – I’m now responsible for maintaining, improving and updating our HOL – and if I’m honest I do feel very much like I’m standing on the shoulders of giants. If you have not checked out the EVO:RAIL HOL it’s over here – HOL-SDC-1428 VMware EVO:RAIL Introduction. Anyway, I wanted to go through the process of reproducing that environment on my homelab, mainly so I could absorb and understand what needed to be done to make it all work. And that’s what inspired this blogpost. It turns out the problem I was experiencing had nothing to do with EVO:RAIL. It was a VSAN issue, and specifically a mistake I had made in the configuration of the vESXI node…

I managed to get the EVO:RAIL part working beautifully. The trouble was the VSAN component was not working as expected.  I kept on getting “Failed to join the host in VSAN Cluster on my 2nd nested EVO:RAIL appliance. Not being terrifically experienced with EVO:RAIL (I’m in Week8) or VSAN (I’m into chapter 4 of Duncan & Cormac’s book) I was bit flummoxed.


I wasn’t initially sure if this was – a problem with EVO:RAIL, a VSAN networking issue (multicast and all that) or some special requirement needed in my personal lab to make it work (like some obscure VMX file entry that everyone else, but me knows about). Looking back there’s some logic here that would have prevented me barking up the wrong tree. For instance, if the first 4-nodes (01-04) successful joined and formed a VSAN cluster – then why wouldn’t nodes (05-08)? As I was working in a nested environment was concerned perhaps I was meeting the network requirements properly. This blogpost was very useful in convincing me this was NOT the case. But I’m referencing it because it’s a bloody good troubleshooting article for situations where it is indeed the network!

You could kinda understand me think it was network related – after all status messages on the host would appear to indicate this as a fact:


But this was merely symptom not a cause. The host COULD communicate with each other – but only if osfsd starts. No osfsd, no VSAN communication. That was indicated by the fact that the VSAN Service whilst enabled, had not started.


and after all the status on the VSAN cluster clearly indicated that networking was not an issue. If it was the status would state a “misconfiguration” in the network status…


As an experiment I setup the first nested EVO:RAIL appliance – and tried doing the 2nd appliance on my own as if it was just another bunch of servers – pretty much I got exactly the same error. That discounted in my mind that this issue had anything to do with EVO:RAIL Configuration engine, and that source of my problem laid elsewhere.

Of course, a resolution had been staring me in the face from way back. Whenever you get errors like this – then google is your friend. In fact (believe it or not) I would go so far to say I love really cryptic and obtuse error messages. Search on “Failed to start osfsd (return code 1)” is like to yield more specific results than some useless generica error message like “Error: An error has occurred”. This took me to this community thread which is quite old. It dates back to 6 months ago or more, and is about some of the changes to VSAN introduce at GA. I must admit I did NOT read it closely enough.

It lead me to Cormac Hogan’s VSAN Part 14 – Host Memory Requirements where I read the following:

At a minimum, it is recommended that a host has at least 6GB of memory. If you configure a host to contain the maximum number of disks (7HDDs x 5 disk groups), then we recommend that the host contains 32GB of memory.

Sure enough following this link to the online pubs page confirmed the same (not that EVER doubted the Mighty Cormac Hogan for second!)

A quick check of my vNested revealed that nodes01-04 had only 5GB of RAM assigned to them, and inexplicably I’d configured nodes05-08 with 4GB RAM. I’d failed to meet the minimum pre-reqs. Of course, you can imagine my response to this – Total FacePalm.

Well you live and don’t learn – always read the pre-reqs and RTFM before jumping in with both boots before embarking on something, especially if you deviating from the normal config.

Comments Off

Posted in EVO:RAIL


VMworld 2014 Europe – HP and HDS join the EVO:RAIL party

14 Oct



Well. The good news is finally out – I’m sorry please to hear that HP and HDS have joined the EVO:RAIL program. If you are at the event this week, we have the HP appliance in our booth… I say booth it actually a Resturant that we have taken over.

I’ll be at the booth all this week and occasionally down at the hang space helping out at the challenge…

Comments Off

Posted in EVO:RAIL


I Don’t Believe IT: HP Printer is out of toner…

10 Oct

Pop-up messages. Arghhhh. If your anything like me when your using a computer (regardless of OS) the incessant harassment of pop-up messages goes beyond belief. One thing I’ve sometimes thought is how little software vendors think about the real usage of a computer from the end-users perspective. It seems entirely reasonable to have helpful pop-up messages. The trouble is you may have 20-30-40-50 programs on your computer, not including the other bits of chatty software such as your AV, and pop-ups from helpful applications like Facebook and Twitter and your email – and once they are all being “helpful” you wind-up shouting – **** OFF, and LEAVE ME ALONE!

One word I’ve coined for this sort of intrusion is “Nagware” (it’s actually a term used to describe free software that nags you to pay – but for me the term can be extended to all software that bugs the living **** out of you.

For me a classic example of this week was an experience my beautiful wife (she told me to write that) who I adore tremendously (she told me to write too) when she was away from her computer – she was only away for 10mins…. Apparently, we need new toner on HP Printer. That’s another one of IT great IDBI – the whole rip off surrounding printers, cartridges and being told your out of ink or toner.

I have an idea for a start-up called “NagAway” which blocks all these pop-up messages. I bet I’d make an absolute fortune!


Comments Off

Posted in IDBI


VMworld EU 2014: Take the EVO:RAIL Challenge!

09 Oct


Yes, I know it sounds like the Pepsi challenge!

Here’s what it is. In the Hangspace at VMworld there will be two EVO:RAILs and to entrants in the challenge will race against each other to setup an EVO:RAIL appliance. The fastest individual will win a coveted golden ticket to VMworld 2015. Space is VERY limited as only a group of 30 attendees to participate in the EVO: RAIL Challenge. If selected, you’ll be given the chance to compete in a race against time to build an EVO:RAIL Appliance. The two challenge participants with the overall best times will compete in the Final Challenge, which will take place on Thursday, 16 October at 12:00 in the Hang Space, Hall 7.0.

So if you want to take part. You need to be quick. You enter by completing this survey

Note: ** Pass valid for VMworld 2015 US or Europe Conference Pass only, all travel, hotel and all other expenses are the responsibility of the winner.

Comments Off

Posted in EVO:RAIL, VMworld


vSoup Mike goes EVO #50

09 Oct

Screen Shot 2014-10-09 at 10.08.41

The other week I had a chance to speak to the vSoup guys on their podcast (now up to episode 50!). In case you don’t know your venerable hosts are Ed Czerwin, Chris Dearden, and Christian Mohn. In case you don’t know – Chris has the dubious honour of being the very first person on my old podcast – The Chinwag…

Anyway, in this podcast the boys grill me about all things EVO:RAIL:


Comments Off

Posted in EVO:RAIL


EVO:RAIL – From Server Consolidation to Hyper Consolidation

06 Oct


A couple of weeks ago I announced my new role in the EVO:RAIL Team – the fact is I’ve been working with the team since just before VMworld US, but it wasn’t really until last week when I had finished up my work with my previous team, and the HR wheels had turned, that I was able to tell folks publically. Not to fear, my new role means I’m even more committed to the community – and that includes projects like Feed4ward and speaking at VMUGs generally. I hope to pick up a schedule of events for next year, and that includes coming to the US to speak at the big VMUG User Conferences too.

So I’ve been thinking about the topic of hyper-convergence generally. It’s popular in the US to talk about the ‘journey’ so I want this article, and the articles that will follow it, to be the basis of my own thinking around the subject – call it my own Personal IP. At some stage I need to write my own VMUG style presentation about hyper-convergence. Folks who know me will know I hardly ever use the standard corporate decks when talking about VMware technologies. Right now I am – but going forward I want to put my own personal “Mike Laverick” stamp on that. My hope is that this series of blog-posts will help me develop my own stance and perspective. So here goes…

 EVO:RAIL. Already there are innumerable spellings (evo:rail, evo-rail, EVO-rail, Evo:Rail and so on), and it doesn’t help that you can’t actually pronounce “: “. So for the record it’s meant to be EVO:RAIL in capitals. One of the funny things about the name is if said too quickly it could sound like EVIL:RAIL, prompting a typical Dr EVIL:RAIL meme:



I’ve been using this to good effect when anyone asks how much EVO:RAIL costs. The reality is VMware doesn’t set the price. The six different Qualified EVO:RAIL Partners (QEPs) do. VMware licenses the EVO:RAIL engine and associated software (vSphere, VSAN, LogInsight) to the partner and the partner sells the final product to the customer – one throat to choke, as my US friends are fond of saying.

The other amusing thing is what happens if you type EVO:RAIL into Google Images. Along side pictures of servers and logos, you’ll also see pictures like this:


EVO:RAIL – Your weapon of choice for the datacenter!

It’s heartening to see an interest in VMware EVO:RAIL is starting to work its way through the Google Images algorithm. A couple of weeks ago there were just photos of semi-automatic weapons! Of course, my lame pun has been – “VMware EVO:RAIL – Your Weapon of choice for the Datacenter”.

So for the record EVO is for evolution – the natural evolution of VMware technologies to be consumed in a hyper-converged fashion. As for RAIL, well if you have ever been into a datacenter hall and racked up some gear… you get the picture…

Hyper-Consolidation – How did we get here?

Although I quite like the term hyper-convergence, I can see already a parlour game of “define your terms” is already cranking up. Sometimes that can be dangerous because if a vendor defines hyper-convergence they are likely to use a definition that presents their product in the best light – it’s a phenomena that customers will be have to cautious about. Personally, I find this game of defining terms slightly pedantic. But sadly, its unavoidable – I think the simplest definition is a server (call it an appliance if you must!) which provides both compute, networking and storage in a single bundle. It’s an approach that scales-out the environment by adding more servers in a block-by-block method with each new server joining the existing environment seamlessly.

Phew, definition done.

What interests me more is where we have come from, and how we got here as an industry and community. For me it seems only right and proper for VMware to deliver (with partners!) a hyper-converged solution, after all VMware was the company that brought server-consolidation to the masses and popularized virtualization. That’s partly the pun in my title – from server-consolidation to hyper-consolidation. For me it makes perfect sense – or the perfect EVOlution for the company that brought server consolidation to the market, to also popularise hyper-consolidation too. Incidentally, I’m not trying to coin a new term here. It’s hyper-convergence, not hyper-consolidation, right? I’m just using the term as metaphor…

Way, way back in 2003, VMware could have taken this route. They could have bought cheap commodity servers from the white box market, yanked the bezel off, and replaced with a VMware bezel and sold that to customers. Mercifully they didn’t take that route at all. They took the market route of being a software vendor, and working with partners and the channel to get VMware ESX and VMware VirtualCenter (to use its old name for a moment) as a way of popularizing virtualization. It’s an approach that helped take the company from a handful of employees with a couple of hundred customers – to being a billion dollar company with 20K+ employees, and 450K customers. So what I like about EVO:RAIL is it delivers on the appliance model for the consumption of VMware technologies, whilst at the same time staying within a tradition and business-model that has been tried and tested before.

I remember when ESXi first came out, and it became possible to boot the hypervisor from a USB/SD-Card. There was talk back then of server vendors shipping the hardware with VMware ESXi already pre-installed, in a sort of embedded format. In my mind I imagined that when the server first booted, you’d merely select which hypervisor you wanted (the best one, of course!). As far as I could tell that never really ever happened. Quite possibly because there wasn’t enough add-value for either the server vendors or customers to make it happen. The truth is installing VMware ESXi is totally trivial event – the fun starts in the post-configuration phases. That’s why I think EVO:RAIL will be successful. Looking back over the years, I’ve personally done a lot of automation. It started with simple “bash” shell scripts in ESX 2.x, and then evolved to using the UDA to install ESXi 3.x from a PXE boot environment. About the time of vSphere4 I moved away from bash shell scripting to building out environments with PowerCLI. It has literally hundreds of cmdlets and can handle not just ESXi but vCenter configuration too. I burned a lot of time building and testing these various deployment methods. Now, EVO:RAIL has come along allows me to do that in less than 15mins.

For me that doesn’t mean all that previous hard work has been for naught – after all I believe there is still legs in other model for delivering infrastucture. I still will still support those methods, but what EVO:RAIL has delivered is much more automated, standardized and simpler method of doing the same thing. As independent it always sort of irked me that VMware didn’t have a pre-package, shrink wrapped method of putting down vSphere, and it was sort of left to the community to develop its own methodology. The trouble with that is everyone has their own personal taste on how that should be done. And we all know that leads to things being not standard between organizations, and in some cases within organizations. Despite ITIL and change-management controls, configuration drift from one BU or geo to another is a reality for many organizations. I see EVO:RAIL as offering not just hyper-converge consumption model, but an opportunity to standardize – especially for companies with lots of branch offices and remote locations.

It means that VMware has empowered the partners to enter the market place rapidly, and provide to customer’s choice, bringing much needed competition to the hyper-converged space where previously there was much narrower range of options. It’s also a further step down the road to hyper-convergence becoming as mainstream as virtualization. Once companies such as VMware, Dell, Fujitsu, EMC, and SuperMicro join the party, hyper-convergence will come to the attention of everyone in the industry – from the guy who racks ‘n’ stacks to the management team.

Comments Off

Posted in EVO:RAIL


vNews Podcast – Talking EVO:RAIL

01 Oct

This week it was my pleasure to take part in a Q/A with the guys who run the vNews Podcast. You might recall before I joined VMware in 2012, I used to co-host the podcast with Stu McHugh. I moved on because I felt it should be run by folks who were vendor-neutral.

So this episode is all about EVO:RAIL – and guys asked some really great questions – really great one’s I’d not had before which is always more interesting.

You can subscribe/listen to the podcast from from iTunes by searching for vNews or download from

or you can listen directly from here

In the podcast I mention a Spiceworks article by Trevor Potts and also the .JSON configuration file for the EVO:RAIL. That’s referrence over here:



Comments Off

Posted in EVO:RAIL, Podcasts


Apply for my old job…

26 Sep

Screen Shot 2014-09-26 at 12.04.37

As you might know I changed roles to move into the EVO:RAIL team in what’s essentially a tech-marketing/evangelist position. With that happening my old role is now open, and VMware is welcoming external applications from those with the right skills and background. Don’t be put off by the referrences to Palo Alto, if you’d rather stay where you are that’s okay. As for relocation – I don’t have a clue. The link to the job req is below… Please read the pre-requisites closely…

Competitive Marketing Engineer

Comments Off

Posted in Announcements


Comments are off…

26 Sep

702947781_1395366929I’ve been think of doing this for a while – but today I decided to globally turn off comments on my blog. If was Scott Lowe’s decision to do this on Tuesday, that finally convinced me it was the right thing to do. Like Scott I have all the tools designed to stop comment spam – and in the past these have worked pretty well in keeping comment spam down to minimum. These tools just aren’t working any longer.

So its with a heavy heart that I decided to do this. Blogging is meant to be two way street, and all bloggers will tell you they love to get positive and constructive feedback on what they post, and as well as forthright and well argued discussions. However, once most of your traffic is 99% spam, and 1% geniune responses something has to give.

I will review this decision every so often, and if I find the old tools improve or better anti-spam comment tools become available. I will turn comments back on…

Comments Off

Posted in Announcements