Cupfighter.net

Powering the cloud. Multi marketing of course, but what is happening in the storage world? What does it mean for mission critical environments? These are the questions I am hoping to get answered today and tomorrow. Currently three sessions done. 1. Introduction to Data protection by Chriss Sop, 2. Optimizing storage in a cloudy, virtualized world by The 451 Group and 3. Enterprise Tiered Storage by John Locky.

First two sessions were somewhat low quality from a contect perspective. Too basic from on technology and on new innovations. Even for me as a non engineer. The difference between full backup, incrementals and differentials is not the thing we came here for. Although i must say that merging incrementals on the back end to always have full backups available sounds interesting. Curious to see this working in real life. How transprrent will that be? Lets ask Commvault later today. And if i can find them Quest as well. Would be nice to learn a bit on automated restore testing as well. Guaranteeing back ups remains an issue. Especially on tapes.

When i get answers, you’ll probably read more about it on cf.net or twitter.

Wrong Way ... Way Wrong a CC NC SA image from Bob.Fornal's Flickr stream

Virtualization aims to save money, make things simple and quick to deploy. Saving money and quick deployment are arch enemies of security

Virtualization products require security on the hypervisor level. Being able to hop from one virtual machine to another is not acceptable. Also there are a lot of products that focus on the security in the virtual machines, but virtualized infrastructure are complex by nature.

Relative lame bugs like XSS can be a big deal in virtualization infrastructures

Claudio demonstrates that live on stage, by exploiting a XSS bug in VMWare vCenter which took 1.5 years to patch.

Claudio showed us how an unprivileged user on the vCenter machine able to read a logfile contain the administrator SOAP session ID. Using this ID and Vasto administrator privileges where obtained. Until the last patch read-only access to vCenter meant that the user could take over the virtual infrastructure using standard tools.

Next attack demonstrated is against an Oracle virtual machine. Using standard “lame” exploits Claudio was able to hope from the application level administrator to the system root account.

So there are still some very simple vulnerabilities in this software.

Virtualization software is broken today, and we have to treat it accordingly. We have to make people aware that it is broken.

Virtualization infrastructures should be setup in such a way that a XSS in the management layer cannot lead to a disaster.

Read more…

ESXi over NFS works just great!

But what if you have an issue with NFS and you need a network dump? 

In ESXi tipically you don’t have a local datastore where you can write files from the network dump and your datastore over NFS is not availabe!

Before running into the Data Centre and stick a USB disk or even better a SCSI disk you might want to try this.

One trick I used that worked out pretty well for me, with a little help of my a linux machine, is to send the tcpdump output to a FIFO and from a remote host (might be a VM in a different ESXi host) over SSH cat the FIFO to a local file.

How To:
On the ESXi host logon via SSH as root and create a named pipe:

root@yourESXihost# mkfifo /tmp/pipe.dmp

and from a remote linux machine launch the following:

you@yourlinuxhost > ssh root@youresxihost "cat /tmp/pipe.dmp" > capture-for-wireshark.cap

Now from a new ssh session to ESXi as root lauch

root@yourESXihost# tcpdump-uw -n -s 1524 -i vmk# -w /tmp/pipe.dmp

OR even better from the remote machine:

you@yourlinuxhost > ssh root@youresxihost "tcpdump-uw -n -s 1524 -i vmk# -w /tmp/pipe.dmp"
(replace the # with the proper vmk port number)

Reproduce your issue and when you finished just hit  “Cotrol+C” to stop the network dump and the cat.
Now you can open your file directly in wireshark (that’s what I use at least!)

This little trick of course can be used to troubleshoot network problems in a VM as well, dumping the traffic from a VMK# nic for the entire dvPortGroup. You just need to make sure that the the VM’s vNIC and the vmk# nic are connected to the same dvPortGroup and you must remember to allow promiscuous mode (not allowed by default)

Good Luck!

Please note: your network can be very chatty so the file can grow very fast and/or your ESXi host might not like the tcpdump so use it at your own risk and only if you really know what you are doing!

For troubleshooting purposes sometimes it might be usefull to perform a tcpdump on the ESXi host instead of from a VM.

After removing the security setting in the dvPortGroup explicitly allowing promiscous mode you can connect a vmkernle port to the dvPortGroup and perform a tcpdump via the commandline

The help for tcpdump-uw is the following:


~ # tcpdump-uw --help
tcpdump-uw version 4.0.0
libpcap version 1.0.0
Usage: tcpdump-uw [-aAdDefIKlLnNOpqRStuUvxX] [ -B size ] [ -c count ]
                [ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds ]
                [ -i interface ] [ -M secret ] [ -r file ]
                [ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ]
                [ -y datalinktype ] [ -z command ] [ -Z user ]
                [ expression ]

Control+C to interrupt the capture

Like over 6000 other people, myself and two other colleagues went to VMworld 2010 two weeks ago, in Copenhagen. This event, one of the largest vendor organized events in the world, is three days packed full of all things VMware (and virtualization) related. There are a lot of exellent posts out there already offering specifics on the subjects presented. This post is not meant to replicate that, but offer some insight into my experience there. Three very important aspects outside of the normal presentations for me were the Solutions Exchange, the VMware labs and of course networking (social, not the hardware kind).

Solutions Exchange

First of all, let me start with the Solutions Exchange. With over 113 vendors showing off their wares, iPad give-aways everywhere, booth-babes distracting you and booth staff trying to get your attention this can be very overwhelming at first. However, you are there for three days. Because of this, you have the time to walk around, get to know who is where and pick your interests. What also helps a lot is the fact you have 30 minutes between sessions. Going from one session to another takes maximum 5 minutes, meaning you have 25 minutes to spare. While that is a limited time, the vendors know this as well and keep very short presentations on a plethora of subjects related to their product. Added benefit is you don’t have to sit through an hour of slides and sales talk, they get right to the point.
In the end, I got a lot of value out of the Solutions Exchange. Of course I had some companies I knew I wanted to visit beforehand (Veeam being a good example). A lot of others however I might not have visited per se (like the VCE alliance with their awesome vBlocks) or hadn’t even heard of (like Nimsoft with their monitoring solution).

VMware labs

Another aspect I was very anxious to try were the VMware labs. The labs offer you the unique opportunity of trying out any and all VMware products in a pre-deployed environment. All labs are virtualized in a big cloud. Even the ESX(i) servers were virtualized and then deployed on demand. This is ‘eating your own dogfood’ in the purest form. The disadvantage here was the performance: my first lab was almost unworkable the first ten minutes. After that time, it was still slow but workable. With a couple of thousand VM’s running off of a cloud in the USA this can be expected though I guess.
The physical lab setup is quite nice: it is a dual-screen setup, where the actual labs are on your left screen. The right screen will have the lab instructions, any remarks and tips and this will allow you to not keep having to switch back and forth between screens. For those with a dual-screen setup, you know what I mean (personally, I’ve arrived at three screens which is even better ).
Some labs are free-form. I got to try CapacityIQ, Chargeback and Orchestrator in the Sandbox for example. The sandbox was a complete vSphere environment, with all the bells and whistles installed and configured. Since I can’t spend the time on doing a Proof of Concept per se, being able to play around and mess with the config proved to be invaluable for me. The lab instructions here consisted of some pointers but the idea really was to mess around and try out anything you wanted. Again, this was not some kind of demo but an actual environment created just for you.
Other labs are more strict (using strict very loose, you can still mess things up if you want). I did the installation lab for VMware View 4.5, to see what is was all about. The lab instructions will guide you through the install step by step, clearly describing the actions needed to complete. Even though I was unfamiliar with this specific product and how to set this up, I felt very comfortable going through this and ended up with a working View setup in under an hour!

General networking

This is the biggest VMware related event in Europe. The who’s who in virtualization land is here. Most of the top Dutch bloggers (like Duncan Epping, Eric Sloof, Gabrie van Zanten, Arne Fokkema, Arnim van Lieshout, Joep Piscaer, etc etc etc), a lot of the 3rd level guys from vendors you normally don’t get to talk to a lot and of course big names in VMware like Paul Maritz and Steve Herrod. If there is any chance to shake hands and meet these guys (and girls) in person, it’s here! This is one of the fields where I would take a different approach next time I visit VMworld though. While I had my sessions and labs planned out, my networking was a lot of free format. This works for meeting the vendor people (they aren’t going anywhere), especially the bloggers are a lot harder to shake hands with. It’s not that they have a security cordon around them, but these guys are popular! I hate to intrude on an ongoing conversation but in the end this meant I did not get to shake as many hands as i had hoped for. Next year I’ll be making sure I have at least pinged some of these guys beforehand, so they expect me to walk over and shake hands. Makes it less of an intrusion and more of a friendly talk.

Sessions

In the end I still spent most of my time visiting the sessions. As I mentioned earlier there are already excellent posts online on all of them (most Dutch bloggers were posting real-time, check Gabes Virtual World or NT Pro for example). I will do a quick summary of what I visited and hope you get some value out of that.

• Keynote
  The keynote was done by Paul Maritz ad Steve Herrod. These guys know how to do a presentation but there was of course a lot of marketing. What it boils down to: Cloud Computing is not just a marketing term. Their goal is to make everything you could consider a resource (storage, network, processing power, applications, etc) available as flexible as possible. This allows you, the administrator, to offer IT as a service to the end-user. It makes his experience transparent and gives him the flexibility he requires. Sorry, can’t explain it any better with less buzzwords, feel free to correct me in the comments .
• Troubleshooting using esxtop
  esxtop has always been a powerful tool for troubleshooting your problems on an ESX server (and of course resxtop for ESXi). With the release of 4.1 new counters are shown (note: values were already recorded, simply not shown yet). Some of the most importent ones are LAT_C and LAT_M, showing CPU and memory contention. Don’t forget: you can export esxtop into csv and analyze further with perfmon or esxplot (open source, available at VMware labs).
• VMware View 4.5 Technical Overview
  VMware View 4.5 offers some major improvements over the previous release. Most important are full support for Windows 7 32- and 64-bit, PCoIP authentication with smart card and offline support. The last one is a major performance improvement for road warriors and now allows them to work without a connection.
• Intelligent HA: Application awareness with VMware HA
  VMware teamed up with Symantec to bring you this one. It boils down to application clustering support for applications with the intelligent layer of VM underneath. It is based on the Symantec clustering product and the years of experience they have there.
• Planning and designing a HA infrastructure
  This session, given by the awesome Duncan Epping, focused on all those little (and big) things you need to keep in mind when using VMware HA. If you regularly read his blog you will know most already. Some highlights: keep in mind you can only have 5 primary nodes in a cluster, design for management network redundancy (a VM network is not a HA network!), change your failure detection values when adding a secondary Service Console. Very important improvements in 4.1 are the fact you are now able to see the master status of nodes and in the vCenter client you have a Cluster Operational Status. Furthermore, in 4.1 U2 detection for lost lock and thus partial split-brain has been built in. Duncan also discussed an often missed ‘Golden Nugget’: vm monitoring. This allows you to quickly recover even from a BSOD and makes a screenshot for you to ease troubleshooting. On the roadmap: storage hearbeating, no more DNS dependency and Improved isolation response actions.
• Transitioning to ESXi
  We’ve all known this was coming and 4.1 is the last release with a full-blown Service Console ESX. ESX is dead, long live ESXi! Whereas with 3.5 and, to a lesser extend, 4.0 not everything could be done with ESXi, those days are gone. With the focus on agent-less management via API, CIM and PowerCLI a powerful framework now exists for ESXi management, more powerful than ESX with the agents in the COS. Worth noting: Tech Support Mode (TSH) local and via SSH is now fully supported, with all commands sent to syslog. You can still disable it and go a couple of steps further with lockdown mode.
• 10 best free tools for vSphere management
  Every VMware administrator worth his salt should have these installed! For a great list check Kendrick Coleman’s blog, but the ones mentioned were: VMware Guest Console, Veeam FastSCP, Trilead VM explorer, XtraVirt vSphere client RDP Plugin, vEcoshell with the Community Powerpack, VKernel CapacityVIEW, vSphere mini monitor, RVtools, vFoglight QuickView and Xangati for ESX.
• Tech Preview Storage DRS
  Honestly, this was one of the coolest sessions I attended mostly because of the subject. While CPU and Memory have been a resource where your VMware cluster is able to spread the load, storage is coming too. This is a complex challenge (as shown in the presentation) but in the end this will add more awesomeness to your VMware cloud.
• PowerCLI for administrators
  This session, given by Alan Renouf and Luc Dekens (those guys are awesome) was actually not new to me. I’ve heard Alan spread the word on the awesomeness of PowerCLI before so I didn’t need convincing. What it boils down to: use PowerCLI to make your life easier as an admin. Like the tagline on Alan’s blog says: “Everything is poshable!”
• SRM using NetApp
  SRM is a workflow task allowing you to script your Disaster Recovery and actually test it. I am impressed by the product and would love to deploy it as I see some very good use cases. There is a but: it’s hideously expensive in my opinion and I simply can’t get the business case for it worked out. Too bad.
• Best practices to increase availability
  I’m afraid to even summarize this session. It’s given by two people who I consider absolute superstars, being Chad Sakacc (EMC) and Vaughn Stewart (NetApp). The fact alone these guys, who work at competing companies, are able to give a presentation that will grab your attention for the full hour is amazing! The presentation was packed full of tips and tricks. One I want to pick out: CHECK YOUR MISALIGNMENT (yep, that is supposed to be in capitals). They took the time to explain and emphasize this can cost you a lot of performance. Something else they mentioned: there is no ideal protocol. Each protocol is different and has it’s own super-power and kryptonite. What works best is designing for those.
• VDR, all you need to know
  VMware Data Recovery is the backup product you get for free when you have an Enterprise Plus license. While this started as a very simple solution, it has grown out to be very powerful and a ‘good-enough’ solution for a lot of people. As long as you are aware of it’s limitations (and use the most recent version) it could very well be enough for you as well.

Concluding, VMworld 2010 Copenhagen has been an awesome experience for me. I’ve gotten an incredible amount of information, packed in three days. Some of that is not directly applicable to how I am using VMware now, but a lot of it I can apply directly to my day-to-day work. As such, I’m hoping I can make it again at the next VMworld!

More than a week has passed from VMWare World Europe 2010 in Copenhagen

From the homepage of the event you can see:

Ok cool! But why was so important to be there? And above all, was it worth the cost?

Yes. It was an amazing experience in my opinion.

I believe every attendee of the 6000 there brought home something different but lots of information to spend in their every day job.

What about me? What’s my personal baggage I brought back?

A good overview about the new products introduced by VMware in the last year that I never had the chance, so far, to play with.

VMware CapacityIQ (VMware Booth Demo)

VMware Data Recovery (BC6701- VMware Data Recovery – All You Need to Know!)

VMware vShield (Lab on vShield)

An intro to the “What’s new” out there:

Just a few on top of my head are:

HA Application Awareness with VMware HA by VMware and Symantec

EMC2’s Avamar Virtual Edition a backup solution leveraging VMware Changed Block Tracking and deduplication

Cisco’s UCS Virtual Interface Card (VIC) soon available also for Rack Mount Servers

Storage DRS (TA7805 Tech Preview Storage DRS)

and lots more …

Another important thing that made it an amazing experience for me was to meet people there. Gents from all over Europe with a common interest in virtualization and more in general in IT converged in the Bella Center. People working with VMware products (customers and partners) and people working for VMware (VMware employees).  To be totally fair I had not that much time for that (the sessions were very close to each other, too much I believe!) but I made some time for it and was very helpful to get an understanding of how other Companies are designing and deploying their VMware infrastructures.

Confirmation that the assumptions and the choices we made during the last year and a half working vSphere where correct!

I believe this is a lot per se.

Not very much knowledge base was available when we started our first implementation fully vSphere for one of our customers. The implementation in the customer environment I’m working on has now grown (and will grow further) to 52 ESXi servers spread over two Data Center in an Active/Active configuration with a fully redundant 10 GB Eth connectivity. OK we already implemented likewise configurations for other customers so we were already sure it was the right choice!

Didn’t I learn anything new? Yes and no!

Yes because: As I said a very valuable piece of information I brought home is the confirmation that the infrastructures we are running here at Schuberg Philis, based on vSphere 4, are really cutting-edge technology and cutting-edge configurations. I learned though that there is still margin for improvements on the latter characteristic. In fact I’m planning to replace our SD card with a boot from SAN via iSCSI software initiator! (NOT just for the fun of it! )

No because: in my opinion more “in-deep” sessions were missing!

To summarize I hope to be there next year as well! Fine for me if it’s going to be the same  technical level… better if more “in-deep” sessions will be in there!

Now since the recorded sessions are finally available I’m going to enjoy one or two sessions I missed like TA8245 ESXiInternals: Better Understanding for Better Management and Troubleshooting

Last week Schuberg Philis organized an internal (official) VMware troubleshooting training (VST). I had personally already followed this training two months ago. However, as I found the training extremely useful and very dynamic (the training is never quite the same) I decided to follow the last two days of the training again. Another very good reason for me to do so was the fact it was given by superstar Eric Sloof (aka @esloof on twitter). He adds a tremendous amount of value to the VMware community as a whole with his blogposts on ntpro.nl which, if you ask me, should be part of your daily reading material. He’s also given trainings for Schuberg Philis before. From my experience he’s a very knowledgeable and patient trainer who is able to explain the material in a very easy to comprehend form while keeping the pace high. Added bonus: a couple of days before the training he was awarded best freelance VCI (VMware Certified Instructor) of the quarter!

VST is a very good balance between theory and lots and lots of hands-on troubleshooting. Each part will start with some theory, followed by labs. Some labs are the typical ‘type this and see that’ but most value are those where ‘random’ things are broken which you then have to find and fix. Especially the last two days are mostly these kind of labs. The experience with VMware specific troubleshooting you gain here is invaluable and is something which would normally take up months if not years of your professional life. The most valuable lesson of all (as with everything): know the product and your setup and structure you troubleshooting accordingly. This will save you hours of random troubleshooting and creating new issues in the process.  Of course, it also helps having the trainer around who can sometimes give a hint and steer you back in the right direction. Don’t want the training to take weeks…

If you are thinking about doing your VCP, this training will allow you to do the exam. If you already have real-world experience with VMware I would advise you to skip ICM (Installation, Configuration and Management) and go straight for VST. One important note to make is that the VST training material only partially matches the exam material. Most training centers however will add an extra exam training as a package (Schuberg Philis arranged this with Eric as well). Combine the two and you get much more value out of your training budget. And, with a bit of luck, you get a training from a VMware celebrity .

We are using Distributed Virtual Switches to provide a centralized network configuration to the ESXi hosts and naturally we are using HA to provide availability to the infrastructure.

The issue:

In case of and HA event some of the virtual machines restarted by HA are not connected to the network.

In the network connection properties of the VM we can see that

The checkbox “connect at startup” is correctly maintained

The “connected” checkbox is not!!!

In this situation trying to connect the VM to the network (select the checkbox) results in an error message.

‘invalid config for device 0′

Is obvious that this results in an environment where even if all the VMs are running the VMs restarted by HA are not available on the network!

The RCA:

In normal operations in presence of a dvSwitch the hostd process in ESX is charge of creating the dvPort in the dvSwitch to let the VM to use the network (Connect the VM to the port number XX)

In case of an HA event the hosd daemon is not doing that because the HA is using a kind of “shortcut” to start the VM. This results in, the dvSwitch even if present, configured and working in the ESX hosts where the VM are restarted is not aware of the existence of the dvPort where the VM is connected.

Environment:

vSphere ESXi, 4.0.0 build 193498

vCenter version 4.0.0 build 162856 

Workaround:

The most effective workaround so far is to restart, all the VMs affected, from vCenter (please note from vCenter Not from the guest OS itself) so the hostd daemon will properly start and connect the machine

-Or-

Restart the hostd in all the ESX hosts (that in case of ESXi is not very handy to do!) and reconnect all the machines affected to the dvSwitch.

The problem is now escalated to the developers in VMWARE and since they were able to reproduce the problem in house we expect a solution in a reasonable time. A bunch of scripts anyway can haelp us to workaround the problem at the moment.

We are currently implementing a XenDesktop 3.0 environment on vSphere 4.

Today I started to lock down the permissions the Xen Desktop Delivery Controllers (DDC) have on vCenter. There is not much documentation on this. Except for the kb article on VMWare Infrastructure 3 and XenDesktop which is lean and mean. But then i stumbled on this great blog post: http://theether.net/kb/100114

Which describes the solution to the error “This virtual machine could not be retrieved from the hosting infrastructure”
The solution basicly describes the proper permissioning for the accunts which access Virtual Center from the DDC and this even works for vCenter 4!

In VirtualCenter:

- Select View | Administration

- Click Add Role
- Enter the name XenDesktopGlobal
- Check Global | Manage Custom Attributes
- Click OK

- Click Add Role
- Enter the name XenDesktopDataCentre
- Check Datastore | Browse Datastore
- Check Virtual Machine | Inventory | Create
- Check Virtual Machine | Provisioning | Deploy Template
- Check Resource | Assign Virtual Machine to Resource Pool
- Click OK

- Click Add Role
- Enter the name XenDesktop
- Check Global | Set Custom Attribute
- Check Virtual Machine | Interaction | Power On
- Check Virtual Machine | Interaction | Power Off
- Check Virtual Machine | Interaction | Suspend
- Check Virtual Machine | Interaction | Reset
- Click OK

- Select View | Inventory | Hosts And Clusters

- Select Hosts & Clusters
- Select the Permissions tab
- Right click and select Add Permission from the context menu
- Select XenDesktopGlobal for Assigned Role
- Click Add
- Select the account used in the Logon Information properties of the Desktop Group
- Click OK
- Click OK

- Select the Datacentre that contains the virtual desktops
- Select the Permissions tab
- Right click and select Add Permission from the context menu
- Select XenDesktopDataCentre for Assigned Role
- Click Add
- Select the account used in the Logon Information properties of the Desktop Group
- Click OK
- Click OK

- Select the Cluster or Resource Pool that contains the virtual desktops
- Select the Permissions tab
- Right click and select Add Permission from the context menu
- Select XenDesktop for Assigned Role
- Click Add
- Select the account used in the Logon Information properties of the Desktop Group
- Click OK
- Click OK

Source: http://theether.net/kb/100114

VMware has released Labmanager for Vpshere 4. http://www.vmware.com/products/labmanager/

VMware vCenter Lab Manager is the ideal solution for IT organizations who want to provide self-service provisioning and management capabilities to internal teams. Policy-based access control reduces administrative burden for IT, lowers infrastructure management costs and empowers project teams to deliver applications more quickly and with greater agility.

Deliver Higher Service Levels and Lower Infrastructure Costs

Lab Manager offers unique capabilities to simplify management of the internal cloud for dev/test:

• Self Service Portal – Provides on-demand access to a library of virtual machine configurations for end users while eliminating time-consuming provisioning tasks for IT by 95%.
• Automated Resource Management – Allows dynamic allocation of resources in a multi-team environment, enforces quotas and access rights, and reclaims unused infrastructure services.

Enterprise Scalability – Provides long-term return on investment with a scalable architecture for worldwide deployment, best in class performance and seamless integrations with in-house and 3rd party solutions.