Mother Board replacement and ESXi

One of my clients on the west coast is considering replacing the motherboard in one of their ESXi servers with a much faster one and asked the question "Will ESXi still boot and function properly?". This is a standalone ESXi server. The existing motherboard has an older Core 2 Duo and the new one has an i7.

Well the short answer is "probably not, without some work". Probably better in a larger environment to use a host profile, rebuild the host after the motherboard upgrade, then restore the profile.

The client chose the motherboard ad had a local computer shop do the swap. Once the motherboard was replaced, ESXi came up but had no network. A little investigation showed that the replacement motherboard (Asus P6T) has an unsupported network card. I recommended that the client buy a Intel Pro 1000 Desktop card. Once installed, ESXi recognized the card and a little console configuration brought the host back to life. More details later.

Errors on the Storage utility box

I am still seeing intermittent errors on the utility machine that houses NFS and iSCSI target storage. I am tempted to move this off of Win XP (NAV firewalls, etc.) and load Linux to make sure I don't have issues with XP's "Protectionism".

Will probably load up a Linux VM to get the entire config down first.

DRS testing

Cold VM startup functions in DRS are working properly, meaning that when I properly configure a VM on shared switch and shared storage, DRS decides which host to start the VM on and this will change based on the available resources (CPU, memory). So I can make a vMotion test VM start on one host and then stop it, start up 3 more VM's on that box, and the next time I start the vMotion test VM, it starts on the other host.

Log entry below (the test vm is called "win2kmove" and the vSphere cluster I created is called "The Towers"):

Relocating Win2kmove in The Towers from 192.168.1.140 to 192.168.1.175
info 7/28/2010 11:07:59 PM Power On virtual machine Win2kmove

Cold vMotion

I was able to successfully move a Linux VM from one host to another via cold migration. However, my first ESXi host has some issues, so no live vMotion yet....

Will perform a cold Storage vMotion next.

StarWind iSCSI solution issues

I am still fighting with my iSCSI configuration on my utility machine. Everything is configured and both host can see the iSCSI LUNS but I am seeing tons of connectivity errors in the cluster logs. I have NFS running on this box, so this could be an issue. More research...

Down time is over - Windows Server 2008 R2

Have been underwater the last week or so, preparing for the VCP exam. Now that the test is out of the way, back to cases.

Installing Windows Server 2008 R2 (64 bit) as a VM is pretty trivial, except that it comes up without a network.

After a few clicks, I remembered reading that VMware Tools needs to be installed in order to load the AMD driver. So, keep that in mind.

Access to iSCSI on Utility machine

I am seeing some errors in the VCS logs about my iSCSI Datastores going up and down. I am not sure what this is but it might have something to do with XP/Norton on the utility machine and their restrictive firewalls. Need to look into this.

VMware Server changes subnet of Host-Only network after reinstall

A client of mine in California ran into an issue with VMware Server.

A VMWare server instance runs on a Window 7 x64 with a couple of WindowsServer 2008's running inside. These guest machines run on a host only network.

The underlying Windows 7 OS had to be re-installed and after re-installing VMWare Server (and the console plug-in) the machines would initialize but the console came up black.

After determining that this was not related to the Host-Only network, a reboot of the Windows 7 box made the console work again.

Then we discovered that the previous Host-Only subnet had been changed after the re-install (291.168.112.128 --> 192.168.29.128).

After changing the static IP address in the guest machine, all was right with the world again (until the next hiccup).

Problems with vCenter Server Service

My vCenter Server VM (VCS VM) is not currently a member of AD (later), but I have been having a problem with the VCS service and VC Web Service not starting after a reboot of the VCS VM. The VC Web Services already has a dependency on VCS starting before it can start, which is why it won't start.

A peruse of the event logs show that VCS service is trying to log into the SQL express DB before it is ready. To resolve the issue I went into the registry and added the SQL Express service as a dependency on both the VCS service and the VC Web Services service:

Launch Regedit
Navigate to \HKLM\CurrentControlSet\Services

Under the vpxd key on the right, double click on the value DependOnService
press Esc (The data in the dialog might be highlighted)
Add "MSSQL$SQLEXP_VIM" to the bottom
Click OK

Under the vctomcat key on the right, double click on the value DependOnService
press Esc (The data in the dialog might be highlighted)
Add "MSSQL$SQLEXP_VIM" to the bottom
Click OK


**

After a reboot, the VCS services now start properly. This dependency will be removed once the SQL Server 2005 VM is functional and I move the VCS database to it (future).

VM's inaccessible

I had powered down my utility machine (XP w/NFS & iSCSI) last night for another task. This morning, one of my ESXi Hosts was missing. Once reconnected (Connect command from right click of host), two of my VM's were inaccessible.

Of course, the Datastore location of the two VM's was on that machine over iSCSI....Have to remember, Datacenters are always up!

Clustering

I am able to build a vSphere Cluster, add my real and "imagined" ESXi Hosts to it and enable HA and DRS. Did some testing of Fault Tolerance of a VM as well. Created a few Resource Pools in the Cluster. It will be a day or two before I have time to test these features.

Noticed that you cannot create a resource pool in vCenter Server without HA being enabled.

Realization - ESXi inside of ESXi

It is not possible to run a VM inside a VM (doh!). So, the dream of loading ESXi inside of ESXi and loading VM's under this child VM is done. Thanks for playing....

Doing an upgrade from ESXi 3.5 to 4 is still possible, but there are other hurdles to that.

ESXi turns in on itself

There was no easy way of getting around the issue of NO VT support on my Dell 745. I had to break down and get another machine....or go through the process of trying to replace the CPU, which I had no appetite for.

So I now have a Dell 780. It will run ESXi 4 Update 1 and allow me to not only install 64 bit guest OS's but will also support installing ESXi inside ESXi to have multiple Hosts. Theoretically, I could install two VM's of ESXi and have a full vSphere Cluster running inside of an ESXi host. This work should happen in the next few days.

study...study...study...

During my virtual process, I am studying for the VCP 4 exam. Much of the material is difficult to master without actually configuring the products, which is another reason for setting all this up. There is also some memorization involved.

Much of the VMware code can be downloaded and installed on a trial basis which gives the little guys the ability to fully test and learn without that pesky 2nd mortgage being taken out.

For those on a similar track, check out VMware Whitebox for hardware choices.

Datastore for ISO sharing

Needed a place to put ISO images for loading VM guest OS's. I know my old Win2k machine could support NFS and I need to move off of Win2k anyway. The Win2k machine is pretty old so it would make a good utility machine.

I used Microsoft's Services for Unix (SFU) some years back while consulting at IBM working on Citrix. SFU does support NFS services. Also, I wanted a RAID location for storage and backup of various data.

The VMware best practice is to access IP Datastores on a separate high speed network between ESXi and the ISO share. Theoretically, no internet traffic is needed over this link.

Get ready
Install a Intel Dual Gig-e card in ESXi
Upgrade Win2k to WinXP
Install a Intel Pro 1000 into XP
Connect the two gig-e cards to a gig-e switch

Make space
Install SATA RAID card and two 1 TB SATA drives in XP
Set the drives to RAID 1 (mirror)
Dump all the Windows, Linux, VMware, etc. ISO's in a folder

Here comes NFS
Download and install MS Services for Unix (SFU) 3.0 on XP
NFS share out a folder on the raid array

Issues with permissions in SFU
I ran into some trouble getting ESXi to properly connect to the NFS mount point. After some searching, it turns out I needed to pull the "/etc/passwd" and "etc/group" files from ESXi and place them on XP in order for the user/group mapping to work properly.

To get them from ESXi back to XP, I had to log into the console of ESXi and copy the files to the exposed ESXi Datastore directory, then go into the vSphere client, browse to the folder on the Datastore where I placed the files and then download them to XP machine.

Once these were in place, I could add the NFS mount point as a Datastore in ESXi, over the gig-e network.


**
Now, I have a 1 TB Datastore where all my ISO's sit, as well as any other files needed by Windows, Linux, or ESXi.

Later, I will talk about using this same storage system to test and use an iSCSI Datastore.

The new machine works great, but.....

Loading up ESXi onto the new machine was trivial, showing that VMware has done a very good job building the product. I was creating VM's within 30 minutes.

After some playing around and reading, I wondered if I could create a ESXi inside of ESXi to fully test the advanced features (VMotion, DRS, etc.). Of course I must have missed a meeting, because the box I have has a processor that does not have the Intel VT feature, which means, even though ESXi (64 bit) does install, I cannot load any 64 bit applications on it, including ESXi.

This is an issue, but I moved on configuring everything I could.

VMware - Early returns

After deciding to move forward on becoming proficient in vSphere, a new machine was needed to load ESXi 4 on. Some searching turned up an alternate hardware plan than using the vendor's HCL, which would require a second mortgage.

The Whitebox site lists hardware that, while not on the HCL, will work with ESXi:

http://www.vm-help.com//esx40i/esx40_whitebox_HCL.php

From that, I acquired a refurbished Dell Optiplex 745 desktop box.