MOVING ON.........

THIS WAS THE END OF THE VMWARE PROJECT IN 2010 - SEE YOU NEXT TIME!!!!

Moving on to testing more products

Once again, our consulting practice has kept us away from posting here, but we will try and move forward again.

vSphere now has a single point of internet connectivity, which is cape-may, the Ubuntu server that is being used as a Ethernet bridge.

Anyway, Microsoft Office Sharepoint 2007 (MOSS) is next on the list. A new Win2k3 server was built for the work, a server called Asbury-Park. Asbury-Park will use the SQL instance on red-bank for it's database.

New plan for vSphere client target PC

For the final target "client" for accessing vSphere from the equipment closet (EC2), we are contemplating getting a new machine (to replace tibet77), running WinXP and VMware Workstation with a Linux NAT server as the LAN/Internet forwarder. This machine will sit in EC2.

More later.

More on the AD move

As of now, most of the critical VM's have been migrated to Gig-e and the Active Directory DNS has 75% of the new 10.x addresses, leaving the less critical Xenapp and VMware View VM's yet to be migrated (add new Gig-e nic cards; assign new 10.x addresses).

This will be completed by week's end.

Catch-22

We are using one of the Win XP VMware View VM's (dt-xp002) as the client for the vSphere client, accessing it via RDP. While this is a temporary setup during the config for the equipment closets, it presents a catch-22.

Since this VM resides on westwood and the vCenter Server VM resides on northvale, there is no easy way of shutting down either ESXi server without losing control of vSphere.

Until the move is ready, we will be accessing vSphere via Gig-e from external clients. In the end, the Win XP hardware called tibet-77 will move to the closets and be the client machine.

Move Active Directory to Gig-e

First, the Domain Controller (trenton) needed a 2nd NIC card and an assignment to the gig-e network. Once in place, DNS was updated manually to reflect the 10.x address (DNS is running on this DC).

Since the move to the equipment closet will mean that all internet traffic from vSphere will need to be forwarded to the new NAT server, the network card gateway was set to point at cape-may. The DNS setting would point at itself.

Once these changes were made, all domain traffic will go over Gig-e. It will cause all other domain members to be confused until their NIC card and DNS configurations were updated.

Tomorrow....

Moving vSphere hardware - #2

So we were aware that to move the vSphere hardware would require moving all network traffic to the Gig-e network. The biggest issue here is that Windows Active Directory and it's DNS were built on the local LAN.

Also in the plan is to move the ESXi hypervisor networks, as these were built on the LAN as well.

A little planning was needed in order to gracefully move the core VM's to Gig-e. Also, not all domain members were built with dual NIC cards.

Proof of Concept NAT Server

To validate the NAT scenario, a Ubuntu server VM was created called cape-may. This is just the CRM standard Ubuntu build, with IP forwarding and IP Tables configured.

One NIC card on cape-may is connected to the internal LAN while a 2nd NIC was placed on the dvs-Production distributed switch (Gig-e). Client devices that need access to the LAN and internet can be placed on dvs-Production with a gateway address of cape-may on gig-e and a DNS setting of the LAN switch.

Some preliminary testing using the Win2k VM win2kmove validated the config quickly. Both ping access and web browser internet is functioning. It is important to add the DNS entry on the client, since that is how internet names are resolved. This will become more complicated once AD is moved (more on this later).

In the equipment closet configuration, the LAN card will probably be a USB wireless network adapter.

The next step is to configure VMware Tools on cape-may to get better throughput by loading the Tools' NIC drivers. This will happen in the next week or so.

The other item that needs testing is the ingress access which is required to get to the vCenter and ESX servers. This might require additional configuration on the NAT server.

Moving vSphere hardware

It has been a plan for sometime to move the vSphere hardware out of the lab and into one of the equipment closets. One of the issues with this plan has been how to get LAN access to the equipment closets, as well as internet access.

Short of running additional cables, a NAT Linux server with a wireless NIC should be able to forward traffic to the lab LAN and on to the internet. Obviously, the production instance will need to be real hardware since VM's don't support things like USB network adapters.

New Win7 x64 desktop

It has been a while but some progress has been made. The Dell 745 has been installed with Win7 x64 (6 GB memory) and is now the primary workstation. The HP laptop is in it's case (where it belongs), ready for customer site travel.

The dual monitor set up has been transferred to this environment, using the Dynadock.

The faster performance of the 64 bit platform and extra memory is very noticeable.

New Win7 x64 desktop

This new primary workstation is close to being ready for production.

One item at issue is the amount of traffic on the 100Mbps front end network. When performing installs, doing testing, and maintenance on vSphere components and VM's, this front end network becomes very congested.

The plan is to move all vSphere management, Active Directory, and storage traffic to the back door gig-e and run VMware Workstation on the new Win7 workstation for all management access to vSphere. This will ensure that only internet bound VMware traffic will travel over the slower network.

Issues with the vCenter Server

Attempting to log into vCenter, the cluster is unable to be found and an error is presented stating that VUM (VMware Update Manager) cannot be contacted.

Initial troubleshooting plan is to log out of vCenter and directly into northvale, log directly into alpine (win2k8 box housing vCenter) and check the logs to make sure these are no obvious errors. The only error is a Winlogon application error for GPClient.

More digging. Shutting down vCenter VUM services on alpine and moving to red-bank to ensure SQL Server is happy.

First step is to push all new Windows Updates to red-bank, and rebooting. The errors persist.

red-bank event logs were showing logon errors for a system request along with a SQL scheduled job failing for a database. It was not long before we determined that this is a task for a database that is no longer present.

After removing the task from SQL Management Studio, the errors are gone. We will give the SQL server 30 minutes or so to ensure the errors do not return before restarting vCenter and VUM services on alpine.

Windows 7 x64

We are installing a few Windows 7 x64 VM to continue testing VMware View 4.5.

Due to some limitations on the previously decommissioned cresskill, only Windows 7 x32 will go on this puppy.

Maintenance Day

Today will see backups of vCenter, Installation of a (additional) 1TB drive in westwood, the decommissioning of cresskill (utility ESXi 4.1 host), and some data backups.

The additional 1TB of space will be used for additional VMFS space as well as for the testing of OpenFiler for iSCSI shared storage.

Uber backdoor Gig-E not up

Ran into a minor issue with Uber that has been here ever since Uber was moved from cresskill to northvale ESXi hosts. Decided to look into it.

The symptom was that northvale (or any of the ESXi hosts) never recognized shared storage on Uber and showed the Datastores as grayed out. This would also cause the VM's using the shared storage to be disconnected and inaccessible.

After some investigation this morning, it was found that the 2nd network card in Uber VM, that connects the iSCSI and NFS shared storage to the back door Gig-e would not come up without a forced sudo ifup eth7 command to bring it up.

We first attributed this to a race condition of the northvale host coming up completely and doing it's scan for storage before the launch of Uber. This may still be the case for iSCSI.....

Anyway, for those following along, you will remember that all VM's were moved from the utility host (cresskill) to northvale since we needed to move to Win2k8 x64 for vCenter Server and cresskill does not support x64 VM's.

Uber was the last to be moved. When a Ubuntu VM is moved, the NIC cards become different eth numbers. Some fun was had (and documented) getting the correct eth numbers set on Uber. The file "/etc/network/interfaces" on Uber had to be modified to update the eth numbers after the move. Of course, there was a typo in the update for eth7, causing it to fail to load at boot.

Once corrected, Uber's back door Gig-e was loading properly. After a reboot, northvale now recognizes the NFS and iSCSI shares from Uber once Uber is up.

PCoIP working properly

Now that the agent issues are resolved, some preliminary testing shows that the PCoIP issues experienced with View 4.0+ are resolved. Obviously, more testing is needed to make sure other issues have not popped up.

View vDesktop not recognized

After some testing of the failed vDesktop, we were unable to find anything wrong with the installation of the agent.

Finally, the primary connection server was bounced and the vDesktop agent was recognized. This is not a good story, saying that when a new vdesktop is provisioned, the only way to add it to a pools is to restart the primary connection server. More testing is needed to validate this.

VMware View 4.5

Moving forward on the VMware upgrades, the primary connection server (Piermont) was updated to version 4.5. This update went in with much fanfare and the new GUI web site seems to be much better designed and has faster response than the 4.0 version.

All settings (Pools, Desktops, etc.) and configurations seem to have transferred properly. However, booting on of the vDesktops was vary slow so we reverted to a snapshot, which may have caused some recognition issues inside of View 4.5.

The 4.5 agent was removed and reinstalled, but View is still unable to recognize the agent. More troubleshooting is needed.

Citrix XenCenter

Once on the console of win2kmove1 VM, we surfed to the XenServer web server and attempted to install the XenCenter app, where we received a pop-up stating that .NET Framework 2.0 SP1 was required.

After installing this piece, the same error is being reported. a little searching found that installing .NET Framework 3.5 SP1 allowed us to install XenCenter.

Launching the XenCenter app, we were able to add XenServer, however, the server does not have access to the back door Gig-e, where the shared storage resides. Will need to add a 2nd vNIC to XenServer for this to work.

Citrix Xen Server

A small side virtualization project will be to load Xen Sever into a VM and test some of these features.

Xen Server is Citrix's virtual host platform.

Xen Server took some 50 minutes to install in a VM, but it did complete. An existing VM (win2kmove1) will have the XenCenter application installed on it.

vSphere maintenance items

VMkernel Management network
There has been some discussion about moving all management network operations to the back door Gig-e. One main impetus for this is the fact that a number of operations involve moving data from one datastore to another, whether local or on a VM (NFS and iSCSI shared storage runs on the Uber VM today). Normally, vSphere does some of this over the slow network and there seems to be no easy way to predict which network will be used.

Moving everything over will ensure that all of these activities will happen at the speed of Gig-e, as opposed to some of them happening over the 100 Mbps Production network that is connected to the internet. This also frees up the Production network for ISO downloads, etc.

Bridge Network
The other item is the physical location of the vSphere environment. Moving these Hosts would require a bridge network for access to the internet as well as vSphere client access to the environment.

Planning for both of these projects is underway and finalized procedures should be in the can by later next week.

In the mean time, new feature testing and the VMware view 4.5 upgrades will continue.

VMware Tools updates for Wintel

Now that all three ESXi hosts have been updated to vSphere 4.1 and the licenses have been re-applied, we will turn our attention to the maintenance work of updating VMware Tools on all the Wintel VM's.

Alpine, Trenton, and red-bank VM's on northvale are now updated, with the VM's on westwood to happen today. This includes all VMware View VM's. These VM's were set to Manual in DRS such that no automated movement would happen during View 4.0 testing.

The old Wintel VM's that sat on cresskill are now duplicates, but Uber (Ubuntu) VM is still housed there, so it will get updates tomorrow.

Upgrade the ESXi Hosts to 4.1

After additional reading in the vSphere 4.1 upgrade guide, we planned to update the ESXi hosts (cresskill, westwood, and northvale). The 4.0-4.1 update zip file was downloaded from VMware and placed on a local hard drive.

Plan A was to use Update Manager (VUM). The update file was imported into Update Manager, a baseline was configured, and cresskill was chosen as the first host to be updated. Once the VUM configuration was set, we kicked off the update. It failed within 5 minutes due to the fact that we are using eval licenses that are not supported for update through VUM. On to Plan B.....

Plan B was to use the vCLI and the vihostupdate script to perform the updates. Getting the syntax correct took a little work but in the end, all hosts were updated to 4.1. The "vihostupdate" query command was run on each host and they each displayed the following:

ESXi410-GA 2010-09-20T20:44:10 ESXi upgrade Bulletin

ESXi410-GA-esxupdate 2010-09-20T21:18:08 ESXi pre-upgrade Bulletin


We may have an issue later due to the order of the commands run (esxupdate run after GA), but there seems to be no immediate concern.

vCenter Server 4.1 is being brought back up and testing will resume tomorrow.

VMware Update Manager 4.1 (re-install)

After completing the vCenter Server 4.1 install and moving the key VM's around, it was noticed that the VMware Update Manager Plug-in was failing to start. Seems that the plug-in was still pointing to VUM on the vCenter Server 4.0 instance.

A re-install of VUM on alpine was needed. Once this was installed, the Update Manager plug-in can be properly enabled.

Weekend Update - 09/16

At the current moment, here is where we are in vSphere:

• Three ESXi 4.0 Hosts
• vCenter Server 4.1
• VMware Update Manager 4.1
• VMware View 4.0

Items for next week:

• Upgrade the ESXi Hosts to 4.1
• Upgrade VMware Tools on all VM's (including Uber)
• Network configuration change planning
  
• Upgrade VMware View to 4.5

vSphere 4.1 topology

vSphere 4.1 topology #1


vSphere 4.1 topology #2

To give more context to the text here, we are uploading a few diagrams. Above are links to the current layout in vCenter Server and the map. These are now on the upper right side of the main blog page.

Moving VM's - the hard way

On the cresskill ESXi Host, the VM's Red-Bank (SQL Server 2005), Trenton (Win2k3 AD), and Uber (Ubuntu Linux shared storage) all need to migrate to the Towers1 cluster, preferably onto northvale ESXi Host, where alpine VM (vCenter 4.1) resides.

In other environments, the movement of VM's from one host to another would be relatively trivial, as vMotion and Storage vMotion would be the standard solution. However, since cresskill, the utility ESXi host, does not have a supported CPU (no VT) to join the Towers1 cluster and be available for these VMware functions, the live migration features will not work.

The other caveat is a catch-22. Some of the VM's that need to be moved are active participants in vSphere or Active Directory and can not be brought down, even for cold migration, without bringing down those services while the move happens.

The solution to this challenge is cloning. Cloning VM's creates a mirrored snapshot of the original live VM files, in a location of your choosing, and creates a new VM object pointing at the new files.

To that end, VM's Trenton and red-bank were cloned (Trenton2; red-bank2) into the Towers1 cluster, the original VM's were shut down and the new clones were brought up. For red-bank2, vCenter Server 4.1 service was stopped before the switch, since its database is on red-bank. Trenton2's AD was more forgiving, allowing for the absence of directory services while the switch was made.

Due to the size of Uber VM's shared storage (close to 250 GB), this clone will run over night, even on the Gig-e.

vCenter Server 4.1 - Part Deux

After some RTFM in the 4.1 install and upgrade guides, it was clear that the best approach for upgrading to 4.1 is to back up the 4.0 database on red-bank and then install vCenter 4.1, pointing it at the existing DB. This would ensure that all the configuration of the current vSphere 4.0 environment will be migrated properly.

Here is the process:

Shut down the vCenter Services on VCS01 (vCenter Server 4.0)
Backup the VIM_VCDB database on Red-Bank
Create a x64 ODBC DSN on Alpine pointing to the above DB
On Alpine VM, begin the install of vCenter Server 4.1 selecting "Connect to SQL Server"
Select "Upgrade Database"

Since the login credentials for the session on Alpine were user credentials that we could not change at install time, we use them. The vCenter Server and VC Web Server services will be logging on as this user. Once the install is complete, we went into Windows services on Alpine and changed the "log in as" credentials to the domain database user (dbuser01, in this case).

vCenter Server , though some were not active.

After logging into vCenter Server 4.1 all objects were present, however all the ESXi Hosts were disconnected. This was because the hosts are still members of the vCenter 4.0 instance. Right clicking on each host and selecting "Connect" initiated the install of the new vCenter 4.1 host agent and allowed the hosts to be added to vCenter 4.1.

Next is to move some of the VM's on cresskill to the Towers1 cluster.

VMware Update Manager 4.1 install

Once we were comfortable with the install of vCenter 4.1, the install of VMware Update Manager 4.1 (VUM) seemed like a quick next step. A new database instance was created on red-bank, an x32 ODBC DSN was created on Alpine, and the install of VUM 4.1 was completed.

The VUM plugin still needs to be installed and activated, but the code is in place. This will be for fixes and probably will not be used for ESXi 4.1 migration (more RTFM on that needed).

ODBC DSN fun
An interesting thing happened on the way to installing VMware Update Manager 4.1 (VUM). Since the VUM 4.1 code installed on Alpine is x32, the ODBC data source DSN for talking to the database needs to be x32! The warning in VMware documentation goes like this:

IMPORTANT: Although you can install the Update Manager server only on 64-bit machines, Update Manager is a 32-bit application and requires a 32-bit DSN.

If you open up the standard ODBC DSN interface on Win2k8 x64, it will only create an x64 DSN.

A few forums highlighted this point, although we did not see the procedure create an x32 ODBC DSN in the admin guide (must have missed a meeting....).

From the forums: On your 64-bit server, create a 32-bit ODBC DSN with odbc32.exe under:

C:\Windows\System32\odbcad32.exe

This brings up the same GUI but it allows you to create a 32 bit DSN.

Once this was created, the VUM install went smoothly.

vSphere 4.1 - ESXi Hosts

As far as the host upgrades are concerned, they will stay on ESXi 4.0 Update 2 and will be migrated to the new vCenter Server before any host upgrades happen. Some planning will need to take place and the vCenter Server 4.0 will need to have its network mirrored on 4.1.

After all ESXi servers are migrated to Alpine's vCenter, VM's can be migrated off the ESXi hosts in order to upgrade them to 4.1.

Since all shared storage is on Uber (Ubuntu VM on cresskill) or the physical server tibet77, the iSCSI and NFS Datastores on these boxes will just need to be rediscovered once cresskill moves. The new vmkernel ports on vCenter Server 4.1's distributed vSwitch will have different IP addresses, but the VM's will retain theirs.

vCenter Server 4.1 install (x64)

vCenter Server 4.1 "Proof of Concept" was installed on alpine VM, which is running Windows Server 2008 R2 x64. The POC install used the SQL Express local DB.

As with the last rebuild of VCS under vSphere 4.0 U2, we attempted to duplicate the process of moving the VCS DB over to the standalone SQL VM. However, since the current VCS DB is already on red-bank, the internal name of the DB is the same, preventing us from importing the new DB into SQL Server 2005.

After some reading and a bit of testing to change the name of the new DB, It was determined that it would be easier to just create a new DB instance on red-bank and the point the VCS install at it.

First, vCenter 4.1 was uninstalled from alpine. On red-bank, SQL Server Management Studio was used to create a new database named VIM41_VCDB. Back on alpine, an x64 ODBC DSN was created to point to the new database on red-bank. Then, vCenter 4.1 was re-installed.

No objects have been created in the new VCS yet, as we need to do some planning in order to migrate cresskill, as well as the ESXi Hosts in the Towers1 cluster, and to make sure all the networking and storage features make it across the divide.

After some reading, this will probably be re-installed.

vSphere 4.1 - vCenter Server

Currently, we are running vSphere 4.0 Update 2 and VMware View 4.0. With the comments in the forums about compatibility issues between different versions of vSphere and View, the best approach seems to be to go to the latest since many of the interim fixes have been applied.

No doubt, there will be more exciting issues in store with the new versions.

vSphere 4.1 and VMware View 4.5 have been placed out on the 1TB raid repository in tibet77, so these will be the next few upgrades. vSphere 4.1 includes vCenter Server and its components as well as ESXi 4.1 ISO for the hosts.

vCenter Server 4.1 requires an x64 Windows O/S, so, as opposed to installing Windows Server 2003 x64 VM for this process, we can utilize one of the Windows Server 2008 R2 x64 VM's that were loaded in the Towers1 cluster for proof of concept. One of these will be renamed "Alpine" for this purpose.

Cresskill, the ESXi utility host which currently houses Active Directory; iSCSI and NFS shared storage; SQL Server 2005; and vCenter Server 4.0, would be the ideal place for the new vCenter. However, this ESXi host cannot host x64 VM's, since the CPU does not support the VT option. Due to this limitation, the Alpine VM will sit in the Towers1 vSphere cluster for now.

New View Agent testing

All three Desktop-VM's now have the patched View Agent. Testing continues.

Two Desktop-VM's are currently in a VMware View PCoIP Pool. A PCoIP session can be successfully connected from two different clients (Vista x32 laptop; Windows Server 2003 x32). RDP sessions can also be initiated. This validates that the new View Agent code resolves the issue.

Now, where were we??....oh yeah, preparing a standard XP image for Composer....

Also reading a document on Virtual Desktop Manager load balancing.

Connection Broker Testing

While updating the View agents in the background, both Connection Broker VM's have not been powered on. The replica View Connection Broker VM was powered up first. As expected, its Connection Server service failed to start, meaning that there are no CB services available for clients. This tells us that if the primary CB is inaccessible, the entire View service may be down.

Anyway, the primary CB was powered on and the replica's connection server service was restarted. This scenario will be tested further later when we get into VMware View availability and redundancy.

VMware View PCoIP issue resolved

From the initial install on VMware View, accessing View desktop using the PCoIP protocol would fail (the user was still being logged in). However, using the RDP protocol would connect properly to desktop-VM's.

After perusing of the event logs on all 3 devices (client, connection broker, Desktop-VM), the common theme was that something on the Desktop-VM was killing the session connection to the client, after login. I could see the session connection errors in the logs.

Changes were made to the network environment and the available video memory on the agent VM. Several testing scenarios were exercised, but it still seemed to be a Desktop-VM problem.

The issue turned out to be caused by the update of ESXi from 4.0 Update 1 to 4.0 Update 2. This updated VMware Tools for the platform, which created an incompatibility between it and the VMware View Agent, which caused the PCoIP protocol to no longer function.

The solution was to download the patched View agent software, uninstall the current agent and install the patched version. This forum post highlights the issue and the resolution (at least for us):

Upgrading VMware Tools in a virtual desktop causes PCoIP connections to fail

VM dt-xp001 was the first to be patched. There were some issues getting the broker to recognize the new agent (reboot of the broker) but once it was recognized, PCoIP was working properly. We are now able to connect over PCoIP from Vista (x32) and Windows Server 2003 VM (x32).

The other Desktop-VM's will be patched over the weekend.

VMware Update Manager (VUM)

The errors in vCenter related to VUM are now resolved. A reinstall of VUM did not resolve it, as the VUM plug-in was actually the source of the error.

The domain id provisioned for connecting to the database (SQL Server 2005) was applied to the VMware Update Manager Service on VCS01 (the id is already used for the VMware vCenter Server Service). Previously, the VUM service was using the local system account for VCS01. This is legacy from the reinstall of vCenter last week.

After applying the id and restarting the service, the VUM plug-in could be re-enabled and vSphere fixes were being updated in VUM. A maintenance cycle will be scheduled for early next week to play around more with VUM. vSphere 4.1 is still on the list.

More on PCoIP Issues

While RDP testing continues to work properly for VMware View testing, PCoIP protocol is still logging the user in and then disconnecting session (user is logged into the vDesktop). The connection server log is showing this error:

Conveyor 0 IOException. Remote address is null: An existing connection was forcibly closed by the remote host

Recommendations from the communities have ranged from:

Firewall issues
To ensure this is not an issue, we moved all VM's in the VMware View test environment to one ESXi host (Manual Automation Level in DRS), including the client itself. Also, all VM's (CB, desktops) are on the same vStandard Switch on the ESXi host (no policies set).

Group Policy issues stopping port 5002
Created a new OU and moved all the desktop VM's into it.

Video drivers
The latest VMware Tools is installed.

**
The network sniffer is not providing any smoking guns either, so we continue to dig.

Going to use RDP for now to move forward on testing View Composer and the virtual hardware load balancer (NLB). We will come back to the PCoIP issues.

New deployment of View desktops

Currently, there are 2 connection servers and 3 XP desktop VM's. These VM's were deployed individually.

Additional applications were installed on dt-xp001 to begin the process of making a master image. Once the install was completed, a snapshot was created, as well as a template for cloning.

Open filer
This appliance VM (called atlantic-city) was installed in the Towers cluster for testing. Since the current vSphere shared storage for vMotion is a home grown Ubuntu Linux VM and is completely custom (Uber serving NFS and iSCSI), Open filer should provide a standardized solution for shared storage.

Other items
Time for maintenance, scheduled for this coming weekend.

• Backup the vCenter DB's
• Reload the NFS ISO library repository on Uber
  
• Sync up the NFS ISO library with tibet77's ISO share
• Prepare for the use of VMware Composer to deploy truly shared desktops.

PCoIP - New Desktops

Moving forward with VMware View, RDP connections from the View Client to Desktop Pools works, but PCoIP seems to start the session but will disconnect the client. In View Manager, the PCoIP session is active but disconnected. Opening the console of the Desktop (dt) VM on the vSphere client shows that the View session is in fact active (locked desktop).

Some local LAN DNS issues are cropping up as well, which need to be resolved to ensure these are not attributed to View itself.

More than one Connection Server (CB)
Additional connection servers (called replicas) will be created, and a Proof of Concept "hardware load balancer" (LB) will be deployed to truly simulate multiple CB servers ahead of clients.

RDP testing
From RDP, application and utility launch performance is good but motion video leaves a bit to be desired. This may have something to do with the View client matching the varying resolutions of the target client (Dual monitor with 1260x1080 & 1024x768). At the moment, this is not a critical issue.

Desktop Image creation
A new master will be built on dt-xp001 (first desktop VM) with the install MS Office and a few other real world applications.

Tasks will be to create a new VM template from dt-xp001, new desktop VM's from the template, and begin to create and set Domain level policies (GPO's) to lock down the desktop for users.

More on View

A successful test of VMware View was conducted. We now have a single connection server (piermont) and 3 Windows XP Pro VM's installed as Desktop clients (View Agent installed).

There were a few hitches with the initial test but these are just AD configuration items that will be resolved early next week.

After consulting a colleague, It was confirmed that the View connection servers (CB) are not multi-master, meaning that the 1st one installed is the primary and all additional CB's are replicas. The primary is the one that admins would log into and it contains the ADAM database with the View configuration, which obviates frequent backup to remote storage.

Also, a DNS alias and a hardware load balancer should be used ahead of the clients to ensure View connection availability and redundancy.

VM View client

A Windows XP VM was built and a template created from it. From the template, a new VM was built and the View Agent was installed.

Once the desktop users were created and provisioned inside of View Manager, the client was launched, pointed at the connection server, and the single desktop was available. However, the new desktop was not found so there is more RTFM to be done.

Also, the Openfiler [open source] SAN was downloaded. Of course, we rolled out our own shared storage solution (NFS and iSCSI from Uber on cresskill) but Openfiler is the standard test method for such storage so this will go on the list for testing.

VMware View components

The View Connection Server component (CB) has been installed on the VM piermont. The Connection Sever application is used for desktop clients connecting to View desktops as well as View administration. View Manager is now accessible from piermont's web server.

Next is the install of View Composer on VCS01 (vCenter Server). During the install, the View Composer service would not start. Some perusing found that the service needed a logon id. The dbuser id was added to the service but this needs investigation, there was no method to add this during install.

VMware View

The reinstall of vCenter went smoothly, although there is a minor error with VUM.

The real old vCenter VM has been renamed "Piermont", moved to cresskill, and will be used for the next Proof of concept process: VMware View.

Some RTFM is needed.

New vCenter Server

The time has come to re-install vCenter Server. The settings for the NY100 Data Center should all be captured in the databases on red-bank (vCenter and VUM) so the process should be to snapshot to existing image, uninstall VCS and VUM, then reinstall them, pointing at red-bank.

The rebuild was completed without incident.....

All ESXi hosts were upgraded from ESXi 4.0 Update 1 to ESXi 4.0 Update 2. ESXi version 4.1 is out but that will happen in the future.

Update on back end network speed

During one of my Storage vMotion tests, there were flashing lights on the front door 100 Mbps switch, and no flashing lights on the back door Gig-e. This signaled that the Storage vMotion operation was actually being performed on the slow network. Also, the progress was slow.

First change was to disconnect Uber's front door vNIC (100Mbps). Also, made sure that the source host (local disk on westwood) and the destination host (shared iSCSI LUN served from Uber on cresskill) had vMotion-enabled vmkernel ports on the same dvSwitch port group.

Once the migration was restarted, the stats were showing 30Mbps network throughput outbound (local-to-shared) and closer to 50Mbps inbound. More testing of the network configuration is needed to verify why vMotion defaulted to the slow network.

VMware Tools on Uber(s)

While both Uber and Uber2 now have NFS and iSCSI shared storage, a bottleneck was noticed on the network. With iSCSI configured on Uber, the initial cold Storage vMotion of a Windows Server 2008 VM (10GB partition) took the better part of 4 hours to complete.

Looking at network performance, the NIC seemed to be peaking at about 3Mbps, which calculated out correctly with the speed of the migration. At the moment, the E1000 virtual NIC is being used on Linux.

It is well documented that VMware Tools has the VMXNET network drivers that are optimized for VM's. While there is no built package for VMware Tools on Ubuntu, VMware did release an open source version, but this needs to be built.

Some searching found this procedure:

http://www.gorillapond.com/2009/03/09/install-vmware-tools-on-ubuntu-linux-reborn/

Once again, Uber2 was used for the test install of VMware Tools and then Uber was updated.

After the code install and before changing the NIC card, network throughput while copying ISO files from Uber to Uber2 improved slightly to about 3.8Mbps. Once the VMXNET 3 NIC was configured, this jumped up to 7.7Mbps.

To test this with a VMware function, another Ubuntu server VM with 16GB of thick storage was Storage vMotion'ed from shared storage (Uber on cresskill) to local ESXi host storage (local disk on westwood). This cold Storage vMotion took about 6 minutes.

Will redo (cold migration) the initial Win2k8001 test to see a real apples-to-apples test, but there was a significant improvement after VMware Tools. Also, going to look at "ethtool" which can set the link speed. The NIC cards could still be running at 10Mbps.

iSCSI on Uber

Uber, the primary Ubuntu VM housing shared storage for the NY100 Data Center is serving shared storage for ISO images and VM's via NFS on separate Datastores.

Uber2 VM was used to validate the configuration of an iSCSI target server on Ubuntu. Once the testing was completed successfully, the final process was to port the configuration to Uber.

Uber now has a 100GB image file sitting on its 200GB vHard Drive (/dev/sdc). This 100GB sparse file was configured in iSCSI as LUN0. Once the LUN was configured, iSCSI target started.

On Northvale (one ESXi host in the Towers1 cluster), a new iSCSI storage adapter was configured and the 100GB iSCSI LUN on Uber was discovered . A new Datastore was added to northvale, formatted as VMFS-3, and this Datastore will be added to all hosts in the NY100 Data Center.

The plan is to move all VM storage from:

NFS - Datastore name shared-VM-store01
to
iSCSI - Datastore name shared-iscsci-store01

Will test Cold migration first, and then live Storage vMotion.

The build ISO images on the Datastore named shared-ISO-store01 will continue to be served via NFS.

Uber2 iSCSI working

Uber2 now has a virtual 200GB partition which is being shared as an iSCSI target. iSCSI initiators on ESXi Hosts in the Towers1 Cluster can now discover the LUNs on Uber2 and connect to them.

Next will be to set this up on Uber, adding a 100GB partition, configuring the Ubuntu iSCSI initiator, and testing DRS from iSCSI. Once complete, move all Cluster aware VM's to iSCSI and only use NFS for the ISO share.

The logistics of moving cresskill VM's to thin provisioning is still on the list.

Space issues

All VM's in the NY100 datacenter were created with Thick provisioning. This is beginning to cause space issues, mainly on cresskill, where the Uber VM is providing virtual shared storage for the Towers Cluster. The conversion from Thick to thin can be performed by "Storage vMotion", as one of the option in the wizard is to change the disk provisioning. Cloning would also work.

The clone function was tried first, on Uber2 (iSCSI test Ubuntu). Once cloned, one of the NIC cards would not load properly in Ubuntu, even though both cards were present in the VM's settings. After some of the following commands, the 2nd NIC was up:

modprobe
lshw -C network (showed 1 NIC disabled)
iwconfig (showed the actual selected "eth" numbers for the NICs)

Once the correct eth number was placed in /etc/network/interfaces, Uber2 was back in business.

The storage vMotion method was used going forward.

Cold vMotion of VCS01, red-bank, and Trenton into the Towers1 cluster will be needed in order to allow enough space for Uber to be migrated, since Uber has a 200GB partition for shared storage.

Uber2 iSCSI almost configured

The 2nd utility test VM (Ubuntu) was ready for install of the iscsitarget services. Once installed, the configuration of the ietd.conf file went quickly. A 2nd vHard drive of 8GB was created as a test LUN.

While the ESXi server initiators could see the target under discovery and Rescan, there was an issue with the format. Will figure this out tomorrow.

VMware Update Manager installed

VUM was installed on vcs01, using red-bank's database engine. With the work moving vCenter off of MSDE, the VUM configuration to SQL was trivial.

vcs01.crm.netx was selected as the point of install for VUM and we ran into some issues trying to install the VUM plug-in (name resolution). The vSphere client machine was not using the AD DNS pointer. Once this was "resolved", the plugin went right in.

More configuration work is needed before attempting to let VUM do an upgrade on a newly built VM.

The ESXi servers are already up to date.

Uber down overnight

During the backup maintenance to and from Uber (Ubuntu VM that holds cluster shared storage and ISO files) and the XP Physical machine (currently named tibet77 with 1TB raid), a large copy process hung Uber and the reboot caused it to hang while running fsck. It was left over night and the reboot in the morning seemed to fix the issue. The vCenter logs are still showing connectivity problems to tibet77.

More work needed on the utility002 testing of iSCSI and the possible replacement of tibet77 with a Ubuntu build and enabling all three protocols available (SMB, NFS, & iSCSI).

Maintenance before VUM

Working on some maintenance items in the NY100 vSphere Data Center. Since a number of items moved forward today, it is time to do some backups, grab additional ISO's and files for Uber's shared storage, and get ready for iSCSI testing on Utility002 VM (Uber2?). This VM is currently sitting in the Towers1 vSphere cluster so it will be cold vMotion'ed to cresskill.

Also, VUM will be installed and a few VM's will be updated with this tool. Tomorrow, a webinar is scheduled discussing vSphere monitoring.

SQL Server auth, scheduling, Limited disk in VCS01

VCS01 and red-bank VM's on cresskill are now members of AD and a domain id was created for SQL connections.

The process of getting vCenter to talk to SQL Server turned out to be very exciting. In the end, the database for vcs is on red-bank (SQL Server VM). Obviously, VM snapshots played a large roll in the testing and troubleshooting.

"vCenter server service" and "vCenter web service" were configured to logon with the new SQL domain id.

The SQL Express service dependency that was added previously to the vCenter services was removed from both services allowing the local SQL Express services to be disabled, reducing the required resources on the vCenter VM.

The shutdown/startup scheduling was also tweaked on cresskill. Generally speaking, the process is to manually shut all VM's down before cresskill is brought down, but the restart of the VM's should be automatic and orderly (cresskill is not part of the DRS cluster and is a utility ESXi host).

Finally, the 8GB virtual disk (8GB C: partition) created for VCS01 was down to 1GB free. After increasing the virtual disk to 16GB, Dell's extpart.exe was used grow the C: partition it to 16 GB. Now that the vCenter database is on the SQL VM, VCS01 should not grow much.

The crm.netx Domain; cresskill maintenance

Trenton is the lone Domain Controller (DC) and DNS is configured on it. VCS01 has been added to the domain (crm.netx) but still has issues connecting to red-bank's SQL engine, so vCenter was pointed back to VCS01's SQL Express instance to continue testing (good exercise, backtracking and making sure the local DB connection still works).

Performance trends will provide useful information on cresskill with all 4 VM's up. Start up and shutdown schedules are also being tweaked to ensure a smooth build up and tear down. Here is the startup sequence:

- Uber (utility Linux) needs to be up to make shared storage available
- Trenton needs to start next to make the AD domain available
- red-bank needs to be up next in order to have SQL 2005 available for clients
- VCS01 needs to be up to activate:
......- the vSphere DataCenter
......- the Towers1 Cluster
......- the Distributed virtual Switch (DvS)

The startup timing will move after I run through several reboot cycles of cresskill.

Active Directory

Another win2k3 VM was built to house a Domain Controller for the environment. The new server (named Trenton) will also house the DNS role, forwarding requests to the internet router.

This VM server, along with VCS01, red-bank, and utility01 will all sit on the ESXi host cresskill, which is not part of the vSphere Cluster "The Towers1" and will not participate in DRS, HA, etc..

VCS01 and red-bank will join AD but will eventually use the backdoor gig-e for vCenter communications.

Issues with the vCenter DB connectivity

After installing SQL server 2005 on red-bank (new win2k3 server), it was time to move the vCenter files from VCS01's MSDE to red-bank:

- Shut down vCenter Server Services
- Detach the vCenter DB from MSDE
- Copy the two VCS files to Red-bank's SQL data folder
- Attach the DB to SQL Server 2005

Once completed, the ODBC connection needed to be updated to point at red-bank (admin Tools -> data Sources (ODBC)). While ODBC testing was successful, vCenter would not start, which is probably related to server service authentication from VCS01 to red-bank.

Will move forward on building AD such that authentication is centralized.

SQL Server, iSCSI test server

The new Win2k3 Server VM (named Red-bank) that will house SQL Server 2005 is up and running. The next step is to install SQL Server 2005 on it and then migrate the current vCenter DB from MSDE.

A new Ubuntu server VM (named "utility002") is up and the iSCSI target code is installed. A small (8GB) second vHard Drive was created for the testing. NFS client was also configured such that files from shared ISO's on utility001 could be pulled.

Finally, a webinar on "Deploying Exchange 2010 on vSphere" is giving the team more depth in deploying VMware for this new Enterprise messaging solution.

The week's Agenda

Here are a few items for this week:

- Move vCenter Server database from MSDE to SQL Server (VM name = Red-Bank)
- Build an Active Directory domain with DNS (and possibly DHCP)
- Load and configure WMware Update Manager (VUM)
- Fully vet the particulars of vShpere 4.1 install
- Test Linux iSCSI and determine if cresskill should move off of NFS shared storage
- Begin testing HA

ESX 3.5 Install

The install of ESX 3.5 completed but it took close to 1 hour to boot up and it still was not accessible. The host CPU pegged at 100 throughout bootup.

A reinstall may be attempted again on cresskill with all VM's off (cresskill has 6 GB of physical RAM).

DRS in action

On the Towers Cluster, 3 VM's are running on westwood (win2kmove, win2k8001, testwin2k) with no VM's running on northvale. Distributed Resource Scheduling (DRS) thresholds are set to "moderately aggressive".

win2k8001 is a windows Server 2008 x64 VM.

A small looping vbscript program is placed on the win2kmove VM. Each instance of this program will continuously run 3 million sine calculations. This will busy the cpu on westwood to simulate heavy activity.

Four of these looping scripts are started on VM win2kmove, which causes westwood's CPU to raise to over 60%. This exceeds the DRS thresholds and within 4 minutes, vSphere's DRS moves VM win2k8001 from westwood to northvale. At the moment, there are no specific resource pools defined.

This test confirms DRS functionality.

vMotion, ESX 3.5 in a VM

Continuing to test manual vMotion and changing configuration to see the results.

Since we were unable to load ESX 3.5 onto any bare metal and the old server has not been put back together, an attempt will be made to load ESX 3.5 into a VM. This is just so that we can try an upgrade to 4.0 on it.

The ESX 3.5 VM install went fine but the boot process has taken a large amount of time, as the CPU is pegged on the ESXi 4 host. Early in this process all other active VM's were vMotion'ed off of this host to give it maximum resources.

Migration, vMotion, Uber maintenance, etc.

After vMotioning VM's back and forth from westwood to northvale to make sure the configuration is bullet proof, I moved my vCenter Server VM over from westwood to cresskill (utility host) and put vCenter's files in the shared storage on NFS via the utility VM (lets call this VM Uber, the Ubuntu build).

In my testing haste, I did shutdown Uber while vCenter had its files on shared storage! After realizing this, I powered Uber back up. While vCenter's Win2k3 event logs showed the missing disk, the server came back alive.

I am now thinking it is better to use the local cresskill host datastore such that I can shut down Uber while I am still logged into vCenter. I just activated Storage vMotion to move vCenter's files. The down side is of course single point of failure (will resolve this next week).

I am getting yellow and red warnings on Uber about high CPU and memory usage. This is partially being caused by the migration vCenter's files to local storage on cresskill. Also, various Cluster VM's continue to read & write to Uber's shared storage.

Uber currently has 1.5 GB of virtual memory. I will push it to 2.5 GB shortly. This Ubuntu VM will be a memory and network workhorse going forward since I plan to put the files from most VM's on it.

I brought up my legacy XP utility machine to do some data backups onto its 1 TB Raid 1 array. Tomorrow, DRS testing and building a few more VM's, including x64 builds. Next week, make vCenter Server Fault Tolerant. Once done, consider making Uber FT.

Host vMotion successful

After much digging into CPUID and vSphere EVC mode, I was able to properly configure the cluster in EVC mode with two of my three hosts.

I am now able to successfully vMotion a Win2k3 VM from westwood to northvale (ESXi host names) and back again, leaving the network and storage as shared.

With the Win2k3 VM not doing any real work, the vMotion takes less than 1 minute. I am going to run some spin type scripts on the VM to see if I can detect any hiccups.

During the vMotion migration wizard, if I select cresskill (my ESXi utility host) I get a CPU incompatibility message. This host has a Pentium D, while the other two have Core 2 CPU's of different flavors.

vSphere Maintenance

Now that all three ESXi Hosts are joined to The Towers vSphere Data Center, I am using the vSphere Host Update Utility to make sure that all hosts are patched to the same level from the base (vSphere 4.0 Update 1).

I have been trying to figure out a way to move the vCenter Server VM (Win2k3) to my Utility host but there is an obvious catch-22 doing that, since vCenter needs to be active in order to do a cold migrate and I am still fighting the war of CPU versions for vMotion to function.

During patching of the esxi-cresskill-175 (new name for the Dell 745 utility host), I realized that the only way to do this is to log into esxi-westwood-140 (new name for Dell 780 host), shutdown vCenter Server and make a local Datastore copy of the files. Then, copy the new folder out to the new NFS shared storage on cresskill once vCenter is back up.

In the future, things like vCenter will always sit on shared storage.

For future reference, the new build host with the Intel D975XBX motherboard and Intel E6700 CPU will now be referred to as esxi-northvale-195 or "northvale" for short.

New Host, more on NFS, still no vMotion

The new machine (we can call it northvale for now) is fully functional as a ESXi host with access to the internet and the backdoor Gig-e network.

NFS fun
One of the plans was to build a Linux VM and serve shared storage from it over NFS. I now have a 200 GB NFS mount off of the Linux VM talking to all hosts over the Gig-e. I can add additional storage as needed from the local datastore where this VM resides.

This will simulate having shared storage from a remote server: Virtual Shared DASD (VSD). I did a cold move of this VM to the Dell 745 which will probably become a test/utility ESXi host (desktop machine) until I repurpose the machine to Windows 7 x64 later in the year.

vMotion
vMotion is still elusive based on the two ESXi hosts in my cluster (CPU's are E6700 & E8400). More digging....More RTFM....

Hardware
Finally for today, the 3rd and last hard drive arrived. Hopefully, this one will actually work!

First Boot

After assembling the new machine, I decided to wait on installing the PCI Intel Pro 1000 (which will connect to the back door gig-e) until I get power up and until the machine looks like it will work. It has an on board NIC that I can use for management access.

First boot saw the PXE try to find its foot (no go) then no boot image. A few clicks in the bios and a ESX 3.5 CD in the DVD and the box beagn loading ESX. As I saw with my Dell 745, 3.5 seems none too happy with these new boxes.

So I removed the 3.5 CD and inserted a ESXi 4.0 Update 1 CD and quickly installed it and a configured a static IP. Logging into the ESXi from vSphere client told me everything looks good.

Moving on to vCenter to add this new host to the Towers Data Center.

New ESXi hardware is in

All the components of the new build are in. This new machine will have power up tomorrow.

ESX 3.5 Upgrade wil not happen today

I was all set to install ESX 3.5 on my Dell 745 until....the install does not recognize the SATA controller. OK, so I did not scrutinize the 3.5 HCL. Oh well, time to rebuild my old and slow IDE chassis and just have a 3.5 box in the vSphere Towers cluster for drill.

Hardware is enjoyable

I received the replacement hard drive yesterday for the original Dell 745 (the box without VT on the CPU). The new drive was also bad.

Fortunately, since I needed a hard drive for the new motherboard, I bought and received two 500GB SATA drives so I had a spare. I had already put one of those drives in the Dell machine and reinstalled ESXi 4, so I will put that back into the machine and will proceed with a fresh ESX 3.5 install.

The supplier apologized, said they would pull another drive, test it, and send.

NFS, SMB, and ESX 3.5 Upgrade

Still waiting for parts for the new build (mid-week??) so I moved forward on my Linux VM NFS testing.

My Linux utility VM now has an SMB mount point (client) from my Utility XP machine and some of the build ISO's have been copied over. Also, a 2nd vHard drive was added to the Linux VM to accommodate the ISO's.

One thing that is very clear: copying files from XP to Linux VM over the back door gig-e is much faster than copying from XP to the ESXi datastores. I have not looked closely with a sniffer to determine why this is, but it is something for the list.

An NFS mount point was added to Linux VM and a new Datastore was added to the Towers Cluster. A Win2k3 build was performed to test this scenario. Obviously, with the new NFS mount point sitting on a VM on the virtual Distributed switch, the Win2k3 build happens much faster. Wait until sysprep in configured!

Finally, the ISO for ESX 3.5 was procured from a colleague to allow testing of ESX 3.5 to ESXi 4 upgrade. This will happen in the next day or two.

First ESXi machine is awake

Some parts have arrived for the new build.

After replacing the hard drive in the first ESXi machine, I installed ESXi 4.0 Update 1 and then joined the machine to the Towers cluster. The drive I installed is not the hard drive that is being sent as a replacement but it moves me off the dime so I can work on that machine again.

No comment on the new machine, as I now have several components that are not compatible. Sometimes it is easier to just buy a machine!!

Testing iSCSI and NFS on Linux

While I wait for hardware to arrive for the new ESXi build and hard drive replacement in the first ESXi host, I have turned my attention back to my utility machine "tibet77" (XP, NFS, StarWind iSCSI target).

There still seem to be hiccups when accessing the 1TB raid array from ESXi, whether via NFS or iSCSI.

I installed a few Ubuntu VM's to begin the process of building a test utility machine on Linux running the NFS and iSCSI services. Once I have this down, I will install Ubuntu onto tibet77. Of course, the new build will need to talk turkey to the 1TB Raid array which is formatted NTFS today.

I would rather not archive the data, reformat as ext3, then restore. We will see how successful I will be at that.

ESXi Host Hard Drive failure - New Motherboard

Hearing some "ticking" from my Dell 745 resolved to be the hard drive going south. A new drive is supposedly being sent by my supplier. The rebuild should not take too long and this box was going to be used to test ESXi 3.5 to 4.0 upgrade anyway.

I have a new motherboard coming in order to bring a 3rd ESXi Host into the Towers vSphere cluster, which will go into a spare case I have in the lab.

Mother Board replacement and ESXi

One of my clients on the west coast is considering replacing the motherboard in one of their ESXi servers with a much faster one and asked the question "Will ESXi still boot and function properly?". This is a standalone ESXi server. The existing motherboard has an older Core 2 Duo and the new one has an i7.

Well the short answer is "probably not, without some work". Probably better in a larger environment to use a host profile, rebuild the host after the motherboard upgrade, then restore the profile.

The client chose the motherboard ad had a local computer shop do the swap. Once the motherboard was replaced, ESXi came up but had no network. A little investigation showed that the replacement motherboard (Asus P6T) has an unsupported network card. I recommended that the client buy a Intel Pro 1000 Desktop card. Once installed, ESXi recognized the card and a little console configuration brought the host back to life. More details later.

Errors on the Storage utility box

I am still seeing intermittent errors on the utility machine that houses NFS and iSCSI target storage. I am tempted to move this off of Win XP (NAV firewalls, etc.) and load Linux to make sure I don't have issues with XP's "Protectionism".

Will probably load up a Linux VM to get the entire config down first.

DRS testing

Cold VM startup functions in DRS are working properly, meaning that when I properly configure a VM on shared switch and shared storage, DRS decides which host to start the VM on and this will change based on the available resources (CPU, memory). So I can make a vMotion test VM start on one host and then stop it, start up 3 more VM's on that box, and the next time I start the vMotion test VM, it starts on the other host.

Log entry below (the test vm is called "win2kmove" and the vSphere cluster I created is called "The Towers"):

Relocating Win2kmove in The Towers from 192.168.1.140 to 192.168.1.175
info 7/28/2010 11:07:59 PM Power On virtual machine Win2kmove

Cold vMotion

I was able to successfully move a Linux VM from one host to another via cold migration. However, my first ESXi host has some issues, so no live vMotion yet....

Will perform a cold Storage vMotion next.

StarWind iSCSI solution issues

I am still fighting with my iSCSI configuration on my utility machine. Everything is configured and both host can see the iSCSI LUNS but I am seeing tons of connectivity errors in the cluster logs. I have NFS running on this box, so this could be an issue. More research...

Down time is over - Windows Server 2008 R2

Have been underwater the last week or so, preparing for the VCP exam. Now that the test is out of the way, back to cases.

Installing Windows Server 2008 R2 (64 bit) as a VM is pretty trivial, except that it comes up without a network.

After a few clicks, I remembered reading that VMware Tools needs to be installed in order to load the AMD driver. So, keep that in mind.

Access to iSCSI on Utility machine

I am seeing some errors in the VCS logs about my iSCSI Datastores going up and down. I am not sure what this is but it might have something to do with XP/Norton on the utility machine and their restrictive firewalls. Need to look into this.

VMware Server changes subnet of Host-Only network after reinstall

A client of mine in California ran into an issue with VMware Server.

A VMWare server instance runs on a Window 7 x64 with a couple of WindowsServer 2008's running inside. These guest machines run on a host only network.

The underlying Windows 7 OS had to be re-installed and after re-installing VMWare Server (and the console plug-in) the machines would initialize but the console came up black.

After determining that this was not related to the Host-Only network, a reboot of the Windows 7 box made the console work again.

Then we discovered that the previous Host-Only subnet had been changed after the re-install (291.168.112.128 --> 192.168.29.128).

After changing the static IP address in the guest machine, all was right with the world again (until the next hiccup).

Problems with vCenter Server Service

My vCenter Server VM (VCS VM) is not currently a member of AD (later), but I have been having a problem with the VCS service and VC Web Service not starting after a reboot of the VCS VM. The VC Web Services already has a dependency on VCS starting before it can start, which is why it won't start.

A peruse of the event logs show that VCS service is trying to log into the SQL express DB before it is ready. To resolve the issue I went into the registry and added the SQL Express service as a dependency on both the VCS service and the VC Web Services service:

Launch Regedit
Navigate to \HKLM\CurrentControlSet\Services

Under the vpxd key on the right, double click on the value DependOnService
press Esc (The data in the dialog might be highlighted)
Add "MSSQL$SQLEXP_VIM" to the bottom
Click OK

Under the vctomcat key on the right, double click on the value DependOnService
press Esc (The data in the dialog might be highlighted)
Add "MSSQL$SQLEXP_VIM" to the bottom
Click OK


**

After a reboot, the VCS services now start properly. This dependency will be removed once the SQL Server 2005 VM is functional and I move the VCS database to it (future).

VM's inaccessible

I had powered down my utility machine (XP w/NFS & iSCSI) last night for another task. This morning, one of my ESXi Hosts was missing. Once reconnected (Connect command from right click of host), two of my VM's were inaccessible.

Of course, the Datastore location of the two VM's was on that machine over iSCSI....Have to remember, Datacenters are always up!

Clustering

I am able to build a vSphere Cluster, add my real and "imagined" ESXi Hosts to it and enable HA and DRS. Did some testing of Fault Tolerance of a VM as well. Created a few Resource Pools in the Cluster. It will be a day or two before I have time to test these features.

Noticed that you cannot create a resource pool in vCenter Server without HA being enabled.

Realization - ESXi inside of ESXi

It is not possible to run a VM inside a VM (doh!). So, the dream of loading ESXi inside of ESXi and loading VM's under this child VM is done. Thanks for playing....

Doing an upgrade from ESXi 3.5 to 4 is still possible, but there are other hurdles to that.

ESXi turns in on itself

There was no easy way of getting around the issue of NO VT support on my Dell 745. I had to break down and get another machine....or go through the process of trying to replace the CPU, which I had no appetite for.

So I now have a Dell 780. It will run ESXi 4 Update 1 and allow me to not only install 64 bit guest OS's but will also support installing ESXi inside ESXi to have multiple Hosts. Theoretically, I could install two VM's of ESXi and have a full vSphere Cluster running inside of an ESXi host. This work should happen in the next few days.

study...study...study...

During my virtual process, I am studying for the VCP 4 exam. Much of the material is difficult to master without actually configuring the products, which is another reason for setting all this up. There is also some memorization involved.

Much of the VMware code can be downloaded and installed on a trial basis which gives the little guys the ability to fully test and learn without that pesky 2nd mortgage being taken out.

For those on a similar track, check out VMware Whitebox for hardware choices.

Datastore for ISO sharing

Needed a place to put ISO images for loading VM guest OS's. I know my old Win2k machine could support NFS and I need to move off of Win2k anyway. The Win2k machine is pretty old so it would make a good utility machine.

I used Microsoft's Services for Unix (SFU) some years back while consulting at IBM working on Citrix. SFU does support NFS services. Also, I wanted a RAID location for storage and backup of various data.

The VMware best practice is to access IP Datastores on a separate high speed network between ESXi and the ISO share. Theoretically, no internet traffic is needed over this link.

Get ready
Install a Intel Dual Gig-e card in ESXi
Upgrade Win2k to WinXP
Install a Intel Pro 1000 into XP
Connect the two gig-e cards to a gig-e switch

Make space
Install SATA RAID card and two 1 TB SATA drives in XP
Set the drives to RAID 1 (mirror)
Dump all the Windows, Linux, VMware, etc. ISO's in a folder

Here comes NFS
Download and install MS Services for Unix (SFU) 3.0 on XP
NFS share out a folder on the raid array

Issues with permissions in SFU
I ran into some trouble getting ESXi to properly connect to the NFS mount point. After some searching, it turns out I needed to pull the "/etc/passwd" and "etc/group" files from ESXi and place them on XP in order for the user/group mapping to work properly.

To get them from ESXi back to XP, I had to log into the console of ESXi and copy the files to the exposed ESXi Datastore directory, then go into the vSphere client, browse to the folder on the Datastore where I placed the files and then download them to XP machine.

Once these were in place, I could add the NFS mount point as a Datastore in ESXi, over the gig-e network.


**
Now, I have a 1 TB Datastore where all my ISO's sit, as well as any other files needed by Windows, Linux, or ESXi.

Later, I will talk about using this same storage system to test and use an iSCSI Datastore.

The new machine works great, but.....

Loading up ESXi onto the new machine was trivial, showing that VMware has done a very good job building the product. I was creating VM's within 30 minutes.

After some playing around and reading, I wondered if I could create a ESXi inside of ESXi to fully test the advanced features (VMotion, DRS, etc.). Of course I must have missed a meeting, because the box I have has a processor that does not have the Intel VT feature, which means, even though ESXi (64 bit) does install, I cannot load any 64 bit applications on it, including ESXi.

This is an issue, but I moved on configuring everything I could.

VMware - Early returns

After deciding to move forward on becoming proficient in vSphere, a new machine was needed to load ESXi 4 on. Some searching turned up an alternate hardware plan than using the vendor's HCL, which would require a second mortgage.

The Whitebox site lists hardware that, while not on the HCL, will work with ESXi:

http://www.vm-help.com//esx40i/esx40_whitebox_HCL.php

From that, I acquired a refurbished Dell Optiplex 745 desktop box.