Thursday, September 30, 2010

Windows 7 x64

We are installing a few Windows 7 x64 VM to continue testing VMware View 4.5.

Due to some limitations on the previously decommissioned cresskill, only Windows 7 x32 will go on this puppy.

Wednesday, September 29, 2010

Maintenance Day

Today will see backups of vCenter, Installation of a (additional) 1TB drive in westwood, the decommissioning of cresskill (utility ESXi 4.1 host), and some data backups.

The additional 1TB of space will be used for additional VMFS space as well as for the testing of OpenFiler for iSCSI shared storage.

Tuesday, September 28, 2010

Uber backdoor Gig-E not up

Ran into a minor issue with Uber that has been here ever since Uber was moved from cresskill to northvale ESXi hosts. Decided to look into it.

The symptom was that northvale (or any of the ESXi hosts) never recognized shared storage on Uber and showed the Datastores as grayed out. This would also cause the VM's using the shared storage to be disconnected and inaccessible.

After some investigation this morning, it was found that the 2nd network card in Uber VM, that connects the iSCSI and NFS shared storage to the back door Gig-e would not come up without a forced sudo ifup eth7 command to bring it up.

We first attributed this to a race condition of the northvale host coming up completely and doing it's scan for storage before the launch of Uber. This may still be the case for iSCSI.....

Anyway, for those following along, you will remember that all VM's were moved from the utility host (cresskill) to northvale since we needed to move to Win2k8 x64 for vCenter Server and cresskill does not support x64 VM's.

Uber was the last to be moved. When a Ubuntu VM is moved, the NIC cards become different eth numbers. Some fun was had (and documented) getting the correct eth numbers set on Uber. The file "/etc/network/interfaces" on Uber had to be modified to update the eth numbers after the move. Of course, there was a typo in the update for eth7, causing it to fail to load at boot.

Once corrected, Uber's back door Gig-e was loading properly. After a reboot, northvale now recognizes the NFS and iSCSI shares from Uber once Uber is up.

Monday, September 27, 2010

PCoIP working properly

Now that the agent issues are resolved, some preliminary testing shows that the PCoIP issues experienced with View 4.0+ are resolved. Obviously, more testing is needed to make sure other issues have not popped up.

View vDesktop not recognized

After some testing of the failed vDesktop, we were unable to find anything wrong with the installation of the agent.

Finally, the primary connection server was bounced and the vDesktop agent was recognized. This is not a good story, saying that when a new vdesktop is provisioned, the only way to add it to a pools is to restart the primary connection server. More testing is needed to validate this.

VMware View 4.5

Moving forward on the VMware upgrades, the primary connection server (Piermont) was updated to version 4.5. This update went in with much fanfare and the new GUI web site seems to be much better designed and has faster response than the 4.0 version.

All settings (Pools, Desktops, etc.) and configurations seem to have transferred properly. However, booting on of the vDesktops was vary slow so we reverted to a snapshot, which may have caused some recognition issues inside of View 4.5.

The 4.5 agent was removed and reinstalled, but View is still unable to recognize the agent. More troubleshooting is needed.

Thursday, September 23, 2010

Citrix XenCenter

Once on the console of win2kmove1 VM, we surfed to the XenServer web server and attempted to install the XenCenter app, where we received a pop-up stating that .NET Framework 2.0 SP1 was required.

After installing this piece, the same error is being reported. a little searching found that installing .NET Framework 3.5 SP1 allowed us to install XenCenter.

Launching the XenCenter app, we were able to add XenServer, however, the server does not have access to the back door Gig-e, where the shared storage resides. Will need to add a 2nd vNIC to XenServer for this to work.

Citrix Xen Server

A small side virtualization project will be to load Xen Sever into a VM and test some of these features.

Xen Server is Citrix's virtual host platform.

Xen Server took some 50 minutes to install in a VM, but it did complete. An existing VM (win2kmove1) will have the XenCenter application installed on it.

Wednesday, September 22, 2010

vSphere maintenance items

VMkernel Management network
There has been some discussion about moving all management network operations to the back door Gig-e. One main impetus for this is the fact that a number of operations involve moving data from one datastore to another, whether local or on a VM (NFS and iSCSI shared storage runs on the Uber VM today). Normally, vSphere does some of this over the slow network and there seems to be no easy way to predict which network will be used.

Moving everything over will ensure that all of these activities will happen at the speed of Gig-e, as opposed to some of them happening over the 100 Mbps Production network that is connected to the internet. This also frees up the Production network for ISO downloads, etc.

Bridge Network
The other item is the physical location of the vSphere environment. Moving these Hosts would require a bridge network for access to the internet as well as vSphere client access to the environment.

Planning for both of these projects is underway and finalized procedures should be in the can by later next week.

In the mean time, new feature testing and the VMware view 4.5 upgrades will continue.

Tuesday, September 21, 2010

VMware Tools updates for Wintel

Now that all three ESXi hosts have been updated to vSphere 4.1 and the licenses have been re-applied, we will turn our attention to the maintenance work of updating VMware Tools on all the Wintel VM's.

Alpine, Trenton, and red-bank VM's on northvale are now updated, with the VM's on westwood to happen today. This includes all VMware View VM's. These VM's were set to Manual in DRS such that no automated movement would happen during View 4.0 testing.

The old Wintel VM's that sat on cresskill are now duplicates, but Uber (Ubuntu) VM is still housed there, so it will get updates tomorrow.

Monday, September 20, 2010

Upgrade the ESXi Hosts to 4.1

After additional reading in the vSphere 4.1 upgrade guide, we planned to update the ESXi hosts (cresskill, westwood, and northvale). The 4.0-4.1 update zip file was downloaded from VMware and placed on a local hard drive.

Plan A was to use Update Manager (VUM). The update file was imported into Update Manager, a baseline was configured, and cresskill was chosen as the first host to be updated. Once the VUM configuration was set, we kicked off the update. It failed within 5 minutes due to the fact that we are using eval licenses that are not supported for update through VUM. On to Plan B.....

Plan B was to use the vCLI and the vihostupdate script to perform the updates. Getting the syntax correct took a little work but in the end, all hosts were updated to 4.1.
The "vihostupdate" query command was run on each host and they each displayed the following:

ESXi410-GA 2010-09-20T20:44:10 ESXi upgrade Bulletin

ESXi410-GA-esxupdate 2010-09-20T21:18:08 ESXi pre-upgrade Bulletin



We may have an issue later due to the order of the commands run (esxupdate run after GA), but there seems to be no immediate concern.

vCenter Server 4.1 is being brought back up and testing will resume tomorrow.

Friday, September 17, 2010

VMware Update Manager 4.1 (re-install)

After completing the vCenter Server 4.1 install and moving the key VM's around, it was noticed that the VMware Update Manager Plug-in was failing to start. Seems that the plug-in was still pointing to VUM on the vCenter Server 4.0 instance.

A re-install of VUM on alpine was needed. Once this was installed, the Update Manager plug-in can be properly enabled.


Weekend Update - 09/16

At the current moment, here is where we are in vSphere:

  • Three ESXi 4.0 Hosts
  • vCenter Server 4.1
  • VMware Update Manager 4.1
  • VMware View 4.0

Items for next week:

  • Upgrade the ESXi Hosts to 4.1
  • Upgrade VMware Tools on all VM's (including Uber)
  • Network configuration change planning
  • Upgrade VMware View to 4.5

vSphere 4.1 topology

vSphere 4.1 topology #1


vSphere 4.1 topology #2

To give more context to the text here, we are uploading a few diagrams. Above are links to the current layout in vCenter Server and the map. These are now on the upper right side of the main blog page.

Thursday, September 16, 2010

Moving VM's - the hard way

On the cresskill ESXi Host, the VM's Red-Bank (SQL Server 2005), Trenton (Win2k3 AD), and Uber (Ubuntu Linux shared storage) all need to migrate to the Towers1 cluster, preferably onto northvale ESXi Host, where alpine VM (vCenter 4.1) resides.

In other environments, the movement of VM's from one host to another would be relatively trivial, as vMotion and Storage vMotion would be the standard solution. However, since cresskill, the utility ESXi host, does not have a supported CPU (no VT) to join the Towers1 cluster and be available for these VMware functions, the live migration features will not work.

The other caveat is a catch-22. Some of the VM's that need to be moved are active participants in vSphere or Active Directory and can not be brought down, even for cold migration, without bringing down those services while the move happens.

The solution to this challenge is cloning. Cloning VM's creates a mirrored snapshot of the original live VM files, in a location of your choosing, and creates a new VM object pointing at the new files.

To that end, VM's Trenton and red-bank were cloned (Trenton2; red-bank2) into the Towers1 cluster, the original VM's were shut down and the new clones were brought up. For red-bank2, vCenter Server 4.1 service was stopped before the switch, since its database is on red-bank. Trenton2's AD was more forgiving, allowing for the absence of directory services while the switch was made.

Due to the size of Uber VM's shared storage (close to 250 GB), this clone will run over night, even on the Gig-e.

vCenter Server 4.1 - Part Deux

After some RTFM in the 4.1 install and upgrade guides, it was clear that the best approach for upgrading to 4.1 is to back up the 4.0 database on red-bank and then install vCenter 4.1, pointing it at the existing DB. This would ensure that all the configuration of the current vSphere 4.0 environment will be migrated properly.

Here is the process:

Shut down the vCenter Services on VCS01 (vCenter Server 4.0)
Backup the VIM_VCDB database on Red-Bank
Create a x64 ODBC DSN on Alpine pointing to the above DB
On Alpine VM, begin the install of vCenter Server 4.1 selecting "Connect to SQL Server"
Select "Upgrade Database"

Since the login credentials for the session on Alpine were user credentials that we could not change at install time, we use them. The vCenter Server and VC Web Server services will be logging on as this user. Once the install is complete, we went into Windows services on Alpine and changed the "log in as" credentials to the domain database user (dbuser01, in this case).

vCenter Server , though some were not active.

After logging into vCenter Server 4.1 all
objects were present, however all the ESXi Hosts were disconnected. This was because the hosts are still members of the vCenter 4.0 instance. Right clicking on each host and selecting "Connect" initiated the install of the new vCenter 4.1 host agent and allowed the hosts to be added to vCenter 4.1.

Next is to move some of the VM's on cresskill to the Towers1 cluster.

Wednesday, September 15, 2010

VMware Update Manager 4.1 install

Once we were comfortable with the install of vCenter 4.1, the install of VMware Update Manager 4.1 (VUM) seemed like a quick next step. A new database instance was created on red-bank, an x32 ODBC DSN was created on Alpine, and the install of VUM 4.1 was completed.

The VUM plugin still needs to be installed and activated, but the code is in place. This will be for fixes and probably will not be used for ESXi 4.1 migration (more RTFM on that needed).

ODBC DSN fun
An interesting thing happened on the way to installing VMware Update Manager 4.1 (VUM). Since the VUM 4.1 code installed on Alpine is x32, the ODBC data source DSN for talking to the database needs to be x32! The warning in VMware documentation goes like this:

IMPORTANT: Although you can install the Update Manager server only on 64-bit machines, Update Manager is a 32-bit application and requires a 32-bit DSN.

If you open up the standard ODBC DSN interface on Win2k8 x64, it will only create an x64 DSN.

A few forums highlighted this point, although we did not see the procedure create an x32 ODBC DSN in the admin guide (must have missed a meeting....).

From the forums: On your 64-bit server, create a 32-bit ODBC DSN with odbc32.exe under:

C:\Windows\System32\odbcad32.exe

This brings up the same GUI but it allows you to create a 32 bit DSN.

Once this was created, the VUM install went smoothly.

vSphere 4.1 - ESXi Hosts

As far as the host upgrades are concerned, they will stay on ESXi 4.0 Update 2 and will be migrated to the new vCenter Server before any host upgrades happen. Some planning will need to take place and the vCenter Server 4.0 will need to have its network mirrored on 4.1.

After all ESXi servers are migrated to Alpine's vCenter, VM's can be migrated off the ESXi hosts in order to upgrade them to 4.1.

Since all shared storage is on Uber (Ubuntu VM on cresskill) or the physical server tibet77, the iSCSI and NFS Datastores on these boxes will just need to be rediscovered once cresskill moves. The new vmkernel ports on vCenter Server 4.1's distributed vSwitch will have different IP addresses, but the VM's will retain theirs.

Tuesday, September 14, 2010

vCenter Server 4.1 install (x64)

vCenter Server 4.1 "Proof of Concept" was installed on alpine VM, which is running Windows Server 2008 R2 x64. The POC install used the SQL Express local DB.

As with the last rebuild of VCS under vSphere 4.0 U2, we attempted to duplicate the process of moving the VCS DB over to the standalone SQL VM. However, since the current VCS DB is already on red-bank, the internal name of the DB is the same, preventing us from importing the new DB into SQL Server 2005.

After some reading and a bit of testing to change the name of the new DB, It was determined that it would be easier to just create a new DB instance on red-bank and the point the VCS install at it.

First, vCenter 4.1 was uninstalled from alpine. On red-bank, SQL Server Management Studio was used to create a new database
named VIM41_VCDB. Back on alpine, an x64 ODBC DSN was created to point to the new database on red-bank. Then, vCenter 4.1 was re-installed.

No objects have been created in the new VCS yet, as we need to do some planning in order to migrate cresskill, as well as the ESXi Hosts in the Towers1 cluster, and to make sure all the networking and storage features make it across the divide.


After some reading, this will probably be re-installed.

vSphere 4.1 - vCenter Server

Currently, we are running vSphere 4.0 Update 2 and VMware View 4.0. With the comments in the forums about compatibility issues between different versions of vSphere and View, the best approach seems to be to go to the latest since many of the interim fixes have been applied.

No doubt, there will be more exciting issues in store with the new versions.

vSphere 4.1 and VMware View 4.5 have been placed out on the 1TB raid repository in tibet77, so these will be the next few upgrades. vSphere 4.1 includes vCenter Server and its components as well as ESXi 4.1 ISO for the hosts.

vCenter Server 4.1 requires an x64 Windows O/S, so, as opposed to installing Windows Server 2003 x64 VM for this process, we can utilize one of the Windows Server 2008 R2 x64 VM's that were loaded in the Towers1 cluster for proof of concept. One of these will be renamed "Alpine" for this purpose.

Cresskill, the ESXi utility host which currently houses Active Directory; iSCSI and NFS shared storage; SQL Server 2005; and vCenter Server 4.0, would be the ideal place for the new vCenter. However, this ESXi host cannot host x64 VM's, since the CPU does not support the VT option. Due to this limitation, the Alpine VM will sit in the Towers1 vSphere cluster for now.

Saturday, September 11, 2010

New View Agent testing

All three Desktop-VM's now have the patched View Agent. Testing continues.

Two Desktop-VM's are currently in a VMware View PCoIP Pool. A PCoIP session can be successfully connected from two different clients (Vista x32 laptop; Windows Server 2003 x32). RDP sessions can also be initiated. This validates that the new View Agent code resolves the issue.

Now, where were we??....oh yeah, preparing a standard XP image for Composer....

Also reading a document on Virtual Desktop Manager load balancing.

Connection Broker Testing

While updating the View agents in the background, both Connection Broker VM's have not been powered on. The replica View Connection Broker VM was powered up first. As expected, its Connection Server service failed to start, meaning that there are no CB services available for clients. This tells us that if the primary CB is inaccessible, the entire View service may be down.

Anyway, the primary CB was powered on and the replica's connection server service was restarted. This scenario will be tested further later when we get into VMware View availability and redundancy.

Friday, September 10, 2010

VMware View PCoIP issue resolved

From the initial install on VMware View, accessing View desktop using the PCoIP protocol would fail (the user was still being logged in). However, using the RDP protocol would connect properly to desktop-VM's.

After perusing of the event logs on all 3 devices (client, connection broker, Desktop-VM), the common theme was that something on the Desktop-VM was killing the session connection to the client, after login. I could see the session connection errors in the logs.

Changes were made to the network environment and the available video memory on the agent VM. Several testing scenarios were exercised, but it still seemed to be a Desktop-VM problem.

The issue turned out to be caused by the update of ESXi from 4.0 Update 1 to 4.0 Update 2. This updated VMware Tools for the platform, which created an incompatibility between it and the VMware View Agent, which caused the PCoIP protocol to no longer function.

The solution was to download the patched View agent software, uninstall the current agent and install the patched version. This forum post highlights the issue and the resolution (at least for us):

Upgrading VMware Tools in a virtual desktop causes PCoIP connections to fail

VM dt-xp001 was the first to be patched. There were some issues getting the broker to recognize the new agent (reboot of the broker) but once it was recognized, PCoIP was working properly. We are now able to connect over PCoIP from Vista (x32) and Windows Server 2003 VM (x32).

The other Desktop-VM's will be patched over the weekend.

Thursday, September 9, 2010

VMware Update Manager (VUM)

The errors in vCenter related to VUM are now resolved. A reinstall of VUM did not resolve it, as the VUM plug-in was actually the source of the error.

The domain id provisioned for connecting to the database (SQL Server 2005) was applied to the VMware Update Manager Service on VCS01 (the id is already used for the VMware vCenter Server Service). Previously, the VUM service was using the local system account for VCS01. This is legacy from the reinstall of vCenter last week.

After applying the id and restarting the service, the VUM plug-in could be re-enabled and vSphere fixes were being updated in VUM. A maintenance cycle will be scheduled for early next week to play around more with VUM. vSphere 4.1 is still on the list.

Wednesday, September 8, 2010

More on PCoIP Issues

While RDP testing continues to work properly for VMware View testing, PCoIP protocol is still logging the user in and then disconnecting session (user is logged into the vDesktop). The connection server log is showing this error:

Conveyor 0 IOException. Remote address is null: An existing connection was forcibly closed by the remote host

Recommendations from the communities have ranged from:

Firewall issues
To ensure this is not an issue, we moved all VM's in the VMware View test environment to one ESXi host (Manual Automation Level in DRS), including the client itself. Also, all VM's (CB, desktops) are on the same vStandard Switch on the ESXi host (no policies set).


Group Policy issues stopping port 5002
Created a new OU and moved all the desktop VM's into it.

Video drivers
The latest VMware Tools is installed.

**
The network sniffer is not providing any smoking guns either, so we continue to dig.

Going to use RDP for now to move forward on testing View Composer and the virtual hardware load balancer (NLB). We will come back to the PCoIP issues.

Tuesday, September 7, 2010

New deployment of View desktops

Currently, there are 2 connection servers and 3 XP desktop VM's. These VM's were deployed individually.

Additional applications were installed on dt-xp001 to begin the process of making a master image. Once the install was completed, a snapshot was created, as well as a template for cloning.

Open filer
This appliance VM (called atlantic-city) was installed in the Towers cluster for testing. Since the current vSphere shared storage for vMotion is a home grown Ubuntu Linux VM and is completely custom (Uber serving NFS and iSCSI), Open filer should provide a standardized solution for shared storage.

Other items
Time for maintenance, scheduled for this coming weekend.
  • Backup the vCenter DB's
  • Reload the NFS ISO library repository on Uber
  • Sync up the NFS ISO library with tibet77's ISO share
  • Prepare for the use of VMware Composer to deploy truly shared desktops.

Monday, September 6, 2010

PCoIP - New Desktops

Moving forward with VMware View, RDP connections from the View Client to Desktop Pools works, but PCoIP seems to start the session but will disconnect the client. In View Manager, the PCoIP session is active but disconnected. Opening the console of the Desktop (dt) VM on the vSphere client shows that the View session is in fact active (locked desktop).

Some local LAN DNS issues are cropping up as well, which need to be resolved to ensure these are not attributed to View itself.

More than one Connection Server (CB)
Additional connection servers (called replicas) will be created, and a Proof of Concept "hardware load balancer" (LB) will be deployed to truly simulate multiple CB servers ahead of clients.

RDP testing
From RDP, application and utility launch performance is good but motion video leaves a bit to be desired. This may have something to do with the View client matching the varying resolutions of the target client (Dual monitor with 1260x1080 & 1024x768). At the moment, this is not a critical issue.

Desktop Image creation
A new master will be built on dt-xp001 (first desktop VM) with the install MS Office and a few other real world applications.

Tasks will be to create a new VM template from dt-xp001, new desktop VM's from the template, and begin to create and set Domain level policies (GPO's) to lock down the desktop for users.

Friday, September 3, 2010

More on View

A successful test of VMware View was conducted. We now have a single connection server (piermont) and 3 Windows XP Pro VM's installed as Desktop clients (View Agent installed).

There were a few hitches with the initial test but these are just AD configuration items that will be resolved early next week.

After consulting a colleague, It was confirmed that the View connection servers (CB) are not multi-master, meaning that the 1st one installed is the primary and all additional CB's are replicas. The primary is the one that admins would log into and it contains the ADAM database with the View configuration, which obviates frequent backup to remote storage.

Also, a DNS alias and a hardware load balancer should be used ahead of the clients to ensure View connection availability and redundancy.

Wednesday, September 1, 2010

VM View client

A Windows XP VM was built and a template created from it. From the template, a new VM was built and the View Agent was installed.

Once the desktop users were created and provisioned inside of View Manager, the client was launched, pointed at the connection server, and the single desktop was available. However, the new desktop was not found so there is more RTFM to be done.

Also, the Openfiler [open source] SAN was downloaded. Of course, we rolled out our own shared storage solution (NFS and iSCSI from Uber on cresskill) but Openfiler is the standard test method for such storage so this will go on the list for testing.