Friday, January 27, 2012

Error 0x800F0A12 Installing SP1 on Windows 2008 R2 Veeam System

I ran into a little issue trying to update my Windows 2008 R2 system that serves as my Veeam Backup server.  It did not matter whether I tried the SP1 install from WSUS distribution or the full download from Microsoft, it always ended fairly quickly with the error 0x800F0A12.

A little searching turned up this article on the website.  Following the guidance in the article it became clear the issue was due to disabled auto-mounting of volumes.  If your Veeam server directly access your SAN VMFS volumes, disabling auto-mounting of volumes is a critical step to keep Windows from squashing your VMFS volumes while still allowing Veeam to directly pull data from them.  I performed the following steps as resolution:

  1. Disable any Veeam jobs scheduled to start in the next 90-120 minutes.
  2. Removed all iSCSI Targets and Favorites from the Microsoft iSCSI initiator.  If you have a vendor specific tool be sure to check it as well (Equallogic HIT, etc).
  3. Disabled network access between the Veeam server and the iSCSI san - not taking any risks here.
  4. Followed the article's guidance and ran: mountvol /E to enable auto-mounting of volumes, then rebooted
  5. Installed SP1, Reboot
  6. Ran mountvol /N to disable auto-mounting of volumes, then rebooted
  7. Re-established iSCSI network connectivity
  8. Added back iSCSI Initiator configurations
  9. Verified Veeam could access VMFS volumes (test job)
  10. Re-Enabled all previously disabled Veeam jobs.
If you have any concern about using mountvol versus diskpart to disable / enable automount, they do the same thing.  Check this thread for the details and critical registry key to validate the claim.

Stumble Upon Toolbar

Tuesday, January 3, 2012

Fix Disappearing Windows 7 Desktop Shortcuts

Case of the Houdini Shortcuts
Since moving to Windows 7, a few users in the office have reported many of their desktop shortcuts would periodically disappear.  These reports would come in after users had been working offsite.  The effected users often had more than 10 desktop shortcuts to files out on the network.

Mystery Revealed
Some searching around led me to two resources that answered what was happening and largely how to adjust it.  MS KB978980 explained what was happening at a high level and how to prevent it manually.  Basically the system scheduled task "Scheduled" will delete broken desktop shortcuts of the currently logged in user whenever it finds more than four of them on the Desktop.  The MS fix is to have four or fewer such shortcuts, or disable Computer Maintenance - manually.  So users with more than four shortcuts to network resources would power up offsite, not using VPN, and the Schedule task would eventually fire off (1:00AM by default) and make their shortcuts disappear - pretty good trick.

Make it Stop
So with a manual fix in hand, I sought options to automate this for all machines.  Further searching turned up this blog post by Alex Verboon (@alexverboon) detailing where in Group Policy the needed changes could be made, but there was a problem.  From a 2008 R2 Domain controller, the setting/folder Alex described (Specifically "Scheduled Maintenance") was not present, another stumble.

I proceeded to fire up Group Policy Management on a Windows 7 SP1 system with RSAT installed hoping connecting from a client class system would reveal the Scheduled Maintenance settings.  I was rewarded with the "Scheduled Maintenance" folder and setting that Alex described.

A short bit of testing later and success.  No longer did my batch of test desktop shortcuts disappear with the PC disconnected from the network and manually invoking the "Scheduled" scheduled task.  A side effect of the Group Policy change is the "broken" shortcuts are reported in Action Center as "Broken Shortcuts".  If a user subsequently clicks on this in Action Center - poof - the shortcuts will be removed!  If you so wish, Action Center can be disabled via GPO also.

Stumble Upon Toolbar

Friday, September 23, 2011

CrashPlan IOS & Android Mobile App Now Available

My favorite online backup provider, Crashplan, has released IOS and Android apps allowing access to your backed up files! Via the app you can download and easily keep up-to-date any files in your backup to local storage on your mobile device. One great use of this is syncing your Music collection from your backup versus some other added service. Check them out in your device's respective market/store.

Stumble Upon Toolbar

Thursday, September 22, 2011

Working with Veeam SureBackup

I began using Veeam Backup and Replication several months ago in the vSphere environment that I manage.  I can honestly say since switching to Veeam, backups have no longer been a pain in my posterior.  With Veeam, my backups just work - quickly and with low impact to the virtual environment.  So now that I have these great backups taking place what is next?

One of the great things about Veeam Backup and Replication is that is comes with a suite of great, unique features - some of the big reasons it ended up being my "backup" solution.  The focus of this article is the SureBackup feature that in essence is a live verification of your backups.  I'm not talking some data hash comparison between the source and backup data.  I'm talking about live booted systems from the backup data including application checks - all automated.

In a nutshell, SureBackup fires up the backed up image(s) of the virtual guests inside of a network bubble(Virtual Lab) on a designated ESX(i) host and then runs some checks against those VM's to ensure they are running properly indicating a successful backup - then optionally tears the whole bubble down when done.  As part of the configuration, you  decide which VM's are brought to life inside the bubble and checked, typically the list is based on dependencies of the systems being verifed.  For example - to check almost any Windows based server that is part of a Windows domain, you will need an Active Directory / DNS server included.

So a bit more about this network bubble.  In order to run the backed up VM's and not need to re-address them on the network, worry about name collisions, etc., Veeam uses the concept of a Virtual Lab.  A Virtual Lab is really composed of small linux proxy / router appliance and some network settings that are automatically deployed to a specific ESX(i) host.  Behind the scenes isolated portgroup(s) are created on the ESX(i) host which are interconnected by the linux proxy appliance.  It does take a basic grasp of networking to understand how to answer the network configuration for the proxy appliance - possibly the biggest hurdle to creating a Virtual Lab and SureBackup jobs.  Check with your friendly network administrator if you have any questions - it really isn't so difficult but if not done properly will cause you and the network headaches!  Once setup, the proxy creates isolated networks that are identical to your production networks so the VM's being run there have no idea anything is different.  The proxy keeps their traffic off of your production network and can provide NAT'd access from the production network to the isolated systems for testing / restore activities - very cool!

What can you do with SureBackup?  I've already mentioned the primary purpose of SureBackup - to validate your backups by actually bringing them online and testing the OS and applications automatically on a schedule. There is another very handy way to leverage the SureBackup feature that is more so implied by the name of the Virtual Lab configuration.  An option in a SureBackup job is to leave things running and not tear the job down automatically.  And what this really allows for is potentially automated rebuilding of a copy of production systems in an isolated environment for testing of patches, configurations, upgrades, etc.  And given the NAT features of the proxy appliance, on a properly configured network end users can easily have access to these test systems.  I'm sure there are other creative uses - please mention them in your comments!

During my time thus far learning and working with SureBackup, I've gathered the following tips:

  1. Be realistic about the IO load you put on your backup target storage.  The SureBackup VM's are running off this storage so align your space and IO requirements with the underlying storage accordingly.  Small numbers of large SATA drives, or a highly de-duplicated disk system may not be well suited to acceptably performing SureBackup jobs.
  2. Be cognizant of the load the SureBackup VM's will put on your ESX(i) host.  If you create large SureBackup jobs with many VM's, they require RAM and CPU like any normal VM.  A handy feature is the ability to, per VM, have Veeam automatically scale back the RAM allocated by some percentage - use it.
  3. Set Timers accordingly.  SureBackup has timer settings used to determine if a job is taking too long or the OS / application is not responding(i.e. a backup did not work!).  Depending on your storage performance, etc these may need adjusting - the defaults should be fairly sufficient.  I found I needed to slightly increase the Application Initialization timeout for VM's with apps that really hit up the disk on start-up (SQL, Exchange).
  4. Access to the backup files.  As it stands today a SureBackup job ran against a backup done with Reverse Increment will lock the files and prevent a subsequent backup from running successfully while the SureBackup Job is active.  For long running lab / testing situations the current suggestion is to copy the backup to another Veeam server, import it, and run SureBackup there against the copy.  If you use Forward Incremental backups check the Veeam forums for any concerns.
  5. Don't adjust the proxy appliance or isolated networks for the Virtual Lab from the vSphere client.  Doing so will create a logical disconnect from what the Veeam Server expects and what it finds.  It will break the Virtual Lab configuration.
As I continue to explore and work with SureBackup I'm sure more ideas will surface on ways to further leverage this great technology.  Please post a comment with your ideas!

Stumble Upon Toolbar

Friday, August 5, 2011

Blah Blah Cloud

If you are in the position of: answering management's questions about the cloud, making the actual decisions about using the cloud, or really anything about the "blah blah cloud" - you owe yourself a listen to a Packet Pushers Show#55. Lots of great questions to ask your (potential) cloud provider and other points to ponder. While often spiking a high level on the technical / nerd meter, this show is a great listen for IT folk, purchasing, and maybe even legal types.

Stumble Upon Toolbar

Friday, July 29, 2011

VMware vSphere 5.0 vRAM License Change

Gabrie van Zanten (@gabvirtualworld) has posted on his blog what I'd expect should be a positively received rumor about VMware updating vRAM entitlements for vSphere 5.0.

I've been a fan of the vRAM based entitlement because it is tied to what is often the most key scaling factor in a virtual environment - RAM. Even though it did not impact my environment, I felt the initial entitlements were much to low for modern hardware, applications, and operating systems. Furthermore as VMware touted increased scalability in 5.0, the initial entitlement calculations indicated one would pay heavily from the pocketbook to take advantage of those improvements.

VMware was listening and has responded more quickly than could have been expected! Given the constant improvements in hardware and decreasing costs, I am a bit curious how much of a tail chasing situation this could end up being.

Go read Gabe's article for the details.

Stumble Upon Toolbar

Thursday, July 28, 2011

Excel 2003 Slow to Open Network Files After File Validation Update Installed

We've mostly completed the cut-over to Windows 7 and Office 2010 where I work. That said there are a few PC's with more 'complicated' configurations we have delayed updating that still run Windows XP and Office 2003. So accordingly we have maintained our WSUS environment to keep Office 2003 updates coming through.

In our last round of monthly PC patching, the update addressing KB2501584 was approved for PCs to install. Shortly afterwards reports started coming in from the few XP/Office 2003 computers in service, that Excel was hanging up when trying to open Excel shortcuts pointing out to files on the network - hmm. The long and short of the story is this MS article explaining how the Office File Validation function slows down opening network files in Excel 2003.

Two of the article's provided solutions I found to be "corporate unfriendly": either copy the file to the local disk first, or upgrade to Office 2007 or 2010. The third option was split into two paths: A) to use a MS Fix IT installer (normal users cannot do this), or B) Create a registry entry to disable OFV in Excel(again most users cannot / are not capable of doing this). Option 3 fix regardless of how it was done created a HKCU registry key - so now we have a per user, per computer situation to remedy - no "PC-wide" fix option.

Thankfully we already push out a few user scope registry entries via a custom Active Directory Group Policy, so we had a familiar delivery mechanism available to us. After some updating to that custom .ADM file and some testing we finally had a corporate quality fix for the issue.

Why didn't we just uninstall the OFV update? Removing security features isn't really a path we prefer to take. Yes - we have effectively disabled it for Excel, but the other Office application are still protected and didn't suffer the same problem that Excel did. One of the many usability versus security trade offs.

There are lots of great resources out there for making your own ADM files for use in Group Policies. If you work with AD / Group Policies it is a great skill to have in you toolbox.

Stumble Upon Toolbar