Jeff Hengesbach

Error 0x800F0A12 Installing SP1 on Windows 2008 R2 Veeam System

2012-01-27T16:40:00.001-05:00

I ran into a little issue trying to update my Windows 2008 R2 system that serves as my Veeam Backup server. It did not matter whether I tried the SP1 install from WSUS distribution or the full download from Microsoft, it always ended fairly quickly with the error 0x800F0A12.

A little searching turned up this article on the windows.microsoft.com website. Following the guidance in the article it became clear the issue was due to disabled auto-mounting of volumes. If your Veeam server directly access your SAN VMFS volumes, disabling auto-mounting of volumes is a critical step to keep Windows from squashing your VMFS volumes while still allowing Veeam to directly pull data from them. I performed the following steps as resolution:

Disable any Veeam jobs scheduled to start in the next 90-120 minutes.
Removed all iSCSI Targets and Favorites from the Microsoft iSCSI initiator. If you have a vendor specific tool be sure to check it as well (Equallogic HIT, etc).
Disabled network access between the Veeam server and the iSCSI san - not taking any risks here.
Followed the article's guidance and ran: mountvol /E to enable auto-mounting of volumes, then rebooted
Installed SP1, Reboot
Ran mountvol /N to disable auto-mounting of volumes, then rebooted
Re-established iSCSI network connectivity
Added back iSCSI Initiator configurations
Verified Veeam could access VMFS volumes (test job)
Re-Enabled all previously disabled Veeam jobs.

If you have any concern about using mountvol versus diskpart to disable / enable automount, they do the same thing. Check this thread for the details and critical registry key to validate the claim.

Fix Disappearing Windows 7 Desktop Shortcuts

2012-01-03T13:20:00.001-05:00

Case of the Houdini Shortcuts
Since moving to Windows 7, a few users in the office have reported many of their desktop shortcuts would periodically disappear. These reports would come in after users had been working offsite. The effected users often had more than 10 desktop shortcuts to files out on the network.

Mystery Revealed
Some searching around led me to two resources that answered what was happening and largely how to adjust it. MS KB978980 explained what was happening at a high level and how to prevent it manually. Basically the system scheduled task "Scheduled" will delete broken desktop shortcuts of the currently logged in user whenever it finds more than four of them on the Desktop. The MS fix is to have four or fewer such shortcuts, or disable Computer Maintenance - manually. So users with more than four shortcuts to network resources would power up offsite, not using VPN, and the Schedule task would eventually fire off (1:00AM by default) and make their shortcuts disappear - pretty good trick.

Make it Stop
So with a manual fix in hand, I sought options to automate this for all machines. Further searching turned up this blog post by Alex Verboon (@alexverboon) detailing where in Group Policy the needed changes could be made, but there was a problem. From a 2008 R2 Domain controller, the setting/folder Alex described (Specifically "Scheduled Maintenance") was not present, another stumble.

I proceeded to fire up Group Policy Management on a Windows 7 SP1 system with RSAT installed hoping connecting from a client class system would reveal the Scheduled Maintenance settings. I was rewarded with the "Scheduled Maintenance" folder and setting that Alex described.

Results

A short bit of testing later and success. No longer did my batch of test desktop shortcuts disappear with the PC disconnected from the network and manually invoking the "Scheduled" scheduled task. A side effect of the Group Policy change is the "broken" shortcuts are reported in Action Center as "Broken Shortcuts". If a user subsequently clicks on this in Action Center - poof - the shortcuts will be removed! If you so wish, Action Center can be disabled via GPO also.

CrashPlan IOS & Android Mobile App Now Available

2011-09-23T11:21:00.003-04:00

My favorite online backup provider, Crashplan, has released IOS and Android apps allowing access to your backed up files! Via the app you can download and easily keep up-to-date any files in your backup to local storage on your mobile device. One great use of this is syncing your Music collection from your backup versus some other added service. Check them out in your device's respective market/store.

Working with Veeam SureBackup

2011-09-22T15:21:00.000-04:00

I began using Veeam Backup and Replication several months ago in the vSphere environment that I manage. I can honestly say since switching to Veeam, backups have no longer been a pain in my posterior. With Veeam, my backups just work - quickly and with low impact to the virtual environment. So now that I have these great backups taking place what is next?

One of the great things about Veeam Backup and Replication is that is comes with a suite of great, unique features - some of the big reasons it ended up being my "backup" solution. The focus of this article is the SureBackup feature that in essence is a live verification of your backups. I'm not talking some data hash comparison between the source and backup data. I'm talking about live booted systems from the backup data including application checks - all automated.

In a nutshell, SureBackup fires up the backed up image(s) of the virtual guests inside of a network bubble(Virtual Lab) on a designated ESX(i) host and then runs some checks against those VM's to ensure they are running properly indicating a successful backup - then optionally tears the whole bubble down when done. As part of the configuration, you decide which VM's are brought to life inside the bubble and checked, typically the list is based on dependencies of the systems being verifed. For example - to check almost any Windows based server that is part of a Windows domain, you will need an Active Directory / DNS server included.

So a bit more about this network bubble. In order to run the backed up VM's and not need to re-address them on the network, worry about name collisions, etc., Veeam uses the concept of a Virtual Lab. A Virtual Lab is really composed of small linux proxy / router appliance and some network settings that are automatically deployed to a specific ESX(i) host. Behind the scenes isolated portgroup(s) are created on the ESX(i) host which are interconnected by the linux proxy appliance. It does take a basic grasp of networking to understand how to answer the network configuration for the proxy appliance - possibly the biggest hurdle to creating a Virtual Lab and SureBackup jobs. Check with your friendly network administrator if you have any questions - it really isn't so difficult but if not done properly will cause you and the network headaches! Once setup, the proxy creates isolated networks that are identical to your production networks so the VM's being run there have no idea anything is different. The proxy keeps their traffic off of your production network and can provide NAT'd access from the production network to the isolated systems for testing / restore activities - very cool!

What can you do with SureBackup? I've already mentioned the primary purpose of SureBackup - to validate your backups by actually bringing them online and testing the OS and applications automatically on a schedule. There is another very handy way to leverage the SureBackup feature that is more so implied by the name of the Virtual Lab configuration. An option in a SureBackup job is to leave things running and not tear the job down automatically. And what this really allows for is potentially automated rebuilding of a copy of production systems in an isolated environment for testing of patches, configurations, upgrades, etc. And given the NAT features of the proxy appliance, on a properly configured network end users can easily have access to these test systems. I'm sure there are other creative uses - please mention them in your comments!

During my time thus far learning and working with SureBackup, I've gathered the following tips:

Be realistic about the IO load you put on your backup target storage. The SureBackup VM's are running off this storage so align your space and IO requirements with the underlying storage accordingly. Small numbers of large SATA drives, or a highly de-duplicated disk system may not be well suited to acceptably performing SureBackup jobs.
Be cognizant of the load the SureBackup VM's will put on your ESX(i) host. If you create large SureBackup jobs with many VM's, they require RAM and CPU like any normal VM. A handy feature is the ability to, per VM, have Veeam automatically scale back the RAM allocated by some percentage - use it.
Set Timers accordingly. SureBackup has timer settings used to determine if a job is taking too long or the OS / application is not responding(i.e. a backup did not work!). Depending on your storage performance, etc these may need adjusting - the defaults should be fairly sufficient. I found I needed to slightly increase the Application Initialization timeout for VM's with apps that really hit up the disk on start-up (SQL, Exchange).
Access to the backup files. As it stands today a SureBackup job ran against a backup done with Reverse Increment will lock the files and prevent a subsequent backup from running successfully while the SureBackup Job is active. For long running lab / testing situations the current suggestion is to copy the backup to another Veeam server, import it, and run SureBackup there against the copy. If you use Forward Incremental backups check the Veeam forums for any concerns.
Don't adjust the proxy appliance or isolated networks for the Virtual Lab from the vSphere client. Doing so will create a logical disconnect from what the Veeam Server expects and what it finds. It will break the Virtual Lab configuration.

As I continue to explore and work with SureBackup I'm sure more ideas will surface on ways to further leverage this great technology. Please post a comment with your ideas!

Blah Blah Cloud

2011-08-05T15:59:00.004-04:00

If you are in the position of: answering management's questions about the cloud, making the actual decisions about using the cloud, or really anything about the "blah blah cloud" - you owe yourself a listen to a Packet Pushers Show#55. Lots of great questions to ask your (potential) cloud provider and other points to ponder. While often spiking a high level on the technical / nerd meter, this show is a great listen for IT folk, purchasing, and maybe even legal types.

VMware vSphere 5.0 vRAM License Change

2011-07-29T13:17:00.005-04:00

Gabrie van Zanten (@gabvirtualworld) has posted on his blog what I'd expect should be a positively received rumor about VMware updating vRAM entitlements for vSphere 5.0.

I've been a fan of the vRAM based entitlement because it is tied to what is often the most key scaling factor in a virtual environment - RAM. Even though it did not impact my environment, I felt the initial entitlements were much to low for modern hardware, applications, and operating systems. Furthermore as VMware touted increased scalability in 5.0, the initial entitlement calculations indicated one would pay heavily from the pocketbook to take advantage of those improvements.

VMware was listening and has responded more quickly than could have been expected! Given the constant improvements in hardware and decreasing costs, I am a bit curious how much of a tail chasing situation this could end up being.

Go read Gabe's article for the details.

Excel 2003 Slow to Open Network Files After File Validation Update Installed

2011-07-28T12:01:00.005-04:00

We've mostly completed the cut-over to Windows 7 and Office 2010 where I work. That said there are a few PC's with more 'complicated' configurations we have delayed updating that still run Windows XP and Office 2003. So accordingly we have maintained our WSUS environment to keep Office 2003 updates coming through.

In our last round of monthly PC patching, the update addressing KB2501584 was approved for PCs to install. Shortly afterwards reports started coming in from the few XP/Office 2003 computers in service, that Excel was hanging up when trying to open Excel shortcuts pointing out to files on the network - hmm. The long and short of the story is this MS article explaining how the Office File Validation function slows down opening network files in Excel 2003.

Two of the article's provided solutions I found to be "corporate unfriendly": either copy the file to the local disk first, or upgrade to Office 2007 or 2010. The third option was split into two paths: A) to use a MS Fix IT installer (normal users cannot do this), or B) Create a registry entry to disable OFV in Excel(again most users cannot / are not capable of doing this). Option 3 fix regardless of how it was done created a HKCU registry key - so now we have a per user, per computer situation to remedy - no "PC-wide" fix option.

Thankfully we already push out a few user scope registry entries via a custom Active Directory Group Policy, so we had a familiar delivery mechanism available to us. After some updating to that custom .ADM file and some testing we finally had a corporate quality fix for the issue.

Why didn't we just uninstall the OFV update? Removing security features isn't really a path we prefer to take. Yes - we have effectively disabled it for Excel, but the other Office application are still protected and didn't suffer the same problem that Excel did. One of the many usability versus security trade offs.

There are lots of great resources out there for making your own ADM files for use in Group Policies. If you work with AD / Group Policies it is a great skill to have in you toolbox.

VDI and the Microsoft VDA License

2011-07-19T14:19:00.004-04:00

If you are working on or looking into VDI, you should ensure an appropriate level of attention is given to the licensing side of whichever solution you'll use. Along with the myriad of technical details to keep in mind, the licensing side of a VDI project is just as critical.

I am by no means an expert in Microsoft Licensing and you should ALWAYS consult with your preferred vendor to arrive at a warm fuzzy feeling when it comes to license compliance (is there such a thing with regards to licensing?). If you haven't guessed already this article is going to be Microsoft center, albeit you can have linux and other OS based virtual desktops.

To start off, VDI is the acronym for Virtual Desktop Infrastructure. In short this is the running of (usually multiple) virtual desktop operating systems on a physical system and accessing them remotely. There are troves of articles out on the net about the merits, challenges, and use cases for VDI.

The initial thought for licensing these virtual instances of a desktop OS is often simply to go buy a license for each one and be done. That is not the case when it comes to Microsoft Windows. Microsoft realized there is value in running their software this way and subsequently created new licensing for such use cases. This is where VDA comes into play.

Two methods exists, that I'm aware of, for properly licensing virtual instances of Windows desktops: VDA and SA. SA or Software Assurance comes into play for corporate systems that have their Desktop OS SA maintained. These systems can be used to access VDI instances. The other is the VDA license. The VDA license is a bit more flexible (and costly) than SA because it can apply to a non Microsoft client device as well as non-corporate devices (contractor, employee owned PC,etc). You can read lots more about Microsoft and Virtual Desktops on Microsoft's Website.

The big question that I've always had until recently around the VDA license is: is it just an "Access Right" license or does it also include the virtual desktop OS? In the Microsoft world "Access" licenses, or CALs as they are often called, are very common - so quite naturally I wondered if the annual subscription based VDA license was only a "CAL" or more so. So after receiving a not so clear response from my Microsoft vendor, I consulted Google for other's experiences. I ran across this VMware Community thread some time ago and subscribed to it with hopes a more definitive update would be posted. Just today a post came through from MoffattThomas, quoting a Microsoft representative stating(summarized): the VDA includes the OS key for the virtual desktop. So there you go - Virtual OS and access rights!

VMware Hardware Assisted Virtualization with Windows Server 2003 x32

2011-06-29T15:38:00.005-04:00

VMware recently published KB 2001372 regarding the use of Hardware-assisted virtualization offered by Intel VT-x and AMD-V with Windows Server 2003 SP2 32 bit. To summarize the article:

Pre SP2 it was more efficient to use binary translation (software).
Starting with SP2, using Intel VT-x or AMD-V could provide significant performance improvements, depending on workload of course.
A VM with "Guest OS" set to "Microsoft Windows Server 2003, ...", will use binary translation by default(the Automatic selection).

I wanted to do a few rudimentary tests to see if it might be worth changing on my Windows Server 2003 SP2 VMs. These are by no stretch incredibly technical tests - my intentions were to see if changing to hardware assisted virtualization helped, and hopefully I might appreciate similar results in more typical workloads. I performed two tests: The first was VM boot-up to Login screen, the seconds was using the Memory Performance Tester by Joey Dieckhans that can be downloaded from Eric Sloof's website. I ran two separate Windows 2003 SP2 VM's through the tests.

Starting with boot up times - I found a consistent 3% to 4% improvement with hardware-assisted virtualization turned on (instruction set and MMU). Most probably don't cycle their Windows servers through enough reboots to find value in this alone, but longer term, it will be interesting to record how normal workloads might benefit.

With the Memory Performance Tester tool there was near zero change in the "MB per Second" figure in either configuration. I also monitored the CPU load in the VM during the memory test and found the same - no difference.

I look forward to trying this setting in a few pilot VMs and comparing their performance to past trends.

Veeam Backup and Replication Prep Checklist

2011-05-29T09:59:00.004-04:00

I recently implemented Veeam Backup and Replication in my employers vSphere environment. Previous to using Veeam, 'traditional' in-guest agent backups had been used. While agent / OS based backups aren't necessarily bad, they are not the most effective way to get things done in a virtual environment. To get the most benefit from a virtualized environment and realize some other truly beneficial 'side effects' a hypervisor level backup should be used.

The downsides to agent / OS based backup methods in a virtual environment are: they consume CPU cycles on each VM and consequently take away from the host on the whole, they are slow because they have to check every file, they can also cause access times to be reset on every file which may not be ideal, and in event of a full restore, there is likely some requirement to get a base OS up and running (time consuming). The list goes on but in summary guest OS agent based backups are not time effective or host resource efficient.

Now when working with backups at the hypervisor level, there are some prerequisites to have in place for backups to work properly. One of the more important (painful to correct) items is ensuring any data that will get backup up is on hypervisor backed storage. By this I mean the backup tool, because it operates at the hypervisor level of the stack, can not see storage that a guest may have direct access to via physical RDM, or in Guest iSCSI mapped luns. Resolving this issue (if possible) can be a good source of downtime for those who don't have sVMotion enabled licenses (like myself). There have been a handful of articles that show little to no performance trade-offs between vmdk backed storage and RDM or in guest mapped luns.

So during my Veeam Backup and Replication implementation I created this Guest VM checklist the ensure things when a bit more smoothly the first time through.

Ensure CBT is set to false on all guest vDisks - Veeam will fix automatically
Ensure Data to be backed up is on vmdk or virtual RDM backed storage
Ensure vDisks are not set as persistent - snapshots are key to the backup process
Put Page / Swap file on seperate vdisk and Exclude the vdisk from backup - optional but will save on backup / replicating of pagefiles

This isn't meant to be a advertisement for Veeam but in short it was the selected solution for the following reasons:

Veeam is the leader at innovating backup and disaster recovery features in the virtualization space. It is a product built from the ground up for virtualized environments and the feature additions over each major release are significant.
vPower NFS and Instant Recovery offer incredible value - you must try these if you have not.
Ability to truly validate backups, by starting up the VM's from the backups in an isolated setting to ensure application functionality.
"Lab" functionality that is great for testing patches, upgrades, etc
For you Hyper-V folks, support is coming in Veeam v6!
There is substantially more value in a Veeam solution than just 'backup'

Extend Windows Server 2003 C Drive

2010-11-15T09:13:00.004-05:00

Sooner or later the chances of encountering a Windows based system that is exhausting its free space on C: are pretty good. After years of Windows patching, application updates, etc, even a system that started with a healthy amount of free space on C: can find itself running low. If you happen to be in a virtualized environment you have some good, semi non-disruptive options to resolve this issue.

In the "old days" when the C drive was carved out of physical disk, the only option was to backup the system, repartition, reformat, reinstall, and restore the system(maybe there are/were some advanced partitioning managers that could do this much more simply?). A very lengthy, disruptive and risky operation. With the advent of Windows 7 and Server 2008 R2 (maybe Vista/Server 2008?) Microsoft built in the very handy feature of extending the C: drive live while the system is running - very cool indeed! As long as you have some available space to add, it is a zero downtime operation on those operating systems.

Back to Windows 2000 and 2003. I recently had some Server 2003 virtual machines that have been in service long enough to start running low on C drive space. All the (conservative) cleanup I could muster wasn't going recover enough space for the long haul. If you turn to Google you'll find several ways to accomplish extending the C drive - here I offer what I though was one of the most straight forward and conservative(safe) methods in a VMware environment.

Preparation Steps

Get a good backup of the system to be modified (you already regularly do this though - right?)
Have a "Helper" Windows 2003 or newer virtual machine you can power on / off without disrupting user services. This system will need access to the vmdk to be extended.
Make sure you have free disk space on the vmfs/nas volume to extend your C drive.
Downtime required - although minimal it must be planned accordingly

HowTo

Backup, Backup, Backup the original system
Shutdown the original System
Edit the Virtual Machine's properties and adjust the disk size to the total desired capacity
In the Helper Virtual Machine
Add the existing, newly extended drive as a new hard drive device
Boot up the Helper Virtual Machine
Fire up diskpart, execute list volume
Execute select volume # where # is the volume number to be extended
Execute extend, then exit
Shutdown the Helper Virtual Machine and remove the disk device configuration from it (but don't delete the VMDK!)
Boot up the original system
You may see a checkdisk run - this is OK
You may get the "New hardware install please reboot" prompt, go ahead and reboot
The system will now have a larger C drive!

This method appealed to me for following key reasons:

Uses Windows native tools to modify Windows partitions.
No copying of data from one disk to another.
It's relatively quick. This can manually be done in around 15 minutes(excluding the backup time).

One potential downside stems from a VMware technical matter. For those who might think: "I'll take a snapshot of my virtual machine before messing with its disk size", you'll quickly notice the VMDK size can not be altered when a snapshot is present. If you are looking to the concept of "Snapshot" as a quick fall-back option, you'll have to get that from your storage or backup solution.

Windows Server 2008 SP2 Install Errors: 0x800F0A0A and 0x800F0826

2010-09-27T13:11:00.005-04:00

I was about my merry way of building a blank slate Windows 2008 x64 server when I ran into issues getting SP2 installed. I had pointed this new server to the internal WSUS where I had previously released a very specific list of updates (to identically match a production system) and all started well with a few quick sets of patch downloads, install, reboot, rinse and repeat. That is until it picked up SP2. Many frustrating attempts followed each resulting in errors 0x800F0A0A and 0x800F0826.

So I dug into the C:\Windows\Software Distribution folder and manually kicked off the 500MB+ beast. It very quickly came up a message indicating why the Windows Update attempts failed. Since this server was going to be accessing some iSCSI LUNs on a Dell Equallogic array, I had early on installed the Equallogic Host Integration Tool Kit (aka HIT Kit) to get the benefits of quick configuration of the initiator and the MultiPath IO plugin(DSM). The SP2 interactive installer clearly pointed out there might be a compatibility issue with the DSM driver along with an option to "Ignore and Continue" - which I selected. The SP2 installer went about its merry way, installed without issue and all SAN luns were fine afterwards. A plus in favor of Windows Update is that it plays its cards conservatively (in this case) - a minus was the hexadecimal cryptic error message it provided back.

Lessons learned - Do all your OS related patching before adding anything else to your system and run installers interactively when issues crop up. Share knowledge with others so that Google may more quickly end their similar tribulations ;) .

Identifying Active Directory Primary Group Membership

2010-06-08T11:38:00.006-04:00

I ran across what I thought was an interesting question on serverfault.com: Delete a windows group in Active Directory. No one had posted any answer / comment to it and since I was curious what the answer was, I put one together.

As a side, serverfault.com (superuser.com, and stackoverflow.com too) is not just a great place to ask relevant questions looking for help, but also to read questions that have been answered in order to advance one's knowledge. A good question there is just as valuable as a good answer - so don't be afraid to ask either!

Active Directory allows for a User's primary group to be changed from "Domain Users". This is great if you have applications that need / benefit from it, but how can you get a list of users that have their Primary Group membership set to a particular group? The built-in AD searches do not provide a "Primary Group = xyz" drop down.

You have to be willing to get your hands just a little bit dirty to accomplish a list of users that have their primary group specifically set. Active Directory is of course based on LDAP, as such AD users are objects and objects have attributes and attributes have values. The key attributes and associated values that need to be examined are 1) primaryGroupToken in the Group object, and 2) primaryGroupID in the User object.

Using adsiedit, find the Group in question and examine its properties to find the primaryGroupToken and make note of its numeric value. This is key for later when we search for user objects that have this value for their primaryGroupID attribute's value.

In Active Directory Users and Computers, right-click Saved Queries, and select New, Query. Give your query a Name: "Users-Primary-Group-Is-XYZ", and a Description. Set the Query root to a reasonable OU in your directory structure and click Define Query.

In order to enter a custom ldap query, change the Find drop down to Custom Search, and click on the Advanced tab. The fairly simple query I created is as follows where XXX=the value of primaryGroupToken found earlier:

(&(objectCategory=person)(objectClass=user)(primaryGroupID=XXX))

Click OK, highlite your query in the Saved Queries Navigation folder and click the Refresh icon in the toolbar to get your results!

Virtually Compensating?

2010-04-01T20:19:00.005-04:00

I picked up on a tweet from @jordansissel that on one hand was an "Well of Course" statement but was at the same time something that seems to have been lost in the virtualization buzz. To paraphrase Jordan's tweet: Virtualization shouldn't be taking off because poorly written software can't coexist on the same host.

So what are some of the top key benefits virtualization brings today that have made it the buzz of the industry for the past few years now? 1) Isolation of the virtual system from each other. 2)Resource control of multiple operating systems on the same physical piece of hardware. 3) Improved portability and availability of the virtualized software environments.

To counter these points, go back to Jordan's tweet: Why isn't software written better? Why doesn't my operating system handle things like isolating applications from impacting each other negatively or using too much RAM, CPU or disk I/O? Why aren't my applications more 'self contained' and easily movable to a different host without having to worry about hardware drivers, hidden settings, etc. If I could count on my operating system to handle fairly running multiple Tier 1 applications and those applications to be easily moved between hosts I could effectively utilize modern hardware without adding the hypervisor layer into that stack. Oh and my backup /recovery solution should be able to backup and restore a single application and its data on my host without effecting the other applications or the host. As things set today in order to define an "application" I need a hypervisor (environment), an OS license, application license and associated per user/connection licenses.

I'm not against virtualization, in fact I'm a huge proponent and user of it, but I'm also a fan of effective and simple. What do you think?

My Home Network

2010-02-09T14:31:00.003-05:00

I often get asked by other people: "How many computers do you have at home?". Because I'm an IT professional, the question is sometimes asked along with a rhetorical answer of: "Probably 10", and other times it is asked with genuine interest. I typically begin by laying out the purposes of my technology use at home to help people understand my setup.

I've maintained a home network since my early days of college. 386 and 486 linux based computers running file shares for roommates as well as dial on demand modem and subsequent dsl Internet connections. There has always been a dual purpose of legitimate utility as well as the learning / hobby aspect. This really hasn't changed much today - just the requirements.

Technology changes, life changes, requirements change. Today I have a family, limited spare time, a home, and demanding professional career. And so my requirements have naturally evolved. Low maintenance is a must due to time constraints. Space needs are high due to volumes of pictures and video of family, images of virtual machines for testing Mobility and connectivity around the home for the work and recreational needs of my family. Backup of all those precious digital memories and information. Of course efficient use of electricity and space.

Low maintenance to me at home comes from a few things. Largely this involves controlling the sprawl devices that I need to do upkeep on - consolidation. At one point I had several older computers doing various tasks, each consuming power and requiring periodic attention - those are gone, replaced by one central modern workstation. I've also recently replaced a "sensitive" MythTV based computer with a DVR from the cable company - a huge win on all fronts. Sensibly configured DHCP, DNS and file shares make connecting and sharing easy and reliable.

So what makes up my home network? At the computing center is a home built workstation. An overclocked Intel Quad core based system currently running 4GB of RAM, Windows 7, VMware Workstation with two 1TB Seagate 7200 RPM Sata drives in a RAID 1 configuration. The ASUS motherboard was not well supported by a Linux distribution that would reliably run VMWare Server 2.0, so the Windows 7 x64 route was taken. The Windows 7 x64 install is minimal and is really only there to run VMware Workstation. Inside VMWare Workstation I run at times a multitude of virtual machines. One always on instance is a CentOS guest that is my file server. For "portability" sake this is great as I can move it across hardware without issue and my file shares and Amazon Jungle Disk backup setup never changes. I maintain a Microsoft Technet subscription so I often boot up several other instances for testing various configurations. I also build up test VMware vSphere environments. In the near future I'll be adding 8GB of RAM to this system since 4GB really limits my testing capabilities. A few years ago, I 'finished' the basement of my home. During the project I installed several network drops that run back to a 10/100 switch that lives in an old rack in my storage room. Also in that rack is my cable modem and Linksys wireless G router. There are two systems in retirement there as well - a SGI Indigo2 (R6000) and a SGI Indy system. I spent a good number of years in an SGI equipment based engineering department and love the hardware. At the time it was light years ahead of anything available. I have a very inexpensive Samsung color laser printer attached to a wireless print server for printing from anywhere in the house. My wife and I have two fairly old Dell laptops that are almost nothing more than thin clients and serve their purposes well since all heaving lifting is done on the workstation via RDP, ssh, etc.

So it comes down to: Only 1 primary system(on a UPS) that is always on to do the heavy lifting and run virtualization to contain / isolate the "complicated" parts, along with a few peripheral devices for mobile access and output. This configuration suits my family's and my needs both for personal and work use - how about yours?

Disk Space Cleanup in Windows Server 2008

2010-02-02T10:56:00.007-05:00

I've recently began setting up and working with Windows Server 2008 based (not R2) systems. After the initial install and full round of patches I was a bit surprised at the amount of consumed disk space. It turns out the c:\Windows\winsxs folder was over 10GB in (apparent) size! Gone are the days of being able to go into c:\Windows folder and deleting the $Uninstall folders to get that space back - assuming you didn't want to uninstall any of those patches. I'd like to point out that this folder should not be tinkered with haphazardly. Many of the items in it are hard links out to many critical OS files and rampantly deleting them will make you sad.

I ran across this article explaining the winsxs folder's purpose and subequently its size. It boils down to all the different installed / installable components(aka the good old i386 folder) of the OS are kept there along with patch backups. The arguments are for reliability and availability for things like removing patches, or adding features and roles without need the OS media.

A bit of searching turned up the built in Windows Component Cleanup (compcln.exe) tool which will do some patch cleanup for you. I got back around 1.7GB which was nice but still not enough for my liking. The next bit I turned up was this article by John Daskalakis explaining howto manually delete some of the larger sub-folders. Following that article I was able to reclaim another ~800MB.

Another folder that deserves attention, and is consistent with Window XP and Server 2003, is the C:\Windows\SoftwareDistribution\Download folder. If you have applied all available updates, this can be safely deleted by:

1) Stopping the Windows Update Service

2) Deleting the folder

3) Starting the Windows Update Service.

I reclaimed about 1.2GB from this.

All said and done almost 4GB of space reclaimed and no issues with the system. I still wasn't very comfortable with the overall amount of free space on the system drive so I had so I added 10GB and extended the volume to a total of 40GB (gotta love virtualization).

How I Became a Sysadmin

2009-12-22T08:21:00.005-05:00

An interesting article was posted on the SysAdvent calendar titled "Becoming A Sysadmin". After reading it, I thought why not put up my story. As the article noted the path to becoming a Sysadmin is not a 1-2-3 step process, it is most often an evolution of roles with a pinch of circumstance. That evolution of roles part lends itself to the vague definition of what a Sysadmin's responsibilities are and as the article mentions putting "Sysadmin" in a job description's responsibility list is no indicator of actual responsibilities. Another part of the article raises the point about what characteristics in a person makes them want to be or enjoy being a Sysadmin. I'm hoping this doesn't read too much like a resume.

I'll start at the foundation question - what characteristics drive a person to enjoy or want to be a Sysadmin? I think creativity is one trait of SysAdmins. Creativity plays a part in may aspects of the job: writing scripts / code, problem solving, designing solutions for networks, storage, etc. I've always enjoyed building things - from Legos when younger, to metal fabrication, and even homes. Of course some desire for control needs to exist, but I think the higher purpose analogy is more like a coach or co-captain guiding a team towards a common goal. Sometimes you get to pick (aka design) your team, and sometimes you have to work with what you are given. This may just be my experiences, but a good many of the 'tech' people I know have a strong liking of music - in fact many have even played or play instruments. I stopped playing saxophone going into college because it was a major time involvement - I do miss playing though.

I've heard IT folk tend to stay in a job or position for maybe 5 years or so. Why this is tends to be answered by the IT person's need for new challenges. It is after all one of the draws of technology - ever changing; sometimes one of its drawbacks too. My 12 year history in IT work, the only professional arena I've worked in, follows the pattern fairly well - but mostly due to life circumstances and not my sole decision to change things up.

Onward with my path. I went through college on the Computer Science / Mathematics major/minor track. I started my first 'career' oriented job going into my 3rd year of college. This position was as a programmer in an engineering department using SGI Irix workstations. It was an incredibly exciting role at the time and provided me the opportunity to learn many technologies, as well as my likes and dislikes regarding IT roles. Besides the skills I took away from the job, I found that for me while creating code was OK, I truly enjoyed networking and systems management much more. I was also during this time that I developed a strong skill set with Linux.

I moved onto my next position due to my wife's pursuit of her masters degree at a university in another city. My *Nix background landed my next position managing a HP-UX based environment running SAP on Oracle using some nice EMC storage. I worked with some great people in this position, but it was definitely a very specific set of tasks and the challenge quickly faded. I learned a lot about SANs, a bit about SAP and Oracle, expanded my network skills, and benefited personally by observing great management. I was made redundant in a post acquisition event and subsequently laid off.

I spent the following few months getting my resume out, networking and mowing lawns - which I actually enjoyed for the time I did it.

And that gets to my current and longest held position. I became the "Systems Administrator" mid 2004 at a smallish company (150 people). It is quite the opposite in the realm of responsibilities - my saying is now: "If it plugs into the wall, rings, buzzes, beeps, or takes batteries I'm responsible for it.". It was a very simple environment that provided me lots of opportunity to advance my Windows based management skills while significantly improving operations efficiency, security, and reliability. The business has been in a constant state of positive change and the challenges have been great. Today I work along side a (SAP) Business Analyst, and another person who specializes in Application support and training. I frequently interface with the parent company IT group in Japan as well as a few other US sister companies. I travel very lightly which allows me lots of time with my young children at home and just enough away ;)

That's my story and I'm sticking to it - evolution of a Sysadmin.

New Location ToDo List

2009-10-30T09:43:00.002-04:00

The 'new' site I've been working with over the past several weeks has several traits I would say are pretty typical. A small, very successful business is acquired / merged with another that is owned by a fairly large parent that also happens to own my place of $WORK. I've had the pleasure of being involved to help get their infrastructure, projects, strategy, and support into a more organized, best practice, and business aligned status.

There has been a lot of learning from all sides up to this point. As I had mentioned most of the items needed attention are not surprising knowing the company was a grassroots effort that has grown incrementally over the past 10 years.

VOIP and computer data are traversing the same network
Some of the switching equipment is non-managed grade stuff
Switches are piggy-backed(Did I mention they do CAD work)
The primary IT person wears many hats and is severely overloaded
A new business software is mid implementation
The primary IT person is going on maternity leave within the next month
There is one server doing everything
Telco and Internet services are not cost effective / lined up with the business needs
No automation - Updates, software installs, PC imaging, etc
Remote access needs some help
Lots of processes need documentation / standardization
No tracking of support issues / history

The challenge for me has been uncovering all the details of the organization and infrastructure to ensure the decisions to right things going forward are sensible. Thankfully I have a good team at $WORK, supportive management, and plenty of great people at both locations. This is not an open call for vendors. Solutions to most of the above items have already been devised and some implemented - the timetable has been the biggest challenge to meet.

Latest Happenings

2009-10-12T21:21:00.002-04:00

The last several weeks have been an extremely busy time. I've taken on the responsibility for a recently acquired sister company's technology and people. The site has a lot going on to say the least. Maybe a "Perfect Storm" of needs, changes, growth, and projects would be a well suited analogy. It's a very exciting opportunity for everyone involved that is going to take a lot of hard work - wish us luck!

Trial Run: Aprigo Ninja

2009-09-15T08:06:00.004-04:00

Fellow blogger Matt Simmons had previously blogged about "Ninja" by Aprigo. After bringing it up in passing again I thought it might warrant a try. Shortly after registering for the private beta, I was approved and provided access to the less than 5MB download.

I am not affiliated with Aprigo in any way - not a current customer, relative, friend to employee(s) there. My motivation for trying the product was its visual representation and analysis of data. I was curios how the volumes of file server data and underlying equipment that I manage was being utilized - beyond simple user quota figures.

Install was a breeze, the typical click, next, next, finish. At first run you have to setup a scan by pointing the application to a folder,share, or UNC path and providing a user friendly name. The trial is limited to 500GB per scan and seems to only be able to run one scan at a time - interactively.

Lots of interesting results are available after a scan is completed - most of them with 'drill down' capabilities to get the finer grained details. If a cost is entered, per GB/year, a dollar figure is displayed next to each result. This is great for seeing how much it costs to keep the different categories of data laying around. The non-trial/beta version appears to offer trending reports, cloud storage analysis and access management(very interesting!) as well. Aprigo Ninja is definitely a handy tool if you're getting going on trying to analyze your storage use or Windows File Server Resource Manager isn't cutting it (or you want to analyze data on samba shares). Give it a try, it's worth the small bit of time and effort.

Free vSphere Videos / Demos

2009-08-19T10:59:00.003-04:00

I picked this up on twitter via @DellServerGeek: Mike Laverick has generously posted many vSphere how-to demos on his website. This is a significant body of work to make freely available. Mike goes through the entire process from "soup to nuts". For anyone curious about setting up a vSphere environment these are a must watch! If you like the series - consider procuring his accompanying book. Thanks Mike!

Free Storage for Your vSphere Lab!

2009-08-18T14:09:00.003-04:00

VM Super Genius has a post with a link to free storage solutions that can facilitate trying out some of the more advanced features in VMware ESX / vSphere. As noted many of those features require shared storage (iSCSI, NFS, Fiber SAN) which can be prohibitively expensive for many. The list offers some zero cost appliances / applications that can provide iSCSI and/or NFS storage to virtual hosts. An excellent blog to follow if you are into virtualization.

Playing with DRBD (Replication)

2009-07-21T08:53:00.005-04:00

Over the past 4+ years I've worked towards moving the infrastructure at work from isolated physical systems to a centralized storage(SAN) and virtualized paradigm. I'm happy to say we are definitely a "Virtualize First" and SAN shop now. A conservative comparison of separate physical systems with local storage to the current SAN/virtualized environment shows a nearly 50% cost savings on equipment alone. The benefits of extra rack space, lower power and cooling costs, I/O and compute capacity, etc are nice as well.

This evolution has a next logical progression. Now operating in a very centralized fashion, I can (more easily) begin examining solutions for replication. The base goal was to replicate the SAN to mitigate a 100% loss of the primary server room and not require an enormous amount of time restoring from tape. Tapes are great snapshot points in time like photographs, but like photographs, trying to rebuild your entire set of memories by looking through photographs would be very time consuming (and photographs degrade). Replication is not a replacement for good backups.

From a previous post my SAN is linux based, much akin to an Openfiler solution. It provdes NFS storage to ESXi servers for VM images and iSCSI storage to the VM images that need data volumes. All physical drives (as present by the RAID controllers) are sliced up using LVM. It has been humming along without issue since installation in February 2009.

I have known of the existence of DRBD for several years but not been in a position to utilize it. In short DRBD is a block device you layer into your device chain, just like LVM. It's specialty is taking all the original block level changes, keep track of them, and send them over to another system where they can be duplicated. The DRBD website is excellent, I highly suggest spending a few minutes there. DRBD has a few very nice traits that I'd like to highlight. First off it is smart (and dumb?). It works at the block level and knows which bits may be out of sync and will only send those bits across the wire - it knows nothing of filesystems, files, etc. Secondly it can be non-destructively added to existing data volumes. There's no need to backup/install/restore. DRBD is opensource and freely available - but its creators and primary maintainers, Linbit, offer commercial support and have been around for awhile. Linbit also offers a closed source product, DRBD Proxy, that is designed for long haul, high latency(200ms) connections or greater than 2 node replication situations. If you want to replicate outside of a LAN using DRBD you'll need it. DRBD is also 'good friends' with Hearbeat for high availability / failover situations.

I setup a couple CentOS x64 based VM's for testing. DRBD is available via the standard CentOS repositories but it is naturally a bit behind the current version available directly from the DRBD website. The download is small and if you have a basic compiler toolchain, and kernel-devel package the build / install is quick and painless(make rpm). Did I already mention the DRBD website documentation is fantastic - really go read it. The required configuration to have DRBD work is quite minimum although there are lots of options to fine tune its operation. If your data's rate of change is very high, you will really want to have Gigabit connectivity between your nodes. What you'll find is your DRBD devices will only write about as fast the data can get across the wire (assuming your drives can outrun wirespeed). If you need more than wire speed and your drives are fast, take a look into the DRBD Proxy product. I spent a fair amount of time in different scenarios to see how DRBD would act and what to do as an admin in those situations. Like many things, with a little bit of time and reading, DRBD was easy to work with.

So what was I doing with all this again? The base goal was to replicate the SAN to mitigate a 100% loss of the server room. Since the SAN literally contains everything (VM's, SQL, Exchange databases, file shares) this was a fairly simple move that captures the entire datacenter to another system. To backpedal a bit, my environment is modest in size by any modern measure, but still just as important. That 'size', centralized storage, and a geographically large site made the option of placing the replica system in a local but 'distant' (fiber connected) building a perfect option. The replica runs ESXi with a CentOS VM running DRBD to replicate the data. Why ESXi on the host? What this more or less creates is my datacenter-in-a-box, transportable if needed. The CentOS VM will provide NFS access back to the host ESXi for access to all the server VM's which in turn will use iSCSI access their data. ESXi virtual switches let me create matching, non routed networks local to the replica host for the NFS and iSCSI traffic, meaning zero reconfiguration of the production server VM's. This isn't meant to be a powerhouse / failover solution. What it is, is a very cost effective solution to a worst case situation that hopefully doesn't occur. If the worst was to occur, some scripting magic transforms the replica to production status. When a new production environment is established, DRBD can be used to mirror the data back to it enabling a transition with very little downtime.

Tired of reading yet?

Shared Hosting with Exchange 2003

2009-05-05T12:56:00.003-04:00

This post is as much a placeholder for me of some information I found some time ago as well as something that may be useful to others. In late 2007 I was tasked with providing "hosted" email services for a sister company out of my employer's Exchange 2003 environment. OK I thought, this should not be a big deal. The system has plenty of capacity, bandwidth, and Exchange 2003 has to have some mechanism to pull it off.

I started off by setting up a small environment in VMware: Windows Domain, Exchange server, "Internal" client, "Internet" firewall, and external client. All of these of course sat on private networks internal to the host server. I installed all the same applications to best mimic the real world - Antivirus, Anti Spam, Archiving, etc.

So I have this environment all prepared with a few users at the "primary" email domain, now what? I turned to one of my favorite exchange online resources: www.msexchange.org and without too much searching found Part 1 and Part 2 of Shared Hosting with Exchange 2003. These are both well written articles that made the project pull off without a hitch.

I know Exchange 2003 is on its way out(long gone?) for many people, but at the same time I'm sure there are just as many Exchange 2003 environments out there that will be around for a handful more years.

One key note to keep in mind. During planning to add valid 'foreign' user accounts to your active directory, consider ways to prevent those accounts from accessing other resources tied into the active directory domain!

Vmware New Release Day

2009-04-21T08:33:00.003-04:00

It's a big day in the virtualization world. VMware is publicly releasing vSphere 4 today. Jason Boche has posted some initial information this morning on his site and VMware's site is of course buzzing with all sorts of new information. VMware will be putting on a simulcast at 9:00AM PDT/Noon Eastern - registration required.

Some of the highlight are in the pricing document (thanks to Jason for the link) which is a good read. Of interest: Pricing is based on per processing socket (with a defined limit of cores/socket), vCenter Server (Centralized management) is still an add-on purchase(in most cases), the SMB Essentials (& Plus) offerings appear to be some nice packages for the small business looking to consolidate 5+ servers with centralized management and basic high availability.

I'll be curious to see more technical information as it is released - more hardware/feature support in ESXi4, etc. Initially I don't see where Xenapp Server being freely available has made much impact, but the day is young.

Exchange 2010 Beta Available

2009-04-15T11:38:00.002-04:00

Michael over at sysadmin-network.com pointed out that Exchange 2010 (Exchange 14) Beta is now available. I haven’t followed Exchange 14 development much beyond thinking it was going to be released around Q2 of 2010. The website lists a generaly availability for Exchange 2010 being the second half of 2009. Being an Exchange shop, the guy who gets to plan and do the upgrades, I decided to do some reading. You can find the Exchange 2010 Overview here but the points that really stuck out to me, in my small exchange environment, follow.

Up to 50% IOPs reduction in exchange databases from Exchange 2007. If this is accurate and realistically attainable it alone could justify an upgrade. Two significant things come from this:

1) Support more users per IOPs of storage (Storage $$/More Users = good)

2) Use less expensive storage while maintaining service levels (Lower Storage $$ = good)

Up to 16 replicated copies of each mailbox database. If I’ve read this right, Exchange maintains multiple copies of each database allowing for quick failover. So throw in a couple of JBOD underlying storage arrays instead of just one Raid 10/6/5 and you could have improved storage reliability for less cost if done properly.

Outlook Web Access improvements: Cross browser consistent experience (IE7+, Firefox 3+, Safari 3+), improved search (THANK YOU), and more.

Archiving and Retention add two features as well that previously were significant pains and/or required 3^rd party solutions to accomplish well. First off, the “Personal Archive”. I think of this in terms of a server side PST (I’m not yet certain on how it is pulled off). This appears to be a long overdue, integrated solution to user PSTs floating around on local / network drives. Searches can be done simultaneously against both Personal Archives and the “Inbox”. So an organization can now potentially keep all email inside its Exchange environment and not need a 3^rd party application to effectively archive / vault / retain messages. Can Personal Archives be set as “check in but never check out”? The second item which is a must if Personal Archives are to be a true archiving solution is Multi-Mailbox Search. Authorized persons may perform searches across multiple mailboxes and personal archives. Note that “Authorized Persons” can be people outside of Exchange administrators!

Needless to say I’m quite interested to find time to kick the tires on Exchange 2010.

In addition to this being news specifically about Exchange Server, a few conjectures can be made if things happen as they have in the past. First Windows Server 7/2010 will be released near or at the same time (Hold off on those Server 2008 deployments!). And secondly, Office 2010 will show up near or at the same time.

OpenFire - GPL, Private 'Chat' server

2009-04-07T14:06:00.005-04:00

There are lots of chat services out there so when thinking about utilizing chat one has to start by asking, why want or need to have one's own? Chat has been around for a long time, and along with it are the eternal questions of privacy, security, etc. Those questions are very motivating reasons to operate a chat / collaboration system internally. Large and or geographically spread out organization can realize significant benefits by having chat / presence features available to users, while maintaining a secure (and compliant) environment.

While looking around I ran across OpenFire at igniterealtime.org. Openfire is a XMPP(jabber) based, cross platform solution. A few of the notable highlights: Simple administration, support for secure communications, logging, cluster options, multiple authentication options: internal, ldap, AD, and connectors to external chat systems.

Two of my favorite plugins are Fastpath and Fastpath web. Fastpath allows for setting up chat queues - think of it like call centers for chat. A "Workgroup" and list of user(s) is setup that service incoming chats in a round robin fashion. Fastpath Web extends this outside of the chat client application and provides the mechanism where placing 2 lines of html code on a website allows for clientless chat - ie "Click Here for Live Support". Fastpat Web is designed to be distributed - it can be installed across multiple (java)application servers to ensure redundancy and a balanced load if desired.

My hats off to igniterealtime/jive software. Installation was simple(rpm, deb, msi, tgz,dmg) and configuration is handled through a well designed web tool. We'll be investigating exactly how Openfire may be used at my employer. Any other Openfire users out there with stories to share?

Cobol Call for Help

2009-03-27T11:02:00.004-04:00

Calling out to sys admins for some COBOL / NCR Unix help. I inherited an older NCR WorldMark 4300 box at my current employer running a customzed COBOL application that was replaced by SAP before I began there. The issue comes from the old regime that setup, programmed and maintained the system up and left / retired right after SAP and before my time. Little / none of the legacy data was transfered into SAP (what a great idea!). So I'm left with a technology and hardware dinosaur that houses some valuable historic information (both from an intellecutual and auditory standpoint).

I'm no COBOL guy. There I said it. The system is documented well enought that I have a copy of the datafiles, sources, and file descriptors, but in the event the old hardware lets the smoke out, I'm still left in a very uncomfortable situation. I've been searching for COBOL database file convertors with some limited success and wanted to ping anyone out there for addition knowledge on the old environment / conversion tools (-> SQL Server) / migration paths.

Online Backups

2009-03-19T14:55:00.004-04:00

They've been around for several years and are one more means to and end in a sea of options - online backups. The term Backup has many traits that must be defined by business in order to architetch and execute a 'Backup' successfully (RTO, RTP, Copy, Archive, CDP, Onsite / offsite, etc...). I'm not looking to dig into those any more than necessary, but more so the concept of online backups and why or why not anyone out there has decided to use one.

First an assumption - few if any businesses have less than 50GB of data that needs be backed up over a given week. One small file server with its operating system and data can quickly tally up 50GB of space. Space is cheap right - let's use it! This data volume figure is important because it is going to determine how much an online backup service will cost. Bigger data = bigger cost (and possibly the need for additional Internet bandwidth; even more cost).

My current employer used an online service when I began there. Over a short period of time it became quite clear data was growing and wasn't going to stop to the point were it was very unecconomical to continue with the service. Besides costs, all those terms that a business defines backups as were used to architect a new solution. I'm happy to say the current backup solution meets the business' "backup" needs and also had a very short ROI.

I was recently cold called by an online backup provider and entertained the ensuing web presentation, and it perked my curiosity. The service costs when considering today's modern volumes of data are still somewhat...staggering. These services also generally offer supreme convenience, no hardware, near zero human interaction during normal operation, and are accessable via the web.

I've came up with a few situations where an online service fits:

1) The data is so critical that even having a redundant(!=backup) DC is not sufficient

2) A local "IT" resource is not present / feasable

3) Onsite backup vs Online is not cost effective (I believe these are few)

What do you think??

MS Exchange P2V

2009-02-28T19:46:00.006-05:00

I've been prepping for this one for a long time. Anyone can vouch for the level of importance that email systems play in today's world. Several months back during planning for 2009, my Exchange 2K3 server came up as hitting its end of hardware warranty. While looking into extending it, I was not excited with the cost compared to going virtual(on ESXi).

My Exchange environment is simple - 1 server, less than 200 users, OWA, & OMA. While this is 'simple' it means everything rides on just one box. Sometime down the road it'll move toward a multi server, split role configuration.

Given the high importance of email and the relatively 'young' status of the physical-to-virtual process I had concerns using P2V on Exchange Server. After much searching and reading, largely through the VMware communities, I gained confidence that if done properly it would work well.

The Keys to Exchange P2V

Testing - build a test Exchange Instance and utilize tools like JetStress
Backups
Consistent Exchange DB's
Minimal amount of System activity during P2V
Highly documented process/plan

The summary of my procedure was:
1) Full Backup
2) Cut off Net Access
3) Disable client related services
4) Incremental backup
5) Disable ALL non system critical services
6) Robocopy (local) data drive (Exchange databases) to iSCSI drive & disconnect
7) P2V the system drive then shutdown physical box
8) Bootup the VM, configure the VM network devices, and connect iSCSI drives
9) Startup services and check event logs after each service startup
10) Test "Internal" Clients
11) Open up external client access and Test
12) Full Production
13) Full Backup
14) Monitor, monitor, monitor

The entire process went smoothly and the new Exchange VM is functioning perfectly. Hopefully this is useful for those unsure about running Exchange Server in a virtual environment or considering an Exchange P2V.

Net positives of this operation include: Less power draw, longer UPS standby capacity, less heat generation, free'd up rack space, better performance, simplified DR, reduced maintenance costs, no client setting changes, etc...

Drawbacks - (Yawn) a late night of lost sleep to catch up on.

Side Note: OEM licenses are not transferable to different hardware - you must have a non-oem license for the new virtual system if the physical box was OEM licensed.

Some helpful resources I'd found:
Article 1
Article 2
Article 3(PDF)

Citrix XenServer - Now Free

2009-02-24T10:13:00.003-05:00

A Slashdot article yesterday afternoon points out that Citrix has made their XenServer product freely available. Big news for Xen fans or those still on the bubble on which virtualization platform to adopt. The XenServer product includes many of the advanced options that VMWare charges handsomely for in their VI3 product line (XenMotion, Templates, Centralized Management, etc).

As the Slashdot article also mentions this announcement comes on the crest of VMWare's flagship event - VMWorld. VMWare users should be biting their proverbial nails with anticipation of what VMWare's response to this will be (I know I am). Many have predicted the pending commoditization of hypervisors followed by their advanced features and I believe that time is here.

Article: How VMware Writes I/O to Disk

2009-02-21T21:59:00.003-05:00

Mike D has a blog entry pointing over to a KB article laying out how the various VMWare products handle disk I/O (depends on the product and host OS!). If you've ever been curious / concerned(you should be) about I/O from Guest to Hypervisor (to host OS) to hardware and consistency in the event of a physical crash, it's a great read. VMWare product users on Linux / Mac hosts take note!

New Poll: Virtualization Platform of Choice

2009-02-17T09:15:00.003-05:00

It's been quite some time since I've put up a poll - so here's a new one. There are a plethora of solutions when it comes to selecting a virtualization platform. I (and others) are interested in hearing what your selection for use in the datacenter is(was) and why you chose that route or possibly changed solutions. VMware, one of the long time incumbents in the x86 virtualization arena, I'm guessing is going to be favored (for various reasons).

I'll cast my vote and thoughts. I'd used VMWare products of the workstation variety "many" years ago in order to run Windows and linux concurrently on the same laptop. The early iterations of their products I thought were OKfor desktop/workstation use but I couldn't see using them in the datacenter. Through a couple versions of the workstation product I observed significant improvements, then for a few years virtualization was off my radar for various reasons.

2007 planning begins and as I look at my replacing my aged equipment Virtualization comes back on the radar. The decision is made to 'crawl' into virtualization, in the datacenter, and use the VMware Server 1.x product. Into 2008 and VMware Server 2, virtualization is here to stay. 2009 - new storage, server capacity needs, and the availability of ESXi (free) have found me on the ESXi platform with an excellent commercial hardware and OSS software SAN. My environment is 'small' thus ESXi free is an excellent fit that allows for adding centralized managment and other advanced features when the need arises.

My long time history using VMware, seeing its progress, and the maturity of its current solutions made it my top selection. Other solutions are still in their infancy, some too far away from the metal, some lacking ease of management, corporate stability, cost, etc.

NFS Performance Tips

2009-02-16T10:35:00.000-05:00

Since I use NFS in my environment, I look around every now and then for tidbits on howto make it better perform. I ran across this article titled NFS for Clusters. I'm not running any "Clusters" that utilize NFS storage, but the concept of a well running NFS service applies everywhere. The article contains some key diagnostics to help determine where NFS may need some tweaking (and how to interpret those diagnostics).

Windows Vmware Guest Time Sync KB

2009-02-03T10:52:00.003-05:00

Accurate time is critical in computing environments - logging, compliance, audit-ability, etc. VMWare has an nice short KB article on configuring Windows guests systems time sync, including the case of Domain controllers. A good read considering the mischief an 'off' clock can cause.

2009 SAN Project

2009-02-03T08:43:00.004-05:00

I wrote earlier about investigations into a SAN solution for 2009 and then a bit later about the eventual decision. The summary of those two blogs is that numerous solutions exists and that commercial "boxed" solutions were not the right fit for my employer. My previous SAN was a linux based Thinkmate system loaded up with SATA drives and two 3ware controllers. It was great from a capacity standpoint, but despite any configuration adjustments, just could never provide adequate IO to support more than 5-6 VM's along with file serving - over iSCSI. It is also at the end of its warranty and Thinkmate does not offer warranty support past 3 years.

My new solution is as follows. Equipment is from Dell. A PE 2900 Racked as the storage controller. Yes it is a big box, but it offers 4 PCIe slots: (2) Perc 6E 512MB controllers, and (2) Dual port Nics. The box will run CentOS x64, NFS, and iSCSI. One MD1000 array shelf full of 15K SAS drives to host VM images, and another shelf of 15K SAS for various data volumes. VM's will be shared using NFS out to ESXi hosts. Use of NFS with VMWare is mature and will allow for backup of the VM's via the storage controller and some minor scripting - no special backup licenses, etc. VM's will run software iSCSI initiators when needing access to data LUNs sliced from the MD1000 "data" array. LAN, NFS, iSCSI are all separate networks, VLAN'd on 2 switches with each server having 2 connections to each network(one on each switch).

How does it work? Let's say I'm pleased with the results and completely comfortable with the decision versus a boxed all-in-one SAN solution. Link fail-over is beautiful, IOPs are fantastic, storage capacity is up.

Crossroads - Linux TCP Load Balancer

2009-01-27T08:53:00.003-05:00

I ran across Crossroads while reading an article on VDI (Virtual Desktop Infrastructure) and thought it'd be a good one to share. The quick summary is, Crossroads is a TCP software based load balancer. This works by directing clients to the Crossroads host system which then takes the request and passes it out to (one of many)backend systems in a manor that keeps the number of connections to each backend equal(balanced). Crossroads also keeps track of any backend that may be down so requests are not dolled out to the down system.

At first glance the balancing appears to be strictly based on number of TCP connections to each backend system (not response time based, or other advanced feedback), which may seem simplistic to what some hardware based balancers may offer. Simple can be good though(so is free - GPL)!

FollowUp: Seeking San

2008-12-17T14:31:00.003-05:00

Earlier I wrote about looking into SAN solutions. Many months before that post I had began a learning excursion - finding different vendors, reading through docs on the various vendor's web sites, and scouring the VMWare forums for user experiences with different storage solutions. It goes without saying that I've had a slight bit of information overload. There are many options out there, each with its own set of unique marketing / technical fortes.

In past lives I've worked with EMC Symmetrix and Netapp Filers and they are great systems without question. I've also worked with "white box" systems running linux +samba/nfs/iscsi. In short I've seen and worked with a decent spread of solutions in some very contrasting environments.

In my current environment being cost conscious is critical - even without taking into account existing global economic issues. Risk / Benefit is always a major factor, and thankfully I've had experiences with a handful of my options in this case and am very comfortable with my final decision.

I don't typically evangelize it, but I am a big proponent Open Source, GNU, and linux. I've used linux in SAP ERP solutions, firewall & routers, file serving(NFS & CIFS/samba), network services(dhcp,dns,ldap,squid,etc), iSCSI, etc.. At all levels of importance, I've used it, trust it, and promote it. The biggest "shortfalls" are: 1) Management doesn't understand anything other than Microsoft or maybe Apple, and 2) Admins are a bit fewer and further between.

Back to SANs and storage. During my investigations with various vendors one conclusion became clear quickly and was not surprising, higher costs. My employer was having to swallow a substantial cost/per useable GB increase to implement any commercial SAN solution. Again from previous experience this was not overly surprising although costs have come down a very long way in the past several years. All these solutions came with excellent feature sets, but these features offered little benefit to my "small" environment that has existed with 99.95% uptime (all inclusive) for years without.

My decided solution will be based on Dell hardware, PERC6 controllers, SAS drives(15k & 10K), and linux (CentOS) providing NFS to ESXi servers and iSCSI to Windows via iSCSI Enterprise Target using dedicated networking equipment segments. This solution will provide significant throughput, high flexibility, great backup options, a very reasonable cost per usable GB, and a low operating cost.

Hooray for Holidays

2008-11-26T09:58:00.003-05:00

It's Holiday Season again. As holiday shoppers are heading out and families and friends are getting together, many IT professionals find themselves working. Holidays and long weekends are the haven for IT maintenance activities when the offices and servers are idle. Unless of course you're running e-commerce or healthcare related systems, then it's a time of booming system activity.

While I'll be taking the next few days off, I will be handling several work items on Saturday - patching, moving to VMware server 2.0(tested of course), and installing a shiny new Dell MD1000 array to handle the IO thirsty set of VM's I run.

It has been a busy Fall season between reviewing SAN options (more to come later), managing a CRM project, planning for a lean(er) 2009, and all the other day-to-day items. I'm certainly looking forward to a few extra days off to enjoy family, friends, hang Christmas lights, and catch up on things around the house!

ESXi or Hyper-V Server 2008

2008-11-25T12:28:00.004-05:00

Ran across an article over at VirtualizationAdmin.com regarding Microsoft's answer to ESXi - Hyper-V Server 2008. VMWare made ESXi, the "core" of their flagship offering, available some months ago for no cost. ESXi has no service console and is strictly managed via remote management tools and also won't perform some of its bigger brother's fancy tricks, but non-the-less is the same foundation of the more pricier offerings.

Now I haven't delved into MS Server 2008 really at all, or Hyper-V for that matter - but this is a great move by MS to step upto the plate for low overhead hypervisors. The article presents a good overview of Hyper-V Server 2008 and how it compares to ESXi.

Regardless of your pick / preference, this is great for all virtualization users. Competition breeds innovation = win for consumers. For small shops, there is now one more excellent "free" virtualization option available.

Seeking SAN

2008-10-08T20:29:00.003-04:00

I've written in the past about iSCSI. I'm now nearing a position, a convergence of 2 situations that has me searching for a solution. My current iSCSI server, a linux based system with 3ware controllers with the iSCSI Enterprise Target software, is nearing its end of warranty with no extension option available. This system provides storage for file services, as well as space for a SQL database used for email archiving and up until recent, space for vm images. The second part of the play is my corporation's move to virtualization. Utimately the plan is to move all enterprise services currently in-house to a multi ESXi server environment.

Keys to this type of environment are shared storage and strong IO capacity. Shared storage, namely SAN storage allows ESXi to perform all of its neat tricks with its vmfs cluster filesystem, not to mention it is the best tuned selection for VM performance. NFS is also an option, but removes access to a few of ESXi's great features, and carries processing overhead. IO capacity is critical - all the space in the world can be quickly lost to complaints of lag and slowness if that space does not have IO capacity to perform(Exchange/SQL).

So I've began the adventure of looking for a solution. A solution to handle VM images, SQL, Exchange, and file serving; a lot of different IO profiles. Thus far I've looked into following iSCSI solutions: Dell/Equallogic PS5000 series, Netapp S / FAS lines, and Compellant. They each offer their different strengths and less desirable aspects. Before any decision is made, there will be demo units put through the ringer, but I'd also like to hear from anyone out there with experience in this arena. For my size organization this solution will be a reasonable sized invenstment and it needs to be right the first time.

Data Security: Encryption with TrueCrypt

2008-10-02T10:36:00.004-04:00

The onion analogy is used by security people to describe the many different aspects of a complete technology security program. Application security, network security, operating system security, physical security, people(social) security, etc. A business' technology environment is made up of many components that must each be addressed and analyzed.

The core reasons for comprehensive security can be distilled down to a few items: 1) Availability - keeping business systems running, 2) Integrity - not allowing data to be corrupted/poisoned, 3) Privacy - not allowing data to become public. I'm sure others can chime in other reasons as well, but I use these 3 as my core values with any security considerations. I will however throw in one collateral value that comes from these 3 but is paramount in today's world - corporate image / trust. A company that has lost consumer trust / confidence due to a security breach, even though they have minimized / mitigated any of the 3 core values, will experience severe hardship (and likely legal penalty).

Doom and gloom aside - there is one tool out there that offers some piece of mind for some layers of the technology security onion - TrueCrypt. TrueCrypt is a freely available tool for encrypting data, including entire operating system hard drives. It is a multiplatform tool that runs on OSX, Windows(2000,XP, Vista), Linux and can be ran directly off of removeable media(USB stick, etc). A file / disk encrypted with TrueCrypt can be accessed by any other computer that can run TrueCrypt if the password is known. TrueCrypt can even create 'hidden' encrypted drives that are only detectable by someone who is aware of its existance.

Here are a few common business situations that TrueCrypt can help out with. A lost/stolen laptop who's hard drive has been encrypted with TrueCrypt. A lost / stolen usb drive that has been encrypted. Server(s) stolen from the office / co-location. Securely sending sensative information on a CD/DVDs.

In my office all USB drives are required to be encrypted and catalogued. The next image developed for mobile computers will feature full hard drive encryption as well.

VMWare Server 2 Released

2008-09-24T08:42:00.005-04:00

There were predictions with VMWorld going on this week that the free VMWare Server 2 would be released - it has. Hop on over to vmware.com to read more & get a copy. Some notable new features from the 1.0 line: Passthrough SCSI device for guests allowing direct access to physical devices such as tape drives, USB 2.0 support, 8GB RAM support in guests, VMI interface and hot adding of scsi drives.

I've been a long time user of the 1.0.x product and look forward to testing out and using the improvements in 2.0.

New Poll: Internet Connections

2008-09-22T09:46:00.002-04:00

Five people or 5 Thousand, chances are the office has Internet connectivity. In tech jargon these connections are often called pipes - for good reason. A big misconception / misstatement about Internet connections is one is "faster" than the other. The reality is that electrons all move at the same speed and what people are generally referring to is how long it takes to 'download' something. Going back to the pipe analogy, water moves through a 1 inch pipe at the same speed as a 6 inch pipe, but the 6 inch pipe can move more at the same time.

Wikipedia has a breakdown of bandwidth that different connections offer. An office's selection will generally be based on few requirements. 1) Throughput/bandwidth commonly called speed. 2) Redundancy / resiliency, 3) Service availability.

In my office a dual T1 connection is used for the equal reasons of both bandwidth and redundancy. Things would be ok running on one circuit, just slowed slightly. My location has left few options for alternative connections outside of the phone company's services for the case where a cement truck runs over the large telco box down the street. There is not a cable feed within a reasonable distance and satellite is not known for supporting vpn connections well. I did recently find out that Verizon offers staticly addressed wireless cards for a minor setup cost. In testing, these cards can offer T1 comparable speeds (using an antenna) with only minor latency overhead. Any comments / experiences with using these in this regard are greatly appreciated.

There are too many options to post a poll on what type of connections people are using so please just comment in with your thoughts.

Interesting Tech: Dropbox

2008-09-18T14:24:00.002-04:00

This probably isn't suited (yet) for businesses beyond a handful of people, but Computerworld.com has an article on "Dropbox". Dropbox is a tool for synchronizing files between pc's using their intermediary service built upon Amazon's S3 storage. It appears to be a young tool with some limits and a good bit of promise. Check out the Dropbox FAQ for a good list of what it is and what/how it does it.

Might be a good tool to help keep an updated copy of key documents in case of disaster(if you don't already employ a DR plan that does this) - Or for use at home to keep those precious digital memories backed up. It's not a backup service in a technical sense - it only keeps copies of live documents, no history. It does however offer an "UnDelete" feature.

It claims to use encryption, but per the FAQ you can always put your files in a TrueCrypt volume and then sync that up.

Internal Chargebacks

2008-09-16T16:05:00.002-04:00

My current place of work is 'small' enough that IT does not 'charge' the business units for services provided.

From a business perspective, I favor charge-backs for a few reasons. First, I think it is important that business units have some concept of how much IT resources cost, and probably one of the most direct ways to make this happen is seeing it hit a department's budget. Secondly by 'seeing' how much IT costs, requests for changes / new services are less blindly requested with little concern for costs. Thirdly charge-backs force IT to be aware of costs down to a relatively fine grained level. This last fact should be taken advantage of irregardless of whether or not charge-backs are used or not.

Before I continue on with that last statement, I do acknowledge that charge-backs have draw-backs. Charge-backs introduce a good bit of accounting overhead. IT must very accurately measure and report on services provided. This can be a very complicated task depending on the services provided, the supporting infrastructure, and business organization. Then of course there is all the actual accounting work, internal invoice and bill processing. Execution of this concept definitely adds cost, hopefully with some return/savings when done properly

Again regardless of whether or not IT does charge-backs or not I think knowing, at least within some small margin of error, what services cost is incredibly valuable. Like any other metric, availability for instance, knowledge is power. When looking for new solutions, having an industry standard or competitive per user cost for a service can be a decision make or break factor. When evaluating a replacement / service upgrade, knowing what the 'old' service cost and then being able to compare that to the new service can help make the proper select to keep that per user cost in a target zone.

I started off mentioning that charge-backs are not done in my current workplace. I have however, over the past several months been working through constructing a spreadsheet that exposes the true costs of IT services. It has been a challenge to properly assign metrics to services - ie what is the true capacity of an MS Exchange server, and what are all the associated bits behind an email account(much more than just a CAL). I've finally come to a place with it that I'm fairly happy with. Its made apparent many facts, some good - some which need improvement, that will be very valuable in my future decision making processes.

New Poll: CRM

2008-09-15T09:44:00.004-04:00

I blogged earlier about CRM and wanted to put a poll out there to hear what others are up to regarding CRM. My workplace is still on track to begin a SageCRM implementation in the next few weeks. It is definitely one of those tools the users aren't really sure of what it does or even is. I've been working hard to spread the word it is something that can help significantly improve the level of customer service provided by bringing together the details of various interactions different employees have with customers in one central location. Another major point of any rollout is adoption. A big part of the final vendor selection was usability and I feel confident SageCRM will be viewed as intuitive to use - it is natively web based and integrates incredibly well with MS Outlook. Some department managers have also been tasked to ensure their areas are 'motivated' in a measurable way to utilize the tool.

A few small (15-20 people) sister companies I know live and die by their CRM. What are your CRM experiences - good and bad?

System Imaging

2008-09-11T09:57:00.005-04:00

Anyone who has to manage PC's is probably aware of the time consuming task of having to 'setup' a computer from scratch. I'm talking about formatting disk drives, installing an operating system, applying security updates, installing applications, and configuration. This set of tasks can easily consume 4-6-8 hours of time and from a business standpoint is time poorly spent.

There are solutions out there. One approach is to use thin clients or stripped down pc's that only serve the purpose of connecting to a server for all useful applications. This works well for local users needing a very standard set of applications, not terribly well suited for mobile/disconnected users(However some really cool [aka expensive] solutions do exist for this). I personally utilize low spec'd PC's running a next-to-nothing OS install at my workplace - they work great, are cheap to obtain, and last several years. The other type of option is to use system images. System imaging is the concept of creating a "golden" setup that can then be wholesale copied to different computers. While the process of creating the 'golden' image can be time consuming, the benefits are significant. A PC can be taken out of the box, have the image copied to its drive, up and running in less than 30 minutes with a standard set of sofware and configurations. Apply this process to several PC simultaneously and the time savings are huge.

From a support standpoint the use of images creates a very consistent environment to support. A PC can also very quickly be 're-imaged' and returned to the user if is a spare is unavailable. If you're talking about a physical system, creating an image right before testing a new 'something' is also a great backup method.

I initiated the use of images in my current workplace and I can't begin to count the amount of setup, and support time it has saved. I've used Symantec Ghost for imaging in my MS-centric environment over the years and it works very well. I'd love to hear comments about other imaging solutions people have used, commercial or open source.

Social Networking@ Work

2008-09-09T06:30:00.001-04:00

I began blogging for mostly professional reasons (check out my first entry). Here and there I run across articles on companies that internally (and externally) exploit blogs, and forums for business use. For larger companies I can kind of see the value of this use but still wonder about : A) Don't most of these companies have Intranets for corporate information / happenings, B) How to keep employees from 'blogging' the day away, C) policing of content.

If you're using blogs / forums internally, what was the business motivation to do so, how was is sold to management, and how is it operated(content policies, etc)?

New Poll: Virtualization

2008-09-08T09:01:00.004-04:00

Poll Day Topic: Do You Use Virtualization

Share your experiences, opinions and environments about virtualization. What solution do you use and why. Are you avoiding virtualization? What has worked for you and what has not.

I use it - at work, and at home. In the office it has saved tens of thousands in equipment alone, not to mention some of the 'intangibles': cooling, power, physical space, recoverability, testing, etc. I'm a VMware fan for several reasons, but I'm not looking to start a holy war over the best solution - here anyway. Some key points I've taken away from my virtualization adventures: 1) RAM, RAM, RAM - have lots of RAM in the physical server. RAM is your best friend and it is inexpensive these days. 2) VM's generate lots of small random IO - have a properly designed disk subsystem (aka avoid Raid5). More spidles are your friend. 3) Understand the workload of your VM's. Checkout one of my older posts about virtualization in general and why it's valuable in nearly any environment.

Linkage: Storage Tips

2008-09-05T08:24:00.002-04:00

I've only poured over a few of the many articles, but there's lots of good info about SANs/SCSI/SATA.
-To SATA or not to SATA
-Price vs performance in selecting a RAID configuration

SearchStorage.com

Meeting Room Mayhem

2008-09-05T06:30:00.001-04:00

I work in an office that has a handful of meeting rooms, some large enough for seated groups of 40, others with long boardroom style tables for 20. Each of these rooms has a projector, either ceiling mounted, or simply set on a small rolling table/cart. Each of these rooms also has either a PC or a laptop that is networked an usable by anyone with a login. It never seems to fail when I see a call from an extension in one of these rooms, that the previous occupant proceeded to disconnect any and all wires and completely rearrange the seating. In the past I've tried neatly arranging / wrapping any wiring to make appearances as neat (and safe) as possible with the hopes the overall 'neatness' of the room would be better kept.

I'm searching for any (realistic) concepts of how to (even subconsciously) motivate people into returning a room to a 'standard' when they are finished using the room.

JRH

Linkage

2008-09-04T06:30:00.002-04:00

Thought I'd share a few of the blogs that I frequent. Chances are if you're reading me, you'll like these as well.

Savagenomades.net is done by Jason Benway (An old college classmate of mine).
The Standalone-Sysadmin by Matt is always full of good material
TheGeekDoctor by John Halamka - insightful writings, chances are you know his name already.

I'll pop these into a sidebar gadget for my and your convenience.

JRH

CRM

2008-09-03T06:30:00.002-04:00

Guess what, it's another 3 letter acronym: CRM. Customer Relationship Managment. It's nothing new but has been a consuming item in my career for the past 12 or so months. At its core CRM is a centralized repository for information - typically contacts, freeform notes, that allows many people to see, review, and update this information. CRM solutions exist that are tailered for as few as 1 persons and up to thousands of users.

As a systems / network administrator, I appreciate and acknowledge the value of 'centralized'. Having anything in a highly organized and shareable, consistent format makes those 'anythings' much more valuable and efficient.

In organizations where a 'team' of different people interact with customers, CRM offers many positives. When contacting / being contacted by a customer CRM can be referenced to see what the last communication was, when it was, and who it was with. This type of background is invaluable to providing efficient customer service be it sales support, technical support, or billing/account support.

From a HR standpoint, CRM helps solve turnover issues. Instead of users keeping unstructred notes, who knows where, CRM keeps data in one place - consistently structured. This also applies in situations where customers are handed off between sales account managers, or even internally between departments/divisions.

I was tasked with finding a solution to many of the above 'challenges' and putting together a project through implementation. I started this off with a few small meetings gathering a list of desired features and then prioritizing those into 1: Must Have Now, 2: Must Have Later, 3: Would be Nice. This was invaluable when speaking with vendors on what their CRM product was capable of and developing a staged implementation. Many vendors quickly had to decline due to not being able to fullfil #1 items. At one point I was left with only a single vendor, not a good negotiating standpoint. I was able to find another offering that eventally ended up winning the project and that'll be kicking off just several weeks from now. More on that as it unfolds.

I'd like to stress a things that stick out during my reflections on the project thus far. The first is having a list / knowing what you want to get out of a CRM solution. There are too numerous solutions out there to list out but knowing weather or not ACT! will work for your needs or if a solution as elaborate as saleslogix.com or salesforce.com can really be simplified by knowing your end goals. From technical standpoint there are lots of architechtual differences that play into the decision making process. Some solutions are completely web based(connect from anywhere), for some web connectivity is an ($)add-on, some only offer local network access.

Is your workplace using CRM, what's your impression?

iSCSI - SAN for Small (and big players)

2008-09-02T06:30:00.003-04:00

For years all the buzz in storage has centered around 3 letters SAN. The storage area network, fast, centralized, flexible, resilient. Placing storage directly in each server is costly, inefficient, risky, and slow. And for some number of years vendors and partners and such 'hid' this term from smaller shops due to costs involved with a SAN architecture that was designed to meet all the marketing accolades. Fibre channel equipment was really the only option.

A new technology began to surface 3-4 years ago - i(nternet)SCSI. I won't rehash all the details that Wikipedia can provide except for the main point of iSCSI - commodity interconnects. iSCSI allows for the use of common networking equipment: switches, NICs, cables. iSCSI does not require the use of $1K+ HBA's, fragile cabling and specialized directors/switches, etc. The reality of iSCSI's pratical use is here due to the commoditization of gigabit ethernet and soon to be 10gig ethernet.

iSCSI also brought options to storage subsystems. No longer are costly FC drives the only option, but also SCSI, SAS, and even SATA. A multi terabyte array can be obtained or even constructed for costs well within reach of nearly any business. iSCSI can also (and often does) exist a strictly software world - no HBA'$ required. Some argue the software overhead is detrimental to overall system performance. I'd argue today's systems have sufficient processing capacity to run software iSCSI in situations where iSCSI is a good fit.

With this commoditization of the SAN, vendors have been quick to offer up 'solutions' for smaller shops. I have a caution to throw out that requires a little lesson on storage. First off, the most important item in selecting the proper storage is understanding the application(s) that will access it. I can't elaborate on this enough - don't even start to think about how much space you'll need until the application is understood. In fact refuse to talk about capacity with any vendor until this is ironed out. Try throwing a few virtual machines at solution that only does (or has been configured for) RAID 5, along with a file server, or mail server - it will not make you feel good about the money spent. At one point I had a prominent vendor's rep suggest a 4 spindle SATA system to me to replace a (6) disk file server and an Exchange server. All that person was concerned about was how much space and quoting his cheapest offering to improve the chance of a sale. I don't carry any certification badges, but I am well informed and experienced, and I know when I'm getting a snow job.

I ended up with a 16 SATA drive over two 3ware controllers system running linux and the open source iSCSI Enterprise Target software to provide my iSCSI storage. It's been rock solid. Its 8 drive RAID10 ran up against performance issues after a handful of VM's where thrown on it, but in all fairness virtualization was not in the picture when the system was spec'd. The next iteration will be 15K SAS based for VM storage.

So why use a SAN/iSCSI? A few main concepts are modularization and maximizing the storage investment. By detaching storage from a specific system several benefits are realized: 1) Smaller form factor systems can be purchased since they don't need to be filled with drives. 2) In the event of a system failure the storage can be accessed by a different system. 3) It fits into a virutalization environment for lots of reasons. 4) Designed right it will be faster than local storage. 5) Because multiple systems utilize space on the iSCSI server, the disk investment-to-utilization ratio is typically better.

Downsides include: 1) Slightly complicates overall architecture, 2) Small bit of iSCSI management expertise, 3) Many eggs in one basket situation(get a good warranty). With the right system and financial resources the iSCSI server can be setup to mirror with another system for improved reliability.

Know your applications, read about technology and reviews, search vendor support forums/blogs for real world feedback on your purchasing considerations.

Storage Advisors Have LOTS of great blogs on RAID levels/storage and their INs & OUTs

JRH

Office Phone Systems

2008-09-01T06:30:00.002-04:00

The motivation for this article came from a situation sprung on me a few months ago that I'll elaborate on a bit more later. Chances are if there are more than a handful of phones in your office, you have an office phone system of one type or another. I'm not going to jump into the various types of technologies behind phone systems, quite intentionally so. One quickly discovers that phone systems, much akin to networking, have their own mystique and black magic, and can be quickly overwhelming with acronyms and terminology. Like networking and other 'complicated' subject matter, the experts are often best left with handling the fine grain details of phone systems. This isn't to say a moderately savvy person can not handle the basic / common phone system operations.

Moving closer to my surprise 'situation'. Phone systems are very important parts of business, there isn't any questioning this fact. I can't help but feel this business value is exploited to a certain extent by phone system manufactures and those who install and support them(partners). Anyone who has purchased a system that supports more than a handful of extensions quickly realizes the investment is substantial, not to mention ongoing support. I'm talking the kind of investment that stretches not 5 years but 10 or 15, maybe more.

The meat of the situation. The system I'm responsible for was made by a prominent manufacturer whose name starts with the letter "A". It was a good size investment when installed just 5 year ago and is currently supporting around 250 extensions with room to grow. The surprise I received was that part of the system will no longer be supported post 2008 by the manufacturer. The system is very modular so maybe this isn't that big of a deal, right? Well it seems that the 'suggested' upgrade path is more akin to replacing the entire engine of a car versus just 4 new tires and proportionally so in cost as well. Three things came to mind in this scenario: 1) So I spend this money to have a supportable system without any other added benefit, 2) When will the next 'Upgrade' have to take place, and 3) aren't phone systems supposed to last a long time?.

My next move was to investigate 3rd party support. In searching it seemed to be a very common situation where customers were presented with obsolescence / costly upgrades from manufacturers and end up using 3rd party support. Great I'm not alone. The endgame was that I've found what appears to be a very reputable provider (according to the several references I've spoken with) who can support my system (better than the mfg) long term, and for less cost.

I was not involved in the purchase of the current system thus did not have a chance to ask any questions. This situation does highlight a few pointers for the next time a new system purchase comes around. 1) Consider 3rd party support from the start or immediately after the initial mfg warranty is over. 2) Get feedback from customers with similar equipment and a long history with the manufacturer.

Backup

2008-08-29T08:40:00.002-04:00

Like insurance, computer backups are the type of item that feel like they 'cost' more than they ever return. This feeling is quickly reversed the instant a backup is put to use, but depending on the shop, backups typically happen a lot more often than restores. There isn't any question backups are a necessity, but the real question are: How and Why.

The "Why" question really refers to selecting the best "How" for a shops situation: Why is this my solution. It involves details like: how much data must be backed up, how often must it be backed up, how fast must it be backed up, how fast must it be available for restoration, where must the backup be stored, etc.

I've ran across a few different "How's" in my career. Some of these involved FC attached libraries, autoloaders, single drives. These are all 'local' backups, where the data was copied to some type of local device and managed locally. There is something to be said for today's modern tape technology - it is soo much better than from 5-7 year ago. Speeds are great, capacities are way up, reliability is significantly better. But - tape is labor intensive, having to be handled, moved offsite, back onsite, sensitive to temperature, etc. Good quality tape drives(LTO and up), either stand alone or inside an autoloader/library are costly for smaller businesses. And when a tape drive decides to start acting up, it is your own worst enemy - writing bad data, eating tapes, failing backups.

In recent years an additional local backup has come into popular use - disk. The plummeting cost of disk drives and sky rocketing capacities have made disk drives a very attractive backup option. There are 3 approaches used with disk backup: 1) Disk replaces tapes. Here backups are written to removable drives and then moved offsite / rotated just like a tape would be. 2) Backups are written to disk first, then tape - known as D-D-T(Disk to Disk to Tape). Here the disk acts as an intermediary(with many benefits). 3) Mirrored remote SAN. This is big business stuff mostly using expensive equipment and network connections to actively mirror live data. It's also more of a disaster recovery / failover solution as it does not preserve historical versions.

Going back to #2 above and the "Many Benefits" part. By putting the disk between the data source and the tape drive several good things happen. First off many backup streams can be written to a disk simultaneously, minimizing jobs having to wait for a tape drive to be free. Secondly because these disks are usually in the backup server or 'near' it, when the data is written to tape from them it can be done much more quickly / smoothly, freeing up even more tape drive time. Thirdly, there are now 2 copies of the backup, always a plus. Lastly, if the data is still on the intermediate disk when a restore is requested, the data can be pulled from the disk instantly versus searching for, loading, and seeking through a tape.

There are too numerous to mention local backup software options. The keys to the proper selection mainly involve: The kinds of data being backup up(database, email, normal files, etc), the type of systems being backed up(Windows, Linux, UNIX, etc), and cost. All reputable softwares should support modern media types, be it tape drives/loaders/libraries, VTLs, or disk drives.

Another option exists that can be attractive in certain situations - online backup. Online backups start with an account on a vendor website, an Internet connection, and a small piece of software on the source systems to be backed up. Often times there are options for the backups to run continuously - capturing all file changes in real-time and transferring them securely online for backup, or scheduled backups that only run at certain times. There are a few immediate cautions with these solutions. 1) Is the Internet connection fast enough to transfer the data efficiently? 2) How is the service charged: per MB transferred/per MB stored/per MB on the source system/etc? 3) What options exist for restoring data, both small files or entire systems? An online backup can be a good match given the right billing structure and a desire for low IT resources.

Active Directory Restore Fun

2008-08-28T06:30:00.002-04:00

This adventure all began with testing out a remote access application. In my earlier experience testing a previous version of this application, it pulled a local copy of my Active Directory users and groups during initial setup which then had to be pruned down to those who actually needed remote access. After setting up the most recent version, I promptly began pruning the users and groups shown in the application to the appropriate list - then the phone rang.... Needless to say the current version of the application was directly accessing AD, not just for authentication, but to maintain its user list --- that I had been actively pruning.

Needless to say this was not the best situation. No fear - there are backups for this sort of situation. But beyond backups, the restoration of AD is not a two click and it's done operation. This isn't a story about "Oh no the backups junk" or backup schedule philosophies, but I will comment of the merits of using Disk-Disk-Tape backups quickly - backup & restores are quick. Back to AD. My AD landscape is fairly simple: single physical site, 3 DC's, relatively small database.

I don't carry any MCxx certifications and thankfully the opportunity to perform this type of operation is not a common thing. Thankfully between MS Support Knowledgebase articles and documentation in my backup software the AD restore operation was smooth.

I had restored a DC in a VMware test environement before, but it was a single DC configuration which changes things a bit. Aside from not realizing up front how the remote access application integrated with AD the following lessons can be taken away:
1) If at all possible have a test environment similar to production - and try these things out.
2) Have knowledge of proceedures or how to find it - quickly
3) Take a deep breath when these things happen - think quickly, act thoughtfully
4) The addage of one-application to one-server is priceless in restore situations
5) Having multiple DC's even in a small environment helps keep things moving along.

[Update on Backup / Restore resources]
AD Restore on 2K - Works for 2k3 (MS Support KB)
Good notes on using NTBackup on DCs for Backup / Restore (MS Support KB)
I use Symantec Backup Exec in my environment, but NTBackup is good for just a few systems.

Thanks for the comment Matt & good luck jumping into MS infrastructure (I'm a *nix guy by trade too).

Virtualization in Small Companies

2008-08-27T06:30:00.003-04:00

The news and media have been chock full the term Virtualization for over the past year and very heavily so since 2008. There are various offerings from several prominent vendors and a new option seems to pop up every few weeks. From a small company point of view these offerings appear catered to larger business. After all many companies out there don't have hundreds, let alone tens of servers - and what's a 'SAN' anyway.

I'm not sale rep / associated with a sales rep from any of the virtualization solution companies. I will however say there are very compelling reasons to use the technology, even in the smallest of environments. Akin to this - many of the 'beginner' virtualization solutions are free to no cost and as such can not be used as an excuse to avoid the technology. In fact, the use of virtualization in companies with only a few systems can be very compelling for some of the following reasons.

1) Hardware Independance. When you only have a few servers and suffer an equipment failure, recovering a hardware dependant system is a nightmare and a huge prioirity. VM's are run against a virtual BIOS/hardware that is more or less identical between host virtual server systems. This means a virtualized system can be restored to another host with little to zero hardware / driver issues and be off and running fast - priceless!

2) Hardware Cost Savings. Starting with 2 physcial systems consolidated to one, hardware savings are acheived. For the sake of having a backup host this isn't the best idea but the savings concept is key. Virtualizing only a handful of systems eases the selection of equipment. A modest system by today's standards can run a handful of virtual servers without worrying about having a high performance SAN or similar storage / network environment.

3) Return on equipment investment. Every business purchase is an investment, and getting the most value from each dollar spent is a big factor in buying decisions. Consider 4 servers, each costing 2 to 3 thousand dollars each. This situation presents a total investment of 8 to 12 thousand dollars and lots of computational power generally sitting 85% idle, with 4 sources of heat, space and power consumption. Virtualize those 4 systems to one 4 to 6 thousand dollar server saving thousands up front with one source of heat, power use, and noise. Or conservatively to two 4 thousand dollar servers still spending the same or less on equipment but halving the noise, power use, and heat dissipation and still reaping all the other benefites mention here.

4) Testing. Using virtualization opens the doors to smaller shops where it isn't feasable to have equipment dedicated for testing. A modest desktop with enough memory can run a handful of VM's for configuration testing purposes.

5) All the usual suspects: Less equipment space, less electricity, less heat produced, less noise, longer battery backup runtime, etc.

All this being said there are of course drawbacks. Virtualization brings along with it new concepts in system management that must be understood. Administrators must generally have a strong understanding of system resources to maintain an effective virtual environment: system memory, disk IO, networking. For smaller shops, putting many VM's on one host creates a "Many eggs in one basket" situation. Backups - there are additional backup options avaialbe to VM's that need to be understood. Certain applications do run better on dedicated physical systems versus the virtualization environment a small shop could support.

All in all virtualization is great. Even utilizing the free solutions available, great efficiences can be acheived compared to 'old school' one system to one physcial computer. You don't have to be a 'big' shop or invest thousands to reap the many benefits of virtualization!

Hello World

2008-08-26T09:42:00.000-04:00

I've come to the realization over the past several months I've enjoyed reading a handful of other tech related blogs some more technical, some more business oriented and decided I would try my hand at one. I've found good genuine technical advice, invaluable real world experiences, and just plain enjoyable reading out there and feel I should relay some back.

It's also becoming a staple for most professional careers to document one's experiences. In this world full of (often too) much public information, the importance of personal experience and sound information is invaluable. I'm certainly not the definitive source for some if any topics, but I have to be for both my family and the real world issues I deal with daily.

A short list of the things I deal with each day(Search engine food mostly): Windows Servers, Active Directory, Wireless Networking, SQL Server, MS Exchange, Citrix, Office Applications, LAN, WAN, VPN, Linux, Security, Backups, iSCSI, Phone systems, Remote Access, Capacity Planning, Budgeting, Business Policy, Office Politics, Virtualization, A/V Systems, Firewalls, SAP, etc. All the fun things in today's modern businesses.