New Location ToDo List

The 'new' site I've been working with over the past several weeks has several traits I would say are pretty typical. A small, very successful business is acquired / merged with another that is owned by a fairly large parent that also happens to own my place of $WORK. I've had the pleasure of being involved to help get their infrastructure, projects, strategy, and support into a more organized, best practice, and business aligned status.

There has been a lot of learning from all sides up to this point. As I had mentioned most of the items needed attention are not surprising knowing the company was a grassroots effort that has grown incrementally over the past 10 years.

• VOIP and computer data are traversing the same network
• Some of the switching equipment is non-managed grade stuff
• Switches are piggy-backed(Did I mention they do CAD work)
• The primary IT person wears many hats and is severely overloaded
• A new business software is mid implementation
• The primary IT person is going on maternity leave within the next month
• There is one server doing everything
• Telco and Internet services are not cost effective / lined up with the business needs
• No automation - Updates, software installs, PC imaging, etc
• Remote access needs some help
• Lots of processes need documentation / standardization
• No tracking of support issues / history

The challenge for me has been uncovering all the details of the organization and infrastructure to ensure the decisions to right things going forward are sensible. Thankfully I have a good team at $WORK, supportive management, and plenty of great people at both locations. This is not an open call for vendors. Solutions to most of the above items have already been devised and some implemented - the timetable has been the biggest challenge to meet.

Latest Happenings

The last several weeks have been an extremely busy time. I've taken on the responsibility for a recently acquired sister company's technology and people. The site has a lot going on to say the least. Maybe a "Perfect Storm" of needs, changes, growth, and projects would be a well suited analogy. It's a very exciting opportunity for everyone involved that is going to take a lot of hard work - wish us luck!

Trial Run: Aprigo Ninja

Fellow blogger Matt Simmons had previously blogged about "Ninja" by Aprigo. After bringing it up in passing again I thought it might warrant a try. Shortly after registering for the private beta, I was approved and provided access to the less than 5MB download.

I am not affiliated with Aprigo in any way - not a current customer, relative, friend to employee(s) there. My motivation for trying the product was its visual representation and analysis of data. I was curios how the volumes of file server data and underlying equipment that I manage was being utilized - beyond simple user quota figures.

Install was a breeze, the typical click, next, next, finish. At first run you have to setup a scan by pointing the application to a folder,share, or UNC path and providing a user friendly name. The trial is limited to 500GB per scan and seems to only be able to run one scan at a time - interactively.

Lots of interesting results are available after a scan is completed - most of them with 'drill down' capabilities to get the finer grained details. If a cost is entered, per GB/year, a dollar figure is displayed next to each result. This is great for seeing how much it costs to keep the different categories of data laying around. The non-trial/beta version appears to offer trending reports, cloud storage analysis and access management(very interesting!) as well. Aprigo Ninja is definitely a handy tool if you're getting going on trying to analyze your storage use or Windows File Server Resource Manager isn't cutting it (or you want to analyze data on samba shares). Give it a try, it's worth the small bit of time and effort.

Free vSphere Videos / Demos

I picked this up on twitter via @DellServerGeek: Mike Laverick has generously posted many vSphere how-to demos on his website. This is a significant body of work to make freely available. Mike goes through the entire process from "soup to nuts". For anyone curious about setting up a vSphere environment these are a must watch! If you like the series - consider procuring his accompanying book. Thanks Mike!

Free Storage for Your vSphere Lab!

VM Super Genius has a post with a link to free storage solutions that can facilitate trying out some of the more advanced features in VMware ESX / vSphere. As noted many of those features require shared storage (iSCSI, NFS, Fiber SAN) which can be prohibitively expensive for many. The list offers some zero cost appliances / applications that can provide iSCSI and/or NFS storage to virtual hosts. An excellent blog to follow if you are into virtualization.

Playing with DRBD (Replication)

Over the past 4+ years I've worked towards moving the infrastructure at work from isolated physical systems to a centralized storage(SAN) and virtualized paradigm. I'm happy to say we are definitely a "Virtualize First" and SAN shop now. A conservative comparison of separate physical systems with local storage to the current SAN/virtualized environment shows a nearly 50% cost savings on equipment alone. The benefits of extra rack space, lower power and cooling costs, I/O and compute capacity, etc are nice as well.

This evolution has a next logical progression. Now operating in a very centralized fashion, I can (more easily) begin examining solutions for replication. The base goal was to replicate the SAN to mitigate a 100% loss of the primary server room and not require an enormous amount of time restoring from tape. Tapes are great snapshot points in time like photographs, but like photographs, trying to rebuild your entire set of memories by looking through photographs would be very time consuming (and photographs degrade). Replication is not a replacement for good backups.

From a previous post my SAN is linux based, much akin to an Openfiler solution. It provdes NFS storage to ESXi servers for VM images and iSCSI storage to the VM images that need data volumes. All physical drives (as present by the RAID controllers) are sliced up using LVM. It has been humming along without issue since installation in February 2009.

I have known of the existence of DRBD for several years but not been in a position to utilize it. In short DRBD is a block device you layer into your device chain, just like LVM. It's specialty is taking all the original block level changes, keep track of them, and send them over to another system where they can be duplicated. The DRBD website is excellent, I highly suggest spending a few minutes there. DRBD has a few very nice traits that I'd like to highlight. First off it is smart (and dumb?). It works at the block level and knows which bits may be out of sync and will only send those bits across the wire - it knows nothing of filesystems, files, etc. Secondly it can be non-destructively added to existing data volumes. There's no need to backup/install/restore. DRBD is opensource and freely available - but its creators and primary maintainers, Linbit, offer commercial support and have been around for awhile. Linbit also offers a closed source product, DRBD Proxy, that is designed for long haul, high latency(200ms) connections or greater than 2 node replication situations. If you want to replicate outside of a LAN using DRBD you'll need it. DRBD is also 'good friends' with Hearbeat for high availability / failover situations.

I setup a couple CentOS x64 based VM's for testing. DRBD is available via the standard CentOS repositories but it is naturally a bit behind the current version available directly from the DRBD website. The download is small and if you have a basic compiler toolchain, and kernel-devel package the build / install is quick and painless(make rpm). Did I already mention the DRBD website documentation is fantastic - really go read it. The required configuration to have DRBD work is quite minimum although there are lots of options to fine tune its operation. If your data's rate of change is very high, you will really want to have Gigabit connectivity between your nodes. What you'll find is your DRBD devices will only write about as fast the data can get across the wire (assuming your drives can outrun wirespeed). If you need more than wire speed and your drives are fast, take a look into the DRBD Proxy product. I spent a fair amount of time in different scenarios to see how DRBD would act and what to do as an admin in those situations. Like many things, with a little bit of time and reading, DRBD was easy to work with.

So what was I doing with all this again? The base goal was to replicate the SAN to mitigate a 100% loss of the server room. Since the SAN literally contains everything (VM's, SQL, Exchange databases, file shares) this was a fairly simple move that captures the entire datacenter to another system. To backpedal a bit, my environment is modest in size by any modern measure, but still just as important. That 'size', centralized storage, and a geographically large site made the option of placing the replica system in a local but 'distant' (fiber connected) building a perfect option. The replica runs ESXi with a CentOS VM running DRBD to replicate the data. Why ESXi on the host? What this more or less creates is my datacenter-in-a-box, transportable if needed. The CentOS VM will provide NFS access back to the host ESXi for access to all the server VM's which in turn will use iSCSI access their data. ESXi virtual switches let me create matching, non routed networks local to the replica host for the NFS and iSCSI traffic, meaning zero reconfiguration of the production server VM's. This isn't meant to be a powerhouse / failover solution. What it is, is a very cost effective solution to a worst case situation that hopefully doesn't occur. If the worst was to occur, some scripting magic transforms the replica to production status. When a new production environment is established, DRBD can be used to mirror the data back to it enabling a transition with very little downtime.

Tired of reading yet?

Shared Hosting with Exchange 2003

This post is as much a placeholder for me of some information I found some time ago as well as something that may be useful to others.  In late 2007 I was tasked with providing "hosted" email services for a sister company out of my employer's Exchange 2003 environment.  OK I thought, this should not be a big deal. The system has plenty of capacity, bandwidth, and Exchange 2003 has to have some mechanism to pull it off.

I started off by setting up a small environment in VMware: Windows Domain, Exchange server, "Internal" client, "Internet" firewall, and external client.  All of these of course sat on private networks internal to the host server.  I installed all the same applications to best mimic the real world - Antivirus, Anti Spam, Archiving, etc.

So I have this environment all prepared with a few users at the "primary" email domain, now what?  I turned to one of my favorite exchange online resources: www.msexchange.org and without too much searching found Part 1 and Part 2 of Shared Hosting with Exchange 2003.  These are both well written articles that made the project pull off without a hitch.

I know Exchange 2003 is on its way out(long gone?) for many people, but at the same time I'm sure there are just as many Exchange 2003 environments out there that will be around for a handful more years.

One key note to keep in mind.  During planning to add valid 'foreign' user accounts to your active directory,  consider ways to prevent those accounts from accessing other resources tied into the active directory domain!