Wednesday, December 17, 2008

FollowUp: Seeking San

Earlier I wrote about looking into SAN solutions. Many months before that post I had began a learning excursion - finding different vendors, reading through docs on the various vendor's web sites, and scouring the VMWare forums for user experiences with different storage solutions. It goes without saying that I've had a slight bit of information overload. There are many options out there, each with its own set of unique marketing / technical fortes.

In past lives I've worked with EMC Symmetrix and Netapp Filers and they are great systems without question. I've also worked with "white box" systems running linux +samba/nfs/iscsi. In short I've seen and worked with a decent spread of solutions in some very contrasting environments.

In my current environment being cost conscious is critical - even without taking into account existing global economic issues. Risk / Benefit is always a major factor, and thankfully I've had experiences with a handful of my options in this case and am very comfortable with my final decision.

I don't typically evangelize it, but I am a big proponent Open Source, GNU, and linux. I've used linux in SAP ERP solutions, firewall & routers, file serving(NFS & CIFS/samba), network services(dhcp,dns,ldap,squid,etc), iSCSI, etc.. At all levels of importance, I've used it, trust it, and promote it. The biggest "shortfalls" are: 1) Management doesn't understand anything other than Microsoft or maybe Apple, and 2) Admins are a bit fewer and further between.

Back to SANs and storage. During my investigations with various vendors one conclusion became clear quickly and was not surprising, higher costs. My employer was having to swallow a substantial cost/per useable GB increase to implement any commercial SAN solution. Again from previous experience this was not overly surprising although costs have come down a very long way in the past several years. All these solutions came with excellent feature sets, but these features offered little benefit to my "small" environment that has existed with 99.95% uptime (all inclusive) for years without.

My decided solution will be based on Dell hardware, PERC6 controllers, SAS drives(15k & 10K), and linux (CentOS) providing NFS to ESXi servers and iSCSI to Windows via iSCSI Enterprise Target using dedicated networking equipment segments. This solution will provide significant throughput, high flexibility, great backup options, a very reasonable cost per usable GB, and a low operating cost.

Stumble Upon Toolbar

Wednesday, November 26, 2008

Hooray for Holidays

It's Holiday Season again. As holiday shoppers are heading out and families and friends are getting together, many IT professionals find themselves working. Holidays and long weekends are the haven for IT maintenance activities when the offices and servers are idle. Unless of course you're running e-commerce or healthcare related systems, then it's a time of booming system activity.

While I'll be taking the next few days off, I will be handling several work items on Saturday - patching, moving to VMware server 2.0(tested of course), and installing a shiny new Dell MD1000 array to handle the IO thirsty set of VM's I run.

It has been a busy Fall season between reviewing SAN options (more to come later), managing a CRM project, planning for a lean(er) 2009, and all the other day-to-day items. I'm certainly looking forward to a few extra days off to enjoy family, friends, hang Christmas lights, and catch up on things around the house!

Stumble Upon Toolbar

Tuesday, November 25, 2008

ESXi or Hyper-V Server 2008

Ran across an article over at regarding Microsoft's answer to ESXi - Hyper-V Server 2008. VMWare made ESXi, the "core" of their flagship offering, available some months ago for no cost. ESXi has no service console and is strictly managed via remote management tools and also won't perform some of its bigger brother's fancy tricks, but non-the-less is the same foundation of the more pricier offerings.

Now I haven't delved into MS Server 2008 really at all, or Hyper-V for that matter - but this is a great move by MS to step upto the plate for low overhead hypervisors. The article presents a good overview of Hyper-V Server 2008 and how it compares to ESXi.

Regardless of your pick / preference, this is great for all virtualization users. Competition breeds innovation = win for consumers. For small shops, there is now one more excellent "free" virtualization option available.

Stumble Upon Toolbar

Wednesday, October 8, 2008

Seeking SAN

I've written in the past about iSCSI. I'm now nearing a position, a convergence of 2 situations that has me searching for a solution. My current iSCSI server, a linux based system with 3ware controllers with the iSCSI Enterprise Target software, is nearing its end of warranty with no extension option available. This system provides storage for file services, as well as space for a SQL database used for email archiving and up until recent, space for vm images. The second part of the play is my corporation's move to virtualization. Utimately the plan is to move all enterprise services currently in-house to a multi ESXi server environment.

Keys to this type of environment are shared storage and strong IO capacity. Shared storage, namely SAN storage allows ESXi to perform all of its neat tricks with its vmfs cluster filesystem, not to mention it is the best tuned selection for VM performance. NFS is also an option, but removes access to a few of ESXi's great features, and carries processing overhead. IO capacity is critical - all the space in the world can be quickly lost to complaints of lag and slowness if that space does not have IO capacity to perform(Exchange/SQL).

So I've began the adventure of looking for a solution. A solution to handle VM images, SQL, Exchange, and file serving; a lot of different IO profiles. Thus far I've looked into following iSCSI solutions: Dell/Equallogic PS5000 series, Netapp S / FAS lines, and Compellant. They each offer their different strengths and less desirable aspects. Before any decision is made, there will be demo units put through the ringer, but I'd also like to hear from anyone out there with experience in this arena. For my size organization this solution will be a reasonable sized invenstment and it needs to be right the first time.

Stumble Upon Toolbar

Thursday, October 2, 2008

Data Security: Encryption with TrueCrypt

The onion analogy is used by security people to describe the many different aspects of a complete technology security program. Application security, network security, operating system security, physical security, people(social) security, etc. A business' technology environment is made up of many components that must each be addressed and analyzed.

The core reasons for comprehensive security can be distilled down to a few items: 1) Availability - keeping business systems running, 2) Integrity - not allowing data to be corrupted/poisoned, 3) Privacy - not allowing data to become public. I'm sure others can chime in other reasons as well, but I use these 3 as my core values with any security considerations. I will however throw in one collateral value that comes from these 3 but is paramount in today's world - corporate image / trust. A company that has lost consumer trust / confidence due to a security breach, even though they have minimized / mitigated any of the 3 core values, will experience severe hardship (and likely legal penalty).

Doom and gloom aside - there is one tool out there that offers some piece of mind for some layers of the technology security onion - TrueCrypt. TrueCrypt is a freely available tool for encrypting data, including entire operating system hard drives. It is a multiplatform tool that runs on OSX, Windows(2000,XP, Vista), Linux and can be ran directly off of removeable media(USB stick, etc). A file / disk encrypted with TrueCrypt can be accessed by any other computer that can run TrueCrypt if the password is known. TrueCrypt can even create 'hidden' encrypted drives that are only detectable by someone who is aware of its existance.

Here are a few common business situations that TrueCrypt can help out with. A lost/stolen laptop who's hard drive has been encrypted with TrueCrypt. A lost / stolen usb drive that has been encrypted. Server(s) stolen from the office / co-location. Securely sending sensative information on a CD/DVDs.

In my office all USB drives are required to be encrypted and catalogued. The next image developed for mobile computers will feature full hard drive encryption as well.

Stumble Upon Toolbar

Wednesday, September 24, 2008

VMWare Server 2 Released

There were predictions with VMWorld going on this week that the free VMWare Server 2 would be released - it has. Hop on over to to read more & get a copy. Some notable new features from the 1.0 line: Passthrough SCSI device for guests allowing direct access to physical devices such as tape drives, USB 2.0 support, 8GB RAM support in guests, VMI interface and hot adding of scsi drives.

I've been a long time user of the 1.0.x product and look forward to testing out and using the improvements in 2.0.

Stumble Upon Toolbar

Monday, September 22, 2008

New Poll: Internet Connections

Five people or 5 Thousand, chances are the office has Internet connectivity. In tech jargon these connections are often called pipes - for good reason. A big misconception / misstatement about Internet connections is one is "faster" than the other. The reality is that electrons all move at the same speed and what people are generally referring to is how long it takes to 'download' something. Going back to the pipe analogy, water moves through a 1 inch pipe at the same speed as a 6 inch pipe, but the 6 inch pipe can move more at the same time.

Wikipedia has a breakdown of bandwidth that different connections offer. An office's selection will generally be based on few requirements. 1) Throughput/bandwidth commonly called speed. 2) Redundancy / resiliency, 3) Service availability.

In my office a dual T1 connection is used for the equal reasons of both bandwidth and redundancy. Things would be ok running on one circuit, just slowed slightly. My location has left few options for alternative connections outside of the phone company's services for the case where a cement truck runs over the large telco box down the street. There is not a cable feed within a reasonable distance and satellite is not known for supporting vpn connections well. I did recently find out that Verizon offers staticly addressed wireless cards for a minor setup cost. In testing, these cards can offer T1 comparable speeds (using an antenna) with only minor latency overhead. Any comments / experiences with using these in this regard are greatly appreciated.

There are too many options to post a poll on what type of connections people are using so please just comment in with your thoughts.

Stumble Upon Toolbar

Thursday, September 18, 2008

Interesting Tech: Dropbox

This probably isn't suited (yet) for businesses beyond a handful of people, but has an article on "Dropbox". Dropbox is a tool for synchronizing files between pc's using their intermediary service built upon Amazon's S3 storage. It appears to be a young tool with some limits and a good bit of promise. Check out the Dropbox FAQ for a good list of what it is and what/how it does it.

Might be a good tool to help keep an updated copy of key documents in case of disaster(if you don't already employ a DR plan that does this) - Or for use at home to keep those precious digital memories backed up. It's not a backup service in a technical sense - it only keeps copies of live documents, no history. It does however offer an "UnDelete" feature.

It claims to use encryption, but per the FAQ you can always put your files in a TrueCrypt volume and then sync that up.

Stumble Upon Toolbar

Tuesday, September 16, 2008

Internal Chargebacks

My current place of work is 'small' enough that IT does not 'charge' the business units for services provided.

From a business perspective, I favor charge-backs for a few reasons. First, I think it is important that business units have some concept of how much IT resources cost, and probably one of the most direct ways to make this happen is seeing it hit a department's budget. Secondly by 'seeing' how much IT costs, requests for changes / new services are less blindly requested with little concern for costs. Thirdly charge-backs force IT to be aware of costs down to a relatively fine grained level. This last fact should be taken advantage of irregardless of whether or not charge-backs are used or not.

Before I continue on with that last statement, I do acknowledge that charge-backs have draw-backs. Charge-backs introduce a good bit of accounting overhead. IT must very accurately measure and report on services provided. This can be a very complicated task depending on the services provided, the supporting infrastructure, and business organization. Then of course there is all the actual accounting work, internal invoice and bill processing. Execution of this concept definitely adds cost, hopefully with some return/savings when done properly

Again regardless of whether or not IT does charge-backs or not I think knowing, at least within some small margin of error, what services cost is incredibly valuable. Like any other metric, availability for instance, knowledge is power. When looking for new solutions, having an industry standard or competitive per user cost for a service can be a decision make or break factor. When evaluating a replacement / service upgrade, knowing what the 'old' service cost and then being able to compare that to the new service can help make the proper select to keep that per user cost in a target zone.

I started off mentioning that charge-backs are not done in my current workplace. I have however, over the past several months been working through constructing a spreadsheet that exposes the true costs of IT services. It has been a challenge to properly assign metrics to services - ie what is the true capacity of an MS Exchange server, and what are all the associated bits behind an email account(much more than just a CAL). I've finally come to a place with it that I'm fairly happy with. Its made apparent many facts, some good - some which need improvement, that will be very valuable in my future decision making processes.

Stumble Upon Toolbar

Monday, September 15, 2008

New Poll: CRM

I blogged earlier about CRM and wanted to put a poll out there to hear what others are up to regarding CRM. My workplace is still on track to begin a SageCRM implementation in the next few weeks. It is definitely one of those tools the users aren't really sure of what it does or even is. I've been working hard to spread the word it is something that can help significantly improve the level of customer service provided by bringing together the details of various interactions different employees have with customers in one central location. Another major point of any rollout is adoption. A big part of the final vendor selection was usability and I feel confident SageCRM will be viewed as intuitive to use - it is natively web based and integrates incredibly well with MS Outlook. Some department managers have also been tasked to ensure their areas are 'motivated' in a measurable way to utilize the tool.

A few small (15-20 people) sister companies I know live and die by their CRM. What are your CRM experiences - good and bad?

Stumble Upon Toolbar

Thursday, September 11, 2008

System Imaging

Anyone who has to manage PC's is probably aware of the time consuming task of having to 'setup' a computer from scratch. I'm talking about formatting disk drives, installing an operating system, applying security updates, installing applications, and configuration. This set of tasks can easily consume 4-6-8 hours of time and from a business standpoint is time poorly spent.

There are solutions out there. One approach is to use thin clients or stripped down pc's that only serve the purpose of connecting to a server for all useful applications. This works well for local users needing a very standard set of applications, not terribly well suited for mobile/disconnected users(However some really cool [aka expensive] solutions do exist for this). I personally utilize low spec'd PC's running a next-to-nothing OS install at my workplace - they work great, are cheap to obtain, and last several years. The other type of option is to use system images. System imaging is the concept of creating a "golden" setup that can then be wholesale copied to different computers. While the process of creating the 'golden' image can be time consuming, the benefits are significant. A PC can be taken out of the box, have the image copied to its drive, up and running in less than 30 minutes with a standard set of sofware and configurations. Apply this process to several PC simultaneously and the time savings are huge.

From a support standpoint the use of images creates a very consistent environment to support. A PC can also very quickly be 're-imaged' and returned to the user if is a spare is unavailable. If you're talking about a physical system, creating an image right before testing a new 'something' is also a great backup method.

I initiated the use of images in my current workplace and I can't begin to count the amount of setup, and support time it has saved. I've used Symantec Ghost for imaging in my MS-centric environment over the years and it works very well. I'd love to hear comments about other imaging solutions people have used, commercial or open source.

Stumble Upon Toolbar

Tuesday, September 9, 2008

Social Networking@ Work

I began blogging for mostly professional reasons (check out my first entry). Here and there I run across articles on companies that internally (and externally) exploit blogs, and forums for business use. For larger companies I can kind of see the value of this use but still wonder about : A) Don't most of these companies have Intranets for corporate information / happenings, B) How to keep employees from 'blogging' the day away, C) policing of content.

If you're using blogs / forums internally, what was the business motivation to do so, how was is sold to management, and how is it operated(content policies, etc)?

Stumble Upon Toolbar

Monday, September 8, 2008

New Poll: Virtualization

Poll Day Topic: Do You Use Virtualization

Share your experiences, opinions and environments about virtualization. What solution do you use and why. Are you avoiding virtualization? What has worked for you and what has not.

I use it - at work, and at home. In the office it has saved tens of thousands in equipment alone, not to mention some of the 'intangibles': cooling, power, physical space, recoverability, testing, etc. I'm a VMware fan for several reasons, but I'm not looking to start a holy war over the best solution - here anyway. Some key points I've taken away from my virtualization adventures: 1) RAM, RAM, RAM - have lots of RAM in the physical server. RAM is your best friend and it is inexpensive these days. 2) VM's generate lots of small random IO - have a properly designed disk subsystem (aka avoid Raid5). More spidles are your friend. 3) Understand the workload of your VM's. Checkout one of my older posts about virtualization in general and why it's valuable in nearly any environment.

Stumble Upon Toolbar

Friday, September 5, 2008

Linkage: Storage Tips

I've only poured over a few of the many articles, but there's lots of good info about SANs/SCSI/SATA.
-To SATA or not to SATA
-Price vs performance in selecting a RAID configuration

Stumble Upon Toolbar

Meeting Room Mayhem

I work in an office that has a handful of meeting rooms, some large enough for seated groups of 40, others with long boardroom style tables for 20. Each of these rooms has a projector, either ceiling mounted, or simply set on a small rolling table/cart. Each of these rooms also has either a PC or a laptop that is networked an usable by anyone with a login. It never seems to fail when I see a call from an extension in one of these rooms, that the previous occupant proceeded to disconnect any and all wires and completely rearrange the seating. In the past I've tried neatly arranging / wrapping any wiring to make appearances as neat (and safe) as possible with the hopes the overall 'neatness' of the room would be better kept.

I'm searching for any (realistic) concepts of how to (even subconsciously) motivate people into returning a room to a 'standard' when they are finished using the room.


Stumble Upon Toolbar

Thursday, September 4, 2008


Thought I'd share a few of the blogs that I frequent. Chances are if you're reading me, you'll like these as well. is done by Jason Benway (An old college classmate of mine).
The Standalone-Sysadmin by Matt is always full of good material
TheGeekDoctor by John Halamka - insightful writings, chances are you know his name already.

I'll pop these into a sidebar gadget for my and your convenience.


Stumble Upon Toolbar

Wednesday, September 3, 2008


Guess what, it's another 3 letter acronym: CRM. Customer Relationship Managment. It's nothing new but has been a consuming item in my career for the past 12 or so months. At its core CRM is a centralized repository for information - typically contacts, freeform notes, that allows many people to see, review, and update this information. CRM solutions exist that are tailered for as few as 1 persons and up to thousands of users.

As a systems / network administrator, I appreciate and acknowledge the value of 'centralized'. Having anything in a highly organized and shareable, consistent format makes those 'anythings' much more valuable and efficient.

In organizations where a 'team' of different people interact with customers, CRM offers many positives. When contacting / being contacted by a customer CRM can be referenced to see what the last communication was, when it was, and who it was with. This type of background is invaluable to providing efficient customer service be it sales support, technical support, or billing/account support.

From a HR standpoint, CRM helps solve turnover issues. Instead of users keeping unstructred notes, who knows where, CRM keeps data in one place - consistently structured. This also applies in situations where customers are handed off between sales account managers, or even internally between departments/divisions.

I was tasked with finding a solution to many of the above 'challenges' and putting together a project through implementation. I started this off with a few small meetings gathering a list of desired features and then prioritizing those into 1: Must Have Now, 2: Must Have Later, 3: Would be Nice. This was invaluable when speaking with vendors on what their CRM product was capable of and developing a staged implementation. Many vendors quickly had to decline due to not being able to fullfil #1 items. At one point I was left with only a single vendor, not a good negotiating standpoint. I was able to find another offering that eventally ended up winning the project and that'll be kicking off just several weeks from now. More on that as it unfolds.

I'd like to stress a things that stick out during my reflections on the project thus far. The first is having a list / knowing what you want to get out of a CRM solution. There are too numerous solutions out there to list out but knowing weather or not ACT! will work for your needs or if a solution as elaborate as or can really be simplified by knowing your end goals. From technical standpoint there are lots of architechtual differences that play into the decision making process. Some solutions are completely web based(connect from anywhere), for some web connectivity is an ($)add-on, some only offer local network access.

Is your workplace using CRM, what's your impression?

Stumble Upon Toolbar

Tuesday, September 2, 2008

iSCSI - SAN for Small (and big players)

For years all the buzz in storage has centered around 3 letters SAN. The storage area network, fast, centralized, flexible, resilient. Placing storage directly in each server is costly, inefficient, risky, and slow. And for some number of years vendors and partners and such 'hid' this term from smaller shops due to costs involved with a SAN architecture that was designed to meet all the marketing accolades. Fibre channel equipment was really the only option.

A new technology began to surface 3-4 years ago - i(nternet)SCSI. I won't rehash all the details that Wikipedia can provide except for the main point of iSCSI - commodity interconnects. iSCSI allows for the use of common networking equipment: switches, NICs, cables. iSCSI does not require the use of $1K+ HBA's, fragile cabling and specialized directors/switches, etc. The reality of iSCSI's pratical use is here due to the commoditization of gigabit ethernet and soon to be 10gig ethernet.

iSCSI also brought options to storage subsystems. No longer are costly FC drives the only option, but also SCSI, SAS, and even SATA. A multi terabyte array can be obtained or even constructed for costs well within reach of nearly any business. iSCSI can also (and often does) exist a strictly software world - no HBA'$ required. Some argue the software overhead is detrimental to overall system performance. I'd argue today's systems have sufficient processing capacity to run software iSCSI in situations where iSCSI is a good fit.

With this commoditization of the SAN, vendors have been quick to offer up 'solutions' for smaller shops. I have a caution to throw out that requires a little lesson on storage. First off, the most important item in selecting the proper storage is understanding the application(s) that will access it. I can't elaborate on this enough - don't even start to think about how much space you'll need until the application is understood. In fact refuse to talk about capacity with any vendor until this is ironed out. Try throwing a few virtual machines at solution that only does (or has been configured for) RAID 5, along with a file server, or mail server - it will not make you feel good about the money spent. At one point I had a prominent vendor's rep suggest a 4 spindle SATA system to me to replace a (6) disk file server and an Exchange server. All that person was concerned about was how much space and quoting his cheapest offering to improve the chance of a sale. I don't carry any certification badges, but I am well informed and experienced, and I know when I'm getting a snow job.

I ended up with a 16 SATA drive over two 3ware controllers system running linux and the open source iSCSI Enterprise Target software to provide my iSCSI storage. It's been rock solid. Its 8 drive RAID10 ran up against performance issues after a handful of VM's where thrown on it, but in all fairness virtualization was not in the picture when the system was spec'd. The next iteration will be 15K SAS based for VM storage.

So why use a SAN/iSCSI? A few main concepts are modularization and maximizing the storage investment. By detaching storage from a specific system several benefits are realized: 1) Smaller form factor systems can be purchased since they don't need to be filled with drives. 2) In the event of a system failure the storage can be accessed by a different system. 3) It fits into a virutalization environment for lots of reasons. 4) Designed right it will be faster than local storage. 5) Because multiple systems utilize space on the iSCSI server, the disk investment-to-utilization ratio is typically better.

Downsides include: 1) Slightly complicates overall architecture, 2) Small bit of iSCSI management expertise, 3) Many eggs in one basket situation(get a good warranty). With the right system and financial resources the iSCSI server can be setup to mirror with another system for improved reliability.

Know your applications, read about technology and reviews, search vendor support forums/blogs for real world feedback on your purchasing considerations.

Storage Advisors Have LOTS of great blogs on RAID levels/storage and their INs & OUTs


Stumble Upon Toolbar

Monday, September 1, 2008

Office Phone Systems

The motivation for this article came from a situation sprung on me a few months ago that I'll elaborate on a bit more later. Chances are if there are more than a handful of phones in your office, you have an office phone system of one type or another. I'm not going to jump into the various types of technologies behind phone systems, quite intentionally so. One quickly discovers that phone systems, much akin to networking, have their own mystique and black magic, and can be quickly overwhelming with acronyms and terminology. Like networking and other 'complicated' subject matter, the experts are often best left with handling the fine grain details of phone systems. This isn't to say a moderately savvy person can not handle the basic / common phone system operations.

Moving closer to my surprise 'situation'. Phone systems are very important parts of business, there isn't any questioning this fact. I can't help but feel this business value is exploited to a certain extent by phone system manufactures and those who install and support them(partners). Anyone who has purchased a system that supports more than a handful of extensions quickly realizes the investment is substantial, not to mention ongoing support. I'm talking the kind of investment that stretches not 5 years but 10 or 15, maybe more.

The meat of the situation. The system I'm responsible for was made by a prominent manufacturer whose name starts with the letter "A". It was a good size investment when installed just 5 year ago and is currently supporting around 250 extensions with room to grow. The surprise I received was that part of the system will no longer be supported post 2008 by the manufacturer. The system is very modular so maybe this isn't that big of a deal, right? Well it seems that the 'suggested' upgrade path is more akin to replacing the entire engine of a car versus just 4 new tires and proportionally so in cost as well. Three things came to mind in this scenario: 1) So I spend this money to have a supportable system without any other added benefit, 2) When will the next 'Upgrade' have to take place, and 3) aren't phone systems supposed to last a long time?.

My next move was to investigate 3rd party support. In searching it seemed to be a very common situation where customers were presented with obsolescence / costly upgrades from manufacturers and end up using 3rd party support. Great I'm not alone. The endgame was that I've found what appears to be a very reputable provider (according to the several references I've spoken with) who can support my system (better than the mfg) long term, and for less cost.

I was not involved in the purchase of the current system thus did not have a chance to ask any questions. This situation does highlight a few pointers for the next time a new system purchase comes around. 1) Consider 3rd party support from the start or immediately after the initial mfg warranty is over. 2) Get feedback from customers with similar equipment and a long history with the manufacturer.

Stumble Upon Toolbar

Friday, August 29, 2008


Like insurance, computer backups are the type of item that feel like they 'cost' more than they ever return. This feeling is quickly reversed the instant a backup is put to use, but depending on the shop, backups typically happen a lot more often than restores. There isn't any question backups are a necessity, but the real question are: How and Why.

The "Why" question really refers to selecting the best "How" for a shops situation: Why is this my solution. It involves details like: how much data must be backed up, how often must it be backed up, how fast must it be backed up, how fast must it be available for restoration, where must the backup be stored, etc.

I've ran across a few different "How's" in my career. Some of these involved FC attached libraries, autoloaders, single drives. These are all 'local' backups, where the data was copied to some type of local device and managed locally. There is something to be said for today's modern tape technology - it is soo much better than from 5-7 year ago. Speeds are great, capacities are way up, reliability is significantly better. But - tape is labor intensive, having to be handled, moved offsite, back onsite, sensitive to temperature, etc. Good quality tape drives(LTO and up), either stand alone or inside an autoloader/library are costly for smaller businesses. And when a tape drive decides to start acting up, it is your own worst enemy - writing bad data, eating tapes, failing backups.

In recent years an additional local backup has come into popular use - disk. The plummeting cost of disk drives and sky rocketing capacities have made disk drives a very attractive backup option. There are 3 approaches used with disk backup: 1) Disk replaces tapes. Here backups are written to removable drives and then moved offsite / rotated just like a tape would be. 2) Backups are written to disk first, then tape - known as D-D-T(Disk to Disk to Tape). Here the disk acts as an intermediary(with many benefits). 3) Mirrored remote SAN. This is big business stuff mostly using expensive equipment and network connections to actively mirror live data. It's also more of a disaster recovery / failover solution as it does not preserve historical versions.

Going back to #2 above and the "Many Benefits" part. By putting the disk between the data source and the tape drive several good things happen. First off many backup streams can be written to a disk simultaneously, minimizing jobs having to wait for a tape drive to be free. Secondly because these disks are usually in the backup server or 'near' it, when the data is written to tape from them it can be done much more quickly / smoothly, freeing up even more tape drive time. Thirdly, there are now 2 copies of the backup, always a plus. Lastly, if the data is still on the intermediate disk when a restore is requested, the data can be pulled from the disk instantly versus searching for, loading, and seeking through a tape.

There are too numerous to mention local backup software options. The keys to the proper selection mainly involve: The kinds of data being backup up(database, email, normal files, etc), the type of systems being backed up(Windows, Linux, UNIX, etc), and cost. All reputable softwares should support modern media types, be it tape drives/loaders/libraries, VTLs, or disk drives.

Another option exists that can be attractive in certain situations - online backup. Online backups start with an account on a vendor website, an Internet connection, and a small piece of software on the source systems to be backed up. Often times there are options for the backups to run continuously - capturing all file changes in real-time and transferring them securely online for backup, or scheduled backups that only run at certain times. There are a few immediate cautions with these solutions. 1) Is the Internet connection fast enough to transfer the data efficiently? 2) How is the service charged: per MB transferred/per MB stored/per MB on the source system/etc? 3) What options exist for restoring data, both small files or entire systems? An online backup can be a good match given the right billing structure and a desire for low IT resources.

Stumble Upon Toolbar

Thursday, August 28, 2008

Active Directory Restore Fun

This adventure all began with testing out a remote access application. In my earlier experience testing a previous version of this application, it pulled a local copy of my Active Directory users and groups during initial setup which then had to be pruned down to those who actually needed remote access. After setting up the most recent version, I promptly began pruning the users and groups shown in the application to the appropriate list - then the phone rang.... Needless to say the current version of the application was directly accessing AD, not just for authentication, but to maintain its user list --- that I had been actively pruning.

Needless to say this was not the best situation. No fear - there are backups for this sort of situation. But beyond backups, the restoration of AD is not a two click and it's done operation. This isn't a story about "Oh no the backups junk" or backup schedule philosophies, but I will comment of the merits of using Disk-Disk-Tape backups quickly - backup & restores are quick. Back to AD. My AD landscape is fairly simple: single physical site, 3 DC's, relatively small database.

I don't carry any MCxx certifications and thankfully the opportunity to perform this type of operation is not a common thing. Thankfully between MS Support Knowledgebase articles and documentation in my backup software the AD restore operation was smooth.

I had restored a DC in a VMware test environement before, but it was a single DC configuration which changes things a bit. Aside from not realizing up front how the remote access application integrated with AD the following lessons can be taken away:
1) If at all possible have a test environment similar to production - and try these things out.
2) Have knowledge of proceedures or how to find it - quickly
3) Take a deep breath when these things happen - think quickly, act thoughtfully
4) The addage of one-application to one-server is priceless in restore situations
5) Having multiple DC's even in a small environment helps keep things moving along.

[Update on Backup / Restore resources]
AD Restore on 2K - Works for 2k3 (MS Support KB)
Good notes on using NTBackup on DCs for Backup / Restore (MS Support KB)
I use Symantec Backup Exec in my environment, but NTBackup is good for just a few systems.

Thanks for the comment Matt & good luck jumping into MS infrastructure (I'm a *nix guy by trade too).

Stumble Upon Toolbar

Wednesday, August 27, 2008

Virtualization in Small Companies

The news and media have been chock full the term Virtualization for over the past year and very heavily so since 2008. There are various offerings from several prominent vendors and a new option seems to pop up every few weeks. From a small company point of view these offerings appear catered to larger business. After all many companies out there don't have hundreds, let alone tens of servers - and what's a 'SAN' anyway.

I'm not sale rep / associated with a sales rep from any of the virtualization solution companies. I will however say there are very compelling reasons to use the technology, even in the smallest of environments. Akin to this - many of the 'beginner' virtualization solutions are free to no cost and as such can not be used as an excuse to avoid the technology. In fact, the use of virtualization in companies with only a few systems can be very compelling for some of the following reasons.

1) Hardware Independance. When you only have a few servers and suffer an equipment failure, recovering a hardware dependant system is a nightmare and a huge prioirity. VM's are run against a virtual BIOS/hardware that is more or less identical between host virtual server systems. This means a virtualized system can be restored to another host with little to zero hardware / driver issues and be off and running fast - priceless!

2) Hardware Cost Savings. Starting with 2 physcial systems consolidated to one, hardware savings are acheived. For the sake of having a backup host this isn't the best idea but the savings concept is key. Virtualizing only a handful of systems eases the selection of equipment. A modest system by today's standards can run a handful of virtual servers without worrying about having a high performance SAN or similar storage / network environment.

3) Return on equipment investment. Every business purchase is an investment, and getting the most value from each dollar spent is a big factor in buying decisions. Consider 4 servers, each costing 2 to 3 thousand dollars each. This situation presents a total investment of 8 to 12 thousand dollars and lots of computational power generally sitting 85% idle, with 4 sources of heat, space and power consumption. Virtualize those 4 systems to one 4 to 6 thousand dollar server saving thousands up front with one source of heat, power use, and noise. Or conservatively to two 4 thousand dollar servers still spending the same or less on equipment but halving the noise, power use, and heat dissipation and still reaping all the other benefites mention here.

4) Testing. Using virtualization opens the doors to smaller shops where it isn't feasable to have equipment dedicated for testing. A modest desktop with enough memory can run a handful of VM's for configuration testing purposes.

5) All the usual suspects: Less equipment space, less electricity, less heat produced, less noise, longer battery backup runtime, etc.

All this being said there are of course drawbacks. Virtualization brings along with it new concepts in system management that must be understood. Administrators must generally have a strong understanding of system resources to maintain an effective virtual environment: system memory, disk IO, networking. For smaller shops, putting many VM's on one host creates a "Many eggs in one basket" situation. Backups - there are additional backup options avaialbe to VM's that need to be understood. Certain applications do run better on dedicated physical systems versus the virtualization environment a small shop could support.

All in all virtualization is great. Even utilizing the free solutions available, great efficiences can be acheived compared to 'old school' one system to one physcial computer. You don't have to be a 'big' shop or invest thousands to reap the many benefits of virtualization!

Stumble Upon Toolbar

Tuesday, August 26, 2008

Hello World

I've come to the realization over the past several months I've enjoyed reading a handful of other tech related blogs some more technical, some more business oriented and decided I would try my hand at one. I've found good genuine technical advice, invaluable real world experiences, and just plain enjoyable reading out there and feel I should relay some back.

It's also becoming a staple for most professional careers to document one's experiences. In this world full of (often too) much public information, the importance of personal experience and sound information is invaluable. I'm certainly not the definitive source for some if any topics, but I have to be for both my family and the real world issues I deal with daily.

A short list of the things I deal with each day(Search engine food mostly): Windows Servers, Active Directory, Wireless Networking, SQL Server, MS Exchange, Citrix, Office Applications, LAN, WAN, VPN, Linux, Security, Backups, iSCSI, Phone systems, Remote Access, Capacity Planning, Budgeting, Business Policy, Office Politics, Virtualization, A/V Systems, Firewalls, SAP, etc. All the fun things in today's modern businesses.

Stumble Upon Toolbar