All of us in “the IT biz” have war stories about the things that have gone horribly wrong at one time or another and we all have the battle scars to show off to all and sundry as proof of the battles we have fought (and hopefully won).  And all of us have learned (or will learn, it’s inevitable) some hard truths.  One hard truth is that there can come a time when things are just too broken to fix – servers, networks, domains, whatever – and that the best thing you can do is to nuke things from high orbit and start fresh.  The trick is to learn how to tell when you are too far in to fix things and then make the decision to call in the strike force.  This is a lousy place to be when it’s your own kit that is a mess, it’s a much harder place to be when it is someone else’s and you have been called in to fix things.

I have been working with a new customer over the past few weeks that has ended up in precisely this position and we just had to make the decision to nuke the current Windows domain and start fresh.  The customer’s sole IT admin had been tasked with migrating them from SBS2003 to SBS2011 and he did all the “right things”:  he read up on the migration procedure, he purchased a new well spec’d server (or so he thought) and he attempted to follow the Microsoft migration path step-by-step.  So, how could it all end up so wrong???  The answer is it was a bunch of things that when added together created a “perfect storm” of problems.

Some of the major things were as follows:

Problem 1 was the fact that he was trying to migrate out of a very “customised” SBS2003 installation and he didn’t have enough exposure to SBS to know that the installation was very “non-standard”.  SBS2003 had been installed by a third-party that broke a lot of SBS rules so things were already in bad state, from an SBS migration point of view.  SBS is a wonderful package but you DO have to resist the urge to break out of the SBS way of doing things or all sorts of issues can crop up to bite you.  This is specially true when it comes time to migrate to a newer version.

Problem 2 was caused by a DFS installation that had spanned three servers in two separate physical locations (across a VPN) that was corrupted by the improper removal of the original SBS2003 server as well as the earlier improper removal of a member server on the domain.  DFS was absolutely broken yet all of the AD objects for it were still intact.  The AD on the new SBS2011 server was flooding errors about the broken DFS in all the server logs and we found that we could NOT remove references and links to long-dead servers because of the DFS references.

Problem 3 was caused by the new server hardware itself.  The admin had been forced into buying a whitebox server that was supposed to offer hardware RAID5 as well as a few other features.  The box actually offered “mock RAID” via Intel’s hideous ICH10 on-board chipset.  RAID performance was awful off the raided SATA disks and SBS was initially loaded while the box was missing half its required RAM and a second processor (adding missing RAM and second processor made little difference in the end to performance).

There were a lot of other problems but the above lists the major issues.  By the time we got called in there was not a lot that we could do.  We tried all sorts of things to try and get around the issues including nuking Symantec a/v on the server (for whatever reason it took the SBS box into the basement) and all sorts of band aid fixes to try and make the server perform.  Nothing worked and we really weren’t sure if we were fighting just Windows domain and configuration issues or was there also a fundamental hardware problem at play.  In the end it was decided that the server would be replaced with a “properly” configured Dell T320 (kick ass hardware RAID 10, SAS drives, the whole enchilada) and we would migrate SBS2011 and clean things up during the migration.

What is it they say about the “best laid plans of mice and men”???? 

SBS migration flat out failed to the new Dell; Exchange would not install, SharePoint would not install, IIS would not install and there were many, many other errors.  That’s when the lights came on and we decided to just replace the Windows domain with a net-new domain using a clean SBS2011 install on the new server.  And that’s where we come back to the whole original point of this post which is to learn to recognize when things are just too broken to continue.  If your gut is telling you that things are a total mess then they probably are.  My gut was screaming at me after my first hour on site with the customer but they had to learn, the hard way, that things were just too far gone to cleanly recover.  Initially, there was no way they were going to look at replacing the whitebox server and there was no way they would look at chucking their current Windows domain.  And that’s the trap that most of us fall into – we want to believe that anything we do with our technology is “reversible” and “fixable” and, frankly, it’s not.  A lot of things we do truly are one-way streets so we had better know what we are doing before we hit the buttons.

Was I as culpable in this case as my customer?  Oh yes, without any shadow of doubt.  I ignored what my gut was telling me (and the evidence too, for that matter) and tried to work with what was obviously a horrendously broken domain and server rather than lay the arguments out, up front, to nuke and pave over and start fresh.  We wasted a lot of time and effort that should have been applied to migrating to the new domain.

The takeaway is simple:  stick with what the evidence and your gut is telling you when you have to make the go/no go decision and strive to keep other factors (the emotional ones) out.  If you are faced with the hard IT decision it is best to lay out possible alternatives and then evaluate them as coldly as possible.  There is going to be a lot of work involved no matter what so you want to ensure the work effort will result in the best possible outcome EVEN if it means you have to completely change track. 

Always take the long view when analysing the possible results of your actions, resist the urge to “bandaid” things (and this is true of your initial planning as well as your reaction to the go/no go question).   Wherever possible, have your big IT plans validated by a dispassionate third-party resource just to be sure that you haven’t missed the obvious and, maybe, no-so-obvious flaws in the plan.  This is specially true when you are dealing with things outside of your experience (like a migration to a new O/S)!  And, finally, realize that you truly do only get what you pay for.  There is an old saying in IT that is oh so true, “you can have good, fast and cheap – pick any two”.  If your budget is such that you really DO have to cut corners then you need to be extremely aware of what the impact is going to be on your end results.  You can actually save money overall if you spend a bit more up front to ensure you aren’t  going to shoot yourself in the foot and then spend a lot of time and money to clean up after the fact.

As I mentioned in an earlier post, we have been in the process of migrating a customer to Office365 in preparation for their migration from SBS2003 to SBS2011 Essentials.  We decided to move them to 365 well in advance of the server migration itself in order to lessen the “shock” of the migration to all concerned.

For the most part the migration from on-premise Exchange to Office365 hosted Exchange has gone quite well. Our customer opted for the “Small Business Package” for Office365 and we got them set up inside of a day.  One thing we did do was elect to NOT host their DNS via Office365.  This is something that might be confusing to many people when they first look at 365 as it seems like you MUST move DNS to 365 when you first read the set up documentation.  If you read carefully you will find that this is not actually true and you can definitely use your own DNS management.  You do need to create all of the records that Microsoft suggests or many functions of 365 will fail so “read and heed” the setup docs.  In our case we could create all of the required records save one through the DNS management interface at HostPapa (where customer’s DNS is managed"). We had to request a manual record change be made by HostPapa support to complete all the necessary DNS record changes. One thing to note is that we set up the MX record to point at 365 as a much lower priority than the on-premise Exchange as on-premise MUST remain the primary receiver until after the migration is completed.

Once DNS was set up and the requisite 365 test was made against the customer’s domain we were able to start the actual migration process.  365 offers two distinct process for Exchange migration – Cutover and Staged.  A Cutover migration assumes you are migrating everything within the on-premise Exchange to 365 while a Staged migration assumes you are migrating “chunks” of the current on-premise or you expect to be migrating over a long period of time or you expect to run in a hybrid mode (both on-premise and 365) for an extended period.  There is more work involved with a staged migration and it requires the use of a “connector” tool between your AD and 365. A Cutover migration, on the other hand, relies solely on an RPC over HTTPS connection (think OWA) to “slurp” the data into 365.  Our customer had a very small number of mailboxes to migrate so we elected to use the Cutover method.

All users were instructed to clean up their mailboxes so there was only one really large mailbox (just over 6GB) to migrate, all the rest were well under a gig.  We set up a “migration batch” which defined all of the parameters required for the migration and kicked it off.  Within a couple of minutes we could see that 365 was already importing mailbox data.  The migration itself was started in the late afternoon and we allowed it to run until completed.  The following morning it was obvious the migration had completed and it was also obvious that 365 had performed a synchronizing update with the on-premise Exchange sometime after the batch had completed.  365 will perform a synchronizing update once every 24 hours until the migration process is completed and you have shutdown the on-premise Exchange.  This is because until you complete the migration and point the MX at 365 as the primary receiver, inbound email will still flow directly to the on-premise Exchange.  If there is no sync between the two then new email would not make it into 365 while the MX still points at the on-premise Exchange.

In our case as all required mailboxes had imported properly into 365 we were able to make an immediate MX switch and we made 365 the primary receiver for email.  As soon as the DNS record propagated we sent email to a couple of the users from our own Exchange and the emails arrived in the users 365 Inbox within seconds of being sent (we watched for this using 365 OWA).  At that point we connected all users to 365 by creating a new Outlook profile in their Outlook 2010.  All users with the exception of the user with the 6GB mailbox were fully updated within about 30 minutes from the point where there new profile for 365 was created within Outlook.  The user with the large mailbox took considerably longer to fully populate Outlook as would be expected.

Overall, it took about 2 days from start to finish to go form on-premise Exchange to Office365 Exchange.  It took the users a little bit to get used to being prompted for credentials when they logged into Outlook and they also had to get used to the fact that there might be a bit of a delay experienced when opening new items with large(ish) attachments.  Otherwise, it all went much more smoothly than any of us – itgroove or the users – expected.  We have seen an issue with the user with the large mailbox having some weird “connection” issues on start up of Outlook that, hopefully, will be solved shortly (looks like a corrupt profile problem).  That issue aside, it has gone quite well and is certainly FAR easier than any on-premise migration I’ve ever performed between old and new versions of Exchange.  We have a larger migration looming at another customer in a month or so and it will be interesting to see how it compares to this one as they have about 5 times as many mailboxes to migrate.  I’ll keep you posted on our progress.

I’ve had very mixed results with Blackberry Enterprise Server (BES) installs over the years.  At its best, BES is a pain in the ass to install and manage; at its worst it is a cast-iron nightmare.  We (itgroove) abandoned Blackberry and BES a few years ago and moved to a mixture of iPhones, Androids and WinPhone 7 (Keith).  Frankly, it was a no brainer and we’ve never looked back.  However, we have customers that cling to Blackberry and we’ve had to manage (even install) our share of BES servers.

A big customer of ours had their BES server die after a server reboot.  To be fair, the BES server didn’t die, just the MAPI interface to Exchange (BES installed on the Exchange server).  No amount of massaging would resurrect MAPI and Sean wasted more than a few hours trying.  He passed the problem over to me after I suggested that standing up a new server and moving BES to the new server might be a solution.

I found a few posts on the Web that described various procedures but the one I followed is here:

http://docs.blackberry.com/en/admin/deliverables/18764/Install_BES_Express_cutover_existing_BBCD_1200285_11.jsp

I followed the process to install on a new server but use the SRP from the existing server along with the existing configuration database.  The process worked after I allowed the installation process to upgrade the configuration database and then apply the SRP identifiers to the new server which included created a “pool” DNS record that pointed to the new server.  Once the installation finished I logged into BAS on the new server and migrated all Blackberry users to the new server and then forced policies and service books to be resent to all Blackberries (again, all documented in the link above).  The customer had slightly more than 30 Blackberries in BES, it took about 30 minutes for all the Blackberries to complete the update process before email started to flow again. I then disabled ALL BES services on the original server.

Aside from stupid problems that I encountered with the new server build (VM) that really had nothing to do with BES, the actual BES user migration process worked quite well; surprisingly well considering some of the gong shows we’ve had with BES in the past (like a 30 hour installation on an SBS2008 server, but that’s another story). 

I’m still no fan of BES and we don’t recommend Blackberry and BES to any of our customers as we feel it’s just another layer of un-needed complexity (sorry RIM).  Yes, Blackberry still has better overall security than iPhone, Android, et al but for the majority of our customers the added security simply isn’t worth all the extra overhead and hassle.  But if you ARE a BES user it’s nice to know that you can recover from certain BES failures with relatively little fanfare.

Louis asked a good question the other day as to why you would want to use a Swing Migration over the SBS2008/SBS2011 migration process provided by Microsoft.

Back in the days of SBS2003 this question would not have been asked as there was no Microsoft provided tools for migration into SBS2003 or to a replacement server. The Microsoft process would lead to a net-new server name/identity in a net-new Windows domain which was hardly ideal. The Swing process eliminated all of that as it left you with the same server name for your new SBS server as your original server and you kept your original domain – pretty much a no brainer as far as I was concerned.

Nowadays the choice isn’t quite so clear. So, here’s a short list of the pro’s/con’s for each method:

Microsoft Migration Process

Pro’s

• Built-in process to SBS2008/SBS2011 installation
• Both old and new server (SBS) can co-exist on the same network (with caveat’s)
• Data transfer can be performed via network (server to server)
• Migration Wizards built-in to SBS console

Con’s

• Limited time frame for migration (21 days) from date that new server is loaded
• New server build cannot be loaded/configured off-site, everything is done on site
• Process cannot be easily rolled back (if at all) if something goes wrong
• One way process only, no provisions for moving out of SBS

SwingIT Migration Process

Pro’s

• Leverages built-in SBS process as well as other Microsoft processes/tools
• New server can be built and configured off site minimizing impact on current environment
• Supports “swing in” from both SBS and non-SBS environments
• Supports “swing out” from SBS to non-SBS environment
• Allows for easy rollback if process goes wrong, current server not touched, modified or retired until new server is 100% operational

Con’s

• Requires extra system (can be a VM) for the Swing server (TEMPDC)
• Requires data transfer medium (USB disk, tape, etc) to transfer data from old server to new; old and new server cannot co-exist on same network
• Time required for the data copy can cause a real time crunch during the actual migration (typically a weekend)

We just wrapped up another SBS2003 to SBS2011 migration and we once again made use of the SwingIT migration kit.  This migration was a bit more complicated as the source SBS2003 system was a complete mess as there were many, many fingers in the “administrator” pie since the system was initialized.

Overall, the migration went well.  For those of you that have messed about with SwingIt migrations you will know that the most time-consuming portion of the Swing is the offline data migration.  The data migration is done offline as the two systems – the source system and the new target system – can never be on the same LAN at the same time as the new target system essentially “borgs” the identity of the source system.  The SwingIt kit has all manner of suggestions for methods to copy the data, we usually end up using USB hard drives and a copy tool such as ViceVersa Pro from TGRMN Software because that gives us the best mix of control and speed.  However, and this is the point that you may want to note, we were migrating from an SBS2003 virtual machine hosted on ESX 3.5 going to an SBS2011 virtual machine hosted on ESXi 5.0.  USB was not really going to be an option due to no real support for it under ESX 3.5 and while supported under ESXi 5.0 it is slow.

Our customer supplied a network-enabled WD Live portable drive that we used for the first pass at the data copy.  While it worked and the speed was okay we discovered after the copy on to the target system that the WD’s internal OS managed to strip off the majority of the NTFS permissions during the initial copy on to the WD disk.  A bit of playing about proved out to us that this happened regardless of the tool used to do the copy (we used ViceVersa Pro for the actual copy but then tested with various Windows copy utilities).  Luckily, the first pass copy was of a large chunk of data that had simple permissions overall so the customer’s admin was able to correct permission settings on the target system without having to spend an inordinate amount of time in the process.  We stopped using the WD and as we did not have access to any other network based storage device at this point (like a QNAP) we ended up using a laptop as the copy “shuttle” device.  It took a long time to get everything copied over but we avoided any further problems with permissions.  So, lesson one, ALWAYS make sure your copy device of choice honours NTFS permissions or you could be facing a lot of extra work on the migration.

Our second challenge was SharePoint.  The customer had a WSS 3 SharePoint site on the old SBS2003 machine that was core to their operations.  This site “replaced” the original SharePoint 2 site on the SBS2003 box.  We decided that the SharePoint migration process listed in the SwingIt docs would not be appropriate for migrating this site so I went to my boss, Sean Wallbridge SharePoint MVP, for help.  Sean ended up migrating the site using processes described on his blog (brainlitter.com) to accomplish migration from WSS 3 to SharePoint 2010 Foundation.  He first identified elements on the original site that would not migrate (in this case the theme) so the customer was prepared.  He then performed the migration which went well but took time.  The site did NOT replace Companyweb on the new server, it lives as a separate site.  But as I worked with him to perform the migration I realised that there are many speedbumps that you could hit following the SwingIt SharePoint migration process (or almost any other SharePoint migration process).  You should be prepared to deal with problems that could arise during the SharePoint migration.  Using a site like Sean’s blog (and some of the other sites he references) will help you prepare and, if needed, overcome problems that could arise with the SharePoint migration.  So, lesson two, be prepared when it comes to SharePoint.

The last challenge was the Exchange migration. The migration from Exchange 2003 to Exchange 2010 is always “interesting”.  The SwingIt kit documents the process very clearly but you will have to reference outside sources if you hit problems along the way.  We had no issues with the migration itself although as our customer had maxed out the Exchange datastores on the source server (pushing 80GB of data), the actual migration of mailboxes from the Swing server (2003) to the SBS2011 server took the better part of 48 hours.  Where we hit problems was with the cleanup of the 2003 box (public folders) before we could follow the instructions for removing Exchange 2003 form the domain and the subsequent demotion of the Swing temporary server.  The problems we hit were not covered within the SwingIt documentation and I had to spend a bit of time Googling to find appropriate “fixes”.  Lesson three, things never go as smoothly as you would like so be prepared to spend some time finding answers.

The SwingIt process is still my preferred method for SBS migrations and I give the folks at SBSmigration.com top marks for their kits and their processes because they “just work”.  But migrations are not simple things and you have to be prepared to handle the exceptions that crop up.  The cost of the SwingIt kit is minor compared to the cost of performing a bad migration based on misinformation (and there is a lot of it out there).  So, if you are planning to migrate, invest in the kit, read the documentation thoroughly, do all of the necessary prep work, give yourself the time to do things correctly and be prepared.