New Data Centre – Hong Kong

Mission

Deploy 20 new Windows 2008 servers to enable the region to host its own services based on VMware and NetApp technology. – Simple enough…

The project schedule was given a significant boost by the local team ,as the equipment being racked and cabled, ready to start implementation. The NetApp filer already had a basic configuration setup by the reseller but the root password was not known in HK. This incident highlighted the main problem for the project. Working in a different time zone or more importantly, being 8 hours ahead of your peers means it is not easy just to call someone and ask a question. It was simple to power down the filer and reset the password from the boot menu but it had the scope to create an 8 hour delay while we waited for the UK to wake up.

The filers were upgraded to DoT 8.0.1P1 then ASUP, RLM, CIFS, NFS, iSCSI and vLAN tagging added to the configuration. The 3 ESX servers were based on a scripted install I had developed in the UK, using UDA. Testing had shown it was possible to go from bare metal to production ready in just 20 minutes per server and multiple servers could be installed concurrently. However the test kit was not the same as the new Proliant servers which had been purchased. HP's new internal SAS controllers are different to what has gone before and are not compatible with the scripts which I had prepared. The ESX error messages where limited and it was not easy to work out what was causing the installation to fail. I did not help that it was not possible to repeat the failure and different errors were reported on each attempt. The solution was to install ESX using a DVD and manually configure each server, which took significantly longer than the 20 minutes the script would.

Once ESX was installed it was discovered that HP's Über quad port NIC cards did not work. All 6 cards did not show any link lights and were invisible to the OS. Long story short, a VMware bug was discovered after a long google session and the ESX patch deployed. Quick reboot and IP was flowing.

Not sure what the issue was with vifs/teaming and vLANs but we could not get the switch configuration to work reliably when the ports were configured as multimode (active/active) at either the filer or ESX end. Given the low current bandwidth requirements, the issue was worked around by setting the vifs to single mode or removing the tagging and using multimode where appropriate. {Will review this next week but it was good enough to get the system serving data} Again if I could have phoned some peers I could have probably resolved the issue…

The ESX boxes were added to vCentre and the NetApp VSC was used to configure the ESX servers with optimal settings. Right click apply, job done – Great feature! A test VM was created in the NFS datastore and the torture began. Cables removed one after another to prove all the HA does what it is supposed to do. Past without issues and NFS datastores work great with cluster takeovers, seams much smother than FCP luns but that is just my opinion.

Now that the platform was ready time to start building Windows servers. 3 VMs were created based on Windows 2003 standard server, Windows 2008 standard server and Windows 2008 enterprise server. This is the point when it became clear just how slow the internet can be. In the UK I can get 20+ MBs download speeds from major sites like Microsoft, NetApp and VMware from my house and even more at the office. This was not the case in HK, despite the 15MBs link to the internet. My main recommendation from this adventure is to download all the files, patches, iso and create the VMs before leaving the office. The idea for the local VMware creation was skills transfer to the local team but it took much longer than the 'training' warranted.

The 3 fully patched and configured VMs were converted into templates and the VMware customisation file created. The two W2K3 VMs created without issue but Windows 2008 servers turned out to be a much tougher nut to crack. The VM would be created and then the configuration applied and become stuck in a never ending reboot cycle. – Again, long story short it was a problem in the configuration and when I changed the password used to a longer and complex password and 'ticked' the Window license applied even though the license key part was empty, the install worked. Not sure which was the fault's root cause but this were the solution. It was then a simple matter to create 20 Windows 2008 servers. The Windows page file was moved to a new VM disk which was created in a 2nd NFS datastore. NetApp dedupe was then run on the OS datastore and 50% saving made. In the UK we have seen this climb to 90% once the system has settled down and more VMs added.

Snapdrive 6.3 has a new feature which allows VMs with connected LUNs to be vMotioned to other ESX servers despite the OS connected LUNs. I am happy to report this feature works well and will be using it with SQL server and iSCSI luns going forward.

The home straight and data protection via CommVault, FCP connected tape and NDMP just needed to be configured. The filer had connected to the library without issue on the first day and correctly identified it as a HP MSL4048 with a single LTO 5 drive. However in a final spiteful twist, when I came to configure the CommVault media agent, it became clear the filer could not access the LTO 5,

" HP Ultrium 5-SCSI (Non-qualified tape drive)" – No!!!!!

As the DoT version was the very latest and greatest it did not leave a lot of options. Anyway a updated version 1.2 HP_LTO5_ULTRIUM.TCF file downloaded from the now site resolved the issue. NMDP would still not work however. Given all the IP ports on the filer it was a case of the VM talking on one IP subnet and the filer replying on a different. Resolved using the options ndmp.preferred.port setting to configure the filer to us only the required port and therefore IP subnet.

So 4 days, 3 ESX servers, 2 filers and 20 new servers.

Hope it helps

Bren

54 Responses to “New Data Centre – Hong Kong”

  1. gravatar Lewis Applebee

    whoah this weblog is excellent i like studying your articles. Keep up the great work! You know, many persons are looking around for this information, you can help them greatly. |

    Reply
  2. gravatar Berna Prosenick

    I would like to thank you for the efforts you’ve put in penning this website. I am hoping to view the same high-grade content by you in the future as well. In fact, your creative writing abilities has inspired me to get my own blog now ;)|

    Reply
  3. gravatar Jordon Mcmina

    We are a group of volunteers and opening a new scheme in our community. Your site provided us with valuable information to work on. You have done a formidable job and our entire community will be thankful to you.|

    Reply
  4. gravatar Valentine Aldana

    hello there and thank you for your info – I’ve certainly picked up anything new from right here. I did however expertise a few technical points using this site, since I experienced to reload the site a lot of times previous to I could get it to load correctly. I had been wondering if your web host is OK? Not that I am complaining, but sluggish loading instances times will sometimes affect your placement in google and could damage your quality score if advertising and marketing with Adwords. Well I’m adding this RSS to my e-mail and could look out for much more of your respective exciting content. Ensure that you update this again very soon.|

    Reply

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>