random windows admin musings

NBU 7.0, Windows Storage Server, Citrix, EMC, oh my!

2010-05-11T12:20:00.002-05:00

Started a new job recently and first thing is diving headlong back into Netbackup. How I miss messing around with downed LTO drives... not. As the year progresses, I hope to upgrade the environment to NBU 7.0, bring in Windows Storage Server, learn much more Citrix and get back into EMC storage administration. I will miss NetApp and hope I don't miss VMware too much as I should be able to get in it and work on it later in the year. More posts as I run into issues...

SQL 2008 MSCS cluster verification error

2009-05-11T16:29:00.011-05:00

This error confused me for a while and Google didn't come up with many hits so I thought I write something out there to help others that may hit on this seemingly rare issue. I ran into this problem on my 2nd or 3rd SQL 2008 x64 install on a Windows 2008 x64 cluster and they were running on the same hardware (Dell R900). Only thing I didn't do this time was do the actual install of Windows. I figured how different can someone else install Windows?....

SQL 2008 x64 Enterprise installing on Windows 2008 x64 Enterprise Microsoft Cluster Server.

During SQL install verification, it fails a cluster check even thought you've ran a Cluster Verification on your 2008 x64 cluster.

Rule "Microsoft Cluster Service (MSCS) cluster verification errors" failed.

The cluster either has not been verified or there are errors or failures in the verification report. Refer to KB953748 or SQL Server Books Online for more information.

As of this writing, KB953748 is still an Microsoft internal KB and according to the Premier Support tech I talked to, it doesn't have much info to help either. After spending hours on the phone with them, I gave up for the night and decided to do some digging myself the next day. After reading all I can find online, one guy hinted at using procmon to monitor the install process to see what is wrong with the verification process, this led to the resolution (ie. tricking the installer) to the issue.

First look in your C:\Windows\Cluster\Reports folder and you should see at least 2-3 files if you ran a full Cluster Verification prior to SQL installation. Cluster.log, ValidateStorage.log, and Validation Data for Node Set xxxxxxxxxxx. The Validation Data for Node Set file is the one that the installer is looking for in order to parse and check to see if the cluster has been verified. What we need to do is run AND PASS (warnings are ok, not failures) a cluster verification to generate these files, then run procmon on the installer and find the Validation Data for Node Set file name it is actually looking for and rename the existing one. I had to delete all the other files in the Reports folder before the SQL install would take the Validation Data for Node Set file with the name it is looking for.

Filter your procmon to only look at setup100.exe after the install starts the verification and look for the two highlighted entries. The second entries should say failed or something to that effect and you can copy the file name it is looking for from there. Delete or move all other files after you rename the file in the reports folder and run verification again, it should work for you. I had to do the same thing on the 2nd node as it looked for a different file name but I used the same verification file.

I also ran into this error on the second node, http://support.microsoft.com/kb/957459, The current SKU is invalid. The simplest way is just to delete the DefaultSetup.ini that contains your embedded serial number and just put in the same serial manually during install. I run my installation off of network drives after I copy them from the ISO. Hope this helps!

Edit: it looks like Microsoft may have something that works as well. However, I did try to skip verification with them on the phone but the commands we used might of been different, try this to see if it works first, much easier.

At a command prompt, change to the hard disk drive and to the folder that contains SQL Server Setup (Setup.exe). Then, type one of the following commands to skip the validation rule:

* For an integrated failover setup, run the following command on each node that is being added:
Setup /SkipRules=Cluster_VerifyForErrors /Action=InstallFailoverCluster

* For an advanced or enterprise installation, run the following command:
Setup /SkipRules=Cluster_VerifyForErrors /Action=CompleteFailoverCluster

Netbackup 5.0 Windows cluster migration

2008-02-19T18:59:00.004-06:00

If you are in the same boat as me, Netbackup 5.0 Windows Master Server running on old hardware, you will need to migrate to new hardware before you should upgrade to the latest, in this case Netbackup 6.5.1, version. Here is how I performed the migration relatively painlessly.

First you need the new cluster, see my previous post about installing Netbackup offline. I highly recommend keeping the same Master Server name. If you have a cluster like me, you can have a new cluster name and ip but the virtual Master Server name and ip should remain the same. That is where installing Netbackup offline makes sense. Just use a cheap switch and put in host file entries for everything after you build the cluster online. Make sure your new cluster has every single software needed, I forgot Perl and some modules for Perl on mine. You should work with Symantec support on running NBCC to ensure no catalog issues before migration.

Step 1: Old cluster - Do a catalog backup after you stop all running backups.

Step 2: Old cluster - Shut off Master Server Virtual IP, this should cause Netbackup to go offline as well. Also shut off any addons, like NBAR or Aptare. Leave the shared disk running. Kill any Netbackup processes that may still be running.

Step 3: New cluster - Shut off all Netbackup services including the virtual name and IP, leave the shared disk running.

Step 4: Old cluster - Copy various databases and custom scripts to the new server over the network, make sure you turn off virus scanner as it can slow it down significantly. You need to know which ones to copy, basically your shared data which are the image database and volmgr database. You can see which ones exactly in your environment if you look in your catalog backup setup. You only need the stuff from your master server and nothing from the media server. An alternate option, and one that Symantec tells you to do, is to recover from a catalog backup. I've not had good luck with it, as it doesn't put everything in the right place, even if I had setup the new cluster exactly the same. Of course, if you didn't keep the directory structure the same, it won't be in the right place. Copying manually is just much faster and easier since you don't have to run a restore from command line, know which tape you need, blah blah. Since this isn't a disaster recovery scenario, just copy the data over.

Step 5: New cluster - bring all Netbackup services up, the only one that goes offline should be the device manager if you have a dedicated Master Server with no attached storage or tape drives.

Step 6: New cluster - Go into "Host Properties" in the Netbackup Administration Console, Master Server, and double click your Master Server. Put all your settings back as they were registry settings before, you did write them down right? You will need a restart once you make that change.

Step 7: New cluster - Check if your scratch pool is still there, mine was still there but not marked as a scratch pool, jobs failed when they ran out of tape. I guess that was some sort of registry setting to not get transferred over. Simply check the box marked Scratch pool and you are set.

Step 8: New cluster - Go into your cluster administrator and setup registry replication. You will want the SOFTWARE\Veritas keys replicated as that is all your Master Server settings. That way when you failover, your 2nd node will have all the right settings.

Step 9: New cluster - Double check host file entries, server list and any custom scripts you may have. At this point, Netbackup should be up and running and since you kept the same Master Server name you will be able to run backups once again.

That was about it, hope your migration goes smoothly. If I saved you thousands in consulting fees, please send a check to .....

I should have some experiences on the Netbackup 6.5.1 Windows Cluster upgrade in 2-3 weeks time, hope that goes smoothly as well.

Netbackup 5.0 cluster install work around

2008-02-16T16:34:00.003-06:00

Following the previous post, in order to install Netbackup 5.0 into the MSCS 2003 with a different default drive we had to change the default program files location in the registry. Microsoft apparently does not support it, but they tell you how to anyway. http://support.microsoft.com/kb/933700's title even says so, which is kind of hilarious.

I will change it back after and see if it works, I hope this will not have to happen when we are upgrading to 6.5.

Steps to change the ProgramFilesDir registry value to use the default location for the Program Files folder
Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall the operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

To change the location of the Program Files folder back to the default location, follow these steps:
1. Click Start, click Run, type regedit , and then click OK.
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion
3. In the details pane, right-click the ProgramFilesDir registry value, and then click Modify.
4. In the Value data box, type the default location for the Program Files folder, and then click OK.

NoteThe default location of the Program Files folder is systemdrive\Program Files. For example, if Windows is installed on drive C, type C:\Program Files in the Value data box.
5. Exit Registry Editor.

Netbackup 5.0 Cluster Server move overly difficult

2008-02-09T10:43:00.000-06:00

Here is the scenario, Windows 2000 MSCS running Netbackup 5.0MP6 master server and I want to move to Windows 2003 MSCS or VCS 5 running Netbackup 6.5.1 on new hardware while keeping the same master server name. My original plan was to install Windows 2003 on new hardware, install VCS 5, install Netbackup 5, migrate over, then upgrade after a couple of weeks. The hardest part of that plan is to keep the same name while production is still running.

Problem is, Netbackup 5 is only supported in VCS 2.0. No one wants to downgrade to VCS 2.0 in order to upgrade later to 5.0, such a huge jump. So I scrap that idea and going back to 2003 MSCS solution. I setup a MSCS cluster, add in host file entries, plug all the public NICs to a switch and do a cluster install of Netbackup. When I finally get it to work, turns out Netbackup won't install to the drive I tell it to...

The shared data drive is fine, installs fine to there, except the base install keeps going to C: drive which I changed and tried twice. It even says in the finish screen that I may need to change permissions to the drive I told it to install to. At this point I'm at a loss, this migration/upgrade has been far more difficult than expected. Some may say Netbackup is job security! I'd say source of high blood pressure.

Why Sun Microsystems sales are crap?

2008-02-09T10:35:00.001-06:00

They wonder why, part of the problem is the support. StorageTek might be marketing leaders in physical tape libraries but I believe they are going away sooner than later. Mainly I think because their technical support is crap. I've had nothing but average to bad experiences with them. This last time was in the middle of the night and they were reluctant to come out. When I got sick of waiting and finally fixed it myself I agreed to have them come out the next day they never did come out. They gave excuse after excuse and I finally gave up. Would I recommend them? If you don't have StorageTek products, I'd say no, buy 2 VTLs instead and duplicate offsite.

VCS 5.0 not starting in Windows 2003

2008-01-17T19:08:00.000-06:00

I'm setting a 2 node Windows 2003 cluster using VSF HA 5.0 and in the process of testing I shut off all nodes and only turned 1 on. I couldn't connect to the cluster with the VCS Cluster Manager. If you do the same or lose all nodes or somehow the cluster completely crashes, here how you can solve this problem.

Look in your event viewer and you will see an error in application, source Had or maybe the same message in a popup. VCS CRITICAL V-16-1-11306 Did not receive cluster membership, manual intervention may be needed for seeding. Run the following command in a command prompt.

gabconfig -x

This will enable seed of control port, if it is not in your server path it is under "\Veritas\comms\gab". Connect as normal through your Cluster Manager.

Netbackup 6.5 liveupdate using HTTP does not work either

2008-01-13T20:37:00.000-06:00

Following the last post, I setup a web server on the Windows master server as the repository for the update to try to update another Windows client. Updated all settings on the client and master server. It still had the same issue of not able to find the files LiveUpdate is looking for at the address. If anyone out there has got Netbackup LiveUpdate working, please let me know how it is setup! For now, 6.5.1 will have to get there manually.

Update 1/17/2008

The Netbackup LiveUpdate 6.5.1 package isn't out yet, the current update release is only for manual installation. Set it up all you want, it won't work without this LiveUpdate package.

Netbackup 6.5 Liveupdate using UNC share does not work

2008-01-08T21:31:00.001-06:00

*UPDATE* refer to this post for LiveUpdate, it has been confirmed that there is no update package for Netbackup LiveUpdate at this time.

Netbackup 6.5 is a fairly new major release and 6.5.1 has been out for about a month now. I've been trying to use Netbackup 6.5 Liveupdate from a Windows Master Server with a LAN share (UNC) on the same box to update another Windows client to 6.5.1 that is already running 6.5. You set everything up, you run the liveupdate policy, the job quits with a status 77 and the following message in the activity monitor.

Info nbliveup(pid=somenumber) EXIT STATUS 77
execution of the specified system command returned a nonzero status(77)

According to everything I've read and everything I've tried, this method is broken. Following the install guides and the Symantec class books (Yes, I went to class, and no, it was not worth it) my Netbackup Liveupdate environment is as follows.

Windows 2003 SP1 Netbackup 6.5 Master Server
--------------------------------------------
Master server has a share with the 6.5.1 files unzipped to it. The share is a NULL Share (Windows 2000 term) or a share that allows anonymous access in Windows 2003 world. Reason for this is because of this bug report from Symantec. Now in the bug report it states you need to setup a NULL Share, but if you are running Netbackup 6.5 on Windows 2000, I'd hope you are going to upgrade to 2003. If you are like the rest of the sane system admins, you are running it on Windows 2003. In order for anonymous access to be setup on a Windows 2003 share, you need to do the following:

Go to gpedit.msc > Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options, find the following policy setting:

Network access: Let Everyone permissions apply to anonymous users, set to Enabled.
Network access: Shares that can be accessed anonymously, add your share name.
Network access: Restrict anonymous access to Named Pipes and Shares, set to Disabled.

I enabled the guest account for good measure, and then in my share, I let Everyone have read access in the share permissions and in security, I let Everyone have Read and List access. I also added the ANONYMOUS LOGON id that Symantec wanted you to add in their bug report, might as well open all doors. Keep in mind your machine's share is pretty open if not all open now, not exactly an ideal solution.

Windows 2003 SP1 Client with Netbackup 6.5GA and LiveUpdate Agent
--------------------------------------------
Client has 6.5GA and LiveUpdate Agent set to LAN with the share in the "Location of LiveUpdate Server" field.

After all this, I run the Liveupdate policy again and it still quits with a status 77. After looking at Log.LiveUpdate on the client, (not the server, as the bug report seems to indicate, you are updating the client not the server) you will see HR 0x802A0017 DECODE: E_UNABLE_TO_CONNECT_TO_NETWORK_RESOURCE before you setup the anonymous accessible share on Windows 2003 but should not after the steps above. What I'm seeing in Log.LiveUpdate after the anonymous access is HR 0x802A0033 DECODE: E_CANT_CREATE_FILE. The rest of the logs seems to indicate Netbackup Liveupdate is looking for liveupdate mini-tri files (no idea why not NB 6.5.1), then the full tri file which has the full liveupdate catalog, then fail with a connection failed with a return code of 1814, LiveUpdate could not retrieve the catalog file of available Symantec product and component updates.

There is no flg file, there is no livetri.zip, there is no minitri.flg or any other liveupdate zip file in the 6.5.1 Windows release update package. There are also nothing that says we have to generate those ourselves. So now I'm at a loss, I basically opened up the share so everyone and their mom can access it if it had a connection to the outside world just to let Netbackup LiveUpdate in, and then it can't find what it needs. I'm going to try the http method later on but it is not looking promising. Looking around STN people are either just as confused, or seems to think Netbackup LiveUpdate works the same as the Antivirus version, ie, automatically updates off the web. This is one new feature that I haven't found anyone out there say, yes I have it running and it works well. I'd recommend holding off on this feature, I'm just glad I'm trying to get it to work on a small extranet environment instead of a big intranet environment.

EMC CDL

2008-01-04T16:02:00.000-06:00

So, our DL700 (CX700 with a VTL engine v2.1) has been dog slow and I've been getting media errors. Several cases later, EMC recommended 6 BCCs, because we have ATA disk so no LCCs, and 2 sets of cables to be changed. Also, since we've been getting ghost single fan fault errors on the disk library, we had 2 power supplies changed as well. All the hardware change went well and the library was back on two hours later with the latest frumon code.

The problem became one of the links wouldn't come back on in the VTL engine. I checked our Netbackup servers, I checked the host HBAs, I even tracked down the switch admins our links were going through and had them reset the ports. No luck at all, after 2 reboots, I call EMC in Australia. One more hour of WebX and sending SP collects, X-Rays, and logs to them later, they send out a new VTL engine. 2 more hours of waiting for the parts to arrive. 30 more minutes of installing it since the VTL wouldn't fit because it had a slightly warped bezel. At that point I was ready to use a hammer and kick it in. There is nothing like being at work for 16 hours working on a problem.

The next day I had EMC on the phone again since everything keeps trespassing to SPA, they found the problem on the VTL paths, they somehow all reset to the same one. Ran a path optimizer script, did nothing, then manually repathed everything. So, we are back at the point where we started, let's see what has been fixed.

Slowness: still slow, same speeds.
Media errors: Netbackup still gets them on the VTL and still freezes tapes.
Ghost Single Fan Fault error: every few hours like clock work.

EMC support had the following to say, our test lab CDL runs about the same speeds (5 disk raid 3 running at 30mb/sec? wow!). You shouldn't get anymore media errors, and we have no idea on the single fan fault, maybe reboot the SPs. I can't afford anymore downtime since our duplications are already days behind. Lesson learned? Buy Hitachi if you can afford it!

EMC's response

2007-12-20T17:01:00.000-06:00

I have 2 cases out to EMC at the moment, slow disk speeds and ghost single fan fault errors. EMC has come back and said both are hardware replacement fixes. I recently unfroze about 100+ virtual tapes in our VTL and all of the expired ones went to the scratch pool. 4 days later, there are about 10 frozen virtual tapes now. EMC is recommending that I replace cables and BCC's from 3 enclosures now, update the Frumon on all BCCs.

Of course I expect a 2 hours down time while I work with the Unisys people to replace these things. If this doesn't fix my speed issue, it just added another terabyte or 2 or data that I'm behind on duplication. I was hoping for more of a plausible explanation for the speed issue rather than a hardware problem, since we've been having speed issues before I took over this DL700 and they haven't been having hardware issues all the time.

Fun with EMC DL700

2007-12-19T22:57:00.000-06:00

If you've ever had to use Powerlink for something you'll know how slow it is. Have you ever used the search function on there? Another painful process. I administer a first gen DL700, and recently I've been looking into why we have such slow duplication to physical tape from it. I've got it to the point where I can't think of much else besides the library is operating at hilariously slow speeds. A simple speed test on the virtual tapes reveals 120mb/sec beginning performance and 8-15mb/sec middle and ending performance. Random speeds are all around 5-9mb/sec. For a raid 3 array it is simply unacceptable. It's current flare code is at 2.19 and the latest I can find on Powerlink is 2.26.

Now I opened a case with EMC on Powerlink and sent them some SP collects, I hope to see a response from them that was better than the CX300 case I opened from them. You see I opened a case for the CX300 performing poorly and they sent Unisys out to collect 2000 logs plus more on our switches and then after not hearing anything from them, they simply closed the case a month later. While the CX300 is slow, it is passable for our VMware environment for now, the DL700/CX700's speeds are not for backup. I'm putting more and more data to it, yet speeds seem to be slower if anything. I hope my experience with EMC this time around is better but if they can't fix this speed issue, I am going to push, and push hard, to change SAN vendors. I won't be the only one pushing for a different storage vendor either, several other SAN admins do not like EMC much either.

It is past my bedtime and the last 2 hours spent searching for possible tweaks yielded nothing on Powerlink, I await EMC's response.

Windows Server 2008 Hyper-V

2007-12-18T14:43:00.000-06:00

Microsoft today had a webcast to announce what many people in the industry already knew, Windows Server 2008 Hyper-V. This is not an updated version of Virtual Server 2005 but a hypervisor from Microsoft. My boss expects it take over VMWare in a couple of year but I'm not sold yet. I am surprised at how developed (at least on paper) it already seems to be. Microsoft says it has support for Windows product lines and Linux (Novell now, Redhat in January) in their presentation. You can deploy it with 2008 Server Core as well. Hyper-V costs $28 which seems insanely cheap for Microsoft, the presentation said you can spend $28 and consolidate your Windows 2003 machines on it. I will have to download 2008 RC1 with Hyper-V and check out the features more in depth.

Things that stood out in the presentation.

Hyper-V server costs $28!
Low overhead design from the beginning (for Microsoft).
Server Core.
MMC 3.0 UI.
1/3 cost of VMware (Says Microsoft)

I tend to believe that last point since people are already familiar with the Windows world but not VMware when they are virtualizing. Of course if you are virtualizing mostly none Windows machines I'm not sure if Hyper-V will work. It does seem fatter as the hypervisor / Server Core both need to be (I think) in order to provide all functionality and it needs more hardware (memory) than VMware.

Links: Here are some links for Hyper-V.
Download Hyper-V
Hyper-V Overview

Netbackup VSP tmp file locks

2007-12-18T10:50:00.000-06:00

I work on a Netbackup 5.0MP6 Windows Master Server and sometimes on clients the drives fill up with VSP tmp files. They use every single bit of space and the official fix from Veritas is to reboot the server and then delete the files. Not a very good solution if you have any kind of real server environment. Here is how you can get around that.

The symptoms: Symantec Netbackup creates tmp files on the root drives to backup open files on Windows clients, these are suppose to be deleted after the backup but they often do not ending up filling the drives to capacity. They look like these files.

_vxfiVspCacheFile_1.tmp
_vxfiVspCacheFile_2.tmp
_vxfiVspCacheFile_0.tmp

The cause: Virus scanners, including Symantec's own Norton antivirus puts locks on the VSP (Veritas open file backup) tmp files and they do not get deleted. The only sure fire way to delete them is to reboot the server.

The solution: I have heard this does not always work but I've never had a problem with it on Windows 2000 or 2003 servers. We can use a little program called Process Explorer from Microsoft (System Internals) to unlock the file and delete it. First stop your antivirus from its' real time scanning. Then start up Process Explorer and put in the Vsp tmp file name that shows up on your drives (they could be hidden). Process Explorer will fine the handle locking them and you can right click and close the handle locking these files. I recommend that you try Process Explorer on something else first if you've never used it since you can close any handle and blue screen your box if you don't know what you are doing. Once the handle is closed, you can delete the tmp files, turn antivirus back on and pick up your fat raise... yeah right.

Now how do we prevent it? No good way really, turn off virus scanner? yeah right. Turn off open file backups in VSP? If you don't need it. Don't use Netbackup, haha. With Windows 2003 you can use VSS (Windows built in Volume Shadow Copy Service or Volume Snapshot Service or VSS... Microsoft...) in the newer version of Netbackup (6.5 for sure) but I haven't tried this yet. Maybe it will work better since VSS lets you set max amount of disk space to use.

Links: I used the following in my original search to solve this problem.
MCSE forums
Bob's Blog
Download Process Explorer