Tuesday, February 19, 2008

Netbackup 5.0 Windows cluster migration

If you are in the same boat as me, Netbackup 5.0 Windows Master Server running on old hardware, you will need to migrate to new hardware before you should upgrade to the latest, in this case Netbackup 6.5.1, version. Here is how I performed the migration relatively painlessly.

First you need the new cluster, see my previous post about installing Netbackup offline. I highly recommend keeping the same Master Server name. If you have a cluster like me, you can have a new cluster name and ip but the virtual Master Server name and ip should remain the same. That is where installing Netbackup offline makes sense. Just use a cheap switch and put in host file entries for everything after you build the cluster online. Make sure your new cluster has every single software needed, I forgot Perl and some modules for Perl on mine. You should work with Symantec support on running NBCC to ensure no catalog issues before migration.

Step 1: Old cluster - Do a catalog backup after you stop all running backups.

Saturday, February 16, 2008

Netbackup 5.0 cluster install work around

Following the previous post, in order to install Netbackup 5.0 into the MSCS 2003 with a different default drive we had to change the default program files location in the registry. Microsoft apparently does not support it, but they tell you how to anyway. http://support.microsoft.com/kb/933700's title even says so, which is kind of hilarious.

I will change it back after and see if it works, I hope this will not have to happen when we are upgrading to 6.5.

Steps to change the ProgramFilesDir registry value to use the default location for the Program Files folder
Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall the operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

To change the location of the Program Files folder back to the default location, follow these steps:
1. Click Start, click Run, type regedit , and then click OK.
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion
3. In the details pane, right-click the ProgramFilesDir registry value, and then click Modify.
4. In the Value data box, type the default location for the Program Files folder, and then click OK.

NoteThe default location of the Program Files folder is systemdrive\Program Files. For example, if Windows is installed on drive C, type C:\Program Files in the Value data box.
5. Exit Registry Editor.

Saturday, February 9, 2008

Netbackup 5.0 Cluster Server move overly difficult

Here is the scenario, Windows 2000 MSCS running Netbackup 5.0MP6 master server and I want to move to Windows 2003 MSCS or VCS 5 running Netbackup 6.5.1 on new hardware while keeping the same master server name. My original plan was to install Windows 2003 on new hardware, install VCS 5, install Netbackup 5, migrate over, then upgrade after a couple of weeks. The hardest part of that plan is to keep the same name while production is still running.

Problem is, Netbackup 5 is only supported in VCS 2.0. No one wants to downgrade to VCS 2.0 in order to upgrade later to 5.0, such a huge jump. So I scrap that idea and going back to 2003 MSCS solution. I setup a MSCS cluster, add in host file entries, plug all the public NICs to a switch and do a cluster install of Netbackup. When I finally get it to work, turns out Netbackup won't install to the drive I tell it to...

The shared data drive is fine, installs fine to there, except the base install keeps going to C: drive which I changed and tried twice. It even says in the finish screen that I may need to change permissions to the drive I told it to install to. At this point I'm at a loss, this migration/upgrade has been far more difficult than expected. Some may say Netbackup is job security! I'd say source of high blood pressure.

Why Sun Microsystems sales are crap?

They wonder why, part of the problem is the support. StorageTek might be marketing leaders in physical tape libraries but I believe they are going away sooner than later. Mainly I think because their technical support is crap. I've had nothing but average to bad experiences with them. This last time was in the middle of the night and they were reluctant to come out. When I got sick of waiting and finally fixed it myself I agreed to have them come out the next day they never did come out. They gave excuse after excuse and I finally gave up. Would I recommend them? If you don't have StorageTek products, I'd say no, buy 2 VTLs instead and duplicate offsite.

Thursday, January 17, 2008

VCS 5.0 not starting in Windows 2003

I'm setting a 2 node Windows 2003 cluster using VSF HA 5.0 and in the process of testing I shut off all nodes and only turned 1 on. I couldn't connect to the cluster with the VCS Cluster Manager. If you do the same or lose all nodes or somehow the cluster completely crashes, here how you can solve this problem.

Look in your event viewer and you will see an error in application, source Had or maybe the same message in a popup. VCS CRITICAL V-16-1-11306 Did not receive cluster membership, manual intervention may be needed for seeding. Run the following command in a command prompt.

gabconfig -x

This will enable seed of control port, if it is not in your server path it is under "\Veritas\comms\gab". Connect as normal through your Cluster Manager.

Sunday, January 13, 2008

Netbackup 6.5 liveupdate using HTTP does not work either

Following the last post, I setup a web server on the Windows master server as the repository for the update to try to update another Windows client. Updated all settings on the client and master server. It still had the same issue of not able to find the files LiveUpdate is looking for at the address. If anyone out there has got Netbackup LiveUpdate working, please let me know how it is setup! For now, 6.5.1 will have to get there manually.

Update 1/17/2008

The Netbackup LiveUpdate 6.5.1 package isn't out yet, the current update release is only for manual installation. Set it up all you want, it won't work without this LiveUpdate package.

Tuesday, January 8, 2008

Netbackup 6.5 Liveupdate using UNC share does not work

*UPDATE* refer to this post for LiveUpdate, it has been confirmed that there is no update package for Netbackup LiveUpdate at this time.

Netbackup 6.5 is a fairly new major release and 6.5.1 has been out for about a month now. I've been trying to use Netbackup 6.5 Liveupdate from a Windows Master Server with a LAN share (UNC) on the same box to update another Windows client to 6.5.1 that is already running 6.5. You set everything up, you run the liveupdate policy, the job quits with a status 77 and the following message in the activity monitor.

Info nbliveup(pid=somenumber) EXIT STATUS 77
execution of the specified system command returned a nonzero status(77)

According to everything I've read and everything I've tried, this method is broken. Following the install guides and the Symantec class books (Yes, I went to class, and no, it was not worth it) my Netbackup Liveupdate environment is as follows.

Friday, January 4, 2008

EMC CDL

So, our DL700 (CX700 with a VTL engine v2.1) has been dog slow and I've been getting media errors. Several cases later, EMC recommended 6 BCCs, because we have ATA disk so no LCCs, and 2 sets of cables to be changed. Also, since we've been getting ghost single fan fault errors on the disk library, we had 2 power supplies changed as well. All the hardware change went well and the library was back on two hours later with the latest frumon code.

The problem became one of the links wouldn't come back on in the VTL engine. I checked our Netbackup servers, I checked the host HBAs, I even tracked down the switch admins our links were going through and had them reset the ports. No luck at all, after 2 reboots, I call EMC in Australia. One more hour of WebX and sending SP collects, X-Rays, and logs to them later, they send out a new VTL engine. 2 more hours of waiting for the parts to arrive. 30 more minutes of installing it since the VTL wouldn't fit because it had a slightly warped bezel. At that point I was ready to use a hammer and kick it in. There is nothing like being at work for 16 hours working on a problem.

The next day I had EMC on the phone again since everything keeps trespassing to SPA, they found the problem on the VTL paths, they somehow all reset to the same one. Ran a path optimizer script, did nothing, then manually repathed everything. So, we are back at the point where we started, let's see what has been fixed.

Slowness: still slow, same speeds.
Media errors: Netbackup still gets them on the VTL and still freezes tapes.
Ghost Single Fan Fault error: every few hours like clock work.

EMC support had the following to say, our test lab CDL runs about the same speeds (5 disk raid 3 running at 30mb/sec? wow!). You shouldn't get anymore media errors, and we have no idea on the single fan fault, maybe reboot the SPs. I can't afford anymore downtime since our duplications are already days behind. Lesson learned? Buy Hitachi if you can afford it!