EMC CDL

So, our DL700 (CX700 with a VTL engine v2.1) has been dog slow and I've been getting media errors. Several cases later, EMC recommended 6 BCCs, because we have ATA disk so no LCCs, and 2 sets of cables to be changed. Also, since we've been getting ghost single fan fault errors on the disk library, we had 2 power supplies changed as well. All the hardware change went well and the library was back on two hours later with the latest frumon code.

The problem became one of the links wouldn't come back on in the VTL engine. I checked our Netbackup servers, I checked the host HBAs, I even tracked down the switch admins our links were going through and had them reset the ports. No luck at all, after 2 reboots, I call EMC in Australia. One more hour of WebX and sending SP collects, X-Rays, and logs to them later, they send out a new VTL engine. 2 more hours of waiting for the parts to arrive. 30 more minutes of installing it since the VTL wouldn't fit because it had a slightly warped bezel. At that point I was ready to use a hammer and kick it in. There is nothing like being at work for 16 hours working on a problem.

The next day I had EMC on the phone again since everything keeps trespassing to SPA, they found the problem on the VTL paths, they somehow all reset to the same one. Ran a path optimizer script, did nothing, then manually repathed everything. So, we are back at the point where we started, let's see what has been fixed.

Slowness: still slow, same speeds.
Media errors: Netbackup still gets them on the VTL and still freezes tapes.
Ghost Single Fan Fault error: every few hours like clock work.

EMC support had the following to say, our test lab CDL runs about the same speeds (5 disk raid 3 running at 30mb/sec? wow!). You shouldn't get anymore media errors, and we have no idea on the single fan fault, maybe reboot the SPs. I can't afford anymore downtime since our duplications are already days behind. Lesson learned? Buy Hitachi if you can afford it!

EMC's response

I have 2 cases out to EMC at the moment, slow disk speeds and ghost single fan fault errors. EMC has come back and said both are hardware replacement fixes. I recently unfroze about 100+ virtual tapes in our VTL and all of the expired ones went to the scratch pool. 4 days later, there are about 10 frozen virtual tapes now. EMC is recommending that I replace cables and BCC's from 3 enclosures now, update the Frumon on all BCCs.

Of course I expect a 2 hours down time while I work with the Unisys people to replace these things. If this doesn't fix my speed issue, it just added another terabyte or 2 or data that I'm behind on duplication. I was hoping for more of a plausible explanation for the speed issue rather than a hardware problem, since we've been having speed issues before I took over this DL700 and they haven't been having hardware issues all the time.

Fun with EMC DL700

If you've ever had to use Powerlink for something you'll know how slow it is. Have you ever used the search function on there? Another painful process. I administer a first gen DL700, and recently I've been looking into why we have such slow duplication to physical tape from it. I've got it to the point where I can't think of much else besides the library is operating at hilariously slow speeds. A simple speed test on the virtual tapes reveals 120mb/sec beginning performance and 8-15mb/sec middle and ending performance. Random speeds are all around 5-9mb/sec. For a raid 3 array it is simply unacceptable. It's current flare code is at 2.19 and the latest I can find on Powerlink is 2.26.

Now I opened a case with EMC on Powerlink and sent them some SP collects, I hope to see a response from them that was better than the CX300 case I opened from them. You see I opened a case for the CX300 performing poorly and they sent Unisys out to collect 2000 logs plus more on our switches and then after not hearing anything from them, they simply closed the case a month later. While the CX300 is slow, it is passable for our VMware environment for now, the DL700/CX700's speeds are not for backup. I'm putting more and more data to it, yet speeds seem to be slower if anything. I hope my experience with EMC this time around is better but if they can't fix this speed issue, I am going to push, and push hard, to change SAN vendors. I won't be the only one pushing for a different storage vendor either, several other SAN admins do not like EMC much either.

It is past my bedtime and the last 2 hours spent searching for possible tweaks yielded nothing on Powerlink, I await EMC's response.