September 9, 2011

Using MRTG to monitor Dell EqualLogic

by Brandon

Alright – if you’re here you likely already know what the Dell EqualLogic is and what MRTG is typically used for. You likely also know that there’s a (free) download from Dell called EqualLogic SAN Headquarters that can give you a ton of good information about your SAN. It has everything from volume capacity, replica capacity, firmware versions and even I/O information for the group(s) you have. Unfortunately, there’s a level of granularity that’s missing from SAN HQ. If you want to know which volume or volumes are getting hammered on a daily basis, which ones have higher read or write I/Os – you’re basically out of luck. The same goes for using the standard Group Manager application.

Here’s an example of what I mean:
read more »

September 7, 2011

HMH Access 21st Century, DEP and You!

by Brandon

Alright, if you haven’t had the pleasure of meeting Access 21st Century yet – and most of you never will – then here’s a quick rundown.

First here’s the FAQ / Tech Notes page – you’re going to need it.

Houghton Mifflin Harcourt is the developer/publisher. Long story short with this application, is it provides a learning environment (computer based) for at risk kids. And that’s about the extent of my knowledge for what the product actually does. Sorry – I’m a systems administrator, not a teacher so going hands on in depth with the learning environment just isn’t part of the role.

Anyway, there’s a couple important things to know that may very well trip you up if you end up installing this platform for your district.

First: DO NOT, I repeat DO NOT trust the built in database backup utility in the application. I’m not sure what the story is with it – but 90% of the time it fails to provide a backup at all. When it does – you can bet that you’re going to try 9 or 10 times to get it to restore. So, by all means use the utility but have a backup plan! My personal method (after a nightmare of problems with the application today) is to stop the Access 21st Century service and take a copy of the data directory in program files. Yes, its a manual process, but I see no other option. I’ll continue backing up the VM, but these other two methods of app DB backup and data directory backup might be the only way to get the data restored. Speaking of – when you can’t restore the database properly – try this:

read more »

August 24, 2011

DPM 2010 Deleting Old Restore Points

by Brandon

Ok, so for the most part this happens automagically and you won’t have to mess with it at all. However, for some reason I found that a few recovery points I had manually created with Microsoft Support while troubleshooting an issue were not going away. How do I know? Here’s the script for determining if you have expired but not pruned recovery points (source):

read more »

Tags:
July 21, 2011

Hyper-V Cluster Service Crash During DPM 2010 Backup and EqualLogic Hardware VSS

by Brandon

Hurrah! Another interesting issue while using Hyper-V with CSV. The problem was odd – If I kicked off a backup of a protection group in DPM which is configured to backup using the Child Partition Snapshot of the VMs, on a random Hyper-V node the cluster service would crash, the attached VMs backups would fail, and the VMs would be brought up on another node. Definitely not what I would like to have seen happen. The problem is there wasn’t a whole heck of a lot of information to go off of in the logs. In fact the only really recurring error is Event ID 5121 which is expected – but only if you’re storage provider didn’t supply a hardware VSS component.

Cluster Shared Volume ‘Volume1′ (‘Cluster Disk – CSV’) is no longer directly accessible from this cluster node. I/O access will be redirected to the storage device over the network through the node that owns the volume. This may result in degraded performance. If redirected access is turned on for this volume, please turn it off. If redirected access is turned off, please troubleshoot this node’s connectivity to the storage device and I/O will resume to a healthy state once connectivity to the storage device is reestablished.

EqualLogic does support hardware VSS – but you need to configure it to work. This should remove any event id 5121 errors…. of course it didn’t with mine. (Need to reboot maybe…)

1. Launch Remote setup Wizard from Programs menu->Equallogic->Remote setup
wizard. ** c:\Program Files\EqualLogic\bin on Core
2. In remote setup wizard select “Configure this computer to access a PS
series SAN”
3. If the group is not added then add the group. If the group is added
select the group and click on “Modify”
4. Verify that “Group Name”, Group IP address, Chap Credentials and the
password is entered correctly. Note that the chap credentials for VSS/VDS
access could be different from iSCSI access.
5. restart the equalogic vss provider (net stop eqlvss & net start eqlvss)

Turns out the actual crashing was an easy fix with a patch from Microsoft – KB2494162. Install that, reboot and let DPM run rampant. So far I’ve been running my protection group every hour on the hour (to test) and its not hiccuped once.

Tags: ,
July 18, 2011

Configuring Broadcom iSCSI Offload on Dell R715

by Brandon

See this document - starting on page 91 to page 102. The most important thing to do is make sure you select the correct adapter (page 101). Most documentation is going to show you that you need to select “Microsoft iSCSI Initiator” in the iscsicpl (discovery and targets tabs). This is not correct if you want to use the iSCSI offload functionality of the Broadcom NICs.You need to chose the “Broadcom NetXtreme” device(s) listed in the drop downs. If you select the Microsoft iSCSI Initiator you will still get decent performance – but the iSCSI Offload performance is a decent boost (albeit with a slight impact to CPU performance).

Here’s the difference I saw using iometer — (note – this was done against a iSCSI boot C: drive with a CSV in the cluster volumes directory. I know that’s not good to test against, but its what I had available.)

Without iSCSI Offload:

And with iSCSI Offload

So here’s the basic steps in server core:

  1. Open BACS and expand the NIC you’re planning to use for iSCSI. There will be a NIC icon and iSCSI icon:
  2. Highlight the iSCSI Adpapter in the left panel. In the right hand panel select the configuration tab. Default will be IPv4 DHCP. Disable that and manually enter an address (this is easier than figuring out which adapter is which in the next steps). Click Apply at the bottom. Wash, rinse and repeat for your remaining iSCSI adapters.
  3. Launch the iSCSI Initiator (iscsicpl)
  4. In the discover portal tab, click Discover Portal –> Enter your SAN IP and port # –> click Advanced –> In the local adapter drop down select your ISCSI adapter (Broadcom) –> in the initator IP make sure you select the IP you assigned in step 2.
  5. In the targets tab (refresh your targets if its empty) –> highlight your target –> click connect –>Enable mulitpath if needed –> click advanced –> repeat the selections you made in step 4
  6. You should now see the connections being offloaded in BACS
July 6, 2011

Dell R715 w/ EqualLogic Failover Cluster Validation Failure

by Brandon

Cluster Disk 0 does not support Persistent Reservations. Some storage devices require specific firmware versions or settings to function properly with failover clusters. Please contact your storage administrator or storage vendor to check the configuration of the storage to allow it to function properly with failover clusters.
Well that’s just great. Another stepping stone into my hair turning grey and being yanked out at the roots. The above is what I get when I try to validate the cluster I’m building for Hyper-V. Connected are 3 hosts, each with MPIO iSCSI connections to a pair of EqualLogic PS6000 arrays hosting a 2TB LUN for data. I thought I had followed all the correct steps to get this to work… until this happened.
A little searching brought me to this thread on Technet which basically describes my issue to a T.

The root cause of the problem is the iqn name that is passed to the Equallogic. If you boot diskless and the first 34 characters of your IQN names are identical then you will have this problem with Windows 2008 R2 Datacentre Edition, for clustering.

Somewhere between the Microsoft ISCSI Initiator and the Broadcom driver/Boot ROM the iqn name is being concatenated to 34 characters. So when you attempt to do a Cluster Validation the requests for PR going to the Storage are essentially seen to be coming from the same host by the Equallogic, as the first 34 characters of the iqn names are identical.

So, the workaround is to make sure your IQNs are unique within the first 34 characters. I have adjusted mine to reflect the server names.

The IQN name field is supposed to accept 255 characters. It is unclear at the moment if the problem is a bug in Microsoft or Broadcom. But given the time taken to prove to Dell there actually was a problem….I am happy to go with the workaround.

I’m not going to blame Microsoft at this point… I am beginning to get the suspicion that these Broadcom NICs and drivers/firmware are nothing but complete crap and need to killed with fire.

 

Following those hints above and setting the IQN for each server to the hostname worked perfectly.

June 24, 2011

Dell PowerEdge R715 iSCSI Boot with Server Core 2008 R2

by Brandon

Note: This is written as it comes along and so you’ll get to see the failures and hopefully a wrap up with how to make it actually work.

Alright, so here’s the project: Get Windows Server Core 2008 R2 booting from iSCSI. The why of it is somewhat simple. I want to be able to iSCSI boot so I can have a set of Hyper-V host servers at my primary location completely configured and perfectly happy. Then I will replicate those LUNs to our offsite SAN hardware. When disaster strikes, I can then just configure those servers at the DR location to boot from the replicated LUNs. (in theory)

Yes, there’s easier ways to do this for my scenario like Citrix Essentials for Hyper-V and other SAN replication software which would then allow me to just fail over the setup or configure it as a geo-cluster. But reality is those cost money… and it wasn’t included in the budget for this project. We got the money for the hardware/OS and that’s pretty much it.

Server: Dell PowerEdge R715 – 12 Core AMD – (3) 4 port Broadcom BMC5709C NICs

Storage: Dell EqualLogic SAN Group – 2 PS6000s and 2 PS4000s

(2) Dell PowerConnect 6248 switches.

iSCSI Boot Learning Material:

Broadcom NetXtreme User Guide (Dell) // Original by Broadcom

Dell Instructions to Perform Boot from iSCSI (page 21-24 & 36-38 )

In the BIOS you need to set the boot order to put the Embedded NIC first in the list, followed by DVD then local storage. Second you need to enable the embedded NIC (assuming that’s what you’re using for boot) to allow for iSCSI boot instead of PXE. (UPDATE: SEE BELOW FOR MORE ON THE BIOS CONFIGURATION)

read more »

May 27, 2011

Display Full Access Permission on Exchange Mailbox

by Brandon

This command will show you who has full access to a mailbox – explicitly added permissions only – using the EMS. You could of course do it one by one in the EMC if you like pain.

Get-Mailbox -Server “mailboxserver” | Get-MailboxPermission | where { ($_.AccessRights -eq “FullAccess”) -and ($_.IsInherited -eq $false) -and -not ($_.User -like “NT AUTHORITY\SELF”) } | fl

Source

May 20, 2011

Exchange 2007 Public Folder Calendar and Management

by Brandon

No. Frankly it wasn’t helpful because I have full admin rights to Exchange. I have Public Folder rights to Exchange. I am the administrator. So no, as usual it wasn’t helpful.

Alright, let’s get to fixing the issue of not being able to create a new public folder in Exchange 2007 even though you should apparently have rights to do so.

First things first – I’m assuming you have at least Exchange 2007 SP1 which is when they included the Public Folder management snap-in. If you don’t – why not? Anyway, open up the Exchange Management Console –> Toolbox –> Public Folder Management Console. Expand Default Public Folders and click on that same folder. On the right hand column you’ll see the option to create a New Public Folder.

Click it  and give the folder a name – then click next.

Now head on over to Outlook. Expand Public Folders and your new Test Folder should be there. Right click on the folder and select New Folder. Change the drop down to Calendar Items and give it a name. Click Ok when you’re done.

Wait… did you get the same error as at the top of the post? I did. Here’s why: If your domain is setup like mine you have a normal (mail enabled) user and an admin (not mail enabled) user account. When you created the Public Folder in the EMC it was most likely under the admin user account which now has owner permissions to the folder, but your lowly Outlook user account doesn’t. That little issue will keep you from doing anything worthwhile on the folder.

Head back over to your Exchange box and open the Exchange Management Shell, then type in:

Get-PublicFolder “\Test Folder”| Add-PublicFolderClientPermission -User “domain\normaluser” -AccessRights Owner -whatIf

(NOTE: Remove the -whatIf at the end for the command to really take hold)

If all goes well you’ll get something like this:

Identity               User                      AccessRights
——–                   —-                       ————
\Test Folder     normaluser…          {Owner}

Guess what? You can now go back to Outlook and not only create your Public Folder Calendar – but you can also add other user permissions to the folder/calendar.

Here’s a slightly more handy command should you need to take ownership of multiple folders all at once:

Get-PublicFolder -recurse | Add-PublicFolderClientPermission -User “domain\normaluser” -AccessRights Owner -whatIf

May 17, 2011

Renaming a Data Protection Manager Server

by Brandon


Just an FYI… :)

Follow

Get every new post delivered to your Inbox.

Join 652 other followers