Category: computers

Having troubles running SQL Server Manager on your shiny new Windows 2008 installation?  Make sure you right click and do a “Run as administrator” on it.  Took me awhile to figure this one out.  Otherwise you’ll just get the error “Login failed for user domainuser.  (Microsoft SQL Server, Error: 18456)” even if you’re a member of the local administrators on the SQL box.

Yet another instance of where UAC sucks.

After having issues with a potential bug on Feb the 29th, I’ve finally gotten things straightened out. 

Today I attempted to move the mailboxes again, but received the same error message.

Summary: 1 item(s). 0 succeeded, 1 failed.
Elapsed time: 00:00:08

User’s Mailbox
Failed

Error:
The address list service on the server ‘servername.fqdn’ is not running. The Exchange server address list service failed to respond. This could be because of an address list or email address policy configuration error.

The Exchange server address list service failed to respond. This could be because of an address list or email address policy configuration error.

Exchange Management Shell command attempted:
‘fqdn/Managed Users/User’ | move-mailbox -BadItemLimit ‘10’ -TargetDatabase ‘servernameFirst Storage GroupMailbox Database’

Elapsed Time: 00:00:08

It turns out that all I needed to do was restart the System Attendant service on the machine I was trying to move the mailboxes to.  After I did that, everything moved successfully.  I then attempted to hit OWA on the new machine and it was successful!  So I finished moving all the mailboxes over, changed ISA to point to the new server name, imported the right certificate and I’m good to go.  Almost 5 days of downtime, but no email lost, yay!

Now I just need to decommission the old Exchange box and give the new one more RAM.  Not a bad few days work.

Well, I’ve got the new Exchange box up and running.  However, I can’t move the mailbox from one machine to the other.  Thankfully, I’m not the only one having this problem today.  It appears as though because it is the 29th of February (leap year), there is a bug in Exchange 2007 preventing certain things from completing.  There’s a nice TechNet thread on it, and it appears by setting your date to tomorrow fixes it.  I think I’ll just wait to move the mailboxes till tomorrow or later then 🙂

I know it has been well documented that you cannot upgrade Windows 2003 to Windows 2008 with Exchange 2007 installed and expect Exchange 2007 to keep functioning.  However, let’s say you may have accidentally done the upgrade on a standalone Exchange 2007 box you have, you know, just in case it were to happen (like it did to me).

Prior to doing the upgrade, you’ll notice a few things.  First of all, you’ll be prompted that you need to uninstall Powershell.  However, no where does the compatibility checker say anything about needing to uninstall Exchange 2007 prior to upgrading.  I found this hilarious (in a sad, pissed off way) because I had tried to upgrade my WSUS virtual machine first, and it had told me that I would need to uninstall Powershell and WSUS prior to upgrading.  I’m so glad that I wasn’t told anything about Exchange in a similar fashion.  Ugh.  By the way, I was running Exchange 2007 with SP1 prior to the upgrade…of death!

The first stumbling block, which should have caused me to stop the upgrade process, was uninstalling Powershell.  Since I had installed it prior to installing SP2, uninstalling it becomes a pain.  This is because Powershell is a windows update and if you install a service pack you can’t uninstall any updates prior to the service pack.  Lovely.  Well, in another unsupported way you can uninstall it.  You have to browse to %windir%$ntuninstallkb926139$spuninstall and run the spuninstall.exe.  Now, this may or may not be on your machine anymore either.  On some of my virtual machines it was there, but on my Exchange server it was not, so I copied it over and ran it.

Ok, so now I can upgrade, yay!  Windows does its thing and upgrades everything and restarts successfully.  I was actually fairly impressed when it booted up.  It looked like it actually worked.  However, then I went into the services snap-in.  I usually do this with this machine because it is slightly RAM starved and sometimes all the Exchange services don’t start.  Sure enough, they hadn’t all started.  So I went through and tried to start them all.  All started but the information store and the system attendant service because of a dependent service.  Crap, of course it’s the important ones.

Well, first thing I tried was to reinstall Exchange 2007 SP1, just to see if that would work.  Of course this required me to reinstall Powershell, since that’s a pre-req.  No big deal, installed that easily.  Then when I tried to actually install SP1 it just bombed saying it couldn’t upgrade.  Looking through the eventlogs it was because it was trying to spin up those two services.  Great.

Well, doing some quick registry editing, I found that the service it was dependent on was NtlmSsp.  Needless to say, this service does not exist on Windows 2008, hence the issue.  Two seconds later, I removed that dependency from within the registry and restarted the machine.  The machine reboots, and low and behold all of the services start.  And all the email that was in the queue on my Edge Transport machine left the queue and made it into Exchange.  Downside is that I was doing this all remotely and OWA still didn’t work.

Honestly, I wasn’t that worried about OWA.  I mean, as long as I can get my emails back and then do the correct upgrade (aka, no upgrade at all) I’d be a happy camper.  Heck, even after installing Powershell back on it, I was able to open up Exchange System Manager.  Really, if I didn’t know all about the services and didn’t use OWA, I would’ve never known it wasn’t working.  Oh, well, maybe the exceedingly high CPU utilization, but oh well.

When I got home, I had to test to see if I would be able to access my email.  Sure enough, Outlook worked like a charm.  I received all the queued email that had been sitting there for a day, and I was even able to send an email.  Pure craziness. 

What makes this even better is that the Exchange team actually decided (well, they actually went into it knowing what they were getting into) to try this same thing too.  However, they weren’t able to get things working.  I think the large mess-up was re-installing SP1.  I’m glad I didn’t decide to go down that path, especially since mine worked.  Needless to say I’m working on building a new VM with Windows 2008 and then going to add it t the ORG and move the mailboxes over to the new one.  However, in the meantime, at least my email is functioning 🙂

I’ll be sure to post again on if I ran into any more issues with the mailbox move.  Worst case I suppose I could just do an ExMerge (actually Export-Mailbox for 2007) on the mailboxes or dump the email out of outlook to a PST.  I’d rather not do that, but if that’s what it takes…

Something that is making me very angry with the current project I’m on is the difference between DAS, NAS, and SAN technologies.  The worst is that I’m working with these people on a specific thing not related to storage infrastructure, but instead development architecture and the people that are dealing with the storage infrastructure are the people that don’t know what the hell their talking about.  In particular, the hosting provider that does all of the storage infrastructure work for us doesn’t know what the differences are.  Oh, and don’t get me started on a VMware paper that we had that didn’t know the difference either.  It just drives me nuts. 

For those of you keeping score, I’m going to outline this out.

DAS = Direct Attached Storage.  These are disks that are physically located in your host machine.

NAS = Network Attached Storage.  NAS is file based.  For example a CIFS or NFS share.  This is typically TCP/IP based access.  The NAS device “owns” the data on it.  That is, the NAS device administers the data.  For example, you connect to a NAS device from a windows machine by accessing servernameshare.

SAN = Storage Area Network.  SAN is block based.  This is when LUNs (logical unit numbers) are involved on a host.  The host “owns” the data.  The host is in charge of the partition, formating, and access to the LUN.  You can access a SAN via two protocols: iSCSI (TCP/IP) and/or Fiber Channel (FC). 

I’m so sick of seeing people talk about iSCSI NAS.  There’s no such thing because in a NAS scenario you are sending CIFS or NFS protocols over TCP/IP while in a SAN solution you’re sending SCSI protocols over TCP/IP.  Huge difference.

And yes, you can have a device that serves both NAS and SAN from one filer.  This is called Unified Storage.  All NetApp devices can do this.

Are we clear now?!

I wasn’t necessarily going to post this, but since ISA seems to be the most linked to thing on this site because of only 2 articles, I figure it can’t hurt to talk about it. Especially since it was a very strange problem I had with it and I’m sure I won’t be the only one with it.

Anyways, at work I am utilizing ISA 2006 Std edition in a front and back wall scenario. Site-to-Site VPN terminate on the external firewall, and all of our local VLANs (55 of them) are routed off of the internal firewall. So far, nothing that complex. It’s just a simple DMZ between the external and internal network setup.

Anyways, I had a site-to-site VPN (IP pre-shared key) between a customer and us. Basically, we just need to hit a single machine, so the remote network contained two IP addresses, one for the client’s gateway (this is added by default in ISA 2006, DO NOT delete it, also be sure that the remote site has added your gateway in as a remote network too!) and another for the machine we needed to hit on their local network. Anyways, it was working fine. Well, actually, nobody was using it quite yet, but testing had been completed, and I was able to access everything that the developers would need. Anyways, the customer decides that they need to add another IP address that we’ll need to access. Again, no big deal. I’ll just add the IP to the network list for this client. Just to make sure everything’s working, I test it. Nothing works to the new IP. However, the old IP still works fine. What the hell?!

For those of you unfamiliar with ISA, it’s not like I created a new VPN for this new IP addition, or anything like that. I simply added the new IP to the existing network. All the routing and firewall rules remained the same. Adding the new IP to the list of remote networks should have allowed it to work.

Working with the IT person at the customer, I learn that when I try to hit the new IP address, the Quick Mode authentication was failing because the ISA server was sending the wrong local network that the request was coming from. The local network that was defined in the rule (by putting a subnet destination in the network rule) was 10.254.95.192/27. However, on the client’s side, he was seeing the request coming from 10.254.64.0/19. In order to create the IPsec tunnel, both the local and remote networks on each end of the tunnel must be identical, but switched (i.e. my local is his remote, and his local is my remote). Needless to say, this 10.254.64.0/27 was screwing everything up. However, when I connected to the original IP that worked, it was sending the correct network of 10.254.95.192/27.

Of course, no where in ISA 2006’s logging can you see it making the IKE requests. All I could see is that requests were being routed correctly from the internal ISA to the external ISA, and then from the external out to the correct network for the customer VPN. In essence, traffic was going in to a black hole. I could also see that the VPN connection (Main Mode) was up and running. I was completely reliant on the customer to let me know what was coming down the pipe to him. That right there is not really something I’m comfortable with, but he seemed to be OK with it. I’m sure it’s because he knew it wasn’t on his end, but on mine.

After deleting the VPN multiple times and recreating it to no avail, restarting the machine, etc, I knew that I would have to get some help from someplace else. Thankfully we have an awesome community of people at work that I could bounce ideas off of. Unfortunately, I never received a response. Also, ISAServer.org is a great place to get information. They have forums there that people keep an eye on. Unfortunately, ISA 2006 is still quite new and not as many people deal with it. I also did not receive a response from there. Needless to say, I was on my own for this one. Not a place I really wanted to be, since I thought I was at the end of my ability.

Actually, the IP isn’t really done in ISA server at all. Much like everything else that ISA server does, it’s just an application that sits on top of the OS and utilizes things that are already built into the OS (in my case Windows 2003 R2). This means that all IP policies, rules, etc are done by Windows and this can be monitored using the IP Security Monitor MMC Snap-in.

Since the VPN tunnel was being created successfully, I knew that Main Mode IKE Policies were correct, it was the Quick Mode policies that were causing me grief. Since we have multiple VPN connections terminating on this firewall, there are a lot of Quick Mode IP policies in place. Especially since all of them use pre-shared keys, which require that two IP policies are created, one for inbound and outbound (otherwise you can have one policy that does both inbound and outbound).

Scanning through the policies I was able to find the inbound and outbound policies for the original customer IP address to the 10.254.95.192/27 network, but I wasn’t able to find it for the new customer IP address. Alas, the problem! The next best policy for the new IP address was for the 10.254.64.0/19, since this policy encompasses the 10.254.95.192/27 subnet. Finally, I felt like I was making progress. Unfortunately, ISA should have been creating these policies when I edit the customer VPN networks. Actually, I still have no idea why ISA isn’t creating these policies. This is why I think there’s a bug which I’m going to submit to Microsoft (via this post actually).

Now that I knew the source of the problem, I had to fix it. Some days diagnosing the problems take longer than fixing them, and some days it’s the other way around. Since it had already taken me about a day to find the problem, I hoped that it wouldn’t take that long to actually fix it.

Needless to say, you can’t add IP policies from the IP Security Monitor MMC Snap-in, because, well, it’s a Monitor not an editor. The IP Policy Manager MMC Snap-in was no use either, as it defines computer level policies. Doh. Well, I can finally say that one of my certifications actually came in handy. That “+ Security” portion of my MCSE gave me the knowledge that there is a way to edit IP policies from the command line. Going on this, a quick Google search gave me exactly what I was searching for. Now which command to actually use?

At first I tried to just create a filter. However, I didn’t know of any filterlist, and none of the current filters were a member of a filterlist. Thankfully you can just make up a name and it creates on. Unfortunately this didn’t solve anything. Nothing showed up in the Quick Mode filters. Lets try again, yeah?

Turns out it’s not a static setting, but a dynamic setting, which makes more sense. Anyways, you can add Quick Mode rules pretty much the same. In that I mean, the command is just as long and gross. Just be aware, that since I wanted to add a Quick Mode rule and not a Main Mode rule, I had to put in the Quick Mode Policy variable.

Another thing that made this so confusing was that in IP Monitor, they are called Quick Mode Filters and at the command line they’re called Rules. Ugh. At least it’s taken care of. And now I think I know more than I ever wanted to about ISA and IP.

Well, I have another acronym to stick at the end of my name, NACA.