SharePoint 2010 User Profile Sync: stopped-extension-dll-exception
Well, it's good to see that User Profile Sync can be better in 2010 than it was in 2007. However, there are definitely some issues still.
The first one, which is something we just noticed was that the User Profile Sync jobs were constantly failing. Unfortunately, there isn't really a good way to know without going into the MIISClient program to look at the errors. Basically, if you think, for whatever reason, profile sync is not working, open up the MIISClient.exe (Program Files\Microsoft Office Servers\14.0\14.0\Synchronization Service\UIShell) as the farm account and take a look to see if everything is a success.
For us, we were seeing all the MOSS-{guid} jobs failing with the error stopped-extension-dll-exception as you can see below.
Based on the lovely error message, I'm still amazed that I was able to isolate this issue (event logs reported that CA was being accessed via a non-registered name). However, it turns out it is because of alternate access mappings (AAMs) for the central admin (CA) website. Normally, SharePoint sets up the AAM for CA as the machine name you first install SharePoint on to. However, we changed the AAM to be a more friendly name.
When you update the "Public URL for Zone" for the CA website, it does not propagate the change into the MIISClient. This causes the MIISClient to not correctly access the CA APIs for the user profile sync (or at least I am imagining this is the case).
Fix it with the following steps:
- MIISClient.exe as the farm account.
- Tools > Management Agents (or click the Management Agents in the bar)
- Right-click on the MOSS-{guid} management agent and select Properties
- Go to the Configure Connection Information section in the left-hand pane
- In the connection information box, change the Connect To URL to be the same URL as listed as the "Public URL for Zone" for your CA in the AAM configuration.
- Re-enter the farm account username and password for good measure
- Save the configuration
- Run a full profile sync from CA
New Home Server Setup
I've been meaning to do this for awhile, but I haven't found a suitable replacement until recently. I am decommissioning the server at home. It's loud, large, and sucks down a lot of power for what I use it for (windows home server). It was nice because I could quickly and easily spin up some VMs and poke around, but I'll still be able to do that.
Instead, I picked up a Synology DS1511+ NAS. This little appliance is pretty darn slick. It can pretty much do everything I was doing, in a smaller, quieter, and cooler form factor. Since it uses an Atom processor, it runs a fairly familiar flavor of Linux, so you can do quite a bit with it. Plus, a lot of the default stuff it comes with is quite nice!
I'll be throwing up a few copy/pastes on the site so that I can quickly re-reference. Oh, and there's another SharePoint article in the works too. Busy, busy!
SharePoint 2010 Synthetic File Data
Still trying to work through creating synthetic data for an out-of-the-box SharePoint performance test. To create the data, create a new site collection (so it doesn't interfere with anything else and is easy to clean up), and uploads all the test data. The biggest downside right now is that the data is created and then uploaded, which requires enough disk space to make the data. Not a huge issue for me, but possibly for you.
General idea came from a few places for the upload, and locally for the file creation.
#USER Defined Variables
#Specify the extension type of files you want uploaded
$strDocTypes = @(".docx",".xlsx", ".pptx", ".pdf")
#The max amount of data generated in MB
$maxSize = 50
#The max size one file could be in MB
$maxFileSize = 10
#Intermediate folder where the test data is placed
$fileSource = "F:\TestData"
#New Content Database (for easy removal)
$dbName = "Portal_ContentDB2"
#New Site collection template
$template = "SPSPORTAL#0"
#Account owner
$siteOwner = "TEST\Administrator"
#Web Application address
$webApp = "https://portal"
#Site Collection Address
$siteCollection = "/sites/1"
# DO not edit anything beyond this line
#Create all the test data using FSUTIL
$rand = New-Object system.random
do {
$guid = [guid]::NewGuid()
$guid = $guid.ToString()
$fileName = $guid+$strDocTypes[$rand.next(0,$strDocTypes.length)]
$rand1 = $rand.nextdouble()
$rand2 = $rand.nextdouble()
$rand3 = $rand.nextdouble()
[int]$fileSize = 1048576*$rand1*$rand2*$rand3*$maxFileSize
FSUTIL FILE CREATENEW $fileName $fileSize
$fileTotalBytes = $fileTotalBytes + $fileSize
$fileTotal = $fileTotalBytes/1024/1024
}
#Data generation keeps going until the amount of data is > $maxSize
while ($fileTotal -le $maxSize)
#Creation of the new content database and site collection
$siteCollectionURL = $webApp + $siteCollection
New-SPContentDatabase $dbName -WebApplication $webApp
New-SPSite -url $siteCollectionURL -OwnerAlias $siteOwner -Name "Test Doc Library" -Template $template -ContentDatabase $dbName
#uploading of all the generated data into the $siteCollectionURL/Documents folder
$spWeb = Get-SPWeb -Identity $siteCollectionURL
$listTemplate = [Microsoft.SharePoint.SPListTemplateType]::DocumentLibrary
$spFolder = $spWeb.GetFolder("Documents")
$spFileCollection = $spFolder.Files
Get-ChildItem $fileSource | ForEach {
$spFileCollection.Add("Documents/$($_.Name)",$_.OpenRead(),$true)
}
SharePoint 2010 Load Testing Kit
Was looking for ways to generate synthetic test data for a SharePoint out-of-the-box install today, and ran into the SharePoint 2010 Load Testing Kit. While it doesn't help me in this stage of the project, I could see it being useful later or on other projects.
There appears to be a lot of dependencies though:
- Migration from 2007 to 2010
- As it collects info from your log files, you'll need to have everything migrated for the scripts to work
- Data
- Apps
- Site Collections
- Etc.
Could be hot though!
Search Schedule Script
To setup the crawl configuration for the default local sites, you can use the script below:
$ssaName="Search Service Application" $context=[Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($ssaName) $incremental=New-Object Microsoft.Office.Server.Search.Administration.DailySchedule($context) $incremental.BeginDay="23" $incremental.BeginMonth="10" $incremental.BeginYear="2011" $incremental.StartHour="0" $incremental.StartMinute="00" $incremental.DaysInterval="1" $incremental.RepeatInterval="720" $incremental.RepeatDuration="1440" $fullCrawl=New-Object Microsoft.Office.Server.Search.Administration.WeeklySchedule($context) $fullCrawl.BeginDay="23" $fullCrawl.BeginMonth="10" $fullCrawl.BeginYear="2011" $fullCrawl.StartHour="6" $fullCrawl.StartMinute="00" $fullCrawl.WeeksInterval="1" $contentsource = Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $ssaName -Identity "Local SharePoint Sites" $contentsource.IncrementalCrawlSchedule=$incremental $contentsource.FullCrawlSchedule=$fullCrawl $contentsource.Update()
SQL Server Issues
Last week I was beating my head against the table, because a VM I had quickly created wasn't allowing SQL to install. I kept receiving the following error in the detailed SQL error log:
Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.
External component has thrown an exception.
The configuration failure category of current exception is ConfigurationFailure
Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.
System.Runtime.InteropServices.SEHException: External component has thrown an exception.
at Microsoft.Win32.SafeNativeMethods.CloseHandle(IntPtr handle)
at System.Runtime.InteropServices.SafeHandle.InternalDispose()
at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
at System.Diagnostics.Process.Close()
at System.Diagnostics.Process.Dispose(Boolean disposing)
at System.ComponentModel.Component.Dispose()
at Microsoft.SqlServer.Configuration.SqlEngine.SqlServerServiceBase.WaitSqlServerStart(Process processSql)
at Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineDBStartConfig.ConfigSQLServerSystemDatabases(EffectiveProperties properties, Boolean isConfiguringTemplateDBs, Boolean useInstallInputs)
at Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineDBStartConfig.DoCommonDBStartConfig(ConfigActionTiming timing)
at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.ExecuteAction(String actionId)
at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.Execute(String actionId, TextWriter errorStream)
Exception: System.Runtime.InteropServices.SEHException.
Source: System.
Message: External component has thrown an exception.
It turns out that I accidentally downloaded the debug check build version of Windows 2008 R2 SP1, and well, you can't install SQL with that version. Needless to say, the error message makes this completely obvious. Found the hint to look at the ISO I was using on MSDN social.
LAFHA Math
Just throwing this down as finding this math or an explanation of it has eluded us until we actually could do it based on paystubs. LAFHA is an in/out transaction (deduction and then addition).
- Monthly Taxable Income = Monthly Gross - LAFHA
- Figure Taxes based on Monthly Taxable Income
- Monthly Net Pay = Monthly Taxable Income - Taxes + LAFHA
Net.TCP, IIS7, and Classic AppPools
With the application that I'm helping "make fast" one of the optimizations identified by an architect was to use net.pipe or net.tcp on the application tier. This was because the services on that tier call into each other, and waste a lot of time doing all the encapsulation that comes with wsHttpBinding.
We had tried to first use net.pipe because it is incredibly fast and it is all local. However, because of how they do security here, it didn't work. Next up, net.tcp.
Overall, it wasn't that difficult to setup the services with dual bindings (wsHttpBinding for calls from other servers and net.tcp for calls originating from the same server). Granted, there were a lot of changes, and since all the configs are manually done here, was very error-prone. I will be glad not to do any detailed configurations for a while.
Anyways, we did our testing of the build that incorporated it through 3 different environments, and over the weekend it went into Production. Of course, that is when the issues started.
Bright and early Monday morning, users were presented with a nice 500 error after they logged into the application. On the App tier servers we were getting the following error in the Application event log with every request to the services:
Log Name: Application Source: ASP.NET 4.0.30319.0 Event ID: 1088 Task Category: None Level: Error Keywords: Classic User: N/A Description: Failed to execute request because the App-Domain could not be created. Error: 0x8000ffff Catastrophic failure
Well, since it mentioned the App Domain, we ran an IISReset and all was well. I didn't think too much into it at the time and only did some cursory searching as it was the first time we saw it. However, today it happened again.
Our app is consistently used between 7AM and about 7PM, but during the night it isn't used at all. This is when the appPool is scheduled to be recycled (3AM). It had appeared as if the recycle was what was killing us, as only the App Domain is recycled and not the complete worker process.
Immediately we had the guys here remove the nightly recycle and change it to a full IISReset. At least that way we had a workaround until we could determine the actual root cause, come up with a fix, and test said fix. However, it didn't take long to determine the actual root cause...
One of the interesting things about this issue, was after it started happening in Production, a few of the other environments started exhibiting similar symptoms: Prod-Like and a smaller, sandbox environment. Mind you, neither of these environments had these issues during testing.
So, I took some time to dive in and actually figure out the problem. At first I thought it was because there were some metabase misconfigurations in these environments. I wouldn't say that all of these environments were pristine, nor consistent between each other. I found a few things, but nothing really stood out...until...
While I was doing diffs against the various applicationHost.config files, NotePad++ told me it had been edited and needed to be refreshed (them removing the appPool recycles). However, as soon as this happened the 500 errors started. It didn't help they did both machines at the same time which took down the whole application, but that's another story.
This led me to believe that it wasn't something within the configuration, but it also showed me how to reproduce it, at least part way. The part that I was missing was that I had to first hit the website to invoke the services and then change the configs causing an app domain recycle.
Then I attempted to connect to the worker process with windbg to see what it was doing. However, that was a complete failure as nothing actually happened to the process. No exceptions being thrown, no threads stuck, etc. It appeared to just sit there.
A bit of searching later led me to an article that had the exact same issues we were having, and that changing from a Classic to Integrated appPool fixed it. However, it didn't mention why. Of course I tried it and it worked. To appease the customer's inquiries I knew I needed to find out why though.
I still don't have a great solution, but apparently net.tcp and WAS activation has to be done in Integrated mode. If it isn't, you get the 500 error. But ours works fine until the app domain is recycled. Well, according to SantoshOnline, "if you are using netTcpBinding along with wsHttpBinding on IIS7 with application pool running in Classic Mode, you may notice the 'Server Application Unavailable' errors. This happens if the first request to the application pool is served for a request coming over netTcpBinding. However if the first request for the application pool comes for an http resource served by .net framework, you will not notice this issue."
That would've been nice to know from Microsoft's article on it, or at least a few more details. I remember reading an article about the differences between Integrated and Classic, but I sure don't remember anything specific to this.
Anyways, hope this helps someone who runs into the same issue...
Windows 2008 Performance Alerts
This may seem silly to some of you, but I am still getting used to Windows 2008. Sadly, I don't spend as much time actually administering servers as I used to (silly management), so it usually takes me a bit longer to make my way around 2008 than 2003. I like to think they made everything more complex, but for some reason I'm sure I'll get booed about that.
Anyways, this morning I was attempting to setup some performance alerts on some servers we're having issues with. Basically I wanted to have it email us when it reached a certain threshold. No big deal, thinking I had this, I created the email app, created a performance counter, and then manually added it in.
Needless to say that didn't work. It took me awhile to figure out why too as my little email utility worked fine. So I began a new search in order to find out how stupid I was being.
Turns out, quite a lot of stupid. Instead of using the utility, you can now use scheduled task items...which includes an email action! I basically used the instructions over at Sam Martin's blog, which, I may add, he posted about in April of this year. I'm not the only n00b. Plus, who doesn't have an enterprise system that deals with this sort of stuff already (at least at the types of clients I work with)?
Perfmon
- Open up perfmon
- Create a new User Defined Data Collector Set
- Choose to create manually after naming
- Select Performance Counter Alert
- Add in the performance counter you care about (mine was requests executing in asp.net apps 4.0)
- Choose the user to run it as
- Edit the Data Collector in the Data collector set
- Change the sample interval to whatever works for you (I set mine to 60s so we can be on top of issues prior to the users)
- Under Alert task, give it a name (e.g. EmailAlert) and give it a task argument (you can combine them to form a sentence like "the value at {date} was {value}"
- Start the Data Collector Set
- Open up scheduled tasks
- Create a task, not a basic task
- Name it the exact same name you did in step 9 above (i.e. EmailAlert)
- Check "user is logged in or not" so that it runs all the time
- Create a new email action under the Action tab
- Enter all the info for from, to, subject, etc. To send to multiple people, comma separate the addresses.
- For the body, type whatever you want, and then $(Arg0) will pass the task argument you made in step 9 above.
- Enter the SMTP server.
Done!
Since the performance counter was set to an application pool, whenever that pool disappears (IISReset, idle timeout, etc.) the counter stops.
Currently Reading (could take awhile): 
Dummy Files
We are doing some document uploading to SharePoint, and needed some test files of various sizes. If you have Visual Studio installed, you have the tools required to make these files. Just make sure you run as administrator, and use the following command.
FSUTIL FILE CREATENEW 100MBTest.mdb 104857600 Usage: FSUTIL FILE CREATENEW [Filename] [Size in bytes]