Speed up SharePoint spin-up and stsadm execution time by Jeroen Ritmeijer

Here is the lifesaver of the month! Jeroen’s post was sent to me by Cedric Carrette (Thanks [:D]).


Original post content;


Ever since SharePoint 2007 was introduced I have been really disappointed with the 2 minute spin-up time of the sites and the 30 second wait when launching STSADM.

How could Microsoft have released this? I thought, What is wrong with these people? Why is there no uproar in the SharePoint Community at large?

As it turns out this problem does not occur on all SharePoint installations, it only happens under a certain combination of circumstances. Fortunately I have identified the root cause as well as a solution.

Symptoms for STSADM:


  • You start STSADM without any parameters
  • There is a delay of about 30 seconds
  • While you are waiting, and tearing your hair out because your deployment script has about 60 STSADM commands, there is no CPU activity, swapping or significant network traffic. 

Symptoms for SharePoint sites:


  • You make the first request of the day, the first request after yet another SharePoint crash or the first request after recycling the app pool because you are developing assemblies that sit in the GAC.
  • There is a delay of about 2 minutes
  • While you are waiting, and tearing your remaining hair out because you know you have to do this at least 50 times today as you are trying to work your way around the various event receiver bugs and limitations, there is no CPU activity, swapping or significant network traffic.

So, what is going on here? Quite a few, but not that many, SharePoint developers are complaining about this on the Internet, but no-one has a real solution. Is there a solution or should I start looking for a new job?

After yet another night of Googling around I found the solution in a posting about SQL Server 2005, which appears under certain circumstances to suffer from the same problem as SharePoint 2007. (Note that in my particular situation the problem is caused by SharePoint being slow, not SQL Server.)

The problem is that when loading signed assemblies the .net Framework checks the Internet based certificate revocation list. As our servers have, like most secure environments,  no outgoing connections to the public Internet the connection to crl.microsoft.com times out after what appears to be 30 seconds. It probably does this a couple of times in succession, causing a 2 minute wait when spinning up SharePoint.

After the timeout the assembly is still loaded and the software works as expected, though very slow every time a new signed assembly is loaded for the first time, which happens a lot. The worst thing is that no entries are written to the event log and no exceptions are thrown so you are left completely in the dark about why your application is so bloody slow.

There are a couple of workarounds, which one works best is for you to decide:


  1. Add crl.microsoft.com to your hosts file and point it to your local machine. Some people have reported success with this, but it didn’t work for me.
  2. Allow your servers to directly connect to crl.microsoft.com. If your environment dictates the use of a proxy server, configure it using proxycfg.
  3. Disable the CRL check by modifying the registry for all user accounts that use STSADM and all service accounts used by SharePoint. Find yourself a group policy wizard to help you out or manually modify the registry:

    [HKEY_USERS\<userid>\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing]
    “State”=dword:00023e00

     
  4. Download the CRLs and add them to the server manually (I haven’t tested this, but it may work):
     


    1. Download: http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl
      http://crl.microsoft.com/pki/crl/products/CodeSignPCA2.crl
       
    2. Add them:
      certutil -addstore CA CodeSignPCA.crl
      certutil -addstore CA CodeSignPCA2.crl

We decided to go for Option 3 (disable CRL check) and life is good again….. well as good as it gets when you are doing SharePoint development.

Update – VBScript to apply registry change:

The following script applies the registry change to all users on a server. This will solve the spin-up time for the service accounts, interactive users and new users.

const HKEY_USERS = &H80000003
strComputer = “.”
Set objReg=GetObject(“winmgmts:{impersonationLevel=impersonate}!\\” _
   & strComputer & “\root\default:StdRegProv”)
strKeyPath = “”
objReg.EnumKey HKEY_USERS, strKeyPath, arrSubKeys
strKeyPath = “\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing”
For Each subkey In arrSubKeys
    objReg.SetDWORDValue HKEY_USERS, subkey & strKeyPath, “State”, 146944
Next


With thanks to Nik Shaw for the script.

Relevant links:

10 thoughts on “Speed up SharePoint spin-up and stsadm execution time by Jeroen Ritmeijer

  1. Hi dirk,

    like you mention it, it is relatively difficult to get anything logged about that issue but… the .Net stack can help you, if the problem occurs, you’ll notice these lines:
    cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x12f
    cryptnet!CryptRetrieveObjectByUrlW+0x9b
    cryptnet!RetrieveObjectByUrlValidForSubject+0x5b
    cryptnet!RetrieveTimeValidObjectByUrl+0xbc
    cryptnet!CTVOAgent::GetTimeValidObjectByUrl+0xc2
    cryptnet!CTVOAgent::GetTimeValidObject+0x2f1
    cryptnet!CrlFromCertGetTimeValidObject+0x2d
    cryptnet!CryptGetTimeValidObject+0x58
    cryptnet!GetTimeValidCrl+0x1e0
    cryptnet!GetBaseCrl+0x34
    cryptnet!MicrosoftCertDllVerifyRevocation+0x128

    The reasons why there are very few complaints are usually:
    Many server are allowed to communicate with the Internet (!)
    Many users are used to Asp.Net slow start-up

  2. Hello !!!! 🙂

    I am Piter Kokoniz. Just want to tell, that I like your blog very much!

    And want to ask you: will you continue to post in this blog in future?

    Sorry for my bad english:)

    Thank you!

    Your Piter Kokoniz, from Latvia

  3. I’m so grateful for this article. This really saves me over an hour per day.

    If only I’ve found this a year ago… 🙁

  4. this is a fantastic posting, after installin some additional solution components into a MOSS deployment, we were experiencing timeout errors after each iisreset or app pool recycle i.e. every day….

    this posting helped to resolve those isues

  5. Pingback: One Bored Admin

  6. This was a great fix for 2007. The script or editing the registry doesnt seem to work for SharePoint 2010. I know stsadm command is different, does anyone know of a fix for 2010 stsadm command speed time? On servers with a slow interent connection they seem to take a long time. Has anyone else come across this or a fix for SharePoint 2010 stsadm commands?

    Thanks,
    Tony

Leave a Reply

Your email address will not be published. Required fields are marked *