Unhandled Exceptions that can Crash your IIS Application Pool

Recently my team discovered some nastiness with unhandled exceptions inside our custom SharePoint code.  Specifically we found that unhandled exceptions inside of SharePoint.Publishing.LoginRunningOperationJob can result an IIS Application Pool crash. 

The reason is because this class puts the delegate code onto a separate thread that when aborted can leave the Application Pool in an unstable state.  Which can (and does) result in an Application Pool recycle (which is bad for very large SharePoint sites that take a few minutes to spin-up).

So you need to make sure that your delegate code is wrapped in try/catch and do NOT throw the error from inside your catch (same as unhandled exception).

Technorati Tags: SharePoint,Development,IIS

Exciting year to be in IT

I’ve been in IT for a little while now and I have to say that this is the most excited I’ve been since 2000.  Why, well because of all the great stuff Microsoft plans to ship this year. 

Here is a short list of what has me so excited.

Silverlight 4

I know I’m late to the Silverlight party since a lot of people felt like version 3 was a good product for developers.  Well, I’m late on purpose.  I remember taking a look at Silverlight 1 and 2 and thinking, hmmmm I wonder where this will go.  Nothing there for me to go back to business and say we have to take a hard look at this now.  With Silverlight 3 I finally started to see some real potential for the business, but I wanted see if the adoption rate would be good enough.

Now with Silverlight 4 getting ready to ship I finally feel comfortable standing up and saying lets take a hard look at Silverlight for business application development.

.Net 4

I consider this to really be the 3rd major release of the .Net Framework stack.  I guess I’m most excited about the new parallel features that are coming with this version of the framework.  But, what is more important is the fact that the framework continues to grow and get better. 

MVC 2

I believe MVC has shown that it is here to stay.  The latest improvements in MVC 2 have really addressed some of the rough edges that were in the MVC 1 release. 

I see some debates raging about MVC vs. WebForms.  Frankly I think the debates are a little silly as each technology has its niche. It reminds me of the old VB vs. C++ debates for doing Windows Forms development.  Although MVC web development is no where near as complicated as building Windows Forms applications in C++ :D.

Just noticed that @scottgu published an article about this very subject as I am finishing up this blog posting. 413 Graves Mill RoadIsn’t it ironic. :D

Visual Studio 2010

Visual Studio 2010, all I can say is wow.  Some people will think I’m full of it because on the surface it does not look like Visual Studio 2010 has a lot of improvements.  I agree that a lot of the improvements are in specific areas (ex. SharePoint development), but the new Extension Manager model should not be overlooked.

I’ve seen some work coming out of the SharePoint camps that are taking advantage of the new Extension Manager.  One great example is the work being done by Waldek Mastykarz.

The other thing that really has me excited is the new enhancements inside of Visual Studio 2010 Team System.  Last week I watched a Channel 9 video about the new Test Lab Manager.  The more I learn about these “little” enhancements the more I can vision software development teams increasing productivity and quality.  Good stuff if you are a manager of a software development team.

SharePoint 2010

I sort of saved the best for last in this case.  While SharePoint 2010 will not be taking advantage of a lot of the new technology from Microsoft (MVC 2, .Net 4) there are some new features coming that make development a much better experience.

The new SharePoint Tools for SharePoint 2010 are great.  While there is still room for improvement these show that Microsoft got the message about development experience with SharePoint.

I’m also really excited about the new client object model.  This makes connecting to SharePoint data from AJAX, Javascript and Silverlight a palatable experience. 

I am also really excited about the Developer Dashboard technology.  I got to see this very early on and I almost made a mess in my pants.  The reason is because I had just finished up going through a painstaking process of “find the bottleneck” with SharePoint. 

Finally I’m pumped about the new Services architecture.  This is the only major architecture change I can see in SharePoint 2010 (perhaps I am being myopic).  This is a good thing as I think the upgrade from 2003 to 2007 was a lot to chew on.  Anyway the new Services architecture shows a lot of promise for building new extensions to SharePoint.  As soon as I saw the new model I thought of 2 new services that could add value to anyone running Publishing sites. 

Conclusion

The team at Microsoft is getting ready to ship a lot of products this year.  Big hats off to everyone involved. 

Technorati Tags: .Net Framework,C#,Development,SharePoint,Silverlight

Tapping into the hidden SharePoint API using Reflection

Summary

The SharePoint API provides a very rich experience for software developers as almost everything that can be done with the SharePoint user interfaces can be done through the API.  One could argue that the SharePoint API has contributed to SharePoint’s overall success.

Anyone that has used Reflector to view the internal works of the SharePoint API knows that there is a lot of really interesting functionality that is marked away as private and not to be used by Joe Developer when customizing SharePoint.  Typically this is done by using the C# Internal keyword.  Even though I’m sure Microsoft had our best interest at heart when they locked away some of SharePoint API, sometimes you find something inside of it that you really want/need to use.

Well when you are in that situation (which should be very rarely) .Net Reflection becomes your friend and allows you to send messages to these classes even though they are supposed to be protected from your fingers.

Example: Field.SetFieldBoolValue

Okay, so a few months back I was trying to do some cleanup on some SharePoint Publishing sites.  One of the things I was cleaning was duplicate fields on the Pages list (how we got in this situation is subject for a different blog post).  Well the field I needed to delete was marked as Hidden so it could not be deleted.  Also since the Field definition marked the field as hidden I could NOT set the Hidden value to false. 

So I was looking at the SPField.Hidden property inside of Reflector and discovered that it uses something called SetFieldBoolValue to actually set the Hidden property.

A quick scan of that method reveled that it is an Internal method of the SPField class so I could not access it directly (again this is a good thing since Microsoft wants to protect me from myself).

Even though I respect that someone at Microsoft does not want me to use this field I really needed to toggle the Hidden property so I could delete that field.

So I used a little Reflection kung-fu to give me access to SetFieldBoolValue.

Type type = field.GetType();

MethodInfo mi = type.GetMethod("SetFieldBoolValue", BindingFlags.NonPublic | BindingFlags.Instance);

mi.Invoke(field, new object[] { "Hidden", false });

mi.Invoke(field, new object[] { "CanToggleHidden", true });

field.Update();

So if you are new to .Net Reflection take a look at this introduction.  What makes this work is the BindingFlags on the GetMethod routine.  So now I can use the MethodInfo to send a message that tells the field object to call the SetFieldBoolValue method.  Great Success!

Conclusion

By using .Net Reflection you can gain access to places in the SharePoint API that are marked as Internal and not to be accessed.  Typically you do NOT want to do this, but sometimes you may need to crack open the engine.  One of the best examples I have seen of this was done recently by a developer on our team.  He used reflection to gain access to the underlying SharePoint list collection to see if a list item had a field value.  This allowed us to really cut down on the number of exceptions thrown by our application.

Technorati Tags: SharePoint,C#,Reflection,.Net Framework

Internet Explorer Discussion Toolbar and SharePoint Publishing Sites.

Summary

The Internet Explorer Discussion Toolbar will probe your web site to see if it is using SharePoint (or Front Page Server Extensions). If it finds that you are using SharePoint then it will enabled the toolbars discussion feature which will most likely result in an Access Denied or some other error message to the user.

While this is a very minor thing you may want to consider blocking access to the URLs Internet Explorer Discussion Toolbar uses to determine if a site is using SharePoint. This can easily be done by using an ISAPI Filter and blocking traffic to /_vti_bin/owssvr.dll and /MSOffice/cltreq.asp.

Background

Recently my team launched some public facing SharePoint Publishing Sites and discovered a small issue with the Internet Explorer Discussion Toolbar. When we would browse our guest (anonymous) access URL we would be prompted for a login. We only were seeing it from certain test clients using Internet Explorer. By installing Fiddler on one of the test clients we could quickly see traffic going to the /_vti_bin/owssvr.dll which would return a HTTP 401 messaging indicating that the client was not authorized.

Below is some sample traffic I collected using Fiddler and http://sharepoint.microsoft.com. As you can see the 11th request (3rd line below) is a call to /_vti_bin/owssvr.dll.

With a little trial and error we were able to quickly figure out that this toolbar was generating those requests to /_vti_bin/owssvr.dll. I’m not an Internet Explorer Discussion Toolbar expert, but it appears to send that request every time a request is made to the server.

If it receives a 200 then it enables discussions for the page. Below is a screenshot from http://sharepoint.microsoft.com. The Discussion Toolbar is enabled and ready to go.

Just because the toolbar is enabled does not mean people will be able to attach comments to your web pages. I tried this and discovered that the toolbar will fail with an Access Denied error since it is trying to write to the SharePoint site collection.

How we stopped it.

During the testing I discovered that if the request to /_vti_bin/owssvr.dll fails then the toolbar will display a message stating that discussions are not allowed for this page. Below is a screenshot of the discussion toolbar disabled.

To stop this activity we used ISAPI_Rewrite to deny all requests going to /_vti_bin/owssvr.dll and /MSOffice/cltreq.asp through the IIS sites that support browsing (we have separate IIS site for content editing). We did NOT want to block traffic to /_vti_bin/owssvr.dll through our editing site because we were concerned it would break some of the Office Client integration features.

Technorati Tags: SharePoint,Publishing,Internet Explorer

SharePoint Saturday Charlotte

This past Saturday I had the privilege of presenting at SharePoint Saturday Charlotte Event along side some top talent in the SharePoint community.  It was great to finally meet some of the people I follow on Twitter (too many to name).  Also, big kudos to Dan Lewis @danlewisnet, Brian Gough and all of the #SPSCLT Volunteers, you guys/girls rock!!!

As promised here the slide deck from my talk about Performance Testing with SharePoint.  I was hoping for a little bigger turnout, but when I saw I was in the same time slot as Becky Isserman (@MossLover) and Laura Rogers (@WonderLaura) I knew I would be lucky to get 7 people. :)

I really enjoyed the sessions I got to attend. 

The day started off with a GREAT presentation from Phil Wicklund. Phil had lots of pragmatic advice for managing your SharePoint investment. 

Next, I learned a LOT about how MS has implemented Windows Azure from Rick Taylor (@slkrck).  Rick is a GREAT speaker and had lots of war stories (which I LOVE hearing). 

Then, I got to hear Mike Watson (@mikewat) talk about SharePoint hosting architectures.  He has some really good insight into what it takes to make SharePoint purr. Also, I got to hear that our hosting architecture for SharePoint is in line with what he thinks is the RIGHT way to do it.

Next, I listened into Dan User (@usher) talk about Taxonomies.  I felt much better about my Internet facing solution that will have about 30 Web Applications in one Farm after hearing what he is doing. 

Finally, I listened to Dan Attis (@jdattis) talk about a solution he recently wrapped up that used SharePoint lists to store data for a Web Interface that was not SharePoint.  It sounds like a really cool solution the folks at B&R built.  Also, I learned about Object Initializers in .Net 3.5, really cool stuff.

Definitely will not be my last SharePoint Saturday event.

Technorati Tags: SharePoint,SharePoint Saturday

Big Thanks to Office / SharePoint Teams for TAP Airlift

I had the privilege of attending the Office 14 TAP Air Lift this week in Seattle.  This is my second time coming out to to a SharePoint Air Lift and I must say they never disappoint.  While I cannot share any information from the Air Lift I can say it is exciting times to be working with SharePoint. 

I really want to extend a big thank you to Microsoft for hosting a great event.  The folks in Office / SharePoint development teams have some big deadlines in front of them.  For them to take time out of their busy schedules to spend some one on one time with customers says a LOT. 

More Lessons learned from Performance Testing SharePoint

Abstract

Performance testing with SharePoint, or any web based application, can be quite tricky.  Recently my team launched an upgraded Corporate Web Site based on SharePoint 2007.  The launch was quite challenging mainly due to mistakes made during performance testing Lessons Learned from Intranet Launch.

This post is dedicated to the lessons learned from the performance testing of Corporate Web Site. 

Background

Prior to launch we ran through our performance test scenarios 3 times.  Each time the output showed that we could scale way beyond the existing implantation of our Corporate Web Site (Referred to as Violin from here on). 

The performance test scenarios had been chosen based on traffic patterns and pages determined to be high risk for performance (This was good). 

Our key performance requirements stated that the web servers must support 38 page views / sec with response time < 5 sec (This was good).  This is a nice well defined requirement, although some could argue that 38 page views needs to be broken down into specific types of pages (ex. 10 home page views, 7 chapter page views, …). 

We also had a performance goal stating that processor utilization should not go above 80% on web servers for more than 5 seconds (This was good).

For the final test we replayed traffic from IIS logs that were taken during peak traffic window (when we received the most requests / sec).  This was a bit tricky because my Load Runner resource told me that this was not supported by Load Runner.  So he and I had to message the data inside the IIS logs to get it so Load Runner would support running the tests (this felt wrong at the time, but I cannot say if it is a mistake).

We used Load Runner (sorry I do not know version) for all of the performance tests.  The Load Runner clients were located within the same data center as our web servers, but they were on different network segments.

When we ran the tests we engaged several people from operations team (Network, Windows Server, SQL Server DBA and SharePoint Admin).  These people were tasked with monitoring components related to their area of expertise.  They were also required to collect performance statistics and report those back so they could be included in overall performance test report (This was good).

“Performance is exceptional” or The False Sense of Security

So each time we ran the tests we were able to reach levels of about 90 page views / sec on one server with avg. response time < 5 seconds (we have 4 load balanced WFE in our farm).  So we were hi-fiving and slapping each other on the back.  As far as we were concerned performance requirements were met, check them off we are done.

We did notice an occasional spike w/ CPU, but we were able to correlate this back to pages expiring in Output Cache.  So this was not a concern.

Well once we went live we discovered that something was gravely wrong.

So What Went Wrong

After going live we discovered that the output cache hit ratio was not aligned with the numbers we were seeing during performance testing.  So were were having a LOT less output cache hits.  This resulted in the servers having to do a lot more work than originally anticipated.

What could have happened? We thought we did everything right with the performance tests.  What went wrong?

Well after much soul searching (and re-reading basics of performance testing) it hit me.  "

Oh $hit we didn’t model user variations and think times. 

 

Does that really matter? 

Yeah it does, the reason is because we ran a high number of requests but the proportion of cached requests vs. un-cached requests was out of balance.  Had we have taken into consideration user think times and other variations(browser type, user location) we would have less hits against output cache.

Classic 101 Performance Testing Mistake.  Oh well, you pick yourself up, dust yourself off and vow not to make the same mistake again.

Lessons Learned Summary

1. Think times matter

User think times are critical when doing performance testing (especially for web applications that rely on ASP.Net Output Caching to meet performance goals).

2. End user variations matter

Just as important as think times you need to look at the IIS Logs (or your web analytics reports) to understand browser differences and local differences.  This is extremely critical if you have Output Cache configured so it treats these differences as non cached page requests.

3. Mix up the IP addresses to fool user affinity

While this is not as important as Think Times and End User variations it is important if you are doing performance testing through a load balancer configured with session affinity. 

All of the tests we ran looked like they were coming from 2 IPs.  While I cannot prove this invalidated the test results it looks like there was some sort of caching efficiencies realized somewhere in the stack (Switch, NIC, IIS, …). 

References

Microsoft Patterns and Practices: Performance Testing Guidance for Web Applications

Microsoft Office Server Online: Configure page output cache settings

MSDN: Output Caching and Cache Profiles

Technorati Tags: SharePoint,Performance Testing,Load Testing,Lessons Learned