Load Testing SharePoint (Lessons Learned) (Part 1)

I want to share some experiences from a recent load testing exercise I performed with SharePoint.

I am working on a web content management project that requires A LOT of customizations to SharePoint. The team needed to understand how these customizations would perform and scale. So I dusted off my limited knowledge of load testing with Visual Studio Team Test and went to work.

First I established my testing environment. I started by setting up my test agent. I was able to commandeer an old R&D server (HP 385, Dual Pentium III, 2 GB RAM, 1 GB NIC) for my cause. I wasn't sure, but suspected that this would be more than enough machine for my needs (oh yeah, its business time). Setup was a breeze, I just installed Visual Studio Team Test 2008 and my load agent was ready.

Next I needed to prepare my SharePoint environment. I already had a virtualized (is that a word) environment for SharePoint. Actually I had two to choose from, one dedicated to integration testing for the development team and another for user acceptance testing. I decided to go with the integration test environment because it was located inside of the corporate network (no messy firewalls).

Next I needed to create my web tests. This step required me to put on a thinking cap. There are many different approaches for load testing and I am not going to discuss them in this article. I chose to create tests that would isolate specific pieces of functionality (rather than trying to mimic usage patterns).  By using this approach I could clearly understand how our customizations would compare to out of the box SharePoint.

Here is a list of my tests

ID	Test	Description
1	HTML page with Hello World text	This is not a SharePoint item.  I created it so I could have an overall for the test environment. The results from this test represent the fastest the server can go.  This will act as a high water mark for me to compare my results to.
2	SharePoint Publishing Article Page	This is an out of the box SharePoint item. The results from this test represent a baseline for me to compare my test results against.  My tests results should be in the same neighborhood as these test results.
3	Our Customized SharePoint Page Layout with one customized control.	This represents a very lightweight version of our customizations.
4	Our Customized SharePoint Page Layout with two customized controls.	The idea is to add a small piece of functionality so I can understand how the change impacts the test results.

I ran each one of those tests independently.  The idea was to take a single test and stress the system.

I used goal based load tests in VSTT (a new feature for VS.Net 2008).  My goal was to stress the Web Server so the CPU would run between 70 - 90%.  Goal based testing is why I decided to avoid the firewalls (Goal based testing requires that the load agent communicate with the performance counters on the target machines). 

VSTT will continue to load transactions until the goal is reached.  If the goal is never reached then VSTT will load to the maximum number of requests/users specified in the load test.

Now I had everything ready to go... or so I thought.  In Part 2 I will go into detail about tweaks I had to make to my SharePoint environment so I could get the load tests to work correctly.

SPTD (no its not a venereal disease, but can cause hives)

Test Data Load Tool is a nifty little utility for creating data in your WSS or SharePoint site. You can download the tool at CodePlex http://www.codeplex.com/sptdatapop.

The tool provides a repeatable way to create test data for your SharePoint sites. It is extremely cool and even with the warts it is worth the time to sit and learn it. If you are a tester for SharePoint solutions you need to add this one to your bag of tricks. If you are a developer you will appreciate the base framework and potential.

The tool comes in two flavors (WSSDW.exe) and (MOSSDW.exe). They are in separate download packages so if you want to load data for things like SSO, publishing portals, audiences (SharePoint features) you will need to get both downloads. The tools are fairly easy to grasp, just take some time and go through the help files and samples.

That said the tools are buggy. Here are some things I learned the hard way.

1. The Delete option works about half the time... I finally quite using it and went another route. I'm sure the code is working as intended, but when you say delete you expect everything to go. Perhaps a -force option is needed with the delete.



2. ModifyExisting depends on "ModifyOne" interface being implemented, if the interface is not implemented no updates are made. It is extremely frustrating when you want to do something simple (ex. change the title of a web) and discover that the Webs class does not do anything in ModifyOne.



3. Pages have to be checked out to modify them (this may make you go Duh, but I was hoping to not have to worry about things like checkin/checkout/publish since it is a test tool). I ended up having 3 statements per page (one with OpenorCreate, one with checkout, one with publish).



4. Don't expect your script to work the first (or second, or third) time. Plan on having plenty of time to work through issues as you learn about the subtle nuances of the tool. It took me a good 2 days to get the kinks worked out of my script, but once I had it done I could bounce from environment to environment and setup a site that was test worthy (Seinfeld reference).


5. Depending on the amount / type of work your script needs to do it can take a while to run the script. I uploaded about 200 images and documents. The first run took about 30 minutes to do the upload, but every run after that was < 1 minute.
   

Zune Wiener

I was one of the lucky Zune winners at SPC2008. Yeah! I never win jack so I'm in shock.

SPC 2008 Day 4

This was the final day of SPC 2008. It was a great conference and I'm really glad I came.

The day started with a session from Russ White in PSS. He was talking about how MSFT PSS troubleshoots SharePoint problems. The title was a little misleading since Russ spent about half of the talk providing best practices for capturing information about the problem (nothing to do with SharePoint). The actual SharePoint data was a little lite. Would have been nice to focus on SharePoint.

Next I went to Tyler Butlers presentation on deploying solutions and content throughout the farm. Tyler gave some solid practical advice for the content deployment tools. He also showed some information about known problems with the tools. He mentioned that in the next 90 days an update that addresses several more fixes should be coming out.

The final session was presented by Andy Hopkins and Paul Learning. They talked about their findings from creating a highly scalable SharePoint solution. The information they shared was priceless. I'm so happy that MSFT made the investment to really find out what SharePoint can do. This made me feel much better about what we are doing.

SPC 2008 Day 3

We'll today was chock full of activities. It started with an excellent keynote from Greg Lemond. His story was extremely interesting. And finishedwith a crazy night of XBox games.

The sessions were not as great today (which doesn't mean they were not good, it means I didn't learn as much as I did on day 2).

The Branding sessions from Heather Solomon opened my eyes to the fact that there are no short cuts in learning to brand SharePoint. Takes a lot of time and patience to dig through the CSS. At least I know where to start digging.

Steve Peschka presented some some really interesting LINQ code that works with SharePoint. The demo was really interesting, but I have some concerns about performance. Hopefully MSFT is working on a LINQ provider for SharePoint. What I saw would work for some situations, but CAML is still the preferred query language. Steve also showed a demo that hooks AJAX into SharePoint. I actually saw a similar demo on Monday. Let's just say MSFT has a big opporutinity to make AJAX (and Silverlight) easier to integrate with SharePoint.

Ted Pattison talked about SharePoint deployments with Solutions. He did a great job covering the basics. He also showed everyone a new tool (STSDEV) that greatly reduces the amount of time needed to create a SharePoint solution.

STSADM Find Tip

Got a great tip from Shane Young @ the Microsoft SharePoint 2008 conference.

Basicly you can pipe the output of stsadm to find command to filter the list.

stsadm -o find "exec"

This is great! I can't tell you how many times I've poured through pages of help output to find commands.

Thanks Shane!

SPC 2008 Day 2

The sessions I went to today were excellent.

The day started with a session by Ben Robb on automating build process of SharePoint solutions. Ben's build routine actually generated an install.bat file that contained the proper STSADM commands. This is a really interesting idea.

Next I attend an excellent session by James Petrosky about Performance / Optimization Strategies for MOSS. I learned about the 3 types of cache options (very similar to existing Content Management Server cache options) and how to tune them.

Next I got to check out a session by Joel Olsen and Shane Young. This was another excellent session with the main takeaway for me being avoid Content Deployment (they didn't actually say that, but came very close). This is critical for me because content deployment is critical to my companies SharePoint implementation. We could potentially be pushing between 1 - 22 GB of content a day. Needless to say I'm convinced that Content Deployment is not right for us.

Finally I got to listen to Sean Squires and David Johnson talk about how Microsoft IT implemented internal portals with SharePoint 2007. Really informative information with a nice sprinkling of lessons learned. They do a LOT with the content query web part.

The night finished up with a party at the Museum of Flight.

SPC 2008 Day 1

I'm at the SharePoint 2008 conference in Seattle. I'm excited to be here because I have been looking for opportunities to connect with the larger SharePoint community. I'm not sure how much technical details I will get out of the conference, but there is a lot of energy and excitement.

The first day started out with a great keynote from Bill Gates. His enegy for technology and how it can drive business value is incredible. I was also really impressed with his passion about the Bill Gates Foundation. It was my forth time seeing him in person and he seemed the happiest of all those speeches.

On the negative side MSFT should have done a better job limiting the number of people at the conference. The conference is sold out, but frankly it is oversold. I have already been turned away from one of the sessions I wanted to attend. Second a few of the sessions I have been too didn't really connect back to SharePoint. For example I went to a talk about using AD FS with SharePoint, but there was really no discussion about hooking AD FS with SharePoint. Another example was a session that was geared towards Extranets, but turned into a 60 minute sales speech.

I ended the day with a great talk about how to get find resources for starting a user group. This is something I am seriously considering so I really enjoyed the info.

Hello World

We'll I've been telling myself to start contributing to the SharePont community. So here it is...