Stress testing by end users

Performance testing is an expensive exercise. It takes specialist skills and a significant amount of planning and understanding of the application in order to replicate the hundreds or thousands of users you are expecting. Another option of course is to let your end users do the performance testing for you. Take the announcement from Blizzard below for Diablo III.

Beginning this Friday everyone is invited to log in and help us put the game and servers through their paces in this three day stress test as we march toward the game’s release.

It is really viable to get your end users to stress test your application?

I am not sure how viable it truly is, though it appears to be getting more popular of late. One other example is ‘Things’, a popular task manager for the Mac and iOS devices .

Since we announced the beta, more than 35,000 people have signed up to become test pilots. Over the past months, we’ve gradually been inviting more and more people. […] Some of our users have adopted the beta entirely and created exceptionally large databases. In the coming weeks we are going to work out some kinks and performance bottlenecks related to such large databases.

In many ways, this makes a lot of sense. Why spend countless hours simulating your users activities when you can just let them do it themselves. I think this approach works well for companys that have a loyal and enthusiastic fan base. People are so excited to play a Blizzard game, or get access to cloud syncing on a tool that they will suffer through any issues without much complaining. After all, here is Blizzards answer to that very question.

What if the service is down/laggy/disconnecting me?

It’s very possible that players connecting to the stress test could experience issues with the service. While not ideal, this is exactly why we’re having a stress test. We want to catch and analyze as many bugs as possible during this stress test period, so that we can try to ensure a smooth launch

In reality, there are very few companies with a loyal enough fan base that will put up with a slow or interrupted service just to be one of the first to use a new product or feature. More importantly, there are not many brands that would be happy to risk their image by doing so.

Stress testing by end users it seems is then out of reach for most organisations. This is most likely not helped by the media attention that occurs when it goes wrong, like when the Australian Government launched the MySchool website and it was brought down an hour after the launch.

Inevitably, it all comes down to risk. What are the risks of the site not being able to handle the load generated by our users? What is the impact if it is the public that finds the breaking point? Finally, how will they react when they find the breaking point?

I am yet to come across a project where it’s acceptable to let the public stress test the application like Blizzard, though maybe I just haven’t worked for a company as fun as Blizzard yet.