Monday, February 28, 2005

Performance vs. load vs. stress testing

Here's a good interview question for a tester: how do you define performance/load/stress testing? Many times people use these terms interchangeably, but they have in fact quite different meanings. This post is a quick review of these concepts, based on my own experience, but also using definitions from testing literature -- in particular: "Testing computer software" by Kaner et al, "Software testing techniques" by Loveland et al, and "Testing applications on the Web" by Nguyen et al.

Update July 7th, 2005

From the referrer logs I see that this post comes up fairly often in Google searches. I'm updating it with a link to a later post I wrote called 'More on performance vs. load testing'.

Performance testing

The goal of performance testing is not to find bugs, but to eliminate bottlenecks and establish a baseline for future regression testing. To conduct performance testing is to engage in a carefully controlled process of measurement and analysis. Ideally, the software under test is already stable enough so that this process can proceed smoothly.

A clearly defined set of expectations is essential for meaningful performance testing. If you don't know where you want to go in terms of the performance of the system, then it matters little which direction you take (remember Alice and the Cheshire Cat?). For example, for a Web application, you need to know at least two things:
  • expected load in terms of concurrent users or HTTP connections
  • acceptable response time
Once you know where you want to be, you can start on your way there by constantly increasing the load on the system while looking for bottlenecks. To take again the example of a Web application, these bottlenecks can exist at multiple levels, and to pinpoint them you can use a variety of tools:
  • at the application level, developers can use profilers to spot inefficiencies in their code (for example poor search algorithms)
  • at the database level, developers and DBAs can use database-specific profilers and query optimizers
  • at the operating system level, system engineers can use utilities such as top, vmstat, iostat (on Unix-type systems) and PerfMon (on Windows) to monitor hardware resources such as CPU, memory, swap, disk I/O; specialized kernel monitoring software can also be used
  • at the network level, network engineers can use packet sniffers such as tcpdump, network protocol analyzers such as ethereal, and various utilities such as netstat, MRTG, ntop, mii-tool
From a testing point of view, the activities described above all take a white-box approach, where the system is inspected and monitored "from the inside out" and from a variety of angles. Measurements are taken and analyzed, and as a result, tuning is done.

However, testers also take a black-box approach in running the load tests against the system under test. For a Web application, testers will use tools that simulate concurrent users/HTTP connections and measure response times. Some lightweight open source tools I've used in the past for this purpose are ab, siege, httperf. A more heavyweight tool I haven't used yet is OpenSTA. I also haven't used The Grinder yet, but it is high on my TODO list.

When the results of the load test indicate that performance of the system does not meet its expected goals, it is time for tuning, starting with the application and the database. You want to make sure your code runs as efficiently as possible and your database is optimized on a given OS/hardware configurations. TDD practitioners will find very useful in this context a framework such as Mike Clark's jUnitPerf, which enhances existing unit test code with load test and timed test functionality. Once a particular function or method has been profiled and tuned, developers can then wrap its unit tests in jUnitPerf and ensure that it meets performance requirements of load and timing. Mike Clark calls this "continuous performance testing". I should also mention that I've done an initial port of jUnitPerf to Python -- I called it pyUnitPerf.

If, after tuning the application and the database, the system still doesn't meet its expected goals in terms of performance, a wide array of tuning procedures is available at the all the levels discussed before. Here are some examples of things you can do to enhance the performance of a Web application outside of the application code per se:
  • Use Web cache mechanisms, such as the one provided by Squid
  • Publish highly-requested Web pages statically, so that they don't hit the database
  • Scale the Web server farm horizontally via load balancing
  • Scale the database servers horizontally and split them into read/write servers and read-only servers, then load balance the read-only servers
  • Scale the Web and database servers vertically, by adding more hardware resources (CPU, RAM, disks)
  • Increase the available network bandwidth
Performance tuning can sometimes be more art than science, due to the sheer complexity of the systems involved in a modern Web application. Care must be taken to modify one variable at a time and redo the measurements, otherwise multiple changes can have subtle interactions that are hard to qualify and repeat.

In a standard test environment such as a test lab, it will not always be possible to replicate the production server configuration. In such cases, a staging environment is used which is a subset of the production environment. The expected performance of the system needs to be scaled down accordingly.

The cycle "run load test->measure performance->tune system" is repeated until the system under test achieves the expected levels of performance. At this point, testers have a baseline for how the system behaves under normal conditions. This baseline can then be used in regression tests to gauge how well a new version of the software performs.

Another common goal of performance testing is to establish benchmark numbers for the system under test. There are many industry-standard benchmarks such as the ones published by TPC, and many hardware/software vendors will fine-tune their systems in such ways as to obtain a high ranking in the TCP top-tens. It is common knowledge that one needs to be wary of any performance claims that do not include a detailed specification of all the hardware and software configurations that were used in that particular test.

Load testing

We have already seen load testing as part of the process of performance testing and tuning. In that context, it meant constantly increasing the load on the system via automated tools. For a Web application, the load is defined in terms of concurrent users or HTTP connections.

In the testing literature, the term "load testing" is usually defined as the process of exercising the system under test by feeding it the largest tasks it can operate with. Load testing is sometimes called volume testing, or longevity/endurance testing.

Examples of volume testing:
  • testing a word processor by editing a very large document
  • testing a printer by sending it a very large job
  • testing a mail server with thousands of users mailboxes
  • a specific case of volume testing is zero-volume testing, where the system is fed empty tasks
Examples of longevity/endurance testing:
  • testing a client-server application by running the client in a loop against the server over an extended period of time
Goals of load testing:
  • expose bugs that do not surface in cursory testing, such as memory management bugs, memory leaks, buffer overflows, etc.
  • ensure that the application meets the performance baseline established during performance testing. This is done by running regression tests against the application at a specified maximum load.
Although performance testing and load testing can seem similar, their goals are different. On one hand, performance testing uses load testing techniques and tools for measurement and benchmarking purposes and uses various load levels. On the other hand, load testing operates at a predefined load level, usually the highest load that the system can accept while still functioning properly. Note that load testing does not aim to break the system by overwhelming it, but instead tries to keep the system constantly humming like a well-oiled machine.

In the context of load testing, I want to emphasize the extreme importance of having large datasets available for testing. In my experience, many important bugs simply do not surface unless you deal with very large entities such thousands of users in repositories such as LDAP/NIS/Active Directory, thousands of mail server mailboxes, multi-gigabyte tables in databases, deep file/directory hierarchies on file systems, etc. Testers obviously need automated tools to generate these large data sets, but fortunately any good scripting language worth its salt will do the job.

Stress testing

Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing). The main purpose behind this madness is to make sure that the system fails and recovers gracefully -- this quality is known as recoverability.

Where performance testing demands a controlled environment and repeatable measurements, stress testing joyfully induces chaos and unpredictability. To take again the example of a Web application, here are some ways in which stress can be applied to the system:
  • double the baseline number for concurrent users/HTTP connections
  • randomly shut down and restart ports on the network switches/routers that connect the servers (via SNMP commands for example)
  • take the database offline, then restart it
  • rebuild a RAID array while the system is running
  • run processes that consume resources (CPU, memory, disk, network) on the Web and database servers
I'm sure devious testers can enhance this list with their favorite ways of breaking systems. However, stress testing does not break the system purely for the pleasure of breaking it, but instead it allows testers to observe how the system reacts to failure. Does it save its state or does it crash suddenly? Does it just hang and freeze or does it fail gracefully? On restart, is it able to recover from the last good state? Does it print out meaningful error messages to the user, or does it merely display incomprehensible hex codes? Is the security of the system compromised because of unexpected failures? And the list goes on.

Conclusion

I am aware that I only scratched the surface in terms of issues, tools and techniques that deserve to be mentioned in the context of performance, load and stress testing. I personally find the topic of performance testing and tuning particularly rich and interesting, and I intend to post more articles on this subject in the future.

136 comments:

Anonymous said...

Perhaps of interest:

http://openload.sourceforge.net/

falkenbt said...

Thank you for this nice overview. I would have liked JMeter to be mentioned along with OpenSTA and TheGrinder.

Anonymous said...

Thnaks for sharing very information on all three aspects of testing: Performance, load and stress testing.

Alex said...

You can look at Load Testing Terminology by Scott Stirling

Still don't see a clear cut difference between performance and load testing, they look more like synonyms for me with some small difference in, perhaps, goals of testing.

Senthil said...

Hi,

I still didnt get clear abt the difference btn performance and load testing. What are the parameters we should consider in both testings? Can we run single user/single iteration for performance? Is it neccessary to increase the load? Can we run the performance scripts for a fixed time?

Clearing this willbe very grateful.

Regds
Senthil

Grig Gheorghiu said...

Senthil,

I usually start my performance testing with a single client machine and I constantly increase the number of simulated concurrent users -- for example, I start with 10 concurrent connections to the server and I increase that number to 100 in increments of 10. At some point, either the client or the server will become the bottleneck. If the server is capable of serving a large number of concurrent users, then the client machine will become the bottleneck. At that point, you need to add more clients to your test and continue to increase the load on the server.

As for the main difference between performance and load testing: you do performance testing in order to find any bottlenecks in your application code and eliminate them. When you can't possibly optimize your application anymore, you start doing load testing, which tells you for example how many Web servers you need behind a load balancer, how many database servers you need, etc. -- all this so that you can sustain some pre-defined load that your customers/users require.

senthil said...

Hi Grig Gheorghiu,

Thanx for your explanation.But I have one more question to you.
Let me explain the way how I am doing the performance/load(I dont know how to name it) testing.
We are doing performance testing of a web application from a client machine.
steps:
1) first I will prepare the test case scenarios for the testing based on the need of customer(for example some searches, Reports and some navigations)
2) I will record the steps using OpenSTA.
3)Develop the test task and run the individual tests from 1 v.user to max of 100 users.
4)Before reaching 100 v.users only I will get errors in the results and graph start decreasing,
5)So based on this graph and data I will find out the bottleneck for that scenario.
6) usually we are running each test for a constant period like 10mts or 15 minutes.
7)Here how I can distinguish btn client and server bottleneck.
8) Am I doing the testing in a right way or not. We are calling this as performance testing only.
9) Here how to do load test?

Sorry for putting lot of queries.

Thanks and regds,
Senthil

Grig Gheorghiu said...

Senthil,

As I said before, it's sometimes hard to distinguish between performance and load testing. Many people use them interchangeably and I can't really blame them.

One of the points I was trying to make in my blog entry was that by performance testing people generally mean tuning your application so that you eliminate bottlenecks in your code. In your example, you would need to profile your Web app while you are sending requests from your client. You will then be able to see what functions/methods in your code are the hot spots (i.e. where most of the time is spent). At that point, you can try optimizing your code, or maybe you database access, if the code involves calls to the database.

If there's no possible way to further optimize your code/database logic, then you need to look at scaling the application either vertically (by throwing more hardware at the server), or horizontally (for ex. by adding more Web servers behind a load balancer).

Load testing is what you're doing after your eliminated the bottlenecks in your code. You want to know at that point what the maximum capability of your application is. So I think in your case, since it doesn't seem like you're doing any profiling/benchmarking within the application, we can say you're doing load testing. Your methodology is sound, but at the same time you need to monitor the server via OS-specific utilities (vmstat, top, iostat or Windows Perfmon) and also, as I said, you need to monitor your app via a profiler.

As to how to see whether the client or the server is the bottleneck, you can try running another network-bound process on the client against a different server. If the client can sustain the second process, it means the first server was the bottleneck. If the resources on the client are exhausted, it means the client is the bottleneck. For more info on client-related bottlenecks, see http://www.hpl.hp.com/personal/David_Mosberger/httperf/doc004.html

senthil said...

Hi Grig Gheorghiu,

Thanks for your clean explanation.
Actually Our work starts from discussing with the technical team for getting the right scenarios where we need to do performance testing. After getting the results we will do the analysis using the HTTP DATA List in Excel sheet.
For CPU monitoring we are using a tool called "PerfView" where we can get all server (CPU time,network,I/O usuage and etc)related parameter values. We will get both app server and db server details in that tool. My Question is running for a constant period is right or not. I feel this time is need to scaleup the vusers in server. But I am not sure how we can decide like running for 10 mts or 15mts. Will it make any change in results. We are taking the average time of each urls in excel sheet started from 1 user to max users.

Clearing the time duration to run will make me comfort. I am also looking your other guidance points which definitly will improve our quality and use of testing.

Thanks and Regds,
Senthil

Anonymous said...

Good concept on Load and Performance testing.
Now can we say load testing is some thing where we test the systems to check how they work under specified load?
and performance testing is testing an application with slow ramp up to see the response times under the load?

Correct me if i am wrong. The terms
Performance,Volume,Load,Spike,
stress testing always confuse me.

Suresh Chatakondu

Grig Gheorghiu said...

Suresh, I think you're right in your characterization. I'd also add that for performance testing, you need to be profiling your app and monitoring your servers while you're increasing the load on the system. I tried to explain these terms in a better way in another blog entry: http://agiletesting.blogspot.com/2005/04/more-on-performance-vs-load-testing.html

rajkumar said...

rssd

Anonymous said...

Hi

I have still confussion with Performance,Load Testing, Volume Testing Can you plz one example for each testing

Thanks
S Rajkumar

Rajkumar said...

Hi,

Why winRunner does not support
.net Apllications

Thanks
S Rajkumar

Anonymous said...

Hi,

Are performance, load and stress test components of system test?
Every organization is different. But I would assume the typical test life cycle is this. I would break it down to these 4 major phases.
Unit Test --> Functional Test --> System Test -->Beta Test.
In the System Test phase, I would have these major tests: Integration Test, Performance Test, Load Test, Stress Test and Scalability Test, etc.

Does anyone have any comment on my statement? Recently, we have some debate within our Test Organization about the testing terminologies and methodologies. I really want to know how other testers understand the test life cycle and definitions of its components.

Anonymous said...

http://www.testingreflections.com/node/view/2321

Alejandro Betancur said...

Another cool tool for do performance testing is Eclipse project called Hyades.

With that you can easy test web applications, I really like to use it, its more simple and more especific than OpenSta.

Also you can fin www.scapatech.com and see there a great tool (this is not a free one)

Anonymous said...

Hi all,
Is there any open soure tool for performance testing. If It there, Can support IE 6.0. I heard abt open sta will support only 5.5.

Rajesh Duhan said...

Thanks for your valuable discussion
Rajesh

Rajesh Duhan said...

Hi all
its realy a nice discussion
Rajesh Duhan

luca said...

How do "concurrent users" as set in a load test, relate to real traffic numbers? e.g. if i want my site to cope with a peak traffic of 4000 hits/hour, what number of concurrent hits should it be able to cope with? By my reckoning there is only one answer: it has to cope with 4000 concurrent users, incase those 4000 hits/hour all come at the same time. But then that's ridiculous!

Raghavendra said...

Hi,

The explanation was great and upto the mark. I would like to know how to identify Matrices(like bottle necks, performance of the appilcation). You can mail me at ragu.karnam@gmail.com

Once again good one.....

Thanks
Raghu

Anita said...

Hi,
I am a gaduate student and my project is about web application testing tools. can anybody let me know if Hyades creates a model from each web site and creates the test cases from this model; or it duplicates the action sequences directly from weblog or mimics recorded user actions?

Thanks,
Anita

Anonymous said...

I tried Neoload (published by Neotys) that is an easy to use load testing tool (but not free).
Hope this helps.

Anonymous said...

Hi,
I need to know what are the steps in a performance test.

Thanks

Anonymous said...

Hi,

I am a load tester who hasn't got a project to work on yet.

My question is: how do I work like an expert, I mean what's the best methodology that will impress my boss?
Remember your first load/perfomance project! How did you go about in preparing fo rthe test?

Your contributions are more than wwelcome and please feel free to mail me at randysage2001@yahoo.com

Anonymous said...

just found this one...gives a good overview on some of the fundamental issues to consider.
http://www.intel.com/cd/ids/developer/asmo-na/eng/257829.htm

hemanshu said...

1. double the baseline number for concurrent users/HTTP connections
randomly shut down and restart ports on the network switches/routers that connect the servers (via SNMP commands for example) :
Here how it will justify that you have stress on system as the word itself say that you have to give some stress on system and not the RECOVERY TIME, as this will consider how much time the system will take to retain its stable working.

take the database offline, then restart it
Again this is recovery step...

run processes that consume resources (CPU, memory, disk, network) on the Web and database servers
This step is correct but how will justify the statement where you have said "take away resources ...". I am confused about these statement.
Please clear me.

viji said...

thanks for giving this information

naga said...

i want Performance Testing by using JMeter, can any one help me

naga said...

Performance Testing by using JMeter

Anonymous said...

Ive been a performance and functional testing consultant for about 12 years now. Here is the problem. Many people forget that performance and stress are not interchangable. I dont think Grig couldve been more clearer.

The tools I use are LoadRunner, WinRunner, QuickTest Pro, Scapa, and others.

Right now I am on a project in Woking, UK, where the new system architechture is being tested for performance, load and resilience. During this round of testing we are loading the system with 5000 concurrent users performing normal user actions on the site. The webservers, appservers and DB performance levels are constantly monitored.

Now do you call this a performance test, load test or a stress test.

This scenario is a performance test. The system is being evaluated for its performance under a constant load of users. They want to see if the proposed new architechture is sufficient enough to support this load of users. However, this can be classified as a load test as well, simply because there is a sustained load of users (clients) hitting the system over a period of time.

Another scenario is this:

The system being tested is a new system architechture and tested with increasing number of users to guage the performance of each component compared with the load provided to the system.

This is a load test. Because there is no defined target and the only goal is to see where the breaking point of the system is. This set of testing is not stress test because no part of the system is in distress while the test is conducted.

Is everyone still unclear?

Michael Sky

Siva said...

From what I understand in the context of a website --

Performance Testing aims to answer (one of) the question(s) - How many concurrent users can my site handle and still provide a n average response time to the user of under 1 second?

Stress Testing aims to answer (one of) the question(s) - What would happen if the database server crashes when hundreds of users are transacting online?

Am I thinking right? ;-)

Priti said...

Hi Grig,
Has anyone come across a server-client testing tool, I mean not a web application. Tool which could test application developed in Java.
But mainly I require that whatever test scenrios are developed should be shown in Graphics and also the result should be mesureable.
Would be glad if anyone can provide details of such a tool.

Anonymous said...

luca-
to replicate production test, just limit your resources in proportion to your users. So if you want to test for 4000 sym. users in test, you look at your production machine and half or quarter it. (ie. if production has a 2.7 Ghz box with 2 Gs of ram. In test use a 1.3 Ghz machine with 1 G of ram. Then use 2000 users and so on.)

Anonymous said...

its good but try to keep the difference in more simple words.

daryl.cammack@us.ccl.com said...

This seems focused on web testing. Any suggestions for tools to load test Windows thick-client applications?

Thanks
daryl.cammack@us.ccl.com

Grig Gheorghiu said...

Daryl -- sorry, can't help you very much there, since I haven't used thick Windows clients in a while....One approach would be to roll your own scripts that simulate key presses, mouse events, etc.

Check out the tools (admittedly Python-based all of them) in the "GUI Testing" section of the Python Testing Tools Taxonomy page I'm maintaining (I split the URL in multiple lines, please join them together):

http://pycheesecake.org/
wiki/PythonTestingToolsTaxonomy

HTH,

Grig

Anonymous said...

Hi,
I want some of Test cases for load and performance testing for Webservices

Anonymous said...

Hello Grig, your blog is very informative. Thanks a lot for providing such useful information. But I still have a doubt, is load testing and volume testing the same? is there any specific difference between them.

I do not know weather it is true but i read somewhere that in volume testing we test large sets of data using one user whereas in load testing large sets of data is tested using multiple concurrent user.

Please suggest. waiting for your response

Grig Gheorghiu said...

Anonymous -- I don't think there is much difference between volume and load testing. I would say though that load testing is usually applied to the system as a whole, in order to see what its behavior is under constant load. Volume testing is applied to specific sub-systems, to see how they react in the presence of high volumes of data. But as with performance and load testing, these terms are sometimes interchanged.

As far as I'm concerned, a sound testing strategy should incorporate these types of testing, paying less attention to their names and more attention to actually running them.

Vijay said...

Thanks a lot, Grig.

Ola Alowooja - DORTCOME said...

Hi Grig,

I have to say that the information you have posted is very rich. I happen to have an interview in London (City), and have really benefited from this Blogg of yours. I am taking on a new role as a test analyst and this information has definately cleared most of my doubts.

Could you please reply to my post. Regards.

ola.alowooja@gmail.com

NB: Thanks everyone for your ideas and sharing you knowledge.

Anonymous said...

thanks for your wonderful contexts

Anonymous said...

Hi,

I am trying to figure out the differences between Loadrunner and ACT tools. Can any of u guys help me?
Can u suggest which is a better tool?

Jeff.

Anonymous said...

Hi Grig, i like very much your syntetic and clear explanation about difference in PLS testing approach.
I translate in Italian your article to explain to non-technical users the difference between these tests.
Because I see in it.wikipedia that this definition are absent, I would like to insert my traslation from your article on it. I've seen you use CC license, but I'm not sure about its limitation, can you give me your permission to public on Wikipedia? Thks in advance, regards
Giovanni

Grig Gheorghiu said...

Giovanni -- sure, you have my permission to put your translation on Wikipedia. Thanks!

Grig

swapna said...

Good one

Papa said...

Hey Grig,
like most of the readers (I think) I still don't see a clean line in the sand between the three methodologies. What I think is clear is that depending on the situation, you have to mix and match to get to where you want. If my understanding is clear, we have 3 main metrics here:
. nominal performance
. throughput
. resilience
that can in turn be mapped to:
. performance testing (assessing a baseline)
. load testing (how many clients before performance degrades or, as customers would ask, how many boxes do I need for my X customers)
. stress testing (what are the parameters that need to be monitored to make sure the service remains available)

With that being said, I would like to come back to one of your first sentences: "Ideally, the software under test is already stable..."

Performance is a very interesting metric to have even as the project just starts. It will allow to pinpoint, as features get committed, which has a steep price in terms of performance. Or, when the team focuses on performance improvements throughout the project, it will allow to keep chaos under control by being able to quantify improvements over baseline or over previous measurement.

Thank you for an interesting post that fosters and focuses the interest of the community on one important aspect of testing.

Arnaud

Grig Gheorghiu said...

Arnaud -- thanks for the thoughtful comments, your taxonomy makes sense. I agree that doing performance testing early in the game has lots of benefis, but in my experience very few people/teams do this.

Grig

Anonymous said...

Great article! I tried OpenSTA and Grinder, but found there was a lot of scripting and debugging to get my tests to run properly as we have a very dynamic site.

I eventually came upon OpenLoad, which was much easier to use and relatively inexpensive.

pavan giridhar said...

what is the difference between stress testing and recovery testing?

pavan giridhar said...

what is the difference between stress testing and recovery testing?

Arumugam said...

I dont see any speicfic difference given in the post,it will be better if a comparision in given between the three

Anonymous said...

hi grig

a good overview.

as per my knowledge performance testing usually evaluates 3 things

1 throughput
2 response time and
3 latency


thanks & regards
murali

Srinidhi said...

I do not why people have a confusion between Performance and Stress.

Performance is you are veryfying the performance of an application for a set of users i.e. say when 2000 users acess the system the system is serving them in 5 seconds.

Stress is the same above system is designed for 2000 users but there can be some time in the business calendar when there will be 3000 users, as the application is only designed for 2000 users the application can be slower than regular load may be the system may take 9 seconds to serve a page, now if there are still more users say 1000 more that is total load is 4000, the system can run out of resources and may crash.

Hope you got it

rekha said...

my name is Rekha Can anyone suggest me the goo book for testing which good enough for the preparation of certification test in testing my id is sarma.shruti@rediffmail.com

Anonymous said...

I must appreciate the way you have explained all three types of tests.
Being a non It , even I was able to understand with great ease.
Really worth explaining and wonderful explanation.
Thanks.

Anonymous said...

The companies I worked as performance engineer have been doing the following Performance tests:
Baseline/Benchmarks testing
Load/Stress load testing

In general:
Baseline/Benchmarks testing: need to be exec from build to builds, releases to releases under the same hardware settings, the same test tools, the same workload profiles, the same running time 10-30mins, collected the same required performance matrices, mostly measuring the peak time product matrices, the purpose is to see if the results fit product release requirement (such as web page latency < 3sec, concurrent connections/users > 1k or 5k, %CPU usage < 85%), then comparing the results between releases, build#, different venders' similar products. Most of time, company will use this tests to collect the best numbers to advertise its product. Sale, Marketing depts. are more interesting on those results. In real life, lots of time people may never experience such good numbers. Such as mileages declared by automobile company.

Load/Stress Load testing: we may use the same setup like benchmarks, but mostly we used more complicated workload profiles, the test settings may be modified over the time as long as the tests will be designed to excise the most of product functionalities, the tests running with less intensive load, say if benchmark showing the product can handle 2000 concurrent connections w/ latency less than 3-5sec, the stress load with more complicated workload may only running 50-70% of 2000 conn. But the load duration will be running from overnight about 12hrs to 72hrs or weeks. Under the load, lots of functional tests may be performed. This kind of tests is more simulate real life scenarios.

During both above tests, we all collect product system resources usages by using Win:PerfMon, Unix/Linux:vmstat/mpstat/iostat/netstat/lsof/ps/sar. %CPU, context switch, memory, IO wait/queue, num of established/time-wait sockets are counters helping to identify product’s bottlenecks & defects.

It seems like the different companies; even different projects within the same company may have different process & methodologies to do their performance tests. Performance release requirements & goals are very important as starting point.

Rosie said...

These comments have been invaluable to me as is this whole site. I thank you for your comment.

Neelkanth said...

Thanxs 4 good but explanation
I would like 2 add 1 more thing that
take ex of total tesing of chair==>
normal tresting ==if chair is dsnd 4 100 kg wt,n my wt is 70 kg then that teting is called as nornal testing.
if my wt is 100 kg then that testing is called as load testing,
if my wt is 120 kg then that testing called as stress testing..

Rajiv Walia said...

Hi Neelkanth,
Thanks for clear comments on Stress and load testing. But what about Performance testing.

Thanks,
Rajiv

Rajiv Walia said...

Hi Grig,
Can we check Performance testing by using:

1. Load testing
2. Stress Testing

Right?

Roy said...
This comment has been removed by the author.
Anonymous said...

Actually I reccomend WebLOAD when talking about open source performance, stress or load testing tools.

webload can be found here:
http://www.webload.org

msuman75 said...

hi This is suman, i am new to laod runner, right now i need to test the web application , same like the gmail(option) like

login-
Edit the mails
logout

can any one say what are the stpes to test this application

my email id is msuman75@gmail.com

Balaji said...

Very good compilation. Clearly and practically explains stress, load and performance testing.

Simha said...

Good Article thanks.
Is there any open souce tool to test the network speed during the run of performance other than the utilites like netstat etc. that comes with the OS?

Grig Gheorghiu said...

Simha -- check out iperf at http://dast.nlanr.net/Projects/Iperf/

Grig

Blackpool Hotels said...

large number of concurrent users, then the client machine will become the bottleneck. At that point, you need to add more clients to your test and continue to increase the load on the server.

Posted By Blackpool Hotels
Date: 19th September 2007

Gokul Nadar said...

Let me give you a scenario to help understand my question, There is a web intranet application that is being accessed from different locations. When the users from the Head Office are accessing this application they have no problem, however the remote location users have a prblem of delayed response. The hardware on all the locations including the HO is the same. How do I conclude if the problem is because of slow network access or slow cpu processing. I have tried trace route and that indicates a mojor time lag between the hops for the problematic locations, however the HO tracert is good. Also I have tried accessing a static page from all the locations and the response time for this page is very big from the problematic location. Can I conclude from this that the problem is with the network or is there anything else that I can try. Very importantly, how can I clearly eliminate the possibility of slow cpu response.

Grig Gheorghiu said...

Gokul -- I think there's a simple test you can run to eliminate the possibility of a CPU bottleneck. Run some utility on the server running the Web intranet application. If the server is Unix/Linux-based, run top. If the server is Windows-based, run Performance Monitor. Access your application from the HO and note the CPU utilization. Access your app from a remote office and also note your CPU utilization. My guess is that the 2 numbers will be almost identical. This means the bottleneck is not CPU-bound, but, as you noted from the traceroutes, network-bound.

I'd also run iperf between the clients and the server -- it will tell you what kind of network throughput you have on those links, and it will be another argument for the network bottleneck. Iperf is available here: http://dast.nlanr.net/Projects/Iperf/

HTH,

Grig

Gokul Nadar said...

Thanks a lot Grig, I really appreciate your feedback, it has been very helpful.

Anonymous said...

Hi,

I have a query regarding the performance tools. I am providing my test scenario below:
I have a product installed on client machine.I want to check the performance of the machine after my product is installed.
I know we can use perfmon to get the performance results.
Is there any other tool which can provide analysis as well for the machine after the product is installed.

Thanks

jenni said...

What are the Parameters do we need to consider for Performance Testing (web applications)

Grig Gheorghiu said...

Anonymous -- perfmon should do the trick in your case. Establish a baseline before installing your app, and then run perfmon again to get new stats after installing it. Sounds easy, but of course it's tricky in practice.

Grig

Grig Gheorghiu said...

jenni -- those parameters were the subject of this post.

Gokul Nadar said...

Hey Grig,
This time I have a question regarding my career with respect to software testing.
Academically I have completed my Masters in IT and I have 2 years of experience in software support prior to this. Recently I have joined a software testing firm as a Team Leader for the performance testing team. My management thinks that I'm a very fast learner(based on my interview), hence even though I dont have any testing experience, according to them I would be able to understand the technicalities very soon.
Now since this is a very new industry for me, I dont have much information to plan my personal growth plan wrt s/w testing.
Currently my qualification is that of a master of IT, what are the other certifications that I can do to enhance my performance testing skills and which would help me grow in the management.
What are the capabilities that I require(based on international industry standards) to become a Project Lead and Manager within 2 years. Pls guide me. Thanks in advance for your help.

Grig Gheorghiu said...

Gokul -- I don't believe in certifications. I believe in doing your job passionately and growing 'organically' in your career that way. Setting a goal such as 'become a Project Lead and Manager in 2 years' is kind of unrealistic in my mind. Since you've been out of school for a couple of years only, you'd be better off learning as much as you can about your trade. I'd also recommend joining an open source project and contributing your time and skills there. It's a great way to learn the trade of software engineering.

Grig

Gokul Nadar said...

Hey Grig Gheorghiu,

Thanks a lot for your feedback, I really appreciate it from the bottom of my heart. What is open source project and how do I join that?

Shishir said...

hello sir,
I'm trying to record a script in Silk Performer. Its an internet application. However I'm not able to record the script. What are the possible reasons for the application not getting recorded.

Bishnu Charan said...

hey Grig,

I want to load test an web based application. could u plz provide me the steps?

Grig Gheorghiu said...

Shishir -- I don't use Silk Performer, so I can't help you.

Bishnu -- I don't do free consulting on my blog. If you have a specific question related to web testing, feel free to ask, but I'm not going to respond to your general 'how do I do web testing -- give me the steps' question.

Grig

Grig Gheorghiu said...

Gokul -- for the largest repository of open source projects, go to sourceforge.net. Search for your favorite OS or programming language, and take it from there.

Another repository of open source projects is hosted at Google Code at http://code.google.com/

Grig

Gokul Nadar said...

Thanks a lot Grig, will definitely take it up from here.......

Anonymous said...

I'm analyzing the home page of a bank website, am analyzing it for enhancement. Right now I'm using the tool SILK, what are the other tools that I can use? PLs suggest.

Anonymous said...

Hi Grig,

I wanted to know if we can do performance monitoring of Active Directory. I am pllaning a project where I would monitor active directory 24*7. Please suggest which software would support. Should I use Silk or Rational or Ruby or any other.

P1 said...

Can any one comment on types of Load test Scenarios apart from
1] Real life scenario
2] endurance
3] stress
4] Load
5] Capacity

Appreciate if some1 can provide name of scenario and crisp 1 liner definition . If some1 designed any specific scenario please share. I ll update if I come across any new ones.

Srinivas said...

Hi Grig,

This is srinivas, I done Load Testing on web application and my doubt is what time should be considered for valuing the test and one more thing. For eg: in my test for 25 user I got TPS 0.4 seconds (transaction Per Second), and the elpse time is 9.925 seconds and in this which time i have take in to consideration. This test I had done in Mercury Interactive Load Runner.

Panks said...

Hi,
Quite informative. Most of times in the testing literature I find the definitions and descriptions of various concepts quite vague and many a times the various concepts just intersect at one point or the other making it difficult to distinguish between them (eg. the case of load testing vs stress testing in this case). Looking forward for more such references where the distinction between the seem-to-be-similar concepts is clear.

Regards,
Pankaj. (pankaj_nith@yahoo.com)

M said...

Quite impressive differentiation.Can anyone tell me about performance testing for symbian services?

KELVIN said...

Hello,

I will like to know what tools will i use to perform a Performace testing on a web application which was developed in .net.

thank you

kelvin

Anonymous said...

Hi Grig,

I am new to performance testing .Can you please explain me how to do the performance testing on Stand alone system and web based system? And what are the exact(MAIN) measurements we have to measure?

Atul said...

great explanation for the three aspects of testing:Performance, Load and Stress.

Thanks

Atul

aradhya said...

Have a question guys,

- An online banking system needs to be availbale 24/7. This means it must run continuously without any problems and without needing a restart.

1. How would you test whether such a system has slow memory leaks, which could lead to poor performance over time. ??

Geeta Chaudhari said...

Hi,
This is really a very helpful article. Well, one can also say that Performance testing is testing using a single client/request and check the bottlenecks i.e. profiling. Once this profiling is done as desired, then we can start loading the application and check the performance again.

Rajan said...

Hi All,

Can you explain the differences between Performance, load and stress with a good examples.

prashant said...

Can any one please tell which tools are used for performance, stress and load testing?

Bendz said...

Hi,

Good post with Nice explanation. Needed and Useful information.

:-)

FMH said...

Hi
Thanks for your explanations, but nobody say about how should we schedules vusers? From how many user we start? how increase them? how long we continue our test?
I want to determine the TPS of my web application. I don't know how manage concurrent user to determine tps of my system. for example once I test 10 concurrent user with ramp up 2 user per second and 5 min test duration. Another time I use 100 concurrent users with same condition, but with 5 concurrent user I get better Tps.

DCM said...

We are just starting with performance/load testing. I am still not clear as to what call my test as. We are starting with benchmarking, as we do not have any data on the accepted mem/cpu/time value right now. Does it fall under Performance or Load testing? and surely we are not trying to fine tune the application . We just want to measure the afore said matrices and decide whether those are acceptable are not.

Grig Gheorghiu said...

DCM -- I'd call that 'Load testing'. After you get some baseline numbers (i.e. numbers for which your app still behaves within certain parameters), you can start stress testing -- which is testing your app while increasing the load above what you had for your baseline numbers.

Grig

Anonymous said...

Opensta is a Powerfull tool for Http/Https protocal. Its provides n number of virtual user for free of cost

Opensta Overviews

Anonymous said...

hi Grig,

what are the things to prepare in doing stress testing? can you give me a procedure in doing it?

thank you,
rose

Anonymous said...

Performance: What is our peak processing capability (CPU/DB/Memory within tolerance and steady).

Load: When does our peak processing capability begin to digress (CPU/DB/Memory begins to run out).

Stress: When does our processing capability digress below our expectations. (CPU/DB/Memory gone...)

Management, developers, marketing usually understands this whole method as performance testing.

Anonymous said...

how can test performance manually....pls help...

pram said...

One can tell me freeware tool to trace the performance an ERP application

Harvard said...

Thanks for your informative data, about the " load testing ".
It helps me a lot...

thanks,

harvard

Bruno Guerra said...

Very good article. Greetings

Amit said...

Hi,
I am looking for some help on Mainframes testing in terms of what are the pre-requisites for performing Interface Testing, Regression Testing, Load& Performance Testing and Stress Testing on Mainframe Applications. Wat needs to be studied during the assessment phase?
The applications are all mainframe based using proprietary HPS Language which generates COBOL/PL1 Code.

Anonymous said...

I believe the link to OpenLoad is:

http://openwebload.sourceforge.net/

not

http://openload.sourceforge.net/ as one person posted.

Anonymous said...

There is a difference between stress testing and recovery testing and where u said the purpose of stress testing is to check how system recovers after failure is not correct as this is part of recovery testing, not the stress testing.
Stress testing is checking of application behaviour in least resources.

Randy Hayes said...

Really well done! You might want to look at CapCal Agile Performance Testing - it integrates easily with agile frameworks and best of all it's free! www.capcal.com.

Also, have a look at the CapCal Blog at http://capcalblog.blogspot.com to see about the first million user load test!

Muhammad Naeem said...

I really appreciate this article.Its really worth reading.The terms are synonyms but different in nature.I would like to share some other tools e.g.Load Runner,Rational Robot as well.

sri said...

Hi Grig,

Thanks for all your explanation. It was really helpful in understanding the performance/Load/Stess testing.

Can you please clarify the below scenarios. Iam new to blackbox approach performance testing. Currently I am understanding the HP Loadrunner9.0 tool eval. I was in a process of doing load testing for my application. I am not going to do any tuning for my application as of now. Even if you are not aware of Loadrunner tool, Please provide me the approach for this testing.

My concern is, If i want to do load testing for “Sign up” module or one of the business req, How do we do? Since the user id is unique, we can’t enforce load with the same script (single user id) with 50 users concurrently. I assume script will fail in this scenario. So we are going for parameterize the value (user id), If we parameterize, obviously it comes into iteration. So each and every iteration the new value is replaced. In this scenario how come 50 users concurrently perform the “Sign Up” transaction with different user id?

Another scenario is Login transaction. If i want to perform load test for login transaction around 50 users concurrently with different name in a same script. We can go for parameterize. But the same doubt comes to my mind. How do we do with different user concurrently with the same script?

It would be really great to see some approach to overcome these scenarios.

Thanks in Advance

Regards
Sridhar J

Adam Brown said...

Sridhar et al, some of the questions you ask are answered in my video. You can find this on the homepage at http://www.qutoium.com
The video demonstrates QTest which is developed by Qutoium who bought Cyrano who developed OpenSTA. In fact some of the developers of OpenSTA worked on QTest. Hope this is of some use.

pc said...

Can anyone tell me that what is the performance of "Neo Load" Testing tool for stress testing ??

vishalg said...

Hi,
I am doing masters in comp sci.
Can anyone tell me how to measure the performance of an appserv only?

Phillip said...

Web developers should always be performance testing their sites. New cloud load testing tools are making it cheap enough for all of us independent consultants.

Prince said...

Can you please tell me the difference between Emulator and Simulator.

Anonymous said...

@prashant: NeoLoad is a good load, performance and stress testing tool. You can try it for free.

Adam Brown said...

Prince - in software testing terms an emulator is usually some software that is used to represent a physical device such as a mobile phone or other device with embedded software. These emulators are used to run applications that would usually run on the emulated device. Because an emulator will run on a PC then PC based software testing tools can be used to test the embedded device albeit an emulated embedded device.
A simulator would typically simulate the output of something else. For example a load testing tool like QTest (http://www.quotium.com) would capture the traffic from a real application or device like an iPhone or emulated application and then replay it thereby simulating the client application.

Adam Brown said...

Prince - in software testing terms an emulator is usually some software that is used to represent a physical device such as a mobile phone or other device with embedded software. These emulators are used to run applications that would usually run on the emulated device. Because an emulator will run on a PC then PC based software testing tools can be used to test the embedded device albeit an emulated embedded device.
A simulator would typically simulate the output of something else. For example a load testing tool like QTest (http://www.quotium.com) would capture the traffic from a real application or device like an iPhone or emulated application and then replay it thereby simulating the client application.

suji said...

hello sir,
thanx for overview...can u brief abt the difference between volume/load/performance/stress testing with single real time example...i vill b really thankfull to u, if u help me

suji said...

im undergoing training in testing...first let me tell u what i understood 4m my materials..

performance testing:measuring bottlenecks in performance of a project in controlled environment eg:response time issues

load testing:testing done to know how much maximum load the system handles at particular point of time,eg:making thousands of users hit a server at same time
but i found both above testing are same

next comin stress testing:finding the breaking point of ur system in a unintended environment.

volume testing:testing done to verify how much volume of data that your system handles(i didn understand literally)

storage testing(i didn understand)

Note:in material its explained as such all are different

i think this may be tedious for you to explain all..still waiting for your answers...thanks...have a good day!!!

Computer technical help said...

Performance testing seems to me to be much more broad than load testing. Load testing is usually focused on metrics like requests per second and concurrent users (the cause); whereas performance testing is more concerned with response times (the effect).

VJ said...

Actually the application on which I am working is quite an enterprise level. But I have to do load testing only on smaller part of it, means a module which is totally in Java and also the calls are using REST API, we have to do only the backend code resting and no UI. This includes db, middleware(communication using REST API and Java) and network.
I have planned and also started working with SOAPUI (Groovy Script) which I found to be very useful. But when I searched on Google, I hardly see it in the top list of tools being used for performance or load testing.
Can anyone tell me reason for this?
And it would be great if anyone can list of tools that can be used for such application.

通達人 said...

Hi Grig,
Can I translate this article in Chinese and post it on my blog?
Of course, I will link it to this article.

Grig Gheorghiu said...

Sure, go ahead. Make sure you send me the link to your post too ;-)

通達人 said...

Thanks

Ian Holdsworth said...

Grig

Fantastic article. So many people get confused about the differences between Performance, Load & Stress testing. Very concisely done.

Thank You

Anonymous said...

Thanks a lot for the information
fantastic n concisely defined

Martin Buhr said...

This is a really good overview of the difference in load testing, performance testing and stress testing - personally stress testing scares the living daylights out of me, breaking a system on purpose to check recoverability seems like madness.

It's quite surprising how few teams actually use load and performance testing in their workflow - it could be because the commercial tools are very expensive and the Open Source ones a bit too cryptic to set up.

Thanks,
Martin
Founder Loadzen - Load testing in the cloud

miaka said...

Hi,

Performance testing, is to check the response times of each load on the AUT. The time it takes to perform each operation..

While, load testing is to check how the application behaves for a number of users in a certain point.

Cheers,

Arlene

Dzmitry said...

Hello Grig,
Interesting post, thank you. What I'd like to add is that performance testing allows to define not only bottlenecks in performance, network. Also this kind of testing can help functional testers to find functional bugs, that appear only under high load.
I invite you to visit our blog devoted to performance testing in general and using JMeter in particular - http://blazemeter.com/blog

Anonymous said...

You can try https://loadfocus.com for JMeter Cloud Load and Performance testing very reliable and with thousands of concurrent users