Comparison research for Load, Performance and Stress test tools.

This report compares most affordable and yet practical tools that can be used for Load and Performance testing tasks in xxx.  Nine tools were selected and determined based on all necessary aspect to perform high quality, effective and accurate tests.

The main factors which are necessary for the selected tools are:

·         Can be used for a long time (Reliable).

·         Not very hard to learn and teach (User friendly)

·         Powerful and accurate enough to cover all of our needs. (Compatible)

·         Preferably, have more features and abilities for other purposes (Other types of tests, Development, Life Cycle Management, Bug tracking, etc.)

·         Provides high quality and understandable reports to prevent exhaustive (and costly) manual data analysis.

·         Have a fair support (or very good documentation). For live support, the time zone (and support desk availability) is one of the concerns.

·         Not too expensive while there are better solutions with better price! (Affordable). In this document, less than $2000 is considered as “Low Price”, Up to $10000 is called as “Expensive” and more than $10000 is considered as “Very Expensive”.

Unfortunately, there is no particular tool that covers all above factors. Therefore, we need to use a combination of tools in a way that gives us best performance and quality.

Quick links:

-          Table A: Explanation of features and rates in Table B

-          Table B: Tool assessment by most required features.

-          Table C: Prices, Cons and Pros

-          Conclusion and Suggestions.

Table A: Explanation of features and rates in Table B.

Feature	Explanation	Rating meanings.
Supported Protocols	Shows if the tool supports a wide variety of protocols which may be used by our developers. Some protocols like HTTP, HTTPS, TCP, FTP are vital. Other important protocols are: JSON, REST, SOAP, WPS, WSDL, XML. A.      Very good: Only the Visual Studio Ultimate can support lots of protocols. ( That is one of the advantages of “very expensive” tools ), it is however, very hard to work when LDAP is being used. JMeter has more flexibility with LDAP. B.      Good enough to support our needs:          All except MS Web App Stress Test C.      Poor: D.      eValid (No SOAP, REST, JSON, Java applets)
Record and Playback	The tool’s ability to record series of actions and imitate them as a playback. This feature can save time and energy and reduces the human mistakes. A.      Professional, Fast and Easy (for an expert) APPPerfect Load Test, MS WAS, JMeter+Selenium, SoapUI+LoadUI (Pro), Visual Studio Ultimate, NeoLoad B.      Acceptable, Not so easy and fast. SoapUI+LoadUI (Free), WebLoadUI (Both Free and Pro), eValid C.      Not Available None
Scripting	Using scripts helps to use the tool more effective and flexible. This can be an integrated IDE or by using a plugin or third-party tool. A.      Good script support, well documented. JMeter+ Selenium , SoapUI+LoadUI (Pro), Visual Studio Ultimate B.      Fair script support, Poor documentation. APPPerfect , eValid , SoapUI+LoadUI (Free) , NeoLoad C.      No script support. WebLoadUI (Free and Pro)
Correlation	Ability of comparing recorded results with playback results. This can be automatic or needs to add manual assertions. Very necessary to perform effective tests. A.      Automatic and Powerful Correlation, or easy to do it manually. WebLoadUI (Pro and Free) B.      Can be done using scripts or with some efforts. All others C.      Not supported. None
Parameterization	This feature facilitates the tester to use dynamic values in different ways and create data driven tests. A.      Fully supported All except WebloadUI (Free) and MS WAS B.      Supports with limits WebloadUI (Free) , eValid (weak parameterization) C.      Not Supported None
Server Monitoring	Having the option to monitor server’s resources during a stress test or capacity test, will be a huge advantage. However, this doesn’t mean that having test tools with server monitoring feature is the must. We can use another free resource monitor tool for server surveillance purposes and it would be fairly close to the more accurate report generated from the test tool. A.      Supported (integrated) by the Tool SoapUI+LoadUI (Pro), WebLoadUI Pro, NeoLoad, APPPerfect B.      Can be done with additional plugins or needs some efforts. JMeter (The only free one in this category), Visual Studio Ultimate C.      The tool doesn’t support the feature (but we still can use other monitoring tools) SoapUI+LoadUI FREE , WebLoadUI Free, eValid
Distributed Testing	This feature gives the ability of having multiple load sources at the same time, which is being controlled by the “main” tool. This helps when we want to have a tremendous number of hits/users that can’t be handled by one station. Also it makes the test environment closer to the real life since each server has its own IP address, ping latencies, packet loss and more. We also can use not supported tools on several machines simultaneously, get the reports separately and merge all reports to one; But this would be not an easy task and takes time to create an individual report. A.      Full Support SoapUI+LoadUI (Pro), WebLoadUI Pro, NeoLoad, APPPerfect B.      Needs extra work JMeter (The only free one in this category), Visual Studio Ultimate C.      No Support eValid, SoapUI+LoadUI FREE , WebLoadUI Free
Web Load Testing	Testing a website or webpage (which can be static or dynamic). This test usually uses various protocols and is designed to determine the server’s behavior when several users are using the website. Each page may contain different requests and elements and the test results depends the defined scenario. A.      Full Support All, except SoapUI+LoadUI (Pro and Free) B.      Needs extra work eValid (Browser based, recommends local tests, limited users), SoapUI+LoadUI Pro: Although these tools are not designed for Web load testing, but it is possible to mimic browser behavior using Groovy scripts and use LoadUI as a Web load testing tool. C.      No Support SoapUI+LoadUI Free
Web Services Testing	Simpler than Web Load testing, but needs more precise calculations and sometimes, working with different protocols. A.      Full Support JMeter, SoapUI+LoadUI (Free and Pro), Visual Studio Ultimate B.      Needs extra work APPPerfect, NeoLoad C.      No Support eValid, WebLoadUI (Free and Pro)
High Concurrent Users	This is a key feature for all load and stress tests. Some tools do not support more than a certain number of users/connections intentionally or because of their design limits. A.      High (as much as the workstation handles) MS WAS, JMeter, SoapUI+LoadUI (Free and Pro), Visual Studio Ultimate B.      Medium (Max 500 users or it’s not easy to manage) APPPrefect, NeoLoad, WebLoadUI (Pro and Free) C.      Low (Less than 100 users or it’s hard to manage.) eValid (it is browser based and will consume lots of resources per browser, especially pages designed with JavaScript)
Reports and Analysis	It would be a big advantage if the tool provides easy to understand and comprehensive reports. Some tools even don’t have a basic report template and leave all the analysis tasks on the tester. In addition, it is good to be able to export the report to an easy-editable format and extract necessary elements. A.      High Quality reports APPPerfect, SoapUI+LoadUI (Free and Pro), WebloadUI Pro, NeoLoad B.      Mediocre, but acceptable (Harder to understand and Takes more time to write final reports) WebLoadUI Free, Visual Studio Ultimate, eValid, JMeter C.      Poor reports (Needs manual data extraction and report generation) MS WAS
AJAX/JSON Support	AJAX pages and JSON requests load the content differently and some tools are weak or even unable to discover these contents. A.      Fully compatible. JSON: SoapUI+LoadUI (Free and Pro), AJAX: WebLoadUI (Free and Pro) , Both: Visual Studio Ultimate B.      Needs extra work or additional plugins. APPPerfect, JMeter+Selenium,  NeoLoad, eValid C.      No Support None
Other Capabilities	Although our focus is only on Load/Performance Test tools, however obviously it would be an advantage if the tool can provide more than that. Some important extra features are Security test, Automation, Code coverage, Functional and Unit test, Browser driving, ability to integrate with other tools, Network emulation, JS rendering, etc. A.      The tool is very powerful and has a wide usage capability. JMeter, Selenium and Visual Studio Ultimate are both offering numerous features. B.      The tool has few more extra features. SoapUI and LoadUI  (Free and Pro) can be used for unit test and security tests too. C.      The tool is only designed for the specific purpose (L&P test). APPPerfect, eValid, WebLoadUI (Free and Pro), NeoLoad
Price	Vividly, one of the main factors. This document is not discussing about top quality but irrational expensive tools like HP Load Runner, Borland Silk Performer or IBM Rational Performance Tester. A.      FREE / Open source MS WAS, JMeter + Selenium, SoapUI+LoadUI (Free), WebLoadUI (Free) B.      Less than $5000 APPPerfect ($1,000 to $1,500), eValid (We already have a license) C.      More than $5000 SoapUI+LoadUI Pro ($10,000+), WebLoadUI Pro ($7,500+), MS Studio Ultimate ($13,000+), NeoLoad ($12,000+)

Table B : Tool assessment by most required features.

Tool/Feature

Protocol Support

Rec & Playback

Scripting

Correlation

Parameterization

Server Monitoring

Distributed Testing

Web Load Test

Web Services

Concurrent Users

Reports/ Analysis

AJAX/JSON Support

Other capabilities

Price

APPPerfect

B

A

B

B

A

A

A

A

B

B

A

B

C

B

eValid

C

A

B

B

B

B

C

B

C

B

B

B

C

B

JMETER + Selenium

B

A

A

A

A

B

B

A

A

A

B

B

A

A

SoapUI+LoadUI (Pro Versions)

B

A

A

B

A

A

A

B

A

A

A

A

B

C

SoapUI+LoadUI (Free Versions)

B

B

B

B

A

C

C

C

A

A

A

A

B

A

WebLoadUI Pro

B

B

C

A

A

A

A

A

C

B

A

A

C

C

WebLoadUI Free

B

B

C

A

B

C

C

A

C

B

B

A

C

A

VS Ultimate

A

A

A

B

A

B

B

A

A

A

B

A

A

C

NeoLoad

B

A

B

B

A

A

A

A

B

B

A

A

C

C

Table C : Prices, Cons and Pros

Tool	Pros	Cons	Price
APPPerfect	- Easy to use -  Java compatible - Can add more virtual users ($1 per VU)	- Weak REST support - Poor Documentation - Poor customer support	$950 for 500 VU + $100/year
eValid	- Easy to use - License is already available	- No Web Service support - Is browser-based and consumes extra resources. - Many limitations (link). - Poor reports	Unknown
JMeter + Selenium Webdriver (Free) JMeter is the main tool for L&P testing purposes. Selenium can be used for better robustness.	- VERY flexible, Java compatible - Distributed testing - Server monitoring - Can be used for other purposes and tests. - Free	- No customer support - Poor documentation - Hard to use and integrate - Poor reports - No standard UI	Free
SoapUI + LoadUI Pro	- Comprehensible reports - Java/Groovy Compatible - Remote agents - Very user friendly	-  Very expensive (LoadUI Pro) - Poor documentation (Specially for scripting)	SoapUI $400/year LoadUI $9900/year
SoapUI + LoadUI Free	- Comprehensible reports - Java/Groovy Compatible - Easy to use - Can be worked around to make it more flexible. - Free	- No live customer support - Poor Documentation (Specially for scripting) - Limited Scripting	Free
WebloadUI Pro	- Up to 500 concurrent users (and more) - Good Documentation - Fairly easy to use	- No support for web services - No Scripting - Very expensive	$7500 /year
WebloadUI Free	-  Up to 300 concurrent users - Good Documentation - Fairly easy to use. - Reliable - Free	- No live customer support - No web service test - Weak parameterization, Not data driven. - No scripting - Weak reports	Free
Visual Studio Ultimate	- Excellent Documentation - Very Flexible - One time purchase - Extensive additional features including other types of Tests, ALM, Team Foundation, Full IDE.	- Not user friendly, needs experience. - Not good for Java Unit and experimental testing. - Very expensive (but perpetual) - Limited live customer  support (Max 4 times per license)	$13300 + $4250 upgrade
Neoload	- IP Spoofing (but cloud-based) - Remote Agents	- Low virtual users - Recommends outside cloud servers - Weak REST support - Very expensive	$12000 for 100 VU

Conclusion and Suggestions

There are different approaches to decide which tools are the best.

Here we tried to categorize possible approaches based on what the most important factors could be:

1-      QA team wants the tool to be “Easy to use” and “very cost effective”:

 This can’t be done easily. We will need to use different tools for different purposes. My best suggestion for this approach would be:

·         SoapUI (Pro) + LoadUI (Free) for Web Services + WebLoadUI (Free) for Web pages.

Reason:  These three tools are known as one of the best solutions in most Test communities and forums, because they are easier, cheaper and more supported than JMeter. Each LoadUI (Free) station can handle at least 5K REST requests/sec (link) and WebloadUI (Free) can load up to 300 concurrent (limited – anonymous) users.

Problems:

a.        There are limits in using groovy scripts with free versions (no data driven, loops, libraries)

b.       Weak Parameterization (especially if we want to use WebLoadUI  for user-based scenarios)

c.        No Distributed testing and Agents. (That would be a concern for more than 5K req/sec or for geographical tests to have real-world results).

d.       There is no guaranty that the free versions remain reliable after few years. The reason is, the tool is Java-based and new versions of Java needs new version of the free tool. Old version of SoapUI and LoadUI are not compatible with our present systems and are not being supported.

2-      QA team wants the tool to be “Reliable”, “Powerful (Flexible)” and “Very cost effective”:  

The answer is using several few free tools:

·         JMeter as the main framework with additional plugins.

·         If necessary, Selenium, JUnit or TestNG will be used to give more flexibility to JMeter

·         Occasionally, other free tools (SoapUI+LoadUI+WebLoadUI) will help to fill the small gaps and make tasks easier.

Reason: JMeter is one of the most powerful and (free) tools available from Apache.org it is being used widely by webmasters, testers and developers for different purposes. Also there are lots plugins that improve JMeter dramatically (some of them are not free but not expensive).There are many forums and communities that can provide information and support.

Problems:

a.       JMeter (and other relevant tools) is not easy to learn and teach, there are no good documentation and support at all.

b.      Reports are harder to create and are more time consuming.

c.       Maintenance is needed regularly to keep the scripts updated (compatible with new versions)

3-      QA team wants the tool to be” Reliable”, “Easy to use” and “Flexible” but not too expensive:

 In this case, there are two following suggestions:

SoapUI (Pro) + LoadUI (Pro) + WebLoadUI (Free/Pro) (Total: around $10000 /year and $18000/year with WebloadUI Pro)

Reason: Since most of our projects are related to high-volume web services, LoadUI Pro can make a great difference because we will have distributed testing, remote agents, server monitoring and heavy scripting. On the other hand, as we don’t need to test very complex and high-volume Web load testing, WebloadUI (Free) can fairly satisfy our needs (with some less-important limits that is explained above).  I can even say the  Pro version of above three tools are comparable with HP LoadRunner (and even are better than LR if accompany with AQtime Pro). Another advantages are their ease of use, user friendly interface and more comprehensible reports.

Problem:

a.       Tools can’t be used for automation, Web page smoke test and code coverage purposes,  (but still are good enough for Web Service functional tests and some security checks.)

b.      Scripts won’t work on free versions. Therefore regression testing would be hard if the license get expired (Smart Bear products are based on one year license agreements). However, we still will be able to use those scripts for a fast regression test on trial versions, but with some limits or unforeseen issues.

·        Visual Studio Ultimate (around $13,000 for perpetual full license)

Reason: Microsoft Visual Studio Ultimate offers a wide variety of Test tools and libraries. It can be used to design very complicated tests in which the only limit would be our client/server/networking hardware. We can even use selenium web driver as an extra bonus. We also can link JIRA to TFS (team foundation server) to track and update defect easier and more effective.

Problems:

·         Visual Studio is not as easy as first option and needs expertise.

·         It is not good for Java app unit testing (which is not relevant to the subject of this research)