Visual Studio Ultimate vs SoapUI+LoadUI

STRESS, LOAD, VOLUME, PERFORMANCE AND BENCHMARK

TESTING TOOL EVALUATION AND COMPARISON

1-    Common advantages

Here are the important features that BOTH tools are supporting equally:

-          Unlimited users:

Both tools can handle unlimited users. Most other load testing tools intentionally limit their services to a certain number of concurrent virtual users.

-          Correlation (Automatic testing the actual results against expected results)

Correlation ability in load testing tools, helps to have a simple “functional testing” during the load test to make sure all pages and responses are as same as what we have recorded before. In other words, it compares actual results with expected results.  This feature saves time, energy and reduces mistakes.

-          Parameterization

This is a key feature of a dynamic test. Having this option enables us to make the test more flexible based on dynamic results. It is also the vital part of a good data-driven test.

-          Scenario execution

It is very beneficial if we can run different usage scenarios with various types of users. Both tools provide lots of option to do that.

-          Automatic transaction measurements.

Recording all necessary metrics (i.e. response times, data traffic, packet sizes, etc.)

-          Real-Time analysis graphs (Multiple views)

During long tests such as stress testing, it would be very handful if we can see the live results in a workbench. This can help us to interrupt a test if something is wrong, without having to wait the test finishes. It also helps to do better determination for system’s behavior during some “events”.

-          Additional agents (External load injectors)

Load injectors (Agents) can be used for various purposes such as:

·         When we need to put a massive load on a system that one test machine can’t produce that load alone.

·         If we need to see the system’s behavior in an environment much closer to reality, because each agent will be setup in a different computer (even in offshore) and naturally would have different IP address, ping latencies, data flow, etc.

·          If the client(s) locate in another geographical area while our server is somewhere else, we can put agents remotely and do the load and performance tests “from there”.

2-    Differences, Cons and Pros

	Visual Studio Ultimate 2012 + MSDN	LoadUI Pro (+ SoapUI Pro)
Cost	Good: Tool’s License is perpetual and we won’t need to upgrade it to a new version for a long time (unless change to .Net development). Yearly MSDN renewal may be required for TFS. $10,900 to $13,300 (13300 is the vendor’s price but there are some other stores or licensors who can offer a bit less) Renewal: $800 to $4300 ($800/yr is enough for TFS Server maintenance) http://msdn.microsoft.com/subscriptions/hh442902.aspx	Bad: License will expire after one year and requires renewal with almost the same price. There are Two-Year and Three-Year licenses available but the cost would be multiplied as well. Standard license: $8900 to 9900 for one seat (user), per year. Floating license: $17900 to 19800 one floating seat, per year. http://www.loadui.org/Store-Info/pro-store.html
Licensing limitations & restrictions.	Bad: License is user-based and will be provided to one individual user. Therefore, we are “not allowed” to share the license to others. However, one license can be used to install the VS on many PCs (Work, Home, Portable …). http://www.microsoft.com/en-us/download/details.aspx?id=13350	Worse:  License is not only user-based (like VS), it also is limited to machines. We will be allowed to “move” a license from a computer to another one, only 2 times per year that means we can install the tool on up to 3 machines. http://www.loadui.org/Store-Info/purchasing-faq.html
Test function coverage	Good: Visual studio covers all Load and performance testing functions we need for Web Services and Web pages.	Bad:  LoadUI is VERY powerful to test web services, but it is very weak and limited for Web load testing. There is the web load feature in the LoadUI but it is good only for very simple pages with few elements. To do a Web load testing as good as what Visual Studio offers, we’d need to use LoadUIWeb Pro ($4900 to $7500 per year)
	Bad: Weak  Ramp downs (it can be  done but needs to write heavy code for load profile plugin) Needs scripting for randomized or distributed scenarios.	Good: Very comprehensive and flexible tools for Ramps and distributed scenarios.
Scripting	Bad: We’ll need to use .Net scripting to create good test cases.	Worse: Groovy is the official scripting language, which is almost Java with some occasional differences.
	Good: There is outstanding documentation for .Net programming with or even without MSDN available.	Bad: Poor documentation for using groovy scripts in the SoapUI is available.
Server monitoring	Bad: Easy server monitoring is only available for windows-based servers. For other webservers that use different operating systems (e.g. Linux), we need either to write codes to read the system counters dynamically or use another (not-integrated) resource monitoring tool during the test.	Good: Supports Server monitoring for all common operating systems, needs to install Net-SNMP on Apache.
Support	-          Full MSDN library and forum support (one year) -          Four times live supports with a professional. -          There is lots of external website to help.	-          Pro user forum and offline support. -          Live support and Ticket desk. -          Limited resources are available in other location.
Ease of usage	Bad: Needs .Net development experience and more programming knowledge.	Good: By providing more visual and Drag &  Drop features and  ready-to-use libraries,  less coding is needed.

3-    Other aspects:

a.      LoadUI other benefits:

-          Includes a Free SoapUI Pro. That means if you buy a license of the LoadUI Pro, you will get a free SoapUI pro as well (around $350 value)

-          LoadUI provides better reports with flexible graphs and customized data.

b.      LoadUI other flaws:

-           Scripts created in Pro versions can’t be used in free versions. Therefore, we won’t be able to use our files if the license expires (Although we can temporary use trial versions). In addition this forces us to run our tests only from one system. Visual Studio Ultimate has the same problem. Please read the provided solution in “d. Visual Studio Ultimate Flaws”.

c.       Visual Studio Ultimate benefits:

-          $150/Mo Credits for Windows Azure cloud and VM processing power (that is enough for our normal test purposes). This can help to avoid purchasing unnecessary powerful test machines that be used rarely.

-          Ability to perform Unit testing, Functional testing and Code coverage testing on developed web pages with HTML, JavaScript and CSS

-          Visual Studio can be used for many other development purposes in the future. The only flaw in this benefit is not being Java compatible.

o   It comes with one year full MSDN subscription that provides Team foundation server (TFS) + one client access license. Please read section 4 for more information about TFS.

Other following provided features are not necessary. To maintain only our Team Foundation Server (TFS) in next years, we can renew MSDN subscription without following extras ($800 year):

o   Exchange server

o   SQL Server

o   Office 360

o   Windows, Windows Server

o   Team foundation service with 5+ CAL

d.      Visual Studio Ultimate flaws:

-          Same as LoadUI pro, we can’t run Load and Performance tests in other machines that have no Visual Studio Ultimate installed. Other tests can be used in Visual Studio Test Professional.

There is a solution which is having a “portable test machine”, like a good laptop (intel i7 + 12GB RAM + SSD HDD). You can install Visual Studio (or LoadUI) on it and provide it to anyone who wants to perform the test in the facility.  

4-    Important note about TFS (Team Foundation Server/Service):

TFS is a powerful and the main part of Microsoft application life cycle management (ALM). It integrates with SharePoint and Outlook and gives ability to:

-          Version control, for managing source code and other deliverables that require versioning.

-          Work item tracking, for keeping track of such things as defects, requirements, tasks, and scenarios.

-          Agile development: Compatible with agile methodologies and common agile frameworks like SCRUM, KANBAN, XP and Crystal.

-          Project management functions, which allow the shaping of a team project based on a user-specifiable software process, and which enable planning and tracking using Microsoft Excel and Microsoft Project.

-          Team build, for enabling a common process for building executable products.

-          Data collection and reporting, which aid in the assessment of a team project's state, based on information gleaned from Team Foundation Server tools.

-          The Team Project Portal, which provides a central point of communication for a team project packaged as a Microsoft Windows SharePoint Services site.

Although TFS is included with the Visual Studio, but the tricky part is that provides with 1 CAL (Client Access License) and we will need additional CALs for more CALs. (Cost of each CAL is $500, less for volume purchases.).

The bright side of the TFS and CALs:

·          Each CAL can be used by more than one user (2-3 in a group with some arrangements).

·         Users (e.g. Stake holders) can see reports and output analysis data without having a CAL.

Comparison research for Load, Performance and Stress test tools.

This report compares most affordable and yet practical tools that can be used for Load and Performance testing tasks in xxx.  Nine tools were selected and determined based on all necessary aspect to perform high quality, effective and accurate tests.

The main factors which are necessary for the selected tools are:

·         Can be used for a long time (Reliable).

·         Not very hard to learn and teach (User friendly)

·         Powerful and accurate enough to cover all of our needs. (Compatible)

·         Preferably, have more features and abilities for other purposes (Other types of tests, Development, Life Cycle Management, Bug tracking, etc.)

·         Provides high quality and understandable reports to prevent exhaustive (and costly) manual data analysis.

·         Have a fair support (or very good documentation). For live support, the time zone (and support desk availability) is one of the concerns.

·         Not too expensive while there are better solutions with better price! (Affordable). In this document, less than $2000 is considered as “Low Price”, Up to $10000 is called as “Expensive” and more than $10000 is considered as “Very Expensive”.

Unfortunately, there is no particular tool that covers all above factors. Therefore, we need to use a combination of tools in a way that gives us best performance and quality.

Quick links:

-          Table A: Explanation of features and rates in Table B

-          Table B: Tool assessment by most required features.

-          Table C: Prices, Cons and Pros

-          Conclusion and Suggestions.

Table A: Explanation of features and rates in Table B.

Feature	Explanation	Rating meanings.
Supported Protocols	Shows if the tool supports a wide variety of protocols which may be used by our developers. Some protocols like HTTP, HTTPS, TCP, FTP are vital. Other important protocols are: JSON, REST, SOAP, WPS, WSDL, XML. A.      Very good: Only the Visual Studio Ultimate can support lots of protocols. ( That is one of the advantages of “very expensive” tools ), it is however, very hard to work when LDAP is being used. JMeter has more flexibility with LDAP. B.      Good enough to support our needs:          All except MS Web App Stress Test C.      Poor: D.      eValid (No SOAP, REST, JSON, Java applets)
Record and Playback	The tool’s ability to record series of actions and imitate them as a playback. This feature can save time and energy and reduces the human mistakes. A.      Professional, Fast and Easy (for an expert) APPPerfect Load Test, MS WAS, JMeter+Selenium, SoapUI+LoadUI (Pro), Visual Studio Ultimate, NeoLoad B.      Acceptable, Not so easy and fast. SoapUI+LoadUI (Free), WebLoadUI (Both Free and Pro), eValid C.      Not Available None
Scripting	Using scripts helps to use the tool more effective and flexible. This can be an integrated IDE or by using a plugin or third-party tool. A.      Good script support, well documented. JMeter+ Selenium , SoapUI+LoadUI (Pro), Visual Studio Ultimate B.      Fair script support, Poor documentation. APPPerfect , eValid , SoapUI+LoadUI (Free) , NeoLoad C.      No script support. WebLoadUI (Free and Pro)
Correlation	Ability of comparing recorded results with playback results. This can be automatic or needs to add manual assertions. Very necessary to perform effective tests. A.      Automatic and Powerful Correlation, or easy to do it manually. WebLoadUI (Pro and Free) B.      Can be done using scripts or with some efforts. All others C.      Not supported. None
Parameterization	This feature facilitates the tester to use dynamic values in different ways and create data driven tests. A.      Fully supported All except WebloadUI (Free) and MS WAS B.      Supports with limits WebloadUI (Free) , eValid (weak parameterization) C.      Not Supported None
Server Monitoring	Having the option to monitor server’s resources during a stress test or capacity test, will be a huge advantage. However, this doesn’t mean that having test tools with server monitoring feature is the must. We can use another free resource monitor tool for server surveillance purposes and it would be fairly close to the more accurate report generated from the test tool. A.      Supported (integrated) by the Tool SoapUI+LoadUI (Pro), WebLoadUI Pro, NeoLoad, APPPerfect B.      Can be done with additional plugins or needs some efforts. JMeter (The only free one in this category), Visual Studio Ultimate C.      The tool doesn’t support the feature (but we still can use other monitoring tools) SoapUI+LoadUI FREE , WebLoadUI Free, eValid
Distributed Testing	This feature gives the ability of having multiple load sources at the same time, which is being controlled by the “main” tool. This helps when we want to have a tremendous number of hits/users that can’t be handled by one station. Also it makes the test environment closer to the real life since each server has its own IP address, ping latencies, packet loss and more. We also can use not supported tools on several machines simultaneously, get the reports separately and merge all reports to one; But this would be not an easy task and takes time to create an individual report. A.      Full Support SoapUI+LoadUI (Pro), WebLoadUI Pro, NeoLoad, APPPerfect B.      Needs extra work JMeter (The only free one in this category), Visual Studio Ultimate C.      No Support eValid, SoapUI+LoadUI FREE , WebLoadUI Free
Web Load Testing	Testing a website or webpage (which can be static or dynamic). This test usually uses various protocols and is designed to determine the server’s behavior when several users are using the website. Each page may contain different requests and elements and the test results depends the defined scenario. A.      Full Support All, except SoapUI+LoadUI (Pro and Free) B.      Needs extra work eValid (Browser based, recommends local tests, limited users), SoapUI+LoadUI Pro: Although these tools are not designed for Web load testing, but it is possible to mimic browser behavior using Groovy scripts and use LoadUI as a Web load testing tool. C.      No Support SoapUI+LoadUI Free
Web Services Testing	Simpler than Web Load testing, but needs more precise calculations and sometimes, working with different protocols. A.      Full Support JMeter, SoapUI+LoadUI (Free and Pro), Visual Studio Ultimate B.      Needs extra work APPPerfect, NeoLoad C.      No Support eValid, WebLoadUI (Free and Pro)
High Concurrent Users	This is a key feature for all load and stress tests. Some tools do not support more than a certain number of users/connections intentionally or because of their design limits. A.      High (as much as the workstation handles) MS WAS, JMeter, SoapUI+LoadUI (Free and Pro), Visual Studio Ultimate B.      Medium (Max 500 users or it’s not easy to manage) APPPrefect, NeoLoad, WebLoadUI (Pro and Free) C.      Low (Less than 100 users or it’s hard to manage.) eValid (it is browser based and will consume lots of resources per browser, especially pages designed with JavaScript)
Reports and Analysis	It would be a big advantage if the tool provides easy to understand and comprehensive reports. Some tools even don’t have a basic report template and leave all the analysis tasks on the tester. In addition, it is good to be able to export the report to an easy-editable format and extract necessary elements. A.      High Quality reports APPPerfect, SoapUI+LoadUI (Free and Pro), WebloadUI Pro, NeoLoad B.      Mediocre, but acceptable (Harder to understand and Takes more time to write final reports) WebLoadUI Free, Visual Studio Ultimate, eValid, JMeter C.      Poor reports (Needs manual data extraction and report generation) MS WAS
AJAX/JSON Support	AJAX pages and JSON requests load the content differently and some tools are weak or even unable to discover these contents. A.      Fully compatible. JSON: SoapUI+LoadUI (Free and Pro), AJAX: WebLoadUI (Free and Pro) , Both: Visual Studio Ultimate B.      Needs extra work or additional plugins. APPPerfect, JMeter+Selenium,  NeoLoad, eValid C.      No Support None
Other Capabilities	Although our focus is only on Load/Performance Test tools, however obviously it would be an advantage if the tool can provide more than that. Some important extra features are Security test, Automation, Code coverage, Functional and Unit test, Browser driving, ability to integrate with other tools, Network emulation, JS rendering, etc. A.      The tool is very powerful and has a wide usage capability. JMeter, Selenium and Visual Studio Ultimate are both offering numerous features. B.      The tool has few more extra features. SoapUI and LoadUI  (Free and Pro) can be used for unit test and security tests too. C.      The tool is only designed for the specific purpose (L&P test). APPPerfect, eValid, WebLoadUI (Free and Pro), NeoLoad
Price	Vividly, one of the main factors. This document is not discussing about top quality but irrational expensive tools like HP Load Runner, Borland Silk Performer or IBM Rational Performance Tester. A.      FREE / Open source MS WAS, JMeter + Selenium, SoapUI+LoadUI (Free), WebLoadUI (Free) B.      Less than $5000 APPPerfect ($1,000 to $1,500), eValid (We already have a license) C.      More than $5000 SoapUI+LoadUI Pro ($10,000+), WebLoadUI Pro ($7,500+), MS Studio Ultimate ($13,000+), NeoLoad ($12,000+)

Table B : Tool assessment by most required features.

Tool/Feature

Protocol Support

Rec & Playback

Scripting

Correlation

Parameterization

Server Monitoring

Distributed Testing

Web Load Test

Web Services

Concurrent Users

Reports/ Analysis

AJAX/JSON Support

Other capabilities

Price

APPPerfect

B

A

B

B

A

A

A

A

B

B

A

B

C

B

eValid

C

A

B

B

B

B

C

B

C

B

B

B

C

B

JMETER + Selenium

B

A

A

A

A

B

B

A

A

A

B

B

A

A

SoapUI+LoadUI (Pro Versions)

B

A

A

B

A

A

A

B

A

A

A

A

B

C

SoapUI+LoadUI (Free Versions)

B

B

B

B

A

C

C

C

A

A

A

A

B

A

WebLoadUI Pro

B

B

C

A

A

A

A

A

C

B

A

A

C

C

WebLoadUI Free

B

B

C

A

B

C

C

A

C

B

B

A

C

A

VS Ultimate

A

A

A

B

A

B

B

A

A

A

B

A

A

C

NeoLoad

B

A

B

B

A

A

A

A

B

B

A

A

C

C

Table C : Prices, Cons and Pros

Tool	Pros	Cons	Price
APPPerfect	- Easy to use -  Java compatible - Can add more virtual users ($1 per VU)	- Weak REST support - Poor Documentation - Poor customer support	$950 for 500 VU + $100/year
eValid	- Easy to use - License is already available	- No Web Service support - Is browser-based and consumes extra resources. - Many limitations (link). - Poor reports	Unknown
JMeter + Selenium Webdriver (Free) JMeter is the main tool for L&P testing purposes. Selenium can be used for better robustness.	- VERY flexible, Java compatible - Distributed testing - Server monitoring - Can be used for other purposes and tests. - Free	- No customer support - Poor documentation - Hard to use and integrate - Poor reports - No standard UI	Free
SoapUI + LoadUI Pro	- Comprehensible reports - Java/Groovy Compatible - Remote agents - Very user friendly	-  Very expensive (LoadUI Pro) - Poor documentation (Specially for scripting)	SoapUI $400/year LoadUI $9900/year
SoapUI + LoadUI Free	- Comprehensible reports - Java/Groovy Compatible - Easy to use - Can be worked around to make it more flexible. - Free	- No live customer support - Poor Documentation (Specially for scripting) - Limited Scripting	Free
WebloadUI Pro	- Up to 500 concurrent users (and more) - Good Documentation - Fairly easy to use	- No support for web services - No Scripting - Very expensive	$7500 /year
WebloadUI Free	-  Up to 300 concurrent users - Good Documentation - Fairly easy to use. - Reliable - Free	- No live customer support - No web service test - Weak parameterization, Not data driven. - No scripting - Weak reports	Free
Visual Studio Ultimate	- Excellent Documentation - Very Flexible - One time purchase - Extensive additional features including other types of Tests, ALM, Team Foundation, Full IDE.	- Not user friendly, needs experience. - Not good for Java Unit and experimental testing. - Very expensive (but perpetual) - Limited live customer  support (Max 4 times per license)	$13300 + $4250 upgrade
Neoload	- IP Spoofing (but cloud-based) - Remote Agents	- Low virtual users - Recommends outside cloud servers - Weak REST support - Very expensive	$12000 for 100 VU

Conclusion and Suggestions

There are different approaches to decide which tools are the best.

Here we tried to categorize possible approaches based on what the most important factors could be:

1-      QA team wants the tool to be “Easy to use” and “very cost effective”:

 This can’t be done easily. We will need to use different tools for different purposes. My best suggestion for this approach would be:

·         SoapUI (Pro) + LoadUI (Free) for Web Services + WebLoadUI (Free) for Web pages.

Reason:  These three tools are known as one of the best solutions in most Test communities and forums, because they are easier, cheaper and more supported than JMeter. Each LoadUI (Free) station can handle at least 5K REST requests/sec (link) and WebloadUI (Free) can load up to 300 concurrent (limited – anonymous) users.

Problems:

a.        There are limits in using groovy scripts with free versions (no data driven, loops, libraries)

b.       Weak Parameterization (especially if we want to use WebLoadUI  for user-based scenarios)

c.        No Distributed testing and Agents. (That would be a concern for more than 5K req/sec or for geographical tests to have real-world results).

d.       There is no guaranty that the free versions remain reliable after few years. The reason is, the tool is Java-based and new versions of Java needs new version of the free tool. Old version of SoapUI and LoadUI are not compatible with our present systems and are not being supported.

2-      QA team wants the tool to be “Reliable”, “Powerful (Flexible)” and “Very cost effective”:  

The answer is using several few free tools:

·         JMeter as the main framework with additional plugins.

·         If necessary, Selenium, JUnit or TestNG will be used to give more flexibility to JMeter

·         Occasionally, other free tools (SoapUI+LoadUI+WebLoadUI) will help to fill the small gaps and make tasks easier.

Reason: JMeter is one of the most powerful and (free) tools available from Apache.org it is being used widely by webmasters, testers and developers for different purposes. Also there are lots plugins that improve JMeter dramatically (some of them are not free but not expensive).There are many forums and communities that can provide information and support.

Problems:

a.       JMeter (and other relevant tools) is not easy to learn and teach, there are no good documentation and support at all.

b.      Reports are harder to create and are more time consuming.

c.       Maintenance is needed regularly to keep the scripts updated (compatible with new versions)

3-      QA team wants the tool to be” Reliable”, “Easy to use” and “Flexible” but not too expensive:

 In this case, there are two following suggestions:

SoapUI (Pro) + LoadUI (Pro) + WebLoadUI (Free/Pro) (Total: around $10000 /year and $18000/year with WebloadUI Pro)

Reason: Since most of our projects are related to high-volume web services, LoadUI Pro can make a great difference because we will have distributed testing, remote agents, server monitoring and heavy scripting. On the other hand, as we don’t need to test very complex and high-volume Web load testing, WebloadUI (Free) can fairly satisfy our needs (with some less-important limits that is explained above).  I can even say the  Pro version of above three tools are comparable with HP LoadRunner (and even are better than LR if accompany with AQtime Pro). Another advantages are their ease of use, user friendly interface and more comprehensible reports.

Problem:

a.       Tools can’t be used for automation, Web page smoke test and code coverage purposes,  (but still are good enough for Web Service functional tests and some security checks.)

b.      Scripts won’t work on free versions. Therefore regression testing would be hard if the license get expired (Smart Bear products are based on one year license agreements). However, we still will be able to use those scripts for a fast regression test on trial versions, but with some limits or unforeseen issues.

·        Visual Studio Ultimate (around $13,000 for perpetual full license)

Reason: Microsoft Visual Studio Ultimate offers a wide variety of Test tools and libraries. It can be used to design very complicated tests in which the only limit would be our client/server/networking hardware. We can even use selenium web driver as an extra bonus. We also can link JIRA to TFS (team foundation server) to track and update defect easier and more effective.

Problems:

·         Visual Studio is not as easy as first option and needs expertise.

·         It is not good for Java app unit testing (which is not relevant to the subject of this research)