Visual Studio Ultimate vs SoapUI+LoadUI

STRESS, LOAD, VOLUME, PERFORMANCE AND BENCHMARK

TESTING TOOL EVALUATION AND COMPARISON

1-    Common advantages

Here are the important features that BOTH tools are supporting equally:

-          Unlimited users:

Both tools can handle unlimited users. Most other load testing tools intentionally limit their services to a certain number of concurrent virtual users.

-          Correlation (Automatic testing the actual results against expected results)

Correlation ability in load testing tools, helps to have a simple “functional testing” during the load test to make sure all pages and responses are as same as what we have recorded before. In other words, it compares actual results with expected results.  This feature saves time, energy and reduces mistakes.

-          Parameterization

This is a key feature of a dynamic test. Having this option enables us to make the test more flexible based on dynamic results. It is also the vital part of a good data-driven test.

-          Scenario execution

It is very beneficial if we can run different usage scenarios with various types of users. Both tools provide lots of option to do that.

-          Automatic transaction measurements.

Recording all necessary metrics (i.e. response times, data traffic, packet sizes, etc.)

-          Real-Time analysis graphs (Multiple views)

During long tests such as stress testing, it would be very handful if we can see the live results in a workbench. This can help us to interrupt a test if something is wrong, without having to wait the test finishes. It also helps to do better determination for system’s behavior during some “events”.

-          Additional agents (External load injectors)

Load injectors (Agents) can be used for various purposes such as:

·         When we need to put a massive load on a system that one test machine can’t produce that load alone.

·         If we need to see the system’s behavior in an environment much closer to reality, because each agent will be setup in a different computer (even in offshore) and naturally would have different IP address, ping latencies, data flow, etc.

·          If the client(s) locate in another geographical area while our server is somewhere else, we can put agents remotely and do the load and performance tests “from there”.

2-    Differences, Cons and Pros

	Visual Studio Ultimate 2012 + MSDN	LoadUI Pro (+ SoapUI Pro)
Cost	Good: Tool’s License is perpetual and we won’t need to upgrade it to a new version for a long time (unless change to .Net development). Yearly MSDN renewal may be required for TFS. $10,900 to $13,300 (13300 is the vendor’s price but there are some other stores or licensors who can offer a bit less) Renewal: $800 to $4300 ($800/yr is enough for TFS Server maintenance) http://msdn.microsoft.com/subscriptions/hh442902.aspx	Bad: License will expire after one year and requires renewal with almost the same price. There are Two-Year and Three-Year licenses available but the cost would be multiplied as well. Standard license: $8900 to 9900 for one seat (user), per year. Floating license: $17900 to 19800 one floating seat, per year. http://www.loadui.org/Store-Info/pro-store.html
Licensing limitations & restrictions.	Bad: License is user-based and will be provided to one individual user. Therefore, we are “not allowed” to share the license to others. However, one license can be used to install the VS on many PCs (Work, Home, Portable …). http://www.microsoft.com/en-us/download/details.aspx?id=13350	Worse:  License is not only user-based (like VS), it also is limited to machines. We will be allowed to “move” a license from a computer to another one, only 2 times per year that means we can install the tool on up to 3 machines. http://www.loadui.org/Store-Info/purchasing-faq.html
Test function coverage	Good: Visual studio covers all Load and performance testing functions we need for Web Services and Web pages.	Bad:  LoadUI is VERY powerful to test web services, but it is very weak and limited for Web load testing. There is the web load feature in the LoadUI but it is good only for very simple pages with few elements. To do a Web load testing as good as what Visual Studio offers, we’d need to use LoadUIWeb Pro ($4900 to $7500 per year)
	Bad: Weak  Ramp downs (it can be  done but needs to write heavy code for load profile plugin) Needs scripting for randomized or distributed scenarios.	Good: Very comprehensive and flexible tools for Ramps and distributed scenarios.
Scripting	Bad: We’ll need to use .Net scripting to create good test cases.	Worse: Groovy is the official scripting language, which is almost Java with some occasional differences.
	Good: There is outstanding documentation for .Net programming with or even without MSDN available.	Bad: Poor documentation for using groovy scripts in the SoapUI is available.
Server monitoring	Bad: Easy server monitoring is only available for windows-based servers. For other webservers that use different operating systems (e.g. Linux), we need either to write codes to read the system counters dynamically or use another (not-integrated) resource monitoring tool during the test.	Good: Supports Server monitoring for all common operating systems, needs to install Net-SNMP on Apache.
Support	-          Full MSDN library and forum support (one year) -          Four times live supports with a professional. -          There is lots of external website to help.	-          Pro user forum and offline support. -          Live support and Ticket desk. -          Limited resources are available in other location.
Ease of usage	Bad: Needs .Net development experience and more programming knowledge.	Good: By providing more visual and Drag &  Drop features and  ready-to-use libraries,  less coding is needed.

3-    Other aspects:

a.      LoadUI other benefits:

-          Includes a Free SoapUI Pro. That means if you buy a license of the LoadUI Pro, you will get a free SoapUI pro as well (around $350 value)

-          LoadUI provides better reports with flexible graphs and customized data.

b.      LoadUI other flaws:

-           Scripts created in Pro versions can’t be used in free versions. Therefore, we won’t be able to use our files if the license expires (Although we can temporary use trial versions). In addition this forces us to run our tests only from one system. Visual Studio Ultimate has the same problem. Please read the provided solution in “d. Visual Studio Ultimate Flaws”.

c.       Visual Studio Ultimate benefits:

-          $150/Mo Credits for Windows Azure cloud and VM processing power (that is enough for our normal test purposes). This can help to avoid purchasing unnecessary powerful test machines that be used rarely.

-          Ability to perform Unit testing, Functional testing and Code coverage testing on developed web pages with HTML, JavaScript and CSS

-          Visual Studio can be used for many other development purposes in the future. The only flaw in this benefit is not being Java compatible.

o   It comes with one year full MSDN subscription that provides Team foundation server (TFS) + one client access license. Please read section 4 for more information about TFS.

Other following provided features are not necessary. To maintain only our Team Foundation Server (TFS) in next years, we can renew MSDN subscription without following extras ($800 year):

o   Exchange server

o   SQL Server

o   Office 360

o   Windows, Windows Server

o   Team foundation service with 5+ CAL

d.      Visual Studio Ultimate flaws:

-          Same as LoadUI pro, we can’t run Load and Performance tests in other machines that have no Visual Studio Ultimate installed. Other tests can be used in Visual Studio Test Professional.

There is a solution which is having a “portable test machine”, like a good laptop (intel i7 + 12GB RAM + SSD HDD). You can install Visual Studio (or LoadUI) on it and provide it to anyone who wants to perform the test in the facility.  

4-    Important note about TFS (Team Foundation Server/Service):

TFS is a powerful and the main part of Microsoft application life cycle management (ALM). It integrates with SharePoint and Outlook and gives ability to:

-          Version control, for managing source code and other deliverables that require versioning.

-          Work item tracking, for keeping track of such things as defects, requirements, tasks, and scenarios.

-          Agile development: Compatible with agile methodologies and common agile frameworks like SCRUM, KANBAN, XP and Crystal.

-          Project management functions, which allow the shaping of a team project based on a user-specifiable software process, and which enable planning and tracking using Microsoft Excel and Microsoft Project.

-          Team build, for enabling a common process for building executable products.

-          Data collection and reporting, which aid in the assessment of a team project's state, based on information gleaned from Team Foundation Server tools.

-          The Team Project Portal, which provides a central point of communication for a team project packaged as a Microsoft Windows SharePoint Services site.

Although TFS is included with the Visual Studio, but the tricky part is that provides with 1 CAL (Client Access License) and we will need additional CALs for more CALs. (Cost of each CAL is $500, less for volume purchases.).

The bright side of the TFS and CALs:

·          Each CAL can be used by more than one user (2-3 in a group with some arrangements).

·         Users (e.g. Stake holders) can see reports and output analysis data without having a CAL.