STRESS,
LOAD, VOLUME, PERFORMANCE AND BENCHMARK
TESTING TOOL
EVALUATION AND COMPARISON
1- Common advantages
Here are the important
features that BOTH tools are supporting equally:
-
Unlimited users:
Both tools can
handle unlimited users. Most other load testing tools intentionally limit their
services to a certain number of concurrent virtual users.
-
Correlation (Automatic
testing the actual results against expected results)
Correlation
ability in load testing tools, helps to have a simple “functional testing”
during the load test to make sure all pages and responses are as same as what
we have recorded before. In other words, it compares actual results with
expected results. This feature saves
time, energy and reduces mistakes.
-
Parameterization
This is a key feature of a dynamic test. Having
this option enables us to make the test more flexible based on dynamic results.
It is also the vital part of a good data-driven test.
-
Scenario execution
It is very beneficial if we can run different usage scenarios with
various types of users. Both tools provide lots of option to do that.
-
Automatic transaction
measurements.
Recording all necessary metrics (i.e. response times, data traffic,
packet sizes, etc.)
-
Real-Time analysis
graphs (Multiple views)
During long tests such as stress testing, it would be very handful if we
can see the live results in a workbench. This can help us to interrupt a test
if something is wrong, without having to wait the test finishes. It also helps
to do better determination for system’s behavior during some “events”.
-
Additional agents
(External load injectors)
Load injectors
(Agents) can be used for various purposes such as:
·
When we need to put a
massive load on a system that one test machine can’t produce that load alone.
·
If we need to see the
system’s behavior in an environment much closer to reality, because each agent
will be setup in a different computer (even in offshore) and naturally would
have different IP address, ping latencies, data flow, etc.
·
If the client(s) locate in
another geographical area while our server is somewhere else, we can put agents
remotely and do the load and performance tests “from there”.
2-
Differences, Cons and Pros
|
|
Visual Studio
Ultimate 2012 + MSDN
|
LoadUI Pro (+
SoapUI Pro)
|
|
Cost
|
Good: Tool’s License is perpetual and we won’t need to
upgrade it to a new version for a long time (unless change to .Net
development). Yearly MSDN renewal may be required for TFS.
$10,900 to $13,300 (13300 is the vendor’s price but
there are some other stores or licensors who can offer a bit less)
Renewal:
$800 to $4300 ($800/yr is enough for TFS Server
maintenance)
|
Bad: License will
expire after one year and requires renewal with almost the same price. There
are Two-Year and Three-Year licenses available but the cost would be
multiplied as well.
Standard license:
$8900 to 9900 for one seat (user), per year.
Floating license:
$17900 to 19800 one floating seat, per year.
|
|
Licensing
limitations & restrictions.
|
Bad: License is user-based and will be provided to one individual user.
Therefore, we are “not allowed” to share the license to others. However, one
license can be used to install the VS on many PCs (Work, Home, Portable …).
|
Worse: License is not only user-based (like VS), it
also is limited to machines. We will be allowed to “move” a license from a
computer to another one, only 2 times per year that means we can install the
tool on up to 3 machines.
|
|
Test
function coverage
|
Good: Visual studio
covers all Load and performance testing functions we need for Web Services
and Web pages.
|
Bad: LoadUI is
VERY powerful to test web services, but it is very weak and limited
for Web load testing. There is the web load feature in the LoadUI but it is
good only for very simple pages with few elements. To do a Web load testing
as good as what Visual Studio offers, we’d need to use LoadUIWeb Pro ($4900
to $7500 per year)
|
|
Bad: Weak Ramp downs
(it can be done but needs to write
heavy code for load profile plugin)
Needs
scripting for randomized or distributed scenarios.
|
Good: Very
comprehensive and flexible tools for Ramps and distributed scenarios.
|
|
Scripting
|
Bad: We’ll need to use .Net scripting to create good
test cases.
|
Worse: Groovy is the
official scripting language, which is almost Java with some occasional
differences.
|
|
Good: There is outstanding documentation
for .Net programming with or even without MSDN available.
|
Bad: Poor documentation for using groovy scripts in the SoapUI is available.
|
|
Server
monitoring
|
Bad: Easy server monitoring is only available for
windows-based servers. For other webservers that use different operating
systems (e.g. Linux), we need either to write codes to read the system
counters dynamically or use another (not-integrated) resource monitoring tool
during the test.
|
Good: Supports Server
monitoring for all common operating systems, needs to install Net-SNMP on
Apache.
|
|
Support
|
-
Full MSDN library and forum support (one
year)
-
Four times live supports with a
professional.
-
There is lots of external website to
help.
|
-
Pro user forum and offline support.
-
Live support and Ticket desk.
-
Limited resources are available in
other location.
|
|
Ease
of usage
|
Bad: Needs .Net development experience and more programming
knowledge.
|
Good: By providing more
visual and Drag & Drop features
and ready-to-use libraries, less coding is needed.
|
3-
Other aspects:
a.
LoadUI other benefits:
-
Includes a Free
SoapUI Pro. That means if you buy a license of the LoadUI Pro, you will get a
free SoapUI pro as well (around $350 value)
-
LoadUI provides
better reports with flexible graphs and customized data.
b.
LoadUI other flaws:
-
Scripts created in
Pro versions can’t be used in free versions. Therefore, we won’t be able to use
our files if the license expires (Although we can temporary use trial
versions). In addition this forces us to run our tests only from one system. Visual
Studio Ultimate has the same problem. Please read the provided solution in
“d. Visual Studio Ultimate Flaws”.
c. Visual
Studio Ultimate benefits:
-
$150/Mo Credits for
Windows Azure cloud and VM processing power (that is enough for our normal test
purposes). This can help to avoid purchasing unnecessary powerful test machines
that be used rarely.
-
Ability to perform
Unit testing, Functional testing and Code coverage testing on developed web
pages with HTML, JavaScript and CSS
-
Visual Studio can be
used for many other development purposes in the future. The only flaw in
this benefit is not being Java compatible.
o It comes with one year full MSDN subscription that
provides Team foundation server (TFS) + one client access license. Please read
section 4 for more information about TFS.
Other following provided features are not necessary. To maintain only
our Team Foundation Server (TFS) in next years, we can renew MSDN subscription
without following extras ($800 year):
o Exchange server
o SQL Server
o Office 360
o Windows, Windows Server
o Team foundation service with 5+ CAL
d. Visual
Studio Ultimate flaws:
-
Same as LoadUI pro,
we can’t run Load and Performance tests in other machines that have no
Visual Studio Ultimate installed. Other tests can be used in Visual
Studio Test Professional.
There is a solution which is having a “portable
test machine”, like a good laptop (intel i7 + 12GB RAM + SSD HDD). You can
install Visual Studio (or LoadUI) on it and provide it to anyone who wants to
perform the test in the facility.
4-
Important note about TFS (Team Foundation Server/Service):
TFS is a powerful and the main part of Microsoft application
life cycle management (ALM). It integrates with SharePoint and Outlook and
gives ability to:
-
Version control, for managing source code and other
deliverables that require versioning.
-
Work item tracking, for keeping track of such things
as defects, requirements, tasks, and scenarios.
-
Agile development: Compatible with agile methodologies
and common agile frameworks like SCRUM, KANBAN, XP and Crystal.
-
Project management functions, which allow the shaping
of a team project based on a user-specifiable software process, and which
enable planning and tracking using Microsoft Excel and Microsoft Project.
-
Team build, for enabling a common process for building
executable products.
-
Data collection and reporting, which aid in the
assessment of a team project's state, based on information gleaned from Team
Foundation Server tools.
-
The Team Project Portal, which provides a central
point of communication for a team project packaged as a Microsoft Windows
SharePoint Services site.
Although TFS is included with the Visual Studio,
but the tricky part is that provides with 1 CAL (Client Access
License) and we will need additional CALs for more CALs. (Cost of each CAL
is $500, less for volume purchases.).
The bright side of the TFS and CALs:
·
Each CAL can be
used by more than one user (2-3 in a group with some arrangements).
·
Users (e.g. Stake holders) can see reports and output
analysis data without having a CAL.
