Performance testing

Contents

Testing introduction & quick guide

Introduction

Testing tools for Cyclos have been made available for download
There are two main areas for testing. Firstly the functional tests. These tests can run 'test suites' that will check if the functionalty is working correctly (for example insert an advertisement or making a payment). Functional test are always run by the development team after any code change to make sure nothing has been broken. And secondly performance tests. These tests will test the system load of the main Web or Web services interfaces under various conditions. There are different test approaches to test the performance. For example testing the 'maximum throughput' (e.g. how many payments the system can process at once) or testing a functional flow (per user) within a specific period. This type of test can be used for example to find out the average response time for 1000 concurrent users performing each a sequence of specific actions (e.g. login, update profile, search members, making payment, logout) within a period of 10 seconds. The test tools also include a script to generate test databases (Jmeter).

Quick guide for functional testing

Web tests

Web service tests

Quick guide for performance testing

Database generation (Testbed)

The testbed is an utility to generate a database with sample data, specially member profiles, advertisements and payments.

There is a project called "cyclos_testbed" (available for Cyclos3.6.x) that can be used to generate a database with members, advertisements and payments and images. There are various options/settings to create the database.

The testbed project also include a series of scripts that can be used with Jakarta JMeter to perform load tests on cyclos.

Configuration

The configuration is done in 2 distinct files:

Works just like the regular cyclos.properties in the main Cyclos application. What will really be used here is the database connection configuration. Normally, only the hibernate.connection.url property should be changed, to adapt to the MySQL database name. If your database uses a distinct username and password, make sure to adjust the hibernate.connection.username and hibernate.connection.password properties as well.

This file defines what will be generated. The file is self-commented. You can define how many members will be generated, how many advertisements will they have and how many payments they will do to other members. The database will replaced an existing database. However, there's the option to generate more data in an existing database. If you want this, make sure the internal identifiers for the groups in which members will be generated (a comma separated list may be used - when more than one, the specified number of members will be by group) and the internal identifier for the accounts used are all correct. Another thing that can be changed is the images which will be used for members and ads. You can replace the images/*.gif files.

Be aware that databases with many images will increment the database size considerable. Large databases can take some time to create. A database with 100.000 users and additional data (see cyclos-testbed.properties) took 24 hours to create on an average normal PC.

Creating the database (by running the utility)

Run ./testbed (from the cyclos3_testbed_X.X/sh folder on Linux) or testbed.bat (from the cyclos3_testbed_X.X/bat folder on Windows). Java 6+ is required. To check this, you can run java -version

Performance testing

Web and web service load tests

Introduction

Load tests have been executed to measure the Cluster performance under different hardware configurations. To deliver hi load to the servers simulating multiple users and collect the needed statistics we used JMeter. Payment is one of the most important core functionalities of the system and was selected first to be measured in a cluster environment. Basically, we created a test case that consists on a single test step that performs a payment between two members. This test case is run concurrently and the main statistics are gathered to conform a summary report. The following results are real but they are still far from being optimal and not intended to give an estimate of the overall performance of the system.

Hardware configuration

Processor Memory Ethernet Card Task
Intel(R) Core(TM) i3 CPU 550 @ 3.20GHz 4GB Gigabit MySQL Server Distrib 5.1.58
Intel(R) Core(TM) i3 CPU 550 @ 3.20GHz 4GB Gigabit SoapUI client 4.0.1
Intel(R) Pentium(R) Dual CPU E2140 @ 1.60GHz 3GB Gigabit Tomcat 6.0.26 worker 1 and Apache load balancer
Intel(R) Pentium(R) Dual CPU E2140 @ 1.60GHz 3GB Gigabit Tomcat 6.0.26 worker 2
Intel(R) Pentium(R) Dual CPU E2140 @ 1.60GHz 3GB Gigabit Tomcat 6.0.26 worker 3
Intel(R) Pentium(R) Dual CPU E2140 @ 1.60GHz 3GB Gigabit Tomcat 6.0.26 worker 4

Software configuration

The load tests can be run with or without a cluster environment. Follow this link to see how a cluster environment is configured. In the following sections you'll find some tuning steps to gain some performance.

JMeter

JMeter 2.5.1 was used configured with 1.5 GB of maximum available memory, issuing requests to the Apache load balancer node.

JMeter installation

Inside this zip there is a file called JMeterPlugins.jar that must be placed in the /opt/develop/jakarta-jmeter-2.5.1/lib/ext folder.

JMeter test plan configuration
The following fields should be in cero:
"First, wait for", "using ramp-up", "Finally, stop", "threads every".

The following fields should contain equal values: 
"threads every", "Then hold load for".

In the request label you can input the exact name of a request to tell the plugin to also export specific statistics for that particular request. In the output path you must introduce the output path for the folder were all statistic logs will be saved and afterwards used by the report generator.

MySQL

The default my.cnf was used with the following tuning: 

thread_cache_size                    = 23
max_connections                      = 2800
table_cache                          = 1000
innodb_buffer_pool_size              = 2048M
innodb_additional_mem_pool_size      = 32M
innodb_thread_concurrency            = 8
key_buffer            		     = 28M

A good rule to follow to assign mysql maximum number of connections is: Tomcat maxThreads * 2 * Number of tomcat nodes.

Database

Generate a database using the testbed tool, read this section to learn how to do it. Make sure that the database is generated using the following rules. 

Here is an example of cyclos-testbed.properties that can be used to generate the database following the rules.

cyclos.testbed.membersBufferSize = 50
cyclos.testbed.members = 1000
cyclos.testbed.imagesPerMember = 0
cyclos.testbed.adsPerMember = 0
cyclos.testbed.imagesPerAd = 0
cyclos.testbed.transactionsPerMember = 1 / 2
cyclos.testbed.transactionAmount = 10 / 100
cyclos.testbed.creditLimit = 0
cyclos.testbed.initialCredit = 1000000
cyclos.testbed.loginPassword=1234
cyclos.testbed.transactionPassword=ABCD
cyclos.testbed.createDatabase=true
cyclos.testbed.groupIds=5
cyclos.testbed.debitAccountIds=1
cyclos.testbed.memberAccountTypeIds=5
cyclos.testbed.transferTypeId=13
cyclos.testbed.adCategories=

Linux O.S.

Be sure that the maximum number of open files per user is large enough. In our case, the default (1024) value had to be duplicated. To know how this is configured see the problems and solutions section.

Apache Tomcat and Apache load balancer

The load balancer was configured to accept a maximum of 350 simultaneous connections. Keep in mind that the maximum number of connections configured in the load balancer shouldn't be lower than the maximum number of connections that each tomcat accepts. So, each tomcat was configured to accept up to 350 simultaneous connections. Follow this link if you want more information on how to configure tomcat and load balancer connections. 

	<Connector port="8009" protocol="AJP/1.3" maxThreads="450" />

	MaxClients          400

<Membership className="org.apache.catalina.tribes.membership.McastService"
	...
        frequency="1000"
        dropTime="30000"/>

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
	...                        
	selectorTimeout="1000"

<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
        <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" timeout="10000"/>

Cyclos

In cyclos.properties: Assign the maximum number of connections to the database for each cyclos node. A good value would be twice the maximum number of threads configured in tomcat connector. 

hibernate.c3p0.minPoolSize      = 50
hibernate.c3p0.maxPoolSize      = 700
hibernate.c3p0.acquireIncrement = 20

Once the database is generated (see #Database section):

* Turning off all logs in Cyclos (to get the best performance). 
  Log in as an admin and go to Settings > Log Settings and select Off in all logs. Logs can lead to poor performance.

* Adding a new channel:
	Menu Settings > Channels
	  New channel:
		Introduce the following information:
		* Display name: Web service channel
		* Internal name: Web service channel
		* User identification: Login name
		* Default user identification: Login name
		* Credentials: Login password
		* Supports payment request: unchecked
		* Submit the form and write down the channel id by looking at the current url, we'll suppose: channelId=7
		
* Adding the channel to the transfer type:
	Menu Accounts > Manage Accounts
	  Modify Member account
		  Modify External trade transfer
			 * Check the Web service channel option in Channels
			 * Leave the Max amount per day empty
			 * Write down the transfer type id by looking at the URL in your browser. It should say something like this: transferTypeId=30.
			 * Submit the form.

* Adding an unrestricted web service client:
	Menu Settings > Web service clients
	  New web service client:
		Introduce the following information:
		* Name: Unrestricted web service client
		* Internet address: 192.168.0.1 (jmeter's IP)
		* Channel: Web service channel
		* Credentials required: checked
		* Permissions
			# Perform payments: 
				Check External trade transfer option
		* Submit the form
		
* Adding the channel to the accessible channels of the member's group:
	Menu Users & Groups > Permission Groups
	  Modify Full members
		* Click on change button
		* In Access settings section, in Accessible channels select Web service channel option.
		* Submit the form
		
* Enabling Web service channel to all members:
	In some steps above you wrote down the channel id, if it differs from 7 change the id on the following statement and execute it using MySQL command line or your favorite tool.
	
	insert into members_channels(member_id, channel_id) select id, 7 from members where subclass = 'M';

* Change Member1 notification preferences (this configuration will avoid bottlenecks in load tests were everybody pays to him, N -> 1):
	Logged in as Admin, go to Home and search for member1. Go to it's profile and under Preferences:
	* Click on Submit button for Manage Notifications.
	* Click on Change button.
	* Uncheck Payment events
	* Uncheck Payments made via external channels.
	* Submit the form.

Results

Running the tests

Generating a report

There is a tool that gathers all the exported data (by the exporter plugin) and creates a report with some basic statistics. After running the tests follow the next instructions to get the generated report. 

 e.g: 
 sh loadTestReport ws_report.properties

If no properties file path is passed to the script, it assumes the properties file is called report.properties and is located in the cyclos3_testbed_X.X directory.

Web services results

Follow this link to download a pdf with the web service results.

Some important information about results:

When tests were carried to the maximum load

Web results

Follow this link to download a pdf with the web results.

Problems and solutions

To increase the maximum number of open files follow this article

Previous Work

Configuring and running the JMeter project

Inside JMeter, open the file cyclos3_testbed_X.X/jmeter/Cyclos3.jmx. It contains a script with the following steps:

Each of those steps involves one or more requests, but are organized in groups, so they can be easily disabled if needed. The basic instructions to make it work is:

Fine tuning Cyclos

The performance obtained heavily depends on several configuration variables. First of all, it's recommended to run the JMeter tests from a different machine than the Cyclos server itself, as JMeter is resource heavy and it can slow down Cyclos. Then, here are some aspects which can be tuned (please, note that when the cyclos.properties file is mentioned, it's the server one which is in WEB-INF/classes directory, not the one included in testbed):

hibernate.cache.use_second_level_cache are set to true. This will reduce the database roundtrips.

Load test results

It is hard to give any numbers because the load depends very much on how Cyclos is accessed. For example, the Web access channel demands much more resource than SMS operations.

Also the hardware is very important. The main factors are

Computers used in testing

Setup

1 computer running JMeter (http requests)
3 computers running Tomcat / Cyclos
1 computer running MySQL

Results

See picture at the right:

Results

The tests where done with 500 to 3000 members loging in performing a payment and login out in 10 seconds (per user). Using one tomcat server the response time where good up to 750 users. With three Tomcats 2500 users still gave reasonable results (3 seconds). Be ware that relatively old computers where used. Using a single computer/server with a more modern chips will perform considerably better.

There are a collection of functional tests that are run to ensure a minimal behavior of the application. This section describes how to install the required environment and the steps needed to run and extend this tests. There are two approaches, the Selenium environment is used to test core functionality surfing Cyclos web front end from a browser; the second approach (using SoapUI) is to test core functionality that is exposed through web services.

Functional testing

Selenium Environment

Installation

Links

The basics

Selenium IDE

Open the Selenium IDE: In Firefox, Tools > Selenium IDE.

By default, Selenium starts with a Test Suite containing a unique Test Case named 'Untitled'. You can create new test cases through the top menu or the context menu of the left panel.

At the right panel Selenium displays the content of the selected test case. To select a test case you must double click the test case, otherwise the test case will be highlighted and the content of the test case won't appear at the right panel. The right panel contains two tabs, Table and Source. The Table tab (when it is enabled) displays the list of commands to be executed by the browser. The Source tab displays the list of commands in the selected format. By default, the selected format is HTML. You can switch between different formats through the Options menu. Note that the Table tab will be enabled only when the HTML format is selected.

At the top of the IDE, there is a text box which you need to fill with a base URL (e.g: http://localhost:8080/cyclos/do/). Certain commands need an URL as an argument. The URL will be relative to the base URL. e.g: 

    Command      Target    
    open         login        (login is the relative path, the absolute URL would be: http://localhost:8080/cyclos/do/login).

The base URL is specific to each test case and it's not global to all the test cases. Modifying the base URL located at the top will modify only the selected test case's base URL.

To start recording click on the recording button (red bullet) and start browsing. After a few actions with the browser click again on the recording button and look at the commands that have been detected.

That's it!. To replay your actions click on the green play button (there are two, one to replay the entire test suite and other to replay the selected test case).

Once the test case is finished it can be saved to a file through the menu or the context menu over the test case. A test suite only contains references to test case files. Saving a test suite prompts the user to save every unsaved test case, and after all, prompts to save the test suite file.

Not everything is so easy

Sometimes Selenium IDE doesn't catch all the actions done in the browser and some commands have to be added manually in order to create a complete test case. To do this, open the Table tab and add them in the places you need, helped by the context menu.

Selenium can't handle alerts that rise in the onLoad event of a page. If this happens, the test gets stucked waiting for something to "consume" the alert. In Cyclos this alerts are everywhere, the workaround is to change the javascript function showMessage (head.jsp) so that a div is displayed instead of an alert. Yes, your Cyclos must be patched. 

   function showMessage() {
       ...
       if(booleanValue(readCookie("showMessage"))) {
           alert('${message}');        -----> this line has to be changed by: showMessageDiv('${message}');
           messageDiv = null;          -----> this line should be added.
       }
       deleteCookie("showMessage");
       ...
   }

Locators

Many commands need to refere to an element in the page. The reference is done giving a locator. For example: the click command needs to be done on some element. To write the locator it is very useful to inspect the element you want to locate. To inspect the element, right click the element in the browser and choose inspect element (firebug must be installed first). Locators are ways of locating an element in the page. For example you could locate an element giving an XPATH: (//input[@id='username']). Click on the left panel examples to learn basic XPATH constructions in this tutorial: http://zvon.org/xxl/XPathTutorial/General/examples.html There are serveral other ways to locate an element, someones much more easier. They are all explained here: http://seleniumhq.org/docs/02_selenium_ide.html#locating-elements. Reeding this reference is a must!.

Tips & tricks

    Command      Target       Value
    select       age          label=30

    Command      Target       
    clickAndWait //img[@class='remove' and @id=${adId}]

    Command      Target
    runScript    FCKeditorAPI.GetInstance( 'descriptionText' ).SetHTML( 'The new ad description' )

    The solution is to create a variable with the long representation of the actual date. This will be unique each time you execute it.
    
    Command      Target                  Value
    storeEval    new Date().getTime()    varUsername
    type         username                member${varUsername}     this will produce user names like: member126436363543

Custom Commands

This commands doesn't come as built-in commands, they are part of our own command set and are implemented in the user-extensions.js file. 

    Command                   Target                Value
    storeURLParameterValue    parameterName         variableName
    

    Command                   Target      
    storeActualDate           variableName
    

    Command                   Target      
    storeTestCaseFolderPath   variableName

    Command      Target                  Value
    login        admin
    login        member1
    login        operator1               member1

Note: If you want to export test cases that have custom commands to a certain language like java, the appropriate implementation for this language must be added, but for now this explanation is beyond the scope.

Test standards

e.g: 

   Command                   Target
   assertElementPresent      css=table.defaultTable > tbody > tr:nth(10) > td:nth(1) > span.customFieldContainer > input[name=member(customValues).value]

   Inspecting the element you'll find that the locator can be much more simple, it only takes a few seconds more and you'll make your tests more maintainable. 
   The above example could be easily changed by the following command:
   
   Command                   Target
   assertElementPresent      //input[@fieldname='city']

The login example

   command         target                value
   open            /cyclos/do/login
   type            cyclosUsername        admin
   type            cyclosPassword        1234
   clickAndWait    css=input.button

Sometimes the autocomplete of the browser interferes with the recording and the type cyclosPassword command could have been skipped. Manually insert the command right clicking in the commands window and choosing the appropriate option.

Created tests

There are many tests that have already been created, they can give you an idea of how to do some other things not covered in this introduction. There is a tool (jar) that can collect all the test cases from a folder and it's subfolders and make a single test suite to run them all. This jar is called testSuiteGenerator.jar and is located under the tests folder. It is also frequently used to change the URL of the test case scripts. Execute: java -jar testSuiteGenerator.jar -help from a command line in the cyclos3_testbed_X.X/selenium/tests folder to see all the generator options. The test cases are proven to be successful with the database located in the db folder.

User extensions

To add new commands append the implementations in the user-extensions.js file. Some considerations: 

   Selenium.prototype.do<NameOfTheCommand> = function([firstArgument[, secondArgument]]){
       //implementation goes here
   };


SoapUI Environment

Installation

SoapUI is an open source functional testing tool. To download and learn SoapUI follow this link: http://www.soapui.org/

Introduction

SoapUI by itself doesn't provide the easiest way for us to create web service tests, but offers all the necessary tools for us to do it. So part of this document describes how to write web service tests in a simple way just using an OpenOffice Calc spreadsheet and generating a file that consists on the input that SoapUI takes to generate and run the tests. When designing this tool we focus directly on the target of what we wanted, and that was to specify tests that:

In essence, we need to fill input values of a request, execute the request, make some assertions on the response and probably get some data from the response to be used in other requests.

Specifying tests

As we mentioned in the introduction a spreadsheet is used to write tests. In this section you'll learn how to specify a web service test in the spreadsheet and generate the properties file that SoapUI needs.

generator.properties file

Global configuration is established in the generator.properties file: 

 e.g: 
 ts.execution=PaymentsTestSuite, POSTestSuite, MemberTestSuite
 ts.cyclosURL=http://localhost:8080/cyclos/
 ts.numberOfLastCasesToGenerate=2

In the above example, there should exist three spreadsheets with some tests in each. Only the last two test cases will be generated for each spreadsheet.

The spreadsheet

The spreadsheet contains a macro, so a dialog message appears when the spreadsheet is opened asking if you want to enable macros. Please select “yes”. If no message appears probably the macro will be disabled and you won't be able to generate the properties file. To change this settings go to Tools > Options... > OpenOffice.org | Security | Macro Security button and select the Medium security level. Then try opening the spreadsheet again.

Two main sheets are in the spreadsheet: Input and Configurations. Configuration contains general properties for all the tests we will be specifying in the Input sheet.

The Configuration sheet

Here you can specify general configurations for all the test cases.

Figure 1: Configuration sheet
Comments

Comments can be added by adding a “#” character behind the name of the property in the appropriate row of the Property column. In the above example only three interface URLs are defined, the first group of URLs will be not taken into account since the “#” prefix precedes the property names.

Interfaces or WSDL URL properties

In the figure 1 we declare some WSDL URL properties. This properties follow a specific syntax. The WSDL URL property name consists on three sections separated by a dot “.”: “conf” meaning its a configuration property “url” meaning its an URL property wsdl definition name

The wsdl definition name can be taken from the wsdl definition.

Figure 2: WSDL definition accessed with a browser

The value of the property consists on the relative URL of the WSDL.

Number of decimal places property

This property declares the decimal places when formatting a number. See Format References for more on formatting numbers.

The Input sheet
Generating the property file

To generate the properties file (the final input file for SoapUI) click on the button Generate Properties in the toolbars section. The number of test cases that will be generated is established in the generator.properties file described before, if this property is omitted all test cases form the sheet will be generated.

Figure 3: Generate properties toolbar
General terms

We will go through an example to understand all the concepts. But first we'll introduce some terms.

An interface or WSDL may contain several operations. For example, the payments interface has the doPayment and the reverse operations.

Figure 4: An interface with its operations in SoapUI

An operation can have multiple input parameters. For example, the doPayment operation has amount currency description etc... So whenever an operation is invoked (a request is done), all the input parameters of the request must have the appropriate values.

This kind of parameters are declared in the Input sheet using an “in” prefix as we'll see further on.

The result of a request operation is a response. The response comes with certain data on certain fields. The values of this fields are what we want to hold in output parameters. For example, the field status on the response of a doPayment tell us if the operation has succeeded or not. If we would like to store this value we need to declare an out parameter in the Input sheet.

This kind of parameters are declared in the Input sheet using an “out” prefix as we'll see further on.

Operations needs to carry authentication information on the request headers. This kind of parameters are declared in the Input sheet using an “auth” prefix.

When the response is received, validation stuff can be done. There are different kinds of validations: Validate there is a response from the server. Validate that the response is in the format (xml) that we expect. Validate that in the response doesn't come a soap fault message. Validate that the response contains certain data (e.g: a SUCCESS text in the status field).

The first two validations are done without having to explicitly declare them in the Input sheet but the last ones needs to be specified because it depends on the test context.

This kind of parameters are declared in the Input sheet using the rude “ass” prefix as we'll see further on.

Parameter conventions

Lets look at an example to understand how to specify the above parameters in the Input sheet. Suppose our first test case consists on making a payment. An user member2 wants to pay to member1 1 dollar.

We will specify some parameters for the doPayment operation in the Input sheet.

Figure 5: Do payment specification

Some parameter values will be explained in the next sections, don't worry if you don't understand everything by now.

Each operation must have a unique name that identifies the operation. In the above case, dp1 is the operation identifier.

When a Cyclos web service client is configure to use http credentials, the operation request must carry this credentials. To accomplish this, authentication parameters must be specified for the request. In this case, the web service client's user and password are 'member1'.

The parameter type is “auth” because it's an authentication parameter. 

   The syntax for auth parameters is:
       auth.httpUser
       auth.httpPassword

The values are taken from the sheet and placed encrypted in the header of the operation request.

The values of the input parameters are placed in the operation request.

Figure 6: Fragment of request with some parameters
   To specify input parameters the syntax is:
       in.<Name of the parameter>
   where <Name of the parameter> is the exact name of the request parameter. If an invalid parameter name is specified the SoapUI will detect the problem.

Sometimes the requests have multiple section parameters as in doBulkPayment. In this operation you can specify more than one section of params. So we could have something like this: 

       <params>
               <amount></amount>
               ...
       </params>
       <params>
               <amount></amount>
               ...
       </params>

To refere to the second field amount you need to be more specific in the naming of the parameters. 

   The general syntax for in parameters is:
       in.<tag1[X1]>.<>...<>.<tagN[Xn]>.<Name of the parameter>
   e.g:
       in.params[2].amount to refere to the second amount parameter.
   The parser looks for the following xpath in the request: //params[2]/amount.
   This aplies to all in parameters.

The out parameters are used to hold values from responses and that can potentially be used for filling input parameters or storing important information to be processed later.

Unlike the in parameters, out parameters have an arbitrary name. 

   The syntax for out parameters is:
       out.<Name of the parameter>
   A name example:
       out.myOutParameter
   The value of the property is the xpath of the tag from where to take the value to fill the property. The xpath is specified between “#{xpath: ” and “}” like in the following example:
   A value example:
       #{xpath: //transfer/id}

The ass parameters are used to make validations over an operation response. Following the example described above, we would like to halt the test if we find that the response is a soap fault or if the status code isn't “PROCESSED”. 

   The syntaxes for ass parameters are:
       ass.notSoapFaultAssertion If the value is true, then a response will be valid if the response isn't a soap fault.
       ass.xpath
       ass.xpath.expectedResult

This two properties are combined and the above one doesn't have to be necessarily an xpath expression but can be a whole complex expression. Let's describe some examples:

16.png

The last row of the above examples shows how we can make references to other properties. For more information on xpath assertions follow this link: http://www.soapui.org/Functional-Testing/xpath-and-xquery-assertions.html

General notes on parameters

If we specify the NULL value for a parameter, then: if it is an in parameter, the tag in the request will be removed. If it is an auth parameter, the Authentication header will be removed. If it is an out parameter, then the parameter will not be created for that case.

In all property values we can make references to other property values. For example, we could reference the trace number property of the doPayment operation in the reverse operation. The r.in.traceNumber property should contain the value: #{p: dp.in.traceNumber} #{p: <content>} indicates that the content should be a property name and it's replaced by the value of the specified property.

After evaluating in references, script references are evaluated. Script references are available to compute math expressions or whatever groovy script you may want. Example: To test the credit limit, we could ask for the available credit and make a payment of that amount plus one cent. The system should return a credit exceeded message. So dp.in.amount property should contain the value: #{s: 0.01 + #{p: sah.out.availableBalance}} #{s: <content>} indicates that the content should be evaluated as a groovy script.

sah.out.availableBalance has the available balance amount returned by the search account history operation invoked before.

Sometimes we need to format a number with a certain amount of decimal places. For this purpose we can use the format reference like in the following example: #{f: #{p: sah.out.availableBalance}} if sah.out.availableBalance contain 0.0466 and the declared decimal places is 2, then the result will be 0.05. The decimal places for the format reference needs to be declared in the configuration sheet in the conf.numberOfDecimalPlaces property. #{f: <content>} indicates that the content is numeric and should be formatted following the conf.numberOfDecimalPlaces property.

Sometimes we need a generated unique number. For example, if a test case registers Member 1 with username member1, the first time the test will succeed, but in the followings the member will be already registered and the test case will fail if the test case asserts always the status "Member registered". The test case must be smarter and register a new member each time it runs. To accomplish this, the username could be an incremental number computed each time. This incremental number can be obtained with a Date reference. There are many created tests that use this approach. Check them to see how this is easily done.

Running tests

Figure 9: Generator script test step

To run the tests, open the Generator step and click on the green arrow button at the upper left corner of the window.

How generation works

There is a test suite generator called 'TS - Generator Test Suite'. This is a special test suite that generates and executes the rest of the test suites. To run this test suite (and generate and run the rest of the test suites) double click the test suite and click on the green arrow button at the upper left corner. For each test suite name declared in the generator.properties file, a test suite is generated.

Figure 10: Generator scripts in each generated test suite

If you want to generate and run a particular test suite, just run the test suite. If you want to run a particular test case, just run the test case.

Test steps are run in descending order, so in the above example the first step to execute is the Generator step and then the first SOAP step below the properties step (properties steps cannot be executed).

Figure 11: Run of generated test cases and test steps

After running the Generator test step, a test case is created for each column specified in the Input sheet. There is also a test step request for each specified operation and a properties step with all the properties for that particular test case. The generator test step window has an output log where we can read the messages of the overall run. The test steps start in white colour indicating that the step didn't run yet. When the step is run it turns to green if there where no errors or red otherwise. To see the request and the response of a step just double click on the step and a window (like the above in the image) will show up. You can double click on the assertions to know it's content.