Developer Guide

This document addresses both test and infrastructure (SystemObject) developers. It begins with issues that are more relevant to test developers and then deals with issues that are relevant to SystemObject developers. However, it is strongly recommended that each developer will read both sections to understand better the relationship between infrastructure and test development.

Background readings:

Writing Tests

General

JSystem is based on JUnit and extends it. Each JSystem test is actually a JUnit test with additional services like the following:

Reporting - hierarchical log/report with HTML, XML or database outputs.
SUT (System Under Test) Independence - the ability to run the same test on different setups without changing the test.
Analyzers - the ability to analyze different types of input data, like string, XML files, ResultSet and more.
Shared Fixtures - share fixture (i.e. setup configuration) between tests.
Parametrization - define parameters for tests that can be later changed by the test executor.
Monitors - processes that will run during the test execution and will monitor different aspect of the SUT.
Graphs - collect data during the test execution and present it in a formulated graph.
Scenario - suite a group of tests into a scenario for management and execution purposes.

Since JSystem extends JUnit, you should first learn about JUnit. The following is defined under JUnit:

Every test class should extend the TestCase class.
In order to define a method in this class as a test, it must start with the string 'test'.
In addition, the test method should be public, return void and get no arguments.
Each test class can define setUp() and tearDown() methods.
The setUp method (if defined) is executed automatically before every test execution.
The tearDown method (if defined) is executed after every test execution.
Test is declared as 'pass' if it doesn't throw any Exception.
A set of assertions (check points) methods were defined and can be used during the test.
A code mechanism is defined to group tests (Suite).

JSystem Vs JUnit (highlights):

When using JSystem tests class should extend SystemTestCase.
The tests/method convention is not changed.
setUp() and tearDown() convention are not changed.
You can use fixture to share configurations between tests.
Test fails if an exception is thrown, or if a fail report is submitted (or both).
An extensible set of analyzers can be used to analyze test execution.
The JSystem runner uses XML to manage tests execution list (scenario) and their parameters.

Example 1: basic test



01 package com.aqua.examples;

02 

03 import jsystem.extensions.analyzers.text.FindText;

04 import jsystem.framework.fixture.RootFixture;

05 import junit.framework.SystemTestCase;


06 

07 /**

08  * Example 1: basic test

09  */

10 public class MyTest extends SystemTestCase {


11   

12   public MyTest(){

13     setFixture(BasicFixture.class);

14     setTearDownFixture(RootFixture.class);


15   }

16   /**

17    * The SystemObject in this case is a linux 


18    * machine that is managed using cli/telnet.

19    */

20   Linux linux1 = null;


21 

22   public void setUp() throws Exception {

23     linux1 = (Linux) system.getSystemObject("linux");


24   }

25 

26   /**

27    * Execute ls command and analyze that the 


28    * string 'myFile' is shown in the results

29    * @throws Exception 

30    * 


31    */

32   public void testLs() throws Exception {

33     report.step("Execute the ls command");


34     linux1.command("ls");

35 

36     report.step("Analyze the command output");


37     linux1.analyze(new FindText("myFile"));

38   }

39 }

What can we learn from the example?

Every JSystem test should extend SystemTestCase (line 10).
The system object the is managed in the example is a linux machine. the setUp method is used to init the object (line 22).
The test is a method that start with the string test in our case 'testLs' (line 32).
We uses the report object to add reports to the tests (lines 33,36). Those reports will be seen in the used reporters. See the reporter chapter for more information.
The test include a SystemObject operation 'linux.command("ls")' (line 34). It will execute the ls command on the linux machine. The string result will be save for analyzing.
The result can be analyze using variety of build-in analyzer (you can also write your own analyzers) (line 37). In this case the analyzer analyzes String input but it can be done with any object type. See the analyzer chapter for more information.

There are 3 types of steps in a test:

reports - test step can generate reports. It's much recommended to use this service to document the progress of the test. It will then help in the analysis of the test execution.
SystemObject operation - test step that manage the SUT (System Under Test). It can change it configuration or get information from it. Every SystemObject operation can build a result object for further analyzing. In Example 1, the result object is the cli return string.
Analyzing - points in the test were the execution result being analyzed and compared with expected results. The analyzing is part of the SystemObject.

General guidelines

The test-plan - the test-plan is mandatory for good automatic tests. In manual tests experience engineer can cover for lack of good test-plan, in automatic tests it will not do. The test-plan is a contract that defines the coverage of the automatic tests.
Manual execution - before starting to write tests, take the time to execute the tests manually. Testing the correlation between your test-plan and the reality can save you a lot of time in the end.
Test environment - automatic tests are not execute in a vacuum. The preset of the SUT (System Under Test) should be well define. What is the root configuration of the setup? What type of cleaning every test should perform? What are the things the test can change in the setup and what are the thing it shouldn't? All those questions should be answered before you start writing your first line of code.
Big amount of small tests - design your tests to be as small as possible. Each test should cover the minimal tested functionality that possible. It as a huge affect on the test debug, analysis and reconstruction. You will end up with a lot of tests and it's good!
Documentation - the ability to read and understand your tests will affect the scalability of the project. Your tests should be well documented (regardless of the excellent test-plan).
Don't be too smart - don't over design. You are writing tests and not planning satellites. For your tests to be readable they should be straight-forward (even in the price of not writing the best object oriented code you can).

System and Sut objects

Most of the services given to a tests can be access using public members in the SystemTestCase object. One of those members/services is the SystemManager. The member name is 'system'. You can see a usage of the system service in example 1 line 23.

linux1 = (Linux) system.getSystemObject("linux");

The system service is responsible for few tasks:

Construct and initialize the SystemObjects

Every SUT can be composed from set of SystemObjects. One of the framework requirements is that test that was written for one SUT will be executed on different SUT without changing the test code. All the setup specific information should be placed in the SUT files. The SUT files are XML files that describe the SUT. Every SUT should have its SUT file.

In our case the SUT file should looks something like this:

<sut>
    <linux>
        <class>com.aqua.examples.Linux</class>
        <host>127.0.0.1</host>

        <username>root</username>
        <password>root</password>
    </linux>
</sut>

The SystemManagerImpl uses java reflection to create the linux object. If the Linux object defined setter method (a method that start with the set string) like setHost, it will be called with the XML tag value (in this case '127.0.0.1').

Object repository

The system service is a repository for SystemObjects. By default the SystemObject will be init only once by the first test that require the object. The second test will get it from the object repository. The SystemObject can be configured to TEST_LIFETIME and then it will be construct and init every test.

Object close

The system service responsible of closing the SystemObject. It register a shutdown hoke that will be executed on system exit.

SUT service

The 'sut' service is used to directly access the SUT file. The SUT file is an XML file. To access it we use the XPath techenology.

Following is an example for using the the 'sut' service:

     String password = sut.getValue("/sut/linux/password/text()")

If run on the above XML the password will be 'root'.

The SUT files should be located in the sut directory under the tests root directory. They should be managed as part of the tests code and be copied to the classes/sut directory as part of the tests build process (Eclipse do it automatically). The SUT file name to be used is saved in the jsystem.properties file (sutFile=xml4Test.xml). This file holds all the JSystem execution information. For more details about the jsystem.properties.

Reporting

Reporting is importuned for the scalability of the automation project. The reports should enable to analyze quickly the cause for test execution failure. It's recommended that before you start your automation project to have reporting convention that will be used in all your project tests.

Example 1 show the usage of the report service in line 33.

report.step("Execute the ls command");

report in general have few fields:

title - the title of the report.
message - the message of the report, will be seen in the html reporter as link from the title.
status - the status of the report. The status have 3 states: PASS, FAIL and WARNING. In some of the report method the status is a boolean and can get true for PASS and false for FAIL.
bold - if true the report will be marked as bold.

If report with fail status is submitted, the test will fail (even if no exception will be thrown).

Report step will be seen as bold report in the html reporter. In addition all the steps are collected and are part of SystemTestCase property, it can be seen in general reports and statistics.

Documentation

JSystem uses your test documentation in it's reports. Both the class documentation (lines 07-09) and the test documentation (lines 26-31) are taken and become part of the SystemTestCase properties and will be shown in the JSystem runner the html or other reporters. The documentation should be in Javadoc format.

Analyzers

Analyzers gives easy way to verify SystemObjects operations. The SystemObject operations are used to manage/get information from the SUT. Every such operation can take the result object (in the telnet example it will be a String object) and defined it as the object to be analyzed (using the setTestAgainstObject method). By doing so the analyzers can be used to analyze the object and compare it with some expected result. In example 1 lines 34, 37:

linux.command("ls");
linux.analyze(new FindText("myFile"));

we first execute telnet command (ls) that the output is taken and set as the test against object (it is done internally by the linux SystemObject), then we uses FindText to verify that the string 'myFile' is found in the result.

The analyzing process is responsable for generating success report if the analyze success or fail report if the analyzer fail. In case of a failure an exception will be thrown.

The analyzer interface define additional method that can be used:

analyze(AnalyzerParameter parameter, boolean silent, boolean throwException)

When silent is true the report will be seen only in case of a failure. When throwException is false the analyzing process will not throw exception in case of analyzer failer. isAnalyzeSuccess(AnalyzerParameter parameter) can be used to check if the analyze success. No report will be post and no exception will be thrown in case of failure.

JSystem comes with a variety of analyzers there are analyzers to analyze text, XML document and more. Analyzers are extendable and you can write your own analyzers.

Example 2: FindText analyzer


01 package jsystem.extensions.analyzers.text;


02 

03 import jsystem.framework.analyzer.AnalyzerParameterImpl;

04 

05 /**

06  * Find a text in a string. Support regular expressions.


07  * 

08  * @author Guy Arieli

09  */

10 public class FindText extends AnalyzerParameterImpl {


11   String toFind;

12     /**

13      * 

14      * @param toFind The string to find.


15      */

16     public FindText(String toFind){

17       this.toFind = toFind;


18     }

19 

20     public void analyze() {

21       String testText = (String)testAgainst;


22         String found = toFind;

23         status = (testText.indexOf(toFind) >= 0);


24         if (status){

25             title = "The text:<" + toFind + "> was found";


26         } else {

27             title = "The text: <" + toFind + "> wasn't found";


28         }

29         message =testText.replaceAll(toFind,"<b>" + found + "</b>");

30     }


31     

32     public Class getTestAgainstType(){

33       return String.class;

34     }


35 }

The FindText analyzer example show you how an analyzer can be implemented.

What can you learn from the example?

Analyzers should extends AnalyzerParameterImpl (line 10).
Analyzers should implement analyze method (line 20).
The analyzer will work on the testAgains object. This object is set to the analyzer by the framework (line 21).
The analyze method should set the value of the title (line 25). It will be used in the generate report.
The analyze method should set the value of the message (line 29).
And the status (line 23). The status will indicate the analyze result (true for success, false for fail).
analyzer can implement getTestAgainstType method that return the type of object that is expected by the analyzer. In our case String (line 32).

Fixtures

Fixture represents a state of the SUT (System Under Test). Fixtures dill with the question, what can test assumes on the environment (SUT) it's executed on. Tests can be very easily executed independently. But when you start to group tests, it become complex. Test starts to fail because other tests change the SUT. You can find 2 traditional approaches:

A test can't assume anything about the environment it's executed on. The test then first has to clean the SUT then create the required configurations and then execute the test.
A test can assume that the previous test left the SUT in a known position.

In general the traditional approaches fails in few issues:

You get longer tests execution time.
It creates restrictions for tests execution order.
Tests can’t be executed independently and need other test to be executed before they do.
When one test fails it can affect the other tests. It doesn’t enable a reasonable recovery mechanism. So usually when one of the tests in the tests chain fails other tests will follow.
You have more code and more code repetitions.

In the fixture approach the configuration (or setupping) of a test (or group of tests) is separated from the test itself. The framework is aware of the current configuration status of the SUT and will execute fixture change only if the next test to be executed requires deferent configuration then the current.

JSystem fixture module enables you to create a hierarchy (a tree) of fixtures.

Example 3: BasicFixture


01 package com.aqua.examples;

02 


03 import jsystem.framework.fixture.Fixture;

04 

05 public class BasicFixture extends Fixture {

06   Linux linux1 = null;


07 

08   public void setUp() {

09     linux1 = (Linux) system.getSystemObject("linux");


10     

11     linux1.command("cd /home/mydir");

12   }

13   

14   public void tearDown(){


15     linux1.command("cd /");

16   }

17   

18   public void failTearDown(){


19     linux1.reboot();

20     tearDown();

21   }

22 

23 }

To assign a fixture to test you should you should use setFixture and setTearDownFixture in the test constractor.

public MyTest(){
   setFixture(BasicFixture.class);
   setTearDownFixture(RootFixture.class);
}

What can you learn from the Fixture example?

Fixture is a class that extend the Fixture class (line 05).
In the fixture you can implement 3 method setUp, tearDown and failTearDown.
The fixture can use all the services that available for SystemTestCase, like 'system' (line 09).
To assign a fixture to a test you should use the setFixture in the test constractor.
You can also use the setTearDownFixture. When using this option, in case of test execution fail the framework will navigate to the tear down fixture befor executing the next test. The tear-down will be done in the failTearDown pass.
In the failTearDown you should use more agrisive ways to insure that the next test will be executed with no leftovers (lines 19-20).

Parameterization

JSystem support parameterization for tests. The test writer can define a set of parameters that can be changed by the test executor using the JSystem runner. It will enable the test executor to build a scenario were the same test run several times, each time with deferent parameters.

Example 4: parameters


01 package com.aqua.examples;


02 

03 import junit.framework.SystemTestCase;

04 

05 public class ParametersExampleTest extends SystemTestCase {

06     int packetSize = 64;


07     

08     // this is the actual test

09     public void testTrafficWithPacketSize (){

10         // the test code using the packetSize parameter


11     }

12     /**

13      * Set the test packet size

14      */


15     public void setPacketSize(int  size){

16         packetSize = size;

17      }


18 

19     // this is optional, if set will show the user 64 

20     // as the default packet size


21     

22      public int getPacketSize(){

23         return  packetSize;

24     }

25     // this is optional, if set will let the user to select 


26     // only from the list given.

27      

28     public int[] getPacketSizeOptions (){


29         return new int[] { 64 , 256, 512, 1024,  1518};


30     }

31 }

What can you learn from the Parameters example?

Parameters can be added to any JSystem (SystemTestCase) or JUnit (TestCase) test case (line 05).
Usually you will add a member to the test and create a setter and a getter methods (member line 06, setter line 15 and getter line 22).
The parameterization feature support vereuse types of objects. In the example we can see a usge of int, you can use String, long, float, double and boolean as well.
You can force the test executor to select from a preset of parameters by implementing getParamnameOptions. This method should return an array of optional values (line 28).
The documentation that is written to the set method will be seen at the runner (lines 12-14).

When using the JSystem runner the test parameters are saved as part of the scenario.

Monitors

Monitors are processes (threads) that can be defined per test. The monitor will run during the test execution in deferent thread. The monitor can fail the test if it find a problem with the monitored object.

Example 5: monitor


01 package com.aqua.examples;

02 


03 import jsystem.framework.monitor.Monitor;

04 import jsystem.utils.MiscUtils;

05 

06 public class PingMonitor extends Monitor {


07   String host;

08   public PingMonitor(String host) {

09     super("PingMonitor_" + host);


10     this.host = host;

11   }

12 

13   public void run() {


14     while(true){

15       if(!MiscUtils.isPing(host)){

16         report.report("Fail to ping to host: " + host, false);


17         setFail(true);

18       }

19       try {

20         Thread.sleep(5000);


21       } catch (InterruptedException ex){

22         return;

23       }

24     }


25   }

26 }

The monitor is activating during the test. To activate a monitor you should use the 'monitors' service.

   monitors.startMonitor(new PingMonitor(linux.getHost()));

What can you learn from the Monitor example?

Monitor should extends the Monitor class (line 06).
As it implement the Java Runnable the Monitor should implement a run method (line 13).
Monitor can use JSystem services like 'report' and 'system' (line 16).
Monitor can fail the test by submitting a fail report (line 16).
On interrupt the monitor should exit (line 22).

SystemObject development

SystemObjects are the objects that should be managed as part of the SUT (SystemUnderTest). It can be a Linux server that is managed using Telnet, it can be network equipment that managed using SNMP or SSH or it can be hardware testing device that is manged using GPIB or TCL. The number of object to manage and there complexity is defined by your testing needs. The way to connect to the required objects is changing from object to object:

CLI/Telnet, CLI/SSH, CLI/RS232, GPIB.
SNMP v1, v2 and v3.
Web.
XML-RPC, SOAP, Corba or any type of OSS your solution supports.
API - Tcl, Java, C or any other programmatic API.
And more ...

The SystemObject you are developing should hide the connectivity issues and expose the business logic or the functionality of the object. The SystemObject is a midel-layer that should do the following:

Hide the complexity of the object connectivity.
Expose an API that 'speak' in the languge of the QA engineer.
Take care of reporting.
And error handeling (throw exception in case of faluir).
Support result analysis.

When you designing SystemObject I found it very usefull to start from writing a test example that will use the SystemObject. The big question should be how should the tests looks like.

Example 6: SystemObject


01 package com.aqua.examples;


02 

03 import jsystem.framework.system.SystemObjectImpl;

04 

05 public class Linux extends SystemObjectImpl {

06   Cli cli;


07   String host;

08   String username;

09   String password;

10   

11   public void init() throws Exception{


12     super.init();

13     cli = new Cli(host, username, password);


14   }

15   public void command(String command) throws Exception{

16     String result = null;


17     try {

18       result = cli.command(command);

19       report.report(getName() + " command: " + command,


20           result,

21           true);

22       setTestAgainsObject(result);

23     } catch (Exception e){


24       report.report(getName() + " command fail", e);

25       throw e;


26     }

27   }

28   public void close(){

29     cli.close();


30   }

31   public String getHost() {

32     return host;

33   }


34   public void setHost(String host) {

35     this.host = host;

36   }


37   public String getPassword() {

38     return password;

39   }

40   public void setPassword(String password) {


41     this.password = password;

42   }

43   public String getUsername() {

44     return username;


45   }

46   public void setUsername(String username) {

47     this.username = username;


48   }

49 }

What can we learn from the SystemObject example

Your SystemObject should extends SystemObjectImpl (line 05).
A SystemObject should have a default constractor with no argument (as the framework constract it using java reflection).
SystemObject can implement init and close methods, init will be called after the object constraction, and close on object close (lines 11, 28).
If a set method is defined in the object and a tag is found in the SUT XML, the set method will be called with the tag value. For example you can find the setHost (line 34) method and in the XML you can find a host tag. The value of the host tag (in this case '127.0.0.1') will be passed to the setHost method. It is all done as part of the super.init() (line 12).
The command method is the SystemObject operation. You can find the execution of the command (line 18). The report in case of success (line 19). The setTestAgainstObject is taking the command result and set it as the object to analyze (line 22). In case of execution failer there is a fail report (line 24) and exception throw (line 25).

Lifetime

One of the SystemObject attribures is its lifetime. There are 2 option in this case:

the default is PERMANENT_LIFETIME - on the first request for the object it's created, the next test that will ask for it will get the old object. All the test in the scenario/suite will use the same object.
TEST_LIFETIME - after the test execution the object is close. The next test will get new constacted object.

Hirarchy

You can create and hirarchy of SystemObject by creating a public member of SystemObject inside a parent SystemObject. In the SUT XML you should create an internal tag that will represent the internal object. You can use the getParrent method in the child object to get its parent.

Properties

Every SystemObject tag in the SUT XML (even once without set method) can be access using the getProperty method. The parameter is the tag name.