JSystem 2.0.2 User Manual

Last Modified Date: 04/30/2005
Author: Yoram Shamir

JSystem is a framework for writing and running automated functional/system tests, based on JUnit. It is highly recommended to be familiar with JUnit before starting to work with JSystem.
This framework is a collection of features that target to improve the maintainability and visibility of your test automation project.

JSystem comes with a runner user interface that extends the JUnit user interface. All the features will work using the original UI runner or any other runner. So if you execute your tests using Ant or IntelliJ, Eclipse or any other environment, the features are expected to work.

Table Of Content

JSystem Runner

Scenario
Reporter
Publisher

Fixtures

What is a fixture?
Why do we need fixtures?
Fixtures Management

Enhanced Reporting

Extendable Interface
HTML Reports

SUT Independence
System Objects
Analyzers

JSystem Runner

JSystem Tests Tree, the JSystem runner, is JSystem end user interface. It allows the user to build test scenarios, execute them and publish the test results.
JSystem runner can run with any JUnit runner.

JSystem runner is built of two panes - tests tree and operations tabs. The tests tree lists all test cases for the SUT. The operations tabs includes four tabs - Scenario, Reporter, Fixture and Publisher.

In order to build and run a test scenario the user simply selects tests from the tests tree and set their order. Once the scenario is ready the user can run it and see online report of the test execution. After the execution is complete, the user can publish the test results to an HTML server.

Scenario

The Scenario tab lists all tests within the scenario.

The up and down buttons allow the user to change the test order within the scenario.

When standing on the test tree root and pressing the right mouse button the following pop-up menu will open:

Check Selection - will check (select) the current sub tree.
Uncheck Selection - will uncheck (unselect) the current sub tree.
Set as current - will set this tree root as the current tree root.
Save scenario as … - will save the current scenario in XML format. The scenario will be saved in the Scenarios directory under the tests classes path.
Load scenario - will load a saved scenario.
Export scenario - will export the current scenario, in Jar format. The user can select the destination directory.
Import scenario - will import a scenario in Jar format.
Publish scenario - will publish a scenario in Jar format.

Reporter

The reporter tab allows the user to see an online run report. Only the test titles and their status are available in this report.
The HTML report is available under Tools->Reports->Html menu.

Publisher

The publisher tab allows the user to edit scenario run results and publish them to an HTML server.

The Reload button

allows the user to load the run results. The user can also use this button to undo changes.
The Load button

allows the user to load an external XML report file.
The Filter pull-down menu allows the user to filter results. The default is to view all of the results. The user can select 'Fail only' to view only the results that failed and 'Fail & Ignore' to see also ignored reports.
The Ignore and Delete buttons

allow the user to edit the report. The user can ignore failures or completely delete tests.
The Save button

allows the user to save the report.
Use Publish button

allows the user to send the results to a web server.

Note:
The publisher is not enabled by default. In order to enable it, edit the jsystem.properties file in the root of the extracted directory.
Go to the reporterClasses property and append jsystem.extensions.report.xml.XmlReporter. You must restart JSystem runner.

Fixtures

Fixture management in general enables you to create a tree of fixtures which you can assign/position tests to.

What is a fixture?

A fixture represents a state of the Software/System under test. For example, think of web application testing: one fixture (and obviously the basic) can be a machine with nothing installed on it, this state can be defined as the root fixture. A second fixture can be the machine installed with the application under test. An additional fixture can be the application configured with some data, and yet additional fixture can be the application configured with some different data.

To define a fixture two things should be defined: setup - the way to get from the initial state to the fixture state, and teardown - the way to return from the fixture back to the original state.

In the JSystem model the fixtures are arranged in a tree. Every fixture is pointing to it's parent.

Why do we need fixtures?

Every test, especially a functional/system test, has a fixture. In some of the cases the fixture is very complicated. One way is is to enter the fixture setting and tear-down into the test itself. However, this way the test gets messy with irrelevant details.
The fixture approach takes the details of of fixtures out of the test.

How to build and use fixtures tree?

If not configured, the default system state will be the root fixture.
If you run a test that is not under the current fixture (root), all the fixtures along the path to the required one will be executed first and only then the test will be executed.

Following is code example of new fixture, inheriting from the root fixture:


package jsystem.tests.framework.fixture;

import jsystem.framework.fixture.Fixture;

public class Fixture1 extends Fixture {

    public void setUp() throws Exception {
	    System.out.println("Fixture1-setUp>>");
		// Fixture setup...
	}
	
	public void tearDown() throws Exception {
	    System.out.println("Fixture1-tearDown>>");
		// Fixture tear-down...
	}

}

This code example shows:

Every fixture should extend the Fixture class.

setUp and tearDown methods MUST be implemented.

The default parent fixture is the root fixture.

Following is code example of new fixture, inheriting from the new fixture:


package jsystem.tests.framework.fixture;

import jsystem.framework.fixture.Fixture;

public class Fixture1_1 extends Fixture {

    public Fixture1_1() {
	    setParentFixture(Fixture1.class);
	}
	
	public void setUp() throws Exception {
	    System.out.println("Fixture1_1-setUp>>");
		// Fixture setup...
	}
	
	public void tearDown() throws Exception {
	    System.out.println("Fixture1_1-tearDown>>");
		// Fixture tear-down...
	}
	
	public void failTearDown() {
	    System.out.println("Fixture1_1-failTearDown>>");
		// Fixture tear-down in case of failure...
    }

}

This code example shows:

The setParentFixture method should be used in the constructor to set the fixture parent.

Apart from the setUp and tearDown methods (now optional) there is optional failTearDown method.

Following is code example of simple test using fixture:


package tests.jsystem.framework;

import junit.framework.SystemTestCase;
import tests.jsystem.framework.fixture.Fixture1_1;

public class SimpleSystemTestCase extends SystemTestCase {

    public SimpleSystemTestCase(String name) {
        super(name);
        setFixture(Fixture1_1.class);
    }

    public void testRun() throws Exception {
        System.out.println("SimpleSystemTestCase-testRun>>");
        // Test...
    }

}

This code example shows:

In order to use a fixture the test should extend SystemTestCase.

The fixture that the test will be bound to is set in the constructor of the test.

There is no need to call tearDown. It is called automatically when new fixture is set.

The code examples above creates small tree of fixtures (Fixture1 and Fixture1_1) and connects a test (SimpleSystemTestCase) to one of the tree leaves (Fixture1_1).

Fixture Management

In order to help managing the fixtures tree, the JSystem runner includes a fixture manager, under the Fixture tab.

The blue point represents the current position in the fixture tree.
When JSystem starts it assumes that the system under test is in the root position.
By selecting a fixture and pushing the "go to…" button, the fixture path to the selected fixture will be executed. Likewise, by selecting a fixture and pushing the "fail to…" button, the fixture path to the selected fixture will be teared down.
Pushing the "set current" button will only set the selected fixture as the current fixture (without executing anything).
The "disable" check-box will block any fixture execution.

Note:
The fixture manager is available only if the JSystem TestRunner is executed.

Enhanced Reporting

JUnit supports a very basic level of reporting. Only failures are logged. In system testing where the test complexity increases and the test execution time is longer a mechanism to report every step of the test is required. Such mechanism should allow the user to monitor the test flow, perform root cause analysis and keep an historical run results. JSystem supports such reporting.

JSystem reporting mechanism is composed of two components. One is an extendable interface - TestReporter - that enables the user to add more reporting capabilities. The second is an HTML reporter that gives a detailed and hierarchal view of your tests.

In order to add a report the user should use the report object (public & static) in SystemTestCase.

Extendable Interface

In order to extended the built in reporting capabilities the TestReporter interface should be implemented. The interface has 4 methods:

report: will be called when a new report is added.
getName: should return the reporter name to be presented.
asUI: should return true if this reporter is a UI. UI reporters are listed in the Tools->reporters menu.
initReporterManager: if the reporter is UI (asUI returns true) this method will be launched when the user selects it on the Tools->reporters menu. A frame or dialog should be presented to support the configuring or managing of the reporter.

The reporters can implement two other interfaces - TestListener and FixtureListener. If implemented they will be called automatically on test or fixture events.
To initiate a reporter it's class name should be added to the reporterClasses system parameter.

HTML Reports

HTML reporting is the default enhanced reporter that comes with JSystem. It can be launched from Tools->reporters->html report menu.

The upper left frame shows a list of all packages of the scenario that have been executed. Clicking on one of the packages displays a list of all the tests in this package that have been executed, in the lower left frame. Clicking on one of the tests displays it's details in the right frame.

By setting testsSrc attribute in the jsystem.properties file to the root directory of tests classes (default is testsSrc=./tests) the user can drill down all the way to the test source in the report.

The user can add attributs to the summary report by editing the summary.properties file. This can be used to add information like setup name, description version number and more.

In order to keep historical results, when JSystem runner starts it will zip the previous test results and save them to a file. Old reports can be found in the log/Old directory.

SUT Independence

SUT stands for System/Software Under Test. In most testing labs there are several test setups where the SUT is tested.
For example, think of a web application: in the lab there are few application servers with different IPs and URLs. If the URL is hard coded inside the test code it will run only on one of the application servers. In order to run it on a different server the test code must be changed.

Of-course, tests must run on all setups without making any changes to them (or to the setups). To solve this problem the setup is defined specifics are set outside the test code, using an XML document.

Following is an SUT example of application server setup:


<sut>
    <appServer>
        <url>http://app1/index.html</url>
        <user>guy</user>
        <password>guy</password>
    </appServer>
</sut>

Following is code example of simple test using SUT:


package tests.jsystem.framework.sut;

import junit.framework.SystemTestCase;

public class MyAppTest extends SystemTestCase {

    public void testBasicUrl() throws Exception {
        String url;
        url = sut.getValue("/sut/appServer/url/text()");
        browser.getNavigator().getDocument(url);
    }

}

sut is an object in SystemTestCase that is initialized by the framework. It represents the XML file in the detailed above. It extracts values from the XML document using XPath syntax.
To set a default SUT file the user should set the sutFile property in the jsystem.properties file. The SUT XML file can be anywhere in the classpath (including inside a jar file).

System Objects

System Object is JSystem convention to represent elements in the test setup. The steps in the tests will is represented by system object methods.

Following is an example showing how to represent a simple test setup with system objects.
Think of web application testing where the setup is built of an application server and a database server. The tests use two interfaces: a web interface to work with the application server and an ODBC interface to work with the database server.

The SUT looks as following:


<sut>
    <setup>
        <class>systemobject.examples.AppServerSetup</class>
        <web>
            <class>systemobject.examples.Web</class>
            <url>http://app1/</url>
        </web>
        <db>
            <class>systemobject.examples.DataBase</class>
            <dbServerName>dbserver1</dbServerName>
            <user>guy</user>
            <password>guy</password>
        </db>
    </setup>
</sut>

This SUT represents three objects: the application server setup object, the web interface object and the database object.
Every one of the objects has a class element, which is used to construct the system object.
Now let's look at the implementation of the three classes.
Every system object in the object tree should extend SystemObjectImpl or implement SystemObject.

The AppServerSetup looks as following:


package systemobject.examples;

import jsystem.framework.system.SystemObjectImpl;

public class AppServerSetup extends SystemObjectImpl {
    public DataBase db;
    public Web web;
}

The setup system object class does not have an internal logic, it just holds the elements system objects classes.

The names of the variables holding the system objects (db and web) should be identical to the XML tag. This way, the initialization of these fields is done automatically (using reflection).

All the system object classes should be excluded from the JUnit class loader. In this case the systemobject package is excluded. You can use excluded.properties files located in the junit.jar to exclude other packages.

The DataBase looks as following:


package systemobject.examples;

import jsystem.framework.system.SystemObjectImpl;
import java.sql.Connection;
import java.sql.SQLException;

public class DataBase extends SystemObjectImpl {

    public static final String USERS_TABLE = "users";
    public static final String USER_FIELD = "user";
    public static final String PASSWORD_FIELD = "password";
    
    private String dbServerName = null;
    private String user = null;
    private String password = null;
    private Connection connection = null;

    public void init() throws Exception {
        super.init();
        user = sut.getValue(getXPath() + "/user/text()");
        password = sut.getValue(getXPath() + "/password/text()");
        initConnection();
    }

    public void assertShouldBeFoundInTable(String table, String field, String value) {
        // Implementation...
    }

    public void deleteFieldEntry(String tableName, String fieldName, String value) {
        // Implementation...
    }

    private void initConnection() throws SQLException {
        // Init connection implementation....
    }
	
    public void setDbServerName(String dbServerName) {
        this.dbServerName = dbServerName;
    }

}

Again, DataBase extends SystemObjectImpl. The init method will be called by the framework during the object initialization. The system object has access to the SUT XML file and it can and should be used to init the internal fields (user and password).

Another option to initialize a field is by creating a setter (setDbServerName) with the XML tag name (dbServerName). The setter method should have a string as a parameter. The framework will call the set method automatically.
The setter types that are supported are: int, long and String.

The Web looks as following:


package systemobject.examples;

import jsystem.framework.system.SystemObjectImpl;

public class Web extends SystemObjectImpl {

    private String url = null;

    public void init() throws Exception {
        super.init();
        url = sut.getValue(getXPath() + "/url/text()");
    }
	
    public void addUser(String user, String password) {
        // Implementation...
    }

    public void removeUser(String user, String password) {
        // Implementation...
    }

}

Last, lets look at the implementation of the test using the system objects.

The SystemObjectExampleTest looks as following:


package examples.systemobject;

import junit.framework.SystemTestCase;

public class SystemObjectExampleTest extends SystemTestCase {

    AppServerSetup setup;
	
    public void setUp() throws Exception {
        setup = (AppServerSetup)system.getSystemObject("setup");
        setup.web.addUser("user1", "user1");
    }
    
    public void testAddUser() throws Exception {
        setup.db.assertShouldBeFoundInTable(
            DataBase.USERS_TABLE, DataBase.USER_FIELD, "user1");
    }

}

In the setUp we can see how the system object system is used to initialize the setup object setup. Then the web and db objects are used to manage and test.

Note:
By default the initialized object will be initialized once. The next test that uses the object will get the same object.

Following are guidelines to consider when developing system objects:

Methods should be considered as test steps.
As such they should be complete.
They should have all the information in order to decide if the step has passed or failed.
very step (method) should report to the reporter.
In case of a step failure a report should be added and an exception should be thrown.

Analyzers

Analyzers give an easy way to analyze the results of test steps. Users can write their own analyzers by extending AnalyzerParameterImpl (or by implementing AnalyzerParameter).

Following is an example of an text analyzer:

 
package jsystem.extensions.analyzers.text;

public class FindText extends AnalyzeTextParameter {

    protected boolean isRegExp = false;

    public FindText(String toFind) {
        this.toFind = toFind;
    }

    public FindText(String toFind, boolean isRegExp) {
        this(toFind);
        this.isRegExp = isRegExp;
    }

    public void analyze() {
		
        message = testText;
	
        if (testText == null) {
            title = "Text to analyze is null";
            status = false;
        }
		
        if (isRegExp) {
            status = testText.matches(".*" + toFind + ".*");
        } else {
            status = (testText.indexOf(toFind) > 0);
        }
		
        if (status) {
            title = "The text: >" + toFind + "< was found";
        } else {
            title = "The text: >" + toFind + "< wasn't found";
        }
    
    }

}

FindText is an analyzer that verifies that a certain string is found. It supports regular expressions. The analyze method does most of the work. testText is a string that will be set by the system object using this analyzer by calling setTestAgainsObject method.

The analyze method should set the value of three fields: title, message and status. If the status is set to false the test will fail (throwing an exception). All the results will be reported to the reporter.

Following is an example of a test using analyzer:

 
package tests.jsystem.framework.analyzer;

import junit.framework.SystemTestCase;
import systemobject.tests.Device1;
import jsystem.extensions.analyzers.text.FindText;
import jsystem.framework.analyzer.AnalyzerException;

public class AnalyzerTest extends SystemTestCase {

    Device1 device;
	
	public void setUp() throws Exception {
        device = (Device1)system.getSystemObject("device1");
    }
	
    public void testSetAndAnalyze() throws AnalyzerException {
        device.telnet.dirCommand();
        device.telnet.analyze(new FindText("Program Files"));
    }

}

In the example above the test uses the analyzer to test that the string "Program Files" is found in the output of a dir command. The result of the dir command (in this case a string) is set to the analyzer using the setTestAgainsObject method.