Guidelines to improve automated GUI tests stability
Author:
Adam Sotona,
Jiri Skrivanek
Updated: April 18, 2005
After all work dedicated to improving stability of UI tests in Jemmy
and Jelly, it is not possible to take care about all situations. Test
developers has to pay an extra attention to writing their test cases.
First condition to a good automated test case is a manual scenario
where
all steps have to be verifiable. For atomic actions like JTabbedPane
tab selection or JTree node selection, there is already verification
implemented inside Jemmy. For more complex actions, test developer has
to do verification himself. In majority of cases verification is done
by next step in scenario. For example, pushing Tools -> Options main
menu item will result in opening of Options dialog. Verification that
the dialog is opened is done by command "new OptionsOperator();"
because constructor of an operator waits until appropriate component is
found. In this case it will wait until dialog with title "Options"
appears. Instance of OptionsOperator is then used in the rest of test
case. So there is no need for an additional verification.
In cases where an extra verification is needed, you can use Jemmy or
Jellytools which gives us possibilities how to solve even non standard
cases. If you don't find any instructions how to solve your particular
case in this document, please, send us an email to either
users@jemmy.netbeans.org
or
users@jellytools.netbeans.org.
JellyTestCase usage
JellyTestCase helps you with default initialization.
If tests are intended to be used in
XTest
harness, it is recommended to use
org.netbeans.jellytools.JellyTestCase
as ancestor for all test classes. It initializes jemmy and helps to
easier resolving of failed tests. Its capabilities are:
- redirect output messages from jemmy to file jemmy.log
in
working directory
- wait at least 1000 ms between test cases (waiting is done by new
EventTool().waitNoEvent(1000);)
- create screen shot (screen.png) if test fails
- close all modal dialogs if test fails
- dump xml hierarchy of all components (disabled by default)
Description of JellyTestCase can be found in other
documentation.
Use Jemmy waiter
Jemmy waiter enables to wait for anything for given period of time.
Use jemmy waiter, if you want to wait for an action. You can wait until
dialog dismiss, button is enabled, status line changed and so on. In
the
following example we want to wait until a connection dialog dismiss.
Remember that we have to locate the dialog first and then wait until it
is no more found. Default time to wait is 60 second and it can be
changed as stated in example:
// wait connecting
dialog appears
String connectingTitle =
"Connecting";
new
NbDialogOperator(connectingTitle);
// wait 30 second until
connecting dialog dismiss
try {
JemmyProperties.setCurrentTimeout("Waiter.WaitingTime", 30000);
new
Waiter(new Waitable() {
public Object actionProduced(Object title) {
return JDialogOperator.findJDialog(title.toString(), true, true) ==
null ? Boolean.TRUE : null;
}
public String getDescription() {
return("Wait until Connecting dialog dismiss.");
}
}).waitAction(connectingTitle);
} catch
(InterruptedException e) {
throw new JemmyException("Interrupted.", e);
}
Inactivity waiting
Wait for application inactivity.
If there is no point to be synchronized to and previous operation could
affect next operation, it is recommended to use waiting for inactivity:
new EventTool().waitNoEvent(time);
It will wait until no event is registered for a given number of
milliseconds. The smallest recommended number is 500.
Focus on test cases
Use GUI actions for test cases only and for environment preparation
use IDE API calls as much as possible.
Before test execution there are usually lot of things to prepare and
during these preparations lot of tests fail. It is recommended to split
current test cases into preparation parts and testing parts.
Preparation part could be performed using IDE API calls to avoid
failure before main testing part.Main testing part than can be
performed normally using Jemmy or Jelly.
Example could be JavaCVS validation test suite where some settings
must be changed before testing.
Accessing and changing some settings in Options could be very unstable
set of actions, but using direct API call is simple and stable one
line of code:
((JavaCvsSettings)SystemOption.findObject(JavaCvsSettings.class,
true)).setUiMode(1);
Use default Jemmy Dispatching Model
Use default Jemmy Dispatching Model rather than Robot Model.
Do not change Jemmy Event Dispatching Model. Use Robot mode only for
cases where you need to simulate real-like user interaction.
Switching can be performed by:
JemmyProperties.setCurrentDispatchingModel(JemmyProperties.ROBOT_MODEL_MASK);
or:
JemmyProperties.setCurrentDispatchingModel(JemmyProperties.getDefaultDispatchingModel());
and locally you can change dispatching model:
<operator instance>.setDispatchingModel(...);
Use Debug Timeouts
Testing machines have different performance than your workstation
and occasionally they can a little jammed during tests.
It is not recommended to use debug timeouts for all test cases because
it can slow down test execution time rapidly. But they can help to
stabilize section of a test case where other synchronization is not
possible.
Timeouts should be changed before first test case by:
JemmyProperties.getCurrentTimeouts().loadDebugTimeouts();
They can be reverted back any time during test execution by command:
JemmyProperties.setCurrentTimeouts(new
Timeouts());
Delay start of automated tests
IDE is usually very busy just after first start with empty user
settings, so let IDE do its job first.
Using XTest you can add some
Thread.sleep(time) or
new
EventTool().waitNoEvent(time) into
suite() method
to
postpone test execution until no events are generated by IDE itself. If
you use JellyTestCase as ancestor for your classes, it already waits
1000 ms before each test case is started.
Prolong timeouts in critical sections
Default and sometimes even debug timeouts are not enough in some
cases.
Timeouts should be prolonged in cases like waiting for dialog with
response from some server (database, versioning, update center) or
waiting for some longer actions (compilation, execution).
In case of possible very long response do not block test for this long
time but split critical section into invocation and verification part
and verification part treat as unstable (see section below).
Timeout could be changed globally by:
JemmyProperties.setCurrentTimeout(String name, long
newValue);
or locally:
<operator instance>.getTimeouts().setTimeout(String
name, long newValue);
Set of timeouts can be loaded from external file by:
JemmyProperties.getCurrentTimeouts().load(filename
or input stream);
For example here is content of debug.timeouts file used
by loadDebugTimeouts():
AbstractButtonOperator.PushButtonTimeout=100
ComponentOperator.AfterDragTimeout=100
ComponentOperator.BeforeDragTimeout=100
ComponentOperator.MouseClickTimeout=100
ComponentOperator.PushKeyTimeout=100
DialogWaiter.AfterDialogTimeout=3000
EventDispatcher.RobotAutoDelay=100
FrameWaiter.AfterFrameTimeout=3000
JComboBoxOperator.BeforeSelectingTimeout=100
JComponentOperator.ShowToolTipTimeout=1000
JMenuItemOperator.PushMenuTimeout=100
JMenuOperator.WaitBeforePopupTimeout=100
JScrollBarOperator.OneScrollClickTimeout=10
JSplitPaneOperator.ScrollClickTimeout=10
JTextComponentOperator.BetweenKeysTimeout=100
JTextComponentOperator.PushKeyTimeout=100
JTextComponentOperator.TypeTextTimeout=60000
JTreeOperator.WaitAfterNodeExpandedTimeout=100
WindowWaiter.AfterWindowTimeout=3000
Repeat unstable actions
Repeat unstable actions in case of exception but do not create
infinite loops.
In case that some action is really unstable, repeat action invocation
with catching possible exceptions.
Restrict cycle to some number of tries or timeout and do not forget on
short sleep before action is repeated.
Use supported environment
Tests well-tuned on extraordinary environment probably fail on test
servers.
Use supported window managers and default environment configuration
during tests development.
Some critical settings are:
- disable focus follows mouse
- enable to focus application windows when they first appear
- enable raise windows when focused
- enable dialogs to inherit focus from parents
- set focus on click
- disable display tooltips
- disable screen saver on test servers
