1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
|
# TestBench tests in Vaadin Framework
The project currently supports running TestBench 3+ (java) tests. Each test consists of two files:
* A TestBench-based Java JUnit test; so-called TB3+ test.
* A Vaadin UI class which contains the component under test.
The Java JUnit test uses TestBench/Selenium/browser and therefore needs an UI to connect to; that UI
is provided by the appropriate Vaadin UI class.
All test UI classes go into `uitest/src`. These files are automatically packaged into a war file which is deployed to a Jetty server during the build process so that tests can open and interact with the UI:s. For development purposes, the Jetty server can be started (both in Eclipse, Intellij and from command-line), see running tests in Eclipse.
The project is setup so that `/run` is mapped to a specialized servlet which allows you to add the UI you want to test to the URL, e.g. [http://localhost:8888/run/com.vaadin.tests.component.label.LabelModes](http://localhost:8888/run/com.vaadin.tests.component.label.LabelModes) or just [http://localhost:8888/run/LabelModes](http://localhost:8888/run/LabelModes) if there are no multiple classes named LabelModes. Because of caching, the `?restartApplication` parameter is needed after the first run if you want to run multiple test classes.
# QuickStart: Running one test
The test logic code resides in the TB3+ JUnit tests. However the tests need a webserver to connect to. Read below
on how to run such a webserver.
## Starting the WebServer: Hosting your UI class in Jetty
Prior running your TB3+ JUnit tests, we need to have the webserver
hosting testing UI classes up and running. See below for instructions for particular IDEs; the
instructions for a pure command-line approach are also present.
### Eclipse
The Jetty included in the project can be started in Eclipse by running the “Development Server (Vaadin)” launch configuration in `/eclipse` (right click -> Debug as -> Development Server (Vaadin) ). This deploys all the tests to `localhost:8888` (runs on port 8888, don’t change that unless you want problems). Use `/run/<uiclass>` to open a test.
The DevelopmentServerLauncher has a built-in feature which ensures only one instance can be running at a time. There is therefore no need to first stop the old and and then start a new one. Start the server and the old one will be killed automatically.
### Command-line
```bash
cd uitest
mvn jetty:run-exploded
```
The server should now be running at port 8888. Every testing UI class
is published as [http://localhost:8888/run/<testUI>](http://localhost:8888/run/<testUI>).
For example you can see the testing UI for the `GridEditorTabSkipsNonEditableCellsTest` here: [http://localhost:8888/run/GridEditorTabSkipsNonEditableCells](http://localhost:8888/run/GridEditorTabSkipsNonEditableCells).
### Intellij
We're going to do the same as with the command-line approach - launch the `mvn jetty:run-exploded`:
1. Open *Maven Projects*
1. Open *vaadin-uitest* -> *Plugins* -> *jetty* -> *jetty:run-exploded*
1. Open URL [http://localhost:8888/run/<testUI>](http://localhost:8888/run/<testUI>)
## Setup before running any tests from your IDE
Before running any tests from your IDE you need to
1. copy `uitest/eclipse-run-selected-test.properties` to `work/eclipse-run-selected-test.properties`
2. edit `work/eclipse-run-selected-test.properties`
1. Define `com.vaadin.testbench.screenshot.directory` as the directory where you checked out the screenshots repository (this directory contains the “references” subdirectory)
2. Set `com.vaadin.testbench.deployment.url=http://localhost:8888/`
3. Set `com.vaadin.testbench.runLocally=chrome` to only run tests on Chrome. On Ubuntu you can then install Chrome driver easily:
`sudo apt install chromium-chromedriver`
> Note: the file is named `eclipse-*` but it applies to all IDEs.
## Running TB3+ tests
Now that the UI Jetty server is up and running and the `work/eclipse-run-selected-test.properties` file is properly created,
you can run the actual tests.
TB3+ tests are standard JUnit tests which can be run using the “Run as -> JUnit test” (or Debug as) in Eclipse or Intellij.
Right click on a single test instance in the JUnit result view and select to debug to re-run it on a single browser.
As an example, you can find the `GridEditorTabSkipsNonEditableCellsTest` JUnit test,
right-click the class and chose "Run As JUnit test".
Done: your environment is set up properly. Now you can proceed to writing your own tests.
# Creating a new test
Creating a new test involves two steps: Creating a test UI (typically this should be provided in the ticket) and creating a TestBench test for the UI.
## Creating a new test UI
Test UIs are created inside the `uitest/src` folder in the `com.vaadin.tests` package. For tests for a given component, use the `com.vaadin.tests.components.<component>` package. For other features, use a suitable existing com.vaadin.tests.<something> package or create a new one if no suitable exists.
The test should be named according to what it tests, e.g. EnsureFormTooltipWorks. Names should not refer to a ticket, e.g. Ticket1123 will automatically be rejected during code review as it is non-descriptive.
There are a couple of helper UI super classes which you can use for the test. You should never extend UI directly:
* `AbstractTestUI`
* Automatically sets up a VerticalLayout with a description label on the top. Use addComponent() to add components to the layout
* Supports ?transport=websocket/streaming/xhr parameter for setting push mode. This is for testing core push functionality, typically you should just add @Push to your test UI
* `AbstractTestUIWithLog`
* AbstractTestUI but adds a log at the top to which you can add rows using log(String). Handy for printing state information which the TB test can assert reliably.
* `AbstractComponentTest`, `AbstractFieldTest`, ...
* Base classes for generic component tests. Generates test which have a menu on the top containing options for configuring the component. Classes follow the same component hierarchy as Vaadin component classes and this way automatically gets menu items for setting features the parent class supports.
* Gotcha: If you add a new feature to a menu you need to run and possibly (probably) fix all TB tests which use the class as they will click on the wrong item (fixable by implementing [http://dev.vaadin.com/ticket/11307](http://dev.vaadin.com/ticket/11307))
[Note] `AbstractTestCase` and `TestBase` are old base classes for tests. Don't use them for any new tests. They extend `LegacyApplication` which is a deprecated class.
## Creating a TestBench test for a UI
All new test for the projects must be created as TestBench3+ tests
### Test class naming
TestBench 3+ tests usually follow a naming convention which automatically maps the test to the UI class. By the convention, the TB3 test class should be named **UIClassTest**, e.g. if UI is **LabelModes** then the TB3+ test is **LabelModesTest**. Inside the LabelModesTest class there are 1..N methods which test various aspects of the LabelModes UI. This is the preferred way, but not strictly necessary; sometimes it's more conventient to use a different name for the UI and test class. In that case, remember to override the `AbstractTB3Test::getUIClass()` method in the test.
### Super class
There are a couple of super classes you can use for a TB3+ test:
* `MultiBrowserTest`
* Ensures the test is run on the browsers we support automatically
* **This is what you typically should use**
* `WebsocketTest`
* Run only on browsers which supports websockets
### Creating the test
The actual test is one method in the test class created for the given UI. The method declaration is something like:
@Test
public void testLabelModes() throws Exception {
`@Test` is needed for it to be run at all.
`throws Exception` should usually be added to avoid catching exceptions inside the test, as most often an exception means the test has failed. Unhandled exceptions will always fail the test; if you use checked exceptions, you'll need to handle them somehow or decide they're automatically failures on exception.
The beginning of the test should request the UI using `openTestURL()`;
public void testLabelModes() throws Exception {
// Causes the test to be opened with the debug console open. Typically not needed
// setDebug(true);
// Causes the test to be opened with push enabled even though the UI does not have @Push.
//Typically not needed
// setPush(true);
openTestURL();
After this it is up to you what the test should do. The browser will automatically be started before the test method and shutdown after it. If the test fails, a failure screenshot will automatically be created in addition to an exception being thrown.
When done, run the test locally, as described below, and ensure it passes. Then run it remotely, as described below, a couple of times to ensure it passes reliably.
### Best practices for creating a TB3 test
#### Communicate intent through your test class
JUnit allows you (as opposed to TB2 HTML tests) to describe what the test should do using method names. The test method should be a chain of calls which enables any reader to quickly understand what the test does, all details should be abstracted into methods.
Over the time we should collect useful helper methods into the parent classes (AbstractTB3Test already contain some, e.g. assertGreater/assertLessThan/vaadinElementById/...).
#### Do a sensible amount of things in one test method
There is an overhead before and after each test method when the browser instance is started. The tests are therefore not restricted to testing one single thing. Instead a test method should test one logical group of things (TB3 tests are not pure unit tests).
#### Use ids in your test class
Use ids in your UI class. Define the IDs as constants in the UI class. Use the constants in your Test class. Use static imports to avoid massive prefixes for the constants.
public class NativeButtonIconAndText extends AbstractTestUI implements
ClickListener {
static final String BUTTON_TEXT = "buttonText";
[...]
buttonText.setId(BUTTON_TEXT);
public class NativeButtonIconAndTextTest extends MultiBrowserTest {
@Test
public void testNativeButtonIconAltText() {
openTestURL();
assertAltText(NativeButtonIconAndText.BUTTON_TEXT, "");
## Debugging TB3+ tests
### Debugging remotely
Running remotely on a single browser (as described above) can be used to debug issues with a given browser but with the downside that you cannot see what is happening with the browser. In theory this is possible if you figure out on what machine the test is run (no good way for this at the moment) and use VNC to connect to that.
### Debugging locally
A better option is to run the test on a local browser instance. To do this you need to add a `@RunLocally` annotation to the test class. `@RunLocally` uses Firefox by default but also supports other browsers using `@RunLocally(CHROME)`, `@RunLocally(SAFARI)`, `@RunLocally(Browser.IE11)` and `@RunLocally(PHANTOMJS)`.
By default using `@RunLocally` annotation in Framework tests is not allowed. In order to run a test locally, you need to uncomment the line `com.vaadin.testbench.allowRunLocally=true` in `work/eclipse-run-selected-test.properties`.
Besides, some local configurations are needed for certain browsers, especially if you did not install them in the “standard” locations.
**PhantomJS**: PhantomJS is a headless browser, good especially for fast validation. Download it from the [PhantomJS site](http://phantomjs.org/download.html) and add the binary to your PATH.
**Firefox**: If you have Firefox in your PATH, this is everything you need. If Firefox cannot be started, add a **firefox.path** property to `/work/eclipse-run-selected-test.properties`, pointing to your Firefox binary (e.g.`firefox.path=/Applications/Firefox 17 ESR.app/Contents/MacOS/firefox`).
**Chrome**: You need to [download ChromeDriver](http://chromedriver.storage.googleapis.com/index.html) and add a chrome.driver.path property to `/work/eclipse-run-selected-test.properties`, pointing to the ChromeDriver binary (NOT the Chrome binary).
**IE11**: You need to [download IEDriverServer according to the selenium version](http://selenium-release.storage.googleapis.com/index.html) and add a webdriver.ie.driver property to `/work/eclipse-run-selected-test.properties`, point to the IEDriveServer binary.
**Safari**: At least on Mac, no configuration should be needed.
Remember to remove `@RunLocally()` before committing the test or it will fail during the build.
|
