Java – Setting up an automated testing framework with Selenium

Architecturejavaseleniumtestingwebdriver

I am working on large websites that get updated (new features added, content changed, etc.) at least once a week. testing however, if at all present, is never as thorough as it should be. so a week ago i started looking into automated testing and selenium.

I read that if you wanna do it right, dont rely solely on selenium ide. programming your tests offers a lot more options. i wasn't comfortable with the third party PHP bindings at all so i quickly skipped to using java for programming the tests.

Now my question revolves around how to cleverly setup the system that will be running the tests. my idea was the following:

the tests will/should be totally autonomous
nevertheless i want to have some kind of main application to loop all these tests and run them one after another
the main application should be running 24/7 or at least get started periodically by a cronjob or something
tests should be run in as much browsers (or simulated clients??) as possible. so far i'm using FF, IE and chrome through their respective WebDrivers.
I wanna name some folder and collect all the tests in that folder. the main application will then just take whatever is in that folder and run it. then everybody will just create their tests and put them in that folder, so they'll get added to the queue
it would be nice to have a way to create tests with selenium ide, export them and they immediately work with my main application (i.e. they get run correctly without any modifications needed). ok, i know i said, selenium ide is disadvised but i feel it's a good start for some simple tests. also, people without any programming knowledge can still use it. as it is right now, i'm exporting the tests created with selenium ide as "Java / JUnit 4 / WebDriver" and changing a little bit of code (passing a WebDriver obect as a parameter).

Here is the code for my "main application":

import java.io.File;
import java.io.FilenameFilter;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.concurrent.TimeUnit;

import org.openqa.selenium.WebDriver;

public class SeleniumTest {

    public static void main(String[] args) throws 
    ClassNotFoundException, InstantiationException,IllegalAccessException, 
    NoSuchMethodException, SecurityException, IllegalArgumentException, 
    InvocationTargetException {
        String[] drivers = {
                "org.openqa.selenium.firefox.FirefoxDriver",
                "org.openqa.selenium.ie.InternetExplorerDriver",
                "org.openqa.selenium.chrome.ChromeDriver"
        };
        Class params[] = {WebDriver.class};
        String testFolderName = "lookhere";

        while (true) {
            //get all .java files from some folder. those will be the tests to run
            File testFolder = new File(testFolderName);
            FilenameFilter filter = new FilenameFilter() {
                public boolean accept(File dir, String name) {
                    return name.endsWith(".java");
                }
            };
            String[] tests = testFolder.list(filter);
            if (tests == null) {
                System.exit(0);
            }

            for(int i=0; i<drivers.length; i++) {
                //loop through different drivers, instantiate them
                Class webDriverClass = Class.forName(drivers[i]);
                WebDriver driver = (WebDriver) webDriverClass.newInstance();
                driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

                for (int j=0; j<tests.length; j++) {
                    //loop through tests, instantiate them
                    String currentTestName = tests[j].replace(".java", "");
                    Class testClass = Class.forName(currentTestName);
                    Object test = testClass.newInstance();

                    Method methods[] = testClass.getDeclaredMethods();
                    Method fn;

                    //run every method that starts with "test" (usually just one)
                    //also run setUp, passing the driver object
                    for(int k=0; k<methods.length; k++) {
                        if (methods[k].getName().startsWith("test")) {
                            fn = testClass.getDeclaredMethod(methods[k].getName());
                            fn.invoke(test);
                        } else if(methods[k].getName().equals("setUp")) {
                            fn = testClass.getDeclaredMethod("setUp", params);
                            fn.invoke(test, driver);
                        }
                    }
                }

                driver.quit();
            }
        }
    }
}

As you can probably see, i am not at all an expert at java, so dont be put off by bad form (like the long throws-declaration of the main method).

Now, for my questions:

is my entire concept, that i formulated above, okay? that is: having a main application run all the tests and a queue to which i can add/remove tests on the fly. or are there other methods of best practice regarding this? what other concept/architecture can you think of?
is there a way to create tests with selenium ide, export them and they immediately work with the main application without any modification needed?
ok since i posted code, you probably wanna go crazy and criticize that 😉 i'm open for all kinds of suggestions.
is it possible with selenium to run different versions of IE (or any browser, for that matter) ? it probably just runs the browser version that i have installed. so if i wanna test different versions of one browser, i have to call remote machines which means selenium grid? i'm really just guessing right now 🙂

Best Answer

I would use TestNG to orchestrate tests instead of a main method, it allows for parameterisation as the previous answer states, and if you use selenium GRID 2, you can run tests in parallel and on many different browser/OS version combo's

I would also consider modeling test cases using PageObjects & PageFactories, this will further help in making tests encapsulated and robust.

Related Solutions

Java – What are the effects of exceptions on performance in Java

It depends how exceptions are implemented. The simplest way is using setjmp and longjmp. That means all registers of the CPU are written to the stack (which already takes some time) and possibly some other data needs to be created... all this already happens in the try statement. The throw statement needs to unwind the stack and restore the values of all registers (and possible other values in the VM). So try and throw are equally slow, and that is pretty slow, however if no exception is thrown, exiting the try block takes no time whatsoever in most cases (as everything is put on the stack which cleans up automatically if the method exists).

Sun and others recognized, that this is possibly suboptimal and of course VMs get faster and faster over the time. There is another way to implement exceptions, which makes try itself lightning fast (actually nothing happens for try at all in general - everything that needs to happen is already done when the class is loaded by the VM) and it makes throw not quite as slow. I don't know which JVM uses this new, better technique...

...but are you writing in Java so your code later on only runs on one JVM on one specific system? Since if it may ever run on any other platform or any other JVM version (possibly of any other vendor), who says they also use the fast implementation? The fast one is more complicated than the slow one and not easily possible on all systems. You want to stay portable? Then don't rely on exceptions being fast.

It also makes a big difference what you do within a try block. If you open a try block and never call any method from within this try block, the try block will be ultra fast, as the JIT can then actually treat a throw like a simple goto. It neither needs to save stack-state nor does it need to unwind the stack if an exception is thrown (it only needs to jump to the catch handlers). However, this is not what you usually do. Usually you open a try block and then call a method that might throw an exception, right? And even if you just use the try block within your method, what kind of method will this be, that does not call any other method? Will it just calculate a number? Then what for do you need exceptions? There are much more elegant ways to regulate program flow. For pretty much anything else but simple math, you will have to call an external method and this already destroys the advantage of a local try block.

See the following test code:

public class Test {
    int value;


    public int getValue() {
        return value;
    }

    public void reset() {
        value = 0;
    }

    // Calculates without exception
    public void method1(int i) {
        value = ((value + i) / i) << 1;
        // Will never be true
        if ((i & 0xFFFFFFF) == 1000000000) {
            System.out.println("You'll never see this!");
        }
    }

    // Could in theory throw one, but never will
    public void method2(int i) throws Exception {
        value = ((value + i) / i) << 1;
        // Will never be true
        if ((i & 0xFFFFFFF) == 1000000000) {
            throw new Exception();
        }
    }

    // This one will regularly throw one
    public void method3(int i) throws Exception {
        value = ((value + i) / i) << 1;
        // i & 1 is equally fast to calculate as i & 0xFFFFFFF; it is both
        // an AND operation between two integers. The size of the number plays
        // no role. AND on 32 BIT always ANDs all 32 bits
        if ((i & 0x1) == 1) {
            throw new Exception();
        }
    }

    public static void main(String[] args) {
        int i;
        long l;
        Test t = new Test();

        l = System.currentTimeMillis();
        t.reset();
        for (i = 1; i < 100000000; i++) {
            t.method1(i);
        }
        l = System.currentTimeMillis() - l;
        System.out.println(
            "method1 took " + l + " ms, result was " + t.getValue()
        );

        l = System.currentTimeMillis();
        t.reset();
        for (i = 1; i < 100000000; i++) {
            try {
                t.method2(i);
            } catch (Exception e) {
                System.out.println("You'll never see this!");
            }
        }
        l = System.currentTimeMillis() - l;
        System.out.println(
            "method2 took " + l + " ms, result was " + t.getValue()
        );

        l = System.currentTimeMillis();
        t.reset();
        for (i = 1; i < 100000000; i++) {
            try {
                t.method3(i);
            } catch (Exception e) {
                // Do nothing here, as we will get here
            }
        }
        l = System.currentTimeMillis() - l;
        System.out.println(
            "method3 took " + l + " ms, result was " + t.getValue()
        );
    }
}

Result:

method1 took 972 ms, result was 2
method2 took 1003 ms, result was 2
method3 took 66716 ms, result was 2

The slowdown from the try block is too small to rule out confounding factors such as background processes. But the catch block killed everything and made it 66 times slower!

As I said, the result will not be that bad if you put try/catch and throw all within the same method (method3), but this is a special JIT optimization I would not rely upon. And even when using this optimization, the throw is still pretty slow. So I don't know what you are trying to do here, but there is definitely a better way of doing it than using try/catch/throw.

Java – How to create an executable JAR with dependencies using Maven

<build>
  <plugins>
    <plugin>
      <artifactId>maven-assembly-plugin</artifactId>
      <configuration>
        <archive>
          <manifest>
            <mainClass>fully.qualified.MainClass</mainClass>
          </manifest>
        </archive>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
      </configuration>
    </plugin>
  </plugins>
</build>

and you run it with

mvn clean compile assembly:single

Compile goal should be added before assembly:single or otherwise the code on your own project is not included.

See more details in comments.

Commonly this goal is tied to a build phase to execute automatically. This ensures the JAR is built when executing mvn install or performing a deployment/release.

<plugin>
  <artifactId>maven-assembly-plugin</artifactId>
  <configuration>
    <archive>
      <manifest>
        <mainClass>fully.qualified.MainClass</mainClass>
      </manifest>
    </archive>
    <descriptorRefs>
      <descriptorRef>jar-with-dependencies</descriptorRef>
    </descriptorRefs>
  </configuration>
  <executions>
    <execution>
      <id>make-assembly</id> <!-- this is used for inheritance merges -->
      <phase>package</phase> <!-- bind to the packaging phase -->
      <goals>
        <goal>single</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Best Answer

Related Solutions

Java – What are the effects of exceptions on performance in Java

Java – How to create an executable JAR with dependencies using Maven

Related Topic