Screenshots with Selenium and TestNG
Using TestNG and Selenium together is an excellent way to exercise a web application. TestNG’s data-driven approach to testing makes it easy to get search terms, evil cookies, and credentials from file, making it possible for non-developers to add new test cases. To get the most out of it, however, the suite needs to collect screenshots of test failures. Furthermore, the error message, test parameters, browser type, and other relevant factors should be stored with the image to provide context.
Previously, I generated screenshots from within a TestNG test listener. TestNG makes all the information about a test available in the ITestResult argument to the onTestFailure method. It was a simple job to get the test parameters, URL, and user-agent string and write them into the image.
This arrangement didn’t always work as expected. Sometimes the web driver had moved on before the TestNG listener had a chance to execute. In this case the screenshot would be out of step with the error condition it was meant to be reporting, making it a source of confusion.
At bottom, the solution to the problem is to generate the screenshot in the same thread as the web driver. One could proxy calls to Assert, or figure out some callback magic in Selenium itself, but these approaches would be messy, and add complexity to the test classes. Keeping the test stack as simple as possible is a requirement because testers, for all their virtues, are not necessarily strong Java coders. Keeping the barrier to entry low is necessary if you want to make sure your suite stays up-to-date and finds a permanent home in the toolchain.
I found a different approach to generating screenshots using AspectJ to wrap the calls to Assert, taking a screenshot immediately before it executes, but only writing it out if the assertion fails. The error message, the name of the test class and method can be gleaned from the pointcuts and written into the final image.
A few assumptions:
- The tests are written using the Page Object pattern.
- All page objects extend a BasePage class.
- There is a single instance of the web driver, which is accessible through static methods in the BasePage.
- All tests extend a BaseTest class.
- All test methods are annotated with @Test.
The BasePage class sets up and tears down the web driver, and handles some of the housekeeping. The BaseTest class doesn’t do much and could almost be replaced with a marker interface. Item three is an unattractive, but I couldn’t be bothered find a better way to do get at the Selenium driver from within the aspect.
The class uses two two pointcuts to intercept the calls to Assert’s static methods. The first does nothing itself, but serves as a hook for the annotation marked with @Around.
@Pointcut("call(static * org.testng.Assert.*(..))")
void anyStaticOperation(){}
@Around( "anyStaticOperation()")
public Object around( ProceedingJoinPoint jp ) throws Throwable {
// Make screenshot
// Calculate the invocation count for the current test method
// Proceed with Assert method
// Write screenshot if exception thrown
}
More information on this technique can be found here.
The parameters TestNG sends to the @Test method need to be capured so we can make sense of the output. An additional intercepting method does this work. The current test class and method are derived from the JoinPoint and used as a key for storing the parameters.
@Before("execution(@org.testng.annotations.Test * *(..))")
public void beforeTestMethod( JoinPoint jp ) {
// Get the test method name
// Save test parameters
}
I’m not sure how any of this behaves in a multi-threaded suites. For mental health reasons, I’ve only ever run Selenium in single-threaded tests. Maintaining invocation counts in a map keyed by method name is an admission I don’t fully understand the guts of TestNG and decided to err on the side of cowardice safety.
From the tester’s point of view it “just works” and requires no special configuration or learning to work with anything beyond the TestNG and Web Driver APIs. He or she can stay focused on breaking the application instead of wading through Selenium’ guts.
Dig it:
@Aspect
public class AssertionInterceptor {
Map<String,Integer> invocationCountByMethod;
Map<String,List<Object[]>> invocationParamsByMethod;
public AssertionInterceptor() {
this.invocationCountByMethod = new HashMap<String,Integer>();
this.invocationParamsByMethod = new HashMap<String,List<Object[]>>();
}
@Before("execution(@org.testng.annotations.Test * *(..))")
public void beforeTestMethod( JoinPoint jp ) {
String callingTestName = getMethodKey( jp );
List<Object[]> params = null;
if( this.invocationParamsByMethod.containsKey( callingTestName )) {
params = this.invocationParamsByMethod.get( callingTestName );
} else {
params = new ArrayList<Object[]>();
this.invocationParamsByMethod.put( callingTestName, params );
}
params.add( jp.getArgs() );
}
private String getMethodKey( JoinPoint jp ) {
StringBuilder name = new StringBuilder();
name.append( jp.getThis().getClass().getName() );
name.append( "." );
name.append( getCallingMethod( jp ));
return name.toString();
}
private int updateInvocationCount( JoinPoint jp ) {
String name = getMethodKey( jp );
int count = 0;
if( !this.invocationCountByMethod.containsKey( name )) {
count = 1;
} else {
count = this.invocationCountByMethod.get( name ).intValue() + 1;
}
this.invocationCountByMethod.put( name, count );
return count;
}
@Pointcut("call(static * org.testng.Assert.*(..))")
void anyStaticOperation(){}
@Around( "anyStaticOperation()")
public Object around( ProceedingJoinPoint jp ) throws Throwable {
Object callingInstance = jp.getThis();
if( callingInstance == null || !( callingInstance instanceof BaseTest )) {
return jp.proceed();
}
int invokeCount = updateInvocationCount( jp );
byte[] imageBytes = null;
try {
// Get the screen shot, but do not write to disk
// unless the call to the assertion fails.
imageBytes = BasePage.getBaseScreenshotBytes();
} catch(Exception e ) {
System.err.println( "Error making screenshot:" + e.getMessage() );
e.printStackTrace();
}
Object methodOut = null;
try {
methodOut = jp.proceed();
} catch( Throwable e ) {
String currentMethodName = getMethodKey( jp );
ImageAnnotator screenShotAnnotator = new ImageAnnotator();
StringBuilder filename = new StringBuilder();
filename.append( currentMethodName );
filename.append( "-" );
filename.append( invokeCount );
filename.append( ".png" );
List<String> messages = getMessages( currentMethodName, e.getMessage(), invokeCount );
screenShotAnnotator.annotateImage( filename, imageBytes, messages );
throw e;
}
return methodOut;
}
protected String getCallingMethod( JoinPoint jp ) {
String callingMethodName = null;
try {
Thread current = Thread.currentThread();
StackTraceElement[] stack = current.getStackTrace();
String callingClassName = jp.getSourceLocation().getWithinType().getName();
for ( StackTraceElement stackElement : stack ) {
if( callingClassName.equals( stackElement.getClassName() )) {
callingMethodName = stackElement.getMethodName();
break;
}
}
} catch( Exception e ) {
e.printStackTrace();
}
if( callingMethodName == null ) {
callingMethodName = "[UNKNOWN]";
}
return callingMethodName;
}
protected List<String> getMessages( String methodName, String assertionMsg, int invocationCount ) {
SimpleDateFormat format = new SimpleDateFormat( "yyyy/MM/dd HH:mm:ss.S");
List<Object[]> allParams = this.invocationParamsByMethod.get( methodName );
StringBuilder buff = new StringBuilder();
if( allParams != null && allParams.size() >= invocationCount ) {
Object[] params = allParams.get( invocationCount - 1 );
if( params != null ) {
int count = params.length;
for( int i = 0; i < count; i++ ) {
Object p = params[ i ];
if( p == null ) {
buff.append( "[NULL]" );
} else {
buff.append( p.toString() );
}
if( i < ( count - 1 )) {
buff.append( ", " );
}
}
} else {
buff.append( "[NO PARAMS]" );
}
} else {
buff.append( "[NO PARAMS]" );
}
List<String> messages = new ArrayList<String>();
messages.add( "Test: " + methodName );
messages.add( "Parameters: " + buff );
messages.add( "Invocation: " + invocationCount );
messages.add( "Assertion: "+ assertionMsg );
messages.add( "Browser: "+ BasePage.getUserAgent() );
messages.add( "URL: " + BasePage.getCurrentUrl() );
messages.add( "Date: " + format.format( new Date() ));
return messages;
}
}