Dear Testdroiders,

In case you are in London this week for any of those mobile events, please do check the Appium Meetup where we will be presenting as well. There are two great topics by great presenters discussed today – one by Shashi Jagtap about Appium with Docker and another by Robert Seege about test automation utilizing Appium on real Android and iOS devices.

In case you’re interested to join this event please sign up here.

Our presenter Robert Seege will be discussing and showcasing some of the latest and coolest stuff around Appium, how to efficiently use Appium with real devices, and how to make the best out of it. Furthermore, there are great giveaways in this meetup, so do not miss it out!

If you’d like to meet our team personally, please get in touch with me and we’ll be happy to meet you during this week.

We’re looking forward to meet you in London this week!

We’re continuing our 37 Things You Should Know About Appium with a tip that can be extremely useful for ‘debugging’ your test script execution. When something fails on a test run, quite often you are left with nothing – no trace, no info on why something crashed. Sure, logs can be inspected later on and root cause of failure found, but it is much easier – and instant – to get understanding of failure when you get screenshots aligned with a test run. And those taken when things are going south.

Test Case Creation for Mobile Web Testing and Mobile Game Testing

Regardless of which test automation framework you end up using, test case creation on mobile is often very iterative and takes a lot of trial & error to get to the point where things work out up to scale. Every now and then we see situations where tests are confirmed to work locally on a specific device, but once the same test is executed on a variety of device models, problems arise.

Sometimes the problems lie within the device or app behavior changing on different screen sizes, different OS versions or even with specific manufacturers. You can’t always predict all of these behavioral changes between different devices, which is why it is very important to make your logging verbose enough and easy to read. Most of the time one can pinpoint the problem as long as they know where it happens, but sometimes the easiest way to understand the problem is to take a screenshot of the situation.

Selecting the right framework and creating tests that fully cover your app, game or mobile web testing needs are essential to get things moving forward. It’s a well-known fact that automated testing methods can be used for both validating requirements and reducing the costs of testing through automated test case generation – when done right and preferably in the earliest phase of the development.

Let’s move to concrete tip how to gain understanding of failures – and recording those to visuals…

Testing Try Catch Statement

Our samples at Github include the ways to take screenshots. What we want to do is use these screenshot methods in a situation where the test fails. A simple way to achieve this is to enclose your test case into try catch statement, where the first thing you do is take the screenshot, handle any logging you want to display and finally re-raise the error. Re-raising should be used, since the error we got was in this case unexpected and we want to know why.

Here’s how to quickly try this with Python:

def testSample(self):
  try:
    self.driver.hide_keyboard()
    self.driver.hide_keyboard()
  except:
    self.driver.save_screenshot(self.screenshotDir + "/ERROR.png")
    raise

If you try to execute driver.hide_keyboard() when keyboard isn’t visible, you will get a WebDriverException from the Appium server. I placed the command twice, since some devices launch our sample app with keyboard open.

When any command inside the try statement fails, the code inside except statement will get executed. Placing the ‘raise’ command at end is used here to stop the case execution and ensure that exception traceback at end. If you have multiple test cases, the next test case would start running.

Tips on Function Calls and Views

It’s a good practice to include some kind of cleaning process between multiple test cases. The usual way to do that with Appium is to either use the resetApp() function, which effectively re-installs the app being tested. In some cases simply killing the app process on device through closeApp() and launchApp() functions will work fine as well.

Different syntax versions for these commands can be found here.

Testing Calls on Testdroid Cloud and Mastering This Appium Tip

You can also end a test case with driver.quit and then create a new webdriver for the next one, but keep in mind that this will cause your test runs in Testdroid Cloud to separate each test case into its own test session. We recommend bundling up your cases to use the same webdriver instance throughout your test suite, as your tests will then be bundled neatly inside a single session also in the Cloud. This way it will be easier to look at your test results as well!

Stay tuned – more insightful and useful tips for Appium coming out soon in our blog series.

Happy Testing!

This is the 9th blog in our blog series 37 Things You Should Know About Appium.

One of those annoyances for mobile app testing using whatever test automation framework, real devices and/or real browsers, and different versions of Android and/or iOS platforms have been those automatic dialogs by operating system platform. When your test script is going through your app and doing its work to hammer those specific test details, a notification from OS might suddenly pop up. What to do and how to get rid of those and proceed the execution? We’ll cover that in this blog.

System Notification Problem with Test Automation and Testing

Notifications have been a discussed topic both among Android and iOS developers. Basically this feature that informs something about OS level – or just a push notification from certain app – can make test automation scripts work flaky sometimes. Naturally, there are workarounds and ways to get rid of that notification, but however we do not recommend using any blockers (or blocker apps) as those may have an impact on OS, execution and even performance of your test session. The easiest way to ‘get rid of’ these dialogs, notifications or popups is to generate an auto-accept click either by a run-time click or on an image recognized event. We’ll discuss about approaches here.

Desired Capability to AutoAccept Notification, Dialog and Popup

With Appium, and especially on iOS, automating those OS level popups is problematic. On iOS the permission alerts are system elements rather than something inside the automated app itself, which means you cannot create test steps to handle them. Those popups, notifications and dialogs may occur at random and must be handled instantly after test script sees those.

Appium has a simple solution to this. To solve this problem, you can use a desired capability specifically designed to handle these alerts.

You can either always accept or always dismiss the alerts with these desired capabilities:

autoAcceptAlerts = true
...
capabilities.SetCapability("autoAcceptAlerts", true);

or

autoDismissAlerts = true
...
capabilities.SetCapability("autoDismissAlerts", true);

Furthermore, some of the older versions of Appium haven’t worked with this solution, so you might want to try a small workaround with this:

driver.SwitchTo().Alert().Accept();

Does Your Test Script Need Tweaks to Auto Accept/Dismiss?

To get these auto-accept/dismiss desired capabilities working with your test script and applications, you should definitely experiment which one works better (and faster) so that application or test script execution doesn’t slow down because of these auto-accept/dismiss capabilities.

On Android, pop-ups can be automated, since automation is not restricted only to one app at a time. The problem, though, might be how to identify the pop-up on screen. If a pop-up appears at the launch of your tested app, Appium inspector for example might not be able to bring out the element tree with the information. To solve this, you should use driver.getPageSource(). This command outputs the complete DOM, or Document Object Model, tree, which you can use to see all existing elements on the device. With this it should be simple to find any pop-up button elements and type them into your automated script as a test step.

Using Image Recognition to Get Rid of System Notification Popups

It’s quite common that these system notification dialogs pop up both on Android and iOS – for various reasons. One of the great workaround to get click done instantly on these notification dialogs is to trigger image recognition feature for the execution. When certain visual element or even part of visual element is shown on the screen, the script will trigger a click on “OK”. For example, with the standard Android notification it can be recognized from this visual notification:

With iOS notifications there are typically two buttons to select activity for approving and getting rid of that notification. The image recognition example can work the same way and do the click for you (and for your actual test script). Typically on iOS there are two

If you are interested to learn how image recognition can help in these kinds of system notification popups, take a look at this blog.

We’ll be continuing next week with a new topic so stay hungry and keep testing your mobile apps!

Happy Appium Testing!

This is our 14th blog in 37 Things You Should Know About Appium blog series. We’ve covered lots of basic things, such as setting up an Appium environment, using various programming languages and frameworks (Java, Python, C#, Ruby, JavaScript, Mocha), plus tools and additional utilities that makes your Appium use as smooth and easy as possible. Today, we’ll look some of the best practices, tips and tricks of how to create Appium scripts for your Android and/or iOS apps.

Creating test scripts from scratch can be a very tedious process, especially when you have many different flows to create. If you have no access to the tested app’s source, you’ll have to figure out an efficient way to map the app’s UI. The methods we know of and are about to explain here are all related to using the Appium inspector or uiautomatorviewer for native apps.

Create a Library for App Elements

One good way to handle multiple flows is to first create a library of all the elements you need to interact with. Simply navigate through the whole app with the Appium Inspector or uiautomatorviewer and mark down all the needed id, name and xpath values you’ll be using. This way you’re not limited to using the Appium Inspector or uiautomatorviewer at the time you’re actually writing your scripts. Correctly gathered library should be easy to check elements from and copy to your scripts as needed.

For example in our BitbarSampleApp, you’ll have several UI elements as radio check buttons, input boxes and couple of buttons. When you click through any of these UI elements in your own app, you’ll get to see the details of that app and can pick those for your test script. On picture above, xpath could be easy fetched and the test script could be formulated as follows:

WebElement appUI = driver.findElement(By.xpath("//UIAApplication[1]/UIAWindow[1]/UIAButton[1]"));
TouchActions simpleClick = new TouchActions(driver).tap(appUI);
simpleClick.perform();

Using gestures with Appium is relatively easy. For example this used TouchActions objects can contain several events: press, release, moveTo, tap, wait, longPress, cancel and perform.

Use Local Appium Server for Acceptance

This should be a given, but worth noting in any case. Using a local Appium server with a local test device is always the fastest way to tweak and confirm your scripts. When you confirm that your test scripts work fine on a local setup, do check that you have desired capabilities configured properly for test run on a cloud, and it’s also recommended to add some basic logic to find readily available device for your test.

In order to launch Appium server use the appium.js script that can be found in the bin directory to start the server up. For iOS you should add the -U, or --udid flag to the server, or with the udid desired capability. Note that if your application is already installed on devices where you’ll be testing your scripts out, you can also give it a bundle ID. For example:

desiredcapabilities.bundleId = 'com.bitbar.testdroid.BitbarIOSSample';

In order to get simultaneous test runs on several devices, you can start up multiple instances of Appium server that all use different ports. With multiple simultaneous devices in use at the same time, you need to set --port flag for the actual Appium port (default is 4723) and a --bootstrap-port for communication with the different Android devices. Furthermore, Selendroid and ChromeDriver also need their dedicated ports when automating multiple devices so don’t forget to set --selendroid-port and --chromedriver-port.

Xcode itself limits the use of multiple devices at the time (there are no limitation of this sort at Testdroid Cloud) in local environment. One single host machine can host only one iOS device per time.

Make Sure the Tested App has All That You Need

This isn’t an easy thing to affect on, but sometimes the apps you need to test might be lacking something on the UI side for you to make well-designed scripts.

It might be easy for developers to slack on giving unique ID’s and descriptions, attributes, and names to elements, but it can at the same time be a real pain for the Quality Assurance and testing folks. Depending on xpath values isn’t always the best way to go either, since the index numbers may change depending on how the app behaves in different situations – or – even when the app is further developer, new UI elements are introduced and those test scripts still rely on indexes given in the earlier phase of the test script creation.

This is a common issue, but it can be nicely and soundly fixed by using Appium Inspector to go through those elements before you start hammering your tests on real devices at cloud.

If you have a chance to affect on securing the naming of the UI, you should take it. One thing to note also is that on Android SDK the resource-id is something that exists only from API level 18 and upwards. If you need to test on devices on older Android version than that, you cannot depend on resource-id. You’ll have to do with the other UI element values on older OS.

Nevertheless, with these tips and tricks you’ll be good and will have a way to create and maintain your test script faster!

Next week, we’ll be deep diving in totally new topic so stay tuned and stay hungry when it comes to Appium Testing!

This is the 15th blog in our 37 Things You Should Know About Appium blog series. There are certain limitations that each framework as well as programming language have, so this time around we’ll focus on how to break off from the limitations and how to use some conventional mechanisms during your test runs. For example ADB (Android Debugging Bridge) and how to use it in the context of Appium. Naturally, this is only available and usable with Android. More importantly, ADB is usable and actually very handy when used together with Appium and test runs on execution. This works well for testing native apps, but also with mobile games using image recognition. Furthermore, not only with Appium, but literally with any test automation framework you could use our image recognition method or dynamic scripting with real-time inspection of log data. Let’s dive deeper.

Whenever people talk about test automation with some specific test framework, you would think that you are completely tied and limiting yourself to using just the API of the chosen platform. In many situations this may be true, but here in Testdroid we do have some hacks and advanced workarounds to break off from the chains of closed framework test automation. The more open, usable and inline with open standards the implementation is, the better.

ADB commands via Server-side Appium

Testdroid offers so called Server-side Appium execution, where the Appium Client is executed on the same host as the Appium Server. The main benefit of this is that it is much more efficient way to scale to high amount of test devices. As in normal Appium’s case, you’re launching test for one device (and can only launch one device per instance) we get to use the Testdroid’s own queuing system so that the tests start executing as soon as each device on chosen device group becomes available. This is the true parallelism and provides a neat way to run tests simultaneously even on hundreds of real devices. For some reason this wasn’t well taken into consideration with Appium in the early days.

All the device sessions are executed in individual and isolated containers, which gives us the possibility to allow our customers execute shell scripts and also adb commands before, during and after test execution. This is very powerful when you want to break off from the test framework restrictions and use for example adb commands during your tests also on cloud.

Here is an example of adb commands from a Java based test script, where I force the Google Play Store app to close and then launch right after:

Runtime.getRuntime().exec("adb shell am force-stop com.android.vending");

Runtime.getRuntime().exec("adb shell am start -n com.android.vending/com.google.android.finsky.activities.MainActivity");

See adb shell documentation for all adb commands. Note that using adb commands via Client side Appium test scripts won’t work for a remote server, because the adb commands will be executed on your own workspace instead of the remote server hosting the devices.

Image Recognition

Image recognition is one way for us to detect elements from apps that don’t use native UI elements. Our image recognition technology is based on open sourced OpenCV and we do have a local sample available at Github as well as Testdroid Cloud integration related help page to get you started.

Image recognition is a premium feature, since we’re still working to improve the testing process and will be offering dedicated support if you decide to evaluate this testing method. Take a look at this video to get a good idea how the image recognition process works.

Dynamic Log Scripting

This is a concept we have had success with before and a public sample is under development. The idea is to make the tested app output all the coordinate data through logs of the tested app and have scripts access that data in realtime. The test framework script has access to the log stream and will know about the coordinate data when executing gestures on the device screen.

This concept is one option next to Image recognition for apps that don’t use native UI elements (e.g. Unity game engine apps). The downside is that it does require the tested app to have custom log output, which means extra work for the developers of the app. We are however working on a tool to make the generation of correct type of log stream available for Unity.

Happy Holidays everyone! We’ll be continuing next week with a new topic so stay hungry and keep testing! And remember that test automation can take care of your testing – despite of holidays!

It’s time to look how different elements can be found with Appium and what actually happens on the background when Appium does most of the work for you. This is the 16th tip in our 37 Things You Should Know About Appium and with the following blogs, I’ll be focusing more on Appium commands, how to use them effectively and what tricks may be involved. As always, I’ll provide some code examples for each and every (popular) programming language that are used with Appium.

Mobile App Testing – What’s Under The Hood?

Appium is for mobile app testing and enables users to execute tests on mobile devices regardless of what OS version is used on devices. In short, this is possible as the Appium framework is a wrapper that translates Selenium WebDriver commands to UI Automation (on iOS), uiautomator (Android, API level 17 or higher) or Selendroid (Android, API level 16 or lower) commands, depending on which platform tests are executed on. For example, Android version 4.2 and higher are supported via Appium’s own uiautomator libraries and provides the default automation backend used by Appium.

Android versions 2.3 through 4.2 are supported via Appium’s bundled version of Selendroid, which utilizes Instrumentation. Selendroid has a bit different set of commands than the default Appium, which you need to take into account when targeting both over and under Android 4.2. To access this automation backend, use the automationName capability with the value Selendroid.

Other than the slightly different commands available, the automation backend doesn’t matter much. Appium takes care of running your tests regardless of what OS version will be installed on the device as long as automationName is correctly set.

How to Find Elements with Appium Automation Engine?

Here’s how to find elements with Appium automation engine in use with our Android sample app. The sample app provides very basic menu-like selection of three items – “Buy 101 devices”, “Use Testdroid Cloud” and “Ask mom for help”. In addition, it provides the similar look-and-feel for both Android and iOS app so the same basic flows can be tried for both platforms. This is how it looks on Appium Inspector as an iOS app under inspection:

The same find strategies work on both Android and iOS as long as the actual attribute values are correct. Here are the examples for Android with the sample app illustrated above:

Python:

elem(s) = driver.find_element(s)_by_class_name('android.widget.EditText')

elem(s) = driver.find_element(s)_by_name('Buy 101 devices')

elem(s) = driver.find_element(s)_by_id('com.bitbar.testdroid:id/radio0')

Note that you can find a list of all the elements matching the criteria by using the find_elements_by_ instead of find_element_by_. This list can then be further accessed via index numbers, for example: elems[0].click(). I’ve added the (s) to the examples to show the alternative layout for finding multiple elements.

Ruby:

driver.find_element(s)(:class_name, 'android.widget.EditText')

driver.find_element(s)(:name, 'Buy 101 devices')

driver.find_element(s)(:id, 'com.bitbar.testdroid:id/radio0')

Ruby implementation isn’t much different from others. When testing for mobile web, you can also find elements by Tag name, Link text, Partial link text or CSS  – e.g:

# <div class="highlight-java" style="display: none; ">...</div>
driver.find_element(:tag_name, 'div')

# <input id="id1" name='input' type='text'>…</input>
driver.find_element(:name, 'input')

# <a href="http://www.testdroid.com">testdroid.com</a>
driver.find_element(:link_text, 'testdroid.com')

# <a href="http://www.testdroid.com">testdroid.com</a>
driver.find_element(:partial_link_text, 'droid.com')

Java:

Using Selenium’s By class (import org.openqa.selenium.By;) can also make things easier in an IDE. Class By is a mechanism used to locate elements in any document. In order to create your own locating mechanisms, you can subclass this class and override the protected methods as required. Basic examples with By in use:

driver.findElement(s)(By.className('android.widget.EditText'));

driver.findElement(s)(By.name('Buy 101 devices'));

driver.findElement(s)(By.id('com.bitbar.testdroid:id/radio0'));

C#:

Using Selenium’s By class (using OpenQA.Selenium;)

driver.FindElement(s)(By.ClassName('android.widget.EditText'));

driver.FindElement(s)(By.Name('Buy 101 devices'));

driver.FindElement(s)(By.Id('com.bitbar.testdroid:id/radio0'));

JavaScript:

driver.element(s)ByClassName('UIATextField')

driver.element(s)ByName('Buy 101 devices')

driver.element(s)ById('com.bitbar.testdroid:id/radio0')

Is xpath Useful for Mobile App Testing?

You can also find by xpath, which we’ll look more in detail later. The most basic way is to use the full xpath, which would in our sample app’s case look like this:

elem = driver.find_element_by_xpath('//android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.ScrollView[1]/android.widget.LinearLayout[1]/android.widget.LinearLayout[1]/android.widget.RadioGroup[1]/android.widget.RadioButton[1]')

Okay, that’s a good set of basic examples. Next week we’ll cover how to find elements with Selendroid automation backend in use – and how it can help your mobile app testing.

Dear Testdroiders,

Thank you for being with us at Testdroid Blog. We hope you all had a great holiday and New Year.

When we look back in 2015, we have had fruitful technical innovations that enable us to maintain the market leadership. We’ve also shared plenty of thoughts regarding mobile app development and testing. Today, we would like to wrap up all the blogs in the last year and let’s review the top 10 of them.

The top 10 blogs are selected based on the number of pageviews throughout the year according to our Google Analytics data (All pageviews from Testdroid team members have been ruled out).

• Top 5 Android Testing Frameworks (with Examples)
• The Basics of Mobile Web Testing on Real Devices Using Selenium
• How to Use Espresso v2.0 with Testdroid Cloud Devices
• Free Android App, Game and Web Testing on Intel Devices
• Tips & Tricks: How to Archive and Export .ipa from script
• How to Get Started with Calabash (iOS example)
• How to Get Started with Appium (sample included)
• How to Use Image Recognition for Mobile App and Game Testing
• How to Increase Efficiency and Productivity with Test Automation
• Best Practices in Mobile Game Testing

We hope you find these 10 blogs helpful to your app development and testing in 2016. And we will continue to bring you more inspiring ideas, tips & tricks, best practices to help you win more in the new year.

We’ll be continuing the topic we discussed on our last Appium blog post on finding UI elements and this is the 17th blog in our series of 37 Things You Should Know About Appium. This time we’ll be focusing on Selendroid and how UI element identification works with it. Just like Appium and uiautomator, Selendroid also comes with a useful inspector tool – Selendroid Inspector – which we will also take a look at in this post. And obviously, this applies only for Android.

Selendroid, which is based on Android instrumentation, is essential when targeting Android devices of API level 17 or lower (part of Android Jelly Bean, all of Ice Cream Sandwich, Gingerbread and older). Appium’s default automation is based on uiautomator libraries, which support API level 17 and higher. With this in mind, you will need to use both automation back ends, if you’re targeting both lower and higher than API level 17 (=Android 4.2, 4.2.1 and 4.2.2).

Despite Selendroid being used as one option for Appium, there are lots of great features that you can utilize in your test script creation, test execution and generally in your test automation process:

• Full compatibility with the JSON Wire Protocol
• App under test requires no modifications or tweaks (due to nature of instrumentation)
• Mobile web testing is also possible, even as an embedded in Android app (WebView app)
• UI elements can be found with varying locators
• The concept works the same for native and hybrid apps
• In advanced framework, gestures and other not-just-click interactions are fully supported
• It was done for both emulators and real devices from the beginning, so even large set of devices works fine with Selendroid
• Quick and handy tool – Selendroid Inspector

How to Start Standalone Selendroid?

Selendroid can be executed also as a standalone. In order to get standalone Selendroid up and running, you need to first start the Selendroid-standalone component and then client driver can be instantiated.

This can be done via a shell by running the following command:

java -jar selendroid-standalone-0.17.0-with-dependencies.jar -app bitbarSelendroidSampleApp.apk

Another option is to start the Selendroid-standalone component directly from the test code:

SelendroidConfiguration config = new SelendroidConfiguration();
config.addSupportedApp("src/main/resources/bitbarSelendroidSampleApp.apk");
selendroidServer = new SelendroidLauncher(config);
selendroidServer.launchSelendroid();

You can check that the application(s) and the devices are recognized by opening a browser and navigating to http://localhost:4444/wd/hub/status.

You should see a result similar to this:

{
  status: 0,
  value: {
  "os": {
    "name": "Android"
    },
  "build": {
    "browserName": "Selendroid",
    "version": "0.17.0"
  },
  "supportedApps": [{
    "appId": "bitbar.sample.app:0.17.0",
    "mainActivity": "bitbar.sample.app.HomeScreenActivity",
    "basePackage": "bitbar.sample.app"
  }],
  "supportedDevices": [{
    "screenSize": "1280x800",
    "targetPlatform": "ANDROID",
    "device": "Asus Memo Pad Smart 10 ME102A",
    "avdName": "latest"
  }]
}

Which Back End Is Used – uiautomator or Selendroid?

Our testdroid_android.py test sample illustrates one way to tackle the situation where both back ends are being used as needed:

First, we use the device_API_level function in device_finder.py to determine the API level of our device:

apiLevel = deviceFinder.device_API_level(testdroid_device)

With the apiLevel, we can then choose the testdroid_target accordingly:

if apiLevel > 16:
   desired_capabilities_cloud['testdroid_target'] = 'Android'
else:
   desired_capabilities_cloud['testdroid_target'] = 'Selendroid'

We also place a boolean value to true, when we find out that the capabilities include automationName cap with value Selendroid:

if 'automationName' in self.driver.capabilities:
   if self.driver.capabilities['automationName'] == 'selendroid':
      isSelendroid = True

Now we can use the isSelendroid to let the script know when to use the Selendroid specific driver command and when not to:

if isSelendroid:
   elem = self.driver.find_element_by_link_text('Buy 101 devices')
else:
   elem = self.driver.find_element_by_name('Buy 101 devices')

As you can see, the same attribute of an element may need to use a different find_element convention. Also, Selendroid does not support resource-id attributes at all, since they were introduced on Android version 4.3 (API 18). When automating a test via Selendroid, you will need to use other element types instead.

What Is and How to Use Selendroid Inspector

It’s pretty similar to Appium Inspector and uiautomatorviewer but Selendroid also has its own inspection tool called Selendroid Inspector. Basically It’s a web app which is embedded inside Selendroid test server. Its purpose is to let you inspect the current state of your app’s UI and find elements of your app the same way as with uiautomatorviewer. If you are interested to learn more about these tools for test script generation take a look at this blog post.

Here is how to get it up and running when you have started Selendroid-standalone:

1) Start a test session.

2) Open your web browsers and navigate to http://localhost:4444/inspector.

3) That’s it! You can view the hierarchy of your app, view standalone UI elements and their properties, record clicks and some other actions, display and view the HTML source of any web view, and also use its xpath helper to get the specific xpath.

We’ll be covering xpath more in details in our upcoming blogs.

Happy Testing and stay tuned for more details in best practices with Appium and how to build yet better tests with it!

As we’ve been discussing here at Appium Weekly blog, there are various different ways and tricks to locate elements in a web page. Appium – based on Selenium foundation – provides excellent methods to locate elements by using a XPath function. And in fact, there are lots of different xpath methods you can use, by name, ID, link texts, partial link text, tags, class names and some others. This is the eighteenth blog in our 37 Things You Should Know About Appium – and now, let’s take a look at those different XPath tricks you might find useful.

Using the name, ID or link information (text) for your Appium tests is just fine, but when use cases are becoming more complex and there are more logic in your websites, you might want to consider using XPath. The xpath is originally designed to allow the navigation of XML content with the purpose of locating individual UI elements, attributes and some other things in an XML content.

Going into how XPath parameters are formulated, some of the XPath queries can be made in many different ways. There isn’t necessarily “the right way” to do this, but in general the good advice is to keep it simple and as straightforward as possible. You can either make a full XPath query with specific indexes for each parent/child or make it more generic and use other search criteria, such as attribute values of the element. You can even leave the element type out of the criteria, if you can’t be sure what type of element it will be.

XPath Tutorial and Few Examples for Different Languages

Regardless of what programming language you are using, the XPath parameter is built up the same way. Here are few basic exmaples with appropriate driver->find_element calls.

Python Example with Web Browser Locators

We start with a basic example: to find element “a” of any index, with parent “li” of index 1, with parent “ul” of any index, with any type of parent with id “menu”, the example goes as follows:

driver.find_element_by_xpath('//*[@id="menu"]/ul/li[1]/a')

The following examples allows you to go in sub-divisions of product ‘ID':

driver.find_element_by_xpath('//*[@id="products"]/div[1]/div/div[1]/div[3]/a')
driver.find_element_by_xpath('//*[@id="topBox"]/div[1]/div/a[2]')

Here is the example from http://testdroid.com/. The following HTML code can be found using web browser’s Inspect tool:

driver.find_element_by_xpath("//button[contains(@class, 'navbar-toggle collapsed')]")
driver.find_element_by_xpath("//li[contains(@class, 'menu-why-testdroid')]")
driver.find_element_by_xpath("//a[contains(@class, 'navbar-brand')]")

The second one (menu-why-testdroid) contains a list (menu) and sub-items can be found if index definition is used. The visual inspection should look like this (Chrome Developer Tools -> Inspect):

Ruby with Native iOS Application

As discussed in the two prior blogs about identifying and locating the UI elements, the XPath can be also identified using the uiautomatorviewer or Appium inspector. The indexes can be given the same way for native apps as those are given to websites, for example: (All UIAButtons, with parent UIAWindow index 1, with parent UIAApplication index 1)

@appium_driver.find_elements(:xpath, "//UIAApplication[1]/UIAWindow[1]/UIAButton")

Java Example with Native Android App

For example, certain UI elements can be located using the UI element definition (e.g. RadioButton) and the XPath call with return any given parameter defined inside the function call:

/* Any android.widget.RadioButton with text value 'Use Testdroid Cloud' */
driver.findElement(By.xpath("//android.widget.RadioButton[@text='Use Testdroid Cloud']")); 

/* Any android.widget.EditText with resource-id value 'com.bitbar.testdroid:id/editText1' */
driver.findElement(By.xpath("//android.widget.EditText[@resource-id='com.bitbar.testdroid:id/editText1']"));

/* Button index 1, with parent LinearLayout index 2, with parent LinearLayout index 1, with parent ScrollView index 1. */
driver.findElement(By.xpath("//android.widget.ScrollView[1]/android.widget.LinearLayout[1]/android.widget.LinearLayout[2]/android.widget.Button[1]"));

C#:

The C# example isn’t really different from Java but the syntax is naturally according to C#:

// The Same example as the first on Java, but with c# syntax.
driver.FindElement(By.XPath("//android.widget.RadioButton[@text='Use Testdroid Cloud']"));

Additional Examples

For example, any iOS element type of UIAElement, that contains at least ‘Notifications tab’ in a value. The “contains” call is great, because we can use only a part of the value. e.g. contains ‘Notifi’ would probably also work:

//UIAElement[contains(@value, 'Notifications tab')]

Also, you could for example search for any UIAStaticText element that does not have a name value that contains at least ‘_subtitle’, as illustrated in example below:

//UIAStaticText[not(contains(@name, '_subtitle'))]

And finally, any android.widget.ImageButton that has content-description (content-desc) of ‘Back’. This one must match completely, since we don’t use contains.

//android.widget.ImageButton[@content-desc='Back']

Other Find Element Functions – Usable for Mobile Web Testing

Appium (derived from Selenium) provides the following methods to locate elements in a page – you can find elements by:

• ID
• name
• link text
• partial text in link
• tag name
• name of class
• CSS selector

And to find multiple elements (these methods will return a list) you just add find_elements instead of find_element.

One of the greatest thing XPath is that it provides already options with a function call and you don’t have to build everything for a parameter from scratch. Basically, the find_elements_by_ function calls are dedicated and less is required to define what elements you are looking from your web page.

Remember! Keep those parameters relatively short and understandable – and use many variants of find_elements_by functions to define the location from your page. This makes the test execution faster, easier to maintain and more simple for everyone to interpret.

Next week, we’ll continue with the XPath theme and look more specific tips and tricks – and how to use them efficiently.

Happy XPathing!

Dear Testdroiders,

We’ve highlighted Appium here at Testdroid blog every week and primarily our focus has been in mobile app and game testing. As you know, Appium is based on the WebDriver JSON Wire Protocol and therefore works extremely well also for mobile web testing. Being a subset of the Selenium, it also provides infrastructure, platform and language-agnostic interface that is compatible with all major web browsers – also on Android and iOS.

In this blog I’ll walk you through a basic example of how to use Appium / Selenium for mobile cross browser testing using real Android and iOS devices and real web browsers on these platforms.

As said, Appium works really well as a bridge between the mobile test automation approach and web testing done with Selenium WebDriver. With Appium, the power of Selenium in web testing can be now taken on mobile devices and more importantly – mobile browsers and do mobile cross browser testing across different browsers and device configurations. One of the greatest things with this kind of combination is that it fits perfectly into agile process and makes full use of test automation – but now also for mobile web. You can use Appium to write Selenium test scripts for mobile browsers and then easily scale the number of devices and browsers you use for testing up to hundreds.

Mobile Cross Browser Testing with Any Language for All Platforms

Another awesome aspect of Appium / Selenium is that it works with any programming language as well as both these major OS platforms (Android and iOS). These means you have more freedom on language you use and you can build the same script for both of these OS platforms. The same test script, API and the workflow backs up testing on native apps, mobile games, hybrid apps and the mobile web. Furthermore, your application either as APK or IPA doesn’t need any modifications or tweaks to make it testable with Appium.

As Appium is an open source project that can be downloaded and use freely to run tests on your localhost with emulators and real devices, we always recommend using real devices and real browsers for testing efforts done on any sort of mobile product targeted for mass audience. There are several things you should take into consideration whether using emulators or real devices, but the fact is that all end-users using your stuff will use those real devices. Getting something ‘verified’ on emulators unfortunately doesn’t mean much. In fact, it maybe even difficult – if not impossible – to get certain mobile web browsers working on emulators.

There are different test automation approaches available for Appium: client-side and server-side execution. The following picture illustrates the difference between these two:

The difference between running your test script from localhost and running it on Testdroid Cloud is really small, but benefits are different. In both cases your test script would run the app or website on those mobile devices (and web browser) we host at Testdroid Cloud. In server-side execution you don’t need to configure desired capabilities but things work based on your preferences. For the cross browser testing and having your website running on as many devices and browsers as possible before you launch or made something available for users is always highly recommended. And good news is that both of these approaches are usable for your cross browser testing.

Let’s look at the example how a simple test can be created using Python and have it executed on a real device and real browser on cloud.

Python Example – Run Mobile Web Tests on Real Web Browser

I’ll be using Python here and if you need any instructions on how to set up Python, Selenium and any other required component, take a look at this. It provides a step-by-step instructions on how to get everything properly installed.

First in your test scrips import the following modules:

import os
import time
import unittest
from time import sleep
from selenium import webdriver
from device_finder import DeviceFinder

You can find device_finder.py at Github. Simply download it and locate this file on the same folder where you’ll be running your Python test script.

To make test script output real-time information about the progress or any information that you might need while test is running, it’s a good idea to write few helper functions – for example, to output ‘debugging’ information to your console:

def log(msg):
    print (time.strftime("%H:%M:%S") + ": " + msg)

If you are running the test on server-side this data will be available on Appium log that you find under Device Run view at Testdroid Cloud:

When tests are executed simultaneously on variety of different devices (regardless whether those are Android or iOS) you should always strive to take screenshots as they will illustrate how the app looks on device and if there are any visual aspects to be fixed. Simply, create a function for taking and saving screenshots with the following function:

def screenshot(self, name):
    self.screenShotCount = 1
    screenshotName = str(self.screenShotCount) + "_" + name + ".png" 
    log ("Taking screenshot: " + screenshotName)
    sleep(1) # wait for animations to complete before taking screenshot
    self.driver.save_screenshot(self.screenshotDir + "/" + screenshotName)
    self.screenShotCount += 1

After your test run is done the taken screenshots will be available on Test Run view:

Or if you are using client-side execution and you want to fetch screenshots to your local environment, you defined this with TESTDROID_SCREENSHOTS environment variable:

self.screenshotDir = os.environ.get('TESTDROID_SCREENSHOTS') or "/Users/vvh/pywebtest/screenshots"

In order to establish connection between your localhost and Testdroid Cloud, you’ll need to configure these few parameters:

testdroid_url = os.environ.get('TESTDROID_URL') or "https://cloud.testdroid.com/"
testdroid_apiKey = os.environ.get('TESTDROID_APIKEY') or "123456789abcdefghijklmnopqrstuvx"
appium_url = os.environ.get('TESTDROID_APPIUM_URL') or 'http://appium.testdroid.com/wd/hub'

Note that we’re now using API Key to authorize access between the localhost and Testdroid Cloud. The API Key can be fetched from Testdroid Cloud -> your profile (top right corner) -> My account and API key. Simply copy that and paste in to your testdroid_apiKey variable:

The next thing to do is to secure any available device for your test session. We’ve built a basic device finder function to help in that and speed up the device acquisition for your test runs. Basically it will go through available devices and pick you the one. Furthermore, you can define the device with testdroid_device variable. You can find more information – as well as some tips and tricks using this function in this blog.

deviceFinder = None
testdroid_device = os.environ.get('TESTDROID_DEVICE') or "Samsung Galaxy Tab 3 10.1 GT-P5210 4.4.2"
     
deviceFinder = DeviceFinder(url=testdroid_url)
   if testdroid_device == "":
      # Loop will not exit until free device is found
      while testdroid_device == "":
            testdroid_device = deviceFinder.available_free_android_device()

If you want to find available iOS device for your test session, you would use available_free_ios_device instead of deviceFinder.available_free_android_device() in that last line.

Setting Up Desired Capabilities

If you are using client-side execution you basically need to have all these desired capabilities properly configured in your test script. With the server-side execution, there is no need to configure any of those as app and test is already uploaded on our server and it will fetch all this information based on your projects, device selections and test run preferences.

Here is the example for desired capabilities on client-side execution:

desired_capabilities_cloud = {}
desired_capabilities_cloud['testdroid_apiKey'] = testdroid_apiKey
desired_capabilities_cloud['testdroid_target'] = 'chrome'
desired_capabilities_cloud['testdroid_project'] = 'Appium Chrome Webtest Demo'
desired_capabilities_cloud['testdroid_testrun'] = 'Test Run 1'
desired_capabilities_cloud['testdroid_device'] = testdroid_device
desired_capabilities_cloud['platformName'] = 'Android'
desired_capabilities_cloud['deviceName'] = 'AndroidDevice'
desired_capabilities_cloud['browserName'] = 'chrome'
desired_caps = desired_capabilities_cloud;

These desired capabilities can be configured based on your preferences, how and on what device you want tests to be executed.

Using Real Devices and Browsers with Selenium

To launch several simultaneous device runs (devices running your tests at the same time) you can build an instigator script with array of devices to specify all devices you want to start your run on:

AndroidDeviceList = [“Acer Iconia Tab 8 A1-840FHD US”, 
                        “Asus Google Nexus 7 ME370T 4.3 JWR66Y”,
                        “Dell Venue 8 3840 EU”,
                        “HTC One M7 5.0.2”,
                        “HTC One M9”,
                        “Huawei Honor 6 H60-L01”,
                        “LG Google Nexus 5 6.0”,
                        “Lenovo Lemon K3 K30-T”,
                        “Motorola DROID RAZR HD XT926”,
                        “Motorola Google Nexus 6 XT1100 5.1 EU”,
                        “NVIDIA Shield Tablet”,
                        “Samsung Galaxy Note 5 SM N920R4”,
                        “Samsung Galaxy S5 SM-G900T (T-Mobile) -US”,
                        “Samsung Galaxy Tab 4 10.1 SM-T530NU”,
                        “Sony Xperia Tablet Z”,
                        “Sony Xperia Z SO-02E (Sony Yuga)”,
                        “Xiaomi MI 3 (Tegra 4 powered)”,
                        “Xiaomi Redmi 2 LTE”,
                        “Yota Phone 2 YD201”,
                        “ZTE V5 Red Bull U9180”
]
...
for each device in AndroidDeviceList:
    testdroid_device =  DeviceFinder.getDevice(device)
       if testdroid_device is not None:
          break

With the server-side execution this would be very convenient as you can manually select on which device group you want to use for test session:

Once the device selection is done, you can also specify which browser will be used for the test session (as set in ['browserName'] = 'chrome' variable). With Android we support pretty much any browser available (however, typically the only meaningful ones are the Android Browser or Chrome) and on iOS Safari is pretty much the one you want to use for your test session.

Creating and Running A Mobile Browser Test

After all these steps have been done, let’s look how to use WebDriver to get test running. First, you need to initiate the WebDriver. Note that if you are using client-side execution this really may take some time as the device will be prepared for your session. If you didn’t use Finder Device, the device may not be either available and WebDriverException is thrown saying that device could not be acquired in certain time.

log ("WebDriver request initiated. Waiting for response, this typically takes 2-3 mins")
self.driver = webdriver.Remote(appium_url, desired_caps)

To specify which web page will be used for testing will be done next, with the action to take a screenshot of it, as follows:

log ("Loading page http://testdroid.com")
self.driver.get("http://testdroid.com")
log ("Taking screenshot of home page: '1_home.png'")
self.driver.save_screenshot(self.screenshotDir + "/1_home.png")

Next you can call some interaction (e.g. click) on certain UI elements or links on that page – and take the screenshot of the view:

log ("Finding 'Solutions'")
elem = self.driver.find_element_by_xpath('//*[@id="menu"]/ul/li[1]/a')
log ("Clicking 'Solutions'")
elem.click()
log ("Taking screenshot of 'Products' page: '2_products.png'")
self.driver.save_screenshot(self.screenshotDir + "/2_products.png")

Furthermore, you can use XPath locators to find certain elements in that web page.

Quit the WebDriver Session

Finally, you can call tearDown to quit WebDriver:

def tearDown(self):
   log ("Quitting")
   self.driver.quit()

If you are interested to fetch this example we just walked through you can find the template at Github.

Happy Web Testing folks!

In the previous blog we had a glance on to use different types of XPath locators in Appium and how those function calls work when building your test scripts. Today, we’re continuing with the same theme but this time we’ll focus more on Selenium WebElements and Appium MobileElements and how to use those to explore an ideal way to interact with UI elements. This is the 19th blog in our massive 37 Things You Should Know About Appium blog series. Also, I’d like to thank many of you for providing such a great feedback on our prior blogs.

There is distinctive difference between how Appium interacts with UI elements on an app and how Selenium interacts with a web elements. Identifying UI elements is the key to write test scripts that efficiently test through the UI logic. We’ve discussed about identifying those UI elements with test scripting tools like Appium Inspector and uiautomatorviewer, and how to find those UI elements in the Appium back-end context.

The major difference for testers who have worked with Selenium to identify UI elements is to learn how the same can be done with Appium and in mobile app context.

In short, there are many different ways to interact with the elements of your apps. Let’s take a look at some of the most useful ones.

Java for MobileElements

For example in Java, it’s quite easy to check the API documentation for the MobileElements. Here’s a shortened list of some of the most useful methods for your tests:

Methods inherited from the Selenium WebElements are clear, click, getSize,getText, getAttribute, getLocation, getTagName, isDisplayed, isEnabled, isSelected, sendKeys. When you call any of these methods the auto-refresh check is done to determine if UI element is still valid (visible). The typical issue is that if UI element is not visible anymore (for any reason) it will throw an error and all the following calls will fail. The same thing applies to Appium test script: if any of your call fails, it will fail the following ones as well. Expected failures can be countered with try catch statements.

For comparison, the Appium MobileElement methods include the calls for more advanced gestures and direct UI interaction: getCenter, pinch, swipe, tap, zoom. Most of these methods are convenience methods for writing cleaner scripts. One of the great things about these methods is that they don’t actually have to click/interact with a specific element, but instead can be used towards the whole device screen. For example, it usually makes more sense to target a swipe gesture relative to the device screen dimensions. This way the gesture gets done without the use of UI elements and also never outside of the screen. In case like that Appium would throw outOfBounds error.

As you can see, you can do the basic click interaction as well as sendKeys via the Selenium WebElement class, and also get different useful information from an element with the get and is methods. Appium’s MobileElement class gives a bit more options for gestures in the form of convenience methods such as pinch, swipe and tap.

To take this even further, you could take a look at the TouchAction and MultiTouchAction classes, which give even more advanced choices for gestures. TouchActions are explained more in detail for example at the Appium’s Github docs page.

Okay, to make sure your WebDriver always uses the MobileElement instead of the default Selenium WebElements, you’ll need to cast the MobileElement within AppiumDriver like so:

For Android:

AppiumDriver<MobileElement> driver = new AndroidDriver<MobileElement>(capabilities, url);

For iOS:

AppiumDriver<MobileElement> driver = new IOSDriver<MobileElement>(capabilities, url);

If you don’t include the cast, the WebDriver will always consider the elements it finds as Selenium WebElements, which will then be missing the methods of MobileElement. Of course, a WebElement can be cast to a MobileElement also while finding it:

MobileElement element = (MobileElement)driver.findElement(By.xpath("//xpath"));
element.swipe(SwipeElementDirection.DOWN, 1000);

Rest of the Bindings

When using Appium as the server, you can choose to interact with the app either via the Appium’s own language bindings or directly from the Selenium’s own libraries. The difference is in which API library you’re importing for your test script’s disposal. In general it’s better idea to take the Appium’s library to use, since you do get more possibilities to your disposal. You could also purely stay with the Selenium API to make things more simple and less cluttered.

To see what Appium’s Python bindings have to offer on top of the Selenium API, you could take a look at the Python client’s Github page or directly the webdriver.py class.

For Appium’s Ruby library the docs are separated to generic, Android and iOS specific docs.

Node.js client has different frameworks to work with (such as Mocha), so their page first starts off with examples of using those. Then comes the listing of the API commands.

The page for .NET client in C# has a simple nudge towards how to setup your environment. You could take a look at the AppiumDriverCommand class for a listing of all Appium specific commands, but in the end, this is the same info you’d directly get from your IDE’s code completion once set up.

Just few years ago, the process of mobile test automation was too immature and the entire mobile ecosystem wasn’t ready to adopt large scale test automation setups for mobile app testing. Furthermore, test automation wasn’t necessarily seen as possible to be done across devices, simultaneously, and in parallel mode. Nowadays, things are whole lot different. Test automation, with frameworks like Appium, real devices, variety of different hardware and using network connections and real back-ends intensify testing of Android and iOS apps.

This is the 21st blog in our 37 Things Should Know About Appium – and we’ll go through a basic example of how to get Appium test started on available Android and/or iOS devices, just in seconds using device finder functionality.

One of the hurdle with setting up a large set of devices has been around physical connection of devices to server machines, and setting up everything with ADB or Xcode access. However, acquiring a device for Appium test is trivial and rapid nowadays using Testdroid device finder. In addition, there are no limitations on which programming language you need to use to define preferred steps for your tests.

This blog walks you through an example of how to find an available device for your testing needs from Testdroid Cloud service.

API Key Authentication for Cross Platform Appium Tests

First thing to do is to use a path for API call to get list of all devices. Based on Testdroid API document, you should use https://cloud.testdroid.com/api/v2/devices?limit= for initialization. The better and yet more easier way to handle all connections between your local instance and Testdroid Cloud is the API Key authentication instead of OAuth authentication.

class CrossPlatformDeviceFinder:

    url = None

    def __init__(self, url="https://cloud.testdroid.com", download_buffer_size=65536):
        self.cloud_url = url
        self.download_buffer_size = download_buffer_size
       
    def _build_headers(self, headers=None):
        hdrs = {}
        hdrs["Accept"] = "application/json"
        if headers != None:
            hdrs.update(headers)
        return hdrs

    def get_devices(self, limit=0):
        return self.get("devices?limit=%s" % (limit))
        
    def get(self, path=None, get_headers=None):
        url = "%s/api/v2/%s" % (self.cloud_url, path)
        headers=self._build_headers(get_headers)
        res =  requests.get(url, headers=self._build_headers(get_headers))
        if res.ok:
            return res.json()
        else:
            print "Could not retrieve any devices."
            sys.exit -1

The API key is a unique identifier for every user. It’s generated when signing up for the service and it can be changed any time. In short, the Testdroid API key makes the entire the login procedure and communication faster as you are you are not required to use access token all the time. The API key also improves the security as you are not required to play with your credentials and expose those with your test scripts.

The API key is can fetched from your Testdroid Cloud profile (Profile -> My account -> API key):

In order to reset API key (recommended every now and then), just click ‘Reset API key‘.

Select Devices for Tests in Desired Capabilities

Appium desired capabilities can be used to identify what devices will be used for a test run. In this example, it is just a simple table including all the device names. This kind of approach can be also used as a cross platform device acquisition (e.g. listing both Android and iOS devices).

Here is a basic example of how to list devices in Python for these described scripts:

   AndroidDevices = [“Acer Iconia Tab 8”, 
                     “HTC One M9”,
                     “Motorola Google Nexus 6”,
                     “Samsung Galaxy Note 5”,
                     “Samsung Galaxy Tab 4 10.1”,
                     “Sony Xperia Tablet Z”,
   ]
   ...
   for each device in AndroidDevices:
       testdroidDevice =  CrossPlatformDeviceFinder.getDevice(device)
       if testdroidDevice is not None:
          break
   ...

Acquiring Android Devices for Appium Test Run

In addition to prior pseudo code example, you can use more specific search parameters in your device acquisition. For example, you can combine the script below with your instigator script to acquire and access just specific device you want for your test run.

    def available_android_device(self, limit=0):
        print "Searching Available Android Devices..."

        for device in self.get_devices(limit)['data']:
            if device['locked'] == False and device['osType'] == "ANDROID" 
               and device['softwareVersion']['apiLevel'] > 16:
                print "Found device '%s'" % device['displayName']
                print ""
                return str(device['displayName'])

        print "No available Android devices were found."
        print ""
        return ""

The API level matters in Android and even Appium uses different foundations for older versions of Android. For example, if your app must run on devices with API level below 17, Appium is using Selendroid as a basis and on newer API levels it will use uiautomator. Furthermore, you can use something as follows for checking the API level of any Android device:

    def device_API_level(self, deviceName):
        print "Searching for API level of device '%s'" % deviceName

        try:
            device = self.get(path="devices", get_headers={'search':deviceName})
            apiLevel = device['data'][0]['softwareVersion']['apiLevel']
            print "Found API level: %s" % apiLevel
            return apiLevel
        except Exception, e:
            print "Error: %s" % e
            return

Acquiring iOS Devices for Appium Test Run

In iOS device acquisition, the OS version isn’t important. Instead, you should pre-define those devices in some table before calling the available_ios_device(). Here is one example how to use pre-defined devices for this search:

 
   iOSDevices = [“Apple iPad 3 A1416 8.2”,
                 "Apple iPad Air 2 A1566 8.4",
                 "Apple iPad Mini 3 A1599 9.0.2"
   ]
   ...
   for each device in iOSDevices:
       testdroidDevice =  CrossPlatformDeviceFinder.getDevice(device)
       if testdroidDevice is not None:
          break
   ...

In order to find available iOS devices you can use the following type of script to identify those and get the device up and running with your app and tests:

    def available_ios_device(self, limit=0):
        print "Searching Available iOS Device..."

        for device in self.get_devices(limit)['data']:
            if device['locked'] == False and device['osType'] == "IOS":
                print "Found device '%s'" % device['displayName']
                print ""
                return str(device['displayName'])

        print "No available devices were found."
        print ""
        return ""

Benefits of Cross Platform Device Finder

Appium provides a great framework for test automation but parallel test execution hasn’t been too easy just by defining a group of devices where your app and test should do its heavy-lifting.

With these instructions and few examples, you should be able to build tests that 1) get started instantly (as script acquires those devices that are available at that very moment), 2) enable simultaneous/parallel/concurrent test runs (with instigator script where you define devices that you want to run on) and 3) provide you results from devices that you see important for your app.

Weigh in with a comment below! Also, stay tuned as more tips and tricks for Appium are on their way here Testdroid blog.

Happy Appium Test Automation!

This is the 22nd blog in our 37 Things You Should Know About Appium blog series and this time we’ll be focusing on how to use image recognition when testing mobile apps, games and even web sites. The image recognition has been lately one of the most prominent approach for enabling test automation for mobile apps and games that have graphics content or other content that regular test automation frameworks may not be able to identify with IDs, description or object characters.

We have recently updated our Image Recognition sample to be more diverse and robust. There are now instructions on how to launch the image recognition tests in your local environment, as a Client Side Appium testrun and as a Server Side Appium testrun. The dependencies are now also using the latest version of testdroid-appium-driver, which includes the usage of Appium java-client 3.2.0.

In other words, the underlying libraries are now more up to date and should help you get your tests running more fluidly and with less problems from old known bugs!

The README may seem a bit more complicated now with instructions to all these three types of test execution included, so do take your time when looking through it. It also includes information on how to install the Image recognition libraries to your own machine, so make sure to follow it until the end before attempting to run the tests locally.

Running Appium Image Recognition Tests from Command Line

Once you have installed all the dependencies, you’re ready to setup the project for testing! For local tests, remember to set your Appium server ready as well. Copy the testdroid.properties.sample.properties file into a testdroid.properties file in the project root directory and uncomment the example lines depending on the type of test you’ll be running:

Local test:

testdroid.appiumUrl=http://localhost:4723/wd/hub
## Android
appium.appFile=../apps/builds/BitbarSampleApp.apk
## iOS
# appium.appFile=../apps/builds/BitbarIOSSample.ipa

Once the testdroid.properties are set, use a maven test command with correct test class specified in “-Dtest” flag to launch the tests:

mvn -Dtest=iOSSample test

OR

mvn -Dtest=AndroidSample test

Client Side test:

Uncomment the Client Side example lines of testdroid.properties and also set your Testdroid Cloud user credentials:

testdroid.username=<add username here>
testdroid.password=<add password here>

Configure the following parameters based on which platform and device you’ll be using for testing:

## Android
testdroid.project=Client side Android
testdroid.device=Samsung Galaxy Nexus GT-I9250 4.2.2
appium.appFile=../apps/builds/BitbarSampleApp.apk

## iOS
testdroid.project=Client side iOS
testdroid.device=Apple iPhone 5 A1429 9.2.1
appium.appFile=../apps/builds/BitbarIOSSample.ipa
appium.bundleId=com.bitbar.testdroid.BitbarIOSSample

Finally, run your tests with maven using the same logic as with local tests:

mvn -Dtest=<TestClass> test

In addition, if OpenCV is not installed using Brew, the java.library.path needs to be provided on the command line. In order to get this done, execute the following line with OpenCV java library path in place:

mvn -Djava.library.path=<java-lib-path> -Dtest=<TestClass> test

To get any additional Testdroid properties configured, check out Testdroid-Appium-Driver’s TestdroidAppiumClient.java.

To select device you want to run your test on, you can use either testdroid.device variable in the properties file or alternatively give the device name via command line parameter: (e.g. testdroid device name = “Apple iPhone 6 A1429 9.2.1″)

mvn -Dtestdroid.device="testdroid device name" -Dtest=<TestClass> test

After the test run is finalized (you need to wait until the device run is finalized) you’ll get the results, screenshots and all the other assets from the test run under ./target/reports/.

Running Appium Image Recognition Tests on Server-Side

You will need a Testdroid project of type CALABASH ANDROID or CALABASH IOS, which we have pre-configured for you to act as a Server-Side Appium project. If this has not been done already, please contact sales@bitbar.com or your Testdroid contact person to get a project configured.

Once you have a properly configured Testdroid project available, use the provided scripts to create the test zip from your project:

## For ANDROID - createAndroidZip.sh
cp run-tests_android.sh run-tests.sh
rm android-test.zip
zip -r android-test.zip pom.xml run-tests.sh src akaze/linux akaze/LICENSE lib/testdroid-appium-driver-1.2.0.jar queryimages

## For iOS - createiOSZip.sh
cp run-tests_ios.sh run-tests.sh
rm ios-test.zip
zip -r ios-test.zip pom.xml run-tests.sh src akaze/mac akaze/LICENSE lib/testdroid-appium-driver-1.2.0.jar queryimages

These lines are included as shell script files and both run-tests_android.sh and run-tests_ios.sh can be found from root of the project.

After you have executed these scripts, you should have a properly built test zip file and the application apk/ipa file ready for a test run. Then create a new testrun in your pre-configured Testdroid Cloud project, upload your APK or IPA as well as test zip file using the test run creation wizard. Choose a device group where your Appium test will be executed (you can include any number of devices for your test run and test will run simultaneously on these devices). The last page of the test run creation wizard includes the setting of max timeout for your tests, so make sure it is set high enough for your tests (default is 10 minutes).

Changing the Test Class Name

At any time if you change the name of your Android or iOS test class, you will need to update it to the run-tests_android.sh and/or run-tests_ios.sh TEST and JUNIT variables as appropriate:

# Name of the desired test suite and optionally specific test case, eg: AndroidSample#mainPageTest
TEST=${TEST:="AndroidSample#mainPageTest"}
# JUnit file wont have the #caseName ending
JUNIT="AndroidSample"

Testdroid-Appium-Driver

Testdroid appium driver is a Java project, which utilizes the Testdroid API client for java and Appium’s java-client. The major benefits of using the driver is the ready-made logic for app upload to Testdroid Cloud, taking of Screenshots as well as simplifying the editing of desired capabilities. The testdroid.properties, which is located at the root of the project, is the only file you need to edit when desired capabilities need changing.

Getting Started With Your Own Tests on Top

When you’re ready to start tweaking the actual tests, you will mostly working on the test files in src/test/java/.

Our collection of helper methods in src/main/java/TestdroidImageRecognitionAndroid.java and src/main/java/TestdroidImageRecognitionIos.java should also be a tremendous help when looking for different ways to make use of the image recognition data.

Happy Testing!

Dear Testdroiders,

Emulators and simulators have their own pros and cons when used as a part of the application development. Using emulator is common in the earliest phase of the development and the need of using the real device typically comes up when application reaches first usable version with enough logic and implementation in it. However, this is a logical path for app developers as emulators and simulators have severe limitations, not only in how they handle real hardware details but also how many concurrent instances can be launched simultaneously. Real devices do not have this problem and any number of available devices can be used for development and testing purposes. In this blog, we’ll take a look at parallelism and concurrency of Appium test automation.

Unfortunately, Android emulator has lots of issues with its parallel test run execution. If you try to run several device emulations simultaneously, you’ll soon face issues with memory running out and emulation instances taking too much of your memory and CPU from the development environment itself. It’ll get very challenging (and eventually slow) to run anything with meaningful number of different ‘devices’ in simulated environment.

With iOS simulators and Xcode environment, it’s not even possible to run local parallel test runs on the same machine as iOS simulator cannot be launched multiple times. And those are yet simulators, providing no real hardware, infrastructure, software and all dependencies that are used in the real user-device context.

Appium Parallelism and Concurrency

Test automation is meant to enable parallelism and concurrency. Running a test script on any number of devices should be always the goal when using test automation as it enables fast test run launches and instant results, and possibility to quickly iterate issues on app under development. In addition, manually starting test runs device by device doesn’t fit well to this sort of agile thinking with mobile test automation.

Because of this sort of limitations test automation using Appium should be always unlimited to any concurrent test runs, on any number of real mobile devices. That makes the development and testing agile with faster turnaround, and result in better quality and more robust apps. The major difference is how the test is launched, not in how it is executed or what number of devices are selected for testing. The picture below illustrates in its simplicity how these two things are different:

Quick glance gives an impression that there isn’t much difference on these two. First one (above) illustrates an Appium session where each mobile device is initiated separately, test session will be conducted normally on those devices, and eventually results are pulled back to the environment/computer from from where the test was launched. The below illustrates a use case where test and application has been uploaded to the server that automatically starts sessions on selected (not hardcoded) devices, and maintains results in the same location. Let’s drill deeper with these examples.

Appium desired capabilities take various aspects into consideration: automation engine, full platform definition (OS, version etc), device to be used and its parameters (name, orientation, locale etc), application (name/link, run details, timeouts), and many others related to launch, execution or device session. For instance, setting (hardcoding) something in desired capabilities wouldn’t make it easy to scale up for myriad of devices, but instead all these would need to be hardcoded in these desired capabilities. The nature of this comes from the Selenium and Webdriver where no additional devices were meant to be used for the execution and it only took scale of different browsers etc. into consideration.

Again, due to these sort of limitations and maximize the potential of Appium the test execution should be used in the context where all possible preset and hardcoded details are minimized and test script can be easily used across devices, for sake of concurrency and any number of parallel test runs.

Starting Appium Tests in Consecutive Order

Appium tests can be easily started from the localhost on real mobile devices using cloud service like Testdroid Cloud. If you start your test runs from localhost you have to configure all devices that app will be installed on and tests will run against the app in parallel mode. This is called the client-side execution where tests are launched from the localhost, application is uploaded separately on a server that then automatically takes care of installing application to dedicated device. The device is selected in desired capability and must be defined before the test is launched:

capabilities.setCapability("testdroid_device", "Samsung Galaxy S7 SM-G930F");

After you’ve started this test, you need to start another one with different device. Easiest implementation is naturally to build an array of devices in your test script and make sure tests launched against each and every device listed on that table. The instigator script and table could also use DeviceFinder to go through all possible devices and select the first free one.

The following picture illustrates the logic behind client-side execution. All details for test – setting up a webdriver, desired capabilities and test itself – is done on a local machine. It communicated with Appium Broker that handles all test steps and makes sure the dedicated device is properly in use for the test session. In between there is an Appium Server running on server infrastructure that does the actual communication, passing the test, data and results between server and the device.

As said, it’s possible to get decent level of concurrency with client-side execution, but the test script must be prepared so that each device is reserved, accessed and test will run on it separately. Also, fetching test results (data, logcats, Appium log, screenshots etc) must be done separately for each device and test run.

Starting Appium Tests in Parallel

When it comes to true parallelism with Appium you should strive to start tests at the same time. This is enabled with so called server-side execution where application and tests are manually, automatically from CI/CD or with some other method via API uploaded to the server. There is a significant difference compared to client-side execution as everything will be now resided on the server that communicates, runs and controls the Appium sessions on real physical devices. As the test can pretty much identical for both Android and iOS, however, devices cannot be mixed in the test run. UI elements, the platform and many others work differently on Android and iOS so user must select whether test run will be done on either one of these platforms.

The server-side execution brings in the true parallelism and scales up depending on test execution needs. When the cloud service is implemented correctly, there are no bottlenecks or things slowing down the parallel execution of test runs. This is why Testdroid Cloud provided unlimited number of devices for a server-side Appium test execution.

One of the benefits also to use server-side execution is that you don’t need to configure any test run specific details in desired capabilities, but only those for application, package and how you want it get executed. Parallelism in this context also means that the test package will be identical on each device and even on iOS the system will take care of provisioning packages. In few words, this is the feature that makes Appium to run simultaneously on yet hundreds of unique devices. Try Appium parallel test runs out!

Dear Testdroiders,

Behaviour-Driven Development (BDD) gathers supporters for many obvious reasons as it is readable (by everyone), understandable (across teams and functions) and accountable (metric-driven for success/failures). Cucumber is a great example of optimized tool/framework to create such tests. This allows basically anyone (developers, QA, testers etc.) to write specifications in any spoken language that then gets executed as automated tests against mobile app(s). Furthermore, this sort of BDD approach incorporates the main standards of test-driven development (TDD) and provides a shared view and process for collaboration between all stakeholders.

Let’s look how to setup Cucumber and how to combine it for some standard test automation frameworks (Appium / Selenium).

Download Our Tutorial to Master Cucumber/Calabash for Mobile App Testing.

Cucumber for Test Automation

Cucumber is a neat tool that developers can use to test their stuff. In mobile context, Cucumber is also usable and for example is used with Calabash. Basically, Cucumber can run automated acceptance tests written in a behavior-driven development (BDD) style and is available with Ruby. In addition, there are other Cucumber projects available for some other platforms/programming languages, but we’ll take a look at Ruby side of things here.

One of the best parts of Cucumber is that it also provides documentation and feature specification for mobile app testing. When the test script is written in understandable language it can also serve as a design, development and test specification – and that’s what makes Cucumber highly usable across functions.

Cucumber can be also used to enable a large-scale test automation with real mobile devices. There are myriad of different test automation frameworks that can get you there instantly, but also combined together with Cucumber. Cucumber behavior-driven development together with one of the best test automation frameworks today – Appium – is a great combination and can help to you automate testing of your apps, write nicely documented specifications and make clarity for test execution.

Prerequisites and Installations

First and foremost, check that your Ruby environment is up-to-date, by using the following terminal line:

$ ruby -v

Things may work pretty well on some older Ruby versions, but to get all the best features and the most stable experience, we recommend using the latest (e.g. 2.2.4 or newer).

RVM – Ruby Version Manager

In case you don’t have or have old version of Ruby or you want to use multiple versions of Ruby, RVM (Ruby Version Manager) is a neat command-line tool which allows quick and easy installations, management of multiple versions and Ruby environments with interpreters and any set of gems. You can simply get it install with these instructions:

$ curl -sSL https://get.rvm.io | bash -s stable
$ rvm install ruby

List of all installed versions of Ruby and set one of those as default one:

$ rvm list
$ rvm --default use 2.1.1

The upgrade of RVM can be done with the following lines:

$ rvm get head
$ rvm autolibs homebrew
$ rvm install ruby

Note that this takes care of Homebrew installation as well. Another alternative to install Homebrew is done as follows:

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

After setting up Homebrew you can (and you should) check every now and then if the environment is up to snuff:

$ brew doctor

This should give you Your system is ready to brew but Your Homebrew is outdated is also very common and then keeping Brew up-to-date can be simply done with an update:

$ brew update

Again, it’s worth of checking that it’s Ruby is properly installed by printing out the Ruby version:

$ ruby -v

Update RubyGems and Bundler:

$ gem update --system
$ gem install --no-rdoc --no-ri bundler
$ gem update
$ gem cleanup

Check that RubyGems is >= 2.1.5

$ gem --version

Platform-Specific Tools

Depending on which platform – OS X, Linux or Windows – you’re about to run your tests, you need to setup all required development tools (Xcode for iOS, Android Studio) and other stuff. For detailed instructions on how to setup Appium for any platform, with desired programming language and all dependencies, take a look at our Appium tutorial.

Cucumber Installation

Setting up Cucumber is pretty straightforward.

Create a directory for test example:

$ mkdir cucumber-test
cd cucumber-test

Then create a file under this folder to include the following lines:

source "https://www.rubygems.org"
gem "cucumber"
gem "selenium-webdriver"
gem "rspec"
gem "rspec-expectations"

Now, run bundler at the Ruby example directory to install dependencies:

$ bundle install

Now, we have installed everything that it takes to start writing tests and running those. For detailed information on how to write Cucumber tests, you can check from our tutorial. This tutorial gives instructions to write Calabash tests but overall the syntax is the same and tests, steps and features are done the same way.

Happy Testing!

Dear Testdroiders,

As you know, Appium was originally built on the idea that developers (as well as testers and QA folks) could use their favourite tools, frameworks, and programming languages when creating automated tests for mobile apps. Appium has been an awesome open source project that many people have contributed and many have extended with other additional advanced features. That’s what we’ve been doing here at Testdroid and helped thousands of mobile developers using this great framework to make mobile app testing smooth, easy and feature-rich.

It’s time to look some of the best online resources to get help and make your Appium use yet more frictionless.

We’ve covered lots of different topics in Appium framwork itself, installation, programming languages, tools, and provided tips, tricks and best practices in our Appium Tutorial – and this time we’ll list out few things that are essential when you are working with Appium. Some these things are must in your daily work and surely we’ll help to get the best out of your test automation effort as well as making things work instantly out-of-the-box.

Appium Infrastructure, Real Devices and Cloud

It is always recommended to use real devices with Appium (like with other test automation frameworks too). This way you’ll find the real issues that end-users face when running your app on their devices.

Parallel Test Runs with Appium on Real Devices (unlimited no. of devices)

Tools & Utilities to Make Appium Use Easier

Especially if you are starting out with Appium and do not have much experience of similar test automation frameworks, it’s highly recommended to get all dependencies and especially those tools/utilities installed. You can use the following tools to make sure everything is accessible in your system:

Java SDK 1.8
The openjdk-8-jdk package is recommended and can be installed with the following command line call:

$ sudo apt-get install openjdk-8-jdk

Apache Ant
Apache Ant is a Java library and command-line tool whose mission is to drive processes described in build files as targets and extension points dependent upon each other. The main known usage of Ant is the build of Java applications.

Apache Maven

Apache Maven is a dependency management and build system tool. Based on its concept of a project object model (POM), Maven can manage a project’s build, reporting and documentation from a central piece of information.

unzip apache-maven-3.3.9-bin.zip

or

tar xzvf apache-maven-3.3.9-bin.tar.gz

Ruby Version Manager
RVM (Ruby Version Manager) is a neat command-line tool which allows quick and easy installations, management of multiple versions and Ruby environments with interpreters and any set of gems. You can simply get it install with these instructions:

$ curl -sSL https://get.rvm.io | bash -s stable
$ rvm install ruby
$ ruby -v

And it should point out the version of Ruby you are using and are set up with.

Ruby Package Manager and Ruby Dependency Manager

First, check what version of RubyGems you’ve installed and update it. This will also take care of Bundler update:

$ gem update --system
$ gem install --no-rdoc --no-ri bundler
$ gem update
$ gem cleanup

Check that RubyGems is higher than 2.1.5:

$ gem --version

Execute bundler at the Ruby example directory to install dependencies:

$ bundle install

Node.js and Node.js Package Manager (NPM)
Node.js is an efficient and lightweight JavaScript runtime and with its package ecosystem one of the largest open source library. Start checking first which version is installed on your machine:

$ node -v
It's recommended to run the latest but at least v0.10.32. If you don't have one, just update it with the following command:

$ sudo npm install npm -g

Alternatively, you can also download it manually and update.

Grunt – JavaScript Task Runner
Grunt is a JavaScript task runner and its ecosystem is already pretty large. There are hundreds of different plugins that can help you to automate various activities in your daily tasks. Again, check what version (if any) you have in use, update or install the latest one:

$ npm install -g grunt-cli

Learn more here on Working with an existing Grunt project.

Platforms – Setting Up Development Tools

iOS

First of all, Appium required Mac OS X Lion (10.7+) or newer to run, and it’s always recommended to use the latest and greatest. When developing for iOS make sure your Xcode environment is properly set up. For instance, read How to Install Xcode, Homebrew, Git, RVM, Ruby & Rails on Mac OS X to get all specific details on how to setup Ruby with all depedencies, proper Xcode configuration etc. to get started with Appium for iOS. For Xcode requirements, there seems to be quite a lot of issues with older ones (6.x) so use the latest one and preferably with the latest iOS devices (9.3.1 at the moment).

Android

This probably causes some discussion but setting up Appium for Android is easier that getting things working on Xcode/iOS. Just install the Android Studio. When this is done (and yes, it typically takes some time to install, configure and make sure all Android API levels are properly installed/set up) set $ANDROID_HOME environmental variable to point your Android SDK path. If you unzipped the Android SDK to /usr/local/adt/, for example, you should add this to your shell startup:

export ANDROID_HOME=“/user/path/here"

Video – How to Run Your Appium Tests

Parallel Test Runs with Appium on Real Mobile Devices – Hands-on Webinar from Testdroid Bitbar on Vimeo.

With the proliferation of iOS devices year over year, the threshold of iOS success is higher than ever before. More companies like startups and SMEs find it not financially sustainable to acquire every piece of iOS devices with different OS versions and HW specs. On the other way, testing apps or games manually is not anymore an ideal option for QA process due to low efficiency and scalability issue.

Though iOS is still a more closed operating system compared to Android, many open source test automation frameworks and tools can be utilized to create robust automated tests. Along with the help of a cloud-based testing solution, it enables developers/testers to verify app quality more efficient in a scalable way and makes iOS testing tasks easier.

Today we’d like to go through 5 widely used iOS testing frameworks nowadays with simple code samples to give you a basic understanding of getting started with your iOS testing. If you are also interested in Android test automation frameworks, do not forget to check out Top 5 Android Testing Frameworks, where code samples are provided as well.

1. Appium

Appium is popular due to its flexibility and usability on both Android and iOS, and it works on native, hybrid and web applications. For iOS testing, it uses JSONWireProtocol to engage with iOS applications using Selenium WebDriver. Thanks for that, Appium does support mobile web testing very well and its use cases are very similar as if Selenium would be used for web testing.

Appium code sample:

driver.findElement(By.id("com.example.app:id/radio0")).click();
driver.findElement(By.id("com.example.app:id/radio1")).click();
driver.findElement(By.id("com.example.app:id/radio2")).click();
driver.findElement(By.id("com.example.app:id/editText1")).click();
driver.findElement(By.id("com.example.app:id/editText1")).sendKeys("Simple Test");
driver.findElement(By.name("Answer")).click();
// or alternatively with
driver.findElement(By.id("com.example.app:id/button1")).click();

2. XCTest / KIF

XCTest is tightly coupled with Xcode but is still usable with both real iOS devices and simulators. XCTest allows developers to write tests for components at any level and also provides the framework for UI testing capabilities. XCTest tests are grouped into subclasses of XCTestCase. Writing any tests with XCTest should be trivial to iOS developers as XCTest is fully compatible with both Objective-C and Swift.

KIF (Keep It Functional) is an iOS integration test framework that is closely related to and uses XCTest test targets. KIF tests can be executed directly in XCTestCase or any subclass. KIF allows for easy automation of iOS applications by leveraging the accessibility attributes that the OS system makes available for those with visual disabilities.

Code sample with Objective-C:

- (void)testClicksOnRadioButtons {
[tester tapViewWithAccessibilityLabel:@”Radio1”];
[tester tapViewWithAccessibilityLabel:@”Radio2”];
[tester tapViewWithAccessibilityLabel:@”Radio3”];
[tester enterText:@”Simple Test”
intoViewWithAccessibilityLabel:@”editText1”];
[tester tapViewWithAccessibilityLabel:@”Answer”];
}

Code sample with Swift

testClicksOnRadioButtons() {
let app = XCUIApplication()
app.radiobutton[0].tap()
app.radiobutton[1].tap()
app.radiobutton[2].tap()

app.staticTexts[“Simple Test”]
app.button[0].tap()
}

3. Calabash

Calabash is another great cross-platform framework that works perfectly with Android and iOS apps. One of the major differences to other frameworks is that Calabash tests are written in Cucumber. That means the test is written like a specification and is simple and easy to read even for non-tech people, but still executable by the automation system.

Calabash code sample:

Feature: Answer the Question feature
Scenario: As a valid user I want to answer app question
I wait for text "What is the best way to test application on hundred devices?"
Then I press Radio button 0
Then I press Radio button 1
Then I press Radio button 2
Then I enter text "Simple Test" into field with id "editText1"
Then I press view with id "Button1"

4. EarlGrey

To some degree, EarlGrey is the ‘Espresso for iOS’. It’s also developed and open sourced by Google. Google uses this test framework to test many iOS native apps including Google Calendar, YouTube, etc. As the codename goes, lots of similarities can be found between Espresso and EarlGrey. For example, EarlGrey tests will automatically wait for events (animations, network requests etc.) before trying to interact with the UI.

EarlyGrey code sample:

- (void)testBasicSelectionAndAction {
[[EarlGrey selectElementWithMatcher::grey_accessibilityID(@"ClickHere")]
performAction:grey_tap()];

// Example of long press with EarlGrey matchers
- (void)testLongPress {
[[EarlGrey selectElementWithMatcher::grey_accessibilityLabel(@"Box")]
performAction:grey_longPressWithDuration(0.5f)];
[[EarlGrey selectElementWithMatcher::grey_accessibilityLabel(@"One Long Press")]
assertWithMatcher:grey_sufficientlyVisible()];

// Example of multi-select, visible click on items
- (void)testCollectionMatchers {
id visibleSendButtonMatcher =
grey_allOf(grey_accessibilityID(@"Box"), grey_sufficientlyVisible(), nil);
[[EarlGrey selectElementWithMatcher:visibleSendButtonMatcher]
performAction:grey_tap()];
}

5. Jest / Jasmine

Jest uses Jasmine behavior-driven framework as the basis for testing JavaScript code. Every test case starts from describe() function call, similar to how JUnit uses TestCase class. The describe() function takes 2 parameters – the description/title of the test case and the function to be executed. The it() function includes all the test steps and provides (similar to JUnit) series of expect() functions.

Jasmine code sample:

describe("Player", function() {
var player;
var song;
beforeEach(function() {
player = new Player();
song = new Song();
});
it("should be able to play a Song", function() {
player.play(song);
expect(player.currentlyPlayingSong).toEqual(song);
//demonstrates use of custom matcher
expect(player).toBePlaying(song);
});
describe("when song has been paused", function() {
beforeEach(function() {
player.play(song);
player.pause();
});

it("should indicate the song is paused", function() {
expect(player.isPlaying).toBeFalsy();
// demonstrates use of 'not' with a custom matcher
expect(player).not.toBePlaying(song);
});
it("should be possible to resume", function() {
player.resume();
expect(player.isPlaying).toBeTruthy();
expect(player.currentlyPlayingSong).toEqual(song);
});
});
// demonstrates use of spies to intercept and test method calls
it("tells the current song if the user has made it a favorite", function() {
spyOn(song, 'persistFavoriteStatus');
player.play(song);
player.makeFavorite();
expect(song.persistFavoriteStatus).toHaveBeenCalledWith(true);
});
//demonstrates use of expected exceptions
describe("#resume", function() {
it("should throw an exception if song is already playing", function() {
player.play(song);
expect(function() {
player.resume();
}).toThrow("song is already playing");
});
});
});

BONUS: UI Automation

UI Automation is a tool developed by Apple that automates the testing of iOS applications. Put in another way, UI Automation to iOS is similar with UI Automator to Android. UI Automation tests are written in JavaScript, adhering to an API defined by Apple. Below is the code written on Apple’s Developer portal.

UI Automation code sample:

UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0]
If Edit button is the first button in your code, you can also have the code as below:
UIATarget.localTarget().frontMostApp().mainWindow().buttons()["Edit"]
Then you can use any of these three ways to call an action to tap that button:
UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0].tap();
UIATarget.localTarget().frontMostApp().mainWindow().buttons()["Edit"].tap();
var editButton=UIATarget.localTarget().frontMostApp().mainWindow().buttons()[0];
editButton.tap();
The Automation instrument has a complete element hierarchy that represents your app’s user interface. To view that hierarchy, use the logElementTree method to write an outline of it to the log:
UIATarget.localTarget().frontMostApp().logElementTree()

Conclusion

Though iOS app testing is totally different than Android app testing, you can use either Appium or Calabash to create test scripts that can be used to do cross-platform testing. That being said, every framework has its advantages and everyone of you have different needs for the projects you are working on. It’s recommended to do some further research, learn more about each framework and choose the one you are comfortable with. Happy iOS testing.

Dear Testdroiders,

This is the 25th blog of our 37 Things You Should Know About Appium blog series. Today we are going to quickly talk about the most common test zip mistake with Appium and how to avoid it to get your Appium tests running instantly with no hassle.

As you know, the Appium Server-Side Execution on Testdroid Cloud enables you to run Appium tests on all of our devices in parallel with no need to change complicated setups or modify any desired capabilities which must be done at client side. The process is simple – upload the target .apk or .ipa, upload your test scripts in .zip file and then you can just start the test run after selecting the device group. If you find that the test didn’t run properly, chances are that the test files are wrongly zipped.

If your Appium tests are finished with failure or didn’t even start, then one of the most important things you can check is how your project files are zipped. The typical mistakes of zipping Appium test files look as below – either all files are under a parent directory or part of the files are not found in the root of the zip file.

Mistake case 1: 

Mistake case 2: 

Mistake case 3: 

One thing worth noting is that run-tests.sh is the key to automating the tests (you can rename it as run-tests_android.sh for Android or run-tests_ios.sh for iOS). It is the launch point of testing process and used to execute any of the tests provided. It can also be modified to fit the purposes of the test project. Hence the correct way of a zipped file looks like below – all the files are at the root of the zip file.

Correct way for zipping:

Hope this quick tip will help you avoid the unnecessary check-up on your Appium tests for server-side execution. Please comment below and stay tuned as more tips and tricks for Appium are on their way here Testdroid blog.

Happy Appium Test Automation!

Dear Testdroiders,

Robot Framework is a generic keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD). Surprisingly, many of you have been using this framework (in the past) for a basic acceptance testing that extend the system level testing capabilities with specifications and test cases associated with the actual app testing. However, if you are not familiar with Robot framework or if you look for a way how to use it in Testdroid Cloud, I’ll walk you through of some basic things in this blog.

What is Robot Framework and How Does It Work

Robot Framework is extremely easy to setup, use and modify to get both Android and iOS apps tested. The test syntax that it uses is based on keywords and these keywords are quick to edit and further configure to make a match with the application under test. Furthermore, testing capabilities provided by Robot Framework can be easily extended with the test libraries which can be implemented using Python, Java and even some other languages are supported.

The nifty thing with Robot Framework is how those mentioned keywords work. Users can quickly create new keywords, either using the existing example ones or by writing everything from the scratch.

If you look for more generic information about Robot Framework, there are plenty of great examples and online documentation in GitHub.

The Basic Setup of Robot Framework

To get started with Robot Framework basically you need Python and pip installed. If you have the environment properly configured you can just execute the installation command lines:

pip install robotframework
pip install robotframework-appiumlibrary

After you have installed Robot Framework, you can take a look at the example where we use our Bitbar Sample Application with very basic Robot Framework test examples. For the cloud run you basically need a Python script and shell script that makes sure the test execution on cloud end works fine. For instance, you need to add the pip installation in your shell script to ensure everything gets properly installed on your cloud session as well.

In this example, we’re using Appium as a basis to run the test and make sure everything gets logged for further inspection.

## Cloud setup
echo "Starting Appium ..."

/opt/appium/appium/bin/appium.js --log-no-colors --log-timestamp --command-timeout 120 >appium.log 2>&1 &

# Wait for Appium to fully launch
sleep 10

ps -ef|grep appium

echo "Extracting tests.zip..."
unzip tests.zip

echo "Installing pip"
curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user

echo "Installing requirements"
~/.local/bin/pip install -r ./resources/requirements.txt --target .

## Start test execution
echo "Running test"
pybot -x TEST-all tests/android_example.robot

This should be all what you need for a shell script. To create a compatible tests you need to add few lines to ensure all paths are recognized and test cases can be found from the right fileset. The file structure for test package can be something as follows:

The following example includes both Android and iOS application in the test package (under resources->app folder). The test definition can be found under ‘tests’ and testing libraries are included under libs->examplelib. In addition, you can easily and quickly tweak and create your own test libraries with the instructions shown on the Github documentation of Robot Framework.

What Data Can Robot Framework Produce

When running a Robot Framework test in Testdroid Cloud users do not need to do any significant tweaks for their test runs (or job configs). The test automation and execution works the same way as with any other framework, and results are quickly available from any of those Android and iOS devices.

If you are starting out test automation with Testdroid Cloud and planning to use Robot Framework, simply create a project (Android or iOS), upload your application (APK or IPA), upload the test package, and select whatever devices you want to use for a test run. Just in few minutes the test run will be done (naturally this depends on the length of your test case) and the following data can be fetched from a test run:

This example includes three basic test cases (we’ll come back to those in the next chapter of this blog). However, after the test run is finalized you have now all the data in one, compact view of Testdroid Cloud. If you click any of those files shown in overview you’ll get full log opened in the view below:

For instance, the shell script file that we introduced in the first chapter of this blog extracted the tests.zip, installed the PIP, setup other tools and finally executed the example tests. The output of the log can be also inspected in this view – and for example it will look something like this:

Imports done
==============================================================================
Tests
==============================================================================
Tests.Example
==============================================================================
Simple Smoke Test - Correct Answer | PASS |
------------------------------------------------------------------------------
Simple Smoke Test - Wrong Answer | PASS |
------------------------------------------------------------------------------
Simple Smoke Test - Wrong Answer 2 | PASS |
------------------------------------------------------------------------------
Tests.Example | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
Tests | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
Output: /home/ubuntu/test/output.xml
XUnit: /home/ubuntu/test/TEST-all.xml
Log: /home/ubuntu/test/log.html
Report: /home/ubuntu/test/report.html

In addition, to get comprehensive understanding of how the app (with test) really does on the physical device, you’ll also get performance data about the CPU and memory consumption:

Okay, that’s some fundamentals for creating test runs using Robot Framework and managing the data after tests are done. Let’s take a look of how easy those test cases are to build and what you need to know about Robot Framework to use it efficiently with Testdroid Cloud.

How to Create Robot Framework Compatible Test Cases

First, some generic settings are defined for application and test session. Resource links to a file where all common procedures are provided in the test package. Test Teardown provides instruction of what system should do when test session is done:

*** Settings ***
Resource                 ...${/}resources${/}common.robot
Test Teardown            Close Application

For the variables, creator of a test should define some basic things here. In the Bitbar Sample Application there are name field (editText) and one button (button1). IDs and actions can be defined in this section:

*** Variables ***
${NAME}                  John Doe
${INPUT_NAME_FIELD}      id=com.bitbar.testdroid:id/editText1
${SUBMIT_BUTTON}         id=com.bitbar.testdroid:id/button1

The example application looks like this:

Now, let’s look at those test cases (using Android). To create three different test cases for the application, we start with a simple smoke test where correctness of selection is validated. Correct answer in this scenario is when option to use Testdroid Cloud is clicked and name is entered in the text field. Two other test cases indicate the wrong answer with radio buttons (Option to buy 101 devices and ask mom to help you). Test cases are as simple as follows:

*** Test cases ***

Simple Smoke Test - Correct Answer
   [Tags] cloud
   Set Up And Open Android Application
   Input Name ${NAME}
   Select Option Use Testdroid Cloud
   Submit Selection
   Validate Correct Answer

Simple Smoke Test - Wrong Answer
   [Tags] 101
   Set Up And Open Android Application
   Input Name ${NAME}
   Select Option Buy 101 devices
   Submit Selection
   Validate Wrong Answer

Simple Smoke Test - Wrong Answer 2
   [Tags] mom
   Set Up And Open Android Application
   Input Name ${NAME}
   Select Option Ask mom for help
   Submit Selection
   Validate Wrong Answer

When test execution is started, Robot framework first parses the test data. After that it uses keywords provided with the test library/libraries to interact with the system.

In the keyword example, there is an input name, select of options (3 radio buttons) and submit of a selection (=a button press). In addition, keywords section can include validation of correctness. In the example below, selecting the right radio button uses xpath to find a name included in the actual UI element:

*** Keywords ***

Input Name
   [Arguments]      ${name}
   Input Text       ${INPUT_NAME_FIELD} ${name}
   Hide Keyboard

Select Option
   [Arguments]      ${option}
   Click Element    xpath = //*[contains(@text, '${option}')]

Submit Selection
   Click Element    ${SUBMIT_BUTTON}

Validate Correct Answer
   Wait Until Page Contains      Congratulations ${NAME}       5s

Validate Wrong Answer
   Wait Until Page Contains      Wrong Answer                  5s

Conclusion and Download Link

Robot Framework is very handy and easy to use. Some of you have been using it in the past so in case you have your tests implemented, and you have your Android / iOS app ready for test, give this framework a try at Testdroid Cloud. You’ll be surprised how smoothly things work and how quickly you can do acceptance testing. Furthermore, here is the full test package available for a download (and tweaks).

Have fun with Robot Framework at Testdroid Cloud!

Dear Testdroiders,

Over the past year since we introduced how to use the technology of Image Recognition to run automated Appium tests for mobile games, we’ve noticed a skyrocketing amount of test runs on Testdroid Cloud using image recognition for mobile game test automation. Today, we’d like to share another quick tip to any mobile game developers or QAs who are just new to the image recognition for mobile game testing. To make sure your test scripts work well on different devices with the method of image recognition, you need to prepare images with different resolution.

If the images are too sharp in high resolution, they can possibly not be identified and found on low resolution devices. This also applies to low resolution images on high resolution devices. And the rule of thumb here is that high resolution images should be used to work on high resolution devices (1920×1080), while the low resolution images are for low resolution devices (1280×720).

That being said, you can still try with one set of images with middle resolution in an attempt to work on both high resolution and low resolution devices. But it depends on how big the differences are. Here’s one example that one set of images work for both device resolution. We used one set of query images.

Then we ran it on 5 different devices with different screen resolution, among which Acer Iconia Tab 8 has 1920×1200, Asus Google Nexus 7 has 1280×800, and the rest all have 1920×1080. The test itself went successfully.

However, games are different than each other. For HillClimbRacing, taking one set of images with middle resolution work for different devices. It doesn’t mean that the same method works for you as well. Without the proper images, your tests will eventually fail. The best way to address that is to use two different sets of images for different screen resolutions.

For doing that, you need to take all the same images on a lower resolution device and have them in a separate folder (here we use ‘low_resolution_images’).

Once you are done with the images, you need to add code in your script so that these low resolution images will be used when tests are running on low resolution devices. For example, the added code in Java script could look like

public void setQueryImageFolder() {
Dimension size = driver.manage().window().getSize();
log("Screen size: " + size.toString());
if ((size.getWidth() <= 520) || (size.getHeight() <= 520)) {
screenshotsFolder = "low_resolution_images";
}
}

Change the value of size.getWidth and size.getHeight accordingly based on your targeted devices.

Hope this Appium Tip would help you succeed in image recognition for mobile game testing.

Happy testing with Appium and Image Recognition at Testdroid Cloud