General testing considerations

In every software development it is important to include testing rules in the process. These rules ensure quality and a continuous evolution over new integrated features. For a developer, testing regularly allows more confidence during new implementations – the more developers test their code, the more they will guarantee the quality of it. Of course, this will take them a little while, if they just do it manually.

There are different types of tests depending on the stages of the project.

• Unit tests, which induce a step-by-step verification of the developer’s work by themselves. The idea is to verify that each unit (which can represent a function or a class depending on the projects conventions)work as intended. These tests are more numerous and have the narrowest scope of all the tests.
• Integration tests. Most applications integrate parts outside of the basic production environment (databases, file systems, network calls to other applications) that are common to dissociate during unit tests. The interactions and links that these parts have with each other are tested during integration testing.
• Functional tests, which can be divided into three types:
  • The Human-Machine Interface (HMI) tests: the purpose of the HMI tests is to verify that the corporate visual identity has been respected throughout development in two aspects:
    • The visual presentation: menus, display parameters, window properties, icon bars, screen resolution, edge effects, etc.
    • Navigation: means of navigation, shortcuts, the result of moving around on a screen, etc.
  • Application tests, which verify security and for example compatibility between different browsers
  • Load tests, which scrutinize the project’s performance and robustness.

New implementations in a project are often the vector of instabilities or anomalies, with sometimes unforeseeable consequences. That is the reason why it makes sense to perform non-regression testing, as a type of safety net. Non-regression testing can take a lot of time and effort for teams. In addition, the non-regression test phase is at the end of the chain of all projects carried out in classic or agile mode. This phase is regularly subjected to strong constraints of schedules and budget. This is why it is interesting to automate those tests.

Once operational, tests automation has many advantages:

• Time savings, due to a reduction in the workload of the teams.
• Increase in productivity.
• Reliability and quality of the tests carried out.
• Increase of the level of completeness of the tests carried out.
• Free teams from repetitive tasks in favour of tasks with greater added value (Analysis of results, definition of test cases, planning, etc.).

Automatic testing in the scope of the VRT project

As part of Work Package 3 (WP3), which focuses on the delivery of specific software and hardware components for the project, the consortium organizes regular testing sessions. Latest development and functional improvements are analysed during meetings called Test fests. Lately, these meetings highlighted the need to automate some integration tests that verify links the different components with a main component which manage all of them: the orchestrator.

These tests are time-consuming if performed by humans and seem very simple to validate via an automatic process. Moreover a solution which can emulate the user’s actions could be used later for HMI tests.

In order to evaluate efficiently testing tools, it is important to follow a precise path:

• Know when and how it is possible to automate these tests
• Translate the initial need into a test scenario: without interpretation, without forgetting any important features
• Execute the tests with rigor and follow the correction of anomalies
• Organize the tests according to the means of the company, the size and the complexity of the software First, a functional, UI (focused on the user interface) non regression test will be created.

How to choose the best suitable tools for automatic testing in Unity?

Of course, many test systems exist. The idea is to find the most suitable solution adapted to VRTogether requirements.

Test systems will be evaluated according to the following criteria:

• Windows platform
• Accordance to the concerned testing type.
• Input fields detection on unity
• Price: This research will focus on open source projects.
• Technical Support and Assistance: Automation engineers sometimes need help on critical problems.

The research for the best system induced naturally the experimentations, and so it makes sense to first create an interface that presents the milestones of the VRTogeter’s usage scenario. It’s a way to identify clearly the testing software’s specificities without confusing on the VRT’s. For the sake of efficiency, a usage scenario reduced to its strict minimum should be created in accordance to a simplified interface made on unity.

The intention is to simulate the behavior of a user who interacts with an interface comprising of a login password section and a registration section with writing areas for the name, email, password and confirmation of the password. During a first test the users will identify themselves with a previously saved profile (it takes the form of a text file at the root of the project), and during a second test the users will create their own account in the register section.

The simplified scenario:

• Check that the program is started
• Check the existence of the login button
• Fill in the login text field with the username
• Fill in the password text field with the password
• Click on the login button
• Verify the interface change

It seemed coherent to us to initially focus our search for an automated test solution working with unity on open source solutions. If the outcome of this research had not been conclusive, our research would have turned towards profitable solutions.

This section will provide an overview of the different test solutions that have been explored. Then a more precise explanation of how AltUnity works and why it was chosen as the most adapted solution follows.

Unity Test Runner

docs.unity3d.com/2017.4/Documentation/Manual/testing-editortestsrunner

Unity Test Runner (UTR) is a tool, now directly integrated into Unity, the tests are written in C # with the nunit library. The checks are done thanks to the assert class; it allows to know if the tests are positive or not.

Of course, there is no limit to the number of possible checks.

The tests can be carried out in “edit mode” from the Unity editor which allows code checks without launching the player and in “play mode” which allows you to see the effects of your test in the unity scene. The tests in play mode add more assemblies during the construction of the project, and therefore increase the size and the build time. Unity test runner is primarily useful for unit testing and possibly for integration testing. Thus, its operation in play mode involves a sort of division of the scene to isolate the fragments of code which are tested. The unity editor can also have different performance depending on the computer used (CPU, GPU, OS …). Automated functional tests must be applied to environments as close as possible to production environments. And this is why the Unity Test Runner is not the most suitable tool.

Selenium

https://www.selenium.dev/

Selenium is “a set of web browser automation tools that uses best available techniques to remotely control browser instances and emulate a user’s interaction with the browser” (Selenium, n.d.). The notoriety of the tool and the big community around it are sufficient reasons to explore Selenium.

Selenium works on a web browser, here the experiment is done with the chrome webdriver. Writing a few lines in Java is made easier by selenium IDE. Automating a functional test on a website therefore does not present any difficulties. Unfortunately, selenium does not have a driver allowing to make the link with the Unity editor or the build of the VRT application. This constraint makes Selenium unsuitable. The Selenium solution finds its equivalent for unity, this one is called Appium.

Appium

http://appium.io/

Appium was developed by Dan Cuellar in August 2012, it is actually written in node.js. Appium works with IOS, Android and theoretically Desktop applications. Starting a game with Appium does not pose any difficulties, but identifying the elements of the scene and creating interactions presents a sticking point. These concerns can be resolved with an image recognition system using the open CV library as raised by Ru Cindrela – engineer at Altom- during her conference on Appium given in 2018. VRT’s requirements lead to find a standalone solution that can identify unity objects separately from each other. Appium is therefore ruled out.

Unium

https://github.com/gwaredd/unium

Unium gives a flexible interface that allows to remotely control and inspect a game while it is running. While the API Unium offers some very interesting possibilities to inspect game behavior, object coordinates or latency, it is not a stand-alone solution. Its compatibility with Appium circumvents one of the major problems of Appium, the object detection. The association between Unium and Appium is of great interest in the testing of mobile applications.

Unium is available for free on GitHub. This is a library for Unity that allows test verification. Even if it is Appium compatible, it’s important to clarify that this is not an automation tool. It is presented as an HTTP API which can be used by the testing tools. It comes in two parts:

– A web server directly integrated into the game

– A query system that allows you to search and manipulate the elements of the project.

Autoit

https://www.autoitscript.com

Autoit is a high-level programming language allowing to automate the user’s behavior who evolve under a Microsoft Windows OS, it gives a wide choice of function libraries. There is an editor and an Autoit IDE. If in a Windows environment, the tool has no significant limitations, the creation of functional tests for unity presents blocking points.

In the usage scenario Autoit is not able to identify the existence of the login button, but if it has received the coordinates of this one as well as its colour code, it will be able to check if the pixels of the button have the expected colour at the expected moment.

This solution which considers the coordinates of the pixels, their colour and the location of the mouse cursor makes possible to automate the behavior of a user in an environment that Autoit doesn’t know while still maintaining the execution of the scenario. To get some feedbacks, when launching the VRT project, Autoit generates in a separate document a txt file in which the verifications are notified.

Indeed, if the aspect of the project takes a different form or if it’s responsive, the Autoit script must be modified.

From this observation, the research turned to a solution capable of identifying the elements of a Unity project.

AltUnityTester

https://altom.gitlab.io/altunity/altunitytester/

AltUnity tests the ability to identify the objects present in a Unity scene and to create interactions with them. This solution is an alternative to Appium presented by Ru Cindrea at the SeleniumConf in Chicago. (SeleniumConf, 2018) AltUnity Tester is an open source test automation tool developed by the company Altom, it takes the form of a user interface that should precisely help detect objects in a game and interact with them using written tests in C #, Python or Java. These tests can be run on real devices (mobile, PC, etc.) or in the Unity editor. This unity plugin seems to meet expectations because it offers a possibility to check the functioning of the tests in the editor and even to make a build. This particularity seems to us suitable for an evolution for performance tests. In addition, this tool is compatible with Appium (It could be just as effective for phone applications in other innovation projects).

It works through WebSocket server and a client. A connection can be established via an API and an interface in the unity editor.