Robot Framework is a programming language

9

10. May 2013 by Michael Wowro

Working on a large project (~200 web developers) with Robot Framework (RF) for ~3 month, I consider the notion of “framework” in Robot Framework a bit misleading. Robot Framework is more than a framework, it’s also a programming language. In this article I don’t want to bash Robot Framework as a testautomation language, even if I don’t feel comfortable with its strange syntax. I want to show that Robot Framework is a programming language and draw some conclusions.

RF has its own syntax
RF has variables (with different scopes)
You create them like this:

${hi} = 	Set Variable 	Hello, world!
${hi} = 	Set Test Variable 	Hello, world!
${hi} = 	Set Suite Variable 	Hello, world!
${hi} = 	Set Global Variable 	Hello, world!

BuiltIn-Library Documentation

RF has functions
In RF they are called keywords.
You create them like this:

Return One Value 	[Arguments] 	${arg} 	  	 
	Do Something 	${arg} 	  	 
	${value} = 	Get Some Value 	  	 
	[Return] 	${value}

User Guide

RF has conditional statements
It looks like this:

${result} = 	Run Keyword If 	${rc} == 0 	Zero return value 		
... 	ELSE IF 	0 < ${rc} < 42 	Normal return value 		
... 	ELSE IF 	${rc} < 0 	Negative return value 	${rc} 	arg2
... 	ELSE 	Abnormal return value 	${rc}

BuiltIn-Library Documentation

RF has loops
And they look like that:

Run my hobbies
    :FOR 	${index} 	IN RANGE 	1 	11 	
    \    Watch TV
    \    Plague your neighbor
    \    Play with your dog	  	 

User Guide

RF has its own IDE
It’s called RIDE and it has some bugs.

RF has its own libraries
Look e.g. in the Selenium2Library
You’ll see, that the naming has nothing in common with what you’re used to from the WebDriver API.

… and RF has its own gotchas and bugs
Because the testcase-timeout produces a flaky bug in the reporting engine, I had to implement a timeout on my own … loop … sleep … boilerplate …

Conclusion

  • Test automation is programming. Robot Framework is a programming language. And you have to spend time on learning it.
  • I haven’t found any advantages over Java/TestNG yet (after 3 month working full-time with it).
  • Think of the Java web-devs in your project/scrum team. Do they want to read/maintain test scripts in a foreign programming language?
  • The skill-combination of Robot-Framework and browser-automation is seldom (In XING (leeding business network platform in Germany) you find the skill-combination “Selenium AND Robot Framework” with 12 people. But the skill-combination “Selenium AND Java” you find at 1000 people.
    • So think of stuffing your project: it is easier to find a pro with Java skills than with Robot Framework skills.
    • So think of the size of the community for support, advancing the tools, articles, …

If you are an adventurer, try it with Robot Framework.

Advertisements

9 thoughts on “Robot Framework is a programming language

  1. autumnator says:

    I agree with your findings. However i disagree of your tone/opinion of RF’s ineffectiveness.

    There is a specific niche that RF fills. As you’ve found it may not fit well against a (Java or web) developer-centric group, and if all you is mostly Selenium automation, it may not fit as well.

    If you ever have to involve more junior-level or novice test writers/programmers, or involve non-programmers in specifying/maintaining tests, the domain specific language of RF abstracts away the underlying complexity of the actual automation work involved. E.g. you have a group of novice test writers who write tests against the RF DSL language keywords, and you have a (smaller) group of automation developers/architects who actually build out the RF libraries that implement those keywords (abstracting complexity and interfacing to the actual automation tools or code that do the work). In essence, RF itself can be though of as like an API client that consumes the APIs (RF libraries).

    Another benefit you will start to see if you are involved in a project that involves multiple disparate types of automation & tools due to multiple application endpoints that together represent the complete system you have to “integration” test. E.g. desktop GUI + web GUI + mobile app GUI + telephony interface/API + web services, etc. If you don’t use RF or similar type framework and just go with TestNG or JUnit, you have to completely roll you own custom frramework to glue the different tools together. RF provides a nice library specification that you implement tool specific libraries against making it easy to link the different tools together via a common domain specific language (RF keywords and test syntax). With other frameworks, that is sometimes more work to do. Another benefit in this area is that RF also provides a remote library interface specification (communicating via XML-RPC) for building test libraries in languages or platforms not native to RF (e.g. .NET/C#, Perl, PHP, C/C++, Objective-C, whereas RF natively supports Python, Java via Jython, and .NET via IronPython). On top of that, RF comes with some publicly available premade test libraries like SeleniumLibrary/Selenium2Library, etc. and premade remote library interface servers to build libraries on top of (Perl, .NET, Java, etc.) that save you work from making your own libraries unless you really need to.

    RF has a varying range of simplicity/complexity to cater to different users. You can customize the framework & write own libraries or customize existing ones, and use the programming features you meantion for the complex side (developer-centric), or you can simply call collections of keywords to build a test for the simple side (novice test writer) and only invoke RF programming features you mention as needed.

    To summarize, RF is not a one-size-fits-all framework and so it may or may not work for you depending on your needs. But if you fall into any of the niche groups I mention above, it will be a highly useful and potential framework to evaluate.

    On a side note, some RF test libraries and remote library server implementations can be reused outside RF as useful tools/libraries based on their modular design. I know this because I’ve written some such libraries and remote library servers and have reused some outside of RF adapted for other frameworks.

    • Hi David,

      thanks for your detailed comment.
      To your statement about the advantages of RF in integrating several testing tools, I haven’t a firm answer yet.
      To your statement about mixed teams: I experienced, that scripting the testcases with a predefined set of keywords is not the time-consuming part in a large testautomation project. On the one hand, it’s really time-consuming to get some correct, stable and non-redundant testcase-specifications – that would be the part of the non-programmers. On the other hand it’s time-consuming to solve the complex automation problems (clever timeouts, nested loops, sophisticated xpaths, skip-mechanism, …) – that would be the part of the programmers. If my experience reflects the state in most test(automation)-projects, a (big) team of (non-programming/novice) testerscripters wouldn’t raise the overall efficiency at all. IMHO keyword driven testing is a dead end in testautomation evolution, because it solves the wrong problem. Non-programmers should create fanatastic test-specifications – and that’s hard work. Programmers should be equipped with a fantastic toolset. A novice testautomater should hurry up and learn programming (sitting next to a senior).

      Yours Michael

      • autumnator says:

        Agree with your thoughts here. Sadly we live in a less than perfect world…

        * non-programmers don’t always create good thorough (and parameterized, minimal hard coded) test specifications. based on my experience

        * we programmers don’t always get the toolsets we desire, with respect to test automation. We’re making headway in mobile and web with Selenium and Appium. But desktop GUI test support sucks, though we’re getting better with native support from Microsoft and Apple, but the APIs pale compared to Selenium in easy of use.

        * novice test automator – we have too many of these folks. And even where they know programming, their competence is lacking compared to a good developer. Thus what automation work they do will suffice for the project, but doesn’t give you the desired long term scalability and maintainability to achieve great automation results and not passable mediocre ones. It is sad when a company has more of these folks than the good test automation architects that build out the good infrastructure and tests. How might we work to improve these folks here or weed them out of the organization to have just the best of the best? In my experience, they are the limiting factor to good automation results. We should either have good manual testers, or good test automaters, but not folks in between that do mediocre at both or are better at one but have to do the other as well for whatever reason.

  2. mkorpela says:

    Interesting post!

    I really like your comment about RIDE “RF has its own IDE It’s called RIDE and it has some bugs.” – as I happen to be one of the more active developers of that tool 😀 and know something about those bugs.

    I do agree that RF has the elements needed for programming, but no one (I hope) is forcing you to go that way.

    If you are more familiar with Java (or Python), you can do all your programming in that language and just let RF to have your high level test cases and produce logs and reports. – Clearly separate tests (intent) from the way you automate them and use directly that familiar Webdriver API in you Java code. No need for those RF libraries.

    This is what in my opinion living documentation is really about. Binding those test-specifications (real non-progammer readable tests) directly to the automation and with a CI system keeping that documentation and automation and system under test on the same level.

    Eventually you will have similar patterns of tests steps in your test cases – and then you can extract those similarities to user keywords if the end result still serves the purpose of those tests..

  3. Hi Mikko,

    thanks for your insightful comment.
    1.) Well, I’m not conviced yet about keyword-driven-testing, as you might have mentioned.
    2.) I needed to workaround because of the buggy test-case timeout (it was troubling the creation of the xml-report). At least in this case I was forced to use the Robot Framework’s own programming language 🙂
    Beside that, I think your direction is absolutely right. If you decided to use Robot Framework in your project, use the programming language most team members are familiar with.

    Yours Michael

  4. Bolto Cavolta says:

    Hi Mikko,

    A very rare kind of perspective but a very great post. For some reason the test automation community seems very convinced of what the Robot framework offers. In my current project, a web app that has a couple of backend sub-systems, we found automating tests using Robot framwork is too limiting. Example:

    1. Complex logics ended up being written in Java and Python
    2. REST API, database, and SSH access (part of our fully automated tests require dynamic data creation … etc) test methods have to be written in Java/Python as well
    3. Keywords are hard to manage despite a few attempts to establish a sound convention
    4. Our test case management tool is TestLink and one of our mandates in automation is to use TestLink’s test plan xml file as the input test suite file and have test results updated in TestLink as well. To do this using Robot framework only adds more overhead as Robot framework does not have annotation like Java does and it was very easy to do it in Java (we simply ended up using TestNG and it has served us well)

    The list goes on and we are wondering why it is such a big deal about using Robot framework in the automation community. We do realize the popular pros of Robot framework but it’s just that when we start using it, it is just really adding a layer of overhead. If you have to write test methods in other scripting languages, why introduce another layer of translation?

    regards,

    BC

    • Hi Bolto,

      thanks for your detailled comment about your experiences and confirming my reservations 😉

      Yours Michael (not Mikko)

      • autumnator says:

        Interesting discussion. Some further comments:

        * Where are we hearing this “test automation community” embracing RF? It might be geographical but I’m not sure. As far as I know, in the US, Silicon Valley, I have not heard of RF being very known or popular. It’s not an unknown either but not everyone (tester, developer, etc.) is aware of it. Even I was not aware of it when I was researching & developing my own generic test framework a few years back and ended up deciding RF was a better approach than to build yet another test framework to meet my needs then. So I don’t think that globally, this is that big a thing to gripe about, but I could be wrong.

        * RF has come a long way over the years, with some expansion in the past few years. When I first started looking at RF, you only had Java (via Jython) and Python support, and basic .NET support via IronPython. Around the time I started contributing, some others did too, and we now have support to integrate RF with .NET (remotely in pure C# or should you choose VB.NET), Java (purely & remotely, outside Jython), node.js, Perl, PHP, and some others. So there’s potential to it, even though it might not be perfect.

        * I’m curious, what you guys state as problems with RF and your alternative solution seems somewhat based on code that’s geared toward specific language and platform. Say you actually had to deal with an application (and thus automation) solution that actually dealt with code/APIs in different languages & platforms (well, we can assume one will be a primary one, and the rest secondary but all important). How would you implement the automation if not going with RF to integrate them? Create your own general DSL to tie together the different components or each one will have their own DSL specific to their own component. Choose one master framework (say in TestNG or JUnit, etc. with custom code) and glue it all together with some glue code/scripting? What I mean here is your entire system needs to interact/test with a mobile component in language A, a desktop component in language B, a web component in language C, all communicating with APIs in language D.

    • autumnator says:

      I’m curious to wonder regarding your thoughts with respect to this article, Bolto and Michael. Is this with respect to RF as a (programming/scripting) language or with respect to it as a framework. Yes, one can say it is both together in ways, but it would be in ways inappropriate to evaluate it on the basis of both, if it happens you don’t like one aspect of it. Because one doesn’t have to use the programmatic features of RF, and go lower level with the real languages. Like RF also offers BDD and data driven capabilities but you don’t have to use it, sticking with keywords only.

      So I’m curious to what you would think of RF solely as a test framework ignoring the programmatic aspects of it and comparing it to say:

      * STAF (http://staf.sourceforge.net)
      * JSystem(Test) – http://at4qa.blogspot.com/2010/03/jsystem-test-automation-framework.html
      * Cucumber, RSpec, and derivatives for other languages
      * Fit/FitNess variants

      The evaluations here about the limitations of RF are in ways like evaluating the limitations of Selenium IDE compared to Selenium RC and WebDriver as they’re all part of the same project but separate components you can choose to use.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: