Yeah, this is one area where webdev is way behind other fields, and i think we're going to see lots of new tooling in this area soon.
We're currently working on a way to help devs test web app functionality and complete user journeys without having to actually write tests in Selenium or whatever. The idea to is let devs write down what they want to test in English ("load the page", "search for a flight", "fill the form", "pay with a credit card", etc), then we'll use NLP to discern intent, and we have ML-trained models to actually execute the test a browser.
You can give us arbitrary assertions, but we also have built-in tests for the page actually loading, the advertising tags you use, page performance, some security stuff (insecure content, malware links). At the end we hand you back your test results, along with video and perf stats. It’s massively faster than writing Selenium, and our tests won’t break every time an XPATH or ID changes.
>tries to do so by using an imprecise, context-dependent language designed for person-to-person communication to instruct a machine
???
Selenium is its own can of worms, but it absolutely sounds like you're using the wrong tool for the job here. The problem stopping people from writing browser-based tests is not that people can't understand specific syntaxes or DSLs, it's actually the opposite: people don't have a good, reliable tool to implement browser-based testing in a predictable and specific way that does what a user would intuitively expect.
Selenium fails here because it has to manage interactions between browsers, because selectors are hard to get right on the first try and continually break as the page's format changes, because JavaScript can do literally anything to a page and that is really hard to anticipate and address reliably from a third-party testing framework like Selenium, especially if components are changing the DOM frequently, etc., because Selenium is subject to random breakage at the WebDriver layer that hangs up your (often long-running) script, and so on.
Whatever the right answers to a next-gen Selenium are, attempting to guess the user's meaning based on Real English by something that is itself an imperfect developing technology like NLP is pretty obviously not the correct toolkit to provide that. Remember, a huge amount of the frustration on Selenium comes from not having the utilities needed to specify your intention once and for all -- the ambiguities of plain English will not help.
If your thing works, it will have to end up as a keyword based DSL like SQL. SQL is usually not so scary to newcomers because a simple statement is pretty accessible, not having any weird symbols or confusing boilerplate, but SQL has a rigid structure and it's parsed in conventional, non-ambiguous ways. "BrowserTestQL" (BTQL) would need to be similar, like "FILL FORM my_form WITH SAMPLE VALUES FROM visa_card_network;"
The biggest piece that's missing in Selenium is probably a new, consistent element hashing selector format; each element on the page should have a machine-generated selector assigned under the covers and that selector should never change for as long as the human is likely to consider it the "same element". The human should then use those identifiers to specify the elements targeted. I don't know how to do that.
The second biggest piece that's missing from Selenium is a consistent, stable WebDriver platform that almost never errors out mid-script; this may involve in some type of compile-time checking against the page's structure or something (which I know is hard/possibly impossible because of JS and everything else).
Totally agree with this. The concerning part for me is that ML is about making a "best guess" given some data. This means that your tests may pass one time and fail another - inconsistent tests aren't tests at all.
Your post gave me deja vu to an automation workflow I cobbled together a few months ago, which I found wonderfully productive for steering a bot: Vimperator keybindings. Selenium can use most of them right out of the box. It's a terrific navigation layer. For instance, pressing "f" enumerates all the visible links on the page and assigns to each one a keybinding. The keybindings are displayed in tooltips and can be trivially extracted with CSS. You can keep sending keys to the browser, and only links that contain the anchor text remain in the set of candidates. Of course the "hashing" of links to keybindings is completely relative to the viewport, so this won't satisfy you completely. But it was an idea I had randomly one day, as an alternative to the trapeze act of navigating through the boughs of the DOM tree, and lo and behold it worked nicely.
Sounds intriguing. There are a few tools for recording interactions with a webpage in order to replay the actions as a test (Ghost Inspector, Selenium IDE, etc) but they tend to be pretty horrible. I've been working on my own as a Chrome extension for a little while. What you're building sounds really interesting though, especially if it can deal with complex Javascript apps. Anything that can make developers more inclined to test things is a good thing.
Alternatively, one of the consequences of React is that the front-end can largely be unit-tested. You can at least get a pretty good idea that the page will render what you expect if it gets the data you expect.
And whether or not it gets that data is a unit test in another place.
I'm not a huge fan of React, or javascript, but having been forced to work in it, this is one of the wins.
You could test the front-end before react though? Also, one can still have a God component. I don't think React changed anything from a testability point of view. Well written modular code is well written modular code.
Frameworks like Knockout have been around for quite some time now. You don't have to use React to not depend on the DOM. There are many alternatives to "jQuery based front-end development" that's not React. Aurelia, for instance, happens to be an amazing framework in my opinion that's also highly testable and that's not React. Like I said, modular code is modular code. You can write good, modular code with just require js modules, and you can also write terrible monolithic React components.
Testability isn't the domain of the view layer.
Abstracting the DOM into a declarative DOM is great for performance, but doesn't lead to necessarily more testable code.
We are using ghostinspector mainly for it ability to compare screenshots between runs. I think the future of testing will be apps like this that don't require you to specify every little div but just record your actions and play them back and catch differences. Right now ghostinspector only takes a screenshot at the very end but they are adding a feature where you can take a shot anytime. As these apps get better at knowing what matters and what to ignore - all the better.
Yeah, I've looked at all the current tools and there's basically two types (other then just writing straight Selenium):
- Test recorders that aren't a great experience and output incomprehensible, brittle tests.
- Test composers that I can best describe as 90's SQL query builders for Selenium.
Complex JS apps are still a challenge for us (especially with some of the WTF code we come across in the wild), but we have a strategy in the works for them. We're still pre-release though. If you're interested, send me an email (donal@unravel.io) and I'll add you to our alpha list.
From reading the web page, I think an Unravel user won't need to specific, pre-defined language, in which case it's very deliberately not a DSL like Gherkin/Cucumber.
We're currently working on a way to help devs test web app functionality and complete user journeys without having to actually write tests in Selenium or whatever. The idea to is let devs write down what they want to test in English ("load the page", "search for a flight", "fill the form", "pay with a credit card", etc), then we'll use NLP to discern intent, and we have ML-trained models to actually execute the test a browser.
You can give us arbitrary assertions, but we also have built-in tests for the page actually loading, the advertising tags you use, page performance, some security stuff (insecure content, malware links). At the end we hand you back your test results, along with video and perf stats. It’s massively faster than writing Selenium, and our tests won’t break every time an XPATH or ID changes.