RFC: Move to a standard JS test framework (e.g., jasmine)
Hi everyone, In an effort to ease some of the cross-browser development aspects, I’m spending some time getting the karma test runner work that Jos started a while ago functional again. This migration is somewhat trickier than it should be, mainly because WebODF has a completely custom test framework (core.UnitTester). This means that: * all karma integration must be written from scratch * it is difficult to run individual tests from karma (very handy when developing from WebStorm) * text reporting from karma is *very* messy * missing shiny features from other test frameworks :'( I was wondering if anyone would have complaints about ripping out the custom framework and replacing with an existing, widely used one (e.g., Jasmine, http://jasmine.github.io/2.0/introduction.html)? The only reason I suggest Jasmine is that I'm familiar with it's features from other projects I work on. Key benefits of switching to Jasmine (probably apply to other frameworks too!): * already has integration for karma via the karma-jasmine plugin * individual tests can be ignored or run directly from any IDE with karma support * has a host of existing addins (e.g., better HTML reporters) which means we can easily improve the usability of the test results from the browser as well as in the qtjsruntimetests (which currently spit out raw html in the console :-/) * good documentation * support for async tests * easier fixture creation
From what I can tell, everything implemented in the existing custom test framework is supported (or supportable without much work) in jasmine without too much work.
I'm happy to do the work... just wanted to get wider approval before spending time on such a patch :-). So… any objections? Cheers, Philip
On 09/15/2014 04:45 AM, Philip Peitsch wrote:
Hi everyone,
In an effort to ease some of the cross-browser development aspects, I’m spending some time getting the karma test runner work that Jos started a while ago functional again.
This migration is somewhat trickier than it should be, mainly because WebODF has a completely custom test framework (core.UnitTester).
This means that: * all karma integration must be written from scratch * it is difficult to run individual tests from karma (very handy when developing from WebStorm) * text reporting from karma is *very* messy * missing shiny features from other test frameworks :'(
I was wondering if anyone would have complaints about ripping out the custom framework and replacing with an existing, widely used one (e.g., Jasmine, http://jasmine.github.io/2.0/introduction.html)?
The only reason I suggest Jasmine is that I'm familiar with it's features from other projects I work on.
Key benefits of switching to Jasmine (probably apply to other frameworks too!): * already has integration for karma via the karma-jasmine plugin * individual tests can be ignored or run directly from any IDE with karma support * has a host of existing addins (e.g., better HTML reporters) which means we can easily improve the usability of the test results from the browser as well as in the qtjsruntimetests (which currently spit out raw html in the console :-/) * good documentation * support for async tests * easier fixture creation
From what I can tell, everything implemented in the existing custom test framework is supported (or supportable without much work) in jasmine without too much work.
I'm happy to do the work... just wanted to get wider approval before spending time on such a patch :-).
So… any objections?
I know and like Jasmine and have used it a few times, see e.g. [1]. It did not yet exist (at least not in my mind) when WebODF started getting unit tests. A few issues that will arise when making this move: 1) Jasmine has no closure compiler typing. An addition to the externs.js should suffice. Jasmine is mainly calls nested "function () { ... }" calls. The expect(), toBe() etc calls can be CCed. 2) Jasmine is a behaviour driven testing method. Each test results in natural language statement that is confirmed of disproved by a test. So porting is not just code porting but also coming up with nice descriptions. This requires some creative thinking and will really improve the readability of the code. 3) In some places tests are generated from xml files. I dont know if Jasmine can handle that. So all in all: I'm all for it. Jasmine is small and light enough to simply include in our repository. Cheers, Jos [1] https://github.com/vandenoever/baredom
I know and like Jasmine and have used it a few times, see e.g. [1]. It did not yet exist (at least not in my mind) when WebODF started getting unit tests. A few issues that will arise when making this move: [snip] 2) Jasmine is a behaviour driven testing method. Each test results in natural language statement that is confirmed of disproved by a test. So porting is not just code porting but also coming up with nice descriptions. This requires some creative thinking and will really improve the readability of the code. I would push to not tackle this part during the port. I’m a little worried that such a task would end up bike-shedding and delaying the landing of the patch :-). My initial idea would be simple to translate calls to the new jasmine approach, and keep the test names relatively unchanged. I am open to doing a rough fix-up though if reviewers promise to not pick at the names I concoct ;-) 3) In some places tests are generated from xml files. I dont know if Jasmine can handle that. Jasmine does this quite easily. Basically, the outer-describe closure is as per normal, and inside, each individual “it” test is specified from the source XML. It should work largely unchanged from the current version. Cheers, Philip
On 09/15/2014 10:05 AM, Philip Peitsch wrote:
A few issues that will arise when making this move:
[snip]
2) Jasmine is a behaviour driven testing method. Each test results in natural language statement that is confirmed of disproved by a test. So porting is not just code porting but also coming up with nice descriptions. This requires some creative thinking and will really improve the readability of the code.
I would push to not tackle this part during the port. I’m a little worried that such a task would end up bike-shedding and delaying the landing of the patch :-). My initial idea would be simple to translate calls to the new jasmine approach, and keep the test names relatively unchanged.
I am open to doing a rough fix-up though if reviewers promise to not pick at the names I concoct ;-)
Simply using the test names as we have them now is fine initially. After jasmine lands we can have a policy to use the behaviour driven testing method and have an effort to improve the preexisting test descriptions.
3) In some places tests are generated from xml files. I dont know if Jasmine can handle that.
Jasmine does this quite easily. Basically, the outer-describe closure is as per normal, and inside, each individual “it” test is specified from the source XML. It should work largely unchanged from the current version.
So we get it("tests.xml, function () { it("test1", function () { } it("test2", function () { } } Having a test factory that creates the tests for the elements in the xml is much easier to maintain. I dont quite get how 'each individual “it” test is specified from the source XML'. Cheers, Jos
So we get it("tests.xml, function () { it("test1", function () { } it("test2", function () { } }
Having a test factory that creates the tests for the elements in the xml is much easier to maintain. I dont quite get how 'each individual “it” test is specified from the source XML'.
With jasmine you nest the describe blocks to group sets tests[1]. So, integrating some XML-based tests could be done like: describe("Operation transform matrix tests", function () { it("constructs successfully", function() { ... }); it("cleans up all things on teardown", function() { ... }); describe("XML-based transform tests", function() { var xmlFile = runtime.readFileSync(...), tests = parseTests(xmlFile); tests.forEach(function(test) { it(test.name, test.func); }); }); });
On 09/15/2014 11:08 AM, Philip Peitsch wrote:
So we get it("tests.xml, function () { it("test1", function () { } it("test2", function () { } }
Having a test factory that creates the tests for the elements in the xml is much easier to maintain. I dont quite get how 'each individual “it” test is specified from the source XML'.
With jasmine you nest the describe blocks to group sets tests[1]. So, integrating some XML-based tests could be done like:
describe("Operation transform matrix tests", function () { it("constructs successfully", function() { ... }); it("cleans up all things on teardown", function() { ... });
describe("XML-based transform tests", function() { var xmlFile = runtime.readFileSync(...), tests = parseTests(xmlFile);
tests.forEach(function(test) { it(test.name, test.func); }); }); });
That is very nice and simple. I was not sure it() was that flexible, but it's absolutely logical to use it like that. I see no remaining roadblocks. Cheers, Jos
participants (3)
-
Jos van den Oever
-
Philip Peitsch
-
Philip Peitsch