Hi all,

So I've spent what extra time I can in the past two weeks taking a
high-level look at JS unit testing; I've looked at a few systems such as
QUnit, SproutCore/core-test, a few others that have been passed my way
(there's one that Bryan Forbes pointed me at this week that cujo is
apparently adopting but for the life of me I can't remember the name of it
right now, begins with a "b") and comparing that with the current state of
DOH. With that, I'd like to make a proposal to rewrite/refactor DOH into
something that is a top-level independent package. The basics of the
proposal follows.


Background:

DOH (The Dojo Objective Harness) has served the Dojo Toolkit well since its
inception. It includes all of the assertion code needed for any decent
unit test package, and it does things that many other unit testing
frameworks do not do, such as DOM testing and async-based tests. Creating
actual unit tests is not a difficult thing (though the registration is a
little clunky), and some other things built on top of it (such as
dojox.robot) have been somewhat ground-breaking.

Over the years, a number of dependencies on aspects of the Dojo Toolkit
have crept into the framework, such as performance analysis using Charting
(optional, currently only required if a specific "has" test is present) and
a hard dependency on dojox.testing, which has not been touched since
Wolfram added it (as far as I can tell). This particular dependency is an
attempt to emulate Python-esque "doctests", where tests can be embedded as
comments and parsed out as run.

In addition, the original interface is still in play--no updates have been
made to it since Alex originally wrote it (with his playful addition of the
Homer Simpson "doh!" to alert someone to test failures)...and the way HTML
tests are set up rely on a strict path-based HTML bootstrap (i.e.
runTests.html).


What has changed since DOH's original conception:

The primary change has been the invention and introduction of AMD, and the
ability (though not really germane here) of dojo.declare to define
anonymous objects (i.e. it returns a constructor function as opposed to
defining it based on a string).


Issues with DOH:

After analyzing (again, at a high level) DOH, it seems that there are
problems that need to be addressed:
1) The creeping in of DTK dependencies, though done with something of a
delicate touch
2) The way DOH has *everything* attached to a single object (i.e.
everything is "doh.foo")
3) Related to both 1 and 2: the lack of separation of concerns. In other
words, assertions == runners == interface.


DOH 2.0 - or "how I learned to love the unit test":
(insert Kubrick reference here ;))

What I propose is the following--

1) Assertions and base runners need to be refactored into library-agnostic
anonymous AMD modules. This will be the foundation for the testing suite,
and will probably need to include some versions of Promises and/or a highly
simplified version of dojo/on.
2) An HTML application should be written that consumes the assertion and
runner modules. This application should be written in such a way as to
both demonstrate how to write an application to consume the assertion and
runner modules, as well as promote teh sexy? that we all know DTK is
capable of.

With regards to the first point, there can be NO dependencies other than an
AMD-compliant loading mechanism. In the end, someone should be able to
take the assertion and runner modules and use them for their own purposes
with RequireJS or any other AMD loader.

On the second point...dependency away on the Dojo Toolkit! Performance
metrics with charting, manipulating the interface using dojo/query et al,
more than fine!


Alternate proposals:

Of course, when talking about a rewrite/refactor, the immediate question
comes to mind--why not just use something else that is out there? And it
is a good question. My main reasoning against it is that I have yet to see
a JS Unit Testing platform that integrates promises and AMD well--and
inevitably whatever "application" is consuming the runner is biased towards
whatever it is primarily aimed at testing. I don't see DTK as an exception
to that tendency.


What we gain:

1) A superior yet simple unit testing framework that is a stand-alone
product.
2) An application to consume that unit testing framework that can be used
as a marketing tool and as a demo application.
3) Something that does not rely on specialized servers, and/or a reliance
on a particular language other than HTML/CSS/JS (some of the unit testing
frameworks out there require things like PHP, Java, Node or Python
installations as an example).


-----

This is more of a high-level proposal than anything else, and feedback is
more than welcome on all fronts. Obviously I have not addressed DOH robot
or something such as TestSwarm, but I do think that starting small is
better than anything. In the very least, the current HTML runner needs a
lot of love--particularly in light of the idea that we are moving to
packages. Better to have a single package that can be grabbed quickly and
run any number of tests than to sprinkle packages with things like
runTests.html with a meta redirect.

Thanks for your time--
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120218/ae347f6d/attachment.htm

Search Discussions

  • Bill Keese at Feb 18, 2012 at 8:24 pm
    Thanks for looking at this. A few comments and questions inline.

    2012/2/19 Tom Trenka <ttrenka at gmail.com>
    Hi all,

    So I've spent what extra time I can in the past two weeks taking a
    high-level look at JS unit testing; I've looked at a few systems such as
    QUnit, SproutCore/core-test, a few others that have been passed my way
    (there's one that Bryan Forbes pointed me at this week that cujo is
    apparently adopting but for the life of me I can't remember the name of it
    right now, begins with a "b")
    Back in August Bryan told me that he was working on a synthetic event
    testing system using Deferreds and AMD etc. Is that what you mean? The
    only thing I can find though is https://github.com/bryanforbes/test-commonjs,
    which seems different.

    There's also selenium webdriver, but that's more of a replacement for robot
    than a replacement for DOH.


    DOH (The Dojo Objective Harness) has served the Dojo Toolkit well since
    its inception. It includes all of the assertion code needed for any decent
    unit test package, and it does things that many other unit testing
    frameworks do not do, such as DOM testing and async-based tests.
    Could you explain what you mean by DOM testing? Is it just the ability
    for DOH to run in the browser, or something more, like assertions about
    whether DOMNodes have certain classes / are visible / are in the right
    position?

    The async testing part is nice, I'm surprised that no other packages
    support that.


    hard dependency on dojox.testing, which has not been touched since Wolfram
    added it (as far as I can tell). This particular dependency is an attempt
    to emulate Python-esque "doctests", where tests can be embedded as comments
    and parsed out as run.
    Wow, I didn't even know about that. I don't think anyone is using that
    feature, and since it doesn't seem to have any internal tests, I'm not even
    sure that it works. (I also note that the AMD conversion is missing
    dependencies, and no one has complained.)


    1) Assertions and base runners need to be refactored into
    library-agnostic anonymous AMD modules. This will be the foundation for
    the testing suite, and will probably need to include some versions of
    Promises and/or a highly simplified version of dojo/on.
    Can you explain what you mean by "base runner"? I guess it means all the
    non-UI code for running tests?


    2) An HTML application should be written that consumes the assertion and
    runner modules. This application should be written in such a way as to
    both demonstrate how to write an application to consume the assertion and
    runner modules, as well as promote teh sexy? that we all know DTK is
    capable of.

    With regards to the first point, there can be NO dependencies other than
    an AMD-compliant loading mechanism. In the end, someone should be able to
    take the assertion and runner modules and use them for their own purposes
    with RequireJS or any other AMD loader.

    On the second point...dependency away on the Dojo Toolkit! Performance
    metrics with charting, manipulating the interface using dojo/query et al,
    more than fine!


    This is more of a high-level proposal than anything else, and feedback is
    more than welcome on all fronts. Obviously I have not addressed DOH robot
    or something such as TestSwarm, but I do think that starting small is
    better than anything. In the very least, the current HTML runner needs a
    lot of love--particularly in light of the idea that we are moving to
    packages. Better to have a single package that can be grabbed quickly and
    run any number of tests than to sprinkle packages with things like
    runTests.html with a meta redirect.
    What's bad about the meta-redirect? I suppose just doing a require() and
    then calling doh.run() seems cleaner, although I'm curious if there's a
    practical issue.

    As I've mentioned before, robot is the pain point for me, but I agree that
    starting small is better than nothing. (I assume above you meant
    "nothing" rather than "anything".)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120219/48a3db2b/attachment-0001.htm
  • Chris Barrett at Feb 18, 2012 at 10:28 pm
    Bill,
    Is this what you are referring to: https://github.com/bryanforbes/eventd ?
    -Chris

    2012/2/18 Bill Keese <bill at dojotoolkit.org>
    Thanks for looking at this. A few comments and questions inline.

    2012/2/19 Tom Trenka <ttrenka at gmail.com>
    Hi all,

    So I've spent what extra time I can in the past two weeks taking a
    high-level look at JS unit testing; I've looked at a few systems such as
    QUnit, SproutCore/core-test, a few others that have been passed my way
    (there's one that Bryan Forbes pointed me at this week that cujo is
    apparently adopting but for the life of me I can't remember the name of it
    right now, begins with a "b")
    Back in August Bryan told me that he was working on a synthetic event
    testing system using Deferreds and AMD etc. Is that what you mean? The
    only thing I can find though is
    https://github.com/bryanforbes/test-commonjs, which seems different.

    There's also selenium webdriver, but that's more of a replacement for
    robot than a replacement for DOH.


    DOH (The Dojo Objective Harness) has served the Dojo Toolkit well since
    its inception. It includes all of the assertion code needed for any decent
    unit test package, and it does things that many other unit testing
    frameworks do not do, such as DOM testing and async-based tests.
    Could you explain what you mean by DOM testing? Is it just the ability
    for DOH to run in the browser, or something more, like assertions about
    whether DOMNodes have certain classes / are visible / are in the right
    position?

    The async testing part is nice, I'm surprised that no other packages
    support that.


    hard dependency on dojox.testing, which has not been touched since Wolfram
    added it (as far as I can tell). This particular dependency is an attempt
    to emulate Python-esque "doctests", where tests can be embedded as comments
    and parsed out as run.
    Wow, I didn't even know about that. I don't think anyone is using that
    feature, and since it doesn't seem to have any internal tests, I'm not even
    sure that it works. (I also note that the AMD conversion is missing
    dependencies, and no one has complained.)


    1) Assertions and base runners need to be refactored into
    library-agnostic anonymous AMD modules. This will be the foundation for
    the testing suite, and will probably need to include some versions of
    Promises and/or a highly simplified version of dojo/on.
    Can you explain what you mean by "base runner"? I guess it means all the
    non-UI code for running tests?


    2) An HTML application should be written that consumes the assertion and
    runner modules. This application should be written in such a way as to
    both demonstrate how to write an application to consume the assertion and
    runner modules, as well as promote teh sexy? that we all know DTK is
    capable of.

    With regards to the first point, there can be NO dependencies other than
    an AMD-compliant loading mechanism. In the end, someone should be able to
    take the assertion and runner modules and use them for their own purposes
    with RequireJS or any other AMD loader.

    On the second point...dependency away on the Dojo Toolkit! Performance
    metrics with charting, manipulating the interface using dojo/query et al,
    more than fine!


    This is more of a high-level proposal than anything else, and feedback is
    more than welcome on all fronts. Obviously I have not addressed DOH robot
    or something such as TestSwarm, but I do think that starting small is
    better than anything. In the very least, the current HTML runner needs a
    lot of love--particularly in light of the idea that we are moving to
    packages. Better to have a single package that can be grabbed quickly and
    run any number of tests than to sprinkle packages with things like
    runTests.html with a meta redirect.
    What's bad about the meta-redirect? I suppose just doing a require() and
    then calling doh.run() seems cleaner, although I'm curious if there's a
    practical issue.

    As I've mentioned before, robot is the pain point for me, but I agree that
    starting small is better than nothing. (I assume above you meant
    "nothing" rather than "anything".)



    _______________________________________________
    dojo-contributors mailing list
    dojo-contributors at mail.dojotoolkit.org
    http://mail.dojotoolkit.org/mailman/listinfo/dojo-contributors

    --
    ______
    /l_,[____],
    l- -L-OlllllllO
    ()_)-()_)--)_)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120218/799343a7/attachment.htm
  • Bill Keese at Feb 18, 2012 at 10:40 pm
    Ah thanks, I guess so, or at least half of what I was thinking of.
  • Tom Trenka at Feb 19, 2012 at 10:34 am
    Inline on your inline...

    2012/2/18 Bill Keese <bill at dojotoolkit.org>
    So I've spent what extra time I can in the past two weeks taking a
    high-level look at JS unit testing; I've looked at a few systems such as
    QUnit, SproutCore/core-test, a few others that have been passed my way
    (there's one that Bryan Forbes pointed me at this week that cujo is
    apparently adopting but for the life of me I can't remember the name of it
    right now, begins with a "b")
    Back in August Bryan told me that he was working on a synthetic event
    testing system using Deferreds and AMD etc. Is that what you mean? The
    only thing I can find though is
    https://github.com/bryanforbes/test-commonjs, which seems different.

    There's also selenium webdriver, but that's more of a replacement for
    robot than a replacement for DOH.
    No, actually...I was referring to this: http://busterjs.org/ Ken reminded
    me about it soon after I posted this email.


    DOH (The Dojo Objective Harness) has served the Dojo Toolkit well since its
    inception. It includes all of the assertion code needed for any decent
    unit test package, and it does things that many other unit testing
    frameworks do not do, such as DOM testing and async-based tests.
    Could you explain what you mean by DOM testing? Is it just the ability
    for DOH to run in the browser, or something more, like assertions about
    whether DOMNodes have certain classes / are visible / are in the right
    position?

    The async testing part is nice, I'm surprised that no other packages
    support that.
    By DOM testing, I mean having a space in the HTML page where animations and
    what not can be done, and measurements taken afterwards...so yes, whether
    DOMNodes have certain classes / are visible / are in the right position.
    Some other unit testing frameworks do this as well, but it can be hit or
    miss.

    Some also do async testing, but the implementations seem to vary widely.

    hard dependency on dojox.testing, which has not been touched since Wolfram
    added it (as far as I can tell). This particular dependency is an attempt
    to emulate Python-esque "doctests", where tests can be embedded as comments
    and parsed out as run.
    Wow, I didn't even know about that. I don't think anyone is using that
    feature, and since it doesn't seem to have any internal tests, I'm not even
    sure that it works. (I also note that the AMD conversion is missing
    dependencies, and no one has complained.)
    Yeah, I'm not sure anyone is either. But it's in there.

    1) Assertions and base runners need to be refactored into
    library-agnostic anonymous AMD modules. This will be the foundation for
    the testing suite, and will probably need to include some versions of
    Promises and/or a highly simplified version of dojo/on.
    Can you explain what you mean by "base runner"? I guess it means all the
    non-UI code for running tests?
    OK. By "base runner", I'm talking about a pure JS module that you register
    tests with, run tests with, and log results with. What I'm not talking
    about is an HTML file that actually executes or consumes this.

    Basically, running tests would be something like:

    var results = require(["./assert", "./runner", "./base",
    "path/to/my/tests"], function(assert, runner, tests){
    // assuming tests is an array for example purposes
    tests.forEach(function(test){
    runner.register(test);
    });
    var promise = runner.run();
    return promise;
    });

    ... or something to that effect. Bear in mind that this is for example
    purposes only, and I'm only guessing at objects here. You'll note that
    this basic pattern can be run headless (i.e. command-line), and that there
    is very little dependency on anything other than the unit test framework
    itself.


    What's bad about the meta-redirect? I suppose just doing a require() and
    then calling doh.run() seems cleaner, although I'm curious if there's a
    practical issue.

    As I've mentioned before, robot is the pain point for me, but I agree that
    starting small is better than nothing. (I assume above you meant
    "nothing" rather than "anything".)
    The problem with the meta redirect is that the approach forces each and
    every project/package/subpackage to have a version of that file included in
    order to run the tests. But I'm thinking that since we have AMD and are
    capable of loading code x-domain, we could have an HTML application in
    which you could simply input a URL pointing at your tests, and run it that
    way. I haven't tried anything like that yet, but it should be possible to
    do...what that does mean though is that a single HTML app can be
    distributed for use, and we could even run unit tests right from the
    nightly output if we wanted to...but I'm really just trying to kick around
    ideas and give ourselves a unit testing platform that kicks ass? and is
    dirt simple to use.

    Cheers--
    Tom
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120219/d26fdc02/attachment.htm
  • Tom Trenka at Feb 19, 2012 at 10:40 am
    In fact...perhaps we should just adapt/adopt
    https://github.com/kriszyp/patrfor unit testing, since Kris already
    wrote most of the approach I'm talking
    about...

    -- Tom
    On Sun, Feb 19, 2012 at 9:34 AM, Tom Trenka wrote:

    Inline on your inline...

    2012/2/18 Bill Keese <bill at dojotoolkit.org>
    So I've spent what extra time I can in the past two weeks taking a
    high-level look at JS unit testing; I've looked at a few systems such as
    QUnit, SproutCore/core-test, a few others that have been passed my way
    (there's one that Bryan Forbes pointed me at this week that cujo is
    apparently adopting but for the life of me I can't remember the name of it
    right now, begins with a "b")
    Back in August Bryan told me that he was working on a synthetic event
    testing system using Deferreds and AMD etc. Is that what you mean? The
    only thing I can find though is
    https://github.com/bryanforbes/test-commonjs, which seems different.

    There's also selenium webdriver, but that's more of a replacement for
    robot than a replacement for DOH.
    No, actually...I was referring to this: http://busterjs.org/ Ken
    reminded me about it soon after I posted this email.


    DOH (The Dojo Objective Harness) has served the Dojo Toolkit well since
    its inception. It includes all of the assertion code needed for any decent
    unit test package, and it does things that many other unit testing
    frameworks do not do, such as DOM testing and async-based tests.
    Could you explain what you mean by DOM testing? Is it just the ability
    for DOH to run in the browser, or something more, like assertions about
    whether DOMNodes have certain classes / are visible / are in the right
    position?

    The async testing part is nice, I'm surprised that no other packages
    support that.
    By DOM testing, I mean having a space in the HTML page where animations
    and what not can be done, and measurements taken afterwards...so yes,
    whether DOMNodes have certain classes / are visible / are in the right
    position. Some other unit testing frameworks do this as well, but it can
    be hit or miss.

    Some also do async testing, but the implementations seem to vary widely.

    hard dependency on dojox.testing, which has not been touched since
    Wolfram added it (as far as I can tell). This particular dependency is an
    attempt to emulate Python-esque "doctests", where tests can be embedded as
    comments and parsed out as run.
    Wow, I didn't even know about that. I don't think anyone is using that
    feature, and since it doesn't seem to have any internal tests, I'm not even
    sure that it works. (I also note that the AMD conversion is missing
    dependencies, and no one has complained.)
    Yeah, I'm not sure anyone is either. But it's in there.

    1) Assertions and base runners need to be refactored into
    library-agnostic anonymous AMD modules. This will be the foundation for
    the testing suite, and will probably need to include some versions of
    Promises and/or a highly simplified version of dojo/on.
    Can you explain what you mean by "base runner"? I guess it means all
    the non-UI code for running tests?
    OK. By "base runner", I'm talking about a pure JS module that you
    register tests with, run tests with, and log results with. What I'm not
    talking about is an HTML file that actually executes or consumes this.

    Basically, running tests would be something like:

    var results = require(["./assert", "./runner", "./base",
    "path/to/my/tests"], function(assert, runner, tests){
    // assuming tests is an array for example purposes
    tests.forEach(function(test){
    runner.register(test);
    });
    var promise = runner.run();
    return promise;
    });

    ... or something to that effect. Bear in mind that this is for example
    purposes only, and I'm only guessing at objects here. You'll note that
    this basic pattern can be run headless (i.e. command-line), and that there
    is very little dependency on anything other than the unit test framework
    itself.


    What's bad about the meta-redirect? I suppose just doing a require()
    and then calling doh.run() seems cleaner, although I'm curious if there's a
    practical issue.

    As I've mentioned before, robot is the pain point for me, but I agree
    that starting small is better than nothing. (I assume above you meant
    "nothing" rather than "anything".)
    The problem with the meta redirect is that the approach forces each and
    every project/package/subpackage to have a version of that file included in
    order to run the tests. But I'm thinking that since we have AMD and are
    capable of loading code x-domain, we could have an HTML application in
    which you could simply input a URL pointing at your tests, and run it that
    way. I haven't tried anything like that yet, but it should be possible to
    do...what that does mean though is that a single HTML app can be
    distributed for use, and we could even run unit tests right from the
    nightly output if we wanted to...but I'm really just trying to kick around
    ideas and give ourselves a unit testing platform that kicks ass? and is
    dirt simple to use.

    Cheers--
    Tom
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120219/2c60a686/attachment-0001.htm
  • Ben hockey at Feb 19, 2012 at 9:39 pm

    On Feb 19, 2012, at 10:40 AM, Tom Trenka wrote:

    In fact...perhaps we should just adapt/adopt https://github.com/kriszyp/patr
    for unit testing, since Kris already wrote most of the approach I'm
    talking about...

    -- Tom
    patr is in the direction of where i think we should be approaching for
    the foundation of our unit testing. in particular, it's extremely
    simple to register (+1 for no explicit register step) and run tests.

    https://github.com/kriskowal/uncommonjs/blob/master/tests/specification.md
    gives a good specification of what you minimally need in order to do
    tests - and is roughly what patr follows. in my opinion https://github.com/Gozala/test-commonjs
    is a better (particularly - better decoupling between components)
    implementation of the same thing that patr is implementing - but there
    are still a few things that could be done differently to make test-
    commonjs even better in my opinion. feel free to press me for details
    if you're interested but the changes are minor.

    even as it is, test-commonjs addresses all the concerns you raised in
    the "Alternate proposals" section - it is asynchronous (understands
    promises) and works with AMD and an "application" could be built
    around it. i'm not necessarily saying we should adopt it but it at
    least serves as a good example - assertions, runner, logger.

    busterjs seems to be a monolith and we certainly should not be trying
    to implement something like that but i wouldn't mind if we considered
    using it - except that it seems to involve a server side component.
    one piece of it that i've been using for a while now is http://sinonjs.org/
    . test cases (ie setUp and tearDown), spies and stubs come in handy
    when writing tests. sinon's testCase isn't (or at least wasn't) aware
    of promises so i've written an adapted version that allows for
    promises to be returned from setUp, tearDown and the test (i don't
    have an official repository that tracks this adaptation but i'm using
    it here https://github.com/neonstalwart/Subject/blob/master/test/promiseTestCase.js)
    . overall, these kinds of things are not needed in the "core" of a
    unit testing framework but are optional extras that can be implemented
    as a layer that sits on top of the unit testing framework.

    i'd like to point out that we don't need an implementation of promises
    in our unit testing framework. the few lines that it takes to
    implement `when` should be sufficient for code under test that
    produces it's own promises. we can even provide an async option for
    applications that don't use promises - see how rawld handles async
    build transforms (i know bryan forbes knows what i'm talking about) by
    passing them a callback for them to return as a way to indicate that
    they intend to call that callback asynchronously. this approach would
    handle async code regardless of whether it used promises or not.
    implementing this would be fairly simple but maybe we're not ready yet
    to talk implementation details.

    overall i'm liking the general direction this conversation is taking
    (minimal testing framework + testing application, no redirects) so i
    just wanted to add my own thoughts to help shape some of the details
    and fill in some of the gaps.

    thanks,

    ben...
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120219/45682265/attachment.htm
  • Bill Keese at Feb 20, 2012 at 2:02 am
    So, both patr and test-commonjs depend on the Node library and/or the
    CommonJS library? Plus, perhaps "advanced" JS features like
    array.forEach(...)? Not sure how we deal w/that when running in the
    browser.

    2012/2/20 ben hockey <neonstalwart at gmail.com>
    On Feb 19, 2012, at 10:40 AM, Tom Trenka wrote:

    In fact...perhaps we should just adapt/adopt
    https://github.com/kriszyp/patr for unit testing, since Kris already
    wrote most of the approach I'm talking about...

    -- Tom


    patr is in the direction of where i think we should be approaching for the
    foundation of our unit testing. in particular, it's extremely simple to
    register (+1 for no explicit register step) and run tests.

    https://github.com/kriskowal/uncommonjs/blob/master/tests/specification.md gives
    a good specification of what you minimally need in order to do tests - and
    is roughly what patr follows. in my opinion
    https://github.com/Gozala/test-commonjs is a better (particularly -
    better decoupling between components) implementation of the same thing that
    patr is implementing - but there are still a few things that could be done
    differently to make test-commonjs even better in my opinion. feel free to
    press me for details if you're interested but the changes are minor.

    even as it is, test-commonjs addresses all the concerns you raised in the
    "Alternate proposals" section - it is asynchronous (understands promises)
    and works with AMD and an "application" could be built around it. i'm not
    necessarily saying we should adopt it but it at least serves as a good
    example - assertions, runner, logger.

    busterjs seems to be a monolith and we certainly should not be trying to
    implement something like that but i wouldn't mind if we considered using it
    - except that it seems to involve a server side component. one piece of it
    that i've been using for a while now is http://sinonjs.org/. test cases
    (ie setUp and tearDown), spies and stubs come in handy when writing tests.
    sinon's testCase isn't (or at least wasn't) aware of promises so i've
    written an adapted version that allows for promises to be returned from
    setUp, tearDown and the test (i don't have an official repository that
    tracks this adaptation but i'm using it here
    https://github.com/neonstalwart/Subject/blob/master/test/promiseTestCase.js).
    overall, these kinds of things are not needed in the "core" of a unit
    testing framework but are optional extras that can be implemented as a
    layer that sits on top of the unit testing framework.

    i'd like to point out that we don't need an implementation of promises in
    our unit testing framework. the few lines that it takes to implement
    `when` should be sufficient for code under test that produces it's own
    promises. we can even provide an async option for applications that don't
    use promises - see how rawld handles async build transforms (i know bryan
    forbes knows what i'm talking about) by passing them a callback for them to
    return as a way to indicate that they intend to call that callback
    asynchronously. this approach would handle async code regardless of
    whether it used promises or not. implementing this would be fairly simple
    but maybe we're not ready yet to talk implementation details.

    overall i'm liking the general direction this conversation is taking
    (minimal testing framework + testing application, no redirects) so i just
    wanted to add my own thoughts to help shape some of the details and fill in
    some of the gaps.

    thanks,

    ben...

    _______________________________________________
    dojo-contributors mailing list
    dojo-contributors at mail.dojotoolkit.org
    http://mail.dojotoolkit.org/mailman/listinfo/dojo-contributors
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120220/1a1fb2a4/attachment-0001.htm
  • Tom Trenka at Feb 20, 2012 at 9:23 am

    patr is in the direction of where i think we should be approaching for the
    foundation of our unit testing. in particular, it's extremely simple to
    register (+1 for no explicit register step) and run tests.
    (snip)

    Right. After looking over both in detail, I do think that they provide
    good guidance as to structure, but I'm not sure about directly using
    either--mostly because both seem to be aimed a little more at the
    server-side. But yes, that's the direction, I think, especially with the
    "no register" aspect.

    busterjs seems to be a monolith and we certainly should not be trying to
    implement something like that but i wouldn't mind if we considered using it
    - except that it seems to involve a server side component. one piece of it
    that i've been using for a while now is http://sinonjs.org/. test cases
    (ie setUp and tearDown), spies and stubs come in handy when writing tests.
    sinon's testCase isn't (or at least wasn't) aware of promises so i've
    written an adapted version that allows for promises to be returned from
    setUp, tearDown and the test (i don't have an official repository that
    tracks this adaptation but i'm using it here
    https://github.com/neonstalwart/Subject/blob/master/test/promiseTestCase.js).
    overall, these kinds of things are not needed in the "core" of a unit
    testing framework but are optional extras that can be implemented as a
    layer that sits on top of the unit testing framework.
    Also correct. I wouldn't recommend using busterjs; I was just holding it
    up as an example of a JS unit testing framework. While the idea of running
    a server is intriguing, it also means an install and I'd prefer to avoid
    that at all costs.

    i'd like to point out that we don't need an implementation of promises in
    our unit testing framework. the few lines that it takes to implement
    `when` should be sufficient for code under test that produces it's own
    promises. we can even provide an async option for applications that don't
    use promises - see how rawld handles async build transforms (i know bryan
    forbes knows what i'm talking about) by passing them a callback for them to
    return as a way to indicate that they intend to call that callback
    asynchronously. this approach would handle async code regardless of
    whether it used promises or not. implementing this would be fairly simple
    but maybe we're not ready yet to talk implementation details.
    I don't know that we're ready for implementation details either,
    but...right now, my feeling is that making it promise-based would
    ultimately be a Good Thing?. Understanding that we don't need a promise
    implementation is helpful, and we can certainly go the callback route via
    test signature if we wanted to. But I don't see anything wrong with, say,
    "stealing" promise.js from promised-io (or something to that effect) as a
    part of the unit test system...


    2012/2/20 Bill Keese <bill at dojotoolkit.org>
    So, both patr and test-commonjs depend on the Node library and/or the
    CommonJS library? Plus, perhaps "advanced" JS features like
    array.forEach(...)? Not sure how we deal w/that when running in the
    browser.
    Neither depend on Node or CommonJS; remember, CommonJS is more of a
    specification than an actual system. That being said...both of the
    mentioned test frameworks do make assumptions about the environment they
    are run in (as in aimed at server-side JS), so yes--they assume ES5. We'd
    need a shim, though not one as extensive as es5-shim--but that is to be
    expected, I think.

    Cheers--
    Tom
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120220/2168cc34/attachment.htm
  • Ben hockey at Feb 20, 2012 at 10:44 am

    On 2/20/2012 9:23 AM, Tom Trenka wrote:
    i'd like to point out that we don't need an implementation of
    promises in our unit testing framework. the few lines that it
    takes to implement `when` should be sufficient for code under test
    that produces it's own promises. we can even provide an async
    option for applications that don't use promises - see how rawld
    handles async build transforms (i know bryan forbes knows what i'm
    talking about) by passing them a callback for them to return as a
    way to indicate that they intend to call that callback
    asynchronously. this approach would handle async code regardless
    of whether it used promises or not. implementing this would be
    fairly simple but maybe we're not ready yet to talk implementation
    details.


    I don't know that we're ready for implementation details either,
    but...right now, my feeling is that making it promise-based would
    ultimately be a Good Thing^(TM). Understanding that we don't need a
    promise implementation is helpful, and we can certainly go the
    callback route via test signature if we wanted to. But I don't see
    anything wrong with, say, "stealing" promise.js from promised-io (or
    something to that effect) as a part of the unit test system...
    we only need a promise implementation in our unit test system if we
    intend/enforce our unit tests themselves to be generating promises -
    note that i'm making a distinction between code under test that
    generates promises and the unit test itself generating promises. i
    don't know if it's worth the weight (lines of code and maintenance) to
    provide an implementation of promises in our unit test system. the
    runner could have code like this to handle async via callback or promises:

    var callback = function (result) {
    // general logic for handling a completed test
    }

    // `when` handles tests that are asynchronous via promises
    when(currentTest(callback), function (result) {
    // if the test returns the callback
    if (result === callback) {
    // it has indicated that it is asynchronous via callback and will
    call callback when done
    }
    else {
    // handle asynchronous (via promise) or synchronous result
    callback(result);
    }
    });


    this would not preclude anyone from creating promises in their unit
    tests but they would just need to bring along their favorite promise
    implementation. by going this route we can avoid maintaining a promise
    library and just maintain the handful of lines (~5) needed to implement
    `when`.

    2012/2/20 Bill Keese <bill at dojotoolkit.org <mailto:bill at dojotoolkit.org>>

    So, both patr and test-commonjs depend on the Node library and/or
    the CommonJS library? Plus, perhaps "advanced" JS features like
    array.forEach(...)? Not sure how we deal w/that when running in
    the browser.


    Neither depend on Node or CommonJS; remember, CommonJS is more of a
    specification than an actual system. That being said...both of the
    mentioned test frameworks do make assumptions about the environment
    they are run in (as in aimed at server-side JS), so yes--they assume
    ES5. We'd need a shim, though not one as extensive as es5-shim--but
    that is to be expected, I think.
    did we ever resolve what our baseline will be for 2.0? i'm in favor of
    es5-shim as a requirement for environments which do not implement es5.
    either way, our own unit test system can be implemented without the need
    for any es5 features if necessary.

    btw, patr has a dependency on an external "assert" module and provides 2
    assertions when that module can not be provided. test-commonjs
    implements a complete assertion module - this is one of the ways in
    which test-commonjs is more complete. i'm not pushing for using either
    library in particular but just mentioned test-commonjs as a more
    complete alternative to patr - more assertions and the runner is not
    tightly coupled with the logger. i'm not opposed to using patr as a
    starting point if that's something we want to do.

    thanks,

    ben...
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120220/07eac61b/attachment-0001.htm
  • Tom Trenka at Feb 20, 2012 at 11:03 am
    2012/2/20 ben hockey <neonstalwart at gmail.com>
    we only need a promise implementation in our unit test system if we
    intend/enforce our unit tests themselves to be generating promises - note
    that i'm making a distinction between code under test that generates
    promises and the unit test itself generating promises. i don't know if
    it's worth the weight (lines of code and maintenance) to provide an
    implementation of promises in our unit test system. the runner could have
    code like this to handle async via callback or promises...
    I was actually thinking more about the runner than the assert module itself
    for promises, but I'm definitely open--and the main reason is for
    convenience more than anything else. If running the runner always returned
    a promise (regardless of whether tests were sync or not sync), it'd help
    any app consuming the runner via simplification. But like I said, I'm open.

    did we ever resolve what our baseline will be for 2.0? i'm in favor of
    es5-shim as a requirement for environments which do not implement es5.
    either way, our own unit test system can be implemented without the need
    for any es5 features if necessary...
    I don't think we've resolved the baseline or decided on es5-shim. I *do*
    think that our unit testing should include polyfills for array iteration
    methods, but I'm not sure we need anything else...for now, I'm using the
    polyfills from dochub.io but again open to other implementations.

    Not sure I was clear about this in my last email, but my current thinking
    is to use patr and test-commonjs as models but not directly fork or consume
    either one. I'm also seriously considering the idea of not one but two
    assertion objects; I don't remember where I saw it, but someone out there
    had an interesting idea in that "assert" should be aimed at truthy
    conditions, and a second object called "refute" dealt with falsy
    conditions...I like the clarity of it, even though its definitely
    non-standard.

    -- Tom
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120220/614fc256/attachment.htm
  • Ben hockey at Feb 20, 2012 at 1:47 pm

    On 2/20/2012 11:03 AM, Tom Trenka wrote:
    2012/2/20 ben hockey <neonstalwart at gmail.com
    <mailto:neonstalwart at gmail.com>>

    we only need a promise implementation in our unit test system if
    we intend/enforce our unit tests themselves to be generating
    promises - note that i'm making a distinction between code under
    test that generates promises and the unit test itself generating
    promises. i don't know if it's worth the weight (lines of code
    and maintenance) to provide an implementation of promises in our
    unit test system. the runner could have code like this to handle
    async via callback or promises...


    I was actually thinking more about the runner than the assert module
    itself for promises, but I'm definitely open--and the main reason is
    for convenience more than anything else. If running the runner always
    returned a promise (regardless of whether tests were sync or not
    sync), it'd help any app consuming the runner via simplification. But
    like I said, I'm open.
    i was talking about the runner too but i see where we may be differing.
    i'm talking about how the runner executes a single test case and it
    seems you're talking about how an app might get feedback from the runner
    while executing a test suite.

    i can see the runner as an EventEmitter being useful which i guess is
    why you mentioned dojo/on previously. multiple events being emitted
    from the runner is maybe a better option than a promise since it allows
    for more granular feedback than a single promise returned from runner.run();
    did we ever resolve what our baseline will be for 2.0? i'm in
    favor of es5-shim as a requirement for environments which do not
    implement es5. either way, our own unit test system can be
    implemented without the need for any es5 features if necessary...


    I don't think we've resolved the baseline or decided on es5-shim. I
    *do* think that our unit testing should include polyfills for array
    iteration methods, but I'm not sure we need anything else...for now,
    I'm using the polyfills from dochub.io <http://dochub.io> but again
    open to other implementations.
    Object.keys is a nice convenience too - for example
    https://github.com/Gozala/test-commonjs/blob/v0.4.4/test.js#L61-63
    returns an array of keys that represent the keys to tests in the object
    passed to the runner.
    Not sure I was clear about this in my last email, but my current
    thinking is to use patr and test-commonjs as models but not directly
    fork or consume either one. I'm also seriously considering the idea of
    not one but two assertion objects; I don't remember where I saw it,
    but someone out there had an interesting idea in that "assert" should
    be aimed at truthy conditions, and a second object called "refute"
    dealt with falsy conditions...I like the clarity of it, even though
    its definitely non-standard.
    that's a buster feature - http://busterjs.org/docs/assertions/#refutations

    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120220/025881e9/attachment.htm
  • Sasha Firsov at Feb 19, 2012 at 12:03 pm
    Not sure whether it related to test suite or to framework itself...
    In preparation to own DKT designer I found that modules/widget's
    properties and serialization capabilities are subject to automated test.
    In another words, all dijits(forgive for old terms) and some generic
    widgets( like data ) need to pass following tests:
    - Serialize and re-instantiate
    - with following public properties matching( each dijit will have
    enumeration of public properties )
    - serialization to different data stores
    - previous dojo releases with compatibility properties used

    Does it fit to Unit Test requirements itself, not 100% sure.

    Sasha
    On 2012-02-18 13:21, Tom Trenka wrote:
    Hi all,

    So I've spent what extra time I can in the past two weeks taking a
    high-level look at JS unit testing; I've looked at a few systems such
    as QUnit, SproutCore/core-test, a few others that have been passed my
    way (there's one that Bryan Forbes pointed me at this week that cujo
    is apparently adopting but for the life of me I can't remember the
    name of it right now, begins with a "b") and comparing that with the
    current state of DOH. With that, I'd like to make a proposal to
    rewrite/refactor DOH into something that is a top-level independent
    package. The basics of the proposal follows.


    Background:

    DOH (The Dojo Objective Harness) has served the Dojo Toolkit well
    since its inception. It includes all of the assertion code needed for
    any decent unit test package, and it does things that many other unit
    testing frameworks do not do, such as DOM testing and async-based
    tests. Creating actual unit tests is not a difficult thing (though
    the registration is a little clunky), and some other things built on
    top of it (such as dojox.robot) have been somewhat ground-breaking.

    Over the years, a number of dependencies on aspects of the Dojo
    Toolkit have crept into the framework, such as performance analysis
    using Charting (optional, currently only required if a specific "has"
    test is present) and a hard dependency on dojox.testing, which has not
    been touched since Wolfram added it (as far as I can tell). This
    particular dependency is an attempt to emulate Python-esque
    "doctests", where tests can be embedded as comments and parsed out as run.

    In addition, the original interface is still in play--no updates have
    been made to it since Alex originally wrote it (with his playful
    addition of the Homer Simpson "doh!" to alert someone to test
    failures)...and the way HTML tests are set up rely on a strict
    path-based HTML bootstrap (i.e. runTests.html).


    What has changed since DOH's original conception:

    The primary change has been the invention and introduction of AMD, and
    the ability (though not really germane here) of dojo.declare to define
    anonymous objects (i.e. it returns a constructor function as opposed
    to defining it based on a string).


    Issues with DOH:

    After analyzing (again, at a high level) DOH, it seems that there are
    problems that need to be addressed:
    1) The creeping in of DTK dependencies, though done with something of
    a delicate touch
    2) The way DOH has *everything* attached to a single object (i.e.
    everything is "doh.foo")
    3) Related to both 1 and 2: the lack of separation of concerns. In
    other words, assertions == runners == interface.


    DOH 2.0 - or "how I learned to love the unit test":
    (insert Kubrick reference here ;))

    What I propose is the following--

    1) Assertions and base runners need to be refactored into
    library-agnostic anonymous AMD modules. This will be the foundation
    for the testing suite, and will probably need to include some versions
    of Promises and/or a highly simplified version of dojo/on.
    2) An HTML application should be written that consumes the assertion
    and runner modules. This application should be written in such a way
    as to both demonstrate how to write an application to consume the
    assertion and runner modules, as well as promote teh sexy^(TM) that we
    all know DTK is capable of.

    With regards to the first point, there can be NO dependencies other
    than an AMD-compliant loading mechanism. In the end, someone should
    be able to take the assertion and runner modules and use them for
    their own purposes with RequireJS or any other AMD loader.

    On the second point...dependency away on the Dojo Toolkit!
    Performance metrics with charting, manipulating the interface using
    dojo/query et al, more than fine!


    Alternate proposals:

    Of course, when talking about a rewrite/refactor, the immediate
    question comes to mind--why not just use something else that is out
    there? And it is a good question. My main reasoning against it is
    that I have yet to see a JS Unit Testing platform that integrates
    promises and AMD well--and inevitably whatever "application" is
    consuming the runner is biased towards whatever it is primarily aimed
    at testing. I don't see DTK as an exception to that tendency.


    What we gain:

    1) A superior yet simple unit testing framework that is a stand-alone
    product.
    2) An application to consume that unit testing framework that can be
    used as a marketing tool and as a demo application.
    3) Something that does not rely on specialized servers, and/or a
    reliance on a particular language other than HTML/CSS/JS (some of the
    unit testing frameworks out there require things like PHP, Java, Node
    or Python installations as an example).


    -----

    This is more of a high-level proposal than anything else, and feedback
    is more than welcome on all fronts. Obviously I have not addressed
    DOH robot or something such as TestSwarm, but I do think that starting
    small is better than anything. In the very least, the current HTML
    runner needs a lot of love--particularly in light of the idea that we
    are moving to packages. Better to have a single package that can be
    grabbed quickly and run any number of tests than to sprinkle packages
    with things like runTests.html with a meta redirect.

    Thanks for your time--
    Tom



    _______________________________________________
    dojo-contributors mailing list
    dojo-contributors at mail.dojotoolkit.org
    http://mail.dojotoolkit.org/mailman/listinfo/dojo-contributors
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20120219/fae6d631/attachment.htm

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdojo-contributors @
categoriesdojo
postedFeb 18, '12 at 4:21p
activeFeb 20, '12 at 1:47p
posts13
users5
websitedojotoolkit.org

People

Translate

site design / logo © 2021 Grokbase