FAQ
Hello, All:


The fda package has tests that run too long for CRAN's current
rules. I'd like to wrap some examples in a construct like the following:


if(!CRAN()){
...
}


I tried the following:


CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
x. <- Sys.getenv(x)
xl <- as.logical(x.)
notCRAN <- is.na(xl) || xl
#
return(!notCRAN)
}


The companion help page included the following example:


if(CRAN()){
stop('CRAN')
} else {
stop('NOT CRAN')
}


This reported "NOT CRAN" even with "R CMD check --as-cran".


Suggestions?
Thanks,
Spencer
sessionInfo()
R version 2.15.1 (2012-06-22)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] sos_1.3-5 brew_1.0-6

loaded via a namespace (and not attached):
[1] tools_2.15.1

Search Discussions

  • Henrik Bengtsson at Sep 3, 2012 at 7:49 am
    Hi, see thread "[Rd] Proposal: Mechanism for controlling the amount of
    testing 'R CMD check' performs" on April 8, 2012:

    https://stat.ethz.ch/pipermail/r-devel/2012-April/063809.html

    /Henrik


    On Sun, Sep 2, 2012 at 10:43 PM, Spencer Graves
    wrote:
    Hello, All:


    The fda package has tests that run too long for CRAN's current rules.
    I'd like to wrap some examples in a construct like the following:


    if(!CRAN()){
    ...
    }


    I tried the following:


    CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
    x. <- Sys.getenv(x)
    xl <- as.logical(x.)
    notCRAN <- is.na(xl) || xl
    #
    return(!notCRAN)
    }


    The companion help page included the following example:


    if(CRAN()){
    stop('CRAN')
    } else {
    stop('NOT CRAN')
    }


    This reported "NOT CRAN" even with "R CMD check --as-cran".


    Suggestions?
    Thanks,
    Spencer
    sessionInfo()
    R version 2.15.1 (2012-06-22)
    Platform: i386-pc-mingw32/i386 (32-bit)

    locale:
    [1] LC_COLLATE=English_United States.1252
    [2] LC_CTYPE=English_United States.1252
    [3] LC_MONETARY=English_United States.1252
    [4] LC_NUMERIC=C
    [5] LC_TIME=English_United States.1252

    attached base packages:
    [1] stats graphics grDevices utils datasets methods base

    other attached packages:
    [1] sos_1.3-5 brew_1.0-6

    loaded via a namespace (and not attached):
    [1] tools_2.15.1

    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Dirk Eddelbuettel at Sep 3, 2012 at 2:34 pm

    On 3 September 2012 at 00:49, Henrik Bengtsson wrote:
    Hi, see thread "[Rd] Proposal: Mechanism for controlling the amount of
    testing 'R CMD check' performs" on April 8, 2012:

    https://stat.ethz.ch/pipermail/r-devel/2012-April/063809.html
    Good proposal, somehow I missed that at the time. Something like this ought
    be to implemented in R proper.

    In Rcpp, I am now using a similar environment-variable-based approach, and it
    doesn't scale particularly well if every (large) package does its own thing.

    Dirk
    /Henrik
    --
    Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
  • Martin Maechler at Sep 3, 2012 at 3:18 pm

    Dirk Eddelbuettel <edd@debian.org>
    on Mon, 3 Sep 2012 09:34:06 -0500 writes:
    On 3 September 2012 at 00:49, Henrik Bengtsson wrote:
    Hi, see thread "[Rd] Proposal: Mechanism for controlling the amount of
    testing 'R CMD check' performs" on April 8, 2012:

    https://stat.ethz.ch/pipermail/r-devel/2012-April/063809.html
    Good proposal, somehow I missed that at the time.
    Something like this ought be to implemented in R proper.
    I agree.... but the CRAN maintainers have to agree too.

    In Rcpp, I am now using a similar
    environment-variable-based approach, and it doesn't scale
    particularly well if every (large) package does its own thing.
    yes, my packages do their own thing too, now, e.g., Matrix
    (not yet released):

    doExtras <- interactive() || nzchar(Sys.getenv("R_MATRIX_CHECK_EXTRA")) ||
    identical("true", unname(Sys.getenv("R_PKG_CHECKING_doExtras")))

    and then I use if(doExtras) { .... }
    but I agree with Henrik that it would make sense to have more
    than just two levels for the amount of testing, not just for CRAN.

    An additional, slightly more interesting feature request that
    you may also want in these cases, is the following:
    Assume, that we have a function testingLevel()
    and the following code should only be run, if the testing level
    is 2 or higher

    if(testingLevel() > 2) {
    .......1
    .......2
    ........
    ........
    .......n
    }

    Now what we all really wanted was that the ........ code
    ran just as in toplevel.
    While I think that's almost impossible,
    as the .....1 to .....n must first be parsed all, and then
    evaluated (which is already different from top level),
    I would at least want that the auto-printing worked like in
    toplevel, so I don't have to write a print(.) around every
    ......i that would autoprint if in toplevel, and of course does
    *not* autoprint inside if(.) { .. }.

    Martin Maechler, ETH Zurich
    Dirk
    /Henrik
    --
    Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
  • Duncan Murdoch at Sep 3, 2012 at 5:41 pm

    On 12-09-03 11:18 AM, Martin Maechler wrote:
    Dirk Eddelbuettel <edd@debian.org>
    on Mon, 3 Sep 2012 09:34:06 -0500 writes:
    On 3 September 2012 at 00:49, Henrik Bengtsson wrote:
    Hi, see thread "[Rd] Proposal: Mechanism for controlling the amount of
    testing 'R CMD check' performs" on April 8, 2012:

    https://stat.ethz.ch/pipermail/r-devel/2012-April/063809.html
    Good proposal, somehow I missed that at the time.
    Something like this ought be to implemented in R proper.
    I agree.... but the CRAN maintainers have to agree too.

    In Rcpp, I am now using a similar
    environment-variable-based approach, and it doesn't scale
    particularly well if every (large) package does its own thing.
    yes, my packages do their own thing too, now, e.g., Matrix
    (not yet released):

    doExtras <- interactive() || nzchar(Sys.getenv("R_MATRIX_CHECK_EXTRA")) ||
    identical("true", unname(Sys.getenv("R_PKG_CHECKING_doExtras")))

    and then I use if(doExtras) { .... }
    but I agree with Henrik that it would make sense to have more
    than just two levels for the amount of testing, not just for CRAN.

    An additional, slightly more interesting feature request that
    you may also want in these cases, is the following:
    Assume, that we have a function testingLevel()
    and the following code should only be run, if the testing level
    is 2 or higher

    if(testingLevel() > 2) {
    .......1
    .......2
    ........
    ........
    .......n
    }

    Now what we all really wanted was that the ........ code
    ran just as in toplevel.
    While I think that's almost impossible,
    as the .....1 to .....n must first be parsed all, and then
    evaluated (which is already different from top level),
    I would at least want that the auto-printing worked like in
    toplevel, so I don't have to write a print(.) around every
    ......i that would autoprint if in toplevel, and of course does
    *not* autoprint inside if(.) { .. }.
    I prefer the current scheme where individual tests can be turned on and
    off. The space of tests is a 2^N factorial space, not a chain of n
    possible levels.

    The only problem I see with the current scheme is that code being tested
    can't find out which tests are being run. In some cases, that's fine:
    there's no point in example code knowing about tests for consistency
    between documentation and implementation. In other cases it might even
    be a good thing: do you want packages to be able to hide dependencies
    on non-CRAN packages, for example?

    Duncan Murdoch
  • Hadley Wickham at Sep 4, 2012 at 2:59 am

    Hi, see thread "[Rd] Proposal: Mechanism for controlling the amount of
    testing 'R CMD check' performs" on April 8, 2012:

    https://stat.ethz.ch/pipermail/r-devel/2012-April/063809.html
    Good proposal, somehow I missed that at the time. Something like this ought
    be to implemented in R proper.

    In Rcpp, I am now using a similar environment-variable-based approach, and it
    doesn't scale particularly well if every (large) package does its own thing.
    Is there any approach that's going to be particularly better than
    scaling out CRAN's build servers? As a particularly profuse package
    author, I'd be happy to pay for the computing time that it takes to
    run a full set of tests on my package (and this seems feasible in the
    era of on-demand cloud computing).

    Hadley

    --
    Assistant Professor
    Department of Statistics / Rice University
    http://had.co.nz/
  • Spencer Graves at Sep 4, 2012 at 4:58 am

    On 9/3/2012 7:59 PM, Hadley Wickham wrote:
    Hi, see thread "[Rd] Proposal: Mechanism for controlling the amount of
    testing 'R CMD check' performs" on April 8, 2012:

    https://stat.ethz.ch/pipermail/r-devel/2012-April/063809.html
    Good proposal, somehow I missed that at the time. Something like this ought
    be to implemented in R proper.

    In Rcpp, I am now using a similar environment-variable-based approach, and it
    doesn't scale particularly well if every (large) package does its own thing.
    Is there any approach that's going to be particularly better than
    scaling out CRAN's build servers? As a particularly profuse package
    author, I'd be happy to pay for the computing time that it takes to
    run a full set of tests on my package (and this seems feasible in the
    era of on-demand cloud computing).

    Hadley
    Jim Ramsay suggested we have CRAN generate a pro-forma invoice
    when a package is first submitted to CRAN and on each anniversary
    thereafter that would be like page charges for journals: If you work
    for an organization that knows how to pay invoices, you submit the
    invoice to that system. Ramsay said he has grant money and would be
    happy to pay a reasonable fee, but he needs an invoice. If you aren't
    in such an organization, you are free to plead poverty and pay anything
    or nothing. I think we should do the same for R-Forge.


    I think the fee should be set high enough that the money
    generated would be enough to pay for more hardware AND for someone to
    maintain the system. The entirety of humanity will benefit if Ripley,
    Hornik, Ligges, Theussl, Zeileis and others spend more time developing
    better statistical algorithms and less struggling with computer hardware
    and operating systems just to keep CRAN and R-Forge functioning. These
    folks and others have done a great job in bringing R to where it is
    today. There are now over 4,000 packages on CRAN. If 25% of those pay,
    say, 200 (Euros or dollars) per year, for example, that should be enough
    to hire a competent sys admin or two with an adequate hardware budget to
    keep it all running, I think.


    Thanks, Hadley.
    Spencer


    --
    Spencer Graves, PE, PhD
    President and Chief Technology Officer
    Structure Inspection and Monitoring, Inc.
    751 Emerson Ct.
    San Jos?, CA 95126
    ph: 408-655-4567
    web: www.structuremonitoring.com
  • Duncan Murdoch at Sep 3, 2012 at 11:41 am

    On 12-09-03 1:43 AM, Spencer Graves wrote:
    Hello, All:


    The fda package has tests that run too long for CRAN's current
    rules. I'd like to wrap some examples in a construct like the following:


    if(!CRAN()){
    ...
    }


    I tried the following:


    CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
    x. <- Sys.getenv(x)
    xl <- as.logical(x.)
    notCRAN <- is.na(xl) || xl
    #
    return(!notCRAN)
    }


    The companion help page included the following example:


    if(CRAN()){
    stop('CRAN')
    } else {
    stop('NOT CRAN')
    }


    This reported "NOT CRAN" even with "R CMD check --as-cran".
    There's no user-visible setting for --as-cran, because it just sets a
    number of other options. You could query one of those. The settings
    that are visible are

    _R_CHECK_TIMINGS_ (which seems most relevant to you)
    _R_CHECK_INSTALL_DEPENDS_
    _R_CHECK_NO_RECOMMENDED_
    _R_SHLIB_BUILD_OBJECTS_SYMBOL_TABLES_

    So I'd check the value in _R_CHECK_TIMINGS_, or maybe just its existence.

    Duncan Murdoch
  • Spencer Graves at Sep 3, 2012 at 11:25 pm
    Hi, Duncan et al.:


    I modified my CRAN() function (see below) to look for
    "_R_CHECK_TIMINGS_" and "R_CHECK_TIMINGS", but got "NOT CRAN" with "R
    CMD check --as-cran" from both.


    Suggestions?
    Thanks,
    Spencer
    p.s. This is fda available from R-Forge via "svn checkout
    svn://svn.r-forge.r-project.org/svnroot/fda/". It currently is
    programmed to stop no matter what. I want it to give an error "CRAN"
    with "R CMD check --as-cran" and "NOT CRAN" without "--as-cran".
    Currently, I get "NOT CRAN" for both.

    On 9/3/2012 4:41 AM, Duncan Murdoch wrote:
    On 12-09-03 1:43 AM, Spencer Graves wrote:
    Hello, All:


    The fda package has tests that run too long for CRAN's current
    rules. I'd like to wrap some examples in a construct like the
    following:


    if(!CRAN()){
    ...
    }


    I tried the following:


    CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
    x. <- Sys.getenv(x)
    xl <- as.logical(x.)
    notCRAN <- is.na(xl) || xl
    #
    return(!notCRAN)
    }


    The companion help page included the following example:


    if(CRAN()){
    stop('CRAN')
    } else {
    stop('NOT CRAN')
    }


    This reported "NOT CRAN" even with "R CMD check --as-cran".
    There's no user-visible setting for --as-cran, because it just sets a
    number of other options. You could query one of those. The settings
    that are visible are

    _R_CHECK_TIMINGS_ (which seems most relevant to you)
    _R_CHECK_INSTALL_DEPENDS_
    _R_CHECK_NO_RECOMMENDED_
    _R_SHLIB_BUILD_OBJECTS_SYMBOL_TABLES_

    So I'd check the value in _R_CHECK_TIMINGS_, or maybe just its existence.

    Duncan Murdoch
  • Duncan Murdoch at Sep 3, 2012 at 11:52 pm

    On 12-09-03 7:25 PM, Spencer Graves wrote:
    Hi, Duncan et al.:


    I modified my CRAN() function (see below) to look for
    "_R_CHECK_TIMINGS_" and "R_CHECK_TIMINGS", but got "NOT CRAN" with "R
    CMD check --as-cran" from both.


    Suggestions?
    Thanks,
    Spencer
    p.s. This is fda available from R-Forge via "svn checkout
    svn://svn.r-forge.r-project.org/svnroot/fda/". It currently is
    programmed to stop no matter what. I want it to give an error "CRAN"
    with "R CMD check --as-cran" and "NOT CRAN" without "--as-cran".
    Currently, I get "NOT CRAN" for both.
    The problem is with your test. If I put print(names(Sys.getenv())) into
    an example, I see _R_CHECK_TIMINGS_ if and only if I do the check with
    --as-cran. The value is supposed to be a number, not a logical.

    Duncan Murdoch

    On 9/3/2012 4:41 AM, Duncan Murdoch wrote:
    On 12-09-03 1:43 AM, Spencer Graves wrote:
    Hello, All:


    The fda package has tests that run too long for CRAN's current
    rules. I'd like to wrap some examples in a construct like the
    following:


    if(!CRAN()){
    ...
    }


    I tried the following:


    CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
    x. <- Sys.getenv(x)
    xl <- as.logical(x.)
    notCRAN <- is.na(xl) || xl
    #
    return(!notCRAN)
    }


    The companion help page included the following example:


    if(CRAN()){
    stop('CRAN')
    } else {
    stop('NOT CRAN')
    }


    This reported "NOT CRAN" even with "R CMD check --as-cran".
    There's no user-visible setting for --as-cran, because it just sets a
    number of other options. You could query one of those. The settings
    that are visible are

    _R_CHECK_TIMINGS_ (which seems most relevant to you)
    _R_CHECK_INSTALL_DEPENDS_
    _R_CHECK_NO_RECOMMENDED_
    _R_SHLIB_BUILD_OBJECTS_SYMBOL_TABLES_

    So I'd check the value in _R_CHECK_TIMINGS_, or maybe just its existence.

    Duncan Murdoch
  • Spencer Graves at Sep 4, 2012 at 1:39 am
    Hi, Duncan:


    Duh... Thanks. I should have been able to figure that out for
    myself. Your help produced a solution much quicker.


    Thank again.
    Spencer

    On 9/3/2012 4:52 PM, Duncan Murdoch wrote:
    On 12-09-03 7:25 PM, Spencer Graves wrote:
    Hi, Duncan et al.:


    I modified my CRAN() function (see below) to look for
    "_R_CHECK_TIMINGS_" and "R_CHECK_TIMINGS", but got "NOT CRAN" with "R
    CMD check --as-cran" from both.


    Suggestions?
    Thanks,
    Spencer
    p.s. This is fda available from R-Forge via "svn checkout
    svn://svn.r-forge.r-project.org/svnroot/fda/". It currently is
    programmed to stop no matter what. I want it to give an error "CRAN"
    with "R CMD check --as-cran" and "NOT CRAN" without "--as-cran".
    Currently, I get "NOT CRAN" for both.
    The problem is with your test. If I put print(names(Sys.getenv()))
    into an example, I see _R_CHECK_TIMINGS_ if and only if I do the check
    with --as-cran. The value is supposed to be a number, not a logical.

    Duncan Murdoch

    On 9/3/2012 4:41 AM, Duncan Murdoch wrote:
    On 12-09-03 1:43 AM, Spencer Graves wrote:
    Hello, All:


    The fda package has tests that run too long for CRAN's
    current
    rules. I'd like to wrap some examples in a construct like the
    following:


    if(!CRAN()){
    ...
    }


    I tried the following:


    CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
    x. <- Sys.getenv(x)
    xl <- as.logical(x.)
    notCRAN <- is.na(xl) || xl
    #
    return(!notCRAN)
    }


    The companion help page included the following example:


    if(CRAN()){
    stop('CRAN')
    } else {
    stop('NOT CRAN')
    }


    This reported "NOT CRAN" even with "R CMD check --as-cran".
    There's no user-visible setting for --as-cran, because it just sets a
    number of other options. You could query one of those. The settings
    that are visible are

    _R_CHECK_TIMINGS_ (which seems most relevant to you)
    _R_CHECK_INSTALL_DEPENDS_
    _R_CHECK_NO_RECOMMENDED_
    _R_SHLIB_BUILD_OBJECTS_SYMBOL_TABLES_

    So I'd check the value in _R_CHECK_TIMINGS_, or maybe just its
    existence.

    Duncan Murdoch

    --
    Spencer Graves, PE, PhD
    President and Chief Technology Officer
    Structure Inspection and Monitoring, Inc.
    751 Emerson Ct.
    San Jos?, CA 95126
    ph: 408-655-4567
    web: www.structuremonitoring.com
  • Martin Maechler at Sep 4, 2012 at 8:45 am

    Spencer Graves <spencer.graves@prodsyse.com>
    on Mon, 3 Sep 2012 18:39:47 -0700 writes:
    Hi, Duncan:
    Duh... Thanks. I should have been able to figure that out for
    myself. Your help produced a solution much quicker.
    Thank again.
    Spencer
    On 9/3/2012 4:52 PM, Duncan Murdoch wrote:
    On 12-09-03 7:25 PM, Spencer Graves wrote:
    Hi, Duncan et al.:
    >>>
    >>>
    I modified my CRAN() function (see below) to look for
    "_R_CHECK_TIMINGS_" and "R_CHECK_TIMINGS", but got "NOT CRAN" with "R
    CMD check --as-cran" from both.
    >>>
    >>>
    Suggestions?
    Thanks,
    Spencer
    p.s. This is fda available from R-Forge via "svn checkout
    svn://svn.r-forge.r-project.org/svnroot/fda/". It currently is
    programmed to stop no matter what. I want it to give an error "CRAN"
    with "R CMD check --as-cran" and "NOT CRAN" without "--as-cran".
    Currently, I get "NOT CRAN" for both.
    >>
    The problem is with your test. If I put print(names(Sys.getenv()))
    into an example, I see _R_CHECK_TIMINGS_ if and only if I do the check
    with --as-cran. The value is supposed to be a number, not a logical.
    >>
    Duncan Murdoch
    But back to the original question.
    Checking for these variables is not at all a general solution;
    If I use 'R CMD check --timings' I want to time my tests and
    actually I find this so reasonable that I've made it the default
    for me.

    I strongly believe we should follow the thread Henrik has
    started and Dirk and I had followed up.

    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    On 9/3/2012 4:41 AM, Duncan Murdoch wrote:
    On 12-09-03 1:43 AM, Spencer Graves wrote:
    Hello, All:
    >>>>>
    >>>>>
    The fda package has tests that run too long for CRAN's
    current
    rules. I'd like to wrap some examples in a construct like the
    following:
    >>>>>
    >>>>>
    if(!CRAN()){
    ...
    }
    >>>>>
    >>>>>
    I tried the following:
    >>>>>
    >>>>>
    CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
    x. <- Sys.getenv(x)
    xl <- as.logical(x.)
    notCRAN <- is.na(xl) || xl
    #
    return(!notCRAN)
    }
    >>>>>
    >>>>>
    The companion help page included the following example:
    >>>>>
    >>>>>
    if(CRAN()){
    stop('CRAN')
    } else {
    stop('NOT CRAN')
    }
    >>>>>
    >>>>>
    This reported "NOT CRAN" even with "R CMD check --as-cran".
    >>>>
    There's no user-visible setting for --as-cran, because it just sets a
    number of other options. You could query one of those. The settings
    that are visible are
    >>>>
    _R_CHECK_TIMINGS_ (which seems most relevant to you)
    _R_CHECK_INSTALL_DEPENDS_
    _R_CHECK_NO_RECOMMENDED_
    _R_SHLIB_BUILD_OBJECTS_SYMBOL_TABLES_
    >>>>
    So I'd check the value in _R_CHECK_TIMINGS_, or maybe just its
    existence.
    >>>>
    Duncan Murdoch
    >>>>
    >>>
    >>

    --
    Spencer Graves, PE, PhD
    President and Chief Technology Officer
    Structure Inspection and Monitoring, Inc.
    751 Emerson Ct.
    San Jos?, CA 95126
    ph: 408-655-4567
    web: www.structuremonitoring.com
    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Duncan Murdoch at Sep 4, 2012 at 11:16 am

    On 12-09-04 4:45 AM, Martin Maechler wrote:
    Spencer Graves <spencer.graves@prodsyse.com>
    on Mon, 3 Sep 2012 18:39:47 -0700 writes:
    Hi, Duncan:
    Duh... Thanks. I should have been able to figure that out for
    myself. Your help produced a solution much quicker.
    Thank again.
    Spencer
    On 9/3/2012 4:52 PM, Duncan Murdoch wrote:
    On 12-09-03 7:25 PM, Spencer Graves wrote:
    Hi, Duncan et al.:


    I modified my CRAN() function (see below) to look for
    "_R_CHECK_TIMINGS_" and "R_CHECK_TIMINGS", but got "NOT CRAN" with "R
    CMD check --as-cran" from both.


    Suggestions?
    Thanks,
    Spencer
    p.s. This is fda available from R-Forge via "svn checkout
    svn://svn.r-forge.r-project.org/svnroot/fda/". It currently is
    programmed to stop no matter what. I want it to give an error "CRAN"
    with "R CMD check --as-cran" and "NOT CRAN" without "--as-cran".
    Currently, I get "NOT CRAN" for both.
    The problem is with your test. If I put print(names(Sys.getenv()))
    into an example, I see _R_CHECK_TIMINGS_ if and only if I do the check
    with --as-cran. The value is supposed to be a number, not a logical.

    Duncan Murdoch
    But back to the original question.
    Checking for these variables is not at all a general solution;
    If I use 'R CMD check --timings' I want to time my tests and
    actually I find this so reasonable that I've made it the default
    for me.

    I strongly believe we should follow the thread Henrik has
    started and Dirk and I had followed up. As did I.
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.
    I don't agree that it is good to think of it that way, as I wrote
    before. Tests don't have a natural ordering. I do agree that example
    (or test) code should be able to determine if certain tests have been
    enabled. I don't know if there are any such tests where it isn't
    currently possible.

    Duncan Murdoch
    Martin
    On 9/3/2012 4:41 AM, Duncan Murdoch wrote:
    On 12-09-03 1:43 AM, Spencer Graves wrote:
    Hello, All:


    The fda package has tests that run too long for CRAN's
    current
    rules. I'd like to wrap some examples in a construct like the
    following:


    if(!CRAN()){
    ...
    }


    I tried the following:


    CRAN <- function(x='_R_CHECK_CRAN_INCOMING_'){
    x. <- Sys.getenv(x)
    xl <- as.logical(x.)
    notCRAN <- is.na(xl) || xl
    #
    return(!notCRAN)
    }


    The companion help page included the following example:


    if(CRAN()){
    stop('CRAN')
    } else {
    stop('NOT CRAN')
    }


    This reported "NOT CRAN" even with "R CMD check --as-cran".
    There's no user-visible setting for --as-cran, because it just sets a
    number of other options. You could query one of those. The settings
    that are visible are

    _R_CHECK_TIMINGS_ (which seems most relevant to you)
    _R_CHECK_INSTALL_DEPENDS_
    _R_CHECK_NO_RECOMMENDED_
    _R_SHLIB_BUILD_OBJECTS_SYMBOL_TABLES_

    So I'd check the value in _R_CHECK_TIMINGS_, or maybe just its
    existence.

    Duncan Murdoch
    --
    Spencer Graves, PE, PhD
    President and Chief Technology Officer
    Structure Inspection and Monitoring, Inc.
    751 Emerson Ct.
    San Jos?, CA 95126
    ph: 408-655-4567
    web: www.structuremonitoring.com
    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Terry Therneau at Sep 4, 2012 at 12:20 pm

    On 09/04/2012 05:00 AM, r-devel-request at r-project.org wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would be the best.
    Then CRAN can choose to set that variable to "3" say, with level 1 for extensive and 2 for
    usual.
    I'm quite willing to put up with the nuisance of print() enclosures. I prefer it to
    having yet another way to subvert the evaluation model.

    I'm a believer in testing everything possible in my packages, and wear it it as a badge
    of honor that the survival package has 4 lines of R code in the tests directory for every
    3 in the R directory. But CRAN only needs to run a small subset of this.

    Terry T
  • Duncan Murdoch at Sep 4, 2012 at 12:38 pm

    On 04/09/2012 8:20 AM, Terry Therneau wrote:
    On 09/04/2012 05:00 AM, r-devel-request at r-project.org wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would be the best.
    Then CRAN can choose to set that variable to "3" say, with level 1 for extensive and 2 for
    usual.
    I'm quite willing to put up with the nuisance of print() enclosures. I prefer it to
    having yet another way to subvert the evaluation model.

    I'm a believer in testing everything possible in my packages, and wear it it as a badge
    of honor that the survival package has 4 lines of R code in the tests directory for every
    3 in the R directory. But CRAN only needs to run a small subset of this.
    We have a mechanism to specify testing level: the --as-cran flag. We
    could presumably make it more elaborate by adding other flags, or option
    levels, or whatever.

    What I think we shouldn't do is try to create an R-level test that says

    if (testingLevel() > 3) {
    doSomething
    }

    because tests can be turned on and off, individually. If testingLevel 3
    specified tests (A, B, C), then is our testingLevel higher if we are
    running tests (A, B, D, E, F, G)? Why not just test for the presence of
    whichever test is most relevant to that particular code block, e.g.

    if ("D" %in% tests()) {
    doSomething
    }

    Duncan Murdoch
  • Warnes, Gregory at Sep 4, 2012 at 6:36 pm
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:
    On 09/04/2012 05:00 AM, r-devel-request at r-project.org wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would be the
    best.
    Then CRAN can choose to set that variable to "3" say, with level 1 for
    extensive and 2 for
    usual.
    I'm quite willing to put up with the nuisance of print()
    enclosures. I prefer it to
    having yet another way to subvert the evaluation model.

    I'm a believer in testing everything possible in my packages, and
    wear it it as a badge
    of honor that the survival package has 4 lines of R code in the tests
    directory for every
    3 in the R directory. But CRAN only needs to run a small subset of
    this.
    We have a mechanism to specify testing level: the --as-cran flag. We
    could presumably make it more elaborate by adding other flags, or option
    levels, or whatever.

    What I think we shouldn't do is try to create an R-level test that says

    if (testingLevel() > 3) {
    doSomething
    }

    because tests can be turned on and off, individually. If testingLevel 3
    specified tests (A, B, C), then is our testingLevel higher if we are
    running tests (A, B, D, E, F, G)? Why not just test for the presence of
    whichever test is most relevant to that particular code block, e.g.

    if ("D" %in% tests()) {
    doSomething
    }

    I would prefer the testingLevel() approach of the "D" %in% tests()
    approach, since testingLevel() provides a natural way to add successively
    greater test details without having to dig into the code to determine the
    set of tests.

    -Greg
  • Duncan Murdoch at Sep 4, 2012 at 6:57 pm

    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:
    On 09/04/2012 05:00 AM, r-devel-request at r-project.org wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would be the
    best.
    Then CRAN can choose to set that variable to "3" say, with level 1 for
    extensive and 2 for
    usual.
    I'm quite willing to put up with the nuisance of print()
    enclosures. I prefer it to
    having yet another way to subvert the evaluation model.

    I'm a believer in testing everything possible in my packages, and
    wear it it as a badge
    of honor that the survival package has 4 lines of R code in the tests
    directory for every
    3 in the R directory. But CRAN only needs to run a small subset of
    this.
    We have a mechanism to specify testing level: the --as-cran flag. We
    could presumably make it more elaborate by adding other flags, or option
    levels, or whatever.

    What I think we shouldn't do is try to create an R-level test that says

    if (testingLevel() > 3) {
    doSomething
    }

    because tests can be turned on and off, individually. If testingLevel 3
    specified tests (A, B, C), then is our testingLevel higher if we are
    running tests (A, B, D, E, F, G)? Why not just test for the presence of
    whichever test is most relevant to that particular code block, e.g.

    if ("D" %in% tests()) {
    doSomething
    }

    I would prefer the testingLevel() approach of the "D" %in% tests()
    approach, since testingLevel() provides a natural way to add successively
    greater test details without having to dig into the code to determine the
    set of tests.
    I don't see how you could possibly calculate a single number in a
    reasonable way. What is the number that should be returned for (A, B,
    D, E, F, G)?

    Duncan Murdoch
  • Terry Therneau at Sep 4, 2012 at 7:44 pm

    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:
    On 09/04/2012 05:00 AM, r-devel-request at r-project.org wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would be the
    best.
    Then CRAN can choose to set that variable to "3" say, with level 1 for
    extensive and 2 for
    usual.
    I'm quite willing to put up with the nuisance of print()
    enclosures. I prefer it to
    having yet another way to subvert the evaluation model.

    I'm a believer in testing everything possible in my packages, and
    wear it it as a badge
    of honor that the survival package has 4 lines of R code in the tests
    directory for every
    3 in the R directory. But CRAN only needs to run a small subset of
    this.
    We have a mechanism to specify testing level: the --as-cran flag. We
    could presumably make it more elaborate by adding other flags, or option
    levels, or whatever.

    What I think we shouldn't do is try to create an R-level test that says

    if (testingLevel() > 3) {
    doSomething
    }

    because tests can be turned on and off, individually. If testingLevel 3
    specified tests (A, B, C), then is our testingLevel higher if we are
    running tests (A, B, D, E, F, G)? Why not just test for the presence of
    whichever test is most relevant to that particular code block, e.g.

    if ("D" %in% tests()) {
    doSomething
    }

    I would prefer the testingLevel() approach of the "D" %in% tests()
    approach, since testingLevel() provides a natural way to add successively
    greater test details without having to dig into the code to determine the
    set of tests.
    I don't see how you could possibly calculate a single number in a reasonable way. What is
    the number that should be returned for (A, B, D, E, F, G)?

    Duncan Murdoch
    Duncan is leapfrogging ahead to another level that I hadn't thought of. An example would
    be to divide my survival package as "cox", "parametric", "survfit", "all", some of whaich
    overlap. Interesting idea, but beyond what I'd use. When I'm focused on a detail I run
    the test of interest directly, not through CMD check. For me low, med, high intensity
    suffices with as-cran invoking the first level and high including those that take an
    exceptionally long time.
    If you went to an A, B, C, ... approach what would be the as-cran default?

    Of course there is then the furthest level, which I recently instituted for survival due
    to the large number of dependencies: before posting a change download all the other
    dependent packages and run their tests too. It's an overnighter, and in that case I'd
    want level=high. Forewarned is forearmed.

    Terry T.
  • Duncan Murdoch at Sep 4, 2012 at 7:58 pm

    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:
    On 09/04/2012 05:00 AM, r-devel-request at r-project.org wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would be the
    best.
    Then CRAN can choose to set that variable to "3" say, with level 1 for
    extensive and 2 for
    usual.
    I'm quite willing to put up with the nuisance of print()
    enclosures. I prefer it to
    having yet another way to subvert the evaluation model.

    I'm a believer in testing everything possible in my packages, and
    wear it it as a badge
    of honor that the survival package has 4 lines of R code in the tests
    directory for every
    3 in the R directory. But CRAN only needs to run a small subset of
    this.
    We have a mechanism to specify testing level: the --as-cran flag. We
    could presumably make it more elaborate by adding other flags, or option
    levels, or whatever.

    What I think we shouldn't do is try to create an R-level test that says

    if (testingLevel() > 3) {
    doSomething
    }

    because tests can be turned on and off, individually. If testingLevel 3
    specified tests (A, B, C), then is our testingLevel higher if we are
    running tests (A, B, D, E, F, G)? Why not just test for the presence of
    whichever test is most relevant to that particular code block, e.g.

    if ("D" %in% tests()) {
    doSomething
    }

    I would prefer the testingLevel() approach of the "D" %in% tests()
    approach, since testingLevel() provides a natural way to add successively
    greater test details without having to dig into the code to determine the
    set of tests.
    I don't see how you could possibly calculate a single number in a reasonable way. What is
    the number that should be returned for (A, B, D, E, F, G)?

    Duncan Murdoch
    Duncan is leapfrogging ahead to another level that I hadn't thought of. An example would
    be to divide my survival package as "cox", "parametric", "survfit", "all", some of whaich
    overlap. Interesting idea, but beyond what I'd use. When I'm focused on a detail I run
    the test of interest directly, not through CMD check. For me low, med, high intensity
    suffices with as-cran invoking the first level and high including those that take an
    exceptionally long time.
    If you went to an A, B, C, ... approach what would be the as-cran default?

    Of course there is then the furthest level, which I recently instituted for survival due
    to the large number of dependencies: before posting a change download all the other
    dependent packages and run their tests too. It's an overnighter, and in that case I'd
    want level=high. Forewarned is forearmed.
    I don't think I'm leapfrogging, I think you're asking about something
    new. The thread was started by Spencer asking how to determine within
    an example whether he was being tested by CRAN, because he didn't want
    to run a long test there (it caused a timeout). This is an instance of
    the general problem of tailoring example and test code to the tests
    being run.

    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you can't
    reasonably convert the set of tests chosen by a tester into a single number.

    What I think you and Greg are talking about is something different. You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals manual).

    Duncan Murdoch
  • Warnes, Gregory at Sep 4, 2012 at 8:53 pm

    On 9/4/12 3:58 PM, "Duncan Murdoch" wrote:
    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    ly in
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:

    On 09/04/2012 05:00 AM, M
    wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would
    be the
    best. Then CRAN can choose to set that variable to "3" say, with
    level
    1 for extensive and 2 for usual.
    [snip]
    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you can't
    reasonably convert the set of tests chosen by a tester into a single
    number.

    What I think you and Greg are talking about is something different. You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals manual).
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.

    -Greg
  • Kasper Daniel Hansen at Sep 4, 2012 at 9:12 pm

    On Tue, Sep 4, 2012 at 4:53 PM, Warnes, Gregory wrote:
    On 9/4/12 3:58 PM, "Duncan Murdoch" wrote:
    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    ly in
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:

    On 09/04/2012 05:00 AM, M
    wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would
    be the
    best. Then CRAN can choose to set that variable to "3" say, with
    level
    1 for extensive and 2 for usual.
    [snip]
    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you can't
    reasonably convert the set of tests chosen by a tester into a single
    number.

    What I think you and Greg are talking about is something different. You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals manual).
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    And given that CRAN recently put down timing requirements (and
    Bioconductor has had them for a long time), it could be extremely
    useful to have one system. It is not clear to me whether it needs
    more than 2 levels ("slow" and "fast"), but I'll leave that up to
    people who have thought longer about this.

    I could certainly use it in several packages to differentiate between
    slow and quick tests.

    Kasper
  • John Fox at Sep 4, 2012 at 9:21 pm
    Dear all,

    I'd like to second this fairly simple request. I currently enclosed some of
    the examples in the effects package in \donttest{} blocks to satisfy the
    CRAN timing requirements for examples. It would be nice to have something
    like a \donttestcran{} block that suppresses the tests when --as-cran is set
    (and on CRAN itself).

    I'm sure that I've missed many of the nuances in this discussion, but this
    seems like a simple solution to me.

    Best,
    John
    -----Original Message-----
    From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-
    project.org] On Behalf Of Kasper Daniel Hansen
    Sent: Tuesday, September 04, 2012 5:12 PM
    To: Warnes, Gregory
    Cc: Terry Therneau; r-devel at r-project.org
    Subject: Re: [Rd] if(--as-cran)?

    On Tue, Sep 4, 2012 at 4:53 PM, Warnes, Gregory
    wrote:
    On 9/4/12 3:58 PM, "Duncan Murdoch" wrote:
    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    ly in
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:

    On 09/04/2012 05:00 AM, M
    wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the
    developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level
    would
    be the
    best. Then CRAN can choose to set that variable to "3" say,
    with
    level
    1 for extensive and 2 for usual.
    [snip]
    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you
    can't
    reasonably convert the set of tests chosen by a tester into a single
    number.

    What I think you and Greg are talking about is something different.
    You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set
    by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals
    manual).
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    And given that CRAN recently put down timing requirements (and
    Bioconductor has had them for a long time), it could be extremely
    useful to have one system. It is not clear to me whether it needs
    more than 2 levels ("slow" and "fast"), but I'll leave that up to
    people who have thought longer about this.

    I could certainly use it in several packages to differentiate between
    slow and quick tests.

    Kasper

    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Duncan Murdoch at Sep 4, 2012 at 9:32 pm

    On 04/09/2012 5:21 PM, John Fox wrote:
    Dear all,

    I'd like to second this fairly simple request. I currently enclosed some of
    the examples in the effects package in \donttest{} blocks to satisfy the
    CRAN timing requirements for examples. It would be nice to have something
    like a \donttestcran{} block that suppresses the tests when --as-cran is set
    (and on CRAN itself).

    I'm sure that I've missed many of the nuances in this discussion, but this
    seems like a simple solution to me.
    That would work for examples, but not tests. Many packages have scripts
    that are in the tests directory, not just test code in the .Rd files.
    What I think you should use is my suggested
    HowMuchTimeLeftBeforeTimeout() function, but if that's not writeable,
    then simply having a TIMELIMIT environment variable or similar, so your
    code could see if the there's a timeout pending.

    Duncan Murdoch
    Best,
    John
    -----Original Message-----
    From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-
    project.org] On Behalf Of Kasper Daniel Hansen
    Sent: Tuesday, September 04, 2012 5:12 PM
    To: Warnes, Gregory
    Cc: Terry Therneau; r-devel at r-project.org
    Subject: Re: [Rd] if(--as-cran)?

    On Tue, Sep 4, 2012 at 4:53 PM, Warnes, Gregory
    wrote:
    On 9/4/12 3:58 PM, "Duncan Murdoch" wrote:
    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    ly in
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" <murdoch.duncan@gmail.com>
    wrote:
    On 04/09/2012 8:20 AM, Terry Therneau wrote:

    On 09/04/2012 05:00 AM, M
    wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the
    developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level
    would
    be the
    best. Then CRAN can choose to set that variable to "3" say,
    with
    level
    1 for extensive and 2 for usual.
    [snip]
    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you
    can't
    reasonably convert the set of tests chosen by a tester into a single
    number.

    What I think you and Greg are talking about is something different.
    You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set
    by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals
    manual).
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    And given that CRAN recently put down timing requirements (and
    Bioconductor has had them for a long time), it could be extremely
    useful to have one system. It is not clear to me whether it needs
    more than 2 levels ("slow" and "fast"), but I'll leave that up to
    people who have thought longer about this.

    I could certainly use it in several packages to differentiate between
    slow and quick tests.

    Kasper

    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Kasper Daniel Hansen at Sep 4, 2012 at 9:46 pm

    On Tue, Sep 4, 2012 at 5:32 PM, Duncan Murdoch wrote:
    On 04/09/2012 5:21 PM, John Fox wrote:

    Dear all,

    I'd like to second this fairly simple request. I currently enclosed some
    of
    the examples in the effects package in \donttest{} blocks to satisfy the
    CRAN timing requirements for examples. It would be nice to have something
    like a \donttestcran{} block that suppresses the tests when --as-cran is
    set
    (and on CRAN itself).

    I'm sure that I've missed many of the nuances in this discussion, but this
    seems like a simple solution to me.

    That would work for examples, but not tests. Many packages have scripts
    that are in the tests directory, not just test code in the .Rd files. What
    I think you should use is my suggested HowMuchTimeLeftBeforeTimeout()
    function, but if that's not writeable, then simply having a TIMELIMIT
    environment variable or similar, so your code could see if the there's a
    timeout pending.
    I don't like this idea, because then I need to think about the order
    in which the tests are run.

    You are right that we could simulate this by having an environment
    variable, and if that variable is set, we could do the full tests.
    Like Dirk is mentioning, the (sensible, I might add) requirement that
    the tests should be "quick" suddenly makes the quick testing the
    default.

    What I - and I think a lot of other people - would like, is to do this
    in a standard way so it would be uniform across packages. I don't
    think it would be good if I start using R_LONG_TESTS and Dirk starts
    to use __NON_CRAN_TESTS or whatever. Then all of us need to know the
    package-specific variables when we test other people packages (and
    that happens occasionally, for example when you have a package that
    other packages depend upon).

    In this use case, standardization would be hugely beneficial in my
    opinion. In my reading of the discussion, this is really what all of
    us are saying.
    Duncan Murdoch

    Best,
    John
    -----Original Message-----
    From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-
    project.org] On Behalf Of Kasper Daniel Hansen
    Sent: Tuesday, September 04, 2012 5:12 PM
    To: Warnes, Gregory
    Cc: Terry Therneau; r-devel at r-project.org
    Subject: Re: [Rd] if(--as-cran)?

    On Tue, Sep 4, 2012 at 4:53 PM, Warnes, Gregory
    wrote:
    On 9/4/12 3:58 PM, "Duncan Murdoch" wrote:
    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    ly in
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" <murdoch.duncan@gmail.com>
    wrote:
    On 04/09/2012 8:20 AM, Terry Therneau wrote:

    On 09/04/2012 05:00 AM, M
    wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the
    developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level
    would
    be the
    best. Then CRAN can choose to set that variable to "3" say,
    with
    level
    1 for extensive and 2 for usual.
    [snip]
    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you
    can't
    reasonably convert the set of tests chosen by a tester into a single
    number.

    What I think you and Greg are talking about is something different.
    You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set
    by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals
    manual).
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    And given that CRAN recently put down timing requirements (and
    Bioconductor has had them for a long time), it could be extremely
    useful to have one system. It is not clear to me whether it needs
    more than 2 levels ("slow" and "fast"), but I'll leave that up to
    people who have thought longer about this.

    I could certainly use it in several packages to differentiate between
    slow and quick tests.

    Kasper

    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Martin Maechler at Sep 5, 2012 at 7:48 am

    Kasper Daniel Hansen <kasperdanielhansen@gmail.com>
    on Tue, 4 Sep 2012 17:41:38 -0400 writes:
    On Tue, Sep 4, 2012 at 5:32 PM, Duncan Murdoch wrote:
    On 04/09/2012 5:21 PM, John Fox wrote:
    >>>
    Dear all,
    >>>
    I'd like to second this fairly simple request. I currently enclosed some
    of
    the examples in the effects package in \donttest{} blocks to satisfy the
    CRAN timing requirements for examples. It would be nice to have something
    like a \donttestcran{} block that suppresses the tests when --as-cran is
    set
    (and on CRAN itself).
    >>>
    I'm sure that I've missed many of the nuances in this discussion, but this
    seems like a simple solution to me.
    >>
    >>
    That would work for examples, but not tests. Many packages have scripts
    that are in the tests directory, not just test code in the .Rd files. What
    I think you should use is my suggested HowMuchTimeLeftBeforeTimeout()
    function, but if that's not writeable, then simply having a TIMELIMIT
    environment variable or similar, so your code could see if the there's a
    timeout pending.
    I don't like this idea, because then I need to think about the order
    in which the tests are run.
    You are right that we could simulate this by having an environment
    variable, and if that variable is set, we could do the full tests.
    Like Dirk is mentioning, the (sensible, I might add) requirement that
    the tests should be "quick" suddenly makes the quick testing the
    default.
    What I - and I think a lot of other people - would like, is to do this
    in a standard way so it would be uniform across packages. I don't
    think it would be good if I start using R_LONG_TESTS and Dirk starts
    to use __NON_CRAN_TESTS or whatever. Then all of us need to know the
    package-specific variables when we test other people packages (and
    that happens occasionally, for example when you have a package that
    other packages depend upon).
    In this use case, standardization would be hugely beneficial in my
    opinion. In my reading of the discussion, this is really what all of
    us are saying.
    Kasper
    Yes!

    ... plus the idea to go one step further:

    Instead of just two levels ( quick tests / extensive tests ),
    one could use 3 or 4 (or 5) such levels, e.g.,
    1: very quick
    2: CRAN-ok-quick
    3: package-developer-usual
    4: extensive

    *and* the CRAN maintainers would say which level is the one
    that runs daily on CRAN and hence needs to meet Duncan's
    TIMELIMIT.

    So instead of just setting an environment variable to
    non-zero length, or "true" (as in my example), one could set
    that environment variable to an 1, 2, ...

    Martin

    -----Original Message-----
    From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-
    project.org] On Behalf Of Kasper Daniel Hansen
    Sent: Tuesday, September 04, 2012 5:12 PM
    To: Warnes, Gregory
    Cc: Terry Therneau; r-devel at r-project.org
    Subject: Re: [Rd] if(--as-cran)?
    >>> >
    On Tue, Sep 4, 2012 at 4:53 PM, Warnes, Gregory
    wrote:
    >>> > >
    On 9/4/12 3:58 PM, "Duncan Murdoch" wrote:
    >>> > >
    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    ly in
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" <murdoch.duncan@gmail.com>
    wrote:
    >>> > >>> >>
    >>> > >>> >>
    On 04/09/2012 8:20 AM, Terry Therneau wrote:
    >>> > >>> >> >>
    On 09/04/2012 05:00 AM, M
    wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the
    developer
    (but not only) should be able to run the extensive tests.
    >>> > >>> >> >> >
    Martin
    >>> > >>> >> >>
    I agree with Martin. A mechanism to specify testing level
    would
    be the
    best. Then CRAN can choose to set that variable to "3" say,
    with
    level
    1 for extensive and 2 for usual.
    >>> > >>>>> >>
    >>> > >>>
    [snip]
    >>> > >
    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you
    can't
    reasonably convert the set of tests chosen by a tester into a single
    number.
    >>> > >>
    What I think you and Greg are talking about is something different.
    You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set
    by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals
    manual).
    >>> > >
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.
    >>> > >
    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    >>> >
    And given that CRAN recently put down timing requirements (and
    Bioconductor has had them for a long time), it could be extremely
    useful to have one system. It is not clear to me whether it needs
    more than 2 levels ("slow" and "fast"), but I'll leave that up to
    people who have thought longer about this.
    >>> >
    I could certainly use it in several packages to differentiate between
    slow and quick tests.
    >>> >
    Kasper
    >>> >
    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
    >>>
    >>
  • Dirk Eddelbuettel at Sep 4, 2012 at 9:14 pm

    On 4 September 2012 at 20:53, Warnes, Gregory wrote:
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    Seconded.

    An add-on argument to the already established option --as-cran may be the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine developing _my_
    package I have remember how to enable what is now (as per CRAN's decree)
    "non-standard behaviour" of full testing. I fully agree with what Terry had
    said: more tests are better (when we develop). I want the full suite at my
    end; that is after all why we wrote it!

    The "non-standard" behaviour really is CRAN, and as it is already being
    accounted for, we may as well add a hook we all can use to disable some tests
    so that the runtime at CRAN remains within desired bounds. But that is the
    special case, and --as-cran should enable it (unless overridden).

    Dirk

    --
    Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
  • Duncan Murdoch at Sep 4, 2012 at 9:26 pm

    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    On 4 September 2012 at 20:53, Warnes, Gregory wrote:
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    Seconded.

    An add-on argument to the already established option --as-cran may be the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine developing _my_
    package I have remember how to enable what is now (as per CRAN's decree)
    "non-standard behaviour" of full testing. I fully agree with what Terry had
    said: more tests are better (when we develop). I want the full suite at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    The "non-standard" behaviour really is CRAN, and as it is already being
    accounted for, we may as well add a hook we all can use to disable some tests
    so that the runtime at CRAN remains within desired bounds. But that is the
    special case, and --as-cran should enable it (unless overridden).
    There are already hooks you can use to enable or disable lots of tests,
    maybe all of them (I haven't checked). It probably doesn't make sense
    to set the --as-cran tests as the default: would you want to be warned
    about version numbers if you're playing with a package locally, or if
    you have no intention to send it CRAN?

    It might make sense to be able to detect that you're running within a
    time limit, which is where this thread started. I don't know if it's
    possible to write a HowMuchTimeLeftBeforeTimeout() function, but if so,
    it would be useful in lots of contexts, not just for CRAN checks. In
    general, there are lots of specific reasons to want to know about
    specific tests, and it's reasonable to ask for those. It's not
    reasonable to ask for an "intensity" number.

    Duncan Murdoch
  • Dirk Eddelbuettel at Sep 4, 2012 at 9:42 pm

    On 4 September 2012 at 17:26, Duncan Murdoch wrote: | On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option --as-cran may be the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine developing _my_
    package I have remember how to enable what is now (as per CRAN's decree)
    "non-standard behaviour" of full testing. I fully agree with what Terry had
    said: more tests are better (when we develop). I want the full suite at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    "In theory, theory and practice are the same. In practice, they are not."

    The main test script long had exactly such a setting; I wrote what I wrote
    because it is _still the wrong way around_ and as I happen to have added to
    unit tests this weekend _having suffered through precisely this setting_.

    But we are on different wavelengths here and I evidently do not get my point
    across to you. And as you are the one who could make a change where it
    matters, I have no choice but to rest my case in frustration.

    Dirk

    --
    Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
  • Duncan Murdoch at Sep 4, 2012 at 10:02 pm

    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:
    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option --as-cran may be the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine developing _my_
    package I have remember how to enable what is now (as per CRAN's decree)
    "non-standard behaviour" of full testing. I fully agree with what Terry had
    said: more tests are better (when we develop). I want the full suite at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    "In theory, theory and practice are the same. In practice, they are not."

    The main test script long had exactly such a setting; I wrote what I wrote
    because it is _still the wrong way around_ and as I happen to have added to
    unit tests this weekend _having suffered through precisely this setting_.

    But we are on different wavelengths here and I evidently do not get my point
    across to you. And as you are the one who could make a change where it
    matters, I have no choice but to rest my case in frustration.
    If you want to give up, then give up, but then don't complain about the
    current behaviour. If you want to fix it, then continue the discussion.

    You're right that we're on different wavelengths. If you want some
    tests to run at home but not on CRAN, then somewhere there has to be a
    conditional. I'm suggesting that the conditional should be "if there's
    a tight time limit, skip this".

    I don't remember if this was your suggestion, but someone has suggested
    "if we're running with the --as-cran option, skip this" and others have
    suggested "if we're running on CRAN, skip this". I don't see why you
    find my suggestion so objectionable. If you want, I'll repeat why I
    find the other two suggestions objectionable.

    Duncan Murdoch
  • Kasper Daniel Hansen at Sep 5, 2012 at 12:19 am

    On Tue, Sep 4, 2012 at 6:02 PM, Duncan Murdoch wrote:
    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:
    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option --as-cran may be
    the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine
    developing _my_
    package I have remember how to enable what is now (as per CRAN's
    decree)
    "non-standard behaviour" of full testing. I fully agree with what
    Terry had
    said: more tests are better (when we develop). I want the full suite
    at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    "In theory, theory and practice are the same. In practice, they are not."

    The main test script long had exactly such a setting; I wrote what I wrote
    because it is _still the wrong way around_ and as I happen to have added
    to
    unit tests this weekend _having suffered through precisely this setting_.

    But we are on different wavelengths here and I evidently do not get my
    point
    across to you. And as you are the one who could make a change where it
    matters, I have no choice but to rest my case in frustration.

    If you want to give up, then give up, but then don't complain about the
    current behaviour. If you want to fix it, then continue the discussion.

    You're right that we're on different wavelengths. If you want some tests to
    run at home but not on CRAN, then somewhere there has to be a conditional.
    I'm suggesting that the conditional should be "if there's a tight time
    limit, skip this".

    I don't remember if this was your suggestion, but someone has suggested "if
    we're running with the --as-cran option, skip this" and others have
    suggested "if we're running on CRAN, skip this". I don't see why you find
    my suggestion so objectionable. If you want, I'll repeat why I find the
    other two suggestions objectionable.
    I agree with Duncan that having an option long/short makes more sense
    than with/without cran, as long as cran sets that option to be short.
    I would also prefer a command line switch to R CMD check to an
    environment variable, but I'll be very happy with a standardized
    environment variable.

    Kasper
    Duncan Murdoch


    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Duncan Murdoch at Sep 5, 2012 at 6:11 am

    On 12-09-04 8:19 PM, Kasper Daniel Hansen wrote:
    On Tue, Sep 4, 2012 at 6:02 PM, Duncan Murdoch wrote:
    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:
    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option --as-cran may be
    the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine
    developing _my_
    package I have remember how to enable what is now (as per CRAN's
    decree)
    "non-standard behaviour" of full testing. I fully agree with what
    Terry had
    said: more tests are better (when we develop). I want the full suite
    at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    "In theory, theory and practice are the same. In practice, they are not."

    The main test script long had exactly such a setting; I wrote what I wrote
    because it is _still the wrong way around_ and as I happen to have added
    to
    unit tests this weekend _having suffered through precisely this setting_.

    But we are on different wavelengths here and I evidently do not get my
    point
    across to you. And as you are the one who could make a change where it
    matters, I have no choice but to rest my case in frustration.

    If you want to give up, then give up, but then don't complain about the
    current behaviour. If you want to fix it, then continue the discussion.

    You're right that we're on different wavelengths. If you want some tests to
    run at home but not on CRAN, then somewhere there has to be a conditional.
    I'm suggesting that the conditional should be "if there's a tight time
    limit, skip this".

    I don't remember if this was your suggestion, but someone has suggested "if
    we're running with the --as-cran option, skip this" and others have
    suggested "if we're running on CRAN, skip this". I don't see why you find
    my suggestion so objectionable. If you want, I'll repeat why I find the
    other two suggestions objectionable.
    I agree with Duncan that having an option long/short makes more sense
    than with/without cran, as long as cran sets that option to be short.

    I would also prefer a command line switch to R CMD check to an
    environment variable, but I'll be very happy with a standardized
    environment variable.
    I honestly don't see the need for a standardized variable. I've told
    you how to detect that you are running --as-cran; if that isn't
    sufficient information, then you, the package author, need to set up
    something more elaborate, and assume that if it's not set up, then
    someone else (maybe CRAN) is running the test.

    I asked the CRAN powers-that-be about the possibility of querying the
    amount of time remaining before a timeout; since the different platforms
    all use different mechanisms to enforce a timeout, that's not really
    practical. So the best you could hope for is to know that a timeout is
    in effect. Before I wrote any code, I'd need to hear why --as-cran
    detection isn't sufficient.

    Duncan Murdoch

    Kasper
    Duncan Murdoch


    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Deepayan Sarkar at Sep 5, 2012 at 6:19 am

    On Wed, Sep 5, 2012 at 11:41 AM, Duncan Murdoch wrote:
    On 12-09-04 8:19 PM, Kasper Daniel Hansen wrote:

    On Tue, Sep 4, 2012 at 6:02 PM, Duncan Murdoch <murdoch.duncan@gmail.com>
    wrote:
    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:

    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option --as-cran may
    be
    the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine
    developing _my_
    package I have remember how to enable what is now (as per CRAN's
    decree)
    "non-standard behaviour" of full testing. I fully agree with what
    Terry had
    said: more tests are better (when we develop). I want the full
    suite
    at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    "In theory, theory and practice are the same. In practice, they are
    not."

    The main test script long had exactly such a setting; I wrote what I
    wrote
    because it is _still the wrong way around_ and as I happen to have added
    to
    unit tests this weekend _having suffered through precisely this
    setting_.

    But we are on different wavelengths here and I evidently do not get my
    point
    across to you. And as you are the one who could make a change where it
    matters, I have no choice but to rest my case in frustration.


    If you want to give up, then give up, but then don't complain about the
    current behaviour. If you want to fix it, then continue the discussion.

    You're right that we're on different wavelengths. If you want some tests
    to
    run at home but not on CRAN, then somewhere there has to be a
    conditional.
    I'm suggesting that the conditional should be "if there's a tight time
    limit, skip this".

    I don't remember if this was your suggestion, but someone has suggested
    "if
    we're running with the --as-cran option, skip this" and others have
    suggested "if we're running on CRAN, skip this". I don't see why you
    find
    my suggestion so objectionable. If you want, I'll repeat why I find the
    other two suggestions objectionable.

    I agree with Duncan that having an option long/short makes more sense
    than with/without cran, as long as cran sets that option to be short.

    I would also prefer a command line switch to R CMD check to an
    environment variable, but I'll be very happy with a standardized
    environment variable.

    I honestly don't see the need for a standardized variable. I've told you
    how to detect that you are running --as-cran; if that isn't sufficient
    information, then you, the package author, need to set up something more
    elaborate, and assume that if it's not set up, then someone else (maybe
    CRAN) is running the test.
    So maybe documenting that (_R_CHECK_TIMINGS_) more formally in R-exts
    would be sufficient?

    -Deepayan

    I asked the CRAN powers-that-be about the possibility of querying the amount
    of time remaining before a timeout; since the different platforms all use
    different mechanisms to enforce a timeout, that's not really practical. So
    the best you could hope for is to know that a timeout is in effect. Before
    I wrote any code, I'd need to hear why --as-cran detection isn't sufficient.
  • Martin Maechler at Sep 5, 2012 at 10:25 am

    Deepayan Sarkar <deepayan.sarkar@gmail.com>
    on Wed, 5 Sep 2012 11:49:37 +0530 writes:
    On Wed, Sep 5, 2012 at 11:41 AM, Duncan Murdoch
    wrote:
    On 12-09-04 8:19 PM, Kasper Daniel Hansen wrote:
    >>>
    On Tue, Sep 4, 2012 at 6:02 PM, Duncan Murdoch <murdoch.duncan@gmail.com>
    wrote:
    >>>>
    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:
    >>>>>
    >>>>>
    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option --as-cran may
    be
    the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine
    developing _my_
    package I have remember how to enable what is now (as per CRAN's
    decree)
    "non-standard behaviour" of full testing. I fully agree with what
    Terry had
    said: more tests are better (when we develop). I want the full
    suite
    at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    >>>>>
    "In theory, theory and practice are the same. In practice, they are
    not."
    >>>>>
    The main test script long had exactly such a setting; I wrote what I
    wrote
    because it is _still the wrong way around_ and as I happen to have added
    to
    unit tests this weekend _having suffered through precisely this
    setting_.
    >>>>>
    But we are on different wavelengths here and I evidently do not get my
    point
    across to you. And as you are the one who could make a change where it
    matters, I have no choice but to rest my case in frustration.
    >>>>
    >>>>
    >>>>
    If you want to give up, then give up, but then don't complain about the
    current behaviour. If you want to fix it, then continue the discussion.
    >>>>
    You're right that we're on different wavelengths. If you want some tests
    to
    run at home but not on CRAN, then somewhere there has to be a
    conditional.
    I'm suggesting that the conditional should be "if there's a tight time
    limit, skip this".
    >>>>
    I don't remember if this was your suggestion, but someone has suggested
    "if
    we're running with the --as-cran option, skip this" and others have
    suggested "if we're running on CRAN, skip this". I don't see why you
    find
    my suggestion so objectionable. If you want, I'll repeat why I find the
    other two suggestions objectionable.
    >>>
    >>>
    I agree with Duncan that having an option long/short makes more sense
    than with/without cran, as long as cran sets that option to be short.
    >>
    >>>
    >>>
    I would also prefer a command line switch to R CMD check to an
    environment variable, but I'll be very happy with a standardized
    environment variable.
    >>
    >>
    I honestly don't see the need for a standardized variable. I've told you
    how to detect that you are running --as-cran; if that isn't sufficient
    information, then you, the package author, need to set up something more
    elaborate, and assume that if it's not set up, then someone else (maybe
    CRAN) is running the test.
    So maybe documenting that (_R_CHECK_TIMINGS_) more formally in R-exts
    would be sufficient?
    -Deepayan
    yes and no.
    As Duncan said very early, --as-cran is just turning on several
    already existing options, one of them being the
    _R_CHECK_TIMINGS_ -- which you can *also* enable by --timings.

    So, checking for _R_CHECK_TIMINGS_ is *not* checking for
    the presence of --as-cran !
    .. in the sense that it is necessary but not sufficient.
    Also, if I run the "long checks" I may still want to use
    --timings. For me as package developer, the timings may be even more
    important in the "long" than in the "short" checks case.

    So, one solution that is little work would be that --as-cran
    sets an *extra* environment variable that is *only* set by
    --as-cran, but by no other command line switch.

    Still a pity that it seems people want to live in a 0/1
    setting when it would be very natural to adopt a 0/1/2/3
    (or so) one.
    It's a bit like prefering verbose = FALSE/TRUE as argument to
    an R function where eventually I often found that using a
    tracing = 0/1/2 (or then 'verbose' = 0/1/2 which is almost
    back compatible to FALSE/TRUE)
    was more useful.


    I asked the CRAN powers-that-be about the possibility of querying the amount
    of time remaining before a timeout; since the different platforms all use
    different mechanisms to enforce a timeout, that's not really practical. So
    the best you could hope for is to know that a timeout is in effect. Before
    I wrote any code, I'd need to hear why --as-cran detection isn't sufficient.
    I agree that *reliable* --as-cran detection solves the OP's
    and most other correspondents problem.
    But as argued above, _R_CHECK_TIMINGS_ is not sufficient.

    Martin
  • Duncan Murdoch at Sep 5, 2012 at 1:10 pm
    This post contains two incorrect statements. Since Martin is too busy
    to post a retraction, I will point them out:

    --timings and --as-cran do not set _R_CHECK_TIMINGS_ in the same way.

    As far as I recall, nobody has suggested that package writers be limited
    to two choices for test suites. I certainly haven't.

    Duncan Murdoch
    On 05/09/2012 6:25 AM, Martin Maechler wrote:
    Deepayan Sarkar <deepayan.sarkar@gmail.com>
    on Wed, 5 Sep 2012 11:49:37 +0530 writes:
    On Wed, Sep 5, 2012 at 11:41 AM, Duncan Murdoch
    wrote:
    On 12-09-04 8:19 PM, Kasper Daniel Hansen wrote:

    On Tue, Sep 4, 2012 at 6:02 PM, Duncan Murdoch <murdoch.duncan@gmail.com>
    wrote:
    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:

    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option --as-cran may
    be
    the
    best.

    And to iterate, what bugs me is that for _me_ on _my_ machine
    developing _my_
    package I have remember how to enable what is now (as per CRAN's
    decree)
    "non-standard behaviour" of full testing. I fully agree with what
    Terry had
    said: more tests are better (when we develop). I want the full
    suite
    at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out once, write a
    script that sets the environment variables that enable it, and then you
    can forget it.
    "In theory, theory and practice are the same. In practice, they are
    not."

    The main test script long had exactly such a setting; I wrote what I
    wrote
    because it is _still the wrong way around_ and as I happen to have added
    to
    unit tests this weekend _having suffered through precisely this
    setting_.

    But we are on different wavelengths here and I evidently do not get my
    point
    across to you. And as you are the one who could make a change where it
    matters, I have no choice but to rest my case in frustration.


    If you want to give up, then give up, but then don't complain about the
    current behaviour. If you want to fix it, then continue the discussion.

    You're right that we're on different wavelengths. If you want some tests
    to
    run at home but not on CRAN, then somewhere there has to be a
    conditional.
    I'm suggesting that the conditional should be "if there's a tight time
    limit, skip this".

    I don't remember if this was your suggestion, but someone has suggested
    "if
    we're running with the --as-cran option, skip this" and others have
    suggested "if we're running on CRAN, skip this". I don't see why you
    find
    my suggestion so objectionable. If you want, I'll repeat why I find the
    other two suggestions objectionable.

    I agree with Duncan that having an option long/short makes more sense
    than with/without cran, as long as cran sets that option to be short.

    I would also prefer a command line switch to R CMD check to an
    environment variable, but I'll be very happy with a standardized
    environment variable.

    I honestly don't see the need for a standardized variable. I've told you
    how to detect that you are running --as-cran; if that isn't sufficient
    information, then you, the package author, need to set up something more
    elaborate, and assume that if it's not set up, then someone else (maybe
    CRAN) is running the test.
    So maybe documenting that (_R_CHECK_TIMINGS_) more formally in R-exts
    would be sufficient?
    -Deepayan
    yes and no.
    As Duncan said very early, --as-cran is just turning on several
    already existing options, one of them being the
    _R_CHECK_TIMINGS_ -- which you can *also* enable by --timings.

    So, checking for _R_CHECK_TIMINGS_ is *not* checking for
    the presence of --as-cran !
    .. in the sense that it is necessary but not sufficient.
    Also, if I run the "long checks" I may still want to use
    --timings. For me as package developer, the timings may be even more
    important in the "long" than in the "short" checks case.

    So, one solution that is little work would be that --as-cran
    sets an *extra* environment variable that is *only* set by
    --as-cran, but by no other command line switch.

    Still a pity that it seems people want to live in a 0/1
    setting when it would be very natural to adopt a 0/1/2/3
    (or so) one.
    It's a bit like prefering verbose = FALSE/TRUE as argument to
    an R function where eventually I often found that using a
    tracing = 0/1/2 (or then 'verbose' = 0/1/2 which is almost
    back compatible to FALSE/TRUE)
    was more useful.


    I asked the CRAN powers-that-be about the possibility of querying the amount
    of time remaining before a timeout; since the different platforms all use
    different mechanisms to enforce a timeout, that's not really practical. So
    the best you could hope for is to know that a timeout is in effect. Before
    I wrote any code, I'd need to hear why --as-cran detection isn't sufficient.
    I agree that *reliable* --as-cran detection solves the OP's
    and most other correspondents problem.
    But as argued above, _R_CHECK_TIMINGS_ is not sufficient.

    Martin
  • Kasper Daniel Hansen at Sep 6, 2012 at 1:51 am
    Duncan,

    you are right that from the perspective of a single developer: it is
    entirely possible to make tests conditional and ensure that 'long'
    tests does not get run by CRAN.

    What I, and several others, advocate is a uniform set of conventions,
    so we avoid having multiple package specific solutions.

    Martin has explained one reason for why using _R_CHECK_TIMINGS_ is not
    ideal (the fact that we - at least I and Martin - are most interested
    in the timings for the long tests). Another reason is that I (and I
    suspect most others) would prefer something that looks a bit more
    official and supported. High-jacking an environment variable set for
    (what appears to be) a different reason does not seem like something
    that could be considered supported long-term, and which have a uniform
    interface (across packages).

    Of course, adding an extra environment variable or an extra function
    (as suggested by Henrik) is additional work, and I know it is easy for
    us to suggest things. My perspective is that I would like something
    that will continue to be supported in the future, and where I can use
    the same interface across the packages I maintain and others outside
    my control that I occasionally test. Testing is not just being done
    by the package author(s) or CRAN. I suspect many packages already
    have half-baked solutions to the 'long' test vs 'short' test issue,
    and I don't think we will see any movement toward a unified interface
    without support in base R.

    And let me end by saying that I do agree with Martin: having numerical
    levels is better (together with a clear statement what --as-cran sets
    the level to). But I would rather have a binary state than nothing.

    Kasper

    On Wed, Sep 5, 2012 at 9:10 AM, Duncan Murdoch wrote:
    This post contains two incorrect statements. Since Martin is too busy to
    post a retraction, I will point them out:

    --timings and --as-cran do not set _R_CHECK_TIMINGS_ in the same way.

    As far as I recall, nobody has suggested that package writers be limited to
    two choices for test suites. I certainly haven't.

    Duncan Murdoch

    On 05/09/2012 6:25 AM, Martin Maechler wrote:

    Deepayan Sarkar <deepayan.sarkar@gmail.com>
    on Wed, 5 Sep 2012 11:49:37 +0530 writes:
    On Wed, Sep 5, 2012 at 11:41 AM, Duncan Murdoch
    wrote:
    On 12-09-04 8:19 PM, Kasper Daniel Hansen wrote:

    On Tue, Sep 4, 2012 at 6:02 PM, Duncan Murdoch
    <murdoch.duncan@gmail.com>
    wrote:
    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:

    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option
    --as-cran may
    be
    the
    best.

    And to iterate, what bugs me is that for _me_ on _my_
    machine
    developing _my_
    package I have remember how to enable what is now (as per
    CRAN's
    decree)
    "non-standard behaviour" of full testing. I fully agree
    with what
    Terry had
    said: more tests are better (when we develop). I want the
    full
    suite
    at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out
    once, write
    a
    script that sets the environment variables that enable it,
    and then
    you
    can forget it.
    "In theory, theory and practice are the same. In practice, they
    are
    not."

    The main test script long had exactly such a setting; I wrote
    what I
    wrote
    because it is _still the wrong way around_ and as I happen to
    have added
    to
    unit tests this weekend _having suffered through precisely this
    setting_.

    But we are on different wavelengths here and I evidently do not
    get my
    point
    across to you. And as you are the one who could make a change
    where it
    matters, I have no choice but to rest my case in frustration.


    If you want to give up, then give up, but then don't complain
    about the
    current behaviour. If you want to fix it, then continue the
    discussion.
    You're right that we're on different wavelengths. If you want
    some tests
    to
    run at home but not on CRAN, then somewhere there has to be a
    conditional.
    I'm suggesting that the conditional should be "if there's a
    tight time
    limit, skip this".

    I don't remember if this was your suggestion, but someone has
    suggested
    "if
    we're running with the --as-cran option, skip this" and others
    have
    suggested "if we're running on CRAN, skip this". I don't see
    why you
    find
    my suggestion so objectionable. If you want, I'll repeat why I
    find the
    other two suggestions objectionable.

    I agree with Duncan that having an option long/short makes more
    sense
    than with/without cran, as long as cran sets that option to be
    short.

    I would also prefer a command line switch to R CMD check to an
    environment variable, but I'll be very happy with a standardized
    environment variable.

    I honestly don't see the need for a standardized variable. I've
    told you
    how to detect that you are running --as-cran; if that isn't
    sufficient
    information, then you, the package author, need to set up
    something more
    elaborate, and assume that if it's not set up, then someone else
    (maybe
    CRAN) is running the test.
    So maybe documenting that (_R_CHECK_TIMINGS_) more formally in R-exts
    would be sufficient?
    -Deepayan
    yes and no.
    As Duncan said very early, --as-cran is just turning on several
    already existing options, one of them being the
    _R_CHECK_TIMINGS_ -- which you can *also* enable by --timings.

    So, checking for _R_CHECK_TIMINGS_ is *not* checking for
    the presence of --as-cran !
    .. in the sense that it is necessary but not sufficient.
    Also, if I run the "long checks" I may still want to use
    --timings. For me as package developer, the timings may be even more
    important in the "long" than in the "short" checks case.

    So, one solution that is little work would be that --as-cran
    sets an *extra* environment variable that is *only* set by
    --as-cran, but by no other command line switch.

    Still a pity that it seems people want to live in a 0/1
    setting when it would be very natural to adopt a 0/1/2/3
    (or so) one.
    It's a bit like prefering verbose = FALSE/TRUE as argument to
    an R function where eventually I often found that using a
    tracing = 0/1/2 (or then 'verbose' = 0/1/2 which is almost
    back compatible to FALSE/TRUE)
    was more useful.


    I asked the CRAN powers-that-be about the possibility of querying
    the amount
    of time remaining before a timeout; since the different platforms
    all use
    different mechanisms to enforce a timeout, that's not really
    practical. So
    the best you could hope for is to know that a timeout is in
    effect. Before
    I wrote any code, I'd need to hear why --as-cran detection isn't
    sufficient.

    I agree that *reliable* --as-cran detection solves the OP's
    and most other correspondents problem.
    But as argued above, _R_CHECK_TIMINGS_ is not sufficient.

    Martin

    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Duncan Murdoch at Sep 6, 2012 at 11:41 am

    On 12-09-05 9:51 PM, Kasper Daniel Hansen wrote:
    Duncan,

    you are right that from the perspective of a single developer: it is
    entirely possible to make tests conditional and ensure that 'long'
    tests does not get run by CRAN.

    What I, and several others, advocate is a uniform set of conventions,
    so we avoid having multiple package specific solutions.

    Martin has explained one reason for why using _R_CHECK_TIMINGS_ is not
    ideal (the fact that we - at least I and Martin - are most interested
    in the timings for the long tests).
    You do realize that his objections were wrong, don't you?


    Another reason is that I (and I
    suspect most others) would prefer something that looks a bit more
    official and supported.
    The _R_CHECK_TIMINGS_ environment variable is documented and official.


    High-jacking an environment variable set for
    (what appears to be) a different reason does not seem like something
    that could be considered supported long-term, and which have a uniform
    interface (across packages).
    Using it to detect reports about long-running examples is exactly what
    it is designed for. It is not "high-jacking".
    Of course, adding an extra environment variable or an extra function
    (as suggested by Henrik) is additional work, and I know it is easy for
    us to suggest things. My perspective is that I would like something
    that will continue to be supported in the future, and where I can use
    the same interface across the packages I maintain and others outside
    my control that I occasionally test. Testing is not just being done
    by the package author(s) or CRAN. I suspect many packages already
    have half-baked solutions to the 'long' test vs 'short' test issue,
    and I don't think we will see any movement toward a unified interface
    without support in base R.
    If Martin or I added a new environment variable _R_CHECK_LEVEL_, and
    perhaps support code to let you set it in R CMD check or elsewhere, and
    support functions to let you read it without the work of calling
    Sys.getenv(), and documentation for its purpose -- that doesn't mean it
    would be ever be used. I would have no objection whatsoever if he wants
    to do that work, but I won't, as I think everyone who would ever use it
    has already invented their own method for doing the same thing. It
    would waste my time to implement it and would waste their time making
    their code compatible with the new system.
    And let me end by saying that I do agree with Martin: having numerical
    levels is better (together with a clear statement what --as-cran sets
    the level to). But I would rather have a binary state than nothing.
    It was pointed out to me by a member of CRAN that detecting --as-cran is
    insufficient. CRAN only runs incoming tests with settings equivalent to
    that. The daily tests use different options. I don't know what those
    are, and CRAN is unwilling to discuss their internal policies on this
    list. (I would guess they find the tantrums and false claims here to be
    frustrating. I do.)

    So if Martin wants to implement something, I'd suggest that he
    coordinate offline with CRAN.

    Duncan Murdoch
    Kasper

    On Wed, Sep 5, 2012 at 9:10 AM, Duncan Murdoch wrote:
    This post contains two incorrect statements. Since Martin is too busy to
    post a retraction, I will point them out:

    --timings and --as-cran do not set _R_CHECK_TIMINGS_ in the same way.

    As far as I recall, nobody has suggested that package writers be limited to
    two choices for test suites. I certainly haven't.

    Duncan Murdoch

    On 05/09/2012 6:25 AM, Martin Maechler wrote:

    Deepayan Sarkar <deepayan.sarkar@gmail.com>
    on Wed, 5 Sep 2012 11:49:37 +0530 writes:
    On Wed, Sep 5, 2012 at 11:41 AM, Duncan Murdoch
    wrote:
    On 12-09-04 8:19 PM, Kasper Daniel Hansen wrote:

    On Tue, Sep 4, 2012 at 6:02 PM, Duncan Murdoch
    <murdoch.duncan@gmail.com>
    wrote:
    On 04/09/2012 5:42 PM, Dirk Eddelbuettel wrote:

    On 4 September 2012 at 17:26, Duncan Murdoch wrote:
    On 04/09/2012 5:14 PM, Dirk Eddelbuettel wrote:
    An add-on argument to the already established option
    --as-cran may
    be
    the
    best.

    And to iterate, what bugs me is that for _me_ on _my_
    machine
    developing _my_
    package I have remember how to enable what is now (as per
    CRAN's
    decree)
    "non-standard behaviour" of full testing. I fully agree
    with what
    Terry had
    said: more tests are better (when we develop). I want the
    full
    suite
    at my
    end; that is after all why we wrote it!
    You don't have to remember that, you need to figure it out
    once, write
    a
    script that sets the environment variables that enable it,
    and then
    you
    can forget it.
    "In theory, theory and practice are the same. In practice, they
    are
    not."

    The main test script long had exactly such a setting; I wrote
    what I
    wrote
    because it is _still the wrong way around_ and as I happen to
    have added
    to
    unit tests this weekend _having suffered through precisely this
    setting_.

    But we are on different wavelengths here and I evidently do not
    get my
    point
    across to you. And as you are the one who could make a change
    where it
    matters, I have no choice but to rest my case in frustration.


    If you want to give up, then give up, but then don't complain
    about the
    current behaviour. If you want to fix it, then continue the
    discussion.
    You're right that we're on different wavelengths. If you want
    some tests
    to
    run at home but not on CRAN, then somewhere there has to be a
    conditional.
    I'm suggesting that the conditional should be "if there's a
    tight time
    limit, skip this".

    I don't remember if this was your suggestion, but someone has
    suggested
    "if
    we're running with the --as-cran option, skip this" and others
    have
    suggested "if we're running on CRAN, skip this". I don't see
    why you
    find
    my suggestion so objectionable. If you want, I'll repeat why I
    find the
    other two suggestions objectionable.

    I agree with Duncan that having an option long/short makes more
    sense
    than with/without cran, as long as cran sets that option to be
    short.

    I would also prefer a command line switch to R CMD check to an
    environment variable, but I'll be very happy with a standardized
    environment variable.

    I honestly don't see the need for a standardized variable. I've
    told you
    how to detect that you are running --as-cran; if that isn't
    sufficient
    information, then you, the package author, need to set up
    something more
    elaborate, and assume that if it's not set up, then someone else
    (maybe
    CRAN) is running the test.
    So maybe documenting that (_R_CHECK_TIMINGS_) more formally in R-exts
    would be sufficient?
    -Deepayan
    yes and no.
    As Duncan said very early, --as-cran is just turning on several
    already existing options, one of them being the
    _R_CHECK_TIMINGS_ -- which you can *also* enable by --timings.

    So, checking for _R_CHECK_TIMINGS_ is *not* checking for
    the presence of --as-cran !
    .. in the sense that it is necessary but not sufficient.
    Also, if I run the "long checks" I may still want to use
    --timings. For me as package developer, the timings may be even more
    important in the "long" than in the "short" checks case.

    So, one solution that is little work would be that --as-cran
    sets an *extra* environment variable that is *only* set by
    --as-cran, but by no other command line switch.

    Still a pity that it seems people want to live in a 0/1
    setting when it would be very natural to adopt a 0/1/2/3
    (or so) one.
    It's a bit like prefering verbose = FALSE/TRUE as argument to
    an R function where eventually I often found that using a
    tracing = 0/1/2 (or then 'verbose' = 0/1/2 which is almost
    back compatible to FALSE/TRUE)
    was more useful.


    I asked the CRAN powers-that-be about the possibility of querying
    the amount
    of time remaining before a timeout; since the different platforms
    all use
    different mechanisms to enforce a timeout, that's not really
    practical. So
    the best you could hope for is to know that a timeout is in
    effect. Before
    I wrote any code, I'd need to hear why --as-cran detection isn't
    sufficient.

    I agree that *reliable* --as-cran detection solves the OP's
    and most other correspondents problem.
    But as argued above, _R_CHECK_TIMINGS_ is not sufficient.

    Martin

    ______________________________________________
    R-devel at r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/r-devel
  • Hadley Wickham at Sep 6, 2012 at 1:25 pm

    It was pointed out to me by a member of CRAN that detecting --as-cran is
    insufficient. CRAN only runs incoming tests with settings equivalent to
    that. The daily tests use different options. I don't know what those are,
    and CRAN is unwilling to discuss their internal policies on this list. (I
    would guess they find the tantrums and false claims here to be frustrating.
    I do.)
    It's equally frustrating to not know how CRAN works. I feel like we
    share common goals (making quality software) but somehow there is a
    communication gap that has lead to an overly combative stance between
    package authors and CRAN. I appreciate that the CRAN maintainers are
    busy, but I would hope that communicating exactly what's going on
    publicly would save time by avoiding individual conversations with
    package authors.

    Hadley

    --
    Assistant Professor
    Department of Statistics / Rice University
    http://had.co.nz/
  • Warnes, Gregory at Sep 6, 2012 at 1:33 pm

    On 9/6/12 7:41 AM, "Duncan Murdoch" wrote:
    If Martin or I added a new environment variable _R_CHECK_LEVEL_, and
    perhaps support code to let you set it in R CMD check or elsewhere, and
    support functions to let you read it without the work of calling
    Sys.getenv(), and documentation for its purpose -- that doesn't mean it
    would be ever be used. I would have no objection whatsoever if he wants
    to do that work, [..]
    [...]
    So if Martin wants to implement something, I'd suggest that he
    coordinate offline with CRAN.
    With this in mind, I've taken the time to create a proposed implementation:

    1) New environement variable _R_CHECK_LEVEL_ which can take values from
    {NONE, MINOR, CRAN, DEFAULT, MAJOR, FULL}
    2) New R CMD CHECK flag --level={NONE, MINOR, CRAN, DEFAULT, MAJOR, FULL},
    which sets the value of _R_CHECK_LEVEL_, defaulting to "DEFAULT".

    3) R CMD CHECK flag --as-cran sets _R_CHECK_LEVEL_ to CRAN (overwriting
    any previous environment variable or --level setting)
    4) A new function: tools::getCheckLevel() that returns an ordered factor
    with levels {NONE, MINOR, CRAN, DEFAULT, MAJOR, FULL}.
    5) An associated man page, with the example:

    if(getCheckLevel() > "CRAN") {
    cat("## Do time-intenstive test ##\n")
    } else {
    cat("## Do brief test\n")
    }

    Duncan, how should I coordinate this with CRAN, simply drop an email to
    cran at r-project.org?

    Everyone: I'll post a pointer to the patch as soon as I've tested it.

    -Greg
  • Duncan Murdoch at Sep 4, 2012 at 9:15 pm

    On 04/09/2012 4:53 PM, Warnes, Gregory wrote:
    On 9/4/12 3:58 PM, "Duncan Murdoch" wrote:
    On 04/09/2012 3:44 PM, Terry Therneau wrote:
    ly in
    On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
    On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
    On 9/4/12 8:38 AM, "Duncan Murdoch" wrote:

    On 04/09/2012 8:20 AM, Terry Therneau wrote:

    On 09/04/2012 05:00 AM, M
    wrote:
    The issue is not just about "CRAN" vs "off CRAN".
    It is good to think about a more general scheme of
    "light testing" vs "normal testing" vs "extensive testing",
    e.g., for the situation where the package implements
    (simulation/bootstrap/ ..) based inference, and the developer
    (but not only) should be able to run the extensive tests.

    Martin
    I agree with Martin. A mechanism to specify testing level would
    be the
    best. Then CRAN can choose to set that variable to "3" say, with
    level
    1 for extensive and 2 for usual.
    [snip]
    The testingLevel() function is supposed to be a way to know that a
    certain level of testing is being done, to allow such tailoring.
    However, I don't think it's practical. I think you can ask whether a
    specific test is being run (my "D" %in% tests() example), but you can't
    reasonably convert the set of tests chosen by a tester into a single
    number.

    What I think you and Greg are talking about is something different. You
    are asking that we set up more suites of tests, corresponding to
    numerical levels. Currently we have two suites: the default, and the
    --as-cran suite. But we also have completely customized suites, set by
    users who want to check specific things. They can do that the way you
    do (by calling the tests explicitly), or by setting environment
    variables (as described in the Tools chapter of the R Internals manual).
    No! We're not asking for the r-core to create more test suites, or even
    to do anything different based on the test intensity level.

    We're just asking for a standard way to control the intensity of the tests
    *we* write to prevent us from duplicating this functionality in our own
    packages, probably in incompatible ways.
    But you do have ways to control the tests you write. The standard way
    is to set an environment variable, private to your needs. If you see
    the variable, do what it says. If you don't see it, do the default.

    What I think is being asked for is a way to detect the suite of tests
    that CRAN runs (i.e. the --as-cran test), or perhaps to detect that it's
    actually CRAN doing the testing. I've given a way to detect --as-cran,
    and despite what Martin says, it doesn't conflict with his practice. I
    don't think it's a good idea to detect who is doing the testing, because
    I'd like to be able to duplicate the CRAN results for your package.

    Duncan Murdoch
  • David L Lorenz at Sep 4, 2012 at 1:47 pm
    All,
    I seem to be missing some key point about the construction of vignettes.
    I have created a vignette script that works fine when run interactively
    using Sweave. The top part of the .Rnw file is below.

    \documentclass{article}
    \parskip 3pt
    \parindent 30pt
    %\VignetteIndexEntry{Box Plot Examples}
    %\VignetteDepends{USGSgraphs}

    \begin{document}

    \title{Box Plot Examples}

    \author{Dave Lorenz}

    \maketitle

    ... text omittted from this example ...
    <<echo=TRUE>>=
    # Generate a random sample for the box plot
    set.seed(27036)
    BP <- rchisq(32, 3)
    # setSweave is a specialized function that sets up the graphics page
    setSweave("boxplot01", 6 ,6)
    # Set layout for 4 graphs
    AA.lo <- setLayout(width=rep(1.25, 4), height=4, xtop=1.5)

    ... remainder omitted ...

    The functions setSweave and setLayout are functions within the
    USGSgraphs library that I am building. As I said, the script runs just
    fine when I run interactively. I do have a version of the USGSgraphs
    library attached when I run the script. When I run R CMD check, or R CMD
    build on the source, I get the error:

    Error: processing vignette 'boxplots.Rnw' failed with diagnostics:
    chunk 1
    Error in eval(expr, envir, enclos) : could not find function "setSweave"
    Execution halted

    I would have expected that the contents of the library being built would
    have been available to the script (based on section 1.3.2 in R-exts.html).
    Barring that, I expected that "%\VignetteDepends{USGSgraphs}" would have
    made those functions available to the script. I have found that if I
    include this code before the first real example, it will run.

    <<echo=FALSE>>=
    library(USGSgraphs)
    @

    But I have no reason to believe that it would run on any system that did
    not already have a version of USGSgraphs installed. Note that if I use the
    default graphics output, then I get the same error on setLayout.
    I use the default process for building the vignettes--no makefile.
    What do I need to do to get this vignette to run? I know I can set
    BuildVignettes to FALSE in the DESCRIPTION file or try to build with the
    --no-vignettes option, but that does not really address the more general
    issue.
    Thanks.
    Dave
  • Duncan Murdoch at Sep 4, 2012 at 2:33 pm

    On 04/09/2012 9:47 AM, David L Lorenz wrote:
    All,
    I seem to be missing some key point about the construction of vignettes.
    I have created a vignette script that works fine when run interactively
    using Sweave. The top part of the .Rnw file is below.

    \documentclass{article}
    \parskip 3pt
    \parindent 30pt
    %\VignetteIndexEntry{Box Plot Examples}
    %\VignetteDepends{USGSgraphs}

    \begin{document}

    \title{Box Plot Examples}

    \author{Dave Lorenz}

    \maketitle

    ... text omittted from this example ...
    <<echo=TRUE>>> # Generate a random sample for the box plot
    set.seed(27036)
    BP <- rchisq(32, 3)
    # setSweave is a specialized function that sets up the graphics page
    setSweave("boxplot01", 6 ,6)
    # Set layout for 4 graphs
    AA.lo <- setLayout(width=rep(1.25, 4), height=4, xtop=1.5)

    ... remainder omitted ...

    The functions setSweave and setLayout are functions within the
    USGSgraphs library that I am building. As I said, the script runs just
    fine when I run interactively. I do have a version of the USGSgraphs
    library attached when I run the script. When I run R CMD check, or R CMD
    build on the source, I get the error:

    Error: processing vignette 'boxplots.Rnw' failed with diagnostics:
    chunk 1
    Error in eval(expr, envir, enclos) : could not find function "setSweave"
    Execution halted

    I would have expected that the contents of the library being built would
    have been available to the script (based on section 1.3.2 in R-exts.html).
    Barring that, I expected that "%\VignetteDepends{USGSgraphs}" would have
    made those functions available to the script. I have found that if I
    include this code before the first real example, it will run.

    <<echoúLSE>>> library(USGSgraphs)
    @

    But I have no reason to believe that it would run on any system that did
    not already have a version of USGSgraphs installed. Note that if I use the
    default graphics output, then I get the same error on setLayout.
    I use the default process for building the vignettes--no makefile.
    What do I need to do to get this vignette to run? I know I can set
    BuildVignettes to FALSE in the DESCRIPTION file or try to build with the
    --no-vignettes option, but that does not really address the more general
    issue.
    Vignettes are run in pretty standard R sessions when the package is
    built, so you need the explicit library(USGSgraphs) call to make
    functions from your package available in that session. What the manual
    is saying is just that that call will succeed: R will install a copy of
    your package in order to build the vignette. It goes into a "temporary
    library tree", because you might not want that particular version
    installed in your main tree.

    Duncan Murdoch
  • David L Lorenz at Sep 4, 2012 at 2:54 pm
    Duncan,
    That does make sense. But then what does %\VignetteDepends{USGSgraphs}
    do if it does not actually make the library available from the temporary
    library tree?
    Dave



    From:
    Duncan Murdoch <murdoch.duncan@gmail.com>
    To:
    David L Lorenz <lorenz@usgs.gov>
    Cc:
    r-devel@r-project.org
    Date:
    09/04/2012 09:34 AM
    Subject:
    Re: [Rd] Vignette question


    On 04/09/2012 9:47 AM, David L Lorenz wrote:
    All,
    I seem to be missing some key point about the construction of
    vignettes.
    I have created a vignette script that works fine when run interactively
    using Sweave. The top part of the .Rnw file is below.

    \documentclass{article}
    \parskip 3pt
    \parindent 30pt
    %\VignetteIndexEntry{Box Plot Examples}
    %\VignetteDepends{USGSgraphs}

    \begin{document}

    \title{Box Plot Examples}

    \author{Dave Lorenz}

    \maketitle

    ... text omittted from this example ...
    <<echo=TRUE>>=
    # Generate a random sample for the box plot
    set.seed(27036)
    BP <- rchisq(32, 3)
    # setSweave is a specialized function that sets up the graphics page
    setSweave("boxplot01", 6 ,6)
    # Set layout for 4 graphs
    AA.lo <- setLayout(width=rep(1.25, 4), height=4, xtop=1.5)

    ... remainder omitted ...

    The functions setSweave and setLayout are functions within the
    USGSgraphs library that I am building. As I said, the script runs just
    fine when I run interactively. I do have a version of the USGSgraphs
    library attached when I run the script. When I run R CMD check, or R CMD
    build on the source, I get the error:

    Error: processing vignette 'boxplots.Rnw' failed with diagnostics:
    chunk 1
    Error in eval(expr, envir, enclos) : could not find function "setSweave"
    Execution halted

    I would have expected that the contents of the library being built would
    have been available to the script (based on section 1.3.2 in
    R-exts.html).
    Barring that, I expected that "%\VignetteDepends{USGSgraphs}" would have
    made those functions available to the script. I have found that if I
    include this code before the first real example, it will run.

    <<echo=FALSE>>=
    library(USGSgraphs)
    @

    But I have no reason to believe that it would run on any system that did
    not already have a version of USGSgraphs installed. Note that if I use the
    default graphics output, then I get the same error on setLayout.
    I use the default process for building the vignettes--no makefile.
    What do I need to do to get this vignette to run? I know I can set
    BuildVignettes to FALSE in the DESCRIPTION file or try to build with the
    --no-vignettes option, but that does not really address the more general
    issue.
    Vignettes are run in pretty standard R sessions when the package is
    built, so you need the explicit library(USGSgraphs) call to make
    functions from your package available in that session. What the manual
    is saying is just that that call will succeed: R will install a copy of
    your package in order to build the vignette. It goes into a "temporary
    library tree", because you might not want that particular version
    installed in your main tree.

    Duncan Murdoch
  • Duncan Murdoch at Sep 4, 2012 at 5:40 pm

    On 04/09/2012 10:54 AM, David L Lorenz wrote:
    Duncan,
    That does make sense. But then what does %\VignetteDepends{USGSgraphs}
    do if it does not actually make the library available from the temporary
    library tree?
    It's not documented to do anything. I think it is supported purely for
    back compatibility.

    Duncan Murdoch
    Dave



    From:
    Duncan Murdoch <murdoch.duncan@gmail.com>
    To:
    David L Lorenz <lorenz@usgs.gov>
    Cc:
    r-devel at r-project.org
    Date:
    09/04/2012 09:34 AM
    Subject:
    Re: [Rd] Vignette question


    On 04/09/2012 9:47 AM, David L Lorenz wrote:
    All,
    I seem to be missing some key point about the construction of
    vignettes.
    I have created a vignette script that works fine when run interactively
    using Sweave. The top part of the .Rnw file is below.

    \documentclass{article}
    \parskip 3pt
    \parindent 30pt
    %\VignetteIndexEntry{Box Plot Examples}
    %\VignetteDepends{USGSgraphs}

    \begin{document}

    \title{Box Plot Examples}

    \author{Dave Lorenz}

    \maketitle

    ... text omittted from this example ...
    <<echo=TRUE>>> > # Generate a random sample for the box plot
    set.seed(27036)
    BP <- rchisq(32, 3)
    # setSweave is a specialized function that sets up the graphics page
    setSweave("boxplot01", 6 ,6)
    # Set layout for 4 graphs
    AA.lo <- setLayout(width=rep(1.25, 4), height=4, xtop=1.5)

    ... remainder omitted ...

    The functions setSweave and setLayout are functions within the
    USGSgraphs library that I am building. As I said, the script runs just
    fine when I run interactively. I do have a version of the USGSgraphs
    library attached when I run the script. When I run R CMD check, or R CMD
    build on the source, I get the error:

    Error: processing vignette 'boxplots.Rnw' failed with diagnostics:
    chunk 1
    Error in eval(expr, envir, enclos) : could not find function "setSweave"
    Execution halted

    I would have expected that the contents of the library being built would
    have been available to the script (based on section 1.3.2 in
    R-exts.html).
    Barring that, I expected that "%\VignetteDepends{USGSgraphs}" would have
    made those functions available to the script. I have found that if I
    include this code before the first real example, it will run.

    <<echoúLSE>>> > library(USGSgraphs)
    @

    But I have no reason to believe that it would run on any system that did
    not already have a version of USGSgraphs installed. Note that if I use the
    default graphics output, then I get the same error on setLayout.
    I use the default process for building the vignettes--no makefile.
    What do I need to do to get this vignette to run? I know I can set
    BuildVignettes to FALSE in the DESCRIPTION file or try to build with the
    --no-vignettes option, but that does not really address the more general
    issue.
    Vignettes are run in pretty standard R sessions when the package is
    built, so you need the explicit library(USGSgraphs) call to make
    functions from your package available in that session. What the manual
    is saying is just that that call will succeed: R will install a copy of
    your package in order to build the vignette. It goes into a "temporary
    library tree", because you might not want that particular version
    installed in your main tree.

    Duncan Murdoch

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupr-devel @
categoriesr
postedSep 3, '12 at 5:43a
activeSep 6, '12 at 1:33p
posts43
users12
websiter-project.org
irc#r

People

Translate

site design / logo © 2022 Grokbase