FAQ
Hello, these tests fail for me too, ubuntu 11.04 x86, both trunk and
5.4 branches.

2011/9/12 Ferenc Kovacs <tyra3l@gmail.com>:
On Sun, Sep 11, 2011 at 11:04 PM, Stas Malyshev wrote:
Hi!
On 9/11/11 8:44 AM, Ferenc Kovacs wrote:

AFAIK you shoud get it(as I did on my debian machines), as both
display_startup_errors and error_reporting is force enabled for the
tests:
http://svn.php.net/viewvc/php/php-src/trunk/run-tests.php?view=markup#l232
http://svn.php.net/viewvc/php/php-src/trunk/run-tests.php?view=markup#l230
Well, maybe I should, but I am not. If you think it's a bug and have a fix
you're welcome to propose it.
I did, and you just reverted it, this is why I'm curious. :)
could you please run the following (basically running those 2 tests
with verbose output):
TEST_PHP_EXECUTABLE=auto TEST_PHP_CGI_EXECUTABLE=auto ./sapi/cli/php
run-tests.php -g 'SKIP,FAIL,XFAIL' -n -c tmp-php.ini -v
ext/session/tests/rfc1867_invalid_settings*.phpt
I would like to know how does your COMMAND look like.
mine is something like:
/home/tyrael/checkouts/php-src/trunk/sapi/cli/php  -n -c 'tmp-php.ini'
-d "output_handler=" -d "open_basedir=" -d "safe_mode=0" -d
"disable_functions=" -d "output_buffering=Off" -d
"error_reporting=32767" -d "display_errors=1" -d
"display_startup_errors=1" -d "log_errors=0" -d "html_errors=0" -d
"track_errors=1" -d "report_memleaks=1" -d "report_zend_debug=0" -d
"docref_root=" -d "docref_ext=.html" -d "error_prepend_string=" -d
"error_append_string=" -d "auto_prepend_file=" -d "auto_append_file="
-d "magic_quotes_runtime=0" -d "ignore_repeated_errors=0" -d
"precision=14" -d "unicode.runtime_encoding=ISO-8859-1" -d
"unicode.script_encoding=UTF-8" -d "unicode.output_encoding=UTF-8" -d
"unicode.from_error_mode=U_INVALID_SUBSTITUTE" -d
"session.auto_start=0" -d "tidy.clean_output=0" -d
"zlib.output_compression=Off" -d "mbstring.func_overload=0" -d
"session.upload_progress.freq=-1" -f
"/home/tyrael/checkouts/php-src/trunk/ext/session/tests/rfc1867_invalid_settings.php"
2>&1

and the second thing:
could you run the following:
./sapi/cli/php -n -d session.upload_progress.freq=200% -d
error_reporting=-1 -d display_errors=1 -d display_startup_errors=1  -r
''
this outputs for me:
PHP Warning:  PHP Startup: session.upload_progress.freq cannot be over
100% in Unknown on line 0

Warning: PHP Startup: session.upload_progress.freq cannot be over 100%
in Unknown on line 0

I verified this with multiple php version on multiple machines, if you
have both display_errors and display_startup_errors (and appropriate
error_reporting level), you should see those two lines.
btw: you can see from the reports that others also seeing this behaviour:
http://qa.php.net/reports/viewreports.php?version=5.4.0beta1-dev&test=%2Fext%2Fsession%2Ftests%2Frfc1867_invalid_settings.phpt
http://qa.php.net/reports/viewreports.php?version=5.4.0beta1-dev&test=%2Fext%2Fsession%2Ftests%2Frfc1867_invalid_settings_2.phpt

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php


--
Regards,
Shein Alexey

Search Discussions

  • Ferenc Kovacs at Sep 12, 2011 at 6:18 pm
    I forget to reply-all to the list :/
    On Mon, Sep 12, 2011 at 7:46 PM, Stas Malyshev wrote:
    Hi!
    On 9/12/11 3:14 AM, Ferenc Kovacs wrote:

    you should pass both -n and -c as the make test pass those to the
    run-tests.php AFAIK, see:
    That's not what I see happening.
    as I mentioned above, you should have the -n -c arguments, if you
    don't then this can cause the differencies.
    That's not what I see run-tests is doing.
    what do you mean by "my php.ini"? for the test runs you shouldn't use
    any external php.ini, as it can cause differences between the test
    results.
    Maybe it shouldn't, but that's what happens. It doesn't use neither -c nor
    -n option.
    before we continue further with debugging, could someone else verify
    my reasoning, or repeat the those test steps?
    I'm pretty sure that the make test/run-test.php should run the tests
    without any external php.inis except the tmp-php.ini (passing -n -c
    tmp-php.ini).
    it is a possibility that Stas somehow borked up his environment, but
    as the gcov machine also generated the same output (at least for those
    session tests) I would like to know that what causes this different
    behavior for Stas and me, and which is the correct.
    thanks.

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu
  • Alexey Shein at Sep 13, 2011 at 2:13 pm
    Just wanted to say, that I managed to reproduce your results for
    ./PHP_5_4/ext/session/tests/rfc1867_invalid_settings.phpt by adding
    date.timezone=UTC
    error_log=file
    settings into --INI-- section, this way test passes. I.e. logged
    message should go into the file and do not mess the output which is
    grabbed by the test and date.timezone setting prevents another startup
    warning about unconfigured timezone.
    Hope it helps.

    2011/9/12 Ferenc Kovacs <tyra3l@gmail.com>:
    I forget to reply-all to the list :/
    On Mon, Sep 12, 2011 at 7:46 PM, Stas Malyshev wrote:
    Hi!
    On 9/12/11 3:14 AM, Ferenc Kovacs wrote:

    you should pass both -n and -c as the make test pass those to the
    run-tests.php AFAIK, see:
    That's not what I see happening.
    as I mentioned above, you should have the -n -c arguments, if you
    don't then this can cause the differencies.
    That's not what I see run-tests is doing.
    what do you mean by "my php.ini"? for the test runs you shouldn't use
    any external php.ini, as it can cause differences between the test
    results.
    Maybe it shouldn't, but that's what happens. It doesn't use neither -c nor
    -n option.
    before we continue further with debugging, could someone else verify
    my reasoning, or repeat the those test steps?
    I'm pretty sure that the make test/run-test.php should run the tests
    without any external php.inis except the tmp-php.ini (passing -n -c
    tmp-php.ini).
    it is a possibility that Stas somehow borked up his environment, but
    as the gcov machine also generated the same output (at least for those
    session tests) I would like to know that what causes this different
    behavior for Stas and me, and which is the correct.
    thanks.

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu

    --
    PHP Internals - PHP Runtime Development Mailing List
    To unsubscribe, visit: http://www.php.net/unsub.php


    --
    Regards,
    Shein Alexey
  • Ferenc Kovacs at Sep 13, 2011 at 2:49 pm
    After a discussion on irc, I was thinking about something similar:
    adding my patch back, but adding -error-log to the --INI--
    Stas, I still don't know why do have a custom php.ini, and why is it used.
    The make test should create tmp-php.ini for you (and copy your
    additional ini files there if you passed --with-config-file-scan-dir
    for the build) and pass -n -c tmp-php.ini as an argument to the
    invidual tests.
    On Tue, Sep 13, 2011 at 4:12 PM, Alexey Shein wrote:
    Just wanted to say, that I managed to reproduce your results for
    ./PHP_5_4/ext/session/tests/rfc1867_invalid_settings.phpt by adding
    date.timezone=UTC
    error_log=file
    settings into --INI-- section, this way test passes. I.e. logged
    message should go into the file and do not mess the output which is
    grabbed by the test and date.timezone setting prevents another startup
    warning about unconfigured timezone.
    Hope it helps.

    2011/9/12 Ferenc Kovacs <tyra3l@gmail.com>:
    I forget to reply-all to the list :/
    On Mon, Sep 12, 2011 at 7:46 PM, Stas Malyshev wrote:
    Hi!
    On 9/12/11 3:14 AM, Ferenc Kovacs wrote:

    you should pass both -n and -c as the make test pass those to the
    run-tests.php AFAIK, see:
    That's not what I see happening.
    as I mentioned above, you should have the -n -c arguments, if you
    don't then this can cause the differencies.
    That's not what I see run-tests is doing.
    what do you mean by "my php.ini"? for the test runs you shouldn't use
    any external php.ini, as it can cause differences between the test
    results.
    Maybe it shouldn't, but that's what happens. It doesn't use neither -c nor
    -n option.
    before we continue further with debugging, could someone else verify
    my reasoning, or repeat the those test steps?
    I'm pretty sure that the make test/run-test.php should run the tests
    without any external php.inis except the tmp-php.ini (passing -n -c
    tmp-php.ini).
    it is a possibility that Stas somehow borked up his environment, but
    as the gcov machine also generated the same output (at least for those
    session tests) I would like to know that what causes this different
    behavior for Stas and me, and which is the correct.
    thanks.

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu

    --
    PHP Internals - PHP Runtime Development Mailing List
    To unsubscribe, visit: http://www.php.net/unsub.php


    --
    Regards,
    Shein Alexey


    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu
  • Stas Malyshev at Sep 13, 2011 at 4:23 pm
    Hi!
    On 9/13/11 7:49 AM, Ferenc Kovacs wrote:
    Stas, I still don't know why do have a custom php.ini, and why is it used.
    The make test should create tmp-php.ini for you (and copy your
    Everybody has custom php.ini, what do you mean "why"? Nobody uses just
    defaults. It is used because -n -c is not passed when you run run-tests.
    additional ini files there if you passed --with-config-file-scan-dir
    for the build) and pass -n -c tmp-php.ini as an argument to the
    invidual tests.
    That is not what I see happening. I've already sent you the actual
    command line, there's no -n and -c there.
    --
    Stanislav Malyshev, Software Architect
    SugarCRM: http://www.sugarcrm.com/
    (408)454-6900 ext. 227
  • Pierre Joye at Sep 13, 2011 at 4:27 pm
    hi,
    On Tue, Sep 13, 2011 at 6:23 PM, Stas Malyshev wrote:
    Hi!
    On 9/13/11 7:49 AM, Ferenc Kovacs wrote:

    Stas, I still don't know why do have a custom php.ini, and why is it used.
    The make test should create tmp-php.ini for you (and copy your
    Everybody has custom php.ini, what do you mean "why"? Nobody uses just
    defaults. It is used because -n -c is not passed when you run run-tests.
    Right, but the goal of these tests is to test behavior under certain
    conditions. If a test can't be executed in a generic way then its
    setting must be changed. It seems to be the case with this one.

    Cheers,
  • Richard Quadling at Sep 14, 2011 at 11:32 am

    On 13 September 2011 17:27, Pierre Joye wrote:
    hi,
    On Tue, Sep 13, 2011 at 6:23 PM, Stas Malyshev wrote:
    Hi!
    On 9/13/11 7:49 AM, Ferenc Kovacs wrote:

    Stas, I still don't know why do have a custom php.ini, and why is it used.
    The make test should create tmp-php.ini for you (and copy your
    Everybody has custom php.ini, what do you mean "why"? Nobody uses just
    defaults. It is used because -n -c is not passed when you run run-tests.
    Right, but the goal of these tests is to test behavior under certain
    conditions. If a test can't be executed in a generic way then its
    setting must be changed. It seems to be the case with this one.
    Hi all.

    Surely the test must first determine if it is an appropriate test for
    the environment.

    Just like we have tests check to see if an extension is available
    before running then tests that are tuned to a specific value from a
    php.ini file (default or custom) must first check that the test is
    appropriate to the value being tested.

    As a daft example.

    If you were testing that the data written to a session file was
    correct, the format of the serialisation, as defined in
    session.serialize_handler, has to be taken into consideration. If the
    php.ini said to use WDDX, testing the "default" format would be
    incorrect.

    So, if there are radically different outcomes, depending upon the
    values read from INI files (default or custom), then multiple tests
    are needed for each value, or the test needs to branch accordingly and
    indicate that a particular value is untested - highlighting unexpected
    changes in the default value (maybe).


    With reference to the discussion a while ago about what PHP considers
    the "right" settings for php.ini, in my mind the "default" settings
    for testing should be the production values. I think this is the case
    in 5.4, that is running tests with no php.ini is the equivalent to
    running with php.ini-production which is the defaults built into the
    engine. If not, then each developer will be having a different INI
    file and testing will be for different value and test "fixes" on one
    platform could result in a failure on another.


    If a test requires a different INI file, then it should set it itself.
    If it is the case that every developer creates the same ini file for a
    specific value when testing, then maybe the default value is
    incorrect.

    As I see it, testing for different values in INI files can only be
    done by having multiple tests or by a single test generating multiple
    sub-tests to check for the known values.

    Richard.

    --
    Richard Quadling
    Twitter : EE : Zend : PHPDoc
    @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY : bit.ly/lFnVea
  • Ferenc Kovacs at Sep 13, 2011 at 6:04 pm

    On Tue, Sep 13, 2011 at 6:23 PM, Stas Malyshev wrote:
    Hi!
    On 9/13/11 7:49 AM, Ferenc Kovacs wrote:

    Stas, I still don't know why do have a custom php.ini, and why is it used.
    The make test should create tmp-php.ini for you (and copy your
    Everybody has custom php.ini, what do you mean "why"? Nobody uses just
    defaults. It is used because -n -c is not passed when you run run-tests.
    not everybody, but I guess you are right that most people has a system
    wide php installed with a custom php.ini on the machine that runs the
    test suite.
    to name an exception: I have a dedicate vm for building php and
    running the tests, and there is no system wide php installed there,
    hence no custom php.ini.
    as I mentioned before, make test (at least it should) passes the -n -c
    argument to run-tests.php
    see http://svn.php.net/viewvc/php/php-src/trunk/Makefile.global?view=markup#l104
    which means if you run make test that can produce different result,
    than simply running run-tests.php
    additional ini files there if you passed --with-config-file-scan-dir
    for the build) and pass -n -c tmp-php.ini as an argument to the
    invidual tests.
    That is not what I see happening. I've already sent you the actual command
    line, there's no -n and -c there.
    And I already mentioned and pointed out, that make test should pass
    those, the test target in the Makefile is pretty straightforward, so I
    can't see how can your make test not pass -n -c tmp-php.ini

    For the record, before our discussion I wasn't aware that the
    tmp-php.ini built by the make test can have "random" values(through
    --with-config-file-scan-dir) and I expected that most of the test
    results are generated through make test.
    Now knowing this, I now understand that theoretically each test should
    define every --INI-- value (except those set on
    http://svn.php.net/viewvc/php/php-src/trunk/run-tests.php?view=markup#l224
    ) which could influence the result of the test.
    Even stuff like memory_limit, as I could have a custom php.ini with
    memory_limit=1

    I'm pretty sure that I could screw up 99% of test results with a
    properly adjusted php.ini.

    I think that it would be a good idea to set up some baseline(or extend
    http://svn.php.net/viewvc/php/php-src/trunk/run-tests.php?view=markup#l224
    ) php.ini for the test suite, otherwise we will test not the testsuite
    but the configuration. (somebody mentioned that it is intentional, but
    I think it is a bad practice)

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu
  • Stas Malyshev at Sep 13, 2011 at 6:23 pm
    Hi!
    On 9/13/11 11:04 AM, Ferenc Kovacs wrote:
    as I mentioned before, make test (at least it should) passes the -n -c
    argument to run-tests.php
    see http://svn.php.net/viewvc/php/php-src/trunk/Makefile.global?view=markup#l104
    which means if you run make test that can produce different result,
    than simply running run-tests.php
    That needs to be fixed, make test and run-tests.php should run exactly
    the same code and produce identical results.
    I'm pretty sure that I could screw up 99% of test results with a
    properly adjusted php.ini.
    This needs to be fixed too - tests should have values they depend on
    either in command line or in INI section.

    --
    Stanislav Malyshev, Software Architect
    SugarCRM: http://www.sugarcrm.com/
    (408)454-6900 ext. 227
  • Ferenc Kovacs at Sep 13, 2011 at 6:38 pm

    On Tue, Sep 13, 2011 at 8:23 PM, Stas Malyshev wrote:
    Hi!
    On 9/13/11 11:04 AM, Ferenc Kovacs wrote:

    as I mentioned before, make test (at least it should) passes the -n -c
    argument to run-tests.php
    see
    http://svn.php.net/viewvc/php/php-src/trunk/Makefile.global?view=markup#l104
    which means if you run make test that can produce different result,
    than simply running run-tests.php
    That needs to be fixed, make test and run-tests.php should run exactly the
    same code and produce identical results.
    I'm pretty sure that I could screw up 99% of test results with a
    properly adjusted php.ini.
    This needs to be fixed too - tests should have values they depend on either
    in command line or in INI section.
    agree.
    I think that it is relevant to mention, that we discussed on irc that
    maybe we should create a wrapper script for running the testsuite.
    so we could create a make-test.sh(and a make-test.bat), and we could
    move the logic from the Makefile there.
    so if you run make test, that would only execute the shell script,
    which would execute the run-tests.php with the proper arguments.
    this would also mean that if you run ./run-tests or make test, you
    would get the same result, and the advanced users could still use
    run-tests.php if they want, but we would have a good baseline.
    this would also solve the problem, that on some platforms the
    run-tests.php won't work with the default configuration (AFAIR
    Christian Weiske, Gustavo Lopes and Alexey Shein all run into the
    issue, that they php.ini from their distribution didn't had the 'E' in
    variables_order, so the $_ENV superglobal wasn't populated which
    caused the run-tests.php to ignore the environment variables :/).

    btw: most features which is available through arguments of
    run-tests.php also available through environmental variables for make
    test, but AFAIK there is no documentation, so I think that most people
    who uses run-tests.php directly isn't really need it, but
    run-tests.php has a --help option, while make test has not.

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu
  • Ferenc Kovacs at Sep 13, 2011 at 8:11 pm

    On Tue, Sep 13, 2011 at 8:23 PM, Stas Malyshev wrote:
    Hi!
    On 9/13/11 11:04 AM, Ferenc Kovacs wrote:

    as I mentioned before, make test (at least it should) passes the -n -c
    argument to run-tests.php
    see
    http://svn.php.net/viewvc/php/php-src/trunk/Makefile.global?view=markup#l104
    which means if you run make test that can produce different result,
    than simply running run-tests.php
    That needs to be fixed, make test and run-tests.php should run exactly the
    same code and produce identical results.
    I'm pretty sure that I could screw up 99% of test results with a
    properly adjusted php.ini.
    This needs to be fixed too - tests should have values they depend on either
    in command line or in INI section.
    Stas, I commited my patch again with the addition --INI-- setting:
    http://news.php.net/php.cvs/66511
    could you test it?

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu
  • Stas Malyshev at Sep 13, 2011 at 8:19 pm
    Hi!
    On 9/13/11 1:11 PM, Ferenc Kovacs wrote:
    Stas, I commited my patch again with the addition --INI-- setting:
    http://news.php.net/php.cvs/66511
    could you test it?
    Passes for me.
    --
    Stanislav Malyshev, Software Architect
    SugarCRM: http://www.sugarcrm.com/
    (408)454-6900 ext. 227
  • Alexey Shein at Sep 14, 2011 at 10:03 am

    2011/9/13 Ferenc Kovacs <tyra3l@gmail.com>:
    On Tue, Sep 13, 2011 at 6:23 PM, Stas Malyshev wrote:
    Hi!
    On 9/13/11 7:49 AM, Ferenc Kovacs wrote:

    Stas, I still don't know why do have a custom php.ini, and why is it used.
    The make test should create tmp-php.ini for you (and copy your
    Everybody has custom php.ini, what do you mean "why"? Nobody uses just
    defaults. It is used because -n -c is not passed when you run run-tests.
    not everybody, but I guess you are right that most people has a system
    wide php installed with a custom php.ini on the machine that runs the
    test suite.
    to name an exception: I have a dedicate vm for building php and
    running the tests, and there is no system wide php installed there,
    hence no custom php.ini.
    as I mentioned before, make test (at least it should) passes the -n -c
    argument to run-tests.php
    see http://svn.php.net/viewvc/php/php-src/trunk/Makefile.global?view=markup#l104
    which means if you run make test that can produce different result,
    than simply running run-tests.php
    Why do we even have this tmp-php.ini? Why not just make test without
    any .ini files, i.e. just with -n option?
    additional ini files there if you passed --with-config-file-scan-dir
    for the build) and pass -n -c tmp-php.ini as an argument to the
    invidual tests.
    That is not what I see happening. I've already sent you the actual command
    line, there's no -n and -c there.
    And I already mentioned and pointed out, that make test should pass
    those, the test target in the Makefile is pretty straightforward, so I
    can't see how can your make test not pass -n -c tmp-php.ini

    For the record, before our discussion I wasn't aware that the
    tmp-php.ini built by the make test can have "random" values(through
    --with-config-file-scan-dir) and I expected that most of the test
    results are generated through make  test.
    Now knowing this, I now understand that theoretically each test should
    define every --INI-- value (except those set on
    http://svn.php.net/viewvc/php/php-src/trunk/run-tests.php?view=markup#l224
    ) which could influence the result of the test.
    Even stuff like memory_limit, as I could have a custom php.ini with
    memory_limit=1

    I'm pretty sure that I could screw up 99% of test results with a
    properly adjusted php.ini.

    I think that it would be a good idea to set up some baseline(or extend
    http://svn.php.net/viewvc/php/php-src/trunk/run-tests.php?view=markup#l224
    ) php.ini for the test suite, otherwise we will test not the testsuite
    but the configuration. (somebody mentioned that it is intentional, but
    I think it is a bad practice)

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu


    --
    Regards,
    Shein Alexey
  • Ferenc Kovacs at Sep 14, 2011 at 10:17 am

    Why do we even have this tmp-php.ini? Why not just make test without
    any .ini files, i.e. just with -n option?
    [20:44:09] <bjori> Tyrael: tmp-php.ini is built from the system ini
    [20:44:45] <bjori> Tyrael: if he builds php with
    --with-config-file-scan-dir=/where/ever/he/has/another/php.ini, then
    it will use that ini
    [20:45:09] <bjori> Tyrael: and thats the whole point of tmp-php.ini,
    to not only test the default ini, but also random user configurations

    I agree that it is a bad practice to test random user configuration in
    our test suite.
    that's just another moving part in the system.

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu
  • Alexey Shein at Sep 14, 2011 at 10:37 am

    2011/9/14 Ferenc Kovacs <tyra3l@gmail.com>:

    Why do we even have this tmp-php.ini? Why not just make test without
    any .ini files, i.e. just with -n option?
    [20:44:09] <bjori> Tyrael: tmp-php.ini is built from the system ini
    [20:44:45] <bjori> Tyrael: if he builds php with
    --with-config-file-scan-dir=/where/ever/he/has/another/php.ini, then
    it will use that ini
    [20:45:09] <bjori> Tyrael: and thats the whole point of tmp-php.ini,
    to not only test the default ini, but also random user configurations

    I agree that it is a bad practice to test random user configuration in
    our test suite.
    that's just another moving part in the system.
    I understand how it works, maybe should we do then this behavior
    optional (if user wants to test his current configuration) and default
    make test will run without any php.inis?
    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu


    --
    Regards,
    Shein Alexey
  • Hannes Magnusson at Sep 15, 2011 at 10:07 am

    On Wed, Sep 14, 2011 at 12:36, Alexey Shein wrote:
    2011/9/14 Ferenc Kovacs <tyra3l@gmail.com>:
    Why do we even have this tmp-php.ini? Why not just make test without
    any .ini files, i.e. just with -n option?
    [20:44:09] <bjori> Tyrael: tmp-php.ini is built from the system ini
    [20:44:45] <bjori> Tyrael: if he builds php with
    --with-config-file-scan-dir=/where/ever/he/has/another/php.ini, then
    it will use that ini
    [20:45:09] <bjori> Tyrael: and thats the whole point of tmp-php.ini,
    to not only test the default ini, but also random user configurations

    I agree that it is a bad practice to test random user configuration in
    our test suite.
    that's just another moving part in the system.
    I understand how it works, maybe should we do then this behavior
    optional (if user wants to test his current configuration) and default
    make test will run without any php.inis?
    No?

    For tests that require certain ini options we have the --INI-- section.
    For all other tests, we need to ensure they work in all environments,
    be it under openbasedir, different precision,
    call_time_pass_reference, variable order, and what have you.

    And we do want to load the shared extensions the user is using to test
    them too and ensure there isn't any new unexpected conflicts.


    Running tests exclusively in an environment known to work doesn't help us.

    When you write a test case you need to know what you are testing. Just
    writing a test to write one doesn't give us any benefits.
    We can easily write test cases that give us 90%+ code coverage, but
    that has absolutely no meaning if it doesn't actually test anything.

    -Hannes
  • Pierre Joye at Sep 15, 2011 at 11:18 am

    On Thu, Sep 15, 2011 at 12:07 PM, Hannes Magnusson wrote:

    Running tests exclusively in an environment known to work doesn't help us.
    It indeed does, and much more than random failures in random environments.

    However it is not about one or the other, it is about both and about
    having as much tests environments as possible. Unit tests (phpt)
    should be in controlled environment to be useful. If a case is not
    tested, then a new test is required. This is the only to have usable
    delta (if something is broken :) between commits or releases. Ideally
    we will have many of such environments running our phpts, but the
    configuration of each of them must be known and constants to be really
    useful.

    It is also not possible to imagine all possible cases and that's why
    further testing are required, like application testing and unit tests
    of these applications. For example we do run applications as part of
    the CI in our labs, many bugs or regressions have been found this way,
    some of them have made it back to the core as simple test case. But
    this is not always possible (cross requests issue or complex cases).

    Cheers,
  • Alexey Shein at Sep 16, 2011 at 8:40 am

    2011/9/15 Hannes Magnusson <hannes.magnusson@gmail.com>:
    On Wed, Sep 14, 2011 at 12:36, Alexey Shein wrote:
    2011/9/14 Ferenc Kovacs <tyra3l@gmail.com>:
    Why do we even have this tmp-php.ini? Why not just make test without
    any .ini files, i.e. just with -n option?
    [20:44:09] <bjori> Tyrael: tmp-php.ini is built from the system ini
    [20:44:45] <bjori> Tyrael: if he builds php with
    --with-config-file-scan-dir=/where/ever/he/has/another/php.ini, then
    it will use that ini
    [20:45:09] <bjori> Tyrael: and thats the whole point of tmp-php.ini,
    to not only test the default ini, but also random user configurations

    I agree that it is a bad practice to test random user configuration in
    our test suite.
    that's just another moving part in the system.
    I understand how it works, maybe should we do then this behavior
    optional (if user wants to test his current configuration) and default
    make test will run without any php.inis?
    No?

    For tests that require certain ini options we have the --INI-- section.
    For all other tests, we need to ensure they work in all environments,
    be it under openbasedir, different precision,
    call_time_pass_reference, variable order, and what have you.

    And we do want to load the shared extensions the user is using to test
    them too and ensure there isn't any new unexpected conflicts.


    Running tests exclusively in an environment known to work doesn't help us.
    Agreed. But you should cross the line somewhere. The test must be
    repeatable or it won't be useful, you will have all kinds of erratic
    tests (described for example here:
    http://xunitpatterns.com/Erratic%20Test.html).
    How do you know if the test failed for true or is that user borked his
    php.ini? It could create a lot of false negative bug reports on
    qa.php.net (but it's no problem since nobody is looking there :) )
    Maybe run failed tests again without user's php.ini?
    When you write a test case you need to know what you are testing. Just
    writing a test to write one doesn't give us any benefits.
    We can easily write test cases that give us 90%+ code coverage, but
    that has absolutely no meaning if it doesn't actually test anything.

    -Hannes
    --
    Regards,
    Shein Alexey
  • Ferenc Kovacs at Sep 26, 2011 at 11:33 pm
    I've just noticed that we had a related bugreport, I thought that I
    should mention it:
    https://bugs.php.net/bug.php?id=55479

    --
    Ferenc Kovács
    @Tyr43l - http://tyrael.hu

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupphp-internals @
categoriesphp
postedSep 12, '11 at 10:57a
activeSep 26, '11 at 11:33p
posts19
users6
websitephp.net

People

Translate

site design / logo © 2022 Grokbase