FAQ
Hello,

This is a proposition for a new XML file for the REST server, which can
speed up the 'list-upgrades' command significantly.


First off, I like REST: It's a conceptually simple idea and it works
great. The only current disavdvantage is that some PEAR commands need to
open a lot of different XML-files, which makes it slow.

Let's investigate the 'list-upgrades' command, which shows if there is
an upgrade available for a package you have installed. In
REST/10.php::listLatestUpgrades()
p/packages.xml is fetched (~10k)
for every package you have installed:
r/<package>/allreleases.xml is fetched (~1k)
r/<package/<state>.xml is fetched (poss. cached) (~5B)
done

A minimal amount of data is downloaded, _but_ the amount of files opened
grows with the amount of packages installed.
If you have 4 packages installed (everybody has) you download up to 9
files each time(~14k). If you have 15 packages installed, you download
up to 31 files each time(~25k) !
I will presume you know the cost of opening/closing connections and
files, it is significant, and certainly in todays broadband times.

The listLatestUpgrades() command uses only the package names, and the
different states with its latest version of these packages. For one
server, this information could be gathered in one file, eg.
latestpackages.xml
<!-- example xml file with packages and latest version info -->
<!-- full descriptive names used for clarity -->
<channel>pear.php.net</channel>
<package>
<name>Example_Tias</name>
<latest>
<row><state>stable</state><version>1.3.2</version></row>
<row><state>beta</state><version>1.4.0a2</version></row>
</latest>
</package>
<package>
...

This file would contain for every package its name, and every 'highest'
state which has a different version number. The latest is needed for
state-friendly package management: only the highest state and its
version are shown, if a lower state has a higher version than that one,
it is shown too.


This file would be at most 3 times as big as the current allpackages.xml
file (thus ~30k), yet listLatestUpgrades() would only need 1 file to
download: a significant speedup.


If there are any questions, do ask.
Greetings,
Tias

Search Discussions

  • Igor Feghali at Mar 26, 2007 at 11:15 pm
  • David Coallier at Mar 26, 2007 at 11:25 pm
    +1 always good to improve performances :)
    On 3/26/07, Igor Feghali wrote:
    +1

    --
    PEAR Development Mailing List (http://pear.php.net/)
    To unsubscribe, visit: http://www.php.net/unsub.php

    --
    David Coallier,
    Founder & Software Architect,
    Agora Production (http://agoraproduction.com)
    51.42.06.70.18
  • Christian Weiske at Mar 27, 2007 at 5:12 am

    This is a proposition for a new XML file for the REST server, which can
    speed up the 'list-upgrades' command significantly.
    It would be good to speed up list-all as well.
    This command does retrieve about 800 files currently..

    --
    Regards/Mit freundlichen Grüßen
    Christian Weiske
  • David Coallier at Mar 27, 2007 at 5:19 am
    I think it'd be pretty easy to speed-up list-all using a REST
    webservice as well but having all the results cached. This way we
    could just parse an xml file. I mean, it's not very expensive to parse
    an xml file of 800 packages.. list-all could perhaps retrieve the REST
    result and store it on localhost, then parse the data from there.
    That'd be pretty effective imho (unless one has a 56k connection of
    course)

    However, I mentionned this but I have no clue currently how the pear
    installer is actually retrieving this, perhaps this could be something
    greg could tell us more about.

    If I see no answers by tomorow, I'll start giving a look and propose a
    solution before friday.

    :)
    On 3/27/07, Christian Weiske wrote:
    This is a proposition for a new XML file for the REST server, which can
    speed up the 'list-upgrades' command significantly.
    It would be good to speed up list-all as well.
    This command does retrieve about 800 files currently..

    --
    Regards/Mit freundlichen Grüßen
    Christian Weiske


    --
    David Coallier,
    Founder & Software Architect,
    Agora Production (http://agoraproduction.com)
    51.42.06.70.18
  • Gregory Beaver at Mar 27, 2007 at 5:31 am

    David Coallier wrote:
    I think it'd be pretty easy to speed-up list-all using a REST
    webservice as well but having all the results cached. This way we
    could just parse an xml file. I mean, it's not very expensive to parse
    an xml file of 800 packages.. list-all could perhaps retrieve the REST
    result and store it on localhost, then parse the data from there.
    That'd be pretty effective imho (unless one has a 56k connection of
    course)

    However, I mentionned this but I have no clue currently how the pear
    installer is actually retrieving this, perhaps this could be something
    greg could tell us more about.

    If I see no answers by tomorow, I'll start giving a look and propose a
    solution before friday.
    Hi,

    I just spent about 5 hours fixing subtle but potentially serious
    problems/BC breaks in another bugfix for PEAR that was related to
    package installation, could we please put any discussion of changes to
    the REST/installer infrastructure off until PEAR 1.6.0, and most
    definitely until after the elections are over (May 1)?

    I would really appreciate this, things are just too crazy right now, and
    I don't think I can handle (mentally) a broken installer + revamping the
    entire political structure of PEAR at the same time. :)

    Thanks,
    Greg
  • David Coallier at Mar 27, 2007 at 5:37 am
    Haha yeah sure :) it's true, we've got enough work on the plate
    already :) Let's wait until after the elections, that might keep us
    alive :)
    On 3/27/07, Gregory Beaver wrote:
    David Coallier wrote:
    I think it'd be pretty easy to speed-up list-all using a REST
    webservice as well but having all the results cached. This way we
    could just parse an xml file. I mean, it's not very expensive to parse
    an xml file of 800 packages.. list-all could perhaps retrieve the REST
    result and store it on localhost, then parse the data from there.
    That'd be pretty effective imho (unless one has a 56k connection of
    course)

    However, I mentionned this but I have no clue currently how the pear
    installer is actually retrieving this, perhaps this could be something
    greg could tell us more about.

    If I see no answers by tomorow, I'll start giving a look and propose a
    solution before friday.
    Hi,

    I just spent about 5 hours fixing subtle but potentially serious
    problems/BC breaks in another bugfix for PEAR that was related to
    package installation, could we please put any discussion of changes to
    the REST/installer infrastructure off until PEAR 1.6.0, and most
    definitely until after the elections are over (May 1)?

    I would really appreciate this, things are just too crazy right now, and
    I don't think I can handle (mentally) a broken installer + revamping the
    entire political structure of PEAR at the same time. :)

    Thanks,
    Greg

    --
    David Coallier,
    Founder & Software Architect,
    Agora Production (http://agoraproduction.com)
    51.42.06.70.18
  • Gregory Beaver at Mar 27, 2007 at 4:16 pm
    Hi,

    This is not going to be forgotten. FYI:

    http://pear.php.net/bugs/roadmap.php?roadmapdetail=1.6.0&package=PEAR

    I've added a roadmap TODO item for this request. Thanks for your
    understanding and patience.

    Greg

    Tias wrote:
    Hello,

    This is a proposition for a new XML file for the REST server, which can
    speed up the 'list-upgrades' command significantly.


    First off, I like REST: It's a conceptually simple idea and it works
    great. The only current disavdvantage is that some PEAR commands need to
    open a lot of different XML-files, which makes it slow.

    Let's investigate the 'list-upgrades' command, which shows if there is
    an upgrade available for a package you have installed. In
    REST/10.php::listLatestUpgrades()
    p/packages.xml is fetched (~10k)
    for every package you have installed:
    r/<package>/allreleases.xml is fetched (~1k)
    r/<package/<state>.xml is fetched (poss. cached) (~5B)
    done

    A minimal amount of data is downloaded, _but_ the amount of files opened
    grows with the amount of packages installed.
    If you have 4 packages installed (everybody has) you download up to 9
    files each time(~14k). If you have 15 packages installed, you download
    up to 31 files each time(~25k) !
    I will presume you know the cost of opening/closing connections and
    files, it is significant, and certainly in todays broadband times.

    The listLatestUpgrades() command uses only the package names, and the
    different states with its latest version of these packages. For one
    server, this information could be gathered in one file, eg.
    latestpackages.xml
    <!-- example xml file with packages and latest version info -->
    <!-- full descriptive names used for clarity -->
    <channel>pear.php.net</channel>
    <package>
    <name>Example_Tias</name>
    <latest>
    <row><state>stable</state><version>1.3.2</version></row>
    <row><state>beta</state><version>1.4.0a2</version></row>
    </latest>
    </package>
    <package>
    ...

    This file would contain for every package its name, and every 'highest'
    state which has a different version number. The latest is needed for
    state-friendly package management: only the highest state and its
    version are shown, if a lower state has a higher version than that one,
    it is shown too.


    This file would be at most 3 times as big as the current allpackages.xml
    file (thus ~30k), yet listLatestUpgrades() would only need 1 file to
    download: a significant speedup.


    If there are any questions, do ask.
    Greetings,
    Tias

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppear-dev @
categoriesphp
postedMar 26, '07 at 10:25p
activeMar 27, '07 at 4:16p
posts8
users6
websitepear.php.net

People

Translate

site design / logo © 2018 Grokbase