Grokbase Groups Perl qa February 2015
FAQ
Hi,

This Saturday at FOSDEM in the hallway I had some discussions with
leont, Tux and later also with .. oh I guess that was RJBS? I did not
introduce myself, very bad. Hi!

Basically I think the whole CPAN setup with 200+ mirrors sounded great
back in the 1990s and it is still widely touted as a feature of CPAN.
But I'm a bit concerned about package integrity. Most Linux
distributions (where the packages and ISOs are typically LOTS bigger)
who use mirrors have a system in place where they verify their
packages with GPG keys. If you do that then having many mirrors
outside of your control using plain HTTP is not a problem, but Perl
does not *really* have something like that. Yeah of course there is
the signatures list, which is GPG signed, but this signature is not
checked 'out of the box' as far as I know.

So assuming you can't really verify the integrity of a module on a
mirror from the client, I think it would be best not to use any
mirrors.
As far as I know, with StrawberryPerl or a client like cpanm, you only
use one mirror anyway. Maybe the parties involved can share how much
bandwith it takes them to see if it would be feasible to switch to
*one* source for CPAN with possibly a CDN underneath. The metacpan
seems to have a decent CDN now, has SSL certificates and a complete
index. I think they should be able to handle the additional data, but
this is just based on my gut feeling of scale of the thing, average
dist size, and such and not on actual facts.

The other problem is how to securely connect to the mirror. There is
no support for SSL in core perl. But I think in many cases, it would
be an acceptable solution to install IO::Socket::SSL from your linux
distro's distribution, and then have the CPAN client 'auto-select' the
https version of the cpan mirror. If desired the CPAN client could
complain about not having SSL when IO::Socket::SSL is not installed.

Please let me know if this would be feasible and what your concerns would be.

I'd be willing to contribute patches to the cpanpm client to use HTTPS
if available, and to rip out the mirrorlist stuff.
--
Michiel

Search Discussions

  • Mike Doherty at Feb 4, 2015 at 12:16 am
    Doesn't cpan know how to use curl or wget if the system has it installed?
    Probably easier to bootstrap TLS support in perl that way.

    -Mike
    On Feb 3, 2015 2:26 PM, "Michiel Beijen" wrote:

    Hi,

    This Saturday at FOSDEM in the hallway I had some discussions with
    leont, Tux and later also with .. oh I guess that was RJBS? I did not
    introduce myself, very bad. Hi!

    Basically I think the whole CPAN setup with 200+ mirrors sounded great
    back in the 1990s and it is still widely touted as a feature of CPAN.
    But I'm a bit concerned about package integrity. Most Linux
    distributions (where the packages and ISOs are typically LOTS bigger)
    who use mirrors have a system in place where they verify their
    packages with GPG keys. If you do that then having many mirrors
    outside of your control using plain HTTP is not a problem, but Perl
    does not *really* have something like that. Yeah of course there is
    the signatures list, which is GPG signed, but this signature is not
    checked 'out of the box' as far as I know.

    So assuming you can't really verify the integrity of a module on a
    mirror from the client, I think it would be best not to use any
    mirrors.
    As far as I know, with StrawberryPerl or a client like cpanm, you only
    use one mirror anyway. Maybe the parties involved can share how much
    bandwith it takes them to see if it would be feasible to switch to
    *one* source for CPAN with possibly a CDN underneath. The metacpan
    seems to have a decent CDN now, has SSL certificates and a complete
    index. I think they should be able to handle the additional data, but
    this is just based on my gut feeling of scale of the thing, average
    dist size, and such and not on actual facts.

    The other problem is how to securely connect to the mirror. There is
    no support for SSL in core perl. But I think in many cases, it would
    be an acceptable solution to install IO::Socket::SSL from your linux
    distro's distribution, and then have the CPAN client 'auto-select' the
    https version of the cpan mirror. If desired the CPAN client could
    complain about not having SSL when IO::Socket::SSL is not installed.

    Please let me know if this would be feasible and what your concerns would
    be.

    I'd be willing to contribute patches to the cpanpm client to use HTTPS
    if available, and to rip out the mirrorlist stuff.
    --
    Michiel
  • Michiel Beijen at Feb 4, 2015 at 7:18 am
    Absolutely correct! I forgot about that; it is used for perl < 5.14 where
    there is no HTTP::Tiny or LWP in core. And yes, this would be the best way
    to go about this I think.

    Can anyone fill in on the feasibility of directing all cpan cients to *one*
    site, i.e. https://cpan.metacpan.org/ ?
    --
    Michiel

    Op woensdag 4 februari 2015 heeft Mike Doherty <mike@mikedoherty.ca> het
    volgende geschreven:
    Doesn't cpan know how to use curl or wget if the system has it installed?
    Probably easier to bootstrap TLS support in perl that way.

    -Mike
    On Feb 3, 2015 2:26 PM, "Michiel Beijen" wrote:

    Hi,

    This Saturday at FOSDEM in the hallway I had some discussions with
    leont, Tux and later also with .. oh I guess that was RJBS? I did not
    introduce myself, very bad. Hi!

    Basically I think the whole CPAN setup with 200+ mirrors sounded great
    back in the 1990s and it is still widely touted as a feature of CPAN.
    But I'm a bit concerned about package integrity. Most Linux
    distributions (where the packages and ISOs are typically LOTS bigger)
    who use mirrors have a system in place where they verify their
    packages with GPG keys. If you do that then having many mirrors
    outside of your control using plain HTTP is not a problem, but Perl
    does not *really* have something like that. Yeah of course there is
    the signatures list, which is GPG signed, but this signature is not
    checked 'out of the box' as far as I know.

    So assuming you can't really verify the integrity of a module on a
    mirror from the client, I think it would be best not to use any
    mirrors.
    As far as I know, with StrawberryPerl or a client like cpanm, you only
    use one mirror anyway. Maybe the parties involved can share how much
    bandwith it takes them to see if it would be feasible to switch to
    *one* source for CPAN with possibly a CDN underneath. The metacpan
    seems to have a decent CDN now, has SSL certificates and a complete
    index. I think they should be able to handle the additional data, but
    this is just based on my gut feeling of scale of the thing, average
    dist size, and such and not on actual facts.

    The other problem is how to securely connect to the mirror. There is
    no support for SSL in core perl. But I think in many cases, it would
    be an acceptable solution to install IO::Socket::SSL from your linux
    distro's distribution, and then have the CPAN client 'auto-select' the
    https version of the cpan mirror. If desired the CPAN client could
    complain about not having SSL when IO::Socket::SSL is not installed.

    Please let me know if this would be feasible and what your concerns would
    be.

    I'd be willing to contribute patches to the cpanpm client to use HTTPS
    if available, and to rip out the mirrorlist stuff.
    --
    Michiel
  • Cosimo Streppone at Feb 4, 2015 at 9:30 am

    On 04. feb. 2015 08:17, Michiel Beijen wrote:

    Can anyone fill in on the feasibility of directing all cpan cients to
    *one* site, i.e. https://cpan.metacpan.org/ ?
    Having multiple mirrors is IMO one of the many things
    that CPAN got right from the start.

    Other similar but centralized package repositories
    have failed (and continue to fail) miserably.
    Why go centralised if the problem is elsewhere?

    CDNs, while being distributed, are managed centrally
    by one entity, who also pays the bandwidth/service cost.

    --
    Cosimo
  • Michiel Beijen at Feb 4, 2015 at 9:37 am
    Hi Cosimo,
    On Wed, Feb 4, 2015 at 10:29 AM, Cosimo Streppone wrote:
    On 04. feb. 2015 08:17, Michiel Beijen wrote:

    Can anyone fill in on the feasibility of directing all cpan cients to
    *one* site, i.e. https://cpan.metacpan.org/ ?
    Having multiple mirrors is IMO one of the many things
    that CPAN got right from the start.

    Other similar but centralized package repositories
    have failed (and continue to fail) miserably.
    Why go centralised if the problem is elsewhere?

    CDNs, while being distributed, are managed centrally
    by one entity, who also pays the bandwidth/service cost.
    Yeah of course, it should not be one **host** - but it can still be
    one URL which leverages a CDN right? Are you saying "CDN, bad,
    mirrors, good?

    I understand this might mean more bandwidth cost for the one who pays
    the bills - therefore I've asked about how we could find out about the
    amount of traffic that would be involved.

    If bandwidth would be much of a concern it could also be a possibility
    to get the checksums for the dists from one source via HTTPS, perform
    the download from a mirror and then verify the checksum.

    --
    Michiel
  • Cosimo Streppone at Feb 4, 2015 at 10:28 am

    On 04. feb. 2015 10:36, Michiel Beijen wrote:
    Hi Cosimo,
    On Wed, Feb 4, 2015 at 10:29 AM, Cosimo Streppone wrote:
    On 04. feb. 2015 08:17, Michiel Beijen wrote:

    Can anyone fill in on the feasibility of directing all cpan cients to
    *one* site, i.e. https://cpan.metacpan.org/ ?
    Having multiple mirrors is IMO one of the many things
    that CPAN got right from the start.
    [...]
    CDNs, while being distributed, are managed centrally
    by one entity, who also pays the bandwidth/service cost.
    Yeah of course, it should not be one **host** - but it can still be
    one URL which leverages a CDN right? Are you saying "CDN, bad,
    mirrors, good?
    Of course I am not saying that :-)

    Just, it's easy to look at the neighbor's garden and think
    the grass is greener.

    I've been in other gardens, and it's often not the case :)

    /C
  • David Cantrell at Feb 4, 2015 at 11:46 am

    On Tue, Feb 03, 2015 at 11:25:26PM +0100, Michiel Beijen wrote:

    Basically I think the whole CPAN setup with 200+ mirrors sounded great
    back in the 1990s and it is still widely touted as a feature of CPAN.
    Having a zillion mirrors is no longer a killer feature - the net is now
    much better connected, bandwidth is cheap, and site reliability is much
    higher than it used to be. However, the ability to easily create a
    mirror is still a nifty feature. It makes it dead easy to:

    * have a mirror on my laptop for hacking on the move;
    * have a customised module repository where all the normal tools "just
       work"

    The latter is really important. It lets companies add their non-public
    code to a CPAN mirror-a-like. It lets you "pin" some of your
    dependencies to particular versions. It lets you do things like the
    cpXXXan.

    --
    David Cantrell | Godless Liberal Elitist

    " In My Egotistical Opinion, most people's ... programs should be
       indented six feet downward and covered with dirt. "
           --Blair P. Houghton
  • Michiel Beijen at Feb 4, 2015 at 12:41 pm
    Hi David,
    On Wed, Feb 4, 2015 at 12:46 PM, David Cantrell wrote:

    Having a zillion mirrors is no longer a killer feature - the net is now
    much better connected, bandwidth is cheap, and site reliability is much
    higher than it used to be. However, the ability to easily create a
    mirror is still a nifty feature. It makes it dead easy to:

    * have a mirror on my laptop for hacking on the move;
    * have a customised module repository where all the normal tools "just
    work"

    The latter is really important. It lets companies add their non-public
    code to a CPAN mirror-a-like. It lets you "pin" some of your
    dependencies to particular versions. It lets you do things like the
    cpXXXan.
    I'm not saying that all mirrors should go, and I'm not saying that you
    should not be able to insert your own servers (or file locations) in
    your urllist! That's a useful feature and should absolutely stay.

    What I'm saying is that I think the *default* out-of-box setup should
    go use some central SSL-enabled website - which now, on latest CPAN,
    uses http://www.cpan.org by default.
    --
    Michiel

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupqa @
categoriesperl
postedFeb 3, '15 at 10:25p
activeFeb 4, '15 at 12:41p
posts8
users4
websiteqa.perl.org

People

Translate

site design / logo © 2019 Grokbase