Grokbase Groups Perl qa May 2011
FAQ

[Perl-QA] RFC: Private CPAN In A Box

Jeffrey Thalhammer
May 20, 2011 at 7:51 am
NOTE: This was also posted on perlmonks at http://perlmonks.org/?node_id=905878. I'm trying to reach a wide audience, so I'm posting it here too.

Over the last few years, I've helped build private CPANs (DarkPANs or DPANs as brian d foy calls them) for 3 different organizations. Each time I cobbled together some combination of different CPAN::Site, CPAN::Mini::Inject and CPAN modules with various shell scripts, commit hooks, and cron jobs. Although they were generally effective, I feel they were clunky, highly specialized, and hard to maintain.

Once again, I'm faced with building another private CPAN. But this time, I have an opportunity to build something that could have broader appeal in the Perl community. In fact, the explicit goal is to produce an open source, turnkey framework for creating, deploying, and maintaining a private CPAN.

So with that in mind, I'm looking for feedback on what you might want from such a framework. Here are some questions I've been asking myself -- hopefully they will help stir your mind:

How would it provide an identifiable & reproducible stack of dependencies that developers can use to write, test, and deploy their code against?
How would it allow one to define and deliver a standardized set of dependencies to all your environments?
How would it enable developers to experiment with dependencies that are not part of the standard environment?
How would it enable different teams to work against different versions of their dependencies?
How would it enable teams to independently upgrade, add, and remove their dependencies in a controlled, reproducible manner?
How would teams use it to distribute and share their own modules and applications within the organization?
Which parts of the larger CPAN ecosystem would be most valuable in a private CPAN (e.g. CPAN Testers, AnnoCPAN, search.cpan.org, CPAN Ratings)?
How would one migrate their legacy code into CPAN-style distributions?
How would one migrate from their existing dependency management infrastructure to a private CPAN?
How might one want incorporate a private CPAN with their other development infrastructure, such as bug trackers & continuous integration servers?
Thanks for sharing your thoughts!

-Jef
reply

Search Discussions

39 responses

  • David E. Wheeler at May 20, 2011 at 3:08 pm

    On May 20, 2011, at 3:51 AM, Jeffrey Thalhammer wrote:

    Once again, I'm faced with building another private CPAN. But this time, I have an opportunity to build something that could have broader appeal in the Perl community. In fact, the explicit goal is to produce an open source, turnkey framework for creating, deploying, and maintaining a private CPAN.
    FWIW, I've written one, called PGXN. The various parts are managed here:

    https://github.com/pgxn/

    PGXN::Manager == PAUSE
    PGXN::API && PGXN::Site == search.cpan.org
    pgxnclient = CPAN.pm

    The client was actually written by someone else, in Python, just using the API.

    https://github.com/dvarrazzo/pgxnclient/

    He was able to do it pretty quickly because almost everything is represented by a static JSON file. The mirror and API APIs are documented here:

    https://github.com/pgxn/pgxn-api/wiki

    Might be of interest to you, FWIW. I wrote it for PostgreSQL Extensions, but tried to implement a generalized solution that could be used by any language or community -- or privately, if you like.
    So with that in mind, I'm looking for feedback on what you might want from such a framework. Here are some questions I've been asking myself -- hopefully they will help stir your mind:

    How would it provide an identifiable & reproducible stack of dependencies that developers can use to write, test, and deploy their code against?
    Do you mean dependencies to build the framework itself, or how to identify dependencies in distributions released to the network? META.json works great for both of these, IMHO.
    How would it allow one to define and deliver a standardized set of dependencies to all your environments?
    Not sure I understand the question here…
    How would it enable developers to experiment with dependencies that are not part of the standard environment?
    What is "the standard environment"?
    How would it enable different teams to work against different versions of their dependencies?
    You mean other than just preserving earlier versions of distributions?
    How would it enable teams to independently upgrade, add, and remove their dependencies in a controlled, reproducible manner?
    I think your use of the term "dependencies" confuses me. What are "dependencies"?
    How would teams use it to distribute and share their own modules and applications within the organization?
    I don't understand. You upload your module or app to the upload server.
    Which parts of the larger CPAN ecosystem would be most valuable in a private CPAN (e.g. CPAN Testers, AnnoCPAN, search.cpan.org, CPAN Ratings)?
    That's really going to depend on the requirements of the organization, I would think. Some will think testing is important; others that ratings or wikis are important.
    How would one migrate their legacy code into CPAN-style distributions?
    Fortunately, all one needs to do is add a META.json file and you're read to go. Unless you want a single method of installing distributions, in which case you would either need to build an installation client that recognizes and executes many different approaches to building, or get your users to adopt one.
    How would one migrate from their existing dependency management infrastructure to a private CPAN?
    Again, just define dependencies in the META.json file.
    How might one want incorporate a private CPAN with their other development infrastructure, such as bug trackers & continuous integration servers?
    Provide a robust API. :-)
    Thanks for sharing your thoughts!
    HTH, sorry I get a bit lost with some of your questions.

    Best,

    David
  • Leo Lapworth at May 20, 2011 at 3:30 pm
    Hi Jeffrey,
    On 20 May 2011 08:51, Jeffrey Thalhammer wrote:
    NOTE: This was also posted on perlmonks at http://perlmonks.org/?node_id=905878.  I'm trying to reach a wide audience, so I'm posting it here too.

    Over the last few years, I've helped build private CPANs (DarkPANs or DPANs as brian d foy calls them) for 3 different organizations. Each time I cobbled together some combination of different CPAN::Site, CPAN::Mini::Inject and CPAN modules with various shell scripts, commit hooks, and cron jobs. Although they were generally effective, I feel they were clunky, highly specialized, and hard to maintain.
    You might want to check out:

    http://beta.metacpan.org/ the front and backend are here:
    https://github.com/cpan-api

    http://blogs.perl.org/mt/mt-search.fcgi?limit=20&search=metacpan has
    some history and discuss with them on irc.perl.org #metacpan - I've
    pointed them to your perlmonk post

    https://github.com/szabgab/CPAN-Digger also exists - I think Gabour
    is discussing if there is any overlap with #metacpan which they're
    discussing.

    http://blogs.perl.org/users/michael_j/2011/04/deploying-perl-code-with-git-locallib-minicpan-and-cpanminus.html
    is actually the setup we use at work, but this only works because we
    only have one version of all code for all projects.

    Hope that helps.

    Leo
  • Jeffrey Thalhammer at May 20, 2011 at 7:04 pm

    On May 20, 2011, at 8:07 AM, David E. Wheeler wrote:

    HTH, sorry I get a bit lost with some of your questions.

    No, this is very helpful. I think you've shown that I need to take a step back and look at the underlying problem.

    I'm not attempting to build an all-encompasing dependency management system. I've tried to avoid thinking about it in terms of CPAN as we know it, since in theory, that is an implementation detail. But in practice, I'm will stipulate that a "dependency" is just a Perl module in some CPAN repository.

    As I see it, there are two sides to the problem. The first is creating tools for creating, managing, and using a private CPAN repository. There is already a lot of prior art for this, but I feel that it is fragmented. So this might just be a matter of assembling the right combination of existing technologies.

    The second (and probably more difficult) side is establishing patterns for using these tools, which enable developers to write, test, and deploy their code against an identifiable and reproducible set of dependencies. I acknowledge that one size will not fit all, but I strongly believe there is (or should be) some common ground here.

    The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1]. This creates problems for developers that want to use the CPAN tool chain, but need to have a stable set of dependencies. At the same time, they need flexibility to evolve those dependencies in a systematic way.

    Thanks for bringing PGXN to my attention. The architecture is very similar to what I was thinking. I will dig deeper to see if we can leverage PGXN.

    [1] I think a lot of this trouble would go away if the CPAN tool chain simply permitted authors to express precisely which $VERSION of something they require. So I have to wonder if that is the right place to focus, rather than building something on top of the tool chain.


    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • Jeffrey Thalhammer at May 20, 2011 at 7:04 pm

    On May 20, 2011, at 8:29 AM, Leo Lapworth wrote:

    http://beta.metacpan.org/
    https://github.com/szabgab/CPAN-Digger
    Thanks! MetaCPAN and CPAN::Digger are definitely part of the picture. But I see these more as adjunct features of a private CPAN. I think the meat of the problem lies in how one actually utilizes a private CPAN in the development cycle.
    http://blogs.perl.org/users/michael_j/2011/04/deploying-perl-code-with-git-locallib-minicpan-and-cpanminus.html
    That's exactly what I'm talking about. I think this kind of pattern is popping up in a lot of organizations. Somehow, I want to try and package that idea in a way that might appeal to a wide audience.

    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • David Cantrell at May 20, 2011 at 8:01 pm

    On Fri, May 20, 2011 at 11:04:24AM -0700, Jeffrey Thalhammer wrote:

    The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1].
    http://cp2010-05an.barnyard.co.uk/ and all its date/time-stamped friends
    and relations aren't.

    A feature I keep meaning to add is some way for users to pick one of my
    mirrors and then further restrict it so that, for example, you might
    want the CPAN as at the beginning of 2011 but with Some::Module (and all
    its pre-requisites) stuck at some earlier point in time, or even a way
    of applying that to an up-to-date CPAN mirror (an up-to-date mirror is
    just a special case of my mirrors anyway, one with no filtering at all).

    I want to do this in a way that:

    * won't require manual intervention from me to set up a new restriction
    * will work with any CPAN client
    * is efficient with storage and CPU
    * provides free ponies and kittens

    --
    David Cantrell | Minister for Arbitrary Justice
  • Jeffrey Thalhammer at May 23, 2011 at 6:52 pm
    After looking at David Wheeler's PGXN and David Cantrell's CPAN snapshots, I've come up with a story. Tell me if you think this makes sense (or not):

    Historically, CPAN mirrors have always been static files. In other words, the paths in the 02packages file always pointed directly to some physical file on disk. This made it possible to access mirrors the with the older ftp:// and file:// protocols supported by CPAN.pm. It also made synchronizing the mirrors easy with tools like rsync.

    This has worked well for a long time. But the need to support older protocols limits the flexibility of a CPAN mirror. If you drop support for ftp:// and file://, and just focus on http:// then things can become more interesting. This open the door for a CPAN server (like the PGXN::Manager) that can do all sorts of things. For example, it could serve up distros that are spread across multiple remote machines. Or they could be stored in version control system or database. Or you could present a view of the mirror that meets certain criteria, such as by time, or cpantesters results, or by author, or by kwalitee scores, or by some other metadata (like http://cp2011-05an.barnyard.co.uk/). The concept of a CPAN server also provides a convenient hook for plugging in other services, like a private PAUSE or MetaCPAN.

    I'm not saying that *the* CPAN should work this way. I'm just suggesting that a *private* CPAN could work this way.

    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • David E. Wheeler at May 23, 2011 at 9:22 pm

    On May 20, 2011, at 11:04 AM, Jeffrey Thalhammer wrote:

    I'm not attempting to build an all-encompasing dependency management system. I've tried to avoid thinking about it in terms of CPAN as we know it, since in theory, that is an implementation detail. But in practice, I'm will stipulate that a "dependency" is just a Perl module in some CPAN repository.
    Yeah, that's what's confusing me. To me, a "dependency" is a Perl module in some CPAN repository that the module I'm trying to install right now depends on. That is, it's listed in in the "prereqs" section of a META.json file.
    The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1]. This creates problems for developers that want to use the CPAN tool chain, but need to have a stable set of dependencies. At the same time, they need flexibility to evolve those dependencies in a systematic way.
    Well, the new version spec in CPAN Meta Spec 2.0 helps with this. You can specify very precise requirements.
    Thanks for bringing PGXN to my attention. The architecture is very similar to what I was thinking. I will dig deeper to see if we can leverage PGXN.

    [1] I think a lot of this trouble would go away if the CPAN tool chain simply permitted authors to express precisely which $VERSION of something they require. So I have to wonder if that is the right place to focus, rather than building something on top of the tool chain.
    You can.

    http://search.cpan.org/~dagolden/CPAN-Meta-2.110930/lib/CPAN/Meta/Spec.pm#Version_Ranges

    Best,

    David
  • David E. Wheeler at May 23, 2011 at 9:27 pm

    On May 23, 2011, at 11:52 AM, Jeffrey Thalhammer wrote:

    Historically, CPAN mirrors have always been static files. In other words, the paths in the 02packages file always pointed directly to some physical file on disk. This made it possible to access mirrors the with the older ftp:// and file:// protocols supported by CPAN.pm. It also made synchronizing the mirrors easy with tools like rsync.
    Yep. PGXN is the same, although I've built an API server that has a dynamic interface for searching.
    This has worked well for a long time. But the need to support older protocols limits the flexibility of a CPAN mirror. If you drop support for ftp:// and file://, and just focus on http:// then things can become more interesting. This open the door for a CPAN server (like the PGXN::Manager) that can do all sorts of things. For example, it could serve up distros that are spread across multiple remote machines. Or they could be stored in version control system or database. Or you could present a view of the mirror that meets certain criteria, such as by time, or cpantesters results, or by author, or by kwalitee scores, or by some other metadata (like http://cp2011-05an.barnyard.co.uk/). The concept of a CPAN server also provides a convenient hook for plugging in other services, like a private PAUSE or MetaCPAN.

    I'm not saying that *the* CPAN should work this way. I'm just suggesting that a *private* CPAN could work this way.
    What I've tried to do with PGXN is to get the best of both worlds. At the lowest level, the mirror generated by PGXN::Manager should be very simple, with all static files. But many of those static files are JSON files, and so every mirror is a lightweight REST API server without any additional work required. Then anyone can build a service using the API. Or they can build new mirrors that do additional stuff with the data, which is what the API server does (adds additional info to some JSON files, generates HTML pages, manages full-text index, etc.).

    So take the static stuff, and build something dynamic on top of it, either using its API or by adding to it.

    Best,

    David
  • David Golden at May 23, 2011 at 9:36 pm
    The problem is that there is no way for a single application to use multiple
    versions of a module. So it's easy to imagine that very specific module
    prereqs could be in conflict and require manual override. That's not the
    right answer.

    What we need is application level dependencies that are specific and non
    conflicting and then to make it easy to install into app specific lib
    directories.

    That's what im
    On May 23, 2011 5:22 PM, "David E. Wheeler" wrote:
    On May 20, 2011, at 11:04 AM, Jeffrey Thalhammer wrote:

    I'm not attempting to build an all-encompasing dependency management
    system. I've tried to avoid thinking about it in terms of CPAN as we know
    it, since in theory, that is an implementation detail. But in practice, I'm
    will stipulate that a "dependency" is just a Perl module in some CPAN
    repository.
    Yeah, that's what's confusing me. To me, a "dependency" is a Perl module
    in some CPAN repository that the module I'm trying to install right now
    depends on. That is, it's listed in in the "prereqs" section of a META.json
    file.
    The trouble with CPANs is that they are a moving target -- modules are
    constantly updated, added, and removed[1]. This creates problems for
    developers that want to use the CPAN tool chain, but need to have a stable
    set of dependencies. At the same time, they need flexibility to evolve those
    dependencies in a systematic way.
    Well, the new version spec in CPAN Meta Spec 2.0 helps with this. You can
    specify very precise requirements.
    Thanks for bringing PGXN to my attention. The architecture is very
    similar to what I was thinking. I will dig deeper to see if we can leverage
    PGXN.
    [1] I think a lot of this trouble would go away if the CPAN tool chain
    simply permitted authors to express precisely which $VERSION of something
    they require. So I have to wonder if that is the right place to focus,
    rather than building something on top of the tool chain.
  • David Golden at May 23, 2011 at 9:47 pm

    On Monday, May 23, 2011, David Golden wrote:
    The problem is that there is no way for a single application to use multiple versions of a module.  So it's easy to imagine that very specific module prereqs could be in conflict and require manual override. That's not the right answer.

    What we need is application level dependencies that are specific and non conflicting and then to make it easy to install into app specific lib directories.
    That's what im
    The SEND button should be harder to press by accident...

    That's what I'm working on in June - trying to tie all the various
    lessons from other toolchain and meta tools into something that can be
    for Perl apps what cpan(p|m) is for Perl modules.

    I think people are trying to make cpan solve the application level
    problem and that's why "custom CPAN mirror" seems like the answer.

    David
  • Cosimo Streppone at May 24, 2011 at 12:38 am

    On Tue, 24 May 2011 07:22:02 +1000, David E. Wheeler wrote:
    On May 20, 2011, at 11:04 AM, Jeffrey Thalhammer wrote:

    [1] I think a lot of this trouble would go away if the CPAN tool chain
    simply permitted authors to express precisely which $VERSION of
    something they require.
    Hi David,
    Whaaat!? :-)

    Let's see if I understand this correctly.
    If, as an example, in this file,

    http://cpansearch.perl.org/src/COSIMO/Net-Prober-0.03/META.json

    I say:

    "prereqs" : {
    ...
    "runtime" : {
    "requires" : {
    "Net::Ping" : "<= 2.35"
    }
    },
    ...
    },

    Where current Net::Ping on CPAN is 2.36,
    than this will install Net::Ping 2.35 as
    prerequisite?

    Or, if Net::Ping on the system is already 2.36
    it will refuse to proceed?

    I have tried fiddling with Makefile.PL/META.yml/META.json
    without success.

    --
    Cosimo
  • David Golden at May 24, 2011 at 10:04 am

    On Mon, May 23, 2011 at 8:38 PM, Cosimo Streppone wrote:
    I say:

    "prereqs" : {
    ...
    "runtime" : {
    "requires" : {
    "Net::Ping" : "<= 2.35"
    }
    },
    ...
    },

    Where current Net::Ping on CPAN is 2.36,
    than this will install Net::Ping 2.35 as
    prerequisite?

    Or, if Net::Ping on the system is already 2.36
    it will refuse to proceed?
    I think the current CPAN clients will all refuse to proceed. This is
    what I meant when I said that version ranges don't accomplish the goal
    because the computer can't know what the right thing to do is.

    -- David
  • David Cantrell at May 24, 2011 at 10:55 am

    On Mon, May 23, 2011 at 11:52:39AM -0700, Jeffrey Thalhammer wrote:

    After looking at David Wheeler's PGXN and David Cantrell's CPAN snapshots, I've come up with a story. Tell me if you think this makes sense (or not):

    Historically, CPAN mirrors have always been static files. In other words, the paths in the 02packages file always pointed directly to some physical file on disk. This made it possible to access mirrors the with the older ftp:// and file:// protocols supported by CPAN.pm. It also made synchronizing the mirrors easy with tools like rsync.

    This has worked well for a long time. But the need to support older protocols limits the flexibility of a CPAN mirror. If you drop support for ftp:// and file://, and just focus on http:// then things can become more interesting. This open the door for a CPAN server (like the PGXN::Manager) that can do all sorts of things.
    Your mirror only has to support file:// if you want it to live on a
    user's local disk, and only has to support ftp:// if you want to give
    FTP access. I realise that this is obvious, but please bear with me :-)

    You can support whatever craziness you want over HTTP.

    The only place in CPAN::Config which mentions the mirror's URL is this:
    urllist => [q[http://cp5.6.2an.barnyard.co.uk/]],

    Crucially, it derives URLs for both the index and the distributions from
    that. So, if you have a magic mirror and are willing to do the heavy
    lifting on the server (and please bear in mind it's *very* heavy lifting -
    my daily rebuilds of static 02packages files take about 7 hours, and
    to build a mirror like cp2011-05an takes about 12 hours) you could
    simply configure CPAN.pm to use:
    urllist => [q[http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22]],

    CPAN.pm would then get its index by requesting:
    http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22/modules/02packages.details.txt.gz

    and you can, of course, construct whatever the hell you want in
    response, provided it has the right format. That file would normally
    read something like ...

    Description: This is a whitespace-seperated file.
    Description: Each line is modulename moduleversion filename.
    Line-Count: 33800
    Last-Updated: Tue, 24 May 2011 06:22:15 GMT

    AAAA::Crypt::DH 0.04 B/BI/BINGOS/AAAA-Crypt-DH-0.04.tar.gz
    ABI 1.0 M/MA/MALAY/ABI-1.0.tar.gz
    ACH::Builder 0.03 T/TK/TKEEFER/ACH-Builder-0.03.tar.gz
    ACME::Error 0.03 C/CW/CWEST/ACME-Error-0.03.tar.gz

    but there's no reason why the meat of it couldn't be stuff like ...

    AAAA::Crypt::DH 99999999 distcontaining/AAAA::Crypt::DH

    which CPAN.pm would resolve to ...

    http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22/authors/id/distcontaining/AAAA::Crypt::DH

    which, apart from the annoying 'authors/id/' in the middle, which could
    be trivially dealt with using mod_rewrite, looks deliciously RESTish.

    --
    David Cantrell | Reality Engineer, Ministry of Information

    Your call is important to me. To see if it's important to
    you I'm going to make you wait on hold for five minutes.
    All calls are recorded for blackmail and amusement purposes.
  • David E. Wheeler at May 24, 2011 at 4:14 pm

    On May 24, 2011, at 3:04 AM, David Golden wrote:

    Or, if Net::Ping on the system is already 2.36
    it will refuse to proceed?
    I think the current CPAN clients will all refuse to proceed. This is
    what I meant when I said that version ranges don't accomplish the goal
    because the computer can't know what the right thing to do is
    Ah, but sometime this summer the clients (and therefor the computer) will?

    Best,

    David
  • David Golden at May 24, 2011 at 4:19 pm

    On Tue, May 24, 2011 at 12:14 PM, David E. Wheeler wrote:
    On May 24, 2011, at 3:04 AM, David Golden wrote:

    Or, if Net::Ping on the system is already 2.36
    it will refuse to proceed?
    I think the current CPAN clients will all refuse to proceed.  This is
    what I meant when I said that version ranges don't accomplish the goal
    because the computer can't know what the right thing to do is
    Ah, but sometime this summer the clients (and therefor the computer) will?
    I doubt it. I have plans for an "application installer" that will
    pre-compute dependencies and reliably set up per-application lib
    directories. Different applications could have different versions of
    a module, but a single application will only have one version
    available to it.

    This is not something that the CPAN clients are really designed to do
    (though they can be made to do the right things with INSTALLBASE and
    so on).

    A better specification and design for application dependencies and
    installation will avoid the need to create N custom CPAN views as a
    hack to make CPAN clients try to be application deployment tools.

    -- David
  • David E. Wheeler at May 24, 2011 at 4:26 pm

    On May 24, 2011, at 9:18 AM, David Golden wrote:

    I doubt it. I have plans for an "application installer" that will
    pre-compute dependencies and reliably set up per-application lib
    directories. Different applications could have different versions of
    a module, but a single application will only have one version
    available to it.

    This is not something that the CPAN clients are really designed to do
    (though they can be made to do the right things with INSTALLBASE and
    so on).
    Well, are you using the same version ranges spec for the META.json file? Seems to me that this might result in a stand-alone library that, given prereqs, could determine what should be installed. Then the existing clients could use it too, eh?
    A better specification and design for application dependencies and
    installation will avoid the need to create N custom CPAN views as a
    hack to make CPAN clients try to be application deployment tools.
    Yeah, that'll be nice.

    Best,

    David
  • David Golden at May 24, 2011 at 8:17 pm

    On Tue, May 24, 2011 at 12:25 PM, David E. Wheeler wrote:
    Well, are you using the same version ranges spec for the META.json file? Seems to me that this might result in a stand-alone library that, given prereqs, could determine what should be installed. Then the existing clients could use it too, eh?
    That's the idea, though I've not worked out the details. But for
    example, I'd like to use the configuration output data (MYMETA.json)
    to determine three sets of modules (tarballs, really):

    - latest versions on CPAN
    - currently installed versions on developers machine
    - minimum versions on CPAN that satisfy the prereq specs

    It's possible that version ranges could conflict, in which case some
    or all of those might not be achievable. Ideally, the dependencies
    would be determined against a specific version of perl without
    additional libraries installed.

    Then, given one of those three ordered list of tarballs that satisfy
    all prereqs, it should be possibly to repeatably deploy an application
    with a known set of module versions, even as the "latest" on CPAN
    evolves.

    Of the three sets, users would have to decide which suit their
    specific needs/purpose, or could substitute versions, etc.

    That's my vision in a nutshell. I have some people in NY that I'm
    going to do some brainstorming with and then I'll try to get a proof
    of concept code out next month.

    -- David
  • Leon Timmermans at May 24, 2011 at 8:32 pm

    On Tue, May 24, 2011 at 10:17 PM, David Golden wrote:
    Then, given one of those three ordered list of tarballs that satisfy
    all prereqs, it should be possibly to repeatably deploy an application
    with a known set of module versions, even as the "latest" on CPAN
    evolves.
    That could get a bit troublesome with the current best practice to
    delete older versions of module from CPAN.

    Leon
  • David Golden at May 24, 2011 at 8:49 pm

    On Tue, May 24, 2011 at 4:32 PM, Leon Timmermans wrote:
    On Tue, May 24, 2011 at 10:17 PM, David Golden wrote:
    Then, given one of those three ordered list of tarballs that satisfy
    all prereqs, it should be possibly to repeatably deploy an application
    with a known set of module versions, even as the "latest" on CPAN
    evolves.
    That could get a bit troublesome with the current best practice to
    delete older versions of module from CPAN.
    http://backpan.perl.org/

    I don't remember when they started keeping the history -- I've seen
    stuff as far back as 1997.

    -- Davie
  • Jan Dubois at May 24, 2011 at 9:27 pm

    On Tue, 24 May 2011, David Golden wrote:
    On Tue, May 24, 2011 at 4:32 PM, Leon Timmermans wrote:
    On Tue, May 24, 2011 at 10:17 PM, David Golden wrote:
    Then, given one of those three ordered list of tarballs that satisfy
    all prereqs, it should be possibly to repeatably deploy an application
    with a known set of module versions, even as the "latest" on CPAN
    evolves.
    That could get a bit troublesome with the current best practice to
    delete older versions of module from CPAN.
    And a somewhat unfortunate practice, as it makes it hard to create diffs
    between arbitrary module versions. And with the new fast-mirror mechanism,
    I'm not sure that the aggressive pruning of CPAN should still be considered
    a "best" practice, as it makes the web-based diff on search.cpan.org
    pretty useless. For example when trying to understand this Locale-Codes
    changelog entry:
    3.16 2011-03-01 sbeck
    * NEW CODE(s)
    I would first try to look here:

    http://search.cpan.org/tools/Locale-Codes-3.16

    But it is not working because even the previous release has already been
    deleted.
    http://backpan.perl.org/

    I don't remember when they started keeping the history -- I've seen
    stuff as far back as 1997.
    Yes, backpan has the complete history with very few exceptions (I think
    some website scraping modules were deleted from backpan when the website
    owner threatened legal action).

    AFAICT there is no good index for backpan though. I have to audit diffs
    to older module versions quite often, so I have my own tools, and found
    that even the links are not comprehensive; I have to crawl both
    backpan/modules/by-module and backpan/modules/by-authors/id to make
    sure I find *all* the different versions.

    Cheers,
    -Jan
  • David Golden at May 24, 2011 at 9:35 pm

    On Tue, May 24, 2011 at 5:27 PM, Jan Dubois wrote:
    AFAICT there is no good index for backpan though.  I have to audit diffs
    to older module versions quite often, so I have my own tools, and found
    that even the links are not comprehensive; I have to crawl both
    backpan/modules/by-module and backpan/modules/by-authors/id to make
    sure I find *all* the different versions.
    I believe that brian foy created a pretty comprehensive one -- I think
    he may have actually investigated how module files changed from
    tarball to tarball. (E.g. Module $VERSION was the same in two
    tarballs, but the file MD5's are different).

    The hard problem is that PAUSE doesn't have any
    maintainer/co-maintainer history -- so given a historical
    DAGOLDEN/Foo-Bar-1.23.tar.gz and RJBS/Foo-Bar-1.23.tar.gz, there's no
    way to tell that one or the other was the official indexed version.
    You can look at who has maintainer rights *today* and assume from that
    who was authorized when the tarball was uploaded.

    I don't know (yet) how many collisions like that there are. I suspect
    it's a relatively small number that can be flagged and resolved with a
    bit of research and email.

    -- David
  • Jeffrey Thalhammer at May 25, 2011 at 5:20 am

    On May 24, 2011, at 1:17 PM, David Golden wrote:

    Then, given one of those three ordered list of tarballs that satisfy
    all prereqs, it should be possibly to repeatably deploy an application
    with a known set of module versions, even as the "latest" on CPAN
    evolves.
    .
    Wow! This is getting juicy.

    It's easy for me to get carried away with all the intricacies of dependency management. But for my task at hand, I think it might be the wrong problem. So for the sake of this discussion, let's put aside the issue of how a private CPAN (really, the 02packages) could be constructed to have a magical mixture of modules. Instead, I want to think about how a private CPAN mirror fits into the development cycle. This might be easiest by walking through a hypothetical scenario...

    Let's suppose I wan to write a new application in Perl. By "application", I really just mean some set of modules and scripts. And let's assume that my development, testing, and deployment environments all have a perfectly clean version of perl -- no modules have been added or updated from the core.

    I want my app to leverage the CPAN tool chain, so I setup a CPAN-style distribution for my source code. It has lib/, bin/, and t/ directories, and all the usual trappings. I'm also going to use Module::Build. And of course, all this is in some source control system (assume svn for now).

    After a couple of Mountain-Dews, I have some code and tests and everything passes. Commit. Now I want to add some feature that requires a module from CPAN. So I go shopping on search.cpan.org and find one I like called Frobulator::Simple. At this point, I might go ask my sysadmin to install Frobulator::Simple (and all its dependencies) in /usr/lib/perl5 on the development machine. Or if I'm smart, I might use local::lib to install it somewhere in my home directory. But these both suck, because changing /usr/lib/perl5 will affect everyone else, and who knows what else I might have installed in my home directory.

    So instead, let's say I add Frobulator::Simple to the "requires" list (or "build_requires", or whatever) in my Build.PL. Now I run the "installdeps" target, which fetches the latest Frobulator::Simple and all its dependencies from a public CPAN and installs them.

    Now this is where it gets interesting. Rather than install in site_perl, it installs into a directory right inside my project (let's call it "dlib" for "dependent libraries"). At the same time, it stashes the tar balls in a directory structure suitable for a CPAN mirror, which is also right inside my project directory (let's call it "lpan", for "local perl archive network"). Finally, it generates the 02packages for the lpan/. Everything in lpan/ will be placed under revision control too.

    Now I have all my application code and my dependencies in one local, revision-controlled location. When I run the "build" target, all my modules are copied into the blib/ directory as usual. And when I run the "test" target, both blib/ and dlib/ are placed on @INC. If the tests pass, then commit. If not, then fix my code. Or I can "svn revert" my lpan/ directory, change the "requires" list in Build.PL to try a new module, and run the "installdeps" target again. Rinse and repeat as needed.

    This goes on for a while as my application evolves until I have several dependencies. Dependencies that I want to keep are in the growing dpan/ directory and committed to svn. Meanwhile, the dlib directory is clouding up with all the modules I've experimented with. Periodically, I want to test the whole stack. So now I run the "realclean" target, which wipes out dlib/. Next, I run an "installdepslocal" target, which basically does an "installdeps", but this time it pulls everything from the lpan. This effectively flushes out any modules that fail since, over time, I may have upgraded some of their dependencies in my lpan/ and those upgrades may not be backward compatible.

    If some of the dependencies do fail, it is my job to figure out what to do. This might mean removing a newer distro from the lpan/ in favor of an older one. Or maybe I'll make a patched version of the distro and put it in the lpan/. Or perhaps I'll just switch to a different dependency altogether. How this all gets done is out of scope for this discussion, but I imagine that we could generate some kind of map that tracks all the dependencies for the app. This could help developers see the interdependencies and figure out how an upgrade or downgrade will impact the app. You could also have special build targets that assist with adding, removing, or upgrading specific sub-trees in the dependency graph.

    But for now, let's assume I've got a working stack.. When I run the "test" target again, I'm now testing my code against a pure set of dependencies. The traditional "disttest" target might also be enhanced to first run "installdepslocal" in the dist directory. Once my application code is golden, I run a "release" target that builds the tar.gz of my application and shoves into the lpan/. Again, the 02packages is automatically updated. Commit.

    Now let's say we use the CPAN tool chain for deployment too. My application and all it's dependencies are in the lpan/, so I can just login to the target host, point to the lpan in SVN, and say "cpan My::App". Again, we'd use local::lib to ensure everything does into an app-specific location. This is especially easy with SVN, since all the tarballs in the lpan can be accessed with HTTP requests. Or if we prefer, we can build the local lib/ directory on a separate host, archive it up, and ship it to the target that way.

    So far, I've described all this in terms of Module::Build targets, but some (perhaps all) of this could be done with Dist::Zilla plugins. One thing I like about this model is that it works well for team development, since all the dependencies converge in the lpan in svn. And since lpan is part of the project, it plays nicely with branching and merging. Also, I like it because it requires absolutely no administration -- each app has its own lpan/, which is totally controlled by the developers. Lastly, it doesn't require rejiggering the entire CPAN tool chain and/or backfilling CPAN distros with some new metadata.

    I realize this doesn't solve the "big" problem of automatically building a particular CPAN mirror with a magical mix modules. But based on my own development patterns, I think this model could provide a workable framework for organically evolving application dependencies, while providing a stable and reproducible dependency stack at the same time.

    What do you think?

    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • Cosimo Streppone at May 25, 2011 at 5:51 am

    On Wed, 25 May 2011 15:19:49 +1000, Jeffrey Thalhammer wrote:
    On May 24, 2011, at 1:17 PM, David Golden wrote:

    Then, given one of those three ordered list of tarballs that satisfy
    all prereqs, it should be possibly to repeatably deploy an application
    with a known set of module versions, even as the "latest" on CPAN
    evolves.
    [...]

    Let's suppose I wan to write a new application in Perl. By
    "application", [...]

    Rather than install in site_perl, it installs into a directory right
    inside my project (let's call it "dlib" for "dependent libraries").
    [...] it stashes the tar balls in a directory structure [...] which
    is also inside my project directory (let's call it "lpan" ...)
    Everything in lpan/ will be placed under revision control too.
    Interesting!
    What do you think?
    When you start a second project that uses a subset
    of project1 dependencies, what do you do?

    Do you duplicate the lpan/ and dlib/ folders?
    If you had to maintain patches, do you apply the patches
    to all your projects lpan/ and dlib/ folders?

    These dlib/ and lpan/ could be shared among projects,
    but they would lose, I imagine, part of the "self-contained-ness"
    of the original idea.

    --
    Cosimo
  • Jeffrey Thalhammer at May 25, 2011 at 6:03 am

    On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote:

    When you start a second project that uses a subset
    of project1 dependencies, what do you do?

    Do you duplicate the lpan/ and dlib/ folders?
    If you had to maintain patches, do you apply the patches
    to all your projects lpan/ and dlib/ folders?

    These dlib/ and lpan/ could be shared among projects,
    but they would lose, I imagine, part of the "self-contained-ness"
    of the original idea.
    I did think about this a bit. The dlib/ is always private to the project -- it is a generated directory that contains the fruit of installing the dependencies from the lpan.

    As you pointed out, most non-trivial applications consist of several distributions. In that case, the lpan could be moved outside the project directory, and you could use an svn:external to link it to each of the associated projects. So the project remains physically self-contained, but logically shares the same lpan with related projects. But I don't know how you would do this with git.

    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • Pedro Melo at May 25, 2011 at 6:07 am
    Hi,

    On Wed, May 25, 2011 at 7:03 AM, Jeffrey Thalhammer
    wrote:
    On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote:

    When you start a second project that uses a subset
    of project1 dependencies, what do you do?

    Do you duplicate the lpan/ and dlib/ folders?
    If you had to maintain patches, do you apply the patches
    to all your projects lpan/ and dlib/ folders?

    These dlib/ and lpan/ could be shared among projects,
    but they would lose, I imagine, part of the "self-contained-ness"
    of the original idea.
    I did think about this a bit.  The dlib/ is always private to the project -- it is a generated directory that contains the fruit of installing the dependencies from the lpan.

    As you pointed out, most non-trivial applications consist of several distributions.  In that case, the lpan could be moved outside the project directory, and you could use an svn:external to link it to each of the associated projects.  So the project remains physically self-contained, but logically shares the same lpan with related projects.  But I don't know how you would do this with git.
    The same way. Create a separate repository with the shared
    dependencies, and include it on your projects as a submodule (git
    version of svn externals).

    Bye,
    --
    Pedro Melo
    http://www.simplicidade.org/
    xmpp:melo@simplicidade.org
    mailto:melo@simplicidade.org
  • Jeffrey Thalhammer at May 25, 2011 at 6:14 am

    On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote:

    When you start a second project that uses a subset
    of project1 dependencies, what do you do?
    We should probably clarify what we mean by "project" and "application".

    I think of an application as all the code that will run against a particular set of dependencies.

    A project is just a distribution, which may depend on other distributions (modules, really).

    An application will consist of one or more related projects.

    All the projects within an application will share a common lpan.

    But each application will have its own lpan.

    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • Jeffrey Thalhammer at May 25, 2011 at 6:41 am

    On May 24, 2011, at 10:51 PM, Cosimo Streppone wrote:

    If you had to maintain patches, do you apply the patches
    to all your projects lpan/ and dlib/ folders?

    [Sorry for the fragmented response. I'm a bit scatter-brained at this hour]

    Locally patched modules are an interesting case. You could stash your patched tarballs elsewhere in the repository, and then manually inject them (by copying) them into the lpan for each application that needs them.

    Or perhaps, you could establish a separate private CPAN just for patched distros and use CPAN::Site to produce an 02packages that falls back to the public CPAN for everything else. That would give you a mechanism for distributing patched distros (or any of your private distros) throughout the organization.

    Either way, each application ultimately gets a separate copy of the patched distro in its lpan. The only difference is how that distro get's delivered into the lpan.

    Off to bed now.

    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • David Cantrell at May 25, 2011 at 10:38 am

    On Tue, May 24, 2011 at 10:19:49PM -0700, Jeffrey Thalhammer wrote:

    It's easy for me to get carried away with all the intricacies of dependency management. But for my task at hand, I think it might be the wrong problem. So for the sake of this discussion, let's put aside the issue of how a private CPAN (really, the 02packages) could be constructed to have a magical mixture of modules. Instead, I want to think about how a private CPAN mirror fits into the development cycle. This might be easiest by walking through a hypothetical scenario...

    Let's suppose I wan to write a new application in Perl. By "application", I really just mean some set of modules and scripts. And let's assume that my development, testing, and deployment environments all have a perfectly clean version of perl -- no modules have been added or updated from the core.

    I want my app to leverage the CPAN tool chain, so I setup a CPAN-style distribution for my source code. It has lib/, bin/, and t/ directories, and all the usual trappings. I'm also going to use Module::Build. And of course, all this is in some source control system (assume svn for now).

    After a couple of Mountain-Dews, I have some code and tests and everything passes. Commit. Now I want to add some feature that requires a module from CPAN. So I go shopping on search.cpan.org and find one I like called Frobulator::Simple...

    So instead, let's say I add Frobulator::Simple to the "requires" list (or "build_requires", or whatever) in my Build.PL. Now I run the "installdeps" target, which fetches the latest Frobulator::Simple and all its dependencies from a public CPAN and installs them.

    Now this is where it gets interesting. Rather than install in site_perl, it installs into a directory right inside my project (let's call it "dlib" for "dependent libraries"). At the same time, it stashes the tar balls in a directory structure suitable for a CPAN mirror, ...
    FWIW, my solution (which I haven't tried) would be:

    * make an up-to-date mirror of the most recent versions of everything on
    the CPAN, using CPAN::Mini;
    * use CPAN::Mini::Inject to add your own modules to the mirror.

    To keep your mirror up to date with new releases to the CPAN, use the
    funky fast update mechanism to get change notifications, and, for each
    new distribution:
    * download it;
    * inject it into a "staging" CPAN;
    * try to test your application, using a clean perl and the staging
    CPAN to resolve dependencies;
    * if it passes, inject the new distribution into your "real" CPAN;
    * if it fails, remove it from the staging CPAN

    Optionally, if adding a distribution to the CPAN makes your app fail its
    tests, put it into a third CPAN mirror and notify your devs that it's
    there and that they should fix the app and then re-inject the
    distribution into your CPAN mirror. However, I can see problems
    here. First, they won't do it because they have more important* things
    to do such as adding features or fixing their own bugs; second, you
    should probably address those failures in order, meaning that that work
    stream will eventually get blocked entirely when something like Catalyst
    has an incompatible change that completely fucks your application over;
    third, what do you do about dists which fail their own tests but don't
    make your app fail its tests?

    * to management

    --
    David Cantrell | Hero of the Information Age

    "IMO, the primary historical significance of Unix is that it marks the
    time in computer history where CPUs became so cheap that it was possible
    to build an operating system without adult supervision."
    -- Russ Holsclaw in a.f.c
  • David Golden at May 25, 2011 at 11:17 am

    On Wed, May 25, 2011 at 1:19 AM, Jeffrey Thalhammer wrote:
    I realize this doesn't solve the "big" problem of automatically building a particular CPAN mirror with a magical mix modules.  But based on my own development patterns, I think this model could provide a workable framework for organically evolving application dependencies, while providing a stable and reproducible dependency stack at the same time.
    I hadn't quite taken it that far in my thinking, but I'm glad to have
    a use case to consider as I work.

    My mental model to date has been along these lines:

    * You're working off a particular version of perl (/opt/perl/5.XX/)
    which has no site_lib modules installed

    * Modules you're adding as dependencies get installed into your
    local::lib (possibly per application, if you need that separation)

    * When you're done with your application, you tar it up and inject it
    into a repository that can be accessed via a URL

    * You add your tarball to a "local+BackPAN" index (local supersedes BackPAN)

    * You run the dependency analysis tool against your tarball, using the
    perl as a base and using the local+BackPAN to resolve dependencies.

    * Tool spits out the ordered list of tarballs (probably as URLs)
    needed to install the application

    If you need to patch a module from CPAN, you treat it just like an
    app, publish locally and add it to the local index.

    To deploy, on some other machine, you run a deployment tool against
    the matching perl on that machine, giving it the ordered list of URLs.

    For instances where you might have App A and App B, the tool should be
    able to factor out a common set of dependencies, which could be
    deployed separately. Then with the right ordering of library paths,
    you could have App A use version X of a module, while App B uses
    version Y of a module and otherwise, they use a common set of modules.

    I hadn't considered the team development case much. My snap reaction
    is that you would regularly "deploy" to a shared development library
    path rather than install modules to a personal local::lib during
    development. As long as the local+BackPAN index can reverse your
    dependency (module name and version) to the right tarball (including
    locally patched ones), it all should just work.

    My though is to write a separate tool for this rather than try to make
    it work with either M::B or dzil.

    -- David
  • Brian d foy at May 26, 2011 at 6:31 pm
    In article <02c001cc1a59$6865b810$39312830$@activestate.com>, Jan
    Dubois wrote:
    AFAICT there is no good index for backpan though.
    There's a good index, but it lives on one of my private computers and
    is rather large. :)
  • Brian d foy at May 26, 2011 at 6:34 pm
    In article <BANLkTinzErQAnW4qthz=T1jdd8p3ioUQuQ@mail.gmail.com>, David
    Golden wrote:
    * You run the dependency analysis tool against your tarball, using the
    perl as a base and using the local+BackPAN to resolve dependencies.

    * Tool spits out the ordered list of tarballs (probably as URLs)
    needed to install the application
    I have this bit done but not public yet.
  • Jeffrey Thalhammer at May 26, 2011 at 6:55 pm

    On May 25, 2011, at 4:16 AM, David Golden wrote:

    My though is to write a separate tool for this rather than try to make
    it work with either M::B or dzil.

    Agreed. A separate tool could be later integrated with either of those.

    Also, because you wouldn't want the tools themselves polluting the pure environment that you're trying to build in.

    Jeffrey Thalhammer
    Imaginative Software Systems
    vcard: http://www.imaginative-software.com/contact/jeff.vcf
  • Steffen Schwigon at Jun 1, 2011 at 8:51 am

    David Cantrell writes:
    FWIW, my solution (which I haven't tried) would be:

    * make an up-to-date mirror of the most recent versions of everything on
    the CPAN, using CPAN::Mini;
    * use CPAN::Mini::Inject to add your own modules to the mirror.
    I used CPAN::Mini a lot but now start using *real* CPAN mirrors for
    the distroprefs mechanism that CPAN.pm provides (needed to work around
    any known problem on CPAN).

    I *think* (but haven't tried yet) that combining both might be good:

    - a local real CPAN mirror for the latest full CPAN
    - an additional CPAN::Mini (plus ::Inject) to add your own lcoal
    modules

    AFARI CPAN.pm looks into other mirrors once it doesn't find a module,
    which would happen for local modules only in CPAN::Mini.

    Kind regards,
    Steffen
    --
    Steffen Schwigon <ss5@renormalist.net>
    Dresden Perl Mongers <http://dresden-pm.org/>
  • Steffen Schwigon at Jun 1, 2011 at 9:09 am

    Jeffrey Thalhammer writes:
    Now this is where it gets interesting. Rather than install in
    site_perl, it installs into a directory right inside my project
    (let's call it "dlib" for "dependent libraries"). At the same time,
    it stashes the tar balls in a directory structure suitable for a
    CPAN mirror, which is also right inside my project directory (let's
    call it "lpan", for "local perl archive network"). Finally, it
    generates the 02packages for the lpan/. Everything in lpan/ will be
    placed under revision control too.
    I like the idea.

    Of course the revision control should not be tied to subversion but
    kept to the developer who uses cvs/svn/git/hg/whatever.

    I realize this doesn't solve the "big" problem of automatically
    building a particular CPAN mirror with a magical mix modules.
    Which is ok imho, as that “big” problem usually was just an
    intermediate one, introduced to solve the actual distro bundle
    problem. It comes back when you want to restore a snapshot back in
    time from nothing, like deploy an app for Perl 5.6, but that's worth
    to skip for now. IMHO.

    Kind regards,
    Steffen
    --
    Steffen Schwigon <ss5@renormalist.net>
    Dresden Perl Mongers <http://dresden-pm.org/>
  • Steffen Schwigon at Jun 1, 2011 at 9:24 am

    Jeffrey Thalhammer writes:
    [build private CPANs]
    Just to add an experience for completeness of this discussion:

    I once tried to snapshot a complete MINICPAN with git every time I
    verified a CPAN::Mini sync with my application's “cpan upgrade ; make
    test”.

    But it did not work out well due to size and speed, even for the damn
    fast git.

    That *might* be relevant again also for the dlib/lpan idea, when the
    dependencies for your app are exploding.

    Kind regards,
    Steffen
    --
    Steffen Schwigon <ss5@renormalist.net>
    Dresden Perl Mongers <http://dresden-pm.org/>
  • Pedro Melo at Jun 1, 2011 at 11:25 am
    Hi,
    On Wed, Jun 1, 2011 at 9:51 AM, Steffen Schwigon wrote:
    I *think* (but haven't tried yet) that combining both might be good:

    - a local real CPAN mirror for the latest full CPAN
    - an additional CPAN::Mini (plus ::Inject) to add your own lcoal
    modules

    AFARI CPAN.pm looks into other mirrors once it doesn't find a module,
    which would happen for local modules only in CPAN::Mini.
    Never tried this either, but maybe start with an empty local CPAN and
    CPAN::Inject into that. Then use that one as the first mirror on your
    CPAN.pm configuration...

    Bye,
    --
    Pedro Melo
    @pedromelo
    http://www.simplicidade.org/
    http://about.me/melo
    xmpp:melo@simplicidade.org
    mailto:melo@simplicidade.org
  • David Cantrell at Jun 1, 2011 at 12:31 pm

    On Wed, Jun 01, 2011 at 10:51:41AM +0200, Steffen Schwigon wrote:
    I used CPAN::Mini a lot but now start using *real* CPAN mirrors for
    the distroprefs mechanism that CPAN.pm provides (needed to work around
    any known problem on CPAN).
    You can, of course, use distroprefs with any CPAN-like mirror, including
    a mini-CPAN.

    Distroprefs are awesome, and not anything like well-known enough.
    I *think* (but haven't tried yet) that combining both might be good:

    - a local real CPAN mirror for the latest full CPAN
    - an additional CPAN::Mini (plus ::Inject) to add your own lcoal
    modules
    AFARI CPAN.pm looks into other mirrors once it doesn't find a module,
    which would happen for local modules only in CPAN::Mini.
    The urllist will indeed take multiple sources. This isn't something
    I've ever done though, so I don't know how well it works or whether it
    introduces any "interesting" gotchas. If you do do this, I suggest that
    one of the edge-cases you should test before relying on it is what
    happens if your local CPAN has a more up-to-date version of some module
    than what the "real" CPAN has. This could happen because you've got
    locally applied patches which the upstream author hasn't yet applied, or
    if you've not yet released the latest version of something to the CPAN,
    for example.

    --
    David Cantrell | even more awesome than a panda-fur coat

    Seven o'clock in the morning is something that
    happens to those less fortunate than me
  • Steffen Schwigon at Jun 2, 2011 at 8:31 am

    David Cantrell writes:
    On Wed, Jun 01, 2011 at 10:51:41AM +0200, Steffen Schwigon wrote:
    I used CPAN::Mini a lot but now start using *real* CPAN mirrors for
    the distroprefs mechanism that CPAN.pm provides (needed to work around
    any known problem on CPAN).
    You can, of course, use distroprefs with any CPAN-like mirror,
    including a mini-CPAN.
    Technically yes. But the existing distroprefs yaml files maintained by
    ANDK in CPAN (and reused by me) are not synced in CPAN::Mini.

    I once synced it manually with rsync on top of a CPAN::Mini mirror but
    I got tired by that eventually.


    Kind regards,
    Steffen
    --
    Steffen Schwigon <ss5@renormalist.net>
    Dresden Perl Mongers <http://dresden-pm.org/>
  • Steffen Schwigon at Jun 2, 2011 at 9:39 am

    Steffen Schwigon writes:
    David Cantrell <david@cantrell.org.uk> writes:
    You can, of course, use distroprefs with any CPAN-like mirror,
    including a mini-CPAN.
    Technically yes. But the existing distroprefs yaml files maintained by
    ANDK in CPAN (and reused by me) are not synced in CPAN::Mini.
    Too early in the morning my brain leaks like a wiki…

    It's the patch collection I mean[1], that is referenced from some
    distroprefs and that is not contained in a CPAN::Mini mirror.

    The yaml files are part of the CPAN distro. These in turn I usually
    rsync manually from CPAN[2] or github directly to keep up to date with
    Andreas' work.

    Steffen

    Footnotes:
    [1] http://mirror.de.leaseweb.net/CPAN/authors/id/A/AN/ANDK/patches/
    [2] http://cpansearch.perl.org/src/ANDK/CPAN-1.9600/distroprefs/
    --
    Steffen Schwigon <ss5@renormalist.net>
    Dresden Perl Mongers <http://dresden-pm.org/>

Related Discussions