[Perl-QA] RFC: Private CPAN In A Box

FWIW, I've written one, called PGXN. The various parts are managed here:

https://github.com/pgxn/

PGXN::Manager == PAUSE
PGXN::API && PGXN::Site == search.cpan.org
pgxnclient = CPAN.pm

The client was actually written by someone else, in Python, just using the API.

https://github.com/dvarrazzo/pgxnclient/

He was able to do it pretty quickly because almost everything is represented by a static JSON file. The mirror and API APIs are documented here:

https://github.com/pgxn/pgxn-api/wiki

Might be of interest to you, FWIW. I wrote it for PostgreSQL Extensions, but tried to implement a generalized solution that could be used by any language or community -- or privately, if you like.
HTH, sorry I get a bit lost with some of your questions.

Best,

David

Hi Jeffrey,
You might want to check out:

http://beta.metacpan.org/ the front and backend are here:
https://github.com/cpan-api

http://blogs.perl.org/mt/mt-search.fcgi?limit=20&search=metacpan has
some history and discuss with them on irc.perl.org #metacpan - I've
pointed them to your perlmonk post

https://github.com/szabgab/CPAN-Digger also exists - I think Gabour
is discussing if there is any overlap with #metacpan which they're
discussing.

http://blogs.perl.org/users/michael_j/2011/04/deploying-perl-code-with-git-locallib-minicpan-and-cpanminus.html
is actually the setup we use at work, but this only works because we
only have one version of all code for all projects.

Hope that helps.

Leo

No, this is very helpful. I think you've shown that I need to take a step back and look at the underlying problem.

I'm not attempting to build an all-encompasing dependency management system. I've tried to avoid thinking about it in terms of CPAN as we know it, since in theory, that is an implementation detail. But in practice, I'm will stipulate that a "dependency" is just a Perl module in some CPAN repository.

As I see it, there are two sides to the problem. The first is creating tools for creating, managing, and using a private CPAN repository. There is already a lot of prior art for this, but I feel that it is fragmented. So this might just be a matter of assembling the right combination of existing technologies.

The second (and probably more difficult) side is establishing patterns for using these tools, which enable developers to write, test, and deploy their code against an identifiable and reproducible set of dependencies. I acknowledge that one size will not fit all, but I strongly believe there is (or should be) some common ground here.

The trouble with CPANs is that they are a moving target -- modules are constantly updated, added, and removed[1]. This creates problems for developers that want to use the CPAN tool chain, but need to have a stable set of dependencies. At the same time, they need flexibility to evolve those dependencies in a systematic way.

Thanks for bringing PGXN to my attention. The architecture is very similar to what I was thinking. I will dig deeper to see if we can leverage PGXN.

[1] I think a lot of this trouble would go away if the CPAN tool chain simply permitted authors to express precisely which $VERSION of something they require. So I have to wonder if that is the right place to focus, rather than building something on top of the tool chain.


Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

That's exactly what I'm talking about. I think this kind of pattern is popping up in a lot of organizations. Somehow, I want to try and package that idea in a way that might appeal to a wide audience.

Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

http://cp2010-05an.barnyard.co.uk/ and all its date/time-stamped friends
and relations aren't.

A feature I keep meaning to add is some way for users to pick one of my
mirrors and then further restrict it so that, for example, you might
want the CPAN as at the beginning of 2011 but with Some::Module (and all
its pre-requisites) stuck at some earlier point in time, or even a way
of applying that to an up-to-date CPAN mirror (an up-to-date mirror is
just a special case of my mirrors anyway, one with no filtering at all).

I want to do this in a way that:

* won't require manual intervention from me to set up a new restriction
* will work with any CPAN client
* is efficient with storage and CPU
* provides free ponies and kittens

After looking at David Wheeler's PGXN and David Cantrell's CPAN snapshots, I've come up with a story. Tell me if you think this makes sense (or not):

Historically, CPAN mirrors have always been static files. In other words, the paths in the 02packages file always pointed directly to some physical file on disk. This made it possible to access mirrors the with the older ftp:// and file:// protocols supported by CPAN.pm. It also made synchronizing the mirrors easy with tools like rsync.

This has worked well for a long time. But the need to support older protocols limits the flexibility of a CPAN mirror. If you drop support for ftp:// and file://, and just focus on http:// then things can become more interesting. This open the door for a CPAN server (like the PGXN::Manager) that can do all sorts of things. For example, it could serve up distros that are spread across multiple remote machines. Or they could be stored in version control system or database. Or you could present a view of the mirror that meets certain criteria, such as by time, or cpantesters results, or by author, or by kwalitee scores, or by some other metadata (like http://cp2011-05an.barnyard.co.uk/). The concept of a CPAN server also provides a convenient hook for plugging in other services, like a private PAUSE or MetaCPAN.

I'm not saying that *the* CPAN should work this way. I'm just suggesting that a *private* CPAN could work this way.

Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

You can.

http://search.cpan.org/~dagolden/CPAN-Meta-2.110930/lib/CPAN/Meta/Spec.pm#Version_Ranges

Best,

David

What I've tried to do with PGXN is to get the best of both worlds. At the lowest level, the mirror generated by PGXN::Manager should be very simple, with all static files. But many of those static files are JSON files, and so every mirror is a lightweight REST API server without any additional work required. Then anyone can build a service using the API. Or they can build new mirrors that do additional stuff with the data, which is what the API server does (adds additional info to some JSON files, generates HTML pages, manages full-text index, etc.).

So take the static stuff, and build something dynamic on top of it, either using its API or by adding to it.

Best,

David

The problem is that there is no way for a single application to use multiple
versions of a module. So it's easy to imagine that very specific module
prereqs could be in conflict and require manual override. That's not the
right answer.

What we need is application level dependencies that are specific and non
conflicting and then to make it easy to install into app specific lib
directories.

That's what im
system. I've tried to avoid thinking about it in terms of CPAN as we know
it, since in theory, that is an implementation detail. But in practice, I'm
will stipulate that a "dependency" is just a Perl module in some CPAN
repository.
in some CPAN repository that the module I'm trying to install right now
depends on. That is, it's listed in in the "prereqs" section of a META.json
file.
constantly updated, added, and removed[1]. This creates problems for
developers that want to use the CPAN tool chain, but need to have a stable
set of dependencies. At the same time, they need flexibility to evolve those
dependencies in a systematic way.
specify very precise requirements.
similar to what I was thinking. I will dig deeper to see if we can leverage
PGXN.
simply permitted authors to express precisely which $VERSION of something
they require. So I have to wonder if that is the right place to focus,
rather than building something on top of the tool chain.

The SEND button should be harder to press by accident...

That's what I'm working on in June - trying to tie all the various
lessons from other toolchain and meta tools into something that can be
for Perl apps what cpan(p|m) is for Perl modules.

I think people are trying to make cpan solve the application level
problem and that's why "custom CPAN mirror" seems like the answer.

David

Hi David,
Whaaat!? :-)

Let's see if I understand this correctly.
If, as an example, in this file,

http://cpansearch.perl.org/src/COSIMO/Net-Prober-0.03/META.json

I say:

"prereqs" : {
...
"runtime" : {
"requires" : {
"Net::Ping" : "<= 2.35"
}
},
...
},

Where current Net::Ping on CPAN is 2.36,
than this will install Net::Ping 2.35 as
prerequisite?

Or, if Net::Ping on the system is already 2.36
it will refuse to proceed?

I have tried fiddling with Makefile.PL/META.yml/META.json
without success.

I think the current CPAN clients will all refuse to proceed. This is
what I meant when I said that version ranges don't accomplish the goal
because the computer can't know what the right thing to do is.

-- David

Your mirror only has to support file:// if you want it to live on a
user's local disk, and only has to support ftp:// if you want to give
FTP access. I realise that this is obvious, but please bear with me :-)

You can support whatever craziness you want over HTTP.

The only place in CPAN::Config which mentions the mirror's URL is this:
urllist => [q[http://cp5.6.2an.barnyard.co.uk/]],

Crucially, it derives URLs for both the index and the distributions from
that. So, if you have a magic mirror and are willing to do the heavy
lifting on the server (and please bear in mind it's *very* heavy lifting -
my daily rebuilds of static 02packages files take about 7 hours, and
to build a mirror like cp2011-05an takes about 12 hours) you could
simply configure CPAN.pm to use:
urllist => [q[http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22]],

CPAN.pm would then get its index by requesting:
http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22/modules/02packages.details.txt.gz

and you can, of course, construct whatever the hell you want in
response, provided it has the right format. That file would normally
read something like ...

Description: This is a whitespace-seperated file.
Description: Each line is modulename moduleversion filename.
Line-Count: 33800
Last-Updated: Tue, 24 May 2011 06:22:15 GMT

AAAA::Crypt::DH 0.04 B/BI/BINGOS/AAAA-Crypt-DH-0.04.tar.gz
ABI 1.0 M/MA/MALAY/ABI-1.0.tar.gz
ACH::Builder 0.03 T/TK/TKEEFER/ACH-Builder-0.03.tar.gz
ACME::Error 0.03 C/CW/CWEST/ACME-Error-0.03.tar.gz

but there's no reason why the meat of it couldn't be stuff like ...

AAAA::Crypt::DH 99999999 distcontaining/AAAA::Crypt::DH

which CPAN.pm would resolve to ...

http://example.com/perl/5.8.8/asat/2008-05-04Z11:43:22/authors/id/distcontaining/AAAA::Crypt::DH

which, apart from the annoying 'authors/id/' in the middle, which could
be trivially dealt with using mod_rewrite, looks deliciously RESTish.

Ah, but sometime this summer the clients (and therefor the computer) will?

Best,

David

I doubt it. I have plans for an "application installer" that will
pre-compute dependencies and reliably set up per-application lib
directories. Different applications could have different versions of
a module, but a single application will only have one version
available to it.

This is not something that the CPAN clients are really designed to do
(though they can be made to do the right things with INSTALLBASE and
so on).

A better specification and design for application dependencies and
installation will avoid the need to create N custom CPAN views as a
hack to make CPAN clients try to be application deployment tools.

-- David

Yeah, that'll be nice.

Best,

David

That's the idea, though I've not worked out the details. But for
example, I'd like to use the configuration output data (MYMETA.json)
to determine three sets of modules (tarballs, really):

- latest versions on CPAN
- currently installed versions on developers machine
- minimum versions on CPAN that satisfy the prereq specs

It's possible that version ranges could conflict, in which case some
or all of those might not be achievable. Ideally, the dependencies
would be determined against a specific version of perl without
additional libraries installed.

Then, given one of those three ordered list of tarballs that satisfy
all prereqs, it should be possibly to repeatably deploy an application
with a known set of module versions, even as the "latest" on CPAN
evolves.

Of the three sets, users would have to decide which suit their
specific needs/purpose, or could substitute versions, etc.

That's my vision in a nutshell. I have some people in NY that I'm
going to do some brainstorming with and then I'll try to get a proof
of concept code out next month.

-- David

That could get a bit troublesome with the current best practice to
delete older versions of module from CPAN.

Leon

http://backpan.perl.org/

I don't remember when they started keeping the history -- I've seen
stuff as far back as 1997.

-- Davie

And a somewhat unfortunate practice, as it makes it hard to create diffs
between arbitrary module versions. And with the new fast-mirror mechanism,
I'm not sure that the aggressive pruning of CPAN should still be considered
a "best" practice, as it makes the web-based diff on search.cpan.org
pretty useless. For example when trying to understand this Locale-Codes
changelog entry:
I would first try to look here:

http://search.cpan.org/tools/Locale-Codes-3.16

But it is not working because even the previous release has already been
deleted.
Yes, backpan has the complete history with very few exceptions (I think
some website scraping modules were deleted from backpan when the website
owner threatened legal action).

AFAICT there is no good index for backpan though. I have to audit diffs
to older module versions quite often, so I have my own tools, and found
that even the links are not comprehensive; I have to crawl both
backpan/modules/by-module and backpan/modules/by-authors/id to make
sure I find *all* the different versions.

Cheers,
-Jan

I believe that brian foy created a pretty comprehensive one -- I think
he may have actually investigated how module files changed from
tarball to tarball. (E.g. Module $VERSION was the same in two
tarballs, but the file MD5's are different).

The hard problem is that PAUSE doesn't have any
maintainer/co-maintainer history -- so given a historical
DAGOLDEN/Foo-Bar-1.23.tar.gz and RJBS/Foo-Bar-1.23.tar.gz, there's no
way to tell that one or the other was the official indexed version.
You can look at who has maintainer rights *today* and assume from that
who was authorized when the tarball was uploaded.

I don't know (yet) how many collisions like that there are. I suspect
it's a relatively small number that can be flagged and resolved with a
bit of research and email.

-- David

.
Wow! This is getting juicy.

It's easy for me to get carried away with all the intricacies of dependency management. But for my task at hand, I think it might be the wrong problem. So for the sake of this discussion, let's put aside the issue of how a private CPAN (really, the 02packages) could be constructed to have a magical mixture of modules. Instead, I want to think about how a private CPAN mirror fits into the development cycle. This might be easiest by walking through a hypothetical scenario...

Let's suppose I wan to write a new application in Perl. By "application", I really just mean some set of modules and scripts. And let's assume that my development, testing, and deployment environments all have a perfectly clean version of perl -- no modules have been added or updated from the core.

I want my app to leverage the CPAN tool chain, so I setup a CPAN-style distribution for my source code. It has lib/, bin/, and t/ directories, and all the usual trappings. I'm also going to use Module::Build. And of course, all this is in some source control system (assume svn for now).

After a couple of Mountain-Dews, I have some code and tests and everything passes. Commit. Now I want to add some feature that requires a module from CPAN. So I go shopping on search.cpan.org and find one I like called Frobulator::Simple. At this point, I might go ask my sysadmin to install Frobulator::Simple (and all its dependencies) in /usr/lib/perl5 on the development machine. Or if I'm smart, I might use local::lib to install it somewhere in my home directory. But these both suck, because changing /usr/lib/perl5 will affect everyone else, and who knows what else I might have installed in my home directory.

So instead, let's say I add Frobulator::Simple to the "requires" list (or "build_requires", or whatever) in my Build.PL. Now I run the "installdeps" target, which fetches the latest Frobulator::Simple and all its dependencies from a public CPAN and installs them.

Now this is where it gets interesting. Rather than install in site_perl, it installs into a directory right inside my project (let's call it "dlib" for "dependent libraries"). At the same time, it stashes the tar balls in a directory structure suitable for a CPAN mirror, which is also right inside my project directory (let's call it "lpan", for "local perl archive network"). Finally, it generates the 02packages for the lpan/. Everything in lpan/ will be placed under revision control too.

Now I have all my application code and my dependencies in one local, revision-controlled location. When I run the "build" target, all my modules are copied into the blib/ directory as usual. And when I run the "test" target, both blib/ and dlib/ are placed on @INC. If the tests pass, then commit. If not, then fix my code. Or I can "svn revert" my lpan/ directory, change the "requires" list in Build.PL to try a new module, and run the "installdeps" target again. Rinse and repeat as needed.

This goes on for a while as my application evolves until I have several dependencies. Dependencies that I want to keep are in the growing dpan/ directory and committed to svn. Meanwhile, the dlib directory is clouding up with all the modules I've experimented with. Periodically, I want to test the whole stack. So now I run the "realclean" target, which wipes out dlib/. Next, I run an "installdepslocal" target, which basically does an "installdeps", but this time it pulls everything from the lpan. This effectively flushes out any modules that fail since, over time, I may have upgraded some of their dependencies in my lpan/ and those upgrades may not be backward compatible.

If some of the dependencies do fail, it is my job to figure out what to do. This might mean removing a newer distro from the lpan/ in favor of an older one. Or maybe I'll make a patched version of the distro and put it in the lpan/. Or perhaps I'll just switch to a different dependency altogether. How this all gets done is out of scope for this discussion, but I imagine that we could generate some kind of map that tracks all the dependencies for the app. This could help developers see the interdependencies and figure out how an upgrade or downgrade will impact the app. You could also have special build targets that assist with adding, removing, or upgrading specific sub-trees in the dependency graph.

But for now, let's assume I've got a working stack.. When I run the "test" target again, I'm now testing my code against a pure set of dependencies. The traditional "disttest" target might also be enhanced to first run "installdepslocal" in the dist directory. Once my application code is golden, I run a "release" target that builds the tar.gz of my application and shoves into the lpan/. Again, the 02packages is automatically updated. Commit.

Now let's say we use the CPAN tool chain for deployment too. My application and all it's dependencies are in the lpan/, so I can just login to the target host, point to the lpan in SVN, and say "cpan My::App". Again, we'd use local::lib to ensure everything does into an app-specific location. This is especially easy with SVN, since all the tarballs in the lpan can be accessed with HTTP requests. Or if we prefer, we can build the local lib/ directory on a separate host, archive it up, and ship it to the target that way.

So far, I've described all this in terms of Module::Build targets, but some (perhaps all) of this could be done with Dist::Zilla plugins. One thing I like about this model is that it works well for team development, since all the dependencies converge in the lpan in svn. And since lpan is part of the project, it plays nicely with branching and merging. Also, I like it because it requires absolutely no administration -- each app has its own lpan/, which is totally controlled by the developers. Lastly, it doesn't require rejiggering the entire CPAN tool chain and/or backfilling CPAN distros with some new metadata.

I realize this doesn't solve the "big" problem of automatically building a particular CPAN mirror with a magical mix modules. But based on my own development patterns, I think this model could provide a workable framework for organically evolving application dependencies, while providing a stable and reproducible dependency stack at the same time.

What do you think?

Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

When you start a second project that uses a subset
of project1 dependencies, what do you do?

Do you duplicate the lpan/ and dlib/ folders?
If you had to maintain patches, do you apply the patches
to all your projects lpan/ and dlib/ folders?

These dlib/ and lpan/ could be shared among projects,
but they would lose, I imagine, part of the "self-contained-ness"
of the original idea.

I did think about this a bit. The dlib/ is always private to the project -- it is a generated directory that contains the fruit of installing the dependencies from the lpan.

As you pointed out, most non-trivial applications consist of several distributions. In that case, the lpan could be moved outside the project directory, and you could use an svn:external to link it to each of the associated projects. So the project remains physically self-contained, but logically shares the same lpan with related projects. But I don't know how you would do this with git.

Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

Hi,

On Wed, May 25, 2011 at 7:03 AM, Jeffrey Thalhammer
wrote:
The same way. Create a separate repository with the shared
dependencies, and include it on your projects as a submodule (git
version of svn externals).

Bye,

We should probably clarify what we mean by "project" and "application".

I think of an application as all the code that will run against a particular set of dependencies.

A project is just a distribution, which may depend on other distributions (modules, really).

An application will consist of one or more related projects.

All the projects within an application will share a common lpan.

But each application will have its own lpan.

Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

[Sorry for the fragmented response. I'm a bit scatter-brained at this hour]

Locally patched modules are an interesting case. You could stash your patched tarballs elsewhere in the repository, and then manually inject them (by copying) them into the lpan for each application that needs them.

Or perhaps, you could establish a separate private CPAN just for patched distros and use CPAN::Site to produce an 02packages that falls back to the public CPAN for everything else. That would give you a mechanism for distributing patched distros (or any of your private distros) throughout the organization.

Either way, each application ultimately gets a separate copy of the patched distro in its lpan. The only difference is how that distro get's delivered into the lpan.

Off to bed now.

Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

FWIW, my solution (which I haven't tried) would be:

* make an up-to-date mirror of the most recent versions of everything on
the CPAN, using CPAN::Mini;
* use CPAN::Mini::Inject to add your own modules to the mirror.

To keep your mirror up to date with new releases to the CPAN, use the
funky fast update mechanism to get change notifications, and, for each
new distribution:
* download it;
* inject it into a "staging" CPAN;
* try to test your application, using a clean perl and the staging
CPAN to resolve dependencies;
* if it passes, inject the new distribution into your "real" CPAN;
* if it fails, remove it from the staging CPAN

Optionally, if adding a distribution to the CPAN makes your app fail its
tests, put it into a third CPAN mirror and notify your devs that it's
there and that they should fix the app and then re-inject the
distribution into your CPAN mirror. However, I can see problems
here. First, they won't do it because they have more important* things
to do such as adding features or fixing their own bugs; second, you
should probably address those failures in order, meaning that that work
stream will eventually get blocked entirely when something like Catalyst
has an incompatible change that completely fucks your application over;
third, what do you do about dists which fail their own tests but don't
make your app fail its tests?

* to management

I hadn't quite taken it that far in my thinking, but I'm glad to have
a use case to consider as I work.

My mental model to date has been along these lines:

* You're working off a particular version of perl (/opt/perl/5.XX/)
which has no site_lib modules installed

* Modules you're adding as dependencies get installed into your
local::lib (possibly per application, if you need that separation)

* When you're done with your application, you tar it up and inject it
into a repository that can be accessed via a URL

* You add your tarball to a "local+BackPAN" index (local supersedes BackPAN)

* You run the dependency analysis tool against your tarball, using the
perl as a base and using the local+BackPAN to resolve dependencies.

* Tool spits out the ordered list of tarballs (probably as URLs)
needed to install the application

If you need to patch a module from CPAN, you treat it just like an
app, publish locally and add it to the local index.

To deploy, on some other machine, you run a deployment tool against
the matching perl on that machine, giving it the ordered list of URLs.

For instances where you might have App A and App B, the tool should be
able to factor out a common set of dependencies, which could be
deployed separately. Then with the right ordering of library paths,
you could have App A use version X of a module, while App B uses
version Y of a module and otherwise, they use a common set of modules.

I hadn't considered the team development case much. My snap reaction
is that you would regularly "deploy" to a shared development library
path rather than install modules to a personal local::lib during
development. As long as the local+BackPAN index can reverse your
dependency (module name and version) to the right tarball (including
locally patched ones), it all should just work.

My though is to write a separate tool for this rather than try to make
it work with either M::B or dzil.

-- David

In article <02c001cc1a59$6865b810$39312830$@activestate.com>, Jan
Dubois wrote:
There's a good index, but it lives on one of my private computers and
is rather large. :)

In article <BANLkTinzErQAnW4qthz=T1jdd8p3ioUQuQ@mail.gmail.com>, David
Golden wrote:
I have this bit done but not public yet.

Agreed. A separate tool could be later integrated with either of those.

Also, because you wouldn't want the tools themselves polluting the pure environment that you're trying to build in.

Jeffrey Thalhammer
Imaginative Software Systems
vcard: http://www.imaginative-software.com/contact/jeff.vcf

I used CPAN::Mini a lot but now start using *real* CPAN mirrors for
the distroprefs mechanism that CPAN.pm provides (needed to work around
any known problem on CPAN).

I *think* (but haven't tried yet) that combining both might be good:

- a local real CPAN mirror for the latest full CPAN
- an additional CPAN::Mini (plus ::Inject) to add your own lcoal
modules

AFARI CPAN.pm looks into other mirrors once it doesn't find a module,
which would happen for local modules only in CPAN::Mini.

Kind regards,
Steffen

I like the idea.

Of course the revision control should not be tied to subversion but
kept to the developer who uses cvs/svn/git/hg/whatever.

Which is ok imho, as that “big” problem usually was just an
intermediate one, introduced to solve the actual distro bundle
problem. It comes back when you want to restore a snapshot back in
time from nothing, like deploy an app for Perl 5.6, but that's worth
to skip for now. IMHO.

Kind regards,
Steffen

Just to add an experience for completeness of this discussion:

I once tried to snapshot a complete MINICPAN with git every time I
verified a CPAN::Mini sync with my application's “cpan upgrade ; make
test”.

But it did not work out well due to size and speed, even for the damn
fast git.

That *might* be relevant again also for the dlib/lpan idea, when the
dependencies for your app are exploding.

Kind regards,
Steffen

Hi,
Never tried this either, but maybe start with an empty local CPAN and
CPAN::Inject into that. Then use that one as the first mirror on your
CPAN.pm configuration...

Bye,

You can, of course, use distroprefs with any CPAN-like mirror, including
a mini-CPAN.

Distroprefs are awesome, and not anything like well-known enough.
The urllist will indeed take multiple sources. This isn't something
I've ever done though, so I don't know how well it works or whether it
introduces any "interesting" gotchas. If you do do this, I suggest that
one of the edge-cases you should test before relying on it is what
happens if your local CPAN has a more up-to-date version of some module
than what the "real" CPAN has. This could happen because you've got
locally applied patches which the upstream author hasn't yet applied, or
if you've not yet released the latest version of something to the CPAN,
for example.

Technically yes. But the existing distroprefs yaml files maintained by
ANDK in CPAN (and reused by me) are not synced in CPAN::Mini.

I once synced it manually with rsync on top of a CPAN::Mini mirror but
I got tired by that eventually.


Kind regards,
Steffen

Too early in the morning my brain leaks like a wiki…

It's the patch collection I mean[1], that is referenced from some
distroprefs and that is not contained in a CPAN::Mini mirror.

The yaml files are part of the CPAN distro. These in turn I usually
rsync manually from CPAN[2] or github directly to keep up to date with
Andreas' work.

Steffen

Footnotes:
[1] http://mirror.de.leaseweb.net/CPAN/authors/id/A/AN/ANDK/patches/
[2] http://cpansearch.perl.org/src/ANDK/CPAN-1.9600/distroprefs/