[P5P] Perl 5.16 and Beyond.

[ What follows is the prose version of the talk I gave at YAPC::NA, OSCON and YAPC::EU this summer. I'll be giving a similar talk at YAPC::Asia. If you prefer reading big sentence fragments on slides, http://www.slideshare.net/obrajesse/perl-516-and-beyond is my slide deck from YAPC::EU]

Porters,

Over the last two years, I've been thinking a lot about where Perl 5 is headed. Well, that's not quite true. I've been thinking a lot about where Perl 5 is headed since way back in 2005 or 2006 when I was the Perl 6 project manager and it become clear to me that Ponie wasn't likely a horse I'd want to bet Perl 5 on.

As a community, we've done an absolutely stellar job of getting Perl 5 and p5p back into a reasonable groove. We know how to do releases. More individuals have released Perl 5 in the last 18 months as in the first 16 years Perl existed. It shouldn't really surprise anybody that knowing that work you do will be released soon is a huge motivating factor. Releasing frequently has dramatically increased Perl 5's rate of change. That's both good and bad.

I'm thrilled that Perl development is incredibly vibrant. 5.14 is a dramatically better Perl than 5.10 was, thanks to the work of approximately 260 porters. At the same time, I've made (or failed to make) some design choices that, in retrospect, I somewhat regret. There are places where Perl 5 is now more baroque than it was before. There have been changes to Perl 5 that have broken backward compatibility where we might have been better served not doing so. These mistakes are entirely my fault. If I had a do-over, I'd make different mistakes..er. decisions.

Various members of the community have asked me about my vision for the future of Perl 5. I was initially very, very hesitant to make any bold statements about the language, since "we're going to put out a development release every month and a new stable release every year" felt rather...bold itself.

Putting out releases is hard work. At this point, it's hard work we have well in hand.

Much of my thinking about the future of Perl 5 stems from the following principles:

* New versions of Perl 5 should not break your existing software
* Backward compatibility must not stop Perl 5 from evolving

Pay particular attention to "should" and "must" there. It is critically important that we not alienate the people, communities and companies who have invested their time and money in Perl 5. Pulling the rug out from under them isn't good for them and isn't good for us. Wherever possible, we need to preserve backward compatibility with earlier versions of Perl 5. At the same time, it could be argued that _any_ change to Perl 5 breaks backward compatibility. ("But I was depending on that segfault!") If Perl 5 is going to continue to flourish, we're going to need to be able to change the language.

* Why we deprecate things

Perl 5 is not Latin. It is a living language, still borrowing liberally from...just about everything. Sometimes we borrow the wrong things. Sometimes we borrow things and use them
wrong. Sometimes we invent things and later wish we hadn't. Perl has always been something of a packrat, but we're in danger of hitting the point of being diagnosed with a pathological hoarding problem. We need to get better at fixing problems and moving forward without hurting old code.

* How we handle language changes

We've always had a messy relationship with backward compatibility. Because we don't have a spec for the language and we have a single implementation, side-effects bugs and ill-considered accidents have become unchangeable parts of the language.

For a project of Perl's age, scale and diversity, we've been astonishingly conservative about how we make incompatible changes. It's not enough. Every release breaks running code. The more the language evolves, the more legacy code will break.

Standing still is not an option. Perl's internals, syntax and semantics have seen some much-needed improvements in the past few years. There are many additional changes we can't make because they may damage too much legacy code.

To date, version declarations in code have been intended as a marker for "minimum acceptable version of perl" -- They've also been used to enable new language features or semantics changes. Many _other_ new changes in Perl are turned on by default without the need for any declaration. Typically, this is because we don't believe those changes will harm earlier code, though it's also the case that conditionally enabling certain changes would be difficult.

Perl 5 doesn't require that a developer declare which version of Perl 5 a program or library was written to run under. Historically, our default has been "no declaration means latest, sort of."

That changes from here on in. Code that does not declare a 'use v5.16;' or later will be presumed to have been written to target v5.14.

If there is no "use v5.xx" line at the top of the code, the runtime should act as it did on v5.14 without a use v5.14 line.

We'll need a nice compact way to declare what version of Perl you want from the commandline for one-liners too.

Along with this, there's a change to how we'll be removing deprecated features from the core.

If a core feature (syntactic or semantic) is removed, a "use v5.xx" declaration for a earlier version of Perl 5 should re-enable the feature.

This will be difficult. I fully expect people to start throwing pies, tomatoes or bricks when I say this. I will not require that the forward-ported implementation of the old feature have performance parity with the old implementation. In certain circumstances, I _will_ make exceptions to the mandatory-forward-port rule. Security fixes are one obvious case. I know we'll run into other cases.

It is my strong preference that features granted R&R (removal and reinstatement) be implemented as modules, so as not to bloat the runtime when they're not needed. This isn't a pipe dream. Classic::Perl already does this for a few features removed in 5.10 and 5.12;

If it's not possible to reinstate a feature we've removed with existing APIs, we'll need to look at the cost of removing the feature vs simply disabling it in the presence of a new-enough v5.xx declaration.

* How quickly can we deprecate things

When I seized power in late 2009, I set us up with a new policy for deprecation timing. Anything we deprecated needed to warn in version .x that it would be removed in version .x+1. At the same time, I put us on a track to ship a new release of Perl 5 each spring. I'm incredibly pleased to see that multiple vendors have already incorporated Perl 5.12 into their stable releases

There's a reasonable chance that some vendors on a slightly longer release cycle may never ship a given major version of Perl, meaning that
our carefully-crafted deprecation warnings will never be seen.

If we manage to implement the R&R policy for deprecated features, we have little to worry about. Code that declares "use v5.14" should continue to function just as it always has, but for code that declares 'use v5.16', we can make changes without a deprecation cycle.

For cases where we _can't_ implement R&R, I think we need to move to a two year deprecation cycle, so as to have as minimal an impact as possible on users who are upgrading.

* Language Modularization

There's a lot of stuff that ships as part of "Perl 5 the distribution" that's not part of "Perl 5 the language" - Over the years, we've done an increasingly good job unwinding many dark corners of what we affectionately refer to as "blead" into CPAN modules. Most of those items have been add-on scripts and tools that have been part of the distribution but weren't part of the language.
Most of those modules continue to ship as part of the distribution and may well do so for many years to come. When I talk about modularization, I'm not proposing that we stop shipping the newly modularized code as part of the traditional core distribution.

Over the past few years, a number of people have spent significant effort to make certain parts of the language runtime a good deal more pluggable. I expect that we have a good amount of work to do before the core is flexible enough for us to fully implement this plan.

It's time for us to start extracting parts of what has traditionally been considered the "language" part of Perl 5 into CPANable modules. To do this successfully, it is imperative that _nothing_ appear out of the ordinary to code that expects to use those features. If code doesn't declare use v5.16, it should still get the 5.14ish environment it would expect.

At the same time, it should be possible for a future Perl to declare that certain previously-integral features are optional and run-time loadable/unloadable. There are many situations where Perl is used that don't need a whole set of the blades on our swiss-army chainsaw. Forcing their inclusion makes Perl a less viable candidate for a wide variety of applications.

Once language features are modularized, it also becomes _possible_ to maintain and improve them without requiring a full Perl upgrade.

Making this work for some things, like grammar changes, may take herculean effort but will invariably result in a more robust and flexible Perl.

I don't know what we'll extract or when we'll extract it, but there are a number of language features that seem like they might make sense to make pluggable: Formats, SysV IPC functions, Socket IO functions, Unix user information functions, Unix network information functions and Process and process group functions. Jesse Luehrs has already built us a first version of an extraction, modularization and replacement of smartmatch.

It should be clear from this list that I don't intend for us to kill, banish or exile these functions from the traditional Perl distribution. They're a useful and important part of our language and our culture. But they're not all needed or even desired everywhere we'd want Perl 5 to run. If we can modularize them in a way that doesn't negatively impact performance, Perl 5's manipulexity gets better without damaging its whipitupitude. The increased flexibility of the core should actually improve our whipitupitude at the same time.

* What should ship in the Perl core

When I say "traditional Perl distribution", I'm talking about the conglomeration of "language" and "toolkit" we currently ship. In 5.001, that totalled up to 59 packages. In 5.005, it grew to 176. By 5.008, it was 338 packages. 5.10 saw 541 packages. 5.12 ballooned to 625 packages. 5.14 was the most modest growth we've had in many versions. It ships 655 packages. Many of these are, of course, part of a smaller number of distributions.

The bar for getting a distribution into blead has varied over time. The "point" of the core distribution has often been up for debate. Is it an SDK for software written in Perl 5? If so, then we ought to be shipping what we currently consider to be the best-practice modules for building software in Perl. Is it "stuff we've always shipped with the Perl core?" If so, we're doomed to a life in a house full of stacks of old line-printer paper and TK50 cartridges.

Right now, what we include in the core distribution is a somewhat eclectic set of modules. They're not a particularly good SDK. They're not a particularly good sysadmin toolkit. They do, however, include an excellent set of modules to bootstrap getting more modules.

During my tenure, the guiding principle for new distributions in the core has been "does it help us bootstrap CPAN module installation or to test the core?"

In this day and age, most users get Perl from their operating system vendor or a third-party packager. Those vendors often look to us to decide what to ship. And they often ignore us. That's 100% ok. There are vendors who choose not to ship Perl's documentation as part of their "perl" package. There are vendors who have considered removing Perl
from their default installation because it's "too big" -- They've talked about splitting the distribution up according to their own logic or sometimes of removing it entirely. We've done some amazing work paring down the installed footprint. So far, I think we've avoided getting the axe.

And then there's the maintenance issue. It takes a lot of work for us to keep up to date with all the modules we include in the core. We do a pretty good job, but it takes a lot out of us.

It's time we made some more work for ourselves. We will continue to ship a distribution of Perl that contains roughly what we've always shipped, but I'd like it to be our secondary product.

The primary product will be "Perl the language" with the minimal set of modules needed to test the core, offer all the language features we intend to offer (Encode, for example) and to bootstrap the installation of new modules from CPAN. While I intend for us to ship the R&R features in the core distribution, it shouldn't actually be necessary.

Since I became involved in the Perl community, there have been constant requests for various "SDKs" for various uses of Perl. The state of the art in how to do such a thing is better than it was, but isn't really all that useful at this point. It should be possible to take a "Perl the language" distribution, drop some modules into a directory, tar/zip it up and hand it off to your user community. And that's how we should be able to build the "traditional" Perl distribution.

* Documentation changes

Perl's documentation is. Well, it is many things. Current, well-edited, complete and completely accurate are four things it is not. We've been getting a little better at splitting out "introduction for newcomers" documentation from "canonical reference" documentation.

Our introduction documentation has always been opinionated. The particular opinions have varied by author, document and phase of the moon. That's ok. Introductory documentation _should_ be opinionated. Introductory documentation should recommend "at least one good way to do it" and point the curious reader who wants to know more to canonical reference documentation. Canonical reference documentation should try to actually spell out how things work with a minimum of opinion.

As we modularize features, we'll need to consider how we reorganize the docs so that they are maximally useful to users who should neither need nor want to know about how Perl's internals work.

We need a new document, built from previous perldeltas that succinctly describes what, other than bug fixes, came or went in each version of Perl 5. Porters should be discouraged from adding or removing a feature without updating this document (and other documentation).

* Tests

In order to understand how well we're doing on back-compat as we move forward, we're going to need to be able to run a new Perl against older versions of the test suite. One step in that direction would be to be able to run the test suite against an installed Perl.

* TL;DR

- New versions of Perl 5 should not break your existing software
- Backward compatibility must not stop Perl 5 from evolving
- From 'use v5.16' forward, Perl should start treating 'use v5.x' statements as "try to give me a Perl that looks like v5.x" rather than "give me at least v5.x"
- We're awesome at modules. Where possible, we should be modularizing core features.


Best,

Jesse