FAQ
Hi internals!

Python and several other languages include support for list
comprehensions and generator expressions and I'd like to see something
similar in PHP too.

I created a hacky proof of concept implementation here:
https://github.com/nikic/php-src/tree/addListComprehensions. It's
really dirty, but it implements both features in about ~150 lines of
code.

Currently I'm using the following syntax:

$firstNames = [foreach ($users as $user) yield $user->firstName];

This code is roughly equivalent to writing:

$firstNames = [];
foreach ($users as $user) {
$firstNames[] = $user->firstName;
}

You may notice that this particular list comprehension provides the
same functionality as array_column(), just in a little more
generalized way. E.g. you could use all of the following without
having special functions for them all:

$firstNames = [foreach ($users as $user) yield $user->firstName];

$firstNames = [foreach ($users as $user) yield $user->getFirstName()];

$firstNames = [foreach ($users as $user) yield $user['firstName']];

It's also possible to explicitly specify a key:

$firstNames = [foreach ($users as $user) yield $user->id =>
$user->firstName];

It is also possible to filter elements using list comprehensions:

$underageUsers = [foreach ($users as $user) if ($user->age < 18)
yield $user];
// or just the names
$underageUserNames = [foreach ($users as $user) if ($user->age <
18) yield $user->firstName];

It is also possible to nest multiple foreach loops:

$aList = ['A', 'B'];
$bList = [1, 2];
$combinations = [foreach ($aList as $a) foreach ($bList as $b)
yield [$a, $b]];
// gives: [ ['A', 1], ['A', 2], ['B', 1], ['B', 2] ]

All the above are list comprehensions (or in PHP rather array
comprehensions), i.e. they create an array as the result.

If this is not needed it is also possible to compute the values lazily
using generator expressions, which use () instead of [].

$firstNames = (foreach ($users as $user) yield $user->firstName);

In this case $firstNames will no longer be an array of first names,
but instead will be a generator producing first names.

This is handy if you only need to iterate the resulting "list" only
once as it saves you holding the whole list in memory.

Also it allows you to work with infinite lists easily:

function *naturalNumbers() {
for ($i = 0; ; ++$i) {
yield $i;
}
}

// all natural numbers
$numbers = naturalNumbers();
// only the odd ones
$oddNumbers = (foreach ($numbers as $n) if ($n % 2) yield $n);
// ...

(At this point I wonder whether one should include support for
for-loops in list comprehensions. So the naturalNumbers() function
could be replaced with (for ($i = 0;; ++$i) yield $i), etc)

So, what do you think? Do we want something like this in PHP?

Nikita

Search Discussions

  • Jordi Boggiano at Jun 28, 2012 at 11:48 am
    Heya,
    So, what do you think? Do we want something like this in PHP?
    I think it looks great, especially the generator functions would be
    useful to avoid having to create a full blown iterator for simple stuff.

    Cheers

    --
    Jordi Boggiano
    @seldaek - http://nelm.io/jordi
  • Sebastian Krebs at Jun 28, 2012 at 1:00 pm
    Hi,

    Whats the difference to the (already existing) function array_map() (except
    the syntax and one more new keyword)?
    $firstNames = array_map(function($user){return $user->firstname;},
    $users);

    Don't want to rewrite every example you gave, but you probably see my point.

    Regards,
    Sebastian

    2012/6/28 Nikita Popov <nikita.ppv@gmail.com>
    Hi internals!

    Python and several other languages include support for list
    comprehensions and generator expressions and I'd like to see something
    similar in PHP too.

    I created a hacky proof of concept implementation here:
    https://github.com/nikic/php-src/tree/addListComprehensions. It's
    really dirty, but it implements both features in about ~150 lines of
    code.

    Currently I'm using the following syntax:

    $firstNames = [foreach ($users as $user) yield $user->firstName];

    This code is roughly equivalent to writing:

    $firstNames = [];
    foreach ($users as $user) {
    $firstNames[] = $user->firstName;
    }

    You may notice that this particular list comprehension provides the
    same functionality as array_column(), just in a little more
    generalized way. E.g. you could use all of the following without
    having special functions for them all:

    $firstNames = [foreach ($users as $user) yield $user->firstName];

    $firstNames = [foreach ($users as $user) yield $user->getFirstName()];

    $firstNames = [foreach ($users as $user) yield $user['firstName']];

    It's also possible to explicitly specify a key:

    $firstNames = [foreach ($users as $user) yield $user->id =>
    $user->firstName];

    It is also possible to filter elements using list comprehensions:

    $underageUsers = [foreach ($users as $user) if ($user->age < 18)
    yield $user];
    // or just the names
    $underageUserNames = [foreach ($users as $user) if ($user->age <
    18) yield $user->firstName];

    It is also possible to nest multiple foreach loops:

    $aList = ['A', 'B'];
    $bList = [1, 2];
    $combinations = [foreach ($aList as $a) foreach ($bList as $b)
    yield [$a, $b]];
    // gives: [ ['A', 1], ['A', 2], ['B', 1], ['B', 2] ]

    All the above are list comprehensions (or in PHP rather array
    comprehensions), i.e. they create an array as the result.

    If this is not needed it is also possible to compute the values lazily
    using generator expressions, which use () instead of [].

    $firstNames = (foreach ($users as $user) yield $user->firstName);

    In this case $firstNames will no longer be an array of first names,
    but instead will be a generator producing first names.

    This is handy if you only need to iterate the resulting "list" only
    once as it saves you holding the whole list in memory.

    Also it allows you to work with infinite lists easily:

    function *naturalNumbers() {
    for ($i = 0; ; ++$i) {
    yield $i;
    }
    }

    // all natural numbers
    $numbers = naturalNumbers();
    // only the odd ones
    $oddNumbers = (foreach ($numbers as $n) if ($n % 2) yield $n);
    // ...

    (At this point I wonder whether one should include support for
    for-loops in list comprehensions. So the naturalNumbers() function
    could be replaced with (for ($i = 0;; ++$i) yield $i), etc)

    So, what do you think? Do we want something like this in PHP?

    Nikita

    --
    PHP Internals - PHP Runtime Development Mailing List
    To unsubscribe, visit: http://www.php.net/unsub.php
  • David Muir at Jun 28, 2012 at 1:22 pm
    I'd assume that array_map() only works with arrays, while list comprehension should work with anything traversable.

    David

    On 28/06/2012, at 10:43 PM, Sebastian Krebs wrote:

    Hi,

    Whats the difference to the (already existing) function array_map() (except
    the syntax and one more new keyword)?
    $firstNames = array_map(function($user){return $user->firstname;},
    $users);

    Don't want to rewrite every example you gave, but you probably see my point.

    Regards,
    Sebastian

    2012/6/28 Nikita Popov <nikita.ppv@gmail.com>
    Hi internals!

    Python and several other languages include support for list
    comprehensions and generator expressions and I'd like to see something
    similar in PHP too.

    I created a hacky proof of concept implementation here:
    https://github.com/nikic/php-src/tree/addListComprehensions. It's
    really dirty, but it implements both features in about ~150 lines of
    code.

    Currently I'm using the following syntax:

    $firstNames = [foreach ($users as $user) yield $user->firstName];

    This code is roughly equivalent to writing:

    $firstNames = [];
    foreach ($users as $user) {
    $firstNames[] = $user->firstName;
    }

    You may notice that this particular list comprehension provides the
    same functionality as array_column(), just in a little more
    generalized way. E.g. you could use all of the following without
    having special functions for them all:

    $firstNames = [foreach ($users as $user) yield $user->firstName];

    $firstNames = [foreach ($users as $user) yield $user->getFirstName()];

    $firstNames = [foreach ($users as $user) yield $user['firstName']];

    It's also possible to explicitly specify a key:

    $firstNames = [foreach ($users as $user) yield $user->id =>
    $user->firstName];

    It is also possible to filter elements using list comprehensions:

    $underageUsers = [foreach ($users as $user) if ($user->age < 18)
    yield $user];
    // or just the names
    $underageUserNames = [foreach ($users as $user) if ($user->age <
    18) yield $user->firstName];

    It is also possible to nest multiple foreach loops:

    $aList = ['A', 'B'];
    $bList = [1, 2];
    $combinations = [foreach ($aList as $a) foreach ($bList as $b)
    yield [$a, $b]];
    // gives: [ ['A', 1], ['A', 2], ['B', 1], ['B', 2] ]

    All the above are list comprehensions (or in PHP rather array
    comprehensions), i.e. they create an array as the result.

    If this is not needed it is also possible to compute the values lazily
    using generator expressions, which use () instead of [].

    $firstNames = (foreach ($users as $user) yield $user->firstName);

    In this case $firstNames will no longer be an array of first names,
    but instead will be a generator producing first names.

    This is handy if you only need to iterate the resulting "list" only
    once as it saves you holding the whole list in memory.

    Also it allows you to work with infinite lists easily:

    function *naturalNumbers() {
    for ($i = 0; ; ++$i) {
    yield $i;
    }
    }

    // all natural numbers
    $numbers = naturalNumbers();
    // only the odd ones
    $oddNumbers = (foreach ($numbers as $n) if ($n % 2) yield $n);
    // ...

    (At this point I wonder whether one should include support for
    for-loops in list comprehensions. So the naturalNumbers() function
    could be replaced with (for ($i = 0;; ++$i) yield $i), etc)

    So, what do you think? Do we want something like this in PHP?

    Nikita

    --
    PHP Internals - PHP Runtime Development Mailing List
    To unsubscribe, visit: http://www.php.net/unsub.php
  • Sebastian Krebs at Jun 28, 2012 at 1:45 pm
    Hi,

    2012/6/28 David Muir <davidkmuir@gmail.com>
    I'd assume that array_map() only works with arrays, while list
    comprehension should work with anything traversable.
    That sounds more like a "bug" (better: "Missing feature") in array_map()
    and less a missing language feature...


    Regards,
    Sebastian


    David

    On 28/06/2012, at 10:43 PM, Sebastian Krebs wrote:

    Hi,

    Whats the difference to the (already existing) function array_map() (except
    the syntax and one more new keyword)?
    $firstNames = array_map(function($user){return $user->firstname;},
    $users);

    Don't want to rewrite every example you gave, but you probably see my point.
    Regards,
    Sebastian

    2012/6/28 Nikita Popov <nikita.ppv@gmail.com>
    Hi internals!

    Python and several other languages include support for list
    comprehensions and generator expressions and I'd like to see something
    similar in PHP too.

    I created a hacky proof of concept implementation here:
    https://github.com/nikic/php-src/tree/addListComprehensions. It's
    really dirty, but it implements both features in about ~150 lines of
    code.

    Currently I'm using the following syntax:

    $firstNames = [foreach ($users as $user) yield $user->firstName];

    This code is roughly equivalent to writing:

    $firstNames = [];
    foreach ($users as $user) {
    $firstNames[] = $user->firstName;
    }

    You may notice that this particular list comprehension provides the
    same functionality as array_column(), just in a little more
    generalized way. E.g. you could use all of the following without
    having special functions for them all:

    $firstNames = [foreach ($users as $user) yield $user->firstName];

    $firstNames = [foreach ($users as $user) yield $user->getFirstName()];

    $firstNames = [foreach ($users as $user) yield $user['firstName']];

    It's also possible to explicitly specify a key:

    $firstNames = [foreach ($users as $user) yield $user->id =>
    $user->firstName];

    It is also possible to filter elements using list comprehensions:

    $underageUsers = [foreach ($users as $user) if ($user->age < 18)
    yield $user];
    // or just the names
    $underageUserNames = [foreach ($users as $user) if ($user->age <
    18) yield $user->firstName];

    It is also possible to nest multiple foreach loops:

    $aList = ['A', 'B'];
    $bList = [1, 2];
    $combinations = [foreach ($aList as $a) foreach ($bList as $b)
    yield [$a, $b]];
    // gives: [ ['A', 1], ['A', 2], ['B', 1], ['B', 2] ]

    All the above are list comprehensions (or in PHP rather array
    comprehensions), i.e. they create an array as the result.

    If this is not needed it is also possible to compute the values lazily
    using generator expressions, which use () instead of [].

    $firstNames = (foreach ($users as $user) yield $user->firstName);

    In this case $firstNames will no longer be an array of first names,
    but instead will be a generator producing first names.

    This is handy if you only need to iterate the resulting "list" only
    once as it saves you holding the whole list in memory.

    Also it allows you to work with infinite lists easily:

    function *naturalNumbers() {
    for ($i = 0; ; ++$i) {
    yield $i;
    }
    }

    // all natural numbers
    $numbers = naturalNumbers();
    // only the odd ones
    $oddNumbers = (foreach ($numbers as $n) if ($n % 2) yield $n);
    // ...

    (At this point I wonder whether one should include support for
    for-loops in list comprehensions. So the naturalNumbers() function
    could be replaced with (for ($i = 0;; ++$i) yield $i), etc)

    So, what do you think? Do we want something like this in PHP?

    Nikita

    --
    PHP Internals - PHP Runtime Development Mailing List
    To unsubscribe, visit: http://www.php.net/unsub.php
  • David Muir at Jun 29, 2012 at 4:46 am

    On 28/06/12 23:45, Sebastian Krebs wrote:
    Hi,

    2012/6/28 David Muir <davidkmuir@gmail.com>
    I'd assume that array_map() only works with arrays, while list
    comprehension should work with anything traversable.
    That sounds more like a "bug" (better: "Missing feature") in array_map()
    and less a missing language feature...
    All the array_* functions only work with arrays, and don't allow
    objects, even ones implementing ArrayAccess.

    It's been there as a feature request since 2007:
    https://bugs.php.net/bug.php?id=43650

    Cheers,
    David
  • Nikita Popov at Jun 28, 2012 at 5:05 pm

    On Thu, Jun 28, 2012 at 2:43 PM, Sebastian Krebs wrote:
    Hi,

    Whats the difference to the (already existing) function array_map() (except
    the syntax and one more new keyword)?
    $firstNames = array_map(function($user){return $user->firstname;},
    $users);

    Don't want to rewrite every example you gave, but you probably see my point.
    They are roughly the same. List comprehensions are basically a way of
    mapping + filtering. Their main advantage is a dedicated, more concise
    syntax and also the ability to combine multiple maps and filters in
    one expression.

    Also PHP is not a particularly lambda-y language in general; using
    array_map for that purpose would feel "wrong" to me. I'd probably
    rather write out a full foreach loop than use map.

    Generator expressions additionally bring the concept of lazy
    evaluation into the mix. This is useful in several situations, e.g.
    when processing large amounts of data, like log files.

    The following Python example is taken from
    http://www.dabeaz.com/generators/Generators.pdf:

    wwwlog = open("access-log")
    bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)
    bytes = (int(x) for x in bytecolumn if x != '-')
    print "Total", sum(bytes)

    The code is written as a set of simple operations, which are all
    defined in terms of iterating a whole file/list. But the actual
    execution is lazy, thus basically applying all the operations in a
    "pipeline". So the whole potentially multi-gig log file is never
    loaded into memory.

    The same could be done in PHP, just a different syntax :)

    Nikita
  • Jared Williams at Jun 28, 2012 at 1:23 pm

    -----Original Message-----
    From: Nikita Popov
    Sent: 28 June 2012 11:49
    To: PHP internals
    Subject: [PHP-DEV] List comprehensions and generator
    expressions for PHP

    Hi internals!

    Python and several other languages include support for list
    comprehensions and generator expressions and I'd like to see something
    similar in PHP too.

    I created a hacky proof of concept implementation here:
    https://github.com/nikic/php-src/tree/addListComprehensions. It's
    really dirty, but it implements both features in about ~150 lines of
    code.

    Currently I'm using the following syntax:

    $firstNames = [foreach ($users as $user) yield
    $user->firstName];
    This code is roughly equivalent to writing:

    $firstNames = [];
    foreach ($users as $user) {
    $firstNames[] = $user->firstName;
    }

    You may notice that this particular list comprehension provides the
    same functionality as array_column(), just in a little more
    generalized way. E.g. you could use all of the following without
    having special functions for them all:

    $firstNames = [foreach ($users as $user) yield
    $user->firstName];
    $firstNames = [foreach ($users as $user) yield
    $user->getFirstName()];

    $firstNames = [foreach ($users as $user) yield
    $user['firstName']];

    It's also possible to explicitly specify a key:

    $firstNames = [foreach ($users as $user) yield $user->id =>
    $user->firstName];

    It is also possible to filter elements using list comprehensions:

    $underageUsers = [foreach ($users as $user) if ($user->age < 18)
    yield $user];
    // or just the names
    $underageUserNames = [foreach ($users as $user) if ($user->age <
    18) yield $user->firstName];

    It is also possible to nest multiple foreach loops:

    $aList = ['A', 'B'];
    $bList = [1, 2];
    $combinations = [foreach ($aList as $a) foreach ($bList as $b)
    yield [$a, $b]];
    // gives: [ ['A', 1], ['A', 2], ['B', 1], ['B', 2] ]

    All the above are list comprehensions (or in PHP rather array
    comprehensions), i.e. they create an array as the result.

    If this is not needed it is also possible to compute the values lazily
    using generator expressions, which use () instead of [].

    $firstNames = (foreach ($users as $user) yield
    $user->firstName);
    In this case $firstNames will no longer be an array of first names,
    but instead will be a generator producing first names.

    This is handy if you only need to iterate the resulting "list" only
    once as it saves you holding the whole list in memory.

    Also it allows you to work with infinite lists easily:

    function *naturalNumbers() {
    for ($i = 0; ; ++$i) {
    yield $i;
    }
    }

    // all natural numbers
    $numbers = naturalNumbers();
    // only the odd ones
    $oddNumbers = (foreach ($numbers as $n) if ($n % 2) yield $n);
    // ...

    (At this point I wonder whether one should include support for
    for-loops in list comprehensions. So the naturalNumbers() function
    could be replaced with (for ($i = 0;; ++$i) yield $i), etc)

    So, what do you think? Do we want something like this in PHP?
    No mention of yielding keys in the comprehensions. Presume that would
    work?

    I would definitely like to see generators in, the amount of
    boilerplate
    code required for the current Iterator interface is not nice.

    Also have been playing with your addGeneratorsSupport branch, trying
    to see
    If can get any benefits from using asynchronous operations.

    The simplest example I could think of using Memcached
    (assuming libmemcached fetch() does return as soon as it recieves an
    item)

    Eg.

    function *getMulti(Memcached $memcached, array $keys)
    {
    if ($memcached->getDelayed($keys)) {
    while ($item = $memcached->fetch())
    yield $item['key'] => $item['value'];
    }
    }

    foreach(getMulti($memcached, ['foo', 'bar', ... ]) as $key => $value)
    {
    doSomeWork();
    }

    So doSomeWork would be called as soon as a single value is available,
    rather
    than having to wait til all values have returned.

    Should in theory work right?

    Jared
  • Nikita Popov at Jun 28, 2012 at 5:28 pm

    No mention of yielding keys in the comprehensions. Presume that would
    work?
    Yup, that works too. I also included an example in the mail:

    $firstNames = [foreach ($users as $user) yield $user->id =>
    $user->firstName];

    This creates a map from ids to first names.
    The simplest example I could think of using Memcached
    (assuming libmemcached fetch() does return as soon as it recieves an
    item)

    Eg.

    function *getMulti(Memcached $memcached, array $keys)
    {
    if ($memcached->getDelayed($keys)) {
    while ($item = $memcached->fetch())
    yield $item['key'] => $item['value'];
    }
    }

    foreach(getMulti($memcached, ['foo', 'bar', ... ]) as $key => $value)
    {
    doSomeWork();
    }

    So doSomeWork would be called as soon as a single value is available,
    rather
    than having to wait til all values have returned.

    Should in theory work right?
    Yup, that should work :)

    Nikita

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupphp-internals @
categoriesphp
postedJun 28, '12 at 11:00a
activeJun 29, '12 at 4:46a
posts9
users5
websitephp.net

People

Translate

site design / logo © 2022 Grokbase