Grokbase Groups Pig user August 2010
FAQ
Hi All,

Just wanted to know has anyone used Pig on a standalone basis in production environment i.e. without integrating with Haddoop? Is that even a good idea?

How is the performance if we try parsing big files using Pig. Typically we use perl for this but often we run into memory management issues and Pig seems
to be much simpler than perl.

Please share if you have any inputs.

Thanks,
Somdip.

Search Discussions

  • Somdip at Aug 27, 2010 at 1:04 am
    Hi All,

    Just wanted to know has anyone used Pig on a standalone basis in production environment i.e. without integrating with Haddoop? Is that even a good idea?

    How is the performance if we try parsing big files using Pig. Typically we use perl for this but often we run into memory management issues and Pig seems
    to be much simpler than perl.

    Please share if you have any inputs.

    Thanks,
    Somdip.
  • Ravi Phulari at Aug 27, 2010 at 1:11 am
    Somdip,
    I will be surprised to know if anyone is using Pig on a standalone basis in production environment . Pig uses Hadoop's power to process huge data.
    HDFS for storing large data and MapReduce for using distributed system for processing huge data.

    You can use Pig in standalone mode for learning purpose but using it on a standalone basis in production environment won't be useful.

    -
    Ravi


    On 8/26/10 5:38 PM, "Somdip" wrote:

    Hi All,

    Just wanted to know has anyone used Pig on a standalone basis in production environment i.e. without integrating with Haddoop? Is that even a good idea?

    How is the performance if we try parsing big files using Pig. Typically we use perl for this but often we run into memory management issues and Pig seems
    to be much simpler than perl.

    Please share if you have any inputs.

    Thanks,
    Somdip.


    --
  • Jeff Zhang at Aug 27, 2010 at 1:07 am
    It is not recommended using Pig standalone mode to process big files
    if you concern the performance. I believe perl's performance will be
    better than pig in local mode.


    On Thu, Aug 26, 2010 at 5:42 PM, Somdip wrote:
    Hi All,

    Just wanted to know has anyone used Pig on a standalone basis in production environment i.e. without integrating with Haddoop? Is that even a good idea?

    How is the performance if we try parsing big files using Pig. Typically we use perl for this but often we run into memory management issues  and Pig seems
    to be much simpler than perl.

    Please share if you have any inputs.

    Thanks,
    Somdip.


    --
    Best Regards

    Jeff Zhang

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedAug 27, '10 at 12:43a
activeAug 27, '10 at 1:11a
posts4
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase