FAQ
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

Search Discussions

  • Mr.SpOOn at Oct 26, 2008 at 9:03 pm

    On Sun, Oct 26, 2008 at 9:54 PM, sonich wrote:
    I need simple web crawler,
    I found Ruya, but it's seems not currently maintained.
    Does anybody know good web crawler on python or with python interface?
    What about BeautifulSoup?

    http://www.crummy.com/software/BeautifulSoup/
  • James Mills at Oct 26, 2008 at 10:25 pm

    On Mon, Oct 27, 2008 at 6:54 AM, sonich wrote:
    I need simple web crawler,
    I found Ruya, but it's seems not currently maintained.
    Does anybody know good web crawler on python or with python interface?
    Simple, but it works. Extend it all you like.

    http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/examples/spider.py

    $ spider.py --help
    Usage: spider.py [options] <url>

    Options:
    --version show program's version number and exit
    -h, --help show this help message and exit
    -q, --quiet Enable quiet mode
    -l, --links Get links for specified url only
    -d DEPTH, --depthÞPTH
    Maximum depth to traverse

    cheers
    James

    --
    --
    -- "Problems are solved by method"
  • Support Desk at Oct 27, 2008 at 8:13 pm
    -----Original Message-----
    From: James Mills [mailto:prologic at shortcircuit.net.au]
    Sent: Sunday, October 26, 2008 5:26 PM
    To: sonich
    Cc: python-list at python.org
    Subject: Re: Web crawler on python
    On Mon, Oct 27, 2008 at 6:54 AM, sonich wrote:
    I need simple web crawler,
    I found Ruya, but it's seems not currently maintained.
    Does anybody know good web crawler on python or with python interface?
    Simple, but it works. Extend it all you like.

    http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/e
    xamples/spider.py

    $ spider.py --help
    Usage: spider.py [options] <url>

    Options:
    --version show program's version number and exit
    -h, --help show this help message and exit
    -q, --quiet Enable quiet mode
    -l, --links Get links for specified url only
    -d DEPTH, --depthÞPTH
    Maximum depth to traverse

    cheers
    James

    --
    --
    -- "Problems are solved by method"
  • Alex at Oct 28, 2008 at 5:22 pm

    On Oct 26, 9:54?pm, sonich wrote:
    I need simple web crawler,
    I found Ruya, but it's seems not currently maintained.
    Does anybody know good web crawler on python or with python interface?
    You should try Orchid http://pypi.python.org/pypi/Orchid/1.1
    or you can have a look at my project on launchpad
    https://code.launchpad.net/~esaurito/jazz-crawler/experimental.
    It's a single site crawler but you can easily modified it.

    Bye.

    Alex
  • Yura at Oct 30, 2008 at 10:13 pm
    I need simple web crawler, I found Ruya, but it's seems not currently
    maintained. Does anybody know good web crawler on python or with
    python interface?
    http://watch-me.890m.com
  • James Mills at Oct 30, 2008 at 10:25 pm

    On Fri, Oct 31, 2008 at 8:13 AM, yura wrote:
    I need simple web crawler, I found Ruya, but it's seems not currently
    maintained. Does anybody know good web crawler on python or with
    python interface?
    http://watch-me.890m.com
    http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/edc08c87ecb7/examples/spider.py

    cheers
    James

    --
    --
    -- "Problems are solved by method"

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedOct 26, '08 at 8:54p
activeOct 30, '08 at 10:25p
posts7
users6
websitepython.org

People

Translate

site design / logo © 2022 Grokbase