|| at Oct 27, 2008 at 8:13 pm
From: James Mills [mailto:prologic at shortcircuit.net.au]
Sent: Sunday, October 26, 2008 5:26 PM
Cc: python-list at python.org
Subject: Re: Web crawler on python
On Mon, Oct 27, 2008 at 6:54 AM, sonich wrote:
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?
Simple, but it works. Extend it all you like.http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/e
$ spider.py --help
Usage: spider.py [options] <url>
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depthÞPTH
Maximum depth to traverse
-- "Problems are solved by method"