FAQ
I wrote spider for this but I haven't kept up with maintaining it:

https://github.com/mikeal/spider

If you end up using it and have improvements to make i'll add you as a contributor.

-Mikeal
On Jan 20, 2014, at 10:44AM, Tim Killian wrote:

Someone has already mentioned using cheerio, and I second that. I built a basic web crawler/scraper using nothing but the requests and cheerio libraries and it worked great. If you already know JQuery you already know how to use Cheerio, which was also a big plus for me.

On Wednesday, January 15, 2014 6:09:48 PM UTC-8, Victor Hooi wrote:
Hi,

I'm wondering if anybody knows of any web-scraping frameworks in Node.JS?

Previously, there was node.io (https://github.com/chriso/node.io), however, the project was recently discontinued.

Googling for Node.JS and web scraping, most of the guides online just talk about using requests and cheerio - it works, but you need to handle a whole bunch of things yourself (throttling, distributing jobs, configuration, managing jobs etc.).

On the Python side, I know of Scrapy (https://github.com/scrapy/scrapy), which is using Twisted for asynchronicity

On the Ruby side, Nokogiri (http://nokogiri.org/) is meant to be good, although I haven't dived into it much.

Is there anything equivalent in the Node world? Or what approaches are people using to tackle this problem?

Cheers,
Victor

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 9 of 35 | next ›
Discussion Overview
groupnodejs @
categoriesnodejs
postedJan 16, '14 at 2:09a
activeJul 3, '14 at 2:54p
posts35
users13
websitenodejs.org
irc#node.js

People

Translate

site design / logo © 2021 Grokbase