FAQ
hi

i am running nutch crawl and i have specified the depth as 200 but in the
console it is showing Stopping at depth=1 - no more URLs to fetch.

it is not crawling the website completely
even i cant find the errors in log file

please help me out

--
View this message in context: http://old.nabble.com/Crawl-depth-problem-tp27284306p27284306.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Search Discussions

  • Lyndon Maydwell at Jan 23, 2010 at 10:07 am
    Have you set up your regex-urlfilter.txt correctly? I've been caught
    out by this before.
    On Sat, Jan 23, 2010 at 4:31 PM, zud wrote:

    hi

    i am running nutch crawl and i have specified the depth as 200 but in the
    console it is showing  Stopping at depth=1 - no more URLs to fetch.

    it is not crawling the website completely
    even i cant find the errors in log file

    please help me out

    --
    View this message in context: http://old.nabble.com/Crawl-depth-problem-tp27284306p27284306.html
    Sent from the Nutch - User mailing list archive at Nabble.com.
  • MilleBii at Jan 23, 2010 at 1:43 pm
    If you use crawl command you need to set crawl-filters.txt for your
    site (not the regex filter)
    Depth of 200 is huge, becareful depth actually means the number of
    fetch/parse/index cycles.

    2010/1/23, Lyndon Maydwell <maydwell@gmail.com>:
    Have you set up your regex-urlfilter.txt correctly? I've been caught
    out by this before.
    On Sat, Jan 23, 2010 at 4:31 PM, zud wrote:

    hi

    i am running nutch crawl and i have specified the depth as 200 but in
    the
    console it is showing  Stopping at depth=1 - no more URLs to fetch.

    it is not crawling the website completely
    even i cant find the errors in log file

    please help me out

    --
    View this message in context:
    http://old.nabble.com/Crawl-depth-problem-tp27284306p27284306.html
    Sent from the Nutch - User mailing list archive at Nabble.com.

    --
    -MilleBii-

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriesnutch, lucene
postedJan 23, '10 at 8:32a
activeJan 23, '10 at 1:43p
posts3
users3
websitenutch.apache.org

3 users in discussion

Zud: 1 post Lyndon Maydwell: 1 post MilleBii: 1 post

People

Translate

site design / logo © 2022 Grokbase