FAQ
Hello

I am trying to pass an object of type LexicalizedParser (which is from an
imported jar file stanford-parser) from the main method to the Mapper Class.
This is to load the trained model first and then apply on every file offset
in the map function in the Mapper Class.
Could you tell me the best way to pass an object of any type to the Mapper?

Thanking you
Rishav


--
Rishav Bhowmick
Computer Science -2010
Carnegie Mellon University
Tel: +974-561-4371
e-mail: rishavb@cmu.edu

Search Discussions

  • J.G.Konrad at Apr 20, 2010 at 6:21 pm
    I would suggest adding a configure() method to your Map class and then load
    the parser and save it in a class object.
    On Tue, Apr 20, 2010 at 4:16 AM, Rishav wrote:

    Hello

    I am trying to pass an object of type LexicalizedParser (which is from an
    imported jar file stanford-parser) from the main method to the Mapper Class.
    This is to load the trained model first and then apply on every file offset
    in the map function in the Mapper Class.
    Could you tell me the best way to pass an object of any type to the Mapper?

    Thanking you
    Rishav


    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu
  • Rishav at Apr 20, 2010 at 7:18 pm
    So My mapper has:
    public static LexicalizedParser lp;

    public void configure(JobConf job) {
    lp = new LexicalizedParser("/user/me/project4/englishPCFG.ser.gz");
    }


    I get a runtimeException of error in configuration object. If you notice
    later on in the exception thread, the parser actually gets loaded. My hunch
    is it is happening in the master node which can access it. The
    englishPCFG.ser.gz trained model is on hdfs.



    10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000000_0, Status : FAILED
    java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
    Caused by: java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
    Caused by: java.lang.RuntimeException: File not found:
    /user/me/project4/englishPCFG.ser.gz
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
    Caused by: java.io.FileNotFoundException:
    /user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

    attempt_201003091315_0078_m_000000_0: Loading parser from serialized file
    /user/me/project4/englishPCFG.ser.gz ...
    10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000001_0, Status : FAILED
    java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
    Caused by: java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
    Caused by: java.lang.RuntimeException: File not found:
    /user/me/project4/englishPCFG.ser.gz
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
    Caused by: java.io.FileNotFoundException:
    /user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

    attempt_201003091315_0078_m_000001_0: Loading parser from serialized file
    /user/me/project4/englishPCFG.ser.gz ...
    10/04/20 22:14:30 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000000_1, Status : FAILED
    java.lang.RuntimeException: Error in configuring object
    On Tue, Apr 20, 2010 at 9:21 PM, J.G.Konrad wrote:

    I would suggest adding a configure() method to your Map class and then load
    the parser and save it in a class object.

    On Tue, Apr 20, 2010 at 4:16 AM, Rishav wrote:

    Hello

    I am trying to pass an object of type LexicalizedParser (which is from an
    imported jar file stanford-parser) from the main method to the Mapper Class.
    This is to load the trained model first and then apply on every file offset
    in the map function in the Mapper Class.
    Could you tell me the best way to pass an object of any type to the
    Mapper?

    Thanking you
    Rishav


    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu

    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu
  • J.G.Konrad at Apr 20, 2010 at 7:42 pm
    The LexicalizedParser does not know how to read from the dfs. You could
    leverage the distributed cache to copy the grammar to the work nodes and
    load them that way. That would look something like

    // initialize cache with file from dfs
    DistributedCache.addCacheFile(new
    URI("/user/me/project4/englishPCFG.ser.gz"), conf);

    // in Map configure
    Path[] ps = DistributedCache.getLocalCacheFiles(conf);
    // assuming only one file
    lp = new LexicalizedParser(ps[0].toString());

    Another option would be to server your grammar from a webserver so then you
    could just give you lp a url and it will download and load the grammer. I
    don't know if you have access to a webserver to do that. If you do have
    access then you would load the grammer like

    lp = new LexicalizedParser("http://some-domain/englishPCFG.ser.gz")


    On Tue, Apr 20, 2010 at 12:18 PM, Rishav wrote:

    So My mapper has:
    public static LexicalizedParser lp;

    public void configure(JobConf job) {
    lp = new LexicalizedParser("/user/me/project4/englishPCFG.ser.gz");
    }


    I get a runtimeException of error in configuration object. If you notice
    later on in the exception thread, the parser actually gets loaded. My hunch
    is it is happening in the master node which can access it. The
    englishPCFG.ser.gz trained model is on hdfs.



    10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000000_0, Status : FAILED
    java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
    Caused by: java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
    Caused by: java.lang.RuntimeException: File not found:
    /user/me/project4/englishPCFG.ser.gz
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
    Caused by: java.io.FileNotFoundException:
    /user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:106)
    at java.io.FileInputStream.<init>(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

    attempt_201003091315_0078_m_000000_0: Loading parser from serialized file
    /user/me/project4/englishPCFG.ser.gz ...
    10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000001_0, Status : FAILED
    java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
    Caused by: java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
    Caused by: java.lang.RuntimeException: File not found:
    /user/me/project4/englishPCFG.ser.gz
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
    Caused by: java.io.FileNotFoundException:
    /user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:106)
    at java.io.FileInputStream.<init>(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

    attempt_201003091315_0078_m_000001_0: Loading parser from serialized file
    /user/me/project4/englishPCFG.ser.gz ...
    10/04/20 22:14:30 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000000_1, Status : FAILED
    java.lang.RuntimeException: Error in configuring object

    On Tue, Apr 20, 2010 at 9:21 PM, J.G.Konrad wrote:

    I would suggest adding a configure() method to your Map class and then
    load the parser and save it in a class object.

    On Tue, Apr 20, 2010 at 4:16 AM, Rishav wrote:

    Hello

    I am trying to pass an object of type LexicalizedParser (which is from an
    imported jar file stanford-parser) from the main method to the Mapper Class.
    This is to load the trained model first and then apply on every file offset
    in the map function in the Mapper Class.
    Could you tell me the best way to pass an object of any type to the
    Mapper?

    Thanking you
    Rishav


    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu

    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu
  • Rishav at Apr 20, 2010 at 8:17 pm
    Thanks. :) That helped a lot.
    -Rishav
    On Tue, Apr 20, 2010 at 10:42 PM, J.G.Konrad wrote:

    The LexicalizedParser does not know how to read from the dfs. You could
    leverage the distributed cache to copy the grammar to the work nodes and
    load them that way. That would look something like

    // initialize cache with file from dfs
    DistributedCache.addCacheFile(new
    URI("/user/me/project4/englishPCFG.ser.gz"), conf);

    // in Map configure
    Path[] ps = DistributedCache.getLocalCacheFiles(conf);
    // assuming only one file
    lp = new LexicalizedParser(ps[0].toString());

    Another option would be to server your grammar from a webserver so then
    you could just give you lp a url and it will download and load the grammer.
    I don't know if you have access to a webserver to do that. If you do have
    access then you would load the grammer like

    lp = new LexicalizedParser("http://some-domain/englishPCFG.ser.gz")


    On Tue, Apr 20, 2010 at 12:18 PM, Rishav wrote:

    So My mapper has:
    public static LexicalizedParser lp;

    public void configure(JobConf job) {
    lp = new
    LexicalizedParser("/user/me/project4/englishPCFG.ser.gz");
    }


    I get a runtimeException of error in configuration object. If you notice
    later on in the exception thread, the parser actually gets loaded. My hunch
    is it is happening in the master node which can access it. The
    englishPCFG.ser.gz trained model is on hdfs.



    10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000000_0, Status : FAILED
    java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
    Caused by: java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
    Caused by: java.lang.RuntimeException: File not found:
    /user/me/project4/englishPCFG.ser.gz
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
    Caused by: java.io.FileNotFoundException:
    /user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:106)
    at java.io.FileInputStream.<init>(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

    attempt_201003091315_0078_m_000000_0: Loading parser from serialized file
    /user/me/project4/englishPCFG.ser.gz ...
    10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000001_0, Status : FAILED
    java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
    Caused by: java.lang.RuntimeException: Error in configuring object
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
    Caused by: java.lang.RuntimeException: File not found:
    /user/me/project4/englishPCFG.ser.gz
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
    Caused by: java.io.FileNotFoundException:
    /user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:106)
    at java.io.FileInputStream.<init>(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
    edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

    attempt_201003091315_0078_m_000001_0: Loading parser from serialized file
    /user/me/project4/englishPCFG.ser.gz ...
    10/04/20 22:14:30 INFO mapred.JobClient: Task Id :
    attempt_201003091315_0078_m_000000_1, Status : FAILED
    java.lang.RuntimeException: Error in configuring object

    On Tue, Apr 20, 2010 at 9:21 PM, J.G.Konrad wrote:

    I would suggest adding a configure() method to your Map class and then
    load the parser and save it in a class object.

    On Tue, Apr 20, 2010 at 4:16 AM, Rishav wrote:

    Hello

    I am trying to pass an object of type LexicalizedParser (which is from
    an imported jar file stanford-parser) from the main method to the Mapper
    Class. This is to load the trained model first and then apply on every file
    offset in the map function in the Mapper Class.
    Could you tell me the best way to pass an object of any type to the
    Mapper?

    Thanking you
    Rishav


    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu

    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu

    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu
  • Kris Nuttycombe at Apr 20, 2010 at 6:38 pm
    Having been down this road recently, the best results I've been able
    to obtain is to use the Distributed Cache and Java serialization; if
    your LexicalizedParser is not serializable then you'll have to jump
    through more hoops, but the basic approach can be the same.

    Here's some Scala code that I use to set things up and retrieve them
    properly through the Distributed Cache:

    package socialmedia.common.hadoop

    import socialmedia.common.util.Util._
    import org.apache.hadoop.conf._
    import org.apache.hadoop.filecache._
    import org.apache.hadoop.fs._
    import org.apache.hadoop.io._
    import java.io.ObjectInputStream
    import java.io.ObjectOutputStream
    import java.io.Serializable
    import java.util.UUID
    import net.lag.logging.Logger
    import DistCacheResources._

    object DistCacheResources {
    private val log = Logger.get
    val serializedResourceDir = "dist_cache_resource.serfile"
    val serializedResourceUUID = "dist_cache_resource.uuid"

    def addToDistCache[T <: Serializable](conf: Configuration, uuid:
    UUID, resource: T): Unit = {
    using(FileSystem.get(conf)) { fs =>
    val path = new Path(fs.getHomeDirectory, new
    Path(serializedResourceDir, uuid.toString + ".ser"))
    log.info("Writing serialized resource to " + path)
    using(fs.create(path)) {
    out => using(new ObjectOutputStream(out)) {
    oout => oout.writeObject(resource)
    }
    }

    conf.set(serializedResourceUUID, uuid.toString)
    DistributedCache.addCacheFile(path.toUri, conf)
    }
    }
    }

    trait DistCacheResources[T] extends Configurable {
    private var _resource: Option[T] = _
    private var _conf: Configuration = _

    def resource: Option[T] = _resource

    override def setConf(conf: Configuration) = _conf = conf
    override def getConf = _conf

    def init(conf: Configuration): Unit = {
    if (conf == null) error("Configuration was null.")
    val uuid = UUID.fromString(conf.get(serializedResourceUUID))
    log.info("Will attempt to obtain serialized resource for uuid: " + uuid)
    val cacheFiles = DistributedCache.getLocalCacheFiles(conf)
    log.info("Available cache files: " + cacheFiles.mkString)
    _resource = cacheFiles.find(_.getName == uuid.toString + ".ser").map {
    resPath => using(FileSystem.getLocal(conf)) {
    fs => using(fs.open(resPath)) {
    in => using(new ObjectInputStream(in)) {
    oin => oin.readObject.asInstanceOf[T]
    }
    }
    }
    }
    }
    }

    Then, you can have something like :

    class MyMapper extends Mapper[BW, BW, Text, BW] with
    DistCacheResources[LexicalizedParser] {
    override protected def setup(context: HMapper[BW, BW, Text,
    BW]#Context): Unit = {
    init(getConf)
    }

    // ...
    }

    And, in your job configuration, you just call
    DistCacheResources.addToDistCache(...)


    Kris
    On Tue, Apr 20, 2010 at 5:16 AM, Rishav wrote:
    Hello

    I am trying to pass an object of type LexicalizedParser (which is from an
    imported jar file stanford-parser) from the main method to the Mapper Class.
    This is to load the trained model first and then apply on every file offset
    in the map function in the Mapper Class.
    Could you tell me the best way to pass an object of any type to the Mapper?

    Thanking you
    Rishav


    --
    Rishav Bhowmick
    Computer Science -2010
    Carnegie Mellon University
    Tel: +974-561-4371
    e-mail: rishavb@cmu.edu

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedApr 20, '10 at 11:17a
activeApr 20, '10 at 8:17p
posts6
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase