FAQ
I can't say I've found anything better -- I'm doing something similar in
Python. After writing several spouts and bolts this way, I put together a
framework so the "standard" code is in a reusable script and you only have
a write a short script with your spout or bolt-specific code.

If you take a look starting at line 147 of this file:

https://github.com/AirSage/Petrel/blob/master/petrel/petrel/package.py

you'll see some of the things we're doing in our startup script such as:

- Setting up a temporary directory for this topology (Storm doesn't like
it if you create additional directories under the one with your scripts)
- Checking that the right version of Python is installed
- Creating a private Python environment (a.k.a. virtualenv) for the
libraries required by this topology
- Installing required libraries (we use a lock file so that all the
spouts and bolts for a topology can share the same library installations)

Barry
On Tuesday, December 4, 2012 3:55:16 PM UTC-5, Jim Baugh wrote:

In the environment that I am running I use perlbrew to set the version of
perl I want to use prior to running a perl script. I do this by sourcing a
file that has all of my env vars set.

What is the best way to do this in storm?

Currently, I am wrapping my perl job with a bash script and call the bash
script from storm.

e.g. splitsentence.sh
#!/bin/bash
path=MultiLangDir
source "${path}/splitsentence.sh"
perl "${path}/splitsentence.pl"

e.g. storm call
public SplitSentence() {
super("sh", "splitsentence.sh");
}

Is there a better way to do this?

Thanks,

James

Search Discussions

  • Chris0wj at Apr 16, 2013 at 12:38 am
    Barry, what did you mean by "Storm doesn't like it if you create additional
    directories under the one with your scripts"? I am trying to debug a shell
    process (Python) issue myself and I have a module in a subdirectory of the
    resources directory.

    Has anyone else had issue with subdirectories in the resources directory?

    Has anyone else deployed or tested with a virtualenv?

    -Chris
    On Thursday, December 6, 2012 9:17:10 AM UTC-5, Barry Hart wrote:

    I can't say I've found anything better -- I'm doing something similar in
    Python. After writing several spouts and bolts this way, I put together a
    framework so the "standard" code is in a reusable script and you only have
    a write a short script with your spout or bolt-specific code.

    If you take a look starting at line 147 of this file:

    https://github.com/AirSage/Petrel/blob/master/petrel/petrel/package.py

    you'll see some of the things we're doing in our startup script such as:

    - Setting up a temporary directory for this topology (Storm doesn't
    like it if you create additional directories under the one with your
    scripts)
    - Checking that the right version of Python is installed
    - Creating a private Python environment (a.k.a. virtualenv) for the
    libraries required by this topology
    - Installing required libraries (we use a lock file so that all the
    spouts and bolts for a topology can share the same library installations)

    Barry
    On Tuesday, December 4, 2012 3:55:16 PM UTC-5, Jim Baugh wrote:

    In the environment that I am running I use perlbrew to set the version of
    perl I want to use prior to running a perl script. I do this by sourcing a
    file that has all of my env vars set.

    What is the best way to do this in storm?

    Currently, I am wrapping my perl job with a bash script and call the bash
    script from storm.

    e.g. splitsentence.sh
    #!/bin/bash
    path=MultiLangDir
    source "${path}/splitsentence.sh"
    perl "${path}/splitsentence.pl"

    e.g. storm call
    public SplitSentence() {
    super("sh", "splitsentence.sh");
    }

    Is there a better way to do this?

    Thanks,

    James
    --
    You received this message because you are subscribed to the Google Groups "storm-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupstorm-user @
postedDec 6, '12 at 2:17p
activeApr 16, '13 at 12:38a
posts2
users2
websitestorm-project.net
irc#storm-user

2 users in discussion

Barry Hart: 1 post Chris0wj: 1 post

People

Translate

site design / logo © 2022 Grokbase