FAQ
Hi Indraneil\Nathan,

I am facing exactly the same issue of "Cannot run 'python'". Could you
please tell me how was this issue resolved. I am downloaded
storm-0.9.0-wip4. Thanks in advance.

Regards,
YogiP
On Tuesday, 29 November 2011 10:13:40 UTC+5:30, indraniel wrote:

Nathan,

Your observation/suggestion seemed to have worked. I am now able to
submit storm jobs to my currently one-node cluster. Both the
ExclamationTopology and the WordCountTopology now work for me.

I also had to supply an argument for both the storm-starter topology
examples to enable remote job submissions as well.

storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology et-test
storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.WordCountTopology wc-test

I have not tried using Ubuntu 10.10 yet. The Linux distributions that
are currently available to me at my place of work, The Genome
Institute at Washington University -- http://genome.wustl.edu, are
based upon the Ubuntu long-term-support (LTS) distributions. Ubuntu
10.10 was not a LTS distribution, but 10.04 was. As an initial foray,
I was experimenting with Storm based upon the latest Ubuntu LTS
distribution available.

I will now continue onwards exploring the possible things I can
attempt with Storm. It seems like an interesting platform to possibly
improve the performance of a few genomics-related pipelines that occur
here. Thanks again for the help and advice so far!

-Indraniel
On Nov 22, 10:11 pm, nathanmarz wrote:
OK, I see what you're doing wrong here. You most definitely should not
touch zookeeper once you have it running.

If you look at the code for ExclamationTopology, you can see that it
submits in local mode if you provide no arguments to the main
function. Otherwise, it submits it in remote mode with the name of the
topology being the first argument to the main function. So you should
try submitting it in remote mode by providing an argument, e.g.:

storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology test-topology

You'll be able to see that it's working by seeing messages being
logged to the worker logs.

It sounds like you've been running WordCountTopology in local mode as
well, and the python isn't working from there. Have you tried using
Ubuntu 10.10?

-Nathan

On Nov 22, 4:15 pm, indraniel wrote:






Okay, I think I got the ExclamationTopology going by finagling the
zookeeper and storm invocations. I started up zookeeper to get nimbus
up
and running, but then stopped zookeeper to enable storm job
submissions
to nimbus.
The process is described in detail towards the end of this email.
After getting the ExclamationTopology (pure java) example going, I
similarly attempted the WordCountTopology (multilang example). But
again,
I got the 'Cannot run program "python"' output.
Is starting and stopping the zookeeper daemon as described below,
appropriate for a one-node cluster test environment? Was the
successful ExclamationTopology invocation really running in "local"
mode?
-Indraniel
# Invocation Details ########
1. Started up 4 xterms
2. xterm[1]: $ zkServer.sh start # started zookeeper
3. xterm[2]: $ storm nimbus # started nimbus daemon
saw the following in the nimbus.log:
...
2011-11-22 14:22:41 ClientCnxn [INFO] Opening socket connection to
server /127.0.0.1:2181
2011-11-22 14:22:41 ClientCnxn [INFO] Socket connection established
to localhost.localdomain/127.0.0.1:2181, initiating session
2011-11-22 14:22:41 ClientCnxn [INFO] Session establishment
complete on server localhost.localdomain/127.0.0.1:2181, sessionid =
0x133ccf10e960001, negotiated timeout = 20000
2011-11-22 14:22:41 zookeeper [INFO] Zookeeper state
update: :connected:none
2011-11-22 14:22:41 ClientCnxn [INFO] EventThread shut down
2011-11-22 14:22:41 nimbus [INFO] Starting Nimbus server...
4. xterm[3]: $ storm supervisor # started supervisor daemon
saw the following in the supervisor.log:
2011-11-22 14:24:20 ZooKeeper [INFO] Initiating client connection,
connectString=127.0.0.1:2181/storm sessionTimeout=20000
watcher=backtype.storm.zookeeper$mk_client$reify__1247@1cecd92c
2011-11-22 14:24:20 ClientCnxn [INFO] Opening socket connection to
server /127.0.0.1:2181
2011-11-22 14:24:20 ClientCnxn [INFO] Socket connection established
to localhost.localdomain/127.0.0.1:2181, initiating session
2011-11-22 14:24:20 ClientCnxn [INFO] Session establishment
complete on server localhost.localdomain/127.0.0.1:2181, sessionid =
0x133ccf10e960003, negotiated timeout = 20000
5. xterm[4]: $ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology
$ # submitted the storm exclamation topology (pure java
example)
saw the following output error:
Running: export STORM_JAR=storm-starter-0.0.1-SNAPSHOT-
standalone.jar; java -client -Djava.library.path=/home/archive/idas/
lib -cp /home/archive/idas/opt/storm-0.5.4/storm-0.5.4.jar:/home/
archive/idas/opt/storm-0.5.4/lib/clj-time-0.3.0.j
ar:/home/archive/idas/opt/storm-0.5.4/lib/clojure-1.2.0.jar:/home/
archive/idas/opt/storm-0.5.4/lib/clojure-contrib-1.2.0.jar:/home/
archive/idas/opt/storm-0.5.4/lib/clout-0.4.1.jar:/home/archive/idas/
opt/storm-0.5.4/lib/commons-codec-1.3.j
ar:/home/archive/idas/opt/storm-0.5.4/lib/commons-exec-1.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/commons-fileupload-1.2.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/commons-io-1.4.jar:/home/archive/
idas/opt/storm-0.5.4/lib/commons-lang-2.5.jar:/home/archive/idas/opt/
storm-0.5.4/lib/commons-logging-1.1.1.jar:/home/archive/idas/opt/
storm-0.5.4/lib/compojure-0.6.4.jar:/home/archive/idas/opt/storm-0.5.4/
lib/core.incubator-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
hiccup-0.3.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpclient-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
httpcore-4.0.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jetty-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/jetty-
util-6.1.26.jar:/home/archive/idas/opt/storm-0.5.4/lib/
jline-0.9.94.jar:/home/archive/idas/opt/storm-0.5.4/lib/joda-
time-1.6.jar:/home/archive/idas/opt/storm-0.5.4/lib/json-
simple-1.1.jar:/home/archive/idas/opt/storm-0.5.4/lib/junit-3.8.1.jar:/
home/archive/idas/opt/storm-0.5.4/lib/jvyaml-1.0.0.jar:/home/archive/
idas/opt/storm-0.5.4/lib/jzmq-2.1.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/libthrift7-0.7.0.jar:/home/archive/idas/opt/
storm-0.5.4/lib/log4j-1.2.16.jar:/home/archive/idas/opt/storm-0.5.4/
lib/ring-core-0.3.10.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
jetty-adapter-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/ring-
servlet-0.3.11.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5-20081211.jar:/home/archive/idas/opt/storm-0.5.4/lib/servlet-
api-2.5.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
api-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/slf4j-
log4j12-1.5.8.jar:/home/archive/idas/opt/storm-0.5.4/lib/
tools.macro-0.1.0.jar:/home/archive/idas/opt/storm-0.5.4/lib/
zookeeper-3.3.2.jar:storm-starter-0.0.1-SNAPSHOT-standalone.jar:/
gscuser/idas/.storm:/home/archive/idas/opt/storm-0.5.4/bin
storm.starter.ExclamationTopology
0 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn -
Thread Thread[main,5,main] died
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:
119)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.apache.zookeeper.server.NIOServerCnxn
$Factory.<init>(NIOServerCnxn.java:144)
at org.apache.zookeeper.server.NIOServerCnxn
$Factory.<init>(NIOServerCnxn.java:125)
at backtype.storm.zookeeper
$mk_inprocess_zookeeper.invoke(zookeeper.clj:101)
at backtype.storm.testing
$mk_local_storm_cluster.doInvoke(testing.clj:101)
at clojure.lang.RestFn.invoke(RestFn.java:398)
at backtype.storm.LocalCluster
$_init.invoke(LocalCluster.clj:10)
at backtype.storm.LocalCluster.<init>(Unknown Source)
at
storm.starter.ExclamationTopology.main(ExclamationTopology.java:67)
6. xterm[1]: $ zkServer.sh stop # stopped the zookeeper daemon
7. xterm[4]: $ storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jar
storm.starter.ExclamationTopology
$ # 2nd time submitted the storm exclamation topology
(pure java example)
saw the following successful abridged output:
0 [main-EventThread] INFO backtype.storm.zookeeper -
Zookeeper state update: :connected:none
69 [main-EventThread] INFO backtype.storm.zookeeper -
Zookeeper state update: :connected:none
78 [main-EventThread] INFO backtype.storm.zookeeper -
Zookeeper state update: :connected:none
97 [main-EventThread] INFO backtype.storm.zookeeper -
Zookeeper state update: :connected:none
160 [main-EventThread] INFO backtype.storm.zookeeper -
Zookeeper state update: :connected:none
227 [main] INFO backtype.storm.daemon.nimbus - Received
topology submission for test with conf {"topology.serializations" {},
"storm.id" "test-1-1321994107", "topology.debug" true}
364 [main] INFO backtype.storm.daemon.nimbus - Task
test-1-1321994107:1 timed out
365 [main] INFO backtype.storm.daemon.nimbus - Task
test-1-1321994107:2 timed out
...
470 [main] INFO backtype.storm.daemon.nimbus - Available slots:
(["03586326-dcf8-45ed-81d6-a6b165c95424" 1] ["03586326-dcf8-45ed-81d6-
a6b165c95424" 2] ["03586326-dcf8-45ed-81d6-a6b165c95424" 3] ["83758bf1-
fde8-49e4-b739-1a80396a9901" 4] ["83758bf1-fde8-49e4-
b739-1a80396a9901" 5] ["83758bf1-fde8-49e4-b739-1a80396a9901" 6])
511 [main] INFO backtype.storm.daemon.nimbus - Setting new
assignment for storm id test-1-1321994107:
#:backtype.storm.daemon.common.Assignment{:master-code-dir "/tmp/
ec3780bc-5ee9-49d7-a3bd-8cd86e1e56a2/nimbus/stormdist/
test-1-1321994107", :node->host {"03586326-dcf8-45ed-81d6-
a6b165c95424" "indraniel.wustl.edu"}, :task->node+port {1 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 2 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 3 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 4
["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 5 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 6 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 7 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 8
["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 9 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 10 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 11 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 12
["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 13 ["03586326-
dcf8-45ed-81d6-a6b165c95424" 1], 14 ["03586326-dcf8-45ed-81d6-
a6b165c95424" 1], 15 ["03586326-dcf8-45ed-81d6-a6b165c95424" 1], 16
["03586326-dcf8-45ed-81d6-a6b165c95424" 1]}, :task->start-time-secs {1
...

read more »

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 2 | next ›
Discussion Overview
groupstorm-user @
postedJan 13, '13 at 5:32a
activeJan 13, '13 at 5:32a
posts2
users1
websitestorm-project.net
irc#storm-user

1 user in discussion

YogiP: 2 posts

People

Translate

site design / logo © 2022 Grokbase