to be worked on in order, so there is no traditional head and tail of
the Queue. The Queue table is simply a set of work orders that can be
fetched randomly or by applying a scan for a particular set of Rows
that can even come out from the middle of the table.
-Jack
On Sat, Jul 16, 2011 at 9:38 AM, Jack Levin wrote:
Hello, we are thinking about using Hbase table as a simple queue which
will dispatch the work for a mapreduce job, as well as real time
fetching of data to present to end user. In simple terms, suppose you
had a data source table and a queue table. The queue table has a
smaller set of Rows that point to Values which in turn point to
Perma-set table, which has large collection of Rows. (so Queue{Row,
Value} -> Perma-Set {Row, Value}). Or Q-Value -> P-Row. Our Goal is
to look up which Rows to retrieve from the Perma-Set table by looking
through the Queue. Once the lookup into the Queue is done, the Row
from the Queue must be deleted to avoid the same process of Perma-Set
lookup be done twice; We expect many concurrent lookups to happen, so
I assume the first thing we need to do is to have a client that does
the work is acquire a lock on the Queue Row, process the work, then
Remove the Queue Row.
Has anyone done something similar before? Any gotchas we should be away of?
Thanks.
-Jack
Hello, we are thinking about using Hbase table as a simple queue which
will dispatch the work for a mapreduce job, as well as real time
fetching of data to present to end user. In simple terms, suppose you
had a data source table and a queue table. The queue table has a
smaller set of Rows that point to Values which in turn point to
Perma-set table, which has large collection of Rows. (so Queue{Row,
Value} -> Perma-Set {Row, Value}). Or Q-Value -> P-Row. Our Goal is
to look up which Rows to retrieve from the Perma-Set table by looking
through the Queue. Once the lookup into the Queue is done, the Row
from the Queue must be deleted to avoid the same process of Perma-Set
lookup be done twice; We expect many concurrent lookups to happen, so
I assume the first thing we need to do is to have a client that does
the work is acquire a lock on the Queue Row, process the work, then
Remove the Queue Row.
Has anyone done something similar before? Any gotchas we should be away of?
Thanks.
-Jack
