FAQ
Hi,

I need to parse a binary file produced by an embedded system, whose
content consists in a set of events laid-out like this:

<event 1> <data 1> <event 2> <data 2> ... <event n> <data n>

Every "event" is a single byte in size, and it indicates how long is
the associated "data". Thus, to parse all events in the file, I need to
take it like a stream and read one event at a time, consuming bytes
according to the event value, and jumping to the next event, until an
EOF is reached.

Since there are dozens of almost completely heterogeneous events and
each one of them may imply different actions on the program parsing the
file, I thought it would be convenient to have one class encapsulating
the logic for every event. The parser would then sit in a loop,
creating objects of different classes and calling a method (say
"execute"). That method (different in every class) is responsible for
consuming the bytes associated with the event.

Hence, as the class the parser needs to instantiate in each iteration
is not known in advance, a factory should be implemented. Somehow the
factory should know how to map an event to a class. I don't know of the
best way I should do that in Python. I made an attempt along the
following lines:

1. Create a base class for the events;
2. For every descendant class declare (in the class body) a public
attribute "eventNum" and assign it the value of the event it will be
responsible for;
3. At runtime, the factory constructor scans the event class hierarchy
and builds a dictionary mapping "eventNum"'s to classes.

A draft of the implementation follows:

#################################

##### <events.py module> #####

class EvtBase:
def __init__(self, file):
self.file = file

def execute(self):
pass

class Evt1(EvtBase):
eventNum = 1
def execute(self):
...

class Evt2(EvtBase):
eventNum = 2
def execute(self):
...

...

class EvtN(EvtBase):
eventNum = N
def execute(self):
...


##### <factory.py module> #####

import inspect
import events

class Factory:
def __isValidEventClass(self, obj):
if inspect.isclass(obj) and obj != events.EvtBase and \
events.EvtBase in inspect.getmro(obj):
for m in inspect.getmembers(obj):
if m[0] == 'eventNum':
return True
return False

def __init__(self):
self.__eventDict = {}
for m in inspect.getmembers(events, self.__isValidEventClass):
cls = m[1]
self.__eventDict.update({cls.eventNum: cls})

def parseEvents(self, file):
while not file.eof():
ev = file.read(1)
self.__eventDict[ev](file).execute()

#################################

I'm using the inspect module to find the event classes. One drawback of
this approach is the need to keep the event classes in a module
different from that of the factory, because the getmembers method
expects an already parsed object or module. (The advantage is keeping
the event number near the class declaration.) I've already had to make
the solution generic and I found it was not straightforward to separate
the common logic while avoiding the need to keep the factory and the
events in two distinct modules.

Is there anything better I can do? I don't have enough experience with
Python, then I don't know whether it offers a more obvious way to
address my problem.

Thanks in advance.

--
Romulo A. Ceccon
'romulo%s\x40yahoo.com.br' % 'ceccon'

Search Discussions

  • George Sakkis at Dec 4, 2006 at 5:13 pm

    googlegroups at romulo.e4ward.com wrote:

    Hi,

    I need to parse a binary file produced by an embedded system, whose
    content consists in a set of events laid-out like this:

    <event 1> <data 1> <event 2> <data 2> ... <event n> <data n>

    Every "event" is a single byte in size, and it indicates how long is
    the associated "data". Thus, to parse all events in the file, I need to
    take it like a stream and read one event at a time, consuming bytes
    according to the event value, and jumping to the next event, until an
    EOF is reached.

    Since there are dozens of almost completely heterogeneous events and
    each one of them may imply different actions on the program parsing the
    file, I thought it would be convenient to have one class encapsulating
    the logic for every event. The parser would then sit in a loop,
    creating objects of different classes and calling a method (say
    "execute"). That method (different in every class) is responsible for
    consuming the bytes associated with the event.

    Hence, as the class the parser needs to instantiate in each iteration
    is not known in advance, a factory should be implemented. Somehow the
    factory should know how to map an event to a class. I don't know of the
    best way I should do that in Python. I made an attempt along the
    following lines:

    1. Create a base class for the events;
    2. For every descendant class declare (in the class body) a public
    attribute "eventNum" and assign it the value of the event it will be
    responsible for;
    3. At runtime, the factory constructor scans the event class hierarchy
    and builds a dictionary mapping "eventNum"'s to classes.

    A draft of the implementation follows:

    #################################

    ##### <events.py module> #####

    class EvtBase:
    def __init__(self, file):
    self.file = file

    def execute(self):
    pass

    class Evt1(EvtBase):
    eventNum = 1
    def execute(self):
    ...

    class Evt2(EvtBase):
    eventNum = 2
    def execute(self):
    ...

    ...

    class EvtN(EvtBase):
    eventNum = N
    def execute(self):
    ...


    ##### <factory.py module> #####

    import inspect
    import events

    class Factory:
    def __isValidEventClass(self, obj):
    if inspect.isclass(obj) and obj != events.EvtBase and \
    events.EvtBase in inspect.getmro(obj):
    for m in inspect.getmembers(obj):
    if m[0] == 'eventNum':
    return True
    return False

    def __init__(self):
    self.__eventDict = {}
    for m in inspect.getmembers(events, self.__isValidEventClass):
    cls = m[1]
    self.__eventDict.update({cls.eventNum: cls})

    def parseEvents(self, file):
    while not file.eof():
    ev = file.read(1)
    self.__eventDict[ev](file).execute()

    #################################

    I'm using the inspect module to find the event classes. One drawback of
    this approach is the need to keep the event classes in a module
    different from that of the factory, because the getmembers method
    expects an already parsed object or module. (The advantage is keeping
    the event number near the class declaration.) I've already had to make
    the solution generic and I found it was not straightforward to separate
    the common logic while avoiding the need to keep the factory and the
    events in two distinct modules.

    Is there anything better I can do? I don't have enough experience with
    Python, then I don't know whether it offers a more obvious way to
    address my problem.

    Thanks in advance.
    If you actually intend to
    1) name your Event subclasses Evt1, Evt2, ... EvtN and not give more
    descriptive (but unrelated to the magic event number) names, and
    2) put them all in one module (events.py),
    you can avoid the code duplication of putting the event number both in
    the class name and as a class attribute. Your dispatcher could then be
    as simple as:

    import events

    def parseEvents(file):
    while not file.eof():
    ev = int(file.read(1))
    cls = getattr(events, 'Evt%d' % ev)
    cls(file).execute()

    By the way, it is not clear from your description if the event number
    equals to the size of the associated data. If it is, you can factor out
    the data extraction part in the factory function and pass just the
    extracted data in the Event constructor instead of the file:

    def parseEvents(file):
    while not file.eof():
    ev = int(file.read(1))
    cls = getattr(events, 'Evt%d' % ev)
    cls(file.read(ev)).execute()


    George
  • Romulo A. Ceccon at Dec 4, 2006 at 5:34 pm

    George Sakkis wrote:

    If you actually intend to
    1) name your Event subclasses Evt1, Evt2, ... EvtN and not give more
    descriptive (but unrelated to the magic event number) names
    No, those names are just an example. The actual classes have
    descriptive names.
    By the way, it is not clear from your description if the event number
    equals to the size of the associated data.
    I'm sorry, George. The event number has nothing to do with the size of
    the associated data. I meant the program has a way to discover the size
    from the event number.

    --
    Romulo A. Ceccon
    'romulo%s\x40yahoo.com.br' % 'ceccon'
  • George Sakkis at Dec 4, 2006 at 6:11 pm

    Romulo A. Ceccon wrote:
    George Sakkis wrote:
    If you actually intend to
    1) name your Event subclasses Evt1, Evt2, ... EvtN and not give more
    descriptive (but unrelated to the magic event number) names
    No, those names are just an example. The actual classes have
    descriptive names.
    Even then, I'd prefer a naming convention plus a global Event registry
    than relying on inspect, both for implementation and (mostly)
    documentation reasons. It's good if a human can browse through a list
    of a few dozen names and immediately know that
    CamelCasedNameEndingWithEvent is an Event subclass. It's also good to
    be able to find in one place the explicit mapping of magic numbers to
    classes rather than searching in the whole file (or worse, multiple
    files) for it. YMMV.

    George
  • Chris Mellon at Dec 4, 2006 at 5:27 pm

    On 4 Dec 2006 08:39:17 -0800, googlegroups at romulo.e4ward.com wrote:
    Hi,

    I need to parse a binary file produced by an embedded system, whose
    content consists in a set of events laid-out like this:

    <event 1> <data 1> <event 2> <data 2> ... <event n> <data n>

    Every "event" is a single byte in size, and it indicates how long is
    the associated "data". Thus, to parse all events in the file, I need to
    take it like a stream and read one event at a time, consuming bytes
    according to the event value, and jumping to the next event, until an
    EOF is reached.

    Since there are dozens of almost completely heterogeneous events and
    each one of them may imply different actions on the program parsing the
    file, I thought it would be convenient to have one class encapsulating
    the logic for every event. The parser would then sit in a loop,
    creating objects of different classes and calling a method (say
    "execute"). That method (different in every class) is responsible for
    consuming the bytes associated with the event.

    Hence, as the class the parser needs to instantiate in each iteration
    is not known in advance, a factory should be implemented. Somehow the
    factory should know how to map an event to a class. I don't know of the
    best way I should do that in Python. I made an attempt along the
    following lines:

    1. Create a base class for the events;
    2. For every descendant class declare (in the class body) a public
    attribute "eventNum" and assign it the value of the event it will be
    responsible for;
    3. At runtime, the factory constructor scans the event class hierarchy
    and builds a dictionary mapping "eventNum"'s to classes.

    A draft of the implementation follows:

    #################################

    ##### <events.py module> #####

    class EvtBase:
    def __init__(self, file):
    self.file = file

    def execute(self):
    pass

    class Evt1(EvtBase):
    eventNum = 1
    def execute(self):
    ...

    class Evt2(EvtBase):
    eventNum = 2
    def execute(self):
    ...

    ...

    class EvtN(EvtBase):
    eventNum = N
    def execute(self):
    ...


    ##### <factory.py module> #####

    import inspect
    import events

    class Factory:
    def __isValidEventClass(self, obj):
    if inspect.isclass(obj) and obj != events.EvtBase and \
    events.EvtBase in inspect.getmro(obj):
    for m in inspect.getmembers(obj):
    if m[0] == 'eventNum':
    return True
    return False

    def __init__(self):
    self.__eventDict = {}
    for m in inspect.getmembers(events, self.__isValidEventClass):
    cls = m[1]
    self.__eventDict.update({cls.eventNum: cls})

    def parseEvents(self, file):
    while not file.eof():
    ev = file.read(1)
    self.__eventDict[ev](file).execute()

    #################################

    I'm using the inspect module to find the event classes. One drawback of
    this approach is the need to keep the event classes in a module
    different from that of the factory, because the getmembers method
    expects an already parsed object or module. (The advantage is keeping
    the event number near the class declaration.) I've already had to make
    the solution generic and I found it was not straightforward to separate
    the common logic while avoiding the need to keep the factory and the
    events in two distinct modules.

    Is there anything better I can do? I don't have enough experience with
    Python, then I don't know whether it offers a more obvious way to
    address my problem.
    I'd have the classes register themselves rather than trying to find
    them. This removes the need to have a common base class (preserves
    duck typing) and lets everything communicate via the factory module.

    #in module Factory.py

    EventMap = {}

    #in module events.py

    import Factory
    class EventHandler:
    Factory.EventMap[1] = EventHandler

    #in module parser.py

    import Factory

    handler = Factory.EventMap[event]()
    handler.handleEvent(data)


    There's probably some way to wrap the registration up in a metaclass
    so it's handled implicitly, but I prefer the explicit approach.

    Thanks in advance.

    --
    Romulo A. Ceccon
    'romulo%s\x40yahoo.com.br' % 'ceccon'

    --
    http://mail.python.org/mailman/listinfo/python-list
  • BearophileHUGS at Dec 4, 2006 at 8:15 pm

    Dennis Lee Bieber:
    Presuming the <event x> is a type code I'd just set up a list of functions:
    Then create a dictionary of them, keyed by the <event x> code
    processors = { "1" : process_1,
    "2" : process_2,
    ....
    "x" : process_x }
    Just a dict of functions was my solution too, I think avoiding more
    complex solutions is positive.

    Bye,
    bearophile
  • Terry Reedy at Dec 4, 2006 at 11:13 pm
    <bearophileHUGS at lycos.com> wrote in message
    news:1165263327.407166.276160 at l12g2000cwl.googlegroups.com...
    Dennis Lee Bieber:
    Presuming the <event x> is a type code I'd just set up a list of
    functions:
    Then create a dictionary of them, keyed by the <event x> code
    processors = { "1" : process_1,
    "2" : process_2,
    ....
    "x" : process_x }
    Just a dict of functions was my solution too, I think avoiding more
    complex solutions is positive.
    If the event codes start at 0 and run sequentially, a tuple or list would
    be even easier.
  • Paul McGuire at Dec 4, 2006 at 11:30 pm
    <bearophileHUGS at lycos.com> wrote in message
    news:1165263327.407166.276160 at l12g2000cwl.googlegroups.com...
    Dennis Lee Bieber:
    Presuming the <event x> is a type code I'd just set up a list of
    functions:
    Then create a dictionary of them, keyed by the <event x> code
    processors = { "1" : process_1,
    "2" : process_2,
    ....
    "x" : process_x }
    Just a dict of functions was my solution too, I think avoiding more
    complex solutions is positive.

    Bye,
    bearophile
    I think I'd go one step up the OO ladder and match each event code to a
    class. Have every class implement a staticmethod something like
    "load(stream)" (using pickle terminology), and then use a dict to dispatch.

    eventTypes = { "1" : BlahEvent, "2" : BlehEvent, "3" : BelchEvent, "4" :
    BlechEvent }

    eventObj = eventTypes[ stream.read(1) ].load( stream )

    Now transcending from plain-old-OO to Pythonic idiom, make this into a
    generator:

    eventTypes = { "1" : BlahEvent, "2" : BlehEvent, "3" : BelchEvent, "4" :
    BlechEvent }
    def eventsFromStream(stream):
    while not stream.EOF:
    evtTyp = stream.read(1)
    yield eventTypes[evtTyp].load(stream)

    and then get them all in a list using

    list( eventsFromStream( stream ) )

    -- Paul
  • Gabriel Genellina at Dec 4, 2006 at 11:39 pm

    At Monday 4/12/2006 13:39, googlegroups at romulo.e4ward.com wrote:
    class Factory:
    def __isValidEventClass(self, obj):
    if inspect.isclass(obj) and obj != events.EvtBase and \
    events.EvtBase in inspect.getmro(obj):
    for m in inspect.getmembers(obj):
    if m[0] == 'eventNum':
    return True
    return False

    def __init__(self):
    self.__eventDict = {}
    for m in inspect.getmembers(events, self.__isValidEventClass):
    cls = m[1]
    self.__eventDict.update({cls.eventNum: cls})

    def parseEvents(self, file):
    while not file.eof():
    ev = file.read(1)
    self.__eventDict[ev](file).execute()
    You already got other ways to go.
    But if you want to use several classes (maybe span along several
    modules), you can code the checking a lot more easily:

    if issubclass(cls, EvtBase) and hasattr(cls, 'eventNum'): ...

    I'd register each class itself, so no need to iterate over the
    members, but anyway you could use vars(module).
    In short, inspect may be good for debugging or documenting tools, but
    hardly needed for usual code.

    BTW, that file format is horrible. Users have to know *every* posible
    event (at least its size), even if the're not interested in them. And
    if you get out of sync, you can't recover. Add a new event, and all
    programs using the file don't work anymore. Ugh!


    --
    Gabriel Genellina
    Softlab SRL

    __________________________________________________
    Correo Yahoo!
    Espacio para todos tus mensajes, antivirus y antispam ?gratis!
    ?Abr? tu cuenta ya! - http://correo.yahoo.com.ar

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedDec 4, '06 at 4:39p
activeDec 4, '06 at 11:39p
posts9
users7
websitepython.org

People

Translate

site design / logo © 2023 Grokbase