FAQ
Hi all,

I think I've found a small bug with multiprocessing package on
Windows. If you try to start a multiprocessing.Process from a Python-
based Windows service, the child process will fail to run. When
running the parent process as a regular Python program, everything
works as expected.

I've tracked the problem down to how main_path is prepared in
multiprocessing.forking.get_preparation_data() (lines 370-377):

def get_preparation_data(name):
[...skipped a few lines...]
if not WINEXE:
main_path = getattr(sys.modules['__main__'], '__file__', None)
if not main_path and sys.argv[0] not in ('', '-c'):
main_path = sys.argv[0]
if main_path is not None:
if not os.path.isabs(main_path) and \
process.ORIGINAL_DIR is not
None:
main_path = os.path.join(process.ORIGINAL_DIR,
main_path)
d['main_path'] = os.path.normpath(main_path)
return d

When the program is running as a Windows service, but is not packaged
into a single executable, main_path will become the path to the
service executable (typically, pythonservice.exe). When this data
makes it to the child process, the prepare() function will treat
main_path as a path to a python module, and will try to import it.
This causes it to fail.

My quick-and-dirty solution was to check in get_preparation_data() if
main_path ends with '.exe', and if it does, to not pass it at all.
This solves the problem in my case, but perhaps there's a better way
to fix this? Here is my version of get_preparation_data():

def get_preparation_data(name):
'''
Return info about parent needed by child to unpickle process
object
'''
from .util import _logger, _log_to_stderr

d = dict(
name=name,
sys_path=sys.path,
sys_argv=sys.argv,
log_to_stderr=_log_to_stderr,
orig_dir=process.ORIGINAL_DIR,
authkey=process.current_process().authkey,
)

if _logger is not None:
d['log_level'] = _logger.getEffectiveLevel()

if not WINEXE:
main_path = getattr(sys.modules['__main__'], '__file__', None)
if not main_path and sys.argv[0] not in ('', '-c'):
main_path = sys.argv[0]
if main_path is not None:
if not os.path.isabs(main_path) and \
process.ORIGINAL_DIR is not
None:
main_path = os.path.join(process.ORIGINAL_DIR,
main_path)
if not main_path.endswith('.exe'):
d['main_path'] = os.path.normpath(main_path)

return d

Search Discussions

  • Mark Hammond at Feb 6, 2009 at 3:50 am

    On 6/02/2009 11:37 AM, Volodya wrote:
    Hi all,

    I think I've found a small bug with multiprocessing package on
    Windows.
    I'd actually argue its a bug in pythonservice.exe - it should set
    sys.argv[] to resemble a normal python process with argv[0] being the
    script. I'll fix it...

    Cheers,

    Mark
  • Mark Hammond at Feb 6, 2009 at 4:26 am

    On 6/02/2009 2:50 PM, Mark Hammond wrote:
    On 6/02/2009 11:37 AM, Volodya wrote:
    Hi all,

    I think I've found a small bug with multiprocessing package on
    Windows.
    I'd actually argue its a bug in pythonservice.exe - it should set
    sys.argv[] to resemble a normal python process with argv[0] being the
    script. I'll fix it...
    Actually it appears I spoke too soon:

    * A bug in pywin32 exists such that when you use 'debug' on a service,
    the argv reflected the full argv of the application, including the
    '-debug' portion of the command-line. However, even if that is fixed,
    the next argument is actually the name of the service (as declared in
    the .py file for the service), not the .py module itself. Thus, I could
    make argv a little more sane in this case, but still the initial problem
    would remain as argv[0] would still not be a .py file.

    * When the service is started by windows itself, there are usually zero
    additional command-line arguments. If there *are* arguments, they are
    likely to be the string the user entered via control panel as a special
    case (Windows doesn't actually remember service args - they are used
    once and discarded). It is important we continue to expose whatever
    argv we actually got from Windows to the service code.

    So unfortunately I don't think I can change pythonservice to resolve the
    issue you reported.

    Cheers,

    Mark
  • Volodymyr Orlenko at Feb 6, 2009 at 5:21 am

    On 05/02/2009 8:26 PM, Mark Hammond wrote:
    On 6/02/2009 2:50 PM, Mark Hammond wrote:
    On 6/02/2009 11:37 AM, Volodya wrote:
    Hi all,

    I think I've found a small bug with multiprocessing package on
    Windows.
    I'd actually argue its a bug in pythonservice.exe - it should set
    sys.argv[] to resemble a normal python process with argv[0] being the
    script. I'll fix it...
    Actually it appears I spoke too soon:

    * A bug in pywin32 exists such that when you use 'debug' on a service,
    the argv reflected the full argv of the application, including the
    '-debug' portion of the command-line. However, even if that is fixed,
    the next argument is actually the name of the service (as declared in
    the .py file for the service), not the .py module itself. Thus, I
    could make argv a little more sane in this case, but still the initial
    problem would remain as argv[0] would still not be a .py file.

    * When the service is started by windows itself, there are usually
    zero additional command-line arguments. If there *are* arguments,
    they are likely to be the string the user entered via control panel as
    a special case (Windows doesn't actually remember service args - they
    are used once and discarded). It is important we continue to expose
    whatever argv we actually got from Windows to the service code.

    So unfortunately I don't think I can change pythonservice to resolve
    the issue you reported.
    Thanks Mark, this makes perfect sense.

    In the patch I submitted, I simply check if the name of the supposed
    module ends with ".exe". It works fine for my case, but maybe this is
    too general. Is there a chance that a Python module would end in ".exe"?
    If so, maybe we should check specifically for "pythonservice.exe". But
    then, if someone renames the executable (as I did, because I wanted to
    see meaningful service names in the process list), the patch will not
    work. Maybe there's another way to fix the forking module?
  • James Mills at Feb 6, 2009 at 5:54 am

    On Fri, Feb 6, 2009 at 3:21 PM, Volodymyr Orlenko wrote:
    In the patch I submitted, I simply check if the name of the supposed module
    ends with ".exe". It works fine for my case, but maybe this is too general.
    Is there a chance that a Python module would end in ".exe"? If so, maybe we
    should check specifically for "pythonservice.exe". But then, if someone
    renames the executable (as I did, because I wanted to see meaningful service
    names in the process list), the patch will not work. Maybe there's another
    way to fix the forking module?
    I believe the best way to fix this is to fix the underlying
    issue that Mark has pointed out (monkey-patching mp won't do).

    --JamesMills
  • Volodymyr Orlenko at Feb 6, 2009 at 7:11 am

    On 05/02/2009 9:54 PM, James Mills wrote:
    On Fri, Feb 6, 2009 at 3:21 PM, Volodymyr Orlenko wrote:

    [...] Maybe there's another
    way to fix the forking module?
    I believe the best way to fix this is to fix the underlying
    issue that Mark has pointed out (monkey-patching mp won't do).
    But according to Mark's second message in this thread, there's no way to
    make pythonservice.exe set its sys.argv[0] to a python module -- it uses
    sys.argv for debug parameters in debug mode, and for custom service
    parameters in normal mode.
  • Mark Hammond at Feb 6, 2009 at 9:54 pm

    On 6/02/2009 4:21 PM, Volodymyr Orlenko wrote:
    In the patch I submitted, I simply check if the name of the supposed
    module ends with ".exe". It works fine for my case, but maybe this is
    too general. Is there a chance that a Python module would end in ".exe"?
    IIRC, py2exe may create executables where sys.argv[0] is the executable
    itself. Maybe if we consider and handle both these cases a patch to mp
    might be looked upon in a better light...

    Cheers,

    Mark

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedFeb 6, '09 at 12:37a
activeFeb 6, '09 at 9:54p
posts7
users3
websitepython.org

People

Translate

site design / logo © 2023 Grokbase