Multiprocessing, Py2exe, and Windows Services

I recently ran in to an issue with using the Python standard library module multiprocessing from within a Windows service that had been frozen with Py2exe. First I’ll give a brief overview of the components involved for those that may not be familiar with them.

A Windows service is a special type of executable that is started by the service control manager (SCM). Generally you can not just run a Windows service by double clicking on it. The service has a service main function and a control handler function that responds to events sent to the service by the SCM. You can read about services on MSDN here. Writing a Windows service in Python also requires the PyWin32 package. PyWin32 can be obtained here

Py2exe is an extension of distutils that turns a python module into an executable file that can run on a Windows system on which there is not an installed Python distribution. Py2exe can handle many types of executables, dlls, exe, windows services, COM objects, etc… Read all about Py2exe here.

Multiprocessing is a part of the standard Python library in Python 2.6 and later. It is an amazing library that allows you to run any callable Python object in a different process. Read about multiprocessing here.

Below is a simple Windows service named MyService in a Python module. NOTE This is not a complete example of a service. The SvcDoRun method must block to keep the service alive. You can accomplish this in many ways, like by waiting on an event. I leave this as an exercise for the reader. The key part of this code is the code after importing the multiprocessing module. You must provide an executable (generally a Python interpreter) that multiprocessing can use to run python scripts because your service executable will not work. In the code below we are indicating to multiprocessing that it should use myapp.exe as the executable file to run processes and that myapp.exe will be in the same directory as our service executable. We will provide another Python module that Py2exe will use to build myapp.exe

import os
import sys
import win32security
import win32service
import win32serviceutil

# Give the multiprocessing module a python interpreter to run
import multiprocessing
executable = os.path.join(os.path.dirname(sys.executable), 'myapp.exe')
multiprocessing.set_executable(executable)
del executable


class MyService(win32serviceutil.ServiceFramework):
   _svc_name_ = 'MyService'
   _svc_display_name_ = 'MyService'
   _svc_description_ = 'MyService Example Program'
#    _exe_name_ = 'pythonservice.exe'   # Defaults to PythonService.exe
#    _svc_deps_ = None                  # sequence of service names on which this depends
#    _exe_args_ = None                  # Defaults to no arguments

   def SvcDoRun(self):

       # Set the current directory to the directory from
       # which this executable was launched.
       currentDir = os.path.dirname(sys.executable)
       os.chdir(currentDir)

       # Tell Windows which privileges you need, others are removed.
       set_privileges((win32security.SE_ASSIGNPRIMARYTOKEN_NAME,
                       win32security.SE_CHANGE_NOTIFY_NAME,
                       win32security.SE_CREATE_GLOBAL_NAME,
                       win32security.SE_SHUTDOWN_NAME))

       #  Implement service here, and block
       #  When this method returns the service is stopped.


if __name__ == '__main__':
   # For a service, this never gets called.
   # 
   # freeze_support must be the first line
   # after the if __name__ == '__main__'
   multiprocessing.freeze_support()

   # Pass the command line to the service utility library.
   # This can handle start, stop, install, remove and other commands.
   win32serviceutil.HandleCommandLine(MyService)

Here is the myapp.py module used to generate our executable for the multiprocessing module. This is a very simple Python script that actually does more than it needs to even for this task.

import multiprocessing
import os

def main ():
   print('myapp: ', os.getpid())


if __name__ == '__main__':
   # freeze_support must be the first line
   # after the if __name__ == '__main__'
   multiprocessing.freeze_support()
   main()

Now all we need is a setup.py script that tells Py2exe how to build our executables. Here is the code.

from distutils.core import setup
import py2exe

#  We must leave the optimization level at 1 if we use the kid template
#  library.  This leaves the doc strings in the library code.  The kid 
#  templating engine parses its doc string at run time, thus if the doc 
#  string is not in the pyo file, the program crashes.

# List of python modules to exclude from the distribution
excludes = [
   "Tkinter",
   "doctest",
   "unittest",
   "pydoc",
   "pdb"
]

# List of dll's (and apparently exe's) to exclude from the distribution
# if any Windows system dll appears in the dist folder, add it to this
# list.
dll_excludes = [
   "API-MS-Win-Core-LocalRegistry-L1-1-0.dll",
   "MPR.dll",
   "MSWSOCK.DLL",
   "POWRPROF.dll",
   "profapi.dll",
   "userenv.dll",
   "w9xpopen.exe",
   "wtsapi32.dll"
]

# List of python modules that are to be manually included.
mod_includes = []

package_includes = []

py2exe_options = {
   "optimize": 2, # 0 (None), 1 (-O), 2 (-OO)
   "excludes": excludes,
   "dll_excludes": dll_excludes,
   "packages": package_includes,
   "xref": False,
   # bundle_files: 1|2|3
   #    1: executable and library.zip
   #    2: executable, Python DLL, library.zip
   #    3: executable, Python DLL, other DLLs and PYDs, library.zip
   "bundle_files": 3
}

setup(service=[{'modules': 'myservice',
               'icon_resources': [(1, 'myapp.ico')],
             }],
     console=[{'script': 'myapp.py',
               'icon_resources': [(1, 'myapp.ico')]
             }],
     version='1.0',
     options={"py2exe": py2exe_options}
    )

I have skimmed this down a bit from an actual setup.py that I use at work. In practice I have found that Py2exe sometimes includes modules and libraries that are not necessary. The excludes list is a list of Python modules that you want to exclude from your distribution. Make sure that you know the modules listed here are not actually used. The dll_excludes option is a list of DLL and possibly EXE files that for whatever reason Py2exe is copying to your dist folder event though they may be Windows system DLLs (that you are most likely not allowed to redistribute), or the old w9xpopen.exe (for Windows 9x only). The mod_includes and package_includes options can be used to force inclusion of Python modules or packages that you must have in your distribution but for some reason Py2exe is not placing them in your dist folder. The work is done in the call to setup. Here we are telling it to build a service from the myservice module and a console application using the myapp.py module. Each of these output files uses the same icon as specified. We pass Py2Exe specific options to Py2Exe via the py2exe_options dictionary. Build the executables by running:

python –OO setup.py py2exe

Now that we have provided an executable for multiprocessing we can do this somewhere in our service:

from multiprocessing import Process

def handle_request(req):
   # do something useful
   pass

def on_request(request):
   Process(target=handle_request, args=(request,)).start()

The Point: In a frozen Windows service, you have to provide an executable to the multiprocessing module that can be used to run Python scripts in a new process.