Multiprocessing, Py2exe, and Windows Services

I recently ran in to an issue with using the Python standard library module multiprocessing from within a Windows service that had been frozen with Py2exe. First I’ll give a brief overview of the components involved for those that may not be familiar with them.

A Windows service is a special type of executable that is started by the service control manager (SCM). Generally you can not just run a Windows service by double clicking on it. The service has a service main function and a control handler function that responds to events sent to the service by the SCM. You can read about services on MSDN here. Writing a Windows service in Python also requires the PyWin32 package. PyWin32 can be obtained here

Py2exe is an extension of distutils that turns a python module into an executable file that can run on a Windows system on which there is not an installed Python distribution. Py2exe can handle many types of executables, dlls, exe, windows services, COM objects, etc… Read all about Py2exe here.

Multiprocessing is a part of the standard Python library in Python 2.6 and later. It is an amazing library that allows you to run any callable Python object in a different process. Read about multiprocessing here.

Below is a simple Windows service named MyService in a Python module. NOTE This is not a complete example of a service. The SvcDoRun method must block to keep the service alive. You can accomplish this in many ways, like by waiting on an event. I leave this as an exercise for the reader. The key part of this code is the code after importing the multiprocessing module. You must provide an executable (generally a Python interpreter) that multiprocessing can use to run python scripts because your service executable will not work. In the code below we are indicating to multiprocessing that it should use myapp.exe as the executable file to run processes and that myapp.exe will be in the same directory as our service executable. We will provide another Python module that Py2exe will use to build myapp.exe

import os
import sys
import win32security
import win32service
import win32serviceutil

# Give the multiprocessing module a python interpreter to run
import multiprocessing
executable = os.path.join(os.path.dirname(sys.executable), 'myapp.exe')
multiprocessing.set_executable(executable)
del executable


class MyService(win32serviceutil.ServiceFramework):
   _svc_name_ = 'MyService'
   _svc_display_name_ = 'MyService'
   _svc_description_ = 'MyService Example Program'
#    _exe_name_ = 'pythonservice.exe'   # Defaults to PythonService.exe
#    _svc_deps_ = None                  # sequence of service names on which this depends
#    _exe_args_ = None                  # Defaults to no arguments

   def SvcDoRun(self):

       # Set the current directory to the directory from
       # which this executable was launched.
       currentDir = os.path.dirname(sys.executable)
       os.chdir(currentDir)

       # Tell Windows which privileges you need, others are removed.
       set_privileges((win32security.SE_ASSIGNPRIMARYTOKEN_NAME,
                       win32security.SE_CHANGE_NOTIFY_NAME,
                       win32security.SE_CREATE_GLOBAL_NAME,
                       win32security.SE_SHUTDOWN_NAME))

       #  Implement service here, and block
       #  When this method returns the service is stopped.


if __name__ == '__main__':
   # For a service, this never gets called.
   # 
   # freeze_support must be the first line
   # after the if __name__ == '__main__'
   multiprocessing.freeze_support()

   # Pass the command line to the service utility library.
   # This can handle start, stop, install, remove and other commands.
   win32serviceutil.HandleCommandLine(MyService)

Here is the myapp.py module used to generate our executable for the multiprocessing module. This is a very simple Python script that actually does more than it needs to even for this task.

import multiprocessing
import os

def main ():
   print('myapp: ', os.getpid())


if __name__ == '__main__':
   # freeze_support must be the first line
   # after the if __name__ == '__main__'
   multiprocessing.freeze_support()
   main()

Now all we need is a setup.py script that tells Py2exe how to build our executables. Here is the code.

from distutils.core import setup
import py2exe

#  We must leave the optimization level at 1 if we use the kid template
#  library.  This leaves the doc strings in the library code.  The kid 
#  templating engine parses its doc string at run time, thus if the doc 
#  string is not in the pyo file, the program crashes.

# List of python modules to exclude from the distribution
excludes = [
   "Tkinter",
   "doctest",
   "unittest",
   "pydoc",
   "pdb"
]

# List of dll's (and apparently exe's) to exclude from the distribution
# if any Windows system dll appears in the dist folder, add it to this
# list.
dll_excludes = [
   "API-MS-Win-Core-LocalRegistry-L1-1-0.dll",
   "MPR.dll",
   "MSWSOCK.DLL",
   "POWRPROF.dll",
   "profapi.dll",
   "userenv.dll",
   "w9xpopen.exe",
   "wtsapi32.dll"
]

# List of python modules that are to be manually included.
mod_includes = []

package_includes = []

py2exe_options = {
   "optimize": 2, # 0 (None), 1 (-O), 2 (-OO)
   "excludes": excludes,
   "dll_excludes": dll_excludes,
   "packages": package_includes,
   "xref": False,
   # bundle_files: 1|2|3
   #    1: executable and library.zip
   #    2: executable, Python DLL, library.zip
   #    3: executable, Python DLL, other DLLs and PYDs, library.zip
   "bundle_files": 3
}

setup(service=[{'modules': 'myservice',
               'icon_resources': [(1, 'myapp.ico')],
             }],
     console=[{'script': 'myapp.py',
               'icon_resources': [(1, 'myapp.ico')]
             }],
     version='1.0',
     options={"py2exe": py2exe_options}
    )

I have skimmed this down a bit from an actual setup.py that I use at work. In practice I have found that Py2exe sometimes includes modules and libraries that are not necessary. The excludes list is a list of Python modules that you want to exclude from your distribution. Make sure that you know the modules listed here are not actually used. The dll_excludes option is a list of DLL and possibly EXE files that for whatever reason Py2exe is copying to your dist folder event though they may be Windows system DLLs (that you are most likely not allowed to redistribute), or the old w9xpopen.exe (for Windows 9x only). The mod_includes and package_includes options can be used to force inclusion of Python modules or packages that you must have in your distribution but for some reason Py2exe is not placing them in your dist folder. The work is done in the call to setup. Here we are telling it to build a service from the myservice module and a console application using the myapp.py module. Each of these output files uses the same icon as specified. We pass Py2Exe specific options to Py2Exe via the py2exe_options dictionary. Build the executables by running:

python –OO setup.py py2exe

Now that we have provided an executable for multiprocessing we can do this somewhere in our service:

from multiprocessing import Process

def handle_request(req):
   # do something useful
   pass

def on_request(request):
   Process(target=handle_request, args=(request,)).start()

The Point: In a frozen Windows service, you have to provide an executable to the multiprocessing module that can be used to run Python scripts in a new process.

Using WMI to Query Monitor Information

I recently had a need to programmatically determine the number, make, and model of each monitor attached to a Windows computer system. Knowing that WMI provides a bewildering array of such functionality, I started here with the WMI Win32_DesktopMonitor class.

Figure 1: Win32_DesktopMonitor WMI class description.

As you can see in figure 1, the information I needed is supplied by this class. I coded up a quick C# application that used this WMI class to report the monitors and thought that I was done. The problem is that each run of the program only ever showed one monitor on my system. I pulled up PythonWin and using Tim Golden's excellent WMI Python module I confirmed that WMI provides only one monitor. Searching the Internet told me that I was not the first to have this problem and in fact on Windows Vista and up (I was running on Windows 7) the Win32_DesktopMonitor class only returns one monitor in many cases. The issues has something to do with the video driver changes that were introduced in Windows Vista and improved in Windows 7. Most people that encountered this problem dropped into the Win32 C API to solve the issue. I still had hope for WMI, after all Windows knows how many monitors I have and what kind they are. Once again I used Tim Golden’s WMI Python module to poke around with various WMI classes and I found the Win32_PnPEntity class.

Figure 2: Win32_PnPEntity WMI class description.

I did a quick query to dump all of these out to see if I could use this WMI class.

import wmi
obj = wmi.WMI().Win32_PnPEntity(ConfigManagerErrorCode=0)

The ConfigManagerErrorCode=0 parameter means that I want all devices that are working properly. Looking at my obj variable, there was a ton of data but I noticed some devices with the string “DISPLAY\\…” within it. Here is some of the output:

I quickly filtered my output in Python with this:

displays = [x for x in obj if 'DISPLAY' in str(x)]
for item in displays:
   print item

And this is the result:

This was the result that I was looking for. I have two Dell E248WFP monitors and finally WMI agreed with me. Every property is exactly the same except for the DeviceID because this is what PnP uses to distinguish between the two monitors.

I quickly re-coded my C# function to use the WMI class Win32_PnPEntity. I did make a change however. The C# version filters devices where the Service = ‘monitor’ rather than looking for ‘DISPLAY’ in the DeviceID.

Special Thanks to Tim Golden who makes WMI enjoyable in Python!