Python 3.2 Released

On Sunday February 20th Python 3.2 was released. Normally I wouldn’t mention when a new point release of Python, but this one is special to me.  It is the first release of Python in which I have played a (very small) part. 

A while ago, I was playing around with Python’s multiprocessing module which, at the time, was new to me.  All of the simple examples that I found would use the os.getpid() and os.getppid() functions to show that the spawned process was indeed a child of the initial Python process.  The problem is that on the Windows platform, os.getppid() was not implemented.  I downloaded a zip of the source code and within a hour I had a working implementation of os.getppid() and could run the samples, great.  Time goes by, and I start to think about contributing my implementation to the actual Python source code.  I had no idea how to go about and do such a thing.  Fortunately, the Python developer guide has wonderful documentation on how you can contribute.  So, I created an issue for os.getppid() on Windows, stumbled through the process of creating a patch, submitted the patch, and then waited.  A core Python developer picked up the issue and told me that I would have to provide a unit test before any check-in would happen.  This meant I would have to spelunk through the source tree to find the unit tests and implement a meaningful test for this.  Well, you all know how life happens, time goes by, and I actually forgot that I had even created this issue.  Eventually I received an email from a core Python developer that moved the issues from the 2.x branch of Python to the 3.x branch.  I got back involved, implemented the unit test and the documentation changes, submitted all the patches and before I knew it, the code was checked in and the issue was closed.

About this time Tim Golden posted a call out to Windows developers to help contribute to Python.  I looked around for other issues to work on.  I ended up being involved on two other issues for Python 3.2 (on Windows).  First, I implemented os.getlogin().  It was just another os module function not available on Windows.  Then, I fixed a bug in the generation of .pyc files.  There was a call to _mkdir to create a directory.  The signature of this function on Windows has one less parameter than on *Nix systems.  Thus, every time _mkdir was called it would push an extra parameter on the stack that was not used.  Being that it is using the C calling convention, the caller cleans the stack, so the extra parameter was actually cleaned up off the stack.  Thus, this was a very minor issue that most likely wouldn’t have caused any problems, but I fixed it anyway.

I would like to write a more in-depth entry about my experience contributing to Python.  I would also like to get involved again in more issues.  Most likely one at a time because:  It can be time consuming, no one is going to pay you, and you must work on the issues on your own time.  It is also great fun and a learning experience.  I think every professional developer should be involved in some open source project.  There are many open source projects, even in the Windows and .NET worlds too (NUnit, NHibernate, IronPython, IronRuby, checkout codeplex).  Go find an open source project and grab an issue from the issue tracker and go for it!      

Using Encrypted Data Between Python and Silverlight

I had a chance to work on a project in which data was encrypted and shared between a Python program on the server side and a Silverlight .NET Framework application on the client side.  Both programming environments offer a rich set of libraries for doing data encryption.  On the Python side I chose to use the excellent PyCrypto library.  At the time that I wrote the code I was using version 2.1 of PyCrypto and version 2.7 of Python.  PyCrypto is in the public domain from the way I understand the license files.  On the .NET side, the System.Security.Cryptography namespace provides the classes needed.  In a Silverlight application the System.Security.Cryptography namespace is much, much smaller than in the full .NET Framework.  Thus AES would have to be the encryption algorithm used. 
AES is the Rijndael symmetric algorithm with a fixed block size (128 bits) and iteration count. For the Silverlight version of this algorithm the cipher mode and padding mode are CBC and PKCS7 respectively and can not be changed.  Padding is used to ensure that the data to be encrypted is a multiple of the block size. 
Here is an example of encrypting the data in Python. You can find the code in this article on GitHub.  Obviously, you will need to have Python and PyCrypto installed to use it.

 1 from Crypto.Cipher import AES
 2 from pkcs7 import PKCS7Encoder
 3 import base64
 4 
 5 key = 'your key 16bytes'
 6 # 16 byte initialization vector
 7 iv = '1234567812345678'
 8 
 9 aes = AES.new(key, AES.MODE_CBC, iv)
10 encoder = PKCS7Encoder()
11 
12 text = 'This is my plain text'
13 
14 # pad the plain text according to PKCS7
15 pad_text = encoder.encode(text)
16 # encrypt the padding text
17 cipher = aes.encrypt(pad_text)
18 # base64 encode the cipher text for transport
19 enc_cipher = base64.b64encode(cipher)
20 
21 print enc_cipher

The data to be encrypted is first run through the padding encoder.  This will ensure that the data is a multiple of the block size (16 bytes in this case).  Note that even if the data is already the correct size it is still padded.  The data is always padded thus the padding must always need to be removed.  This alleviates the programmer from having to know if the data is padded or not. This program will print the following encoded string: ZeYXkFf8wPbvzdC91V4adwx4U56o2zMMOathdDYuBOE=





The PKCS7 padding code is built in to the Silverlight AesManaged class.  On the Python side, I had to write this my self.  PKCS7 is described in RFC 2315 and is actually very simple, here is the code.

 1 import binascii
 2 import StringIO
 3 
 4 class PKCS7Encoder(object):
 5     '''
 6     RFC 2315: PKCS#7 page 21
 7     Some content-encryption algorithms assume the
 8     input length is a multiple of k octets, where k > 1, and
 9     let the application define a method for handling inputs
10     whose lengths are not a multiple of k octets. For such
11     algorithms, the method shall be to pad the input at the
12     trailing end with k - (l mod k) octets all having value k -
13     (l mod k), where l is the length of the input. In other
14     words, the input is padded at the trailing end with one of
15     the following strings:
16 
17              01 -- if l mod k = k-1
18             02 02 -- if l mod k = k-2
19                         .
20                         .
21                         .
22           k k ... k k -- if l mod k = 0
23 
24     The padding can be removed unambiguously since all input is
25     padded and no padding string is a suffix of another. This
26     padding method is well-defined if and only if k < 256;
27     methods for larger k are an open issue for further study.
28     '''
29     def __init__(self, k=16):
30         self.k = k
31 
32     ## @param text The padded text for which the padding is to be removed.
33     # @exception ValueError Raised when the input padding is missing or corrupt.
34     def decode(self, text):
35         '''
36         Remove the PKCS#7 padding from a text string
37         '''
38         nl = len(text)
39         val = int(binascii.hexlify(text[-1]), 16)
40         if val > self.k:
41             raise ValueError('Input is not padded or padding is corrupt')
42 
43         l = nl - val
44         return text[:l]
45 
46     ## @param text The text to encode.
47     def encode(self, text):
48         '''
49         Pad an input string according to PKCS#7
50         '''
51         l = len(text)
52         output = StringIO.StringIO()
53         val = self.k - (l % self.k)
54         for _ in xrange(val):
55             output.write('%02x' % val)
56         return text + binascii.unhexlify(output.getvalue())

Remember that each time the AES key is used to encrypt/decrypt a block of data, the internal state of the key changes.  Thus each time you are preparing encrypted data to transmit to the server you must create a new AES key object from the original key and iv or that data will not be properly transformed.  The server will also have to create a new AES key with the original data as well. If you do not, you will have a hard time keeping the server and client AES keys in sync and data will not be transformed properly on either side.





Decryption on the Silverlight side is very straight forward.  I am presenting this here as a console based program but there should not be any relevant difference when used in a Silverlight program.

 1 using System;
 2 using System.Text;
 3 using System.Security.Cryptography;
 4 
 5 namespace AesTest
 6 {
 7     class Program
 8     {
 9         static void Main(string[] args)
10         {
11             // This was the output of our Python program.
12             string enc_cipher = "ZeYXkFf8wPbvzdC91V4adwx4U56o2zMMOathdDYuBOE=";
13 
14             var textEncoder = new UTF8Encoding();
15 
16             // defaults to CBC and PKCS7
17             var aes = new AesManaged();
18             aes.Key = textEncoder.GetBytes("your key 16bytes");
19             aes.IV = textEncoder.GetBytes("1234567812345678");
20 
21             var decryptor = aes.CreateDecryptor();
22             var cipher = Convert.FromBase64String(enc_cipher);
23             var text_bytes = decryptor.TransformFinalBlock(cipher, 0, cipher.Length);
24 
25             var text = textEncoder.GetString(text_bytes);
26             // Should print 'This is my plain text'
27             Console.WriteLine(text);
28         }
29     }
30 }

It should be easy for any programmer to reverse the process so the the Silverlight client encrypts data that is then decrypted on the python side.  The hard part here is the age old problem of how you share the key to the symmetric encryption algorithm.  For this we used RSA public key cryptography.





PyCrypto once again comes to the rescue as it includes RSA encryption classes, for the most part.  Once again I had to write a padding encoder/decoder to be used along with the RSA classes. RSA can use a variety of padding schemes like PKCS#1 v1.5, and OAEP (optimal asymmetric encryption padding), with OAEP being recommended for new applications.  I wrapped the Crypto.PublicKey.RSA class in my own RSAkey class shown below.  As you can see in the code the data is padded, encrypted, and then base64 encoded.  The data can easily be transmitted via web protocols when it is base64 encoded.

 1 from Crypto.PublicKey import RSA
 2 from pkcs1 import OAEPEncoder
 3 import base64
 4 import binascii
 5 import os
 6 import pickle
 7 
 8 class RSAKey(object):
 9 
10     def __init__(self, keybitsize, encoder=OAEPEncoder()):
11         self._encoder = encoder
12         self._keysize = keybitsize
13         self._key = RSA.generate(keybitsize, os.urandom)
14 
15     @property
16     def key(self):
17         return self._key
18 
19     @property
20     def key_size(self):
21         return self._keysize
22 
23     ## Get the public RSA key used to encrypt data as
24     #  an XML string.
25     #  @param xml_format True if the key should be returned as XML.
26     #  If False, the key is returned as a base64 encoded pickled 
27     #  Python object.
28     #  @return: An XML string representation of the 
29     #           public RSA key. Each node is a base64
30     #           encoded string. It has the following 
31     #           structure.
32     #           \<RSAKeyValue\>
33     #               \<Exponent\>AQAB\</Exponent\>
34     #               \<Modulus\>some data\</Modulus\>
35     #           \</RSAKeyValue\> 
36     def public_key(self, xml_format):
37         pkey = self._key.publickey()
38 
39         if xml_format:
40             # Pads with leading zeros if needed.
41             def ensure_length(hexstr):
42                 if len(hexstr) % 2 != 0:
43                     return '0' + hexstr
44                 else:
45                     return hexstr
46             # make an encoded child node
47             def add_child(tag, n):
48                 str_n = ensure_length('%x' % n)
49                 n_bytes = binascii.unhexlify(str_n)
50 
51                 sub = et.SubElement(root, tag)
52                 sub.text = base64.b64encode(n_bytes)
53 
54             root = et.Element('RSAKeyValue')
55             add_child('Exponent', pkey.e)
56             add_child('Modulus', pkey.n)
57             return tostring(root)
58         else:
59             return base64.b64encode(pickle.dumps(pkey))
60 
61     ## Encrypt data with the public RSA key.
62     #  @param data The data to be encrypted
63     #  @return A base64 encoded string that is the encrypted data.
64     def encrypt(self, data):
65         enc_data = self._encoder.encode(data, keybits=self._keysize)
66         cipher = self._key.encrypt(enc_data, '')
67         return base64.b64encode(cipher[0])
68 
69     ## Decrypt data with the private RSA key.
70     #  @param encoded_cipher A base64 encoded string of encrypted data.
71     #  @return The decrypted data as a string.
72     def decrypt(self, encoded_cipher):
73         cipher = base64.b64decode(encoded_cipher)
74         enc_data = self._key.decrypt(cipher)
75         data = self._encoder.decode(enc_data)
76         return data

Following are the classes I wrote that implement two of the padding schemes (OAEP, PKCS#1 v1.5)described in rfc 2437.  Coming from a C background, I find it hard to work with binary data in python. i know that there must be more efficient ways to handle binary data. The python standard library modules cStringIo, binascii and struct come in very handy.





You may notice that some of the variable names in the following code are not ideal (sorry Uncle Bob!).  They actually reflect the names used within the RFC to make it easier for someone to follow along with the RFC document.  Also, for those of you that do not know Python well, an '_' prefix to a field name in a class is the convention for specifying a private class field. When a Python programmer sees a field like this: self._hash_length, it is clear (to Python programmers) that the author intended this to be private.  This is just a convention that you must enforce upon yourself as the Python language will allow you to access the field via an instance of the class.

  1 import binascii
  2 import cStringIO
  3 import hashlib
  4 import os
  5 import struct
  6 
  7 class PKCS1Error(RuntimeError):
  8     '''
  9     Base class for PKCS1 encoding/decoding errors.
 10     Error of this or derived classes should be caught
 11     by the calling code and then a generic error message
 12     should be returned to the caller.
 13     '''
 14     pass
 15 
 16 class DecoderError(PKCS1Error):
 17     '''
 18     Raised when a decoding error has been detected.
 19     '''
 20     pass
 21 
 22 class EncoderError(PKCS1Error):
 23     '''
 24     Raise when an encoding error has been detected.
 25     '''
 26     pass
 27 
 28 
 29 class PKCSAuxiliary(object):
 30     '''
 31     Auxiliary functions used in RFC 2437
 32     '''
 33 
 34     def __init__(self):
 35         self._hash_length = None
 36 
 37     @property
 38     def hash_length(self):
 39         if not self._hash_length:
 40             hasher = self.create_hasher()
 41             self._hash_length = hasher.digest_size
 42 
 43         return self._hash_length
 44 
 45     @staticmethod
 46     def create_hasher():
 47         return hashlib.sha1()
 48 
 49     @staticmethod
 50     def compute_hash(data, hex_digest=False):
 51         hasher = PKCSAuxiliary.create_hasher()
 52         hasher.update(data)
 53         if hex_digest:
 54             return hasher.hex_digest()
 55         else:
 56             return hasher.digest()
 57 
 58     def mgf(self, seed, length):
 59         '''
 60         RFC 2437 page 28 MFG1
 61         '''
 62         counter = 0
 63         output = cStringIO.StringIO()
 64         try:
 65             limit = length / self.hash_length
 66             while counter <= limit:
 67                 C = self.i2osp(counter)
 68                 output.write(self.compute_hash(seed + C))
 69                 counter += 1
 70 
 71             raw_mask = output.getvalue()
 72             if len(raw_mask) < length:
 73                 raise PKCS1Error("MGF: mask too long")
 74         finally:
 75             output.close()
 76 
 77         mask = raw_mask[:length]
 78         return mask
 79 
 80     def i2osp(self, x):
 81         '''
 82         RFC 2437 page 6 I2OSP
 83         Special case where length = 4
 84         '''
 85         if x > 256 ** 4:
 86             raise PKCS1Error("I2OSP: integer too large")
 87 
 88         sp = (
 89             int((x >> 24) & 0xff),
 90             int((x >> 16) & 0xff),
 91             int((x >> 8) & 0xff),
 92             int((x >> 0) & 0xff)
 93         )
 94 
 95         return struct.pack('BBBB', *sp)
 96 
 97     @staticmethod
 98     def xor(a, b):
 99         '''
100         RFC 2437  bitwise exclusive-or of two octet strings.
101         page 23
102         '''
103         if len(a) != len(b):
104             raise PKCS1Error("XOR: invalid input lengths")
105 
106         output = cStringIO.StringIO()
107 
108         try:
109             for i in xrange(len(a)):
110                 x = int(binascii.hexlify(a[i]), 16)
111                 y = int(binascii.hexlify(b[i]), 16)
112                 output.write('%02x' % (x ^ y))
113 
114             data = output.getvalue()
115 
116         finally:
117             output.close()
118 
119         return binascii.unhexlify(data)
120 
121 
122 class OAEPEncoder(PKCSAuxiliary):
123     '''
124     RFC 2437 9.1.1 EME-OAEP PKCS1-v2.0
125     9.1.1.1 EME-OAEP-ENCODE
126     9.1.1.2 EME-OAEP-DECODE
127     '''
128 
129     def __init__(self):
130         super(OAEPEncoder, self).__init__()
131 
132 
133     def encode(self, msg, salt='', keybits=1024):
134         k = keybits / 8
135         if len(msg) > (k - 2 - 2 * self.hash_length):
136             raise EncoderError("EME-OAEP: message too long")
137 
138         emLen = k - 1
139         if (emLen < (2 * self.hash_length + 1) or
140             len(msg) > (emLen - 1 - 2 * self.hash_length)):
141             raise EncoderError("EME-OAEP: message too long")
142 
143         pslen = emLen - len(msg) - 2 * self.hash_length - 1
144         output = cStringIO.StringIO()
145         try:
146             for _ in xrange(pslen):
147                 output.write('%02x' % 0)
148             ps = binascii.unhexlify(output.getvalue())
149             assert len(ps) == pslen, "PS: invalid length"
150         finally:
151             output.close()
152 
153         shash = self.compute_hash(salt)
154         dbout = cStringIO.StringIO()
155         try:
156             dbout.write(shash)
157             dbout.write(ps)
158             dbout.write('\x01')
159             dbout.write(msg)
160             db = dbout.getvalue()
161         finally:
162             dbout.close()
163 
164         seed = os.urandom(self.hash_length)
165         assert len(seed) == self.hash_length
166 
167         dbMask = self.mgf(seed, emLen - self.hash_length)
168         maskedDB = self.xor(db, dbMask)
169         seedMask = self.mgf(maskedDB, self.hash_length)
170         maskedSeed = self.xor(seed, seedMask)
171         emout = cStringIO.StringIO()
172         try:
173             emout.write(maskedSeed)
174             emout.write(maskedDB)
175             emsg = emout.getvalue()
176         finally:
177             emout.close()
178         return emsg
179 
180 
181     def decode(self, emsg, salt=''):
182         if len(emsg) < (2 * self.hash_length + 1):
183             raise DecoderError("EME-OAEP: decoding error")
184 
185         maskedSeed = emsg[:self.hash_length]
186         maskedDB = emsg[self.hash_length:]
187         seedMask = self.mgf(maskedDB, self.hash_length)
188         seed = self.xor(maskedSeed, seedMask)
189         dbMask = self.mgf(seed, len(emsg) - self.hash_length)
190         db = self.xor(maskedDB, dbMask)
191         shash = self.compute_hash(salt)
192 
193         db_shash = db[:self.hash_length]
194         if db_shash != shash:
195             raise DecoderError("EME-OAEP: decoding error")
196 
197         index = db.find('\x01', self.hash_length)
198         if - 1 == index:
199             raise DecoderError("EME-OAEP: decoding error")
200 
201         return db[index + 1:]
202 
203 
204 
205 class PKCS1v1_5Encoder(object):
206     '''
207     RFC 2437 9.1.2 EME-PKCS1-v1_5
208     
209     9.1.2.1 EME-PKCS1-v1_5-ENCODE
210     9.1.2.2 EME-PKCS1-v1_5-DECODE
211     '''
212 
213     def encode(self, msg, keybits=1024):
214         emLen = keybits / 8 - 1
215         if len(msg) > (emLen - 10):
216             raise EncoderError("PKCS1-V1.5: message too long")
217 
218         ps = self.rnd_non_zero(emLen - len(msg) - 2)
219         assert len(ps) >= 8, "PKCS1-V1.5: invalid PS"
220 
221         emout = cStringIO.StringIO()
222         try:
223             emout.write('\x02')
224             emout.write(ps)
225             emout.write('\x00')
226             emout.write(msg)
227             emsg = emout.getvalue()
228         finally:
229             emout.close()
230 
231         return emsg
232 
233 
234     def decode(self, emsg):
235         if len(emsg) < 10:
236             raise DecoderError("PKCS1-V1.5: decoding error")
237 
238         if '\x02' != emsg[0]:
239             raise DecoderError("PKCS1-V1.5: decoding error")
240 
241         index = emsg.find('\x00')
242         if - 1 == index:
243             raise DecoderError("PKCS1-V1.5: decoding error")
244 
245         ps = emsg[1:index]
246         if len(ps) < 8:
247             raise DecoderError("PKCS1-V1.5: decoding error")
248 
249         return emsg[index + 1:]
250 
251 
252     @staticmethod
253     def rnd_non_zero(length):
254         rnd = os.urandom(length)
255         while - 1 != rnd.find('\x00'):
256             rnd = rnd.replace('\x00', os.urandom(1))
257         return rnd

Silverlight does not provide any RSA cryptography classes in the its version of the .NET framework.  Fortunately, the Scrypt project exists and provides a nice RSA library for Silverlight (version 3+) and windows phone 7!  the Scrypt project is licensed under the Microsoft public license (ms-pl).  the RSA.RSACrypto class has an interface that is very similar to that of System.Security.Cryptography.RSACryptoServiceProvider in the full .NET framework.





In my project the Silverlight client would obtain the RSA public key in XML format via a web service call.  This would be used to encrypt login credentials.  During that process an AES key would be generated for the login session.  Any sensitive data would then be encrypted by the AES key shared between the Python server and the Silverlight client. Here is an example of the encryption on the Silverlight side.

 1 using System;
 2 using System.Text;
 3 using RSA;
 4 
 5 // ... other code ...
 6 
 7 // in this example, e is the GetPublicKeyCompletedEventArgs
 8 // parameter from an asynchronous web service call
 9 var pkey = new RSACrypto();
10 pkey.FromXmlString(e.Result.GetPublicKeyResult);
11 
12 StringBuilder output = new StringBuilder();
13 // fill output with some data...                
14 Byte[] raw_data = Encoding.UTF8.GetBytes(output.ToString());
15 var cipher = pkey.Encrypt(raw_data);
16 var encodedCipher = Convert.ToBase64String(cipher);
17 
18 // now encodedCipher is ready to be transported 
19 // to the Python server.

Remember, with RSA public key cryptography you can only encrypt up to keysize / 8 – 1 bytes of data.  Thus if you have a key that is 128 bytes (1024 bits), you can only encrypt up to 127 bytes (1016 bits) of data (depending on the implementation details, maybe less).  The padding schemes used with RSA ensure that the data you are to encrypt is exactly keysize / 8 – 1 bytes in length.  Thus if your data is short, like a 32 byte AES key (16 byte key, 16 byte iv for 256 bits), the padding scheme will pad out the data to keysize / 8 – 1 bytes before the data is encrypted.





This was a fun project to work on.  I learned a great deal and had a blast. I appreciate all of the hard work that others have put in to make the excellent encryption libraries that exist and are freely available.  Maybe some of you will find my padding encoders useful, Cheers!

Py2Exe and SQLAlchemy

I recently had the task of delivering an application written in Python to a large customer.  This would be the first application developed with Python that my company has distributed.  We chose to use Py2Exe for distribution of this program.  Py2Exe is a Python distutils extension that takes your Python code as input and outputs Windows executables that can run without a local Python installation.  Read about it at www.py2exe.org.  I have blogged about it before as well, you can read that here.

At times Py2Exe will not be able to determine that you are using some Python modules and packages.  In this case you will have to manually include them or your resultant binary file will raise an ImportError and stop running.  This is the case when using SQLAlchemy. 

SQLAlchemy is an awesome Python based SQL toolkit and ORM package.  Read all about it at www.sqlalchemy.org.  SQLAlchemy was completely new to me.  I wanted to learn about it so I forced myself to use it on this project.  I have had a very good experience with SQLAlchemy and will use it again whenever I can.  Below is my setup.py file that I used for my application with names and descriptions replaced with meaningless names. 

from distutils.core import setup
import py2exe #@UnusedImport
import platform
 
# 1. List of python modules to exclude from the distribution
mod_excludes = [
    "Tkinter",
    "doctest",
    "unittest",
    "pydoc",
    "pygments",
    "pdb",
    "email"
]
 
# 2. List of dll's (and apparently exe's) to exclude from the distribution
#    if any Windows system dll appears in the dist folder, add it to this
#    list.
dll_excludes = [
    "API-MS-Win-Core-LocalRegistry-L1-1-0.dll",
    "POWRPROF.dll",
    "w9xpopen.exe"
]
 
# 3. List of python modules that are to be manually included.
mod_includes = [
    "Cheetah.DummyTransaction",
]
 
# 4. List of python packages that are to be manually included.
package_includes = [
    "sqlite3",
    "sqlalchemy.dialects.sqlite"
]
 
# 5. determine the distribution folder.
arch = platform.architecture()
if '32bit' in arch:
    dist_dir = "dist-x86"
elif '64bit' in arch:
    dist_dir = "dist-x64"
else:
    raise RuntimeError("Unsupported architecture!")
 
# 6. Dictionary of options to pass to py2exe
py2exe_options = {
    "optimize": 2, # 0 (None), 1 (-O), 2 (-OO)
    "includes": mod_includes,
    "excludes": mod_excludes,
    "dll_excludes": dll_excludes,
    "packages": package_includes,
    "xref": False,
    # bundle_files: 1|2|3
    #    1: executable and library.zip
    #    2: executable, Python DLL, library.zip
    #    3: executable, Python DLL, other DLLs and PYDs, library.zip
    "bundle_files": 3,
    "dist_dir": dist_dir
}
 
# 7. call setup to create the service and the console app
setup(service=[{'modules': 'myservice',
                'icon_resources': [(1, '..\\my.ico')]
               }],
      console=[{'script': '..\\myexe.py',
                'icon_resources': [(1, '..\\my.ico')]
              }],
      version='1.0',
      description='My Service',
      long_description="My service verbose description.",
      author='Jon Anglin',
      author_email='jonanglin@somewhere.com',
      url='http://japrogbits.blogspot.com',
      options={"py2exe": py2exe_options}
     )

To make your executable run correctly, you need to tell Py2exe to include your database DB-API package and the SQLAlchemy database dialect package that you have used.  You can see this in the code above at 4.  I have indicated that Py2exe should include the sqlite3 and the sqlalchemy.dialects.sqlite packages in the final output.  You can also indicate that Py2exe should include a certain Python module.  I have done this as well (above at 3) as I was using the Cheetah template library for formatting my web reports (read about Cheetah).

Also at times Py2exe may include in your output DLLs or Python package and modules that should not be included.  When a Windows system DLL ends up in my distribution folder I know for sure that I need to exclude it in my setup file.  You can see this at 2 above.  Python modules are not so easy to determine though.  You may exclude a module and suddenly find that you executable will no longer run.  If you are certain that a Python package or module is not used in your application you may exclude it as shown in 1 above.

I also build 32 and 64 bit versions of the software.  This is accomplished simply by running setup.py first with the 32 bit python.exe and then again with the 64 bit python.exe.  I use the currently running platform.architecture() string to create separate distribution folders for each build of the software.  You can see that at 5 above.

Multiprocessing, Py2exe, and Windows Services

I recently ran in to an issue with using the Python standard library module multiprocessing from within a Windows service that had been frozen with Py2exe. First I’ll give a brief overview of the components involved for those that may not be familiar with them.

A Windows service is a special type of executable that is started by the service control manager (SCM). Generally you can not just run a Windows service by double clicking on it. The service has a service main function and a control handler function that responds to events sent to the service by the SCM. You can read about services on MSDN here. Writing a Windows service in Python also requires the PyWin32 package. PyWin32 can be obtained here

Py2exe is an extension of distutils that turns a python module into an executable file that can run on a Windows system on which there is not an installed Python distribution. Py2exe can handle many types of executables, dlls, exe, windows services, COM objects, etc… Read all about Py2exe here.

Multiprocessing is a part of the standard Python library in Python 2.6 and later. It is an amazing library that allows you to run any callable Python object in a different process. Read about multiprocessing here.

Below is a simple Windows service named MyService in a Python module. NOTE This is not a complete example of a service. The SvcDoRun method must block to keep the service alive. You can accomplish this in many ways, like by waiting on an event. I leave this as an exercise for the reader. The key part of this code is the code after importing the multiprocessing module. You must provide an executable (generally a Python interpreter) that multiprocessing can use to run python scripts because your service executable will not work. In the code below we are indicating to multiprocessing that it should use myapp.exe as the executable file to run processes and that myapp.exe will be in the same directory as our service executable. We will provide another Python module that Py2exe will use to build myapp.exe

import os
import sys
import win32security
import win32service
import win32serviceutil

# Give the multiprocessing module a python interpreter to run
import multiprocessing
executable = os.path.join(os.path.dirname(sys.executable), 'myapp.exe')
multiprocessing.set_executable(executable)
del executable


class MyService(win32serviceutil.ServiceFramework):
   _svc_name_ = 'MyService'
   _svc_display_name_ = 'MyService'
   _svc_description_ = 'MyService Example Program'
#    _exe_name_ = 'pythonservice.exe'   # Defaults to PythonService.exe
#    _svc_deps_ = None                  # sequence of service names on which this depends
#    _exe_args_ = None                  # Defaults to no arguments

   def SvcDoRun(self):

       # Set the current directory to the directory from
       # which this executable was launched.
       currentDir = os.path.dirname(sys.executable)
       os.chdir(currentDir)

       # Tell Windows which privileges you need, others are removed.
       set_privileges((win32security.SE_ASSIGNPRIMARYTOKEN_NAME,
                       win32security.SE_CHANGE_NOTIFY_NAME,
                       win32security.SE_CREATE_GLOBAL_NAME,
                       win32security.SE_SHUTDOWN_NAME))

       #  Implement service here, and block
       #  When this method returns the service is stopped.


if __name__ == '__main__':
   # For a service, this never gets called.
   # 
   # freeze_support must be the first line
   # after the if __name__ == '__main__'
   multiprocessing.freeze_support()

   # Pass the command line to the service utility library.
   # This can handle start, stop, install, remove and other commands.
   win32serviceutil.HandleCommandLine(MyService)

Here is the myapp.py module used to generate our executable for the multiprocessing module. This is a very simple Python script that actually does more than it needs to even for this task.

import multiprocessing
import os

def main ():
   print('myapp: ', os.getpid())


if __name__ == '__main__':
   # freeze_support must be the first line
   # after the if __name__ == '__main__'
   multiprocessing.freeze_support()
   main()

Now all we need is a setup.py script that tells Py2exe how to build our executables. Here is the code.

from distutils.core import setup
import py2exe

#  We must leave the optimization level at 1 if we use the kid template
#  library.  This leaves the doc strings in the library code.  The kid 
#  templating engine parses its doc string at run time, thus if the doc 
#  string is not in the pyo file, the program crashes.

# List of python modules to exclude from the distribution
excludes = [
   "Tkinter",
   "doctest",
   "unittest",
   "pydoc",
   "pdb"
]

# List of dll's (and apparently exe's) to exclude from the distribution
# if any Windows system dll appears in the dist folder, add it to this
# list.
dll_excludes = [
   "API-MS-Win-Core-LocalRegistry-L1-1-0.dll",
   "MPR.dll",
   "MSWSOCK.DLL",
   "POWRPROF.dll",
   "profapi.dll",
   "userenv.dll",
   "w9xpopen.exe",
   "wtsapi32.dll"
]

# List of python modules that are to be manually included.
mod_includes = []

package_includes = []

py2exe_options = {
   "optimize": 2, # 0 (None), 1 (-O), 2 (-OO)
   "excludes": excludes,
   "dll_excludes": dll_excludes,
   "packages": package_includes,
   "xref": False,
   # bundle_files: 1|2|3
   #    1: executable and library.zip
   #    2: executable, Python DLL, library.zip
   #    3: executable, Python DLL, other DLLs and PYDs, library.zip
   "bundle_files": 3
}

setup(service=[{'modules': 'myservice',
               'icon_resources': [(1, 'myapp.ico')],
             }],
     console=[{'script': 'myapp.py',
               'icon_resources': [(1, 'myapp.ico')]
             }],
     version='1.0',
     options={"py2exe": py2exe_options}
    )

I have skimmed this down a bit from an actual setup.py that I use at work. In practice I have found that Py2exe sometimes includes modules and libraries that are not necessary. The excludes list is a list of Python modules that you want to exclude from your distribution. Make sure that you know the modules listed here are not actually used. The dll_excludes option is a list of DLL and possibly EXE files that for whatever reason Py2exe is copying to your dist folder event though they may be Windows system DLLs (that you are most likely not allowed to redistribute), or the old w9xpopen.exe (for Windows 9x only). The mod_includes and package_includes options can be used to force inclusion of Python modules or packages that you must have in your distribution but for some reason Py2exe is not placing them in your dist folder. The work is done in the call to setup. Here we are telling it to build a service from the myservice module and a console application using the myapp.py module. Each of these output files uses the same icon as specified. We pass Py2Exe specific options to Py2Exe via the py2exe_options dictionary. Build the executables by running:

python –OO setup.py py2exe

Now that we have provided an executable for multiprocessing we can do this somewhere in our service:

from multiprocessing import Process

def handle_request(req):
   # do something useful
   pass

def on_request(request):
   Process(target=handle_request, args=(request,)).start()

The Point: In a frozen Windows service, you have to provide an executable to the multiprocessing module that can be used to run Python scripts in a new process.