Thursday, November 18, 2010

Enable ADC on Your Windows Phone 7

I received a Windows Phone 7 device by attending the Microsoft PDC in Redmond this year.  I was given an LG E900 device.  The phone came from Vodafone in Germany.  It even has the European style electrical plug, luckily they provided an adaptor. 

The story goes that Microsoft approached AT&T to buy phones for the PDC attendees.  AT&T refused so Microsoft bought phones from Vodafone.  Microsoft then employed someone to switch the language of each phone from German to English.  I don’t know if this is entirely true, but that was the story going around the PDC.

Like many others, I popped the SIM card out of my IPhone and placed it in the E900.  ADC was not enabled so I had to set many properties of the phone carrier that a normal customer would not have to.  My carrier is AT&T.  I first had to set the APN to enable data access.  This can be done by going to the Settings app, choosing cellular from the system page and clicking the ‘edit apn’ button on the bottom of the page.  In the APN text box enter: wap.cingular then click the done button (check mark). 

You may also have to set your SMS number although this was done automatically for me. Settings app, flip to the applications page and choose ‘messaging.’  Here you will find the SMS center number.   You can call AT&T to get your number, mine is +13123149810.

Next you may need to set the voicemail number.  I had to do this.  Settings app, flip to the applications page and choose ‘phone.’  Here you will see the voicemail number.  Once again I obtained this number by calling AT&T, yours may be different, mine is +13174379963.

 

Enabling MMS

Everything was now working on the phone except for MMS.  I was able to get the required MMS settings from AT&T, but there did not appear to be a way to put these settings into the phone.  I had heard that you could reset the phone and at the first boot (with your SIM card in place) you could set the MMS settings.  This turned out to not be true.  Thanks to Google translator I was able to get the phone back to English (it booted up in German after the reset).  The original instructions came from this site, many thanks to the author of that post. Here is how to enable MMS without resetting your phone.  Sorry for the low quality pictures, you can’t take a screen shot of a running app on Windows Phone 7 like you can with an Iphone.

  1. Open the phone app used to make calls
  2. Dial ##634# and click call
    1. This will install the MFG application on your phone (shows up in your installed applications).
  3. In the MFG application enter the password 277634#*#
    1. 2e
  4. In the Factory Menu choose  “Engineer Menu”
    1. 2d
  5. In the Engineer Menu choose “Other Setting”
    1. 2c
  6. In the Other Setting Menu choose “Set ADC” and set it to Enable
    1. 2a 
  7. In the Other Setting Menu choose “Set network profiles”
    1. 2b
  8. On the bottom half of the screen you will see a list of XML files that can be used to configure the network profile on your phone.  You will have to guess which file you will need for your carrier and country.  The currently selected profile will be highlighted in your theme color.  You may want to write down the current setting in case you need to go back. Because I am on AT&T in the US, I chose the file _PROV_ATT_US_310_410.xml.
  9. To view the new settings choose “View current profile” from the “Other Setting” menu.  This will bring up the “View config xml” page. This will show you an XML file that contains the new network profile settings that you just placed into your phone.  Within this XML file on my phone, I can see (among other things) the settings that the AT&T representative gave to enable MMS.
    1. 29
  10. Test it out, mine worked!
  11. The MFG app will now be in your applications page with all of your other apps.  You will still need the password to open it.

Tuesday, September 14, 2010

How to Migrate a Visual SourceSafe Database to Subversion

The software tools involved in this migration are:

  • Collabnet Subversion Edge 1.2 – the subversion server
  • TortoiseSVN 1.6.10 – a subversion client
  • VSS2SVN 0.2.0  - the Visual SourceSafe to Subversion conversion tool
  • Visual SourceSafe 2005 – the VSS software

    I used Collabnet Subversion Edge 1.2 as my subversion server.  It is easy to install and configure and it is free.  This post assumes you can handle the server setup yourself. First we must create a subversion repository to hold the converted VSS database. Open the web interface to Collabnet Subversion (http://your-server-here:3343) and click the Repositories tab.  Click the ‘New Repository’ link.  This will ask you to provide a name for the repository and if you would like to use the standard trunk/branches/tags structure.

    image

    We need to set up a pre-revision hook in the new repository for the conversion tool to successfully save your source code history.  To do this, go to your repository folder in the file system and find the hooks directory.  It is a sub-folder of your newly created repository folder.  In the hooks directory and several tmpl files. These are hook templates.  To get a hook to execute you must place an executable file (exe, script, batch file, etc…) in the hooks folder with the exact name of the hook.  For our purposes, create a file named pre-revprop-change.bat and in this file place the line: exit /b 0.

    image  

    We are done on the server. 

    You most likely have Visual SourceSafe installed on your client computer.  After all this is what you have been using for source control.  If not, you must install and configure Visual SourceSafe 2005 on your client computer.  Next, Install TortoiseSVN on your client computer.  Using TortoiseSVN pull down your empty subversion repository that we created earlier.  Any temp folder will do.  We just need to have TortoiseSVN to have a connection to the subversion server so that the VSS2SVN tool will work properly.

    You may want to copy your entire VSS database to your local computer for the migration.  First make sure that no user has any checked out items in the VSS database.  Next copy the entire VSS database to your local computer.  The conversion tool may take several hours if you have a large database.  The nice thing is that the VSS2SVN tool will allow you to convert one project at a time.

    Download and run the VSS2SVN tool.  If you are on a 64 bit verison of Windows you will need to download the source code for VSS2SVN.  Open the solution in Visual Studio 2008, add an x86 configuration to the project and rebuild.  By default it is built for any CPU, but VSS only provides x86 versions of its automation DLL.  VSS2SVN uses this automation DLL so we must have a 32 bit (x86) build of the VSS2SVN tool.

    The VSS2SVN tool is under-documented.  First you need to set the VSS Parameters.  This includes: Path to Scrsafe.ini, User name, Password, and VSS project to use.  Browse to your copy of your VSS databases srcsafe.ini folder, enter your VSS user name and password, enter the path to the project to be converted and then click the ‘Find file in source safe’ button.

    image

      

    Click ‘OK’ to dismiss this dialog and then the General and SVN parameters become available to be edited.  Set the working directory to something like %TEMP% and check the ‘Delete temporary files when done’ button.  Fill out the ‘SVN path to use’ with the path to the trunk of the repository that you created at the beginning of this blog post and click the ‘Migrate to subversion’ button.  This may take a long time!  It depends on the size of the project and the number of revisions that is has.  Your SourceSafe history will be converted to subversion revisions.

    When this is done you can open the web interface to Collabnet Subversion and browse through your source repository.  Collabnet also provides Windows command line subversion tools, as well as Visual Studio and Eclipse plug-ins.

  • Friday, September 10, 2010

    VC++ Directories in Visual Studio 2010

    The default include, lib, etc… directories in Visual Studio 2010 are no longer project dependent.  You can, if you need to, over-ride the settings on a per project basis.  The directory settings are stored in the following location:

    %USERPROFILE%\AppData\Local\Microsoft\MSBuild\v4.0

    In this directory you will find the files:  Microsoft.Cpp.Win32.user.props, Microsoft.Cpp.x64.user.props, and Microsoft.Itanium.user.props  (the last two only if you have installed the x64 tools).

    These files are UTF-8 encoded XML files.  In them you can define your include, lib, executable, reference, and source paths.  You can also define excluded directories.  These settings become the defaults for all C++ projects.  Of course the Win32 file is for 32 bit Windows builds, while the x64 file is for 64 bit AMD64 builds.  I don’t know anyone that has an Itanium computer so I won’t even discuss those.

    This is great now that you don’t have to search and replace through every .vcproj file (I know .vcxproj file in 2010) every time you want to change an include path.  Just change the paths in these files and all projects are updated.

    Teams:
    You can store your Microsoft.Cpp.xxx.user.props files in source control.  Each team member can pull down the required files so that everyone is on the same path (bad pun, couldn’t resist).

    Friday, September 3, 2010

    My Thanks to the Profiler

    I recently had an issue at work with one of our kernel mode file system filters for Windows.   The CPU was spending way too much time executing our drivers code.  I ran the kernrate sampling profiler tool from the Windows Driver Kit on our driver to see what code was taking so long to execute.  The profiler revealed that over 50% of the time that our driver was executing in a function named RtlUnicodeStStri.  This is a function that I wrote that is similar to the strstr function in the C standard library with the exception that it is case-insensitive and it works on UNICODE_STRING structures. In the Windows kernel, UNICODE_STRING structures are used in place of character arrays.  The structure is defined like this:

    typedef struct _UNICODE_STRING
    {
        USHORT Length;
        USHORT MaximumLength;
        PWCH Buffer;
    } UNICODE_STRING, *PUNICODE_STRING;   


    Windows includes type definitions of USHORT as unsigned short and PWCH as wchar_t*. When using UNICODE_STRING structures you have the length of the string (in bytes) and the maximum length (i.e. the buffer size) of the string (in bytes).  This provides many benefits: you can safely detect if any operation will overrun your string buffer, you can compare lengths before comparing strings, etc…  The strings stored in the Buffer member of the UNICODE_STRING structure are not guaranteed to be NULL terminated and many times they are not.  Thus you can not use the standard C library functions on the Buffer member.
    Although the Windows kernel internally uses the UNICODE_STRING, there is not a very comprehensive library for handling UNICODE_STRINGS (The string handling functions are in the Rtl library).  This causes many developers like myself to write functions like RtlUnicodeStrStri.  Below is the original version of RtlUnicodeStrStri that I had written. 
     1 __drv_maxIRQL(APC_LEVEL)
     2 NTSTATUS
     3 RtlUnicodeStrStri(
     4     __in PCUNICODE_STRING  Str,
     5     __in PCUNICODE_STRING  SubStr,
     6     __out_opt PUNICODE_STRING  Result,
     7     __out_opt int* MatchStartIndex
     8     )
     9 {
    10     USHORT l1=0, l2=0;
    11     USHORT start = 0;
    12     
    13     //  Translate all counts to character counts.
    14     const USHORT StrCharLen = Str->Length / sizeof(WCHAR);
    15     const USHORT SubStrCharLen = SubStr->Length / sizeof(WCHAR);
    16 
    17     ASSERT_VALID_STRING(Str);
    18     ASSERT_VALID_STRING(SubStr);
    19     
    20     while (start < StrCharLen)
    21     {
    22         for (l1=start,l2=0; l1<StrCharLen && l2<SubStrCharLen; ++l1, ++l2)
    23         {
    24             if (RtlUpcaseUnicodeChar(Str->Buffer[l1]) != 
    25                 RtlUpcaseUnicodeChar(SubStr->Buffer[l2]))
    26             {
    27                 break;
    28             }
    29         }
    30     }
    31 
    32     //    other code removed...
    33 
    34 } // RtlUnicodeStrStri

    At first glance, I thought that the function was fairly straightforward.  Then I noticed one particular optimization that should have been made.  In the worst case, when the sub-string is not present, the algorithm continues to look through the entire source string even when the remaining length of the source string is too short to hold the sub-string.  This is an optimization that can not be made in the C version of strstr because you have no idea in advance how long the source string.  The full code is given below.


     1 __drv_maxIRQL(APC_LEVEL)
     2 NTSTATUS
     3 RtlUnicodeStrStri(
     4     __in PCUNICODE_STRING  Str,
     5     __in PCUNICODE_STRING  SubStr,
     6     __out_opt PUNICODE_STRING  Result,
     7     __out_opt int* MatchStartIndex
     8     )
     9 /*++
    10 
    11 Routine Description:
    12 
    13     This routine search Str for the first occurrence of the string SubStr
    14     in a case insensitive manner.  If Result is provided and the search is
    15     successful, it will be filled in with the string starting at the matched 
    16     sub-string.
    17 
    18 Arguments:
    19 
    20     Str - The string to search.
    21     
    22     SubStr - The string to search for.
    23 
    24     Result - Result of the search starting at the sub-string and 
    25              continuing through the rest of Str. This structure 
    26              justs points to the string in Str.
    27 
    28     MatchStartIndex - The starting index within Str at which SubStr
    29                       appears.                                   
    30 
    31 Return Value:
    32 
    33     STATUS_SUCCESS if SubStr was found in Str, 
    34     STATUS_OBJECT_NAME_NOT_FOUND otherwise.
    35 
    36 Unit test:
    37     
    38     UnitTest4() via RtlWStrStri
    39 
    40 --*/
    41 {
    42     USHORT l1=0, l2=0;
    43     USHORT start = 0;
    44     
    45     //  Translate all counts to character counts.
    46     const USHORT StrCharLen = Str->Length / sizeof(WCHAR);
    47     const USHORT SubStrCharLen = SubStr->Length / sizeof(WCHAR);
    48 
    49     ASSERT_VALID_STRING(Str);
    50     ASSERT_VALID_STRING(SubStr);
    51     
    52     // Run the loop while the length of the sub-string (SubStr) is less
    53     // than the remaining length of the input string being searched (Str). 
    54     while (start < StrCharLen && (SubStrCharLen <= (StrCharLen - start)))
    55     {
    56         for (l1=start,l2=0; l1<StrCharLen && l2<SubStrCharLen; ++l1, ++l2)
    57         {
    58             if (RtlUpcaseUnicodeChar(Str->Buffer[l1]) != 
    59                 RtlUpcaseUnicodeChar(SubStr->Buffer[l2]))
    60             {
    61                 break;
    62             }
    63         }
    64 
    65         if (l2 == SubStrCharLen)
    66         {
    67             if (ARGUMENT_PRESENT(Result))
    68             {
    69                 //  Translate start back to byte count.
    70                 l1 = start * sizeof(WCHAR);
    71 
    72                 Result->Length = Str->Length - l1;
    73                 Result->MaximumLength = Str->MaximumLength - l1;
    74                 Result->Buffer = &Str->Buffer[start];
    75             }
    76 
    77             if (ARGUMENT_PRESENT(MatchStartIndex))
    78             {
    79                 *MatchStartIndex = start;
    80             }
    81 
    82             return STATUS_SUCCESS;
    83         }
    84 
    85         ++start;
    86     }
    87 
    88     return STATUS_OBJECT_NAME_NOT_FOUND;
    89 
    90 } // RtlUnicodeStrStri

    This optimization dropped our drivers execution time dramatically, to an acceptable level, though RtlUnicodeStrStri was still responsible for about %30 of our drivers execution time.  I tried to optimize the calls to RtlUpcaseUnicodeChar (A Windows API) by doing quick translations for English ASCII codes (including ignoring chars that were already up-cased).  It turns out that RtlUpcaseUnicodeChar is very very good and any optimization attempt I made just made the driver’s execution time worse.

    It turns out that our programmer’s (including myself) just got lazy and overused RtlUnicodeStrStri.  I mean, why parse a string and do a compare on the part of interest when you can just call RtlUnicodeStrStri?  So I wrote many more UNICODE_STRING handling functions like RtlStringEndsWithSuffix that can replace many calls to RtlUnicodeStrStri and more clearly capture the programmer’s intention in the code.

    Wednesday, September 1, 2010

    Porting C# Code to IronPython, an Example

    Lately, I have been reading Jeff Richter’s book “CLR via C#, 3rd Edition.”  I have read several of Jeff Richter’s programming books over the years, mostly his series of books on programming Windows with C/C++.  I have always liked his writing.  He is not afraid to say that Microsoft has made a mistake by providing a Windows API or technology that is substandard.  He won’t just complain about it, no, he will go on to say how he would have done it (and provide code).  Don’t get me wrong here, he also points out many things that Microsoft has done well.  I just respect the fact that he is not afraid to provide a dissenting opinion.

    In chapter 26 of “CLR via C#, 3rd, Ed.,” “Compute-Bound Asynchronous Operations”  there is a cool example of using Task objects from the System.Threading.Task namespace.  This small example demonstrates several features of Task objects including using a TaskScheduler to sync back to the GUI thread, cancelling a task, and using a continuation task to execute another action after a task completes.  The sample uses a contrived example of a compute bound function named Sum that keeps the CPU busy for a while by adding integers from 0 through some number n.  This program is a simple Windows Forms application.  I have modified it slightly to use a button instead of just detecting a mouse click on the form.  Below is the form and the C# code that appeared in the book (modified by me).

      form

     1 using System;
    2 using System.Windows.Forms;
    3 using System.Threading;
    4 using System.Threading.Tasks;
    5
    6 namespace TaskSchedTest
    7 {
    8 public partial class Form1 : Form
    9 {
    10 private readonly TaskScheduler m_syncContextTaskScheduler;
    11 private CancellationTokenSource m_cts;
    12
    13 public Form1()
    14 {
    15 m_syncContextTaskScheduler =
    16 TaskScheduler.FromCurrentSynchronizationContext();
    17
    18 InitializeComponent();
    19 }
    20
    21 private void button1_Click(object sender, EventArgs e)
    22 {
    23 if (null != m_cts)
    24 {
    25 m_cts.Cancel();
    26 }
    27 else
    28 {
    29 label1.Text = "Operation Running...";
    30 button1.Text = "Cancel Task";
    31
    32 // Define a function to reset the state of the program
    33 // upon task completion.
    34 Func<String, Int32> reset = (String labelText) =>
    35 {
    36 label1.Text = labelText;
    37 m_cts = null;
    38 button1.Text = "Run Task";
    39 return 0;
    40 };
    41
    42 m_cts = new CancellationTokenSource();
    43
    44 // This task uses the default task scheduler and executes
    45 // on a thread pool thread.
    46 var t = new Task<Int64>(() => Sum(m_cts.Token, 200000000), m_cts.Token);
    47 t.Start();
    48
    49 // These tasks use the syn context task schedules and execute
    50 // on the GUI thread.
    51 t.ContinueWith(task => { reset("Result: " + task.Result); },
    52 CancellationToken.None,
    53 TaskContinuationOptions.OnlyOnRanToCompletion,
    54 m_syncContextTaskScheduler);
    55
    56 t.ContinueWith(task => { reset("Operation canceled"); },
    57 CancellationToken.None,
    58 TaskContinuationOptions.OnlyOnCanceled,
    59 m_syncContextTaskScheduler);
    60
    61 t.ContinueWith(task => { reset("Operation faulted"); },
    62 CancellationToken.None,
    63 TaskContinuationOptions.OnlyOnFaulted,
    64 m_syncContextTaskScheduler);
    65 }
    66 }
    67
    68 private static Int64 Sum(CancellationToken ct, Int32 n)
    69 {
    70 Int64 sum = 0;
    71 for (; n > 0; n--)
    72 {
    73 // The following throws OperationCanceledException when Cancel
    74 // is called on the CancellationTokenSource referred by the token
    75 ct.ThrowIfCancellationRequested();
    76 checked { sum += n; }
    77 }
    78
    79 return sum;
    80 }
    81 }
    82 }



    I added the reset function that is used within the lambda expressions in the ContinueWith method calls.  I like this code because it is very succinct and it doesn’t pollute your class namespace with a bunch of private functions that are only used within this one method call.  The performance hit of creating the reset function and the lambda expressions are negligible as well.  The C# compiler actually generates an internal class that contains as method members the lambda expression functions.  You can see this in the image below taken from ildasm.exe.  The compiler generated class is called <>c__DisplayClass7 and the lambda expressions from the ContinueWith method calls are named <button1_Click>b__2, <button1_Click>b__3, and <button1_Click>b__4. You can also see the reset function object as a field.       



    ildasm1



    I liked this sample so much that I wanted to port it to IronPython.  I enjoy Python programming and I have been trying to incorporate IronPython into my work whenever it makes sense to do so.  Being that Python and IronPython are dynamic languages, interfacing the to the .NET Framework can make the syntax cumbersome and not very Pythonic at times.  You must ensure that your IronPython code is using the correct types because the .NET Framework is statically typed. A lot of the time the IronPython interpreter will infer the correct types for you and everything just works.  At other times, the IronPython interpreter does not infer the correct types and you are left with a runtime exception.  For this IronPython example I changed to a Windows Presentation Foundation application, mostly because Visual Studio has a drag and drop WPF editor for IronPython.  Below is the WPF form with XAML and the IronPython code.



    wpf





    xaml




     1 import clr
    2 clr.AddReference('PresentationFramework')
    3
    4 from System import Func
    5 from System.Windows import Application, Window
    6 from System.Threading import CancellationTokenSource, CancellationToken
    7 from System.Threading.Tasks import (Task, TaskScheduler,
    8 TaskContinuationOptions
    9 )
    10
    11 class MyWindow(Window):
    12 def __init__(self):
    13 clr.LoadComponent('WpfApplication1.xaml', self)
    14 self.cts = None
    15
    16 def AppLoaded(self, sender, e):
    17 self.syncContextTaskScheduler = TaskScheduler.FromCurrentSynchronizationContext()
    18
    19 def Button_Click(self, sender, e):
    20 if self.cts is not None:
    21 self.cts.Cancel()
    22 else:
    23 self.label1.Content = "Operation Running...";
    24 self.button1.Content = "Cancel Task";
    25
    26 # Define a function to reset the state of the program
    27 # upon task completion.
    28 def reset(labelText):
    29 self.label1.Content = labelText
    30 self.button1.Content = "Run Task"
    31 self.cts = None
    32
    33 self.cts = CancellationTokenSource();
    34
    35 # This task uses the default task scheduler and executes
    36 # on a thread pool thread.
    37 t = Task[long](lambda: self.Sum(self.cts.Token, 2000000), self.cts.Token)
    38 t.Start()
    39
    40 NoneType = type(None)
    41 # These tasks use the syn context task schedules and execute
    42 # on the GUI thread.
    43 t.ContinueWith[NoneType](
    44 Func[Task[long], NoneType](
    45 lambda task: reset("Result: {0}".format(task.Result))
    46 ),
    47 CancellationToken.None,
    48 TaskContinuationOptions.OnlyOnRanToCompletion,
    49 self.syncContextTaskScheduler)
    50
    51 t.ContinueWith[NoneType](
    52 Func[Task[long], NoneType](
    53 lambda task: reset("Operation canceled.")
    54 ),
    55 CancellationToken.None,
    56 TaskContinuationOptions.OnlyOnCanceled,
    57 self.syncContextTaskScheduler)
    58
    59 t.ContinueWith[NoneType](
    60 Func[Task[long], NoneType](
    61 lambda task: reset("Operation faulted.")
    62 ),
    63 CancellationToken.None,
    64 TaskContinuationOptions.OnlyOnFaulted,
    65 self.syncContextTaskScheduler)
    66
    67 @staticmethod
    68 def Sum(cancellationToken, n):
    69 sum = 0L
    70 for i in xrange(n + 1):
    71 cancellationToken.ThrowIfCancellationRequested()
    72 sum += i
    73 return sum
    74
    75
    76 if __name__ == '__main__':
    77 Application().Run(MyWindow())



    The most difficult part of this port was getting the .NET Framework generic types correct.  When constructing the initial Task object, I had to use the Task[long] notation for generics with IronPython.  Without the generic parameter, the IronPython interpreter would produce a non-generic Task object and this version of the Task object does not have a Result property as the generic parameter is the result type (line 37).  The nice thing about this line is that I could pass in a Python lambda expression directly and the interpreter infers the proper .NET type, Func[long], in this case.  As you can see later in the code this is not the case.  For those that do not know, generic syntax in IronPython differs from C#. For example: Task<TResult>, in C# you may have Task<Int64> where in IronPython you would write Task[long]. 



    The ContinueWith method calls on lines 43, 51, and 59 gave me the most trouble.  I found that if I tried to pass the lambda expressions directly as the first parameter to ContinueWith, the IronPython interpreter would generate a Func[Task, long] object, when in fact I needed to have Func[Task[long], NoneType] objects.  This is because each of these lambda expressions are called with a single parameter of type Task[long] and they do not have a return value.  In C# this would be void, IronPython it would be NoneType.  To make this work, I had to explicitly create Func[Task[long], NoneType] objects and pass them into ContinueWith.  Luckily I could pass the Python lambda expressions directly to the Func constructor.  I also used the generic notation on the ContinueWith[NoneType] calls to state that they do not have any return value. 



    When I first coded this up I had mistakenly used Func[Task[long], long] objects and ContinueWith[long] method calls.  This is stating that the functions will take a Task[long] parameter (which is correct) and return a long value (which is not correct).  This actually seemed to work, at least until the Finalizer thread ran!  When the Func[Task[long], long] object was called by the framework, an exception was thrown because it was expecting a long value to be returned but received NoneType.  It seemed to work because my function had already executed.  The Finalizer thread would see that a Task had thrown an exception.  It would then pack up this exception into an AggregateException and throw that object. 



    I had fun porting this sample to IronPython and I hope to use it more in my daily work.  Interfacing with the .NET Framework with IronPython can be a very non-Pythonic experience, but this just how it is when you cross the dynamic to static type boundary.

    Friday, July 2, 2010

    Py2Exe and SQLAlchemy

    I recently had the task of delivering an application written in Python to a large customer.  This would be the first application developed with Python that my company has distributed.  We chose to use Py2Exe for distribution of this program.  Py2Exe is a Python distutils extension that takes your Python code as input and outputs Windows executables that can run without a local Python installation.  Read about it at www.py2exe.org.  I have blogged about it before as well, you can read that here.

    At times Py2Exe will not be able to determine that you are using some Python modules and packages.  In this case you will have to manually include them or your resultant binary file will raise an ImportError and stop running.  This is the case when using SQLAlchemy. 

    SQLAlchemy is an awesome Python based SQL toolkit and ORM package.  Read all about it at www.sqlalchemy.org.  SQLAlchemy was completely new to me.  I wanted to learn about it so I forced myself to use it on this project.  I have had a very good experience with SQLAlchemy and will use it again whenever I can.  Below is my setup.py file that I used for my application with names and descriptions replaced with meaningless names. 

    from distutils.core import setup
    import py2exe #@UnusedImport
    import platform

    # 1. List of python modules to exclude from the distribution
    mod_excludes = [
    "Tkinter",
    "doctest",
    "unittest",
    "pydoc",
    "pygments",
    "pdb",
    "email"
    ]

    # 2. List of dll's (and apparently exe's) to exclude from the distribution
    # if any Windows system dll appears in the dist folder, add it to this
    # list.
    dll_excludes = [
    "API-MS-Win-Core-LocalRegistry-L1-1-0.dll",
    "POWRPROF.dll",
    "w9xpopen.exe"
    ]

    # 3. List of python modules that are to be manually included.
    mod_includes = [
    "Cheetah.DummyTransaction",
    ]

    # 4. List of python packages that are to be manually included.
    package_includes = [
    "sqlite3",
    "sqlalchemy.dialects.sqlite"
    ]

    # 5. determine the distribution folder.
    arch = platform.architecture()
    if '32bit' in arch:
    dist_dir = "dist-x86"
    elif '64bit' in arch:
    dist_dir = "dist-x64"
    else:
    raise RuntimeError("Unsupported architecture!")

    # 6. Dictionary of options to pass to py2exe
    py2exe_options = {
    "optimize": 2, # 0 (None), 1 (-O), 2 (-OO)
    "includes": mod_includes,
    "excludes": mod_excludes,
    "dll_excludes": dll_excludes,
    "packages": package_includes,
    "xref": False,
    # bundle_files: 1|2|3
    # 1: executable and library.zip
    # 2: executable, Python DLL, library.zip
    # 3: executable, Python DLL, other DLLs and PYDs, library.zip
    "bundle_files": 3,
    "dist_dir": dist_dir
    }

    # 7. call setup to create the service and the console app
    setup(service=[{'modules': 'myservice',
    'icon_resources': [(1, '..\\my.ico')]
    }],
    console=[{'script': '..\\myexe.py',
    'icon_resources': [(1, '..\\my.ico')]
    }],
    version='1.0',
    description='My Service',
    long_description="My service verbose description.",
    author='Jon Anglin',
    author_email='jonanglin@somewhere.com',
    url='http://japrogbits.blogspot.com',
    options={"py2exe": py2exe_options}
    )



    To make your executable run correctly, you need to tell Py2exe to include your database DB-API package and the SQLAlchemy database dialect package that you have used.  You can see this in the code above at 4.  I have indicated that Py2exe should include the sqlite3 and the sqlalchemy.dialects.sqlite packages in the final output.  You can also indicate that Py2exe should include a certain Python module.  I have done this as well (above at 3) as I was using the Cheetah template library for formatting my web reports (read about Cheetah).



    Also at times Py2exe may include in your output DLLs or Python package and modules that should not be included.  When a Windows system DLL ends up in my distribution folder I know for sure that I need to exclude it in my setup file.  You can see this at 2 above.  Python modules are not so easy to determine though.  You may exclude a module and suddenly find that you executable will no longer run.  If you are certain that a Python package or module is not used in your application you may exclude it as shown in 1 above.



    I also build 32 and 64 bit versions of the software.  This is accomplished simply by running setup.py first with the 32 bit python.exe and then again with the 64 bit python.exe.  I use the currently running platform.architecture() string to create separate distribution folders for each build of the software.  You can see that at 5 above.

    Tuesday, May 11, 2010

    Programming MongoDB on Windows – Basics

    Here are a few samples that build on my previous post “Installing MongoDB on a Windows Server.”  The following code snippets were developed in Visual Studio 2010 with the IronPython Tools for Visual Studio installed.  You can grab these tools here
    The first code snippet queries for the one document that we inserted into the database in the previous post.  This just shows a basic connection and query using the C# programming language.  The .NET drivers for MongoDB that I am using here, mongodb-csharp, come from the following link: http://github.com/samus/mongodb-csharp.
    csharp 
    Next is the exact same code only in IronPython.  I had one issue while creating this simple sample.  The IronPython code would not load the MongoDB.Driver.dll that I downloaded from github.  I had to download the source for it and rebuild it.  Then, IronPython would load the assembly.
    ipy   

    Installing MongoDB on a Windows Server

    To start, I downloaded the latest mongo binaries from  the mongo download site.  This was version 1.4.2 at the time of this writing.  I was installing mongo on a 64 bit Windows 2008 server so I picked the Windows 64-bit binary package.

    On the server, I created the folder C:\Program Files\MongoDB to hold the binaries and the folder C:\ProgramData\mongodb\data to hold the mongo databases.  I then extracted the binaries from the downloaded file (mongodb-win32-x86_64-1.4.2.zip) to the C:\Program Files\MongoDB folder.  I then opened a command prompt as Administrator and installed mongo as a service by executing the following line:

    “C:\Program Files\MongoDB\bin\mongod” -–dbpath C:\ProgramData\mongodb\data -–install

    This runs the mongod.exe executable and tells it to use C:\ProgramData\mongodb\data as its database directory and to install itself as a Windows service.  It is important that you use the full path to the mongod.exe executable or else the Windows service control manager will not be able to find the executable to start the service.  The service can now be started by executing the this command from the administrative command prompt:

    net start MongoDB

    I then used the mongo.exe database client application to test the install.  I inserted a simple person record and then queried for that record.  Mongo creates a ‘test’ database and a ‘things’ collection on my behalf and inserts the document into it.  A screen shot is shown below.  You can see the object ID assigned to the document by mongoDB.

    mongo-1

     

    To access mongoDB from another computer the Windows firewall needs to be configured.  I allowed any domain connections to access this installation of mongo.  This may not be acceptable for your needs.  As this is a test database in my environment, this configuration is acceptable.  Here is a screen shot to show the configuration in Windows firewall.

    mongo-2

     

    Now I wanted to connect to the database from another computer.  On my own desktop I then ran the mongo.exe database client to connect to the server and the ‘test’ database.  Then I queried for the known record.  The results are shown below.

    mongo-3

    As you can see you use the <server name>/<database name> pair to connect.  There are many more options of course, but this is sufficient if you are using the defaults. 

    Thursday, March 11, 2010

    Multiprocessing, Py2exe, and Windows Services

    I recently ran in to an issue with using the Python standard library module multiprocessing from within a Windows service that had been frozen with Py2exe. First I’ll give a brief overview of the components involved for those that may not be familiar with them.

    A Windows service is a special type of executable that is started by the service control manager (SCM). Generally you can not just run a Windows service by double clicking on it. The service has a service main function and a control handler function that responds to events sent to the service by the SCM. You can read about services on MSDN here. Writing a Windows service in Python also requires the PyWin32 package. PyWin32 can be obtained here

    Py2exe is an extension of distutils that turns a python module into an executable file that can run on a Windows system on which there is not an installed Python distribution. Py2exe can handle many types of executables, dlls, exe, windows services, COM objects, etc… Read all about Py2exe here.

    Multiprocessing is a part of the standard Python library in Python 2.6 and later. It is an amazing library that allows you to run any callable Python object in a different process. Read about multiprocessing here.

    Below is a simple Windows service named MyService in a Python module. NOTE This is not a complete example of a service. The SvcDoRun method must block to keep the service alive. You can accomplish this in many ways, like by waiting on an event. I leave this as an exercise for the reader. The key part of this code is the code after importing the multiprocessing module. You must provide an executable (generally a Python interpreter) that multiprocessing can use to run python scripts because your service executable will not work. In the code below we are indicating to multiprocessing that it should use myapp.exe as the executable file to run processes and that myapp.exe will be in the same directory as our service executable. We will provide another Python module that Py2exe will use to build myapp.exe

    import os
    import sys
    import win32security
    import win32service
    import win32serviceutil

    # Give the multiprocessing module a python interpreter to run
    import multiprocessing
    executable = os.path.join(os.path.dirname(sys.executable), 'myapp.exe')
    multiprocessing.set_executable(executable)
    del executable


    class MyService(win32serviceutil.ServiceFramework):
    _svc_name_ = 'MyService'
    _svc_display_name_ = 'MyService'
    _svc_description_ = 'MyService Example Program'
    # _exe_name_ = 'pythonservice.exe' # Defaults to PythonService.exe
    # _svc_deps_ = None # sequence of service names on which this depends
    # _exe_args_ = None # Defaults to no arguments

    def SvcDoRun(self):

    # Set the current directory to the directory from
    # which this executable was launched.
    currentDir = os.path.dirname(sys.executable)
    os.chdir(currentDir)

    # Tell Windows which privileges you need, others are removed.
    set_privileges((win32security.SE_ASSIGNPRIMARYTOKEN_NAME,
    win32security.SE_CHANGE_NOTIFY_NAME,
    win32security.SE_CREATE_GLOBAL_NAME,
    win32security.SE_SHUTDOWN_NAME))

    # Implement service here, and block
    # When this method returns the service is stopped.


    if __name__ == '__main__':
    # For a service, this never gets called.
    #
    # freeze_support must be the first line
    # after the if __name__ == '__main__'
    multiprocessing.freeze_support()

    # Pass the command line to the service utility library.
    # This can handle start, stop, install, remove and other commands.
    win32serviceutil.HandleCommandLine(MyService)

    Here is the myapp.py module used to generate our executable for the multiprocessing module. This is a very simple Python script that actually does more than it needs to even for this task.


    import multiprocessing
    import os

    def main ():
    print('myapp: ', os.getpid())


    if __name__ == '__main__':
    # freeze_support must be the first line
    # after the if __name__ == '__main__'
    multiprocessing.freeze_support()
    main()

    Now all we need is a setup.py script that tells Py2exe how to build our executables. Here is the code.


    from distutils.core import setup
    import py2exe

    # We must leave the optimization level at 1 if we use the kid template
    # library. This leaves the doc strings in the library code. The kid
    # templating engine parses its doc string at run time, thus if the doc
    # string is not in the pyo file, the program crashes.

    # List of python modules to exclude from the distribution
    excludes = [
    "Tkinter",
    "doctest",
    "unittest",
    "pydoc",
    "pdb"
    ]

    # List of dll's (and apparently exe's) to exclude from the distribution
    # if any Windows system dll appears in the dist folder, add it to this
    # list.
    dll_excludes = [
    "API-MS-Win-Core-LocalRegistry-L1-1-0.dll",
    "MPR.dll",
    "MSWSOCK.DLL",
    "POWRPROF.dll",
    "profapi.dll",
    "userenv.dll",
    "w9xpopen.exe",
    "wtsapi32.dll"
    ]

    # List of python modules that are to be manually included.
    mod_includes = []

    package_includes = []

    py2exe_options = {
    "optimize": 2, # 0 (None), 1 (-O), 2 (-OO)
    "excludes": excludes,
    "dll_excludes": dll_excludes,
    "packages": package_includes,
    "xref": False,
    # bundle_files: 1|2|3
    # 1: executable and library.zip
    # 2: executable, Python DLL, library.zip
    # 3: executable, Python DLL, other DLLs and PYDs, library.zip
    "bundle_files": 3
    }

    setup(service=[{'modules': 'myservice',
    'icon_resources': [(1, 'myapp.ico')],
    }],
    console=[{'script': 'myapp.py',
    'icon_resources': [(1, 'myapp.ico')]
    }],
    version='1.0',
    options={"py2exe": py2exe_options}
    )

    I have skimmed this down a bit from an actual setup.py that I use at work. In practice I have found that Py2exe sometimes includes modules and libraries that are not necessary. The excludes list is a list of Python modules that you want to exclude from your distribution. Make sure that you know the modules listed here are not actually used. The dll_excludes option is a list of DLL and possibly EXE files that for whatever reason Py2exe is copying to your dist folder event though they may be Windows system DLLs (that you are most likely not allowed to redistribute), or the old w9xpopen.exe (for Windows 9x only). The mod_includes and package_includes options can be used to force inclusion of Python modules or packages that you must have in your distribution but for some reason Py2exe is not placing them in your dist folder. The work is done in the call to setup. Here we are telling it to build a service from the myservice module and a console application using the myapp.py module. Each of these output files uses the same icon as specified. We pass Py2Exe specific options to Py2Exe via the py2exe_options dictionary. Build the executables by running:

    python –OO setup.py py2exe

    Now that we have provided an executable for multiprocessing we can do this somewhere in our service:


    from multiprocessing import Process

    def handle_request(req):
    # do something useful
    pass

    def on_request(request):
    Process(target=handle_request, args=(request,)).start()

    The Point: In a frozen Windows service, you have to provide an executable to the multiprocessing module that can be used to run Python scripts in a new process.

    Tuesday, March 2, 2010

    Using WMI to Query Monitor Information

    I recently had a need to programmatically determine the number, make, and model of each monitor attached to a Windows computer system. Knowing that WMI provides a bewildering array of such functionality, I started here with the WMI Win32_DesktopMonitor class.

    image Figure 1: Win32_DesktopMonitor WMI class description.

    As you can see in figure 1, the information I needed is supplied by this class. I coded up a quick C# application that used this WMI class to report the monitors and thought that I was done. The problem is that each run of the program only ever showed one monitor on my system. I pulled up PythonWin and using Tim Golden's excellent WMI Python module I confirmed that WMI provides only one monitor. Searching the Internet told me that I was not the first to have this problem and in fact on Windows Vista and up (I was running on Windows 7) the Win32_DesktopMonitor class only returns one monitor in many cases. The issues has something to do with the video driver changes that were introduced in Windows Vista and improved in Windows 7. Most people that encountered this problem dropped into the Win32 C API to solve the issue. I still had hope for WMI, after all Windows knows how many monitors I have and what kind they are. Once again I used Tim Golden’s WMI Python module to poke around with various WMI classes and I found the Win32_PnPEntity class.

    image Figure 2: Win32_PnPEntity WMI class description.

    I did a quick query to dump all of these out to see if I could use this WMI class.

    import wmi
    obj = wmi.WMI().Win32_PnPEntity(ConfigManagerErrorCode=0)

    The ConfigManagerErrorCode=0 parameter means that I want all devices that are working properly. Looking at my obj variable, there was a ton of data but I noticed some devices with the string “DISPLAY\\…” within it. Here is some of the output:

    image

    I quickly filtered my output in Python with this:


    displays = [x for x in obj if 'DISPLAY' in str(x)]
    for item in displays:
    print item

    And this is the result:

    image

    This was the result that I was looking for. I have two Dell E248WFP monitors and finally WMI agreed with me. Every property is exactly the same except for the DeviceID because this is what PnP uses to distinguish between the two monitors.

    I quickly re-coded my C# function to use the WMI class Win32_PnPEntity. I did make a change however. The C# version filters devices where the Service = ‘monitor’ rather than looking for ‘DISPLAY’ in the DeviceID.

    Special Thanks to Tim Golden who makes WMI enjoyable in Python!

    Thursday, January 28, 2010

    Current Python Development Environment

    In my day job I program exclusively in the Windows environment.  I pushed hard to get Python into our development environment here.  I love Python, what a great language it is!  The problem with Python is the development tools, or as any Windows programmer will whine: “You can’t use Visual Studio!” (insert sobbing noises here).  Visual Studio spoils us, and when we can’t use it, we’re as sad as a MAC fanboi that just dropped his iPhone in the toilet.  I should say the problem with free Python development tools as I had no budget to try commercial tools.  I have heard that some of the commercial Python development tools, like Wing IDE, are quite good. But, after blowing the budget on Visual Studio, I was left to choose from amongst the free tools.

    Eclipse is almost Visual Studio.  Eclipse is a very nice IDE.  Add the PyDev tools from Aptana and it is a great Python development environment.  It supports Jython and IronPython development as well.  This environment support many features including:

    • You can configure PyLint for static analysis. 
    • Source level debugging, breakpoints, the whole bit.
    • You can configure and use multiple Python interpreters.
    • Constantly updated and improved, (thanks Fabio!)

    Grab the latest Eclipse Galileo at http://www.eclipse.org/downloads/  The PyDev home page is here http://pydev.org  Python is available here http://python.org

    SharpDevelop wants to be Visual Studio.  SharpDevelop is an open source IDE for the .NET world.  It is taking on Visual Studio directly, come one, Visual Studio is like a MacBook Pro, waaayyy overpriced! SharpDevelop supports an impressive array of languages: Boo, C#, F#, ILAsm, IronPython, and VB.NET.  SharpDevelop 3.1.1 comes with IronPython 2.6 in the box.  It also gives you a drag and drop Windows Forms editor just like Visual Studio, except that you can use Python! 

    Get SharpDevelop here http://www.icsharpcode.net/OpenSource/SD/Download/ and IronPython at this Codeplex download page

    Unfortunately I have not had the opportunity to really use IronPython for anything real.  I have written a few toy applications with it.   I plan to spend more time with SharpDevelop and IronPython in the near future.  I have read the wonderful book “IronPython in Action” by Micheal Foord and Christian Muirhead.  My first project will be to get the cherrypy web framework working under IronPython.  Right now about half of the cherrypy test suite passes, it’s a work in progress.  But that is for another blog entry.