Bryce Boe

I’m currently in Chicago, having just attended the 18th ACM Conference on Computer and Communications Security, where Adam Doupé presented our paper, Fear the EAR: Discovering and Mitigating Execution After Redirect Vulnerabilities. As always, when I’m traveling, the problem of how to connect to the Internet arises. Fortunately, we were provided with Internet access via WiFi in our rooms throughout the duration of the conference, however, now that the conference is over we’ll have to pay for the access ourselves. Hotels, like many airports, charge absurd amounts for Internet access simply because they can. $8 a day for an unreliable and slow connection is typical and is absurd considering I’m already paying $30 a month for an unlimited data plan on my cell phone. Of course I can check my email, read reddit, and maybe look up some places on yelp using my Android phone, however, it is simply more efficient to use my laptop.

Naturally, the obvious solution is to tether your laptop to your phone and if you have an Android phone the process is incredibly simple. In fact, there are a number of pages on the Internet that tell you exactly how to do it. Primarily, there is an Android Forums post from November, 2009 that recommends using an Android application, Azilink, in combination with an OS X application Tunnelblick, and a shell script to configure the tether correctly. There is also a great blog post from March of this year that is a bit more user friendly than the forum post in describing how to tether your OS X machine to your Android phone.

Despite the last guide being fairly recent, it unfortunately is slightly out of date due in part to OS X Lion. The older versions of Tunnelblick seemingly don’t work with OS X Lion, thus an update to a newer version of Tunnelblick is required. The newer version (currently 3.2beta32) of Tunnelblick breaks the shell script that the previous guide uses, thus I am writing this post simply to provide the Internet with an update to that shell script. Please note that everything else mentioned on the blog post currently works as described.

The fix to the shell script is actually really simple, just replace the line shown in the first block of code below, with the line shown in the second:
sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn --dev tun \ sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.2.1/openvpn --dev tun \

Finally for your convenience, I created a github gist with the updated shell script. Observant readers will notice that I updated some of the comments in the script. Happy tethering!

#!/bin/bash
#
# azilink for OS X Lion
# based on http://pastie.org/405289 but works with Tunnelblick only
# (no need to install a separate copy of OpenVPN2 from macports 
# or building from source by hand, thankfully)
# Requires: 
# - azilink running on android phone (http://code.google.com/p/azilink/)
#   (run the app and check the box to start the service).
# - adb on system path (comes with the Android SDK; 
#   add its tools folder to your PATH in ~/.profile or 
#   place or symlink the sdk's tools/adb file in e.g. usr/local/bin or somewhere else on the PATH)
# - Tunnelblick, a nice OS X packaging of OpenVPN (http://code.google.com/p/tunnelblick/)
#   Install Tunnelblick to Applications. Tested with Tunnelblick 3.2beta32 (build 2817)

init() {
    adb forward tcp:41927 tcp:41927
    sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.2.1/openvpn --dev tun \
                  --script-security 2\
                  --remote 127.0.0.1 41927 \
                  --proto tcp-client \
                  --ifconfig 192.168.56.2 192.168.56.1 \
                  --route 0.0.0.0 128.0.0.0 \
                  --route 128.0.0.0 128.0.0.0 \
                  --keepalive 10 30 \
                  --up "$0 up" \
                  --down "$0 down"
}


up() {
    tun_dev=$1
    ns=192.168.56.1
    sudo /usr/sbin/scutil << EOF
open
d.init
get State:/Network/Interface/$tun_dev/IPv4
d.add InterfaceName $tun_dev
set State:/Network/Service/openvpn-$tun_dev/IPv4

d.init
d.add ServerAddresses * $ns
set State:/Network/Service/openvpn-$tun_dev/DNS
quit
EOF
}


down() {
    tun_dev=$1
    sudo /usr/sbin/scutil << EOF
open
remove State:/Network/Service/openvpn-$tun_dev/IPv4
remove State:/Network/Service/openvpn-$tun_dev/DNS
quit
EOF
}


case $1 in
    up  ) up $2 ;;  # openvpn will pass tun/tap dev as $2
    down) down $2 ;;
    *   ) init ;;
esac

Yesterday, October 7, 2011, the graduate students of UCSB’s Computer Science department, including myself, hosted the 6th annual Graduate Student Workshop on Computing (GSWC). The workshop is a great opportunity for other students, faculty, and industry professionals to get an overview of the work performed by our department. Part of organizing the workshop is obtaining gift or sponsorship money in order to pay for the facilities, food, proceedings, and all other costs associated with running a workshop.

In exchange for event sponsorship, we include each company’s logo on our website, in the conference proceedings, and in a visible display at the workshop itself. For many of the previous GSWCs, we created and printed up an expensive poster with the theme of the event as well as logos of all the corporate sponsors. While that worked well, it required a decent effort to design the poster, and, as I said before, was considerably expensive to print. Thus, when I was the chair of the GSWC last year, I decided to take a different approach to how we display the corporate logos during the workshop. Rather than using a static poster, I created an animated display that was projected on one of the room’s walls during the conference.

I occasionally provide consulting for a local Santa Barbara company, Worldviz who makes a product, Vizard, that allows one to quickly construct 3D environments in python. Using Vizard I was able to quickly create a 3D logo display in which the logos follow a circular path along the z-axis, as shown in the following photos. While this worked well, Vizard unfortunately only works on Windows and thus could not be run easily from my Mac laptop.

We wanted to use the same display for this year’s GSWC so I sought to rewrite the display in a cross platform and free manor. While I have some previous opengl experience in C, I really wanted to write the display in python so I looked at the various python opengl options. From the StackOverflow thread, OpenGL with Python, I quickly settled on pyglet. Since I already knew the equation to move points around in a circular path, my only challenge was to figure out how to map a texture to a quad so that I could position the quad in 3D space. I quickly asked a StackOverflow question, Moving an image around in 3D space in hopes that while I was in the process of figuring it out, someone would provide me with the solution. Unfortunately, crowdsourcing didn’t pay off, nevertheless, I eventually found the solution and wrote a simple cross platform logo animation program.

In order to run the following program, you will first need to install the required libraries (an exercise left to the reader), create a folder “imgs” and place whatever images you want in that folder and then run the following script (download animator.py). Note: If you want to run this script on 64-bit OS X you’ll need to run it via: VERSIONER_PYTHON_PREFER_32_BIT=yes ./animator.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

#!/usr/bin/env python
import math, os, pyglet, sys
from pyglet.gl import *
 
class World(pyglet.window.Window):
    def __init__(self, scale=10, center_pos=(0, 0, -15), speed=1.0,
                 *args, **kwargs):
        super(World, self).__init__(*args, **kwargs)
        self.scale = scale
        self.center_pos = center_pos
        self.speed = speed
        glClearColor(1.0, 1.0, 1.0, 0.0)
        glEnable(GL_DEPTH_TEST)
        self.textures = self.load_textures()
        self.clock = 0
        pyglet.clock.schedule_interval(self.update, 1 / 60.0)
 
    @staticmethod
    def load_textures():
        img_dir = 'imgs'
        textures = []
        if not os.path.isdir(img_dir):
            print 'Could not find directory "%s" under "%s"' % (img_dir,
                                                                os.getcwd())
            sys.exit(1)
        for image in os.listdir(img_dir):
            try:
                image = pyglet.image.load(os.path.join(img_dir, image))
            except pyglet.image.codecs.dds.DDSException:
                print '"%s" is not a valid image file' % image
                continue
            textures.append(image.get_texture())
 
            glEnable(textures[-1].target)
            glBindTexture(textures[-1].target, textures[-1].id)
            glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, image.width, image.height,
                         0, GL_RGBA, GL_UNSIGNED_BYTE,
                         image.get_image_data().get_data('RGBA',
                                                         image.width * 4))
        if len(textures) == 0:
            print 'Found no textures to load. Exiting'
            sys.exit(0)
        return textures
 
    def update(self, _):
        self.on_draw()
        self.clock += .01
 
    def on_draw(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()
        self.draw_images()
 
    def draw_images(self):
        angle_base = (self.clock * self.speed * 50) % 360
        angle_delta = 360. / len(self.textures)
 
        for i, texture in enumerate(self.textures):
            angle = math.radians((angle_base + i * angle_delta) % 360)
            dx = math.sin(angle) * self.scale
            dz = math.cos(angle) * self.scale
 
            if texture.width > texture.height:
                rect_w = texture.width / float(texture.height)
                rect_h = 1
            else:
                rect_w = 1
                rect_h = texture.height / float(texture.width)
 
            glPushMatrix()
            glTranslatef(dx + self.center_pos[0], self.center_pos[1],
                         dz + self.center_pos[2])
            glBindTexture(texture.target, texture.id)
            glBegin(GL_QUADS)
            glTexCoord2f(0.0, 0.0); glVertex3f(-rect_w, -rect_h, 0.0)
            glTexCoord2f(1.0, 0.0); glVertex3f( rect_w, -rect_h, 0.0)
            glTexCoord2f(1.0, 1.0); glVertex3f( rect_w,  rect_h, 0.0)
            glTexCoord2f(0.0, 1.0); glVertex3f(-rect_w,  rect_h, 0.0)
            glEnd()
            glPopMatrix()
 
    def on_resize(self, width, height):
        glViewport(0, 0, width, height)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(65.0, width / float(height), 0.1, 1000.0)
        glMatrixMode(GL_MODELVIEW)
 
 
if __name__ == "__main__":
    window = World(width=800, height=600)
    pyglet.app.run()

For the third year, I competed with team Shellphish in the Defcon quals. We pulled through with some amazing points at the end to finish in 8th place. My successful contributions, however, were really only with respect to Forensics 100 and 300. My write up for the following are below:

Forensics 100
The forensics 100 challenge indicated to find the key, and provided a png file that was 19025×1 in resolution. Immediately our team thought we could simply change the resolution to 25×761 and would be on to something. After working with the resulting image for sometime I finally thought about converting it to 761×25. That was our first break through when we read some text along the lines of “ILoveMeSomesheepysheepies” followed by binary that includes capital ‘O’s in place of some of the ’0′s. After no success with different permutations of that message we incorporated an idea the other team members had about the blue offset pixels that occur at regular intervals. Our first attempt at wrapping the image at the blue pixel boundaries (every 450 pixels) resulted in success! The key “thankYouSirPleasemayIhaveAnother” appeared and worked. The following is my simple python solution for Forensics 100:

1
2
3
4
5
6
7
8
9
10
11

#!/usr/bin/env python
import sys, Image
 
def main():
    orig = Image.open('f100.png')
    img = Image.new('RGBA', (450, 43))
    img.putdata(orig.getdata())
    img.show()
 
if __name__ == '__main__':
    sys.exit(main())

Forensics 300
Forensics 300 was quite an interesting challenge. I don’t have the original file, nevertheless, one had to extract the initial file with a password to get a dmg containing a dump from an iphone. I came into the challenge a little late, after one of my teammates had gone through all the images, videos, and audio files looking for Waldo and ‘grep’ing for various relevant keywords. Further more, my teammates had previously used the iPhoneTracker on the consolidated.db file to see where the phone had been, however San Antonio didn’t prove to be very useful.

While the iPhoneTracker app seemed pretty cool, I wanted to programmatically see where the phone had been the most. Thus, after figuring out what was what with respect to the consolidated.db file I wrote a little python script to find the most visited places rounded to less precision to account for some variance. The top three results were the following where the first number represents the number of occurrences in that location, and the two numbers between the parenthesis represent the latitude and longitude respectfully.

• 30 (‘-77.846′, ’166.677′)
• 18 (’0.000′, ’0.000′)
• 10 (’36.106′, ‘-115.173′)

When I did a google search for the coordinates -77.846 166.667 I knew immediately that it was no coincidence that I was centered in a small town in Antarctica. Unfortunately, Google maps doesn’t have a name for this location so I had to revert to Bing (for the first time ever) to figure out that this location is called Ross Island. From that point we simply attempted different “places” listed Ross Island’s wikpiedia page until “McMurdo Station” submitted successfully. Below is the script I used to find the coordinates from the consolodated.db input file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#!/usr/bin/env python
import os, sys
 
def main():
    os.system('sqlite3 consolidated.db "select Latitude, Longitude '
              'from CellLocation;" > tmp')
 
    uniq = {}
    for line in open('tmp'):
        pos = tuple('%.3f' % float(x) for x in line.split('|')[:2])
        if pos in uniq:
            uniq[pos] += 1
        else:
            uniq[pos] = 1
 
    for pos, count in sorted(uniq.items(), key=lambda x:x[1]):
        print count, pos
 
if __name__ == '__main__':
    sys.exit(main())

You can find links to solutions to other Defcon 19 Quals challenges at the following locations: Rogunix, negative foo, VNSecurity site.

Last night Adam Doupe wrote up his description on our Execution After Redirect Vulnerability which I wanted to link my followers to. Adam’s primary focus on this project has been adapting a static ruby analyzer to find instances of the EAR vulnerability in thousands of Ruby-On-Rails projects from github. It’s rather exciting.

In other news I was one of four recipients of the 2010-2011 UCSB Academic Senate Outstanding Teaching Assistant Award. Today there was an awards ceremony where I received my first pre-framed certificate. I am very honored to have won this award especially given that I am the first student in the Computer Science Department to win it. It’s a very cool feeling, however, as Nichole put it, with great acknowledgement come greater expectations. I hope to exceed those expectations.

Update 2011/02/23 11:02 PST
Added the lift tag and updated the list.

Update 2011/02/22 13:19 PST
Added the jsf tag (java server faces) and updated the total question count for each item on the list.

Update 2011/02/22 11:14 PST
Adding spring-mvc as that was what I originally was originally supposed to have.

Update 2011/02/22 10:36 PST
For the interested, here is the table used to generate the graph.

Update 2011/02/22 10:05 PST
As per comments on this post, I updated the list by removing hibernate, spring, and sass and added gwt and grails. I also updated the chart reflecting this information, and created an additional chart which plots the frameworks as a percentage of the questions asked each week to hide stackoverflows’s growing popularity.

Adam and I are currently in the process of working on our research about the Execution After Redirect, or EAR, Vulnerability which I previously discussed in my blog post about the 2010 iCTF. While Adam is working on a static analyzer to detect EARs in ruby on rails projects, I am testing how simple it is for a developer to introduce an EAR vulnerability in several popular web frameworks. In order to do that, I first needed to come up with a mostly unbiased list of popular web frameworks.

My first thought was to perform a search on the top web frameworks hoping that the information I seek may already be available. This search provided a few interesting results, such as the site, Best Web-Frameworks as well as the page Framework Usage Statistics by the group BuiltWith. The Best Web-Frameworks page lists and compares various web frameworks by language, however it offers no means to compare the adoption of each. The Framework Usage Statistics page caught my eye as its usage statistics are generated by crawling and fingerprinting various websites in order to determine what frameworks are in use. Their fingerprinting technique, however, is too generic in some cases thus resulting in the labeling of languages like php and perl as frameworks. While these results were a step in the right direction, what I was really hoping to find was a list of top web frameworks that follow the model, view, controller, or MVC, architecture.

After a bit more consideration I realized it wouldn’t be very simple to get a list of frameworks by usage, thus I had to consider alternative metrics. I thought how I could measure the popularity of the framework by either the number of developers using or at least interested in the framework. It was this train of thought that lead me to both Google Trends and StackOverflow. Google Trends allows one to perform a direct comparison of various search queries over time, such as ruby on rails compared to python. The problem, as evidenced by the former link, is that some of the search queries don’t directly apply to the web framework; in this case not all the people searching for django are looking for the web framework. Because of this problem, I decided a more direct approach was needed.

StackOverflow is a website geared towards developers where they can go to ask questions about various programing languages, development environments, algorithms, and, yes, even web frameworks. When someone asks a question, they can add tags to the question to help guide it to the right community. Thus if I had a question about redirects in ruby on rails, I might add the tag ruby-on-rails. Furthermore if I was interested in questions other people had about ruby on rails I might follow the ruby-on-rails tag.

Between the number of questions per tag, the number of answers per tag, and the number of followers per tag, StackOverflow provides a few metrics for measuring the relative level of developer interest in various web frameworks. Success! The next step was to extract these numbers for the tags of various frameworks. For this, I attempted to find StackOverflow tags corresponding to all the frameworks listed on the Best Web-Frameworks site I previously found. I skipped the framework languages CSS and Javascript as they aren’t server side frameworks. I then narrowed the list down to the frameworks which had at least 100 questions asked.

This produced the following frameworks sorted by total number of questions asked:

1. (31156) ruby-on-rails
2. (20587) asp.net-mvc
3. (14951) django
4. (4726) zend-framework
5. (3510) jsf
6. (3336) gwt
7. (3296) cakephp
8. (3127) codeigniter
9. (2731) grails
10. (1976) spring-mvc
11. (1603) symfony
12. (912) struts
13. (538) kohana
14. (515) pylons
15. (514) sinatra
16. (506) dotnetnuke
17. (420) wicket
18. (227) lift
19. (194) yii
20. (163) cherrypy
21. (126) web2py
22. (106) catalyst

• (6609) hibernate (note: not a web framework)
• (5765) spring (note: not a web framework)
• (178) sass (note: not a web framework)

This list alone seems to work fairly well, however, I wanted to take it one step further which was to see the number of questions asked on a per week basis since the start of StackOverflow. Using the StackOverflow API (I used the API to generate the previous list too) I wrote a script to generate a CSV file containing this information. The information is depicted in the interactive chart below for the top 10 frameworks according to total number of StackOverflow questions. Each point in the chart represents the number of questions asked in a one week period starting on the date of the data point (protip: hover over chart to get the exact values).

Note: if the above graph doesn’t load, try this static image.

The data confirms my previous suspicion that ruby on rails is the number one MVC and that django and cakePHP would also appear in the top 10. I must admit that I had never before heard of asp.net MVC, however considering that stackoverflow and all other stackexchange sites run on asp.net MVC, it makes sense that it would rank quite high.

I added the below chart to show the relative percentage of questions per tag over time as per Big Dave’s Gusset’s comment. This hides the growing popularity of stackoverflow.


(Interactive version)

The data for the above chart was extracted using the following script. The script requires the python package py-stackexchange in order to run and can be easily modified to add additional tags or change the filtering methods.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

#!/usr/bin/env python import datetime, sys, time from stackexchange import Site, StackOverflow   frameworks = [# php 'zend-framework', 'cakephp', 'symfony', 'codeigniter', 'seagull', 'prado', 'solar', 'ezcomponents', 'kohana', 'jelix', 'flow3', 'modx', 'sapphire', 'yii', 'limonade', 'tekuna', 'doophp', 'fat-free', 'akelos', 'php-on-trax', 'atk', # ruby 'ruby-on-rails', 'merb', 'ramaze', 'halcyon', 'sinatra', 'webby', 'sass', # perl 'catalyst', 'interchange', 'mason', 'cgi-application', 'jifty', 'gantry', 'dancer', 'mojolicious', # java 'struts', 'hibernate', 'spring', 'wicket', 'play', 'stripes', # python 'django', 'pylons', 'grok', 'turbogears', 'web2py', 'cherrypy', # coldfusion 'cfwheels', 'coldspring', 'model-glue', # asp.net 'asp.net-mvc', 'dotnetnuke', 'monorail', 'vici']   class TagStats(object): DATE_START = 1217540572 WEEK_SECONDS = 604800 def __init__(self, tag_names): self.so = Site(StackOverflow, 'LzYJwh19o0WCIvXK9q6k6g') self.tag_names = tag_names self.tags = [] self.stats = {}   def output_counts(self, html=False): tmp = [] for tag in sorted(self.tags, key=lambda x:x.count, reverse=True): tmp.append((tag.count, tag.name)) if html: print '<ol>' for count, name in tmp: print ('<li>(%d) <a href="http://stackoverflow.com/tags/%s">' '%s</a></li>') % (count, name, name) print '</ol>' else: print '\n'.join(['%8d %s' % x for x in tmp])   def get_tags(self, min_size): for name in self.tag_names: query = self.so.tags(filter=name) for tmp in query: if name == tmp.name: break else: sys.stderr.write('Not found: %s\n' % name) continue if tmp.count < min_size: sys.stderr.write('Too few questions: %s\n' % name) else: self.tags.append(tmp) self.stats = dict(zip([tag.name for tag in self.tags], [[]]*len(self.tags)))   def output_stats_by_week(self, start_week=0): now = int(datetime.datetime.now().strftime('%s')) num_weeks = (now - self.DATE_START) / self.WEEK_SECONDS print ', '.join(tag.name for tag in self.tags) for i in range(start_week, num_weeks): sys.stdout.flush() start = self.DATE_START + i * self.WEEK_SECONDS end = self.DATE_START + (i + 1) * self.WEEK_SECONDS counts = [] for tag in self.tags: try: count = self.so.questions(tagged=str(tag.name), fromdate=start, todate=end).total except Exception, e: sys.stderr.write('Stopped at week %d\n' % i) sys.exit(1) self.stats[tag.name].append(count) counts.append(str(count)) print ', '.join(counts)     def main(): try: start_week = int(sys.argv[1]) except IndexError: start_week = 0 tag_stats = TagStats(frameworks) tag_stats.get_tags(100) #tag_stats.output_counts(html=True) tag_stats.output_stats_by_week(start_week)   if __name__ == '__main__': sys.exit(main())

Happy web-framework coding!

• Share this:
• Reddit
• Share
• Email
• Print
• Facebook
• Digg
• StumbleUpon

The Usenix security deadline is quickly approaching, and that means finalizing everything on my research project. Therefore, today I wanted to quickly parallelize some of my analysis code to take advantage of the eight virtual processors on my machine. I previously wrote about python multiprocessing and keyboard interrupts, so the task of converting my code seemed pretty trivial. Unfortunately, my analysis code deals with large amounts of data which when passed to processes using the multiprocessing queue class, produced unexpected results. Thus the sample I originally used in my previous post, isn’t as robust as I hoped. It will soon be updated. The post has been updated with a reference to this post.

The issue is that when pushing large items onto the queue, the items are essentially buffered, despite the immediate return of the queue’s put function [endnote 1]. Therefore, the queue may have not yet completely added an item by the time a subsequent non-blocking call to the queue’s get function is made. This delay explains why such a non-blocking call to queue.get may return a Queue.Empty error and is the exact problem I encountered in my analysis code.

As usual, I have provided some code demonstrating the problem and a solution.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

#!/usr/bin/env python
import multiprocessing, Queue
 
def worker_bad(jobs):
    while True:
        try:
            tmp = jobs.get(block=False)
        except Queue.Empty:
            break
 
def worker_good(jobs):
    while True:
        tmp = jobs.get()
        if tmp == None:
            break
 
def main(items, size, num_workers, good_test):
    if good_test:
        func = worker_good
    else:
        func = worker_bad
    jobs = multiprocessing.Queue()
    for i in range(items):
        jobs.put(range(size))
    workers = []
    for i in range(num_workers):
        if good_test:
            jobs.put(None)
        tmp = multiprocessing.Process(target=func, args=(jobs,))
        tmp.start()
        workers.append(tmp)
    for worker in workers:
        worker.join()
    return jobs.empty()
 
if __name__ == '__main__':
    workers = 4
    items = workers * 2
    size = 10000
 
    for good_test in [False, True]:
        passed = 0
        for i in range(100):
            passed += main(items, size, workers, good_test=good_test)
        print '%d%% passed (Good Test: %s)' % (passed, good_test)

This code provides a simple test framework to compare two methods for completing tasks in a job queue. The actual workers are virtually identical as they simply continue to pull items off the job queue, until there are no more jobs. An individual test is considered passed when the number of jobs remaining after the workers terminate is zero. When running this script, you should notice that the first group has fewer than 100% of the tests pass, and more importantly that the second group has a 100% pass rate. If your output shows the first with a 100% pass rate, try doubling or further increasing the value of the variable size on line 39. Additionally you may notice I/O errors. These errors likely represent a bug in the multiprocessing queue and further demonstrate that a call to queue.get immediately after queue.put will not always succeed.

I arrived at the solution to this problem, in the response to a slightly similar stackoverflow question titled, Dumping a multiprocessing.Queue into a list. The solution recommended by the stackoverflow response is to enqueue a sentinel after all data in order to clearly delineate the end of the queue rather than utilizing the empty state of the queue. However, a single sentinel will not work in a case where there are multiple worker processes as only one worker can receive the sentinel. Thus the intuitive solution is to enqueue as many sentinels as there are workers on the queue following the data. I chose to represent these sentinels by the value None shown on line 28.

Now I can get back to speeding up my data analysis. Happy coding!

Endnote 1: Observant readers might notice that the queue’s put function has a block argument, however that argument, true by default, is for cases when the queue has a maximum size.

Each year the Security Lab at UCSB hosts the International Capture the Flag competition, an approximately eight-hour security competition pitting security groups at various universities around the world against each other. Last year I had the privilege of contributing significantly to the setup on the iCTF, and later publishing and presenting a paper, “Organizing Large Scale Hacking Competitions” at DIMVA 2010. This year, I finally had an opportunity to take Giovanni Vigna’s security course, thus allowing me to participate in the competition. I led the team, “Tr0llF4ce Pwns You”, and we did decently well considering this was the first hacking competition for many on our team.

Despite my participation, I was still able to contribute a challenge to the competition, which ended up being the 800 point challenge 6 in the competition. The primary reason for the challenge was to draw attention to a previously unpublicized web vulnerability that we are calling the Execution After Redirect, or EAR Vulnerability. This vulnerability is exactly as the name states, however to be a bit more precise our exact definition of the vulnerability is, “code that executes after the developer’s intended termination point”. The developer’s intended termination point is often indicated by a server-initiated redirect, or more precisely an HTTP 301, 302, 303, or 307 status code along with an HTTP Location header. It is important to note that there are cases in which the developer intends server side code to continue executing following a redirect. Such cases are not EAR vulnerabilities.

Adam Doupé and I are in the process of writing a paper describing the complete details of the EAR vulnerability. For the purpose of understanding this challenge, you need only know that when a modern browser receives the redirect, any data sent in conjunction with the server-initiated redirect will not be displayed as the browser automatically fetches the resource indicated by the redirect. Furthermore, tools such as wget, curl and python’s urllib, among others, also automatically handle the redirect in their default configuration thus making the EAR vulnerability that much more difficult to detect.

With that said, the following is a complete walkthrough of solving my iCTF 2010 challege:

Teams were presented with the message, “Obey the error messages and find the secret at http://10.15.3.1:8000.” Upon visiting the URL teams would see the title, “User Administration” this video playing immediately, and a simple file upload submission form. The video was mostly an annoying red herring, however, some of the keywords in the video happened to be valid user names that could be beneficial. The submission form contained an upload field called control and a hidden field called csrf whose value was a randomly generated integer valid for 30 seconds that could only be used once to indicate a valid form

The first real part of the challenge required discovering the control file format. This part was not meant to be difficult, thus the server intentionally leaked a plethora of information through error messages that were delivered to the user via a cookie along with a server-initiated redirect to the same page. When a GET request was issued containing the error message cookie, the error message was simply displayed to the user on the page. While the code to handle the error message did contain an XSS vulnerability, it was irrelevant to the challenge. Sending error messages through cookies followed by redirects was my hint to teams to investigate what else was sent in the first server response.

The specific error messages allowed teams to quickly discover the file had to be three lines and less than 40 characters where the first line was for the username, and the second for the password. At this point, teams had to guess usernames and passwords, however this process was pretty trivial due to error messages indicating that usernames could be at most 6 characters a-z, and the password could be at most two numerical digits. The passwords could be easily brute forced, however the username space was 27⁶-1 in size which is not feasible to brute force. Thus, I populated the database with many guessable usernames such as, ‘user’, ‘ictf’, ‘a’, ‘admin’, ‘root’, ‘dev’, ‘test’, as well as some keywords from the annoying video. Further error messages indicated invalid usernames, incorrect passwords, inactive accounts, and non-administrator accounts allowing teams to guess valid username and password combinations in a minimal amount of time.

The service contained two types of users: administrators, and regular users. Both types of users could be either in an active or inactive state. In this challenge, all guessable administrator accounts were inactive, and all regular users were active. The second part of the challenge was to discover the Execution After Redirect Vulnerability. More specifically, teams needed to discover that upon logging in with a regular user account the administrator console view was sent in the response along with the redirect and cookie containing the error message stating that regular users could not send commands. As previously mentioned, the detection of the EAR vulnerability is not trivial thus this process required teams to do something special in order to view the raw response.

Once the vulnerability was discovered, teams learned, via an error message, that they could send the command “help” to list all the administrator commands, thus informing them about the other commands, “add”, “info”, and “list”. The add command simply exposed the information that the SQLITE database was read only, the info command listed the info field for the specified user, and the list command listed all the active users in the system along with their info field. The teams could use the info command to discover the special admin account that allowed them to successfully view the administrator console without exploiting the EAR vulnerability. The list command also told the teams that there were five disabled users.

The third and final part of the challenge was to discover and exploit the SQL injection vulnerability in the info command. Two caveats of this part were that the SQL injection vulnerability had to be performed without standard whitespace, and the teams had to get the info field for the user secret. Neither of these proved difficult for the teams that made it this far.

During the competition, 69 of the 72 teams attempted to solve my challenge, of which 44 teams were able to submit valid control files. 34 of those teams, successfully guessed regular user accounts, thus exposing them to the Execution After Redirect Vulnerability. However, as indicated by the teams who successfully ran the “help” command, of these 34 teams, only 12 of them discovered the EAR vulnerability. Finally of those 12 teams, 7 successfully exploited the SQL injection vulnerability that provided them with:
The secret is: http://hackiswack.com/videos/viewvideo/23/hack-is-wack/blocka-blocka

The 8 successful control files sent over the course of the competition were:
a\n50\ninfo 'OR(1)LIMIT/**/5,1--\n
zzyzx\n83\ninfo '/**/OR/**/pass<83;/*\n
a\n50\ninfo 0'/**/or/**/pass='66\n
zzyzx\n83\ninfo root'OR'1'='1'LIMIT'5','1\n
user\n50\ninfo ab'or/**/oid=6;--\n
a\n50\ninfo '/**/or/**/oid=6;--\n
zzyzx\n83\ninfo '/**/or/**/user/**/like's%
zzyzx\n83\ninfo 'OR(user='secret')--\n

My own control file used in testing:
a\n50\ninfo z'or"user"='secret

I think the fact that just over one third of the teams in our hacking competition discovered this vulnerability shows how dangerous it can be, and hence the title, Fear the EAR.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

        conn = sqlite3.connect(DATABASE)
        try:
            c = conn.execute('select * from users where user=?', (user,))
            result = c.fetchone()
	    if not result:
		return self.send_redirect(S['nonexist'] % user)
            _, user_pswd, info, is_admin, is_active = result
            if pswd != user_pswd:
                return self.send_redirect(S['mismatch'] % user)
            if not is_active:
                return self.send_redirect(S['inactive'] % user)
            if not is_admin:
                self.send_redirect(S['admin'] % user)
            self.process_command(conn, user, *cmd.split(' '))
        finally:
            conn.close()

You can find the complete source code for this challenge here. It simply requires running a single python file. The bug in the code that produces the vulnerability occurs on line 413 (line 13 above) in which I neglected to return from the function when I called send_redirect as in all other instances.

If you worked on this challenge in the competition let me know what you thought. As always I appreciate all feedback, and challenge related feedback will help me create better challenges for next year’s iCTF.

Recently I’ve done a lot of work requiring heavy computation on large datasets. While python is not a great choice for speed, it can be extended by modules written in C for those speed critical moments. For such moments I always try to find solutions written as C modules. This approach works very well save for one major caveat that seems to be common across many of the existing C modules. This caveat has to do with the nonexistent handling of SIGINT, or the KeyboardInterrupt exception, within the C module.

On linux, the SIGINT interrupt occurs when the user wishes to cancel a command by pressing ctrl+c. Python properly receives this interrupt and internally converts it to a KeyboardInterrupt exception, thus in many cases it’s not necessary for the C module to handle SIGINT itself. However, in cases where the C module function requires a significant amount of time, python will not be able to handle the interrupt until the function call returns. Thus if you are writing a C module with long running functions I suggest you implement some sort of the following. One thing I want to mention: while this should serve as a good example for those who want to write a python C module, please do not think of this as a tutorial.

Let’s start with a simple C program that computes the nth Fibonacci number using naïve recursion. This approach has an exponential running time and therefore is perfect for demonstrating the want to end the process. You’ll notice in the code below that I have already added the SIGINT interrupt handler. This handler normally would be redundant as C will terminate by default when it receives SIGINT, however I added a simple print statement in the interrupt handler to distinguish it from the default operation. I acknowledge that it is a bad practice to use printf in a signal handler, however I am neglecting that concern for demonstration purposes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#include <signal.h>
#include <stdio.h>
 
volatile sig_atomic_t kb_interrupt = 0;
 
void kb_interrupt_handler(int sig) {
  printf("Received kb interrupt\n");
  kb_interrupt = 1;
}
 
int fibo(int n) {
  int a, b;
  if (kb_interrupt) return -1;
  else if (n < 2) return n;
  else if ((a = fibo(n - 1)) < 0) return -1;
  else if ((b = fibo(n - 2)) < 0) return -1;
  else return a + b;
}
 
int main() {
  int n;
  signal(SIGINT, kb_interrupt_handler);
  printf("Number: ");
  fflush(stdout);
  scanf("%d", &n);
  printf("Fibo(%d) = %d\n", n, fibo(n));
}

Now let’s convert this to something that python can use. In the code below, you’ll notice I eliminated the main function and the include statements as they are no longer needed. All code that was added was needed to call the function fibo from the python C module bboe.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

#include <Python.h>
 
volatile sig_atomic_t kb_interrupt = 0;
 
void kb_interrupt_handler(int sig) {
  printf("Received kb interrupt\n");
  kb_interrupt = 1;
}
 
int fibo(int n) {
  int a, b;
  if (kb_interrupt) return -1;
  else if (n < 2) return n;
  else if ((a = fibo(n - 1)) < 0) return -1;
  else if ((b = fibo(n - 2)) < 0) return -1;
  else return a + b;
}
 
PyObject *bboe_fibo(PyObject *self, PyObject *args) {
  __sighandler_t prev;
  int n, result;
  if (!PyArg_ParseTuple(args, "i", &n))
    return NULL;
  prev = signal(SIGINT, kb_interrupt_handler);
  result = fibo(n);
  signal(SIGINT, prev);
  if (result < 0) {
    PyErr_SetObject(PyExc_KeyboardInterrupt, NULL);
    return NULL;
  }
  return Py_BuildValue("i", result);
}
 
PyMethodDef bboe_methods[] = {
  {"fibo", bboe_fibo, METH_VARARGS, "Compute the nth Fibonacci number."},
  {NULL, NULL, 0, NULL}
};
 
PyMODINIT_FUNC initbboe(void) {
  Py_InitModule("bboe", bboe_methods);
}

The proper distutils setup.py script for this C module is shown below.

1
2
3
4
5

from distutils.core import setup, Extension
 
setup(name = 'BBoe',
      version = '1.0',
      ext_modules = [Extension('bboe', ['bboemodule.c'])])

Finally to reproduce the functionality of the C program I’ve written the following. While you can copy and paste this code, you can also grab this tarball that contains all the code listed here, as well as a Makefile which builds both the C and python parts. Simply run make followed by ./fibo_prog for the C version, or ./fibo_prog.py for the python version.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#!/usr/bin/env python
import bboe
 
def main():
    try:
        n = int(raw_input('Number: '))
    except ValueError:
        n = 0
    try:
        res = bboe.fibo(n)
    except KeyboardInterrupt:
        res = -1
    print 'Fibo(%d) = %d' % (n, res)
 
if __name__ == '__main__':
    main()

One last thing I want to mention is to ensure the signal handler is only changed in your C module when required. In my first iteration of this code, I had my call to signal in the initbboe method which had the negative effect of using my kb_interrupt_handler whenever the bboe module was loaded. The fix was to only have my signal handler in place whilst code from my module was running. Lines 24 and 26 accomplish this properly.

Happy python C module coding!

Edit 2010-09-15
I wanted to test the speed between a pure C implementation, a python C module implementation, and a pure python implementation. Thus I have updated the source tarball to include the following python file as well as a script to generate the timing results which are also shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#!/usr/bin/env python
 
def fibo(n):
    if n < 2: return n
    return fibo(n - 1) + fibo(n - 2)    
 
def main():
    try:
        n = int(raw_input('Number: '))
    except ValueError:
        n = 0
    try:
        res = fibo(n)
    except KeyboardInterrupt:
        res = -1
    print 'Fibo(%d) = %d' % (n, res)
 
if __name__ == '__main__':
    main()

fibo(00) pure c: 0.002s c module: 0.016s pure python: 00.015s
fibo(02) pure c: 0.002s c module: 0.014s pure python: 00.015s
fibo(04) pure c: 0.002s c module: 0.014s pure python: 00.014s
fibo(06) pure c: 0.002s c module: 0.015s pure python: 00.015s
fibo(08) pure c: 0.002s c module: 0.015s pure python: 00.013s
fibo(10) pure c: 0.002s c module: 0.017s pure python: 00.016s
fibo(12) pure c: 0.002s c module: 0.013s pure python: 00.014s
fibo(14) pure c: 0.002s c module: 0.015s pure python: 00.016s
fibo(16) pure c: 0.002s c module: 0.014s pure python: 00.015s
fibo(18) pure c: 0.002s c module: 0.014s pure python: 00.017s
fibo(20) pure c: 0.002s c module: 0.014s pure python: 00.018s
fibo(22) pure c: 0.002s c module: 0.012s pure python: 00.021s
fibo(24) pure c: 0.003s c module: 0.013s pure python: 00.037s
fibo(26) pure c: 0.004s c module: 0.017s pure python: 00.074s
fibo(28) pure c: 0.008s c module: 0.020s pure python: 00.167s
fibo(30) pure c: 0.018s c module: 0.027s pure python: 00.406s
fibo(32) pure c: 0.034s c module: 0.040s pure python: 01.061s
fibo(34) pure c: 0.067s c module: 0.080s pure python: 02.716s
fibo(36) pure c: 0.167s c module: 0.183s pure python: 07.121s
fibo(38) pure c: 0.425s c module: 0.450s pure python: 18.722s
fibo(40) pure c: 1.099s c module: 1.153s pure python: 49.901s


These results show first that there is much to be gained by writing a C module for CPU intensive tasks, and second that a pure C implementation doesn’t gain you much in this particular case.

VirusTotal is a web service that essentially performs a virus scan of an uploaded file, or url against many of the top virus scanners (see full list). I recently needed to submit over 100 binaries to VirusTotal, and being a computer scientist I knew this task, like many other things I do, could be perfectly automated. I was thrilled to see that VirusTotal provides both a simple API as well as some python code examples demonstrating the file submission and report checking process.

Two days ago (2010/09/30) I attempted to run their file upload and scan example when I encountered some server errors. I quickly contacted VirusTotal via email to which I received a reply from Emiliano who informed me he corrected the issues on their end. Thanks Emiliano! Meanwhile I wrote a fairly simple, self contained python script which retrieves a VirusTotal report for a given binary, uploading the file if necessary. The following script is loosely based off the API examples, and contains modified code from this snippet that was already required to run the scan file example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

#!/usr/bin/env python
import hashlib, httplib, mimetypes, os, pprint, simplejson, sys, urlparse
 
DEFAULT_TYPE = 'application/octet-stream'
 
REPORT_URL = 'https://www.virustotal.com/api/get_file_report.json'
SCAN_URL = 'https://www.virustotal.com/api/scan_file.json'
 
API_KEY = 'YOUR KEY HERE'
 
# The following function is modified from the snippet at:
# http://code.activestate.com/recipes/146306/
def encode_multipart_formdata(fields, files=()):
    """
    fields is a dictionary of name to value for regular form fields.
    files is a sequence of (name, filename, value) elements for data to be
    uploaded as files.
    Return (content_type, body) ready for httplib.HTTP instance
    """
    BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
    CRLF = '\r\n'
    L = []
    for key, value in fields.items():
        L.append('--' + BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"' % key)
        L.append('')
        L.append(value)
    for (key, filename, value) in files:
        L.append('--' + BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"; filename="%s"' %
                 (key, filename))
        content_type = mimetypes.guess_type(filename)[0] or DEFAULT_TYPE
        L.append('Content-Type: %s' % content_type)
        L.append('')
        L.append(value)
    L.append('--' + BOUNDARY + '--')
    L.append('')
    body = CRLF.join(L)
    content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
    return content_type, body
 
def post_multipart(url, fields, files=()):
    """
    url is the full to send the post request to.
    fields is a dictionary of name to value for regular form fields.
    files is a sequence of (name, filename, value) elements for data to be
    uploaded as files.
    Return body of http response.
    """
    content_type, data = encode_multipart_formdata(fields, files)
    url_parts = urlparse.urlparse(url)
    if url_parts.scheme == 'http':
        h = httplib.HTTPConnection(url_parts.netloc)
    elif url_parts.scheme == 'https':
        h = httplib.HTTPSConnection(url_parts.netloc)
    else:
        raise Exception('Unsupported URL scheme')
    path = urlparse.urlunparse(('', '') + url_parts[2:])
    h.request('POST', path, data, {'content-type':content_type})
    return h.getresponse().read()
 
def scan_file(filename):
    files = [('file', filename, open(filename, 'rb').read())]
    json = post_multipart(SCAN_URL, {'key':API_KEY}, files)
    return simplejson.loads(json)
 
def get_report(filename):
    md5sum = hashlib.md5(open(filename, 'rb').read()).hexdigest()
    json = post_multipart(REPORT_URL, {'resource':md5sum, 'key':API_KEY})
    data = simplejson.loads(json)
    if data['result'] != 1:
        print 'Result not found, submitting file.'
        data = scan_file(filename)
        if data['result'] == 1:
            print 'Submit successful.'
            print 'Please wait a few minutes and try again to receive report.'
        else:
            print 'Submit failed.'
            pprint.pprint(data)
    else:
        pprint.pprint(data['report'])
 
 
if __name__ == '__main__':
    if len(sys.argv) != 2:
        print 'Usage: %s filename' % sys.argv[0]
        sys.exit(1)
 
    filename = sys.argv[1]
    if not os.path.isfile(filename):
        print '%s is not a valid file' % filename
        sys.exit(1)
 
    get_report(filename)

Be sure to change the API_KEY value on line 8 to reflect your VirusTotal API key. The entire script can be downloaded here to save you a copy and paste.

Update 2011/02/03: Added commentary regarding Georges’s comment about this stackoverflow thread.

Update 2011/01/28: There is an issue with this code when passing large objects through the queue. While the code listed below will work in most situations, consider using sentinels to indicate the end of jobs in your queue rather than relying on the Queue.Empty error. You can read about that in my post titled, The Python Multiprocessing Queue and Large Objects.

I was recently working on improving the efficiency of my botnet analysis code by utilizing 100% of the CPU resources available to my machine. In order to do that in python I needed to span multiple processes as multithreading would produce no benefit for these CPU bound events. While python utilizes true threads that have the ability to run on different cores concurrently, the Global Interpreter Lock, or GIL, makes it such that only one of these threads can run “concurrently”. Thus the simplest solution seemed to be utilizing python’s multiprocessing module.

Python’s multiprocessing module is actually quite simple to use, especially if you’ve previously used python’s threading module. Additionally the multiprocessing module contains a pool class which automatically sets up processes to manage a pool of jobs. There is, however, one HUGE caveat. The pool of workers cannot be terminated until all the tasks have been consumed. After some simple experimentation I noticed two key things with the multiprocessing.pool feature. First, while the worker processes can handle the KeyboardInterrupt and call sys.exit, these processes persist and thus receive future tasks. Second, the KeyboardInterrupt is not delivered to the parent process until all jobs are completed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

#!/usr/bin/env python
import multiprocessing, os, time, Queue
 
def do_work():
    print 'Work Started: %d' % os.getpid()
    time.sleep(2)
    return 'Success'
 
def pool_function():
    try:
        return do_work()
    except KeyboardInterrupt:
        return 'KeyboardException'
 
def main():
    pool = multiprocessing.Pool(3)
    try:
        jobs = []
        for i in range(6):
            jobs.append(pool.apply_async(pool_function, args=()))
        pool.close()
        pool.join()
    except KeyboardInterrupt:
        print 'parent received control-c'
        pool.terminate()
 
    for i in jobs:
        if i.successful():
            print i.get()
        else:
            print 'Job failed: %s %s' % (type(i._value), i._value)
 
if __name__ == "__main__":
    main()

I constructed a fairly simple example of this behavior, shown above. Running the code will span three worker processes to handle a total of six jobs. The job is very simple: display a message, sleep for two seconds and return a message to the parent. You’ll notice that when you send a KeyboardInterrupt by pressing ctrl+c (on Linux) it’ll kill the currently running child processes, however the next job will simply take its place until there are no remaining jobs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

#!/usr/bin/env python
import multiprocessing, os, signal, time, Queue
 
def do_work():
    print 'Work Started: %d' % os.getpid()
    time.sleep(2)
    return 'Success'
 
def manual_function(job_queue, result_queue):
    signal.signal(signal.SIGINT, signal.SIG_IGN)
    while not job_queue.empty():
        try:
            job = job_queue.get(block=False)
            result_queue.put(do_work())
        except Queue.Empty:
            pass
        #except KeyboardInterrupt: pass
 
def main():
    job_queue = multiprocessing.Queue()
    result_queue = multiprocessing.Queue()
 
    for i in range(6):
        job_queue.put(None)
 
    workers = []
    for i in range(3):
        tmp = multiprocessing.Process(target=manual_function,
                                      args=(job_queue, result_queue))
        tmp.start()
        workers.append(tmp)
 
    try:
        for worker in workers:
            worker.join()
    except KeyboardInterrupt:
        print 'parent received ctrl-c'
        for worker in workers:
            worker.terminate()
            worker.join()
 
    while not result_queue.empty():
        print result_queue.get(block=False)
 
if __name__ == "__main__":
    main()

In order to behave as I would expect it to, I had to stop using the convenient pool feature and thus I wrote the pool handling code myself as displayed above. By running the code, you’ll notice that I had to implement a job_queue and result_queue to pass information between the parent and children processes. More importantly you’ll notice that when you send a KeyboardInterrupt the remaining tasks are not executed and the parent safely exits almost immediately. One final note is that I chose to ignore KeyboardInterrupts in the child completely by using the signal module. This functionality could additionally be accomplished with the except statement commented out on line 17.

On a separate note, I would like to say that this is my 100th blog post. Awesome!

Response to Georges comment on 2011/02/03
In response to Georges comment I wanted to see how such an approach would work in a case similar to what was done above. The below code demonstrates the approach from the stackoverflow response by Glenn Maynard to call the get function with a timeout.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/usr/bin/env python                                                           
import multiprocessing, os, time, Queue
 
def do_work(i):
    try:
        print 'Work Started: %d %d' % (os.getpid(), i)
        time.sleep(2)
        return 'Success'
    except KeyboardInterrupt, e:
        pass
 
def main():
    pool = multiprocessing.Pool(3)
    p = pool.map_async(do_work, range(6))
    try:
        results = p.get(0xFFFF)
    except KeyboardInterrupt:
        print 'parent received control-c'
        return
 
    for i in results:
        print i
 
if __name__ == "__main__":
    main()

This code works exactly as expected (yay!) however there is one difference between this approach and my solution. With this approach, as is, it is not possible to retrieve the results which completed prior to the keyboard interrupt. If that is not important, than this is a much more elegant solution. Below is the output when Thanks for bringing the solution to my attention Georges.