Immediately from links on their landing page it was apparent that Gogo permitted access to its own website, as well as the airline’s website and a few other third-party sites such as Living Social, and Eventful. Attempting to access any other website resulted in a redirect to Gogo’s landing page. Gogo appears to accomplish this by responding with an HTTP redirect to any standard HTTP request, i.e., port 80, that is sent to a non-permitted IP address.

Two common techniques for bypassing pay-for-wireless providers are TCP-over-ICMP and TCP-over-DNS. In a nutshell, TCP is the protocol required to browse the web, ICMP is the protocol used by the command `ping` to determine if a host is available, and DNS is required to resolve domain names like www.google.com to an IP address. When TCP is not completely available yet either ICMP or DNS is, then it is possible to encapsulate TCP connections over the other available protocol. Gogo blocked my ping attempts, hence TCP-over-ICMP was not possible. However, while Gogo doesn’t permit direct access to external DNS servers, Gogo’s DNS server recursively resolved the DNS queries I made. Therefore, Gogo appears to be susceptible to TCP-over-DNS. Despite not being able to verify this finding, I will simply state that Gogo can remedy this, if necessary, by only responding to white-listed DNS queries for non-authenticated users.

As a quick aside, I would like to mention I made an attempt to responsibly disclose this information to Gogo prior to posting this article [1, 2]. I contacted one of their twitter representatives via email who informed me the right person would contact me. After a few days with no response, I sent a follow up email to the same twitter representative. While that representative has continued to tweet since my followup, I have received no replies. Thus, I have come to the conclusion that Gogo is uninterested. I proceed with the knowledge that this authentication bypass in no way compromises Gogo’s security; therefore, it is of negligible importance to Gogo’s existing customers. Finally, I proceed knowing very well that Gogo’s sysadmins can correct the underlying problem in a very brief period of time and I even present them with the solution.

♫Now back to the good part!♫

I use Google Chrome as my primary browser and along with it I have installed a few extensions that depend on connections to various Google services. When Chrome is open and does not have a connection to the Internet, some of these extensions, such as the Google Reader Notifier, adjust their icon to indicate their disconnected status. As expected, the Google Reader Notifier extension indicated that I was not connected to the Internet. However, as I was browsing around Gogo’s landing page and permitted sites, I was shocked when the Google Voice extension notified me of a new text message and shocked again when I successfully replied. I had mistakenly stumbled across a hole in Gogo’s access policies and decided to dig deeper.

As I previously mentioned, Gogo redirected to their landing page upon any standard HTTP request. However, the same was not true for HTTPS connections, which by default occur on TCP port 443. Thus, for some unknown reason, my computer was able to connect to www.google.com via HTTPS. Immediately I tried other Google services, like mail.google.com for Gmail, and docs.google.com for Google docs; neither worked. Nevertheless, any Google service accessible via HTTPS and addressed under the www.google.com domain worked flawlessly.

After poking around to find a list of Google services meeting these requirements, I had an epiphany. While I probably learned this information at some point in my past, I had a hunch that Google’s front-end web servers would likely provide the correct response to any Google web service. Hence, I opened up my /etc/hosts file on my OS X machine, and added the following line:

74.125.225.40 mail.google.com

Voila! Immediately, I was able to access Gmail. The IP address, 74.125.225.40, corresponds to the IP address my computer was successfully able to connect to in order to access services under the www.google.com domain. By adding this manual entry to my hosts file, I informed my operating system to use the provided IP address when attempting to connect to mail.google.com. In order to additionally receive successful access to Google+, YouTube, Google Docs, Google Code, and Google’s chat interface, I updated the line to the following:

74.125.225.40 mail.google.com plus.google.com youtube.com docs.google.com code.google.com chatenabled.mail.google.com

What I previously described is how I was able to bypass Gogo’s Inflight Internet Authentication in order to access a number of Google services for free. What remains is how to utilize this information to access the entire Internet. The answer to this question lies with Google AppEngine.

Google AppEngine (GAE) is a cloud-based web application hosting service provided by Google. Anyone can run a GAE application on Google’s servers. As I previously described, Google’s front-end web servers respond to requests for any Google web service, including third-party GAE applications. The final piece of the puzzle is that GAE allows its applications to themselves make web requests. Therefore, a well written GAE application can operate as a proxy server. In fact, there are a number of available packages for running simple GAE proxy servers. Digital Inspiration and Windows Guides each provide tutorials for setting up such proxies. The Windows Guides tutorial even links to a working example at https://jttm-server-prox.appspot.com/. A user of Gogo’s Inflight Internet need only add jttm-server-prox.appspot.com to their hosts file in order to utilize this proxy to access much of the Internet.

I have previously described how Gogo’s Inflight Internet authentication can be bypassed through the combination of a custom hosts file and a GAE proxy server. The remainder of this article will detail how Gogo can fix this problem.

The root question, is: why does Gogo allow non-authenticated access to some Google IPs? While my answer to this question is purely speculation, the solution I offer will absolutely solve the problem. I speculate that Gogo allows access to some Google IPs because of Google Analytics and Google Adsense, indicated in client scripts by the domains www.google-analytics.com and ad.doubleclick.net respectively. If correct, Gogo’s sysadmins simply white-listed a few too many IP addresses. Regardless, the point is moot because any Google front-end web server IP address, including those for Analytics and Adsense, will serve the reply for all Google web services, most notably GAE applications. Hence, by white-listing any single Google front-end web server IP address, Gogo is essentially providing access to the entire Internet. It follows, that the simplest solution to this bypass is for Gogo to completely disable direct access to Google IPs from non-authenticated users.

While the above solution absolutely works, it prevents Gogo from tracking users and serving ads using Google services. Of course, serving ads is pointless when non-authenticated users cannot visit the target site of the advertisement, so let’s forget about Google Adsense. Google Analytics on the other hand, is a useful tool for monitoring users’ access to a website. Let’s now proceed under the assumption that it is essential for Gogo’s non-authenticated users to have access to whatever Google service Gogo intended to allow, and for simplicity assume it is the Google Analytics tracking service. The question now is: how can Gogo allow access to the Google Analytics tracking service without allowing access to all other Google services? This answer partially lies with Server Name Indication.

Server Name Indication (SNI) is implemented on most modern web browsers and allows a web server with a single IP address to serve multiple TLS certificates. The proper certificate is returned in response to the hostname provided in the TLS client handshake. The use of SNI is why, despite using the same IP address for all the aforementioned services, Google is able to return the appropriate domain-specific TLS certificate. Therefore, Gogo can additionally white-list on the SNI-hostname in order to prevent the authentication bypass and still allow access to Google Analytics. However, things are not so simple, as Google’s servers do not depend on the presence of the SNI-hostname in the TLS client handshake.

In the absence of SNI, Google’s front-end web servers return the default TLS certificate for that particular server, yet, the server still responds with the requested content as indicated in the HTTP host header. Because the HTTP host header is encrypted, as well as the entire response, it is not possible to discern desired requests from undesired requests. One solution is to only allow such access to Google Analytics if the client supports SNI and the SNI-hostname is in the white-list. While this approach would work, it appears it would restrict access to Google Analytics for some browsers that are still commonly used.

Regardless, I suspect that Google’s front-end web servers do not verify consistency between the SNI-hostname and the HTTP host header. Under that assumption, it would still be possible, in the presence of an SNI-hostname white-list, to bypass Gogo’s authentication with a custom browser (or local proxy server) that negotiates TLS handshakes using only the expected SNI-hostname yet still makes the desired Google web service request. While I don’t believe anything is readily accessible to make this process easy for 99.9% of Gogo’s potential customers, it only requires a single person’s determination to make it so. Thus, the only be-all-end-all solution to prevent this bypass is for Gogo to completely disable direct access to Google IPs from non-authenticated users.

There is, of course, one other possible solution: Gogo could man-in-the-middle the desired Google web services in order to perform filtering on the HTTP host header. However, this approach could have unforeseen consequences. Therefore, I do not recommend it.

In conclusion, I have described how one can currently bypass Gogo’s Inflight Internet authentication to access much of the Internet using a custom host file and a Google AppEngine proxy. I have also described how Gogo can somewhat block this bypass by supplementing their IP white-list with both an SNI requirement and an SNI-hostname white-list. Finally, I speculated that even with the aforementioned solution, it would still be possible for a determined user to bypass their authentication. Thus, I stated the only real solution is for Gogo to not allow any direct access between non-authenticated users and Google web servers.

Happy “free” surfing, for now anyway

March 12, 2012
2:00 PM
HFH 1132

Committee:
Diana Franklin (chair), Tim Sherwood, Kevin Almeroth

Title: Automated Feedback and Assessment of Student Submitted Code

Abstract:
Increases in class sizes pose a problem for computer science instructors. With a larger number of students to educate, instructors are forced to choose between assigning fewer assignments and devoting more of their time to satisfying the additional demands of larger class sizes. The first option, reducing the number of assignments, decreases the overall quality of a course, whereas the latter option, working more, scales only so far. Fortunately, a variety of tools exists which provides instructors with a third option. Automated feedback and assessment systems can provide significantly more detailed feedback to large numbers of students than instructors are capable of providing, and do so nearly instantaneously. The use of such systems allows instructors to continue to assign a similar number of comprehensive assignments despite growing class sizes, and also makes it possible for instructors to focus their time on students who require additional assistance with coursework.

In this talk I will provide a brief history of automated feedback and assessment systems, followed by an in-depth analysis of state-of-the-art automated feedback and assessment systems. Finally, I will provide an overview of additional research opportunities provided in other areas of computer science as a byproduct of using automated feedback and assessment systems.

Everyone is welcome to attend!

In spite of my continuing recommendation to use python for such a task, I was curious what a C solution might look like. Thus, I set out to write, and did write, a simple process-based job management system in C that offers similar functionality as python’s multiprocessing.Pool class. Take note that writing this system required more time than the equivalent python code would have, however, the point is moot because python has the multiprocessing.Pool class. Furthermore, whoever wants to use this system must manually include it in their project, thus introducing a third-party dependency in their code. Finally, this code will not compile in the typical Windows development environment as it relies on POSIX-based system calls. Hence, for all the reasons I previously mentioned, this C implementation is quite inferior to python’s multiprocessing.Pool class.

The source that I reference throughout the remainder of this post can be found and downloaded in entirety. On a POSIX-based machine the code can be compiled via gcc main.c job_management.c and the test program can be run via ./a.out.

The system, or library, is contained in two files: job_management.h and job_management.c. Anyone familiar with C or C++ development knows that job_management.h is a header file which contains the structures and function prototypes whereas job_management.c contains the implementation. The entire library functionality is exposed through three functions: manager_init, manager_run_jobs, and manager_destruct.

void manager_init(struct manager *manager, int num_workers,
		  void (*input_callback)(char *),
		  void (*output_callback)(char *),
		  void (*work_function)(char *));

Aside from the pre-allocated manager structure that all the functions require as their first parameter, the manager_init function requires four other parameters: num_workers, input_callback, output_callback, and work_function. The num_workers parameter specifies an upper bound on how many worker processes to start. Each of the remaining parameters are function pointers to functions that must each have a single char pointer parameter and return no value. I will describe the purpose of these functions in the order that they execute when considering only a single job.

The input_callback function is called by the master process, once per job, in order to fetch the input for the job. The callback function stores the job input in the provided buffer before returning. The work_function function is the reason for the entire system as this function performs the job processing. In this function, the provided buffer serves both as job input and job output. On invocation, the buffer contains the result of a prior call to input_callback which has been sent over a pipe between the master process and worker process. On return, the buffer should contain the job output which will then be sent back to the master process. The output_callback function is called by the master process as each job is completed. In this case, the buffer contains the output of a completed job.

int manager_run_jobs(struct manager *manager, int num_jobs);

Once the manager has been properly set up, a number of jobs can be started via manager_run_jobs which takes one additional argument, num_jobs. Intuitively this parameter represents the number of jobs to process. This function contains the most complicated piece of the system as it is responsible for forking the processes, setting up the pipes between the master and workers, cleaning up the address space and file descriptor table for the workers, and properly using I/O multiplexing in order to both send job input and receive job output.

One additional complication that the manger_run_jobs function has to handle is the issue that started this entire series of blog posts; that is, the want to gracefully exit the job management system when a keyboard interrupt (SIGINT), occurs. Gracefully exiting means any jobs that are running at the time of the keyboard interrupt will successfully complete prior to exiting, thus giving the master process the chance to save partial results. This function handles the desired behavior through the combination of a global volatile value that’s periodically checked within this function’s primary loop, and a SIGINT handler function that updates the value. Worker processes are unaffected by SIGINT as they are set to ignore it.

void manager_destruct(struct manager *manager);

The last function, manager_destruct, conveniently frees the contents of the manager structure. It is important to note that this function is also called by each worker as part of its address space and file descriptor table clean up. For this reason, the manager_destruct function handles the closing of some FILE structures. Otherwise, this function is incredibly simple.

As previously mentioned, my job_management library is simple and thus is no where near as flexible as python’s multiprocessing.Pool class. Furthermore, my library has no support for data serialization as it relies on messages being no larger than MAX_MESSAGE bytes in size, and has a limitation that the message contents span only a single line. These restrictions were chosen for simplicity and are, by no means, a limitation of C. However, again, I must point out that object serialization is provided by default with python, and is built into the multiprocess.Pool class.

Below is an example program that demonstrates using my process-based job management system. Note that this code really isn’t that complicated, especially when compared to the python solution that gracefully handles the keyboard interrupt. Of course, the python solution is without dependencies and much more portable.

Happy coding!

#include <stdio.h>
#include <string.h>
#include "job_manager.h"

int job_num = 0;

/* This function is a worker function that does the work. On invocation, buffer
   contains a single line containing the input for the job. When complete,
   buffer should store the result of the job. The contents of the result must
   fit on a single line that ends with a single '\n'. */
void do_work(char *buffer) {
  int job = atoi(buffer);
  printf("%d started on job %d\n", getpid(), job);
  sleep(3 + getpid() % 3);
  snprintf(buffer, MAX_MESSAGE, "%d Success\n", job);
}

/* The message to send to the worker must be stored in the provided buffer of
   size MAX_MESSAGE. The contents must fit on a single line that ends with a
   single '\n' */
void prepare_job(char *buffer) {
  snprintf(buffer, MAX_MESSAGE, "%d\n", job_num++);
}

/* Process the single line result returned from the worker */
void process_result(char *buffer) {
  printf("%s", buffer);
}

int main(int argc, char *argv[]) {
  int i, num_jobs;
  struct manager manager;

  if (argc != 3) {
    printf("Usage: %s workers jobs\n", argv[0]);
    return 1;
  }
  num_jobs = atoi(argv[2]);

  manager_init(&manager, atoi(argv[1]), prepare_job, process_result, do_work);
  i = manager_run_jobs(&manager, num_jobs);
  manager_destruct(&manager);
  printf("%d of %d jobs completed\n", i, num_jobs);
  return 0;
}

We started with a discussion of performance. It’s no secret that C will outperform python in any task; after all python is an interpreted language and the standard python implementation is written in C. However, Ali and my discussion was not focused on the performance of the jobs themselves, but rather on the performance of a job management system that could be gracefully interrupted. In such a system, where the majority of work is done in job processing, the question of the job management system’s performance is irrelevant; it makes little difference if C can launch all the workers thirty times faster when python still does it in well under a second. As an aside, it’s trivial to call a C program from python, or even to write a python C module for fast job processing performance.

Once the irrelevance of the job management performance was agreed upon, Ali and I discussed development time, dependencies and portability. Having written both a significant amount of C and python code pertaining to operating system concepts, I know that anyone with equivalent experience can much more efficiently code a process-based job management system in python than in C. This holds true, even without the multiprocessing module and when using only the low level features provided by the os package. The two primary reasons for faster development time are dynamic types, and garbage collection. Additionally, python provides a lot more out-of-the-box functionality, e.g., string manipulation and basic data structures such as lists and dictionaries; all features that contribute to faster development time.

Of course, if one had all the needed libraries in C, then my only argument in favor of python is the lack of types and implicit memory management. The choice between C and python in this case is simply up to personal preference. However, having the required C libraries is a problem on its own. This is where my lack of real-world C experience appears as I’m not familiar with well-written C libraries that provide python-like string manipulation and basic data structures, nor am I familiar with a well written process-based job management library. Nevertheless, even if these libraries were easily available, they are additional dependencies that must be present in order to run any code utilizing them.

Managing dependencies can be a pain, especially in the absence of an automated process for obtaining them. For example, I wrote some code for a computer graphics class a few years ago that I recently wanted to compile. As I recall, it took me a few hours to find and install all the required OpenGL dependencies before I could compile my own code. Compare that to the 90%+ of python programs I’ve written which have no external dependencies. This vast amount of code still works out-of-the-box across multiple platforms.

Portability is my final argument in favor of python over C. While ANSI-C based libraries should work across multiple platforms, dealing with process management is a whole other can of worms as this drastically differs between operating systems. Again, my inexperience shows as I am not familiar with any process management libraries that are compatible with both Windows and POSIX-based systems. Moreover, I am not even sufficiently familiar with the Windows API to know its equivalent functions to fork, exec, pipe and dup that are needed to write a process-based job management system for Windows. Nevertheless, with python, one needn’t worry about such things because the majority of the functionality in the os package is supported on both Windows and POSIX-based systems. Most importantly the multiprocessing module is supported on both, and thus anyone who installs python, independent of their operating system, can run the multiprocessing examples I previously provided [1, 2].

While Ali and I had a great discussion going, we still hadn’t gotten to the root of his question. Ali didn’t care about development time, nor portability. The problem was that he didn’t see any significant advantages to using the multiprocessing.Pool class when he could instead quickly write a process-based job management system in C utilizing fork to spawn processes and pipe to handle the inter-process communication (IPC). As it turns out, Ali wasn’t aware that the python multiprocessing.Pool class internally handles all the IPC. Without the built-in IPC support, Ali’s concerns would be valid as some other communication mechanism would need to be explicitly written to communicate between the master process and its workers. Fortunately, in python we get all that for free with both the multiprocessing.Pool and multiprocessing.Queue classes.

Personally, the choice of using python for such a task is a no-brainer because I can develop the system faster without the need for external dependencies and have confidence that it will work on any operating system that python runs on. Nevertheless, my curiosity peaked and I set out to write a simple process-based job management system in C. The system is complete, though I’ve yet to write about it and I’ve now written about it. Read the followup: Implementation of a Process-Based Job Management System in C

John suggests that by having the worker processes ignore SIGINT, the signal that results in python’s KeyboardInterrupt, the entire problem can be solved. Astute readers will note that I actually used the same approach in my second update to my aforementioned post, which suffered from the problem that intermediate results could not be processed, i.e., jobs that completed prior to the keyboard interrupt. While, John’s solution did educate me as to the existence of the initializer and initargs parameters to the multiprocessing.Pool function, his solution in-fact does not work. The only reason it appears to work is due to the time.sleep(10) in his try block. In most code this sleep call would not exist, rather the code would immediately call join() on the pool object.

In the absence of the delay introduced by the sleep call, John’s code still suffers from the original problem which is the KeyboardInterrupt exception does not reach the main process until all of the jobs have completed. The proper solution to the problem would be to fix the multiprocessing library to allow the join function to be interrupted. Until then, my suggestion of rolling your own pool functionality is the best solution I am aware of.

Below is a verbatim copy of my original solution for your convenience:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

#!/usr/bin/env python
import multiprocessing, os, signal, time, Queue
 
def do_work():
    print 'Work Started: %d' % os.getpid()
    time.sleep(2)
    return 'Success'
 
def manual_function(job_queue, result_queue):
    signal.signal(signal.SIGINT, signal.SIG_IGN)
    while not job_queue.empty():
        try:
            job = job_queue.get(block=False)
            result_queue.put(do_work())
        except Queue.Empty:
            pass
        #except KeyboardInterrupt: pass
 
def main():
    job_queue = multiprocessing.Queue()
    result_queue = multiprocessing.Queue()
 
    for i in range(6):
        job_queue.put(None)
 
    workers = []
    for i in range(3):
        tmp = multiprocessing.Process(target=manual_function,
                                      args=(job_queue, result_queue))
        tmp.start()
        workers.append(tmp)
 
    try:
        for worker in workers:
            worker.join()
    except KeyboardInterrupt:
        print 'parent received ctrl-c'
        for worker in workers:
            worker.terminate()
            worker.join()
 
    while not result_queue.empty():
        print result_queue.get(block=False)
 
if __name__ == "__main__":
    main()

Naturally, the obvious solution is to tether your laptop to your phone and if you have an Android phone the process is incredibly simple. In fact, there are a number of pages on the Internet that tell you exactly how to do it. Primarily, there is an Android Forums post from November, 2009 that recommends using an Android application, Azilink, in combination with an OS X application Tunnelblick, and a shell script to configure the tether correctly. There is also a great blog post from March of this year that is a bit more user friendly than the forum post in describing how to tether your OS X machine to your Android phone.

Despite the last guide being fairly recent, it unfortunately is slightly out of date due in part to OS X Lion. The older versions of Tunnelblick seemingly don’t work with OS X Lion, thus an update to a newer version of Tunnelblick is required. The newer version (currently 3.2beta32) of Tunnelblick breaks the shell script that the previous guide uses, thus I am writing this post simply to provide the Internet with an update to that shell script. Please note that everything else mentioned on the blog post currently works as described.

The fix to the shell script is actually really simple, just replace the line shown in the first block of code below, with the line shown in the second:
sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn --dev tun \ sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.2.1/openvpn --dev tun \

Finally for your convenience, I created a github gist with the updated shell script. Observant readers will notice that I updated some of the comments in the script. Happy tethering!

#!/bin/bash
#
# azilink for OS X Lion
# based on http://pastie.org/405289 but works with Tunnelblick only
# (no need to install a separate copy of OpenVPN2 from macports 
# or building from source by hand, thankfully)
# Requires: 
# - azilink running on android phone (http://code.google.com/p/azilink/)
#   (run the app and check the box to start the service).
# - adb on system path (comes with the Android SDK; 
#   add its tools folder to your PATH in ~/.profile or 
#   place or symlink the sdk's tools/adb file in e.g. usr/local/bin or somewhere else on the PATH)
# - Tunnelblick, a nice OS X packaging of OpenVPN (http://code.google.com/p/tunnelblick/)
#   Install Tunnelblick to Applications. Tested with Tunnelblick 3.2beta32 (build 2817)

init() {
    adb forward tcp:41927 tcp:41927
    sudo /Applications/Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.2.1/openvpn --dev tun \
                  --script-security 2\
                  --remote 127.0.0.1 41927 \
                  --proto tcp-client \
                  --ifconfig 192.168.56.2 192.168.56.1 \
                  --route 0.0.0.0 128.0.0.0 \
                  --route 128.0.0.0 128.0.0.0 \
                  --keepalive 10 30 \
                  --up "$0 up" \
                  --down "$0 down"
}


up() {
    tun_dev=$1
    ns=192.168.56.1
    sudo /usr/sbin/scutil << EOF
open
d.init
get State:/Network/Interface/$tun_dev/IPv4
d.add InterfaceName $tun_dev
set State:/Network/Service/openvpn-$tun_dev/IPv4

d.init
d.add ServerAddresses * $ns
set State:/Network/Service/openvpn-$tun_dev/DNS
quit
EOF
}


down() {
    tun_dev=$1
    sudo /usr/sbin/scutil << EOF
open
remove State:/Network/Service/openvpn-$tun_dev/IPv4
remove State:/Network/Service/openvpn-$tun_dev/DNS
quit
EOF
}


case $1 in
    up  ) up $2 ;;  # openvpn will pass tun/tap dev as $2
    down) down $2 ;;
    *   ) init ;;
esac

In exchange for event sponsorship, we include each company’s logo on our website, in the conference proceedings, and in a visible display at the workshop itself. For many of the previous GSWCs, we created and printed up an expensive poster with the theme of the event as well as logos of all the corporate sponsors. While that worked well, it required a decent effort to design the poster, and, as I said before, was considerably expensive to print. Thus, when I was the chair of the GSWC last year, I decided to take a different approach to how we display the corporate logos during the workshop. Rather than using a static poster, I created an animated display that was projected on one of the room’s walls during the conference.

I occasionally provide consulting for a local Santa Barbara company, Worldviz who makes a product, Vizard, that allows one to quickly construct 3D environments in python. Using Vizard I was able to quickly create a 3D logo display in which the logos follow a circular path along the z-axis, as shown in the following photos. While this worked well, Vizard unfortunately only works on Windows and thus could not be run easily from my Mac laptop.

We wanted to use the same display for this year’s GSWC so I sought to rewrite the display in a cross platform and free manor. While I have some previous opengl experience in C, I really wanted to write the display in python so I looked at the various python opengl options. From the StackOverflow thread, OpenGL with Python, I quickly settled on pyglet. Since I already knew the equation to move points around in a circular path, my only challenge was to figure out how to map a texture to a quad so that I could position the quad in 3D space. I quickly asked a StackOverflow question, Moving an image around in 3D space in hopes that while I was in the process of figuring it out, someone would provide me with the solution. Unfortunately, crowdsourcing didn’t pay off, nevertheless, I eventually found the solution and wrote a simple cross platform logo animation program.

In order to run the following program, you will first need to install the required libraries (an exercise left to the reader), create a folder “imgs” and place whatever images you want in that folder and then run the following script (download animator.py). Note: If you want to run this script on 64-bit OS X you’ll need to run it via: VERSIONER_PYTHON_PREFER_32_BIT=yes ./animator.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

#!/usr/bin/env python
import math, os, pyglet, sys
from pyglet.gl import *
 
class World(pyglet.window.Window):
    def __init__(self, scale=10, center_pos=(0, 0, -15), speed=1.0,
                 *args, **kwargs):
        super(World, self).__init__(*args, **kwargs)
        self.scale = scale
        self.center_pos = center_pos
        self.speed = speed
        glClearColor(1.0, 1.0, 1.0, 0.0)
        glEnable(GL_DEPTH_TEST)
        self.textures = self.load_textures()
        self.clock = 0
        pyglet.clock.schedule_interval(self.update, 1 / 60.0)
 
    @staticmethod
    def load_textures():
        img_dir = 'imgs'
        textures = []
        if not os.path.isdir(img_dir):
            print 'Could not find directory "%s" under "%s"' % (img_dir,
                                                                os.getcwd())
            sys.exit(1)
        for image in os.listdir(img_dir):
            try:
                image = pyglet.image.load(os.path.join(img_dir, image))
            except pyglet.image.codecs.dds.DDSException:
                print '"%s" is not a valid image file' % image
                continue
            textures.append(image.get_texture())
 
            glEnable(textures[-1].target)
            glBindTexture(textures[-1].target, textures[-1].id)
            glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, image.width, image.height,
                         0, GL_RGBA, GL_UNSIGNED_BYTE,
                         image.get_image_data().get_data('RGBA',
                                                         image.width * 4))
        if len(textures) == 0:
            print 'Found no textures to load. Exiting'
            sys.exit(0)
        return textures
 
    def update(self, _):
        self.on_draw()
        self.clock += .01
 
    def on_draw(self):
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glLoadIdentity()
        self.draw_images()
 
    def draw_images(self):
        angle_base = (self.clock * self.speed * 50) % 360
        angle_delta = 360. / len(self.textures)
 
        for i, texture in enumerate(self.textures):
            angle = math.radians((angle_base + i * angle_delta) % 360)
            dx = math.sin(angle) * self.scale
            dz = math.cos(angle) * self.scale
 
            if texture.width > texture.height:
                rect_w = texture.width / float(texture.height)
                rect_h = 1
            else:
                rect_w = 1
                rect_h = texture.height / float(texture.width)
 
            glPushMatrix()
            glTranslatef(dx + self.center_pos[0], self.center_pos[1],
                         dz + self.center_pos[2])
            glBindTexture(texture.target, texture.id)
            glBegin(GL_QUADS)
            glTexCoord2f(0.0, 0.0); glVertex3f(-rect_w, -rect_h, 0.0)
            glTexCoord2f(1.0, 0.0); glVertex3f( rect_w, -rect_h, 0.0)
            glTexCoord2f(1.0, 1.0); glVertex3f( rect_w,  rect_h, 0.0)
            glTexCoord2f(0.0, 1.0); glVertex3f(-rect_w,  rect_h, 0.0)
            glEnd()
            glPopMatrix()
 
    def on_resize(self, width, height):
        glViewport(0, 0, width, height)
        glMatrixMode(GL_PROJECTION)
        glLoadIdentity()
        gluPerspective(65.0, width / float(height), 0.1, 1000.0)
        glMatrixMode(GL_MODELVIEW)
 
 
if __name__ == "__main__":
    window = World(width=800, height=600)
    pyglet.app.run()

Forensics 100
The forensics 100 challenge indicated to find the key, and provided a png file that was 19025×1 in resolution. Immediately our team thought we could simply change the resolution to 25×761 and would be on to something. After working with the resulting image for sometime I finally thought about converting it to 761×25. That was our first break through when we read some text along the lines of “ILoveMeSomesheepysheepies” followed by binary that includes capital ‘O’s in place of some of the ’0′s. After no success with different permutations of that message we incorporated an idea the other team members had about the blue offset pixels that occur at regular intervals. Our first attempt at wrapping the image at the blue pixel boundaries (every 450 pixels) resulted in success! The key “thankYouSirPleasemayIhaveAnother” appeared and worked. The following is my simple python solution for Forensics 100:

1
2
3
4
5
6
7
8
9
10
11

#!/usr/bin/env python
import sys, Image
 
def main():
    orig = Image.open('f100.png')
    img = Image.new('RGBA', (450, 43))
    img.putdata(orig.getdata())
    img.show()
 
if __name__ == '__main__':
    sys.exit(main())

Forensics 300
Forensics 300 was quite an interesting challenge. I don’t have the original file, nevertheless, one had to extract the initial file with a password to get a dmg containing a dump from an iphone. I came into the challenge a little late, after one of my teammates had gone through all the images, videos, and audio files looking for Waldo and ‘grep’ing for various relevant keywords. Further more, my teammates had previously used the iPhoneTracker on the consolidated.db file to see where the phone had been, however San Antonio didn’t prove to be very useful.

While the iPhoneTracker app seemed pretty cool, I wanted to programmatically see where the phone had been the most. Thus, after figuring out what was what with respect to the consolidated.db file I wrote a little python script to find the most visited places rounded to less precision to account for some variance. The top three results were the following where the first number represents the number of occurrences in that location, and the two numbers between the parenthesis represent the latitude and longitude respectfully.

• 30 (‘-77.846′, ’166.677′)
• 18 (’0.000′, ’0.000′)
• 10 (’36.106′, ‘-115.173′)

When I did a google search for the coordinates -77.846 166.667 I knew immediately that it was no coincidence that I was centered in a small town in Antarctica. Unfortunately, Google maps doesn’t have a name for this location so I had to revert to Bing (for the first time ever) to figure out that this location is called Ross Island. From that point we simply attempted different “places” listed Ross Island’s wikpiedia page until “McMurdo Station” submitted successfully. Below is the script I used to find the coordinates from the consolodated.db input file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#!/usr/bin/env python
import os, sys
 
def main():
    os.system('sqlite3 consolidated.db "select Latitude, Longitude '
              'from CellLocation;" > tmp')
 
    uniq = {}
    for line in open('tmp'):
        pos = tuple('%.3f' % float(x) for x in line.split('|')[:2])
        if pos in uniq:
            uniq[pos] += 1
        else:
            uniq[pos] = 1
 
    for pos, count in sorted(uniq.items(), key=lambda x:x[1]):
        print count, pos
 
if __name__ == '__main__':
    sys.exit(main())

You can find links to solutions to other Defcon 19 Quals challenges at the following locations: Rogunix, negative foo, VNSecurity site.

In other news I was one of four recipients of the 2010-2011 UCSB Academic Senate Outstanding Teaching Assistant Award. Today there was an awards ceremony where I received my first pre-framed certificate. I am very honored to have won this award especially given that I am the first student in the Computer Science Department to win it. It’s a very cool feeling, however, as Nichole put it, with great acknowledgement come greater expectations. I hope to exceed those expectations.

Update 2011/02/22 13:19 PST
Added the jsf tag (java server faces) and updated the total question count for each item on the list.

Update 2011/02/22 11:14 PST
Adding spring-mvc as that was what I originally was originally supposed to have.

Update 2011/02/22 10:36 PST
For the interested, here is the table used to generate the graph.

Update 2011/02/22 10:05 PST
As per comments on this post, I updated the list by removing hibernate, spring, and sass and added gwt and grails. I also updated the chart reflecting this information, and created an additional chart which plots the frameworks as a percentage of the questions asked each week to hide stackoverflows’s growing popularity.

Adam and I are currently in the process of working on our research about the Execution After Redirect, or EAR, Vulnerability which I previously discussed in my blog post about the 2010 iCTF. While Adam is working on a static analyzer to detect EARs in ruby on rails projects, I am testing how simple it is for a developer to introduce an EAR vulnerability in several popular web frameworks. In order to do that, I first needed to come up with a mostly unbiased list of popular web frameworks.

My first thought was to perform a search on the top web frameworks hoping that the information I seek may already be available. This search provided a few interesting results, such as the site, Best Web-Frameworks as well as the page Framework Usage Statistics by the group BuiltWith. The Best Web-Frameworks page lists and compares various web frameworks by language, however it offers no means to compare the adoption of each. The Framework Usage Statistics page caught my eye as its usage statistics are generated by crawling and fingerprinting various websites in order to determine what frameworks are in use. Their fingerprinting technique, however, is too generic in some cases thus resulting in the labeling of languages like php and perl as frameworks. While these results were a step in the right direction, what I was really hoping to find was a list of top web frameworks that follow the model, view, controller, or MVC, architecture.

After a bit more consideration I realized it wouldn’t be very simple to get a list of frameworks by usage, thus I had to consider alternative metrics. I thought how I could measure the popularity of the framework by either the number of developers using or at least interested in the framework. It was this train of thought that lead me to both Google Trends and StackOverflow. Google Trends allows one to perform a direct comparison of various search queries over time, such as ruby on rails compared to python. The problem, as evidenced by the former link, is that some of the search queries don’t directly apply to the web framework; in this case not all the people searching for django are looking for the web framework. Because of this problem, I decided a more direct approach was needed.

StackOverflow is a website geared towards developers where they can go to ask questions about various programing languages, development environments, algorithms, and, yes, even web frameworks. When someone asks a question, they can add tags to the question to help guide it to the right community. Thus if I had a question about redirects in ruby on rails, I might add the tag ruby-on-rails. Furthermore if I was interested in questions other people had about ruby on rails I might follow the ruby-on-rails tag.

Between the number of questions per tag, the number of answers per tag, and the number of followers per tag, StackOverflow provides a few metrics for measuring the relative level of developer interest in various web frameworks. Success! The next step was to extract these numbers for the tags of various frameworks. For this, I attempted to find StackOverflow tags corresponding to all the frameworks listed on the Best Web-Frameworks site I previously found. I skipped the framework languages CSS and Javascript as they aren’t server side frameworks. I then narrowed the list down to the frameworks which had at least 100 questions asked.

This produced the following frameworks sorted by total number of questions asked:

1. (31156) ruby-on-rails
2. (20587) asp.net-mvc
3. (14951) django
4. (4726) zend-framework
5. (3510) jsf
6. (3336) gwt
7. (3296) cakephp
8. (3127) codeigniter
9. (2731) grails
10. (1976) spring-mvc
11. (1603) symfony
12. (912) struts
13. (538) kohana
14. (515) pylons
15. (514) sinatra
16. (506) dotnetnuke
17. (420) wicket
18. (227) lift
19. (194) yii
20. (163) cherrypy
21. (126) web2py
22. (106) catalyst

• (6609) hibernate (note: not a web framework)
• (5765) spring (note: not a web framework)
• (178) sass (note: not a web framework)

This list alone seems to work fairly well, however, I wanted to take it one step further which was to see the number of questions asked on a per week basis since the start of StackOverflow. Using the StackOverflow API (I used the API to generate the previous list too) I wrote a script to generate a CSV file containing this information. The information is depicted in the interactive chart below for the top 10 frameworks according to total number of StackOverflow questions. Each point in the chart represents the number of questions asked in a one week period starting on the date of the data point (protip: hover over chart to get the exact values).

Note: if the above graph doesn’t load, try this static image.

The data confirms my previous suspicion that ruby on rails is the number one MVC and that django and cakePHP would also appear in the top 10. I must admit that I had never before heard of asp.net MVC, however considering that stackoverflow and all other stackexchange sites run on asp.net MVC, it makes sense that it would rank quite high.

I added the below chart to show the relative percentage of questions per tag over time as per Big Dave’s Gusset’s comment. This hides the growing popularity of stackoverflow.


(Interactive version)

The data for the above chart was extracted using the following script. The script requires the python package py-stackexchange in order to run and can be easily modified to add additional tags or change the filtering methods.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

#!/usr/bin/env python import datetime, sys, time from stackexchange import Site, StackOverflow   frameworks = [# php 'zend-framework', 'cakephp', 'symfony', 'codeigniter', 'seagull', 'prado', 'solar', 'ezcomponents', 'kohana', 'jelix', 'flow3', 'modx', 'sapphire', 'yii', 'limonade', 'tekuna', 'doophp', 'fat-free', 'akelos', 'php-on-trax', 'atk', # ruby 'ruby-on-rails', 'merb', 'ramaze', 'halcyon', 'sinatra', 'webby', 'sass', # perl 'catalyst', 'interchange', 'mason', 'cgi-application', 'jifty', 'gantry', 'dancer', 'mojolicious', # java 'struts', 'hibernate', 'spring', 'wicket', 'play', 'stripes', # python 'django', 'pylons', 'grok', 'turbogears', 'web2py', 'cherrypy', # coldfusion 'cfwheels', 'coldspring', 'model-glue', # asp.net 'asp.net-mvc', 'dotnetnuke', 'monorail', 'vici']   class TagStats(object): DATE_START = 1217540572 WEEK_SECONDS = 604800 def __init__(self, tag_names): self.so = Site(StackOverflow, 'LzYJwh19o0WCIvXK9q6k6g') self.tag_names = tag_names self.tags = [] self.stats = {}   def output_counts(self, html=False): tmp = [] for tag in sorted(self.tags, key=lambda x:x.count, reverse=True): tmp.append((tag.count, tag.name)) if html: print '<ol>' for count, name in tmp: print ('<li>(%d) <a href="http://stackoverflow.com/tags/%s">' '%s</a></li>') % (count, name, name) print '</ol>' else: print '\n'.join(['%8d %s' % x for x in tmp])   def get_tags(self, min_size): for name in self.tag_names: query = self.so.tags(filter=name) for tmp in query: if name == tmp.name: break else: sys.stderr.write('Not found: %s\n' % name) continue if tmp.count < min_size: sys.stderr.write('Too few questions: %s\n' % name) else: self.tags.append(tmp) self.stats = dict(zip([tag.name for tag in self.tags], [[]]*len(self.tags)))   def output_stats_by_week(self, start_week=0): now = int(datetime.datetime.now().strftime('%s')) num_weeks = (now - self.DATE_START) / self.WEEK_SECONDS print ', '.join(tag.name for tag in self.tags) for i in range(start_week, num_weeks): sys.stdout.flush() start = self.DATE_START + i * self.WEEK_SECONDS end = self.DATE_START + (i + 1) * self.WEEK_SECONDS counts = [] for tag in self.tags: try: count = self.so.questions(tagged=str(tag.name), fromdate=start, todate=end).total except Exception, e: sys.stderr.write('Stopped at week %d\n' % i) sys.exit(1) self.stats[tag.name].append(count) counts.append(str(count)) print ', '.join(counts)     def main(): try: start_week = int(sys.argv[1]) except IndexError: start_week = 0 tag_stats = TagStats(frameworks) tag_stats.get_tags(100) #tag_stats.output_counts(html=True) tag_stats.output_stats_by_week(start_week)   if __name__ == '__main__': sys.exit(main())

Happy web-framework coding!

]]> http://www.bryceboe.com/2011/02/21/using-stackoverflows-api-to-find-the-top-web-frameworks/feed/ 34