Build Geos and Collectstatic

was originally using these buildpacks:

https://github.com/GramercyStudios/heroku-geo-buildpack.git#1.1

https://github.com/heroku/heroku-buildpack-python.git#v40

got this error when i pushed: Collectstatic configuration error. To debug, run …..

when i ran: heroku run …. collectstatic everything was fine!!!

so i had to patch the python buildpack: https://github.com/mattharley/heroku-buildpack-python/commit/807ac1344b01daa8ccc5da0774adeb55fae3b6f5

then i got when i pushed:

—–> Preparing static assets Traceback (most recent call last):

….

django.core.exceptions.ImproperlyConfigured: Could not import user-defined GEOMETRY_BACKEND “geos”.

so i found a newer fork of heroku-geo-buildpack which had two newer commit to fix everything:https://github.com/Tekco/heroku-geo-buildpack/commits/master

Thanks

To this stack overflow post Heroku/Django: Could not import user-defined GEOMETRY_BACKEND “geos”

Backups are as easy as 1-2-3

I like Leo Laporte‘s idea of 1-2-3 backup solutions.

1. Local Copy – I keep anything local that i need to access quickly. every time i format the hard drive or buy a new computer, i only copy across the bare essentials. everything else, i grab from the external drive when needed.
2. Cloud Copy – I use Backblaze for unlimited backup including external drives!
3. Different Hard Drive – I use a simple rsync script, triggered nightly by a cron job, to backup to my external hard drive (which I plug in every night). The rcync script goes something like this:

#!/bin/sh
rsync --progress --recursive -a --exclude="*.Trash*" --exclude="Downloads/*" /Users/matt/ "/Volumes/laptop-2tb/matt-2013/"

Pretty simple hey?

Django + Elastic Beanstalk

Hpflsk has been playing around with Django and AWS Elastic Beanstalk. We’ve hit a few roadblocks along the way, but so far we’re up and running and everything seems to be fine.

Just to summarise our current situation:

• We have 1 Android and 3 iOs developers working hard on version 2 of the app
• These guys and me (sometimes) are the only people hitting the new API
• The new version of the app will be hitting the new AWS Elastic Beanstalk setup

Here are a few roadblocks  and successes we hit along the way:

• Setup using this excellent guide from Grigory Kruglov but each of the .config files, change commands: from commands: to container_commands:. This cost be a few frustrating and headbangingly awful days
• The official docs get you started with a simple example, but in my opinion are really lacking
• The whole config system is wack. The options are spread across two seperate files and one of them seems to update from AWS at random times. Uhh, it sucks and leads to so many headaches.

One thing I wanted to talk about was cost. There’s a guide that is on the Amazon site which estimates costs at something ridiculous – in the $1000s of dollars – and way outside the range of a lean startup. You definitely want to monitor your expenses using a Cloudwatch alarm.

So far, Hpflsk has been charged around $50 AUD a month with only a grand total of 5 people hitting the API regularly. This includes the free tier. This may seem like not that much, but it definitely discourages developers from creating a bunch of apps and seeing what sticks – which seems to be the lean startup model. So if you want to keep everything super lean and mean, you’re better off with a VPS (~$100 AUD per year). It’s that not much of a pain in the ass to setup – took me around a day – and can be used to host multiple projects.

Logging and Application Performance Management Options

Disclaimer: I’m a developer fumbling around in the world of operations. Or in short, I’m trying to becoming as little a devops guy as necessary.

I have deployed the hpflsk backend, a python django app, to an Amazon Elastic Beanstalk (horrible name, pretty good product) instance, which has a generous free tier. Elastic Beanstalk instances have the option to roll, zip and send log files to S3. This is all well and good but what do we do with all those log files? Download, unzip and trawl through them by hand? No way.

So I wanted to find an easy way to index, analyse and store log files which makes it easy for us to find errors, monitor performance and analyse application usage. Here are the current contenders:

• Sentry: Cloud-based or self-hosted open source exception tracking. Integrates with django and web frontends, amongst other things. Free trial 100 events/day, 7 day history
• Loggly: Cloud-based log collection and analysis solution. Free tier 200MB/day of log data and 1 week of log retention. Integrates easily with django
• Papertrail: Another cloud-based log collection and analysis solution. Free tier 100MB/day data, 1 week retention, 48 hours of search. Can integrate with django
• Logstash: Self-hosted open source log collection and analysis. Awesome logo! Can centralize log collection and visualize using the built-in web UI or send to Graphite and view through its web interface
• Graphite with Statsd: Self-hosted open source statistics collection and scalable realtime graphing system. Statsd integrates with django and can send events to Graphite
• New Relic: Cloud-based user (web/app frontend), application and server monitoring. Basic plan is free. Integrates with django

Off we go. May the best man(agement software) win!

Playing with Facebook test users

I’ve been working on the second iteration of the hpflsk backend and have decided to follow best practice and create test Facebook users to play around with (before, we were using our personal accounts as test accounts and causing all sorts of mayhem). The backend is, again, implemented in django – the web framework for hipsters, with deadlines.
My original plan was to create a new batch of test users for every test (in the setUp process) and delete them again after each test (during tearDown). However, I found one of the limitations of the Facebook API is that not all user fields are updatable – notably location and timezone. To get around this, I decided to create 10 static test users and manually update their details by logging in (through the roles sections of the app development page). Hopefully I only need to do this once. Here’s the code to create the test users:

from django.conf import settings
import requests
import json

def create_test_user(name, locale='en_US', permissions=[]):
    r = requests.get("https://graph.facebook.com/%s/accounts/test-users"
            % settings.FACEBOOK_APP_ID,
            params={
                'installed': 'true',
                'name': name,
                'locale': locale,
                'permissions': ','.join(permissions),
                'method': 'post',
                'access_token': settings.FACEBOOK_APP_TOKEN,
                })
    return json.loads(r.text)

def delete_test_user(id, access_token):
    r = requests.get("https://graph.facebook.com/%s?" % id,
            params={
                'method': 'delete',
                'access_token': access_token,           
                })
    return r.text

Another limitation I found with using Facebook test users is that you can’t extend the life of their access token using the method described here. By default, test users are issued with a short term (1 hour) access token. But I want these users to be accessible by my test suite for all eternity! Here’s the trick: every time you fetch a list of your test users, their access tokens will be updated for another hour. So all you need to do is follow the process here (see the Accessing section) and you’ll get a list of all of your test users id’s and updated access tokens. You can then update your test users with the latest access tokens like so (I use the brilliant django-social-auth to handle all my social logins):

from django.conf import settings
from social_auth.models import UserSocialAuth
import requests

def refresh_test_users():
    r = requests.get("https://graph.facebook.com/%s/accounts/test-users" % settings.FACEBOOK_APP_ID,
            params = {
                'access_token': settings.FACEBOOK_APP_TOKEN
            })
    j = json.loads(r.text)
    for u in UserSocialAuth.objects.all():
       user_data = [i for i in j['data'] if i['id'] == u.uid][0]
       u.extra_data['access_token'] = user_data['access_token'] 
       u.save()

I run this refresh script every time my test suite starts so that I have a fresh set of access tokens before I start the integration tests with Facebook.

A Snake and a Plane

Google Appengine + Django

Getting started on a new project can be tough, especially when you have little to no idea about 80% of the technologies involved. As I enjoy making life difficult for myself, I’ve decided to tackle a new web venture from the top down – in other words, tackling some of the big software questions before writing a single line of code – and I think I’m almost there. Hopefully this can act as a template for future web projects.

The Big Decisions

So here’s a brief rundown of the buzz words I was looking for in the platform:

• Cheap! (not really a buzz word) – I want this site to be pretty much free to get up and going
• Scalable – The platform should scale gracefully (both in cost and capacity) with the  amount of traffic
• Portable – The ability to pack up our toys and go elsewhere
• Open – In a number of ways: To be able to clearly communicate about the technologies involved without giving away any trade secrets, to leverage as much open source as possible to begin with and hopefully be able to give back to the open source community as much as possible
• Simple! – Stick to standard technologies so that getting up and running with development is (should be) a breeze

At the end of the day I was won over by Appengine (on pricing, scalability and portability), Django (on openness) and Python (on simplicity). To complement this trio of nerd hipsterness I’ve gone with a Github private repo (some of which will become public) for development, Virtualenv (to keep Python dependencies in check) and most importantly Django-nonrel and the related apps over at allbuttonspressed (to give a portable wrapper over Google Appengine’s Database backend).

For the front end, I’m thinking jQuery and Coffeescript (a nicer and hopefully more familiar syntax for writing Javascript).

If it all goes pear-shaped, at least this post has been deliciously trendy.

The LightSwitch Is On

…but is anyone home?

Ok horrible pun, but here’s the question – is Microsoft LightSwitch worth the price of admission?

A little autobiographical interlude

So I attended my first meeting of the Perth .NET Community of Practice yesterday. It was held in the seminar room at Ennex 100, right in the heart of the CBD. I had no idea that the Perth Microsoft Office was right next door – they must have the most kick ass lunches!

The talk was by Andrew Coates about LightSwitch. The talk itself was great and I think I’ll be attending more of these kind of things. Was a little bemused by the fact that Visual Studio 2010 managed to lock up during the demo several times, especially considering that Andrew’s laptop was rocking 8 cores!

Back to the point – LightSwitch

LightSwitch is a totally RAD, sorry I mean Rapid Application Development, tool which gives non-software people, or as Andrew referred to them: “The Omega Geeks”, the ability to quickly build a forms over data application. My first gripe is with the non-software people part, but I’ll get back to that.

So when LightSwitch fires up, you’re asked to specify where the data is coming from. You can pull in data from a variety of different sources, as long as they are Microsoft SQL Server or Microsoft Sharepoint. Luckily this aspect, along with most other aspects of the product, is extensible.

Next, you’re essentially asked to design the database schema. Woah, hold up! That doesn’t really sound like a non-software person kind of job. Even the plain English definitions of One-to-One, Many-to-One, etc. will probably be lost on the target audience. Validation functions? Computed fields? Forget about it. Second gripe.

Next, you define your screens – essentially the ‘view’ part if we’re talking MVC. This part looks rather neat. There’s a bundle of readily available common screen formats available: row edits, list and details, new entry, grid. Select your screen, point it at a table in your database and BOOM, out pops a… wait, what? A tree view of the screen? This is my third gripe. So now you’re non-software people have to arrange everything in this tree, because apparently non-software people don’t have a clue about HCI either. To see the result you have to build the solution and launch the application. Confused? Imagine how the user is going to feel.

Ok enough hating, now the good bits. Everything is extensible. You name it, it’s extensible. And not in some junky macro language extensible I mean plug in there and start writing C# code extensible. So here’s the idea: Non-software person reaches the limits of their ability. They can then call in Captain Software to the rescue, who dives in and starts developing more powerful features with all the power and flexibility and portability of .NET. Oh wait sorry, forget the portability part.

So I admit, this is much better than the existing alternatives – Microsoft Access databases or (worse) an Excel Spreadsheet! Another biographical aside. I once worked for a company that did all of their company finances in Excel. With macros, cell formulas and the rest of it. No wonder the books didn’t balance. Eeeeek. But is this really a solution for the non-software guy, or is this simply another enabling tool that allows people out of their depth to dig their way into a giant, expensive, ominous mess?

The takeaway: Don’t let the “Omega Geeks” near our software!

I don’t think LightSwitch is the answer to the knowledge gap between us “Alpha Geeks” and the “Omega Geeks”. I can see the point that there is a great divide between these two camps and both of us are just as bad as each other when it comes to communicating effectively.

An example dialogue (paraphrasing from Andrew’s presentation):

“Omega Geek”: I need an application for my book store

“Alpha Geek”: Ok so give me a list of requirements and I’ll get back to you in a few weeks to talk architecture

“Omega Geek”: Archi..wha? I already have a house, I don’t need another one.

MONTHS PASS

“Alpha Geek”: (panting) Ok here it is. It works using a plugin architecture, just enable the plugins you need at runtime. Oh and configuration is done using a backend database. To change the configuration you just need to do a few SQL queries. Simple right? Oh and I had a problem getting it to run on Windows XP, the target platform you originally asked for, so you’re going to have to upgrade everyone to Windows 7. Oh and I didn’t get time to write the user manual. Cool?

“Omega Geek”: Dude. I run a book store. Well I used to. Now I’m broke. Thanks a lot.

Now a counter-example:

“Omega Geek”: I need an application for my book store

“Alpha Geek”: Ok what do you want it to do?

“Omega Geek”: I don’t know. Database or something. Reporting. Automate. Internet. And stuff.

“Alpha Geek”: (sigh) Have you heard of this new awesome LightSwitch thing. It’s awesome. Bye.

I’m not trying to be jerk to anyone here, it’s just the nature of the beast that we attempt to tame every day. So how do you cross this great divide? In my opinion, you build a bridge. You insist on close and consistent communication between both parties. You iterate early and often. You remain flexible to change at all times. Uh wait, sounds like I’m stumbling towards something like Agile. What I’m adamant that you don’t do is give the “Omega Geeks” a hang glider and say JUUUUUUMP!

An alternate use case: Tool for developers and aspiring developers

So just quickly, I’m already past the point of rambling here, what can we do with a broken LightSwitch? How bout we turn the tables. I see LightSwitch as a fantastic tool for “Alpha Geeks” to use as part of their development process. Want a working GUI prototype in seconds? Boom. Want to let your customer populate the back end before the front end has caught up? Boom. Want to get fancier? Reach for your C# and boom! (ok, you get the point. Boom)

What about the kiddies? (I say knowing full well that most “kids” are already far more advanced in their software career than I am, after selling their fart app on the App Store). I see this as a great way for aspiring developers to get started with .NET. Think about it, a rich, graphical, tweak-able application seems a lot more exciting to me than “Hello World”. As their code ninja skills increase, they can delve further and further into the code until they look around one day and say “Holy crap, I’m an Alpha Geek!”.

Ruby on Rails on Windows Quickstart

Just started messing around with Ruby on Rails development. Here’s my quickstart to getting it up and running on Windows XP:

Download the one-click installer from http://rubyforge.org/frs/?group_id=167 (I installed version 1.8.6-27 RC 2)

Use all the defaults for the install (make sure you opt to install the gems package manager)

Download the latest version of Sqlite3 (a small, simple database) without the TCL bindings (i.e. sqlitedll-*.zip). I used version 3.6.13.

Unpack the zip file and copy the contents to your c:\ruby\bin directory (replace c:\ruby with your own install path)

Now open the RubyGems Package Manager command prompt (which should be in the new start menu shortcut under Ruby -> RubyGems)

type:
> gem install rails
> gem install sqlite3-ruby -v 1.2.3

(this last step is necessary as the latest version of sqlite3-ruby doesn’t seem to work for me – see here for more details)

now you can test using: (from the command prompt)
> rails test
> cd test
> ruby script\server
(this should start the local webserver)

navigate to http://localhost:3000/ and you should be greeted by the Ruby on Rails: Welcome aboard page

Laser USB Webcam gets Linux Drivers! (kind of)

Knowing that driver support for USB webcams on Linux is usually quite hit-and-miss, I didn’t have very high hopes after buying two Laser USB Webcams from zazz. The main problem is that device manufacturers usually only release drivers for the majority of their market (i.e Windows users), leaving Mac and all the various *nix’s in the dark. This leads to people like this guy having to pick up the slack, usually with no help from the original manufacturer.

However, one point which does help driver development is that a lot of these cheap re-branded Chinese products use one of only a small number of chipsets (the main brains of the device) – but good luck trying to find the chipset brand in the product documentation. One useful tool for helping to find the product chipset yourself is a free tool called USB sniffer for Windows (I found version 1.5 seemed to work the best for me on a Windows XP SP2 machine). The problem with this method is that you need a Windows box to run this tool. Anyway, using this I found the chipset number was SN9C201 , which is made by a company called Sonix and is employed in numerous USB webcams.

After a bit of googling, I managed to find this which does have a working binary driver for this chipset. The only drawback is that this driver only works under Ubuntu 7.04. So if you’re not a fan of this distro, you’re going to be struggling to get the driver to work (it didn’t work under my Ubuntu Studio 7.04 as this distro uses the low-latency kernel). I’m going to try it on a stock installation of Ubuntu 7.04 later today.

update: ok so it does work under Ubuntu 7.04, just not very well – output is very blocky although framerates are acceptable.

In the mean time, you might want to watch this space for the development of an open source driver.

update: Thanks to a commentor pointing me to this google group, i’ve managed to get an open source solution working. Here’s a rundown on how I got the Laser USB Webcam working under Ubuntu Studio 8.10 (Intrepid Ibex) – I’ve simply augmented the instructions located here:

$ sudo apt-get install git-core
$ sudo apt-get install libv4l-0 libv4l-dev kernel-package build-essential
$ git clone http://repo.or.cz/r/microdia.git
$ cd microdia
$ make

(Don’t worry about the Error 127)

$ sudo modprobe videodev
$ modprobe compat-ioctl32
$ insmod sn9c20x.ko

Test with:

$ sudo apt-get install mplayer
$ LD_PRELOAD=/usr/lib/libv4l/v4l2convert.so mplayer tv:// \
-tv driver=v4l2:width=640:height=480:fps=25:device=/dev/video0 -vo x11

Install with:

$ strip -g sn9c20x.ko
$ mkdir -p /lib/modules/`uname -r`/kernel/drivers/media/video/usbvideo/
$ sudo cp sn9c20x.ko /lib/modules/`uname -r`/kernel/drivers/media/video/usbvideo/
$ sudo depmod -a

Midomi – Music Search Engine

Only recently, I had a friend ask me how hard it would be to create a music search engine that could act like your local (underpaid) CD store worker – where you could come in and say “I heard this song on the radio that kind of goes ‘do do da da doo do baby, yeh’, do you have this one?”. Like anyone, I thought this would be pretty-much impossible as it would need a massive database of melodies (read – every song in the world ever written) and a very powerful analysis engine which could quickly pattern-match whatever you sing into your microphone. That’s why, when I found this website, I was left eating my own hat with humility, as they seem to have done it (although not amazingly well).

Midomi seems to approach the problem by leveraging the very Web-2.0 concepts of user-generated content and social networking. The front page contains a little Flash control which allows you to sing a melody into your microphone and click the “done” button. The search takes around 20 seconds but, sadly, only seems to return a relevant result about 1/4 of the time (for example, “Heaven” by Bryan Adams seems to work but I was very offended when my favorite karaoke song “Total Eclipse of the Heart” by Bonnie Tyler didn’t return any results). It seems that it is the users that have solved the problems of that massive music database, by submitting their own renditions of themselves singing popular tunes. The search engine then attempts to match your input to their massive collection of uploaded content.

It would seem intuitive that as more content is uploaded, the search function will probably get a lot better. On the other hand, the site also, at times, seems to be nothing more than a chance for closet musicians to get their talents out there (this guy has done over 80 renditions of various songs) – which isn’t at all a bad thing, except that there exist zillions of other sites already out there that seem to do the same thing. The success or failure of this site will be whether Midomi can effectively utilise all of this content to help visitors find music online and, eventually, sell them the end product at an affordable price.

Links:

Midomi.com