GrokCode

I ❤ Code Review

Thu, 08 Feb 2018 00:00:00 +0000

Code review has the proven benefit of reducing bugs, but there are so many more reasons to love code review. It is also a great way to improve code clarity, avoid the “hit by a bus” problem, learn new things, and solidify the team.

Find bugs

Catching bugs early is probably the most talked about benefit of code review. The sooner bugs are found, the easier, faster, and cheaper it is to fix them. So the best place to catch bugs is in automated testing, and the next best place is during code review while the code is likely still fresh in your mind. Once defects move into staging or production and are caught by QA or real users, chances are you have long since moved on to developing something else and have to task switch back in and refresh your memory on how that particular area of the code works.

There are a lot of statistics out there on the effectiveness of code review for detecting bugs. Here’s what Steve McConnell says in Code Complete.

The average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent.

Squashing bugs is important, but code review can do a lot more than that.

Improve clarity

Code review can do a lot to improve the clarity and general quality of the codebase. Sometimes code is unnecessarily complex and another set of eyes will see a way to simplify it. Other times the code resists all attempts at complexity reduction. If the reviewer is having trouble understanding it and complexity can’t be reduced, it is probably a sign that there should be more comments.

Another way clarity can be improved during code review is by communicating and enforcing coding standards. Most of these issues can be caught before the code review stage by using a linter as part of an automated test suite. This can catch a lot of different issues like misuse of whitespace, unused imports, malformed variables names, excessive line length, etc, but other coding standards are harder check for in an automated way.

Increase the “bus factor”

The “bus factor” is the minimum number of team members that have to suddenly disappear from a project before the project stalls due to lack of knowledge.

Having one person own a particular part of the codebase can be problematic if they aren’t available to help with modifications or bugfixes. Code review improves the bus factor because more team members have the opportunity to put eyes on each part of the code, ask questions, and think through how everything works during code review.

Image attribution: Adam Prescott

Learn new things

I’ve learned tons of new things by having my code reviewed by other developers. Getting specific feedback or seeing examples of better ways to organize code, increase performance, or simplify logic is helpful when honing your craft.

There is research showing that having someone critique your work is a valuable tool when working to become an expert. Anders Ericsson, a researcher who studies expert performance in all kinds of different domains, writes about the importance having someone who “understands and can demonstrate the proper way to perform various skills” and “who can provide useful feedback”. Peak: Secrets from the New Science of Expertise is full of Ericsson’s research-backed insight into becoming an expert.

On the flip side, It’s OK to stretch the definition of “review” so that reviewers can learn too.

re·view

a formal assessment or examination of something with the possibility or intention of instituting change if necessary.

Not every part of the review needs to be suggesting a change; asking questions is a good thing. Oftentimes the best way to learn is to ask. It could be general questions about the codebase, asking for more info on a particular library or tool, clarifying the need for a particular approach, or anything else.

With the philosophy that no question is out of bounds when giving a code review, it is easier to spread knowledge around and level up everybody’s skills. It also has the awesome side-effect of making code reviews less confrontational and more collaborative.

Praise the awesome

Code reviews are a great venue for showing appreciation for all of your awesome team members. Super elegant solution? Refactor that removed a whole bunch of cruft? Particularly gnarly bug squashed? Perfect test coverage? Code review is a great place to share the love with a shoutout or a sprinkling of emojis that show how you feel about a changeset.

Reviews for everyone, everywhere

With the widespread availability of excellent code review tools, there is really no reason not to do them. For people already using GitHub, pull requests are an easy, lightweight way to incorporate code reviews into the development process. But there are plenty of other tools out there that work with different types of source control.

Review everything, it’s worth it. ❤

Why Move From WordPress to Jekyll?

Thu, 11 Jan 2018 00:00:00 +0000

GrokCode has officially been a Jekyll site for a few months now. Why move from WordPress to a static Jekyll site hosted on GitHub Pages? Lots of reasons!

The Good

A Jekyll site offers a number of advantages versus a WordPress site. One of the biggest wins is being able to switch from a shared hosting environment running WordPress to a static site hosted by GitHub.

Static sites are faster than dynamic sites, and GitHub’s architecture has better uptime, a faster response rate, and is more scalable than shared hosting. It is also more hands off than managing a cloud hosted server. According to Pingdom, after moving from Bluehost to GitHub Pages, average response rates went from ~900ms to ~375ms, and there has been 0 downtime since the switch. This is pretty awesome, and the cherry on top is that GitHub Pages is free versus the $10 a month I was paying for Bluehost.

Another plus is I said goodbye to the last PHP environment I was responsible for maintaining. Sayonara PHP! After using WordPress on grokcode.com for nearly 10 years (wow!), I had accumulated a mishmash of custom plugins and PHP code. Moving to Jekyll was a great way to clean house and start fresh with Ruby and Liquid Templates. Unfortunately, I lost a few minor features along the way, but it looks like many of them are fixable and I’m hoping to implement solutions eventually.

Moving to a git workflow that uses all of my regular development tools feels a lot more natural than editing in WordPress. Having templates, posts, assets, comments, and configuration for the site all under source control in one git repo is a nice way to organize things and has the bonus effect of improving my GitHub vanity metrics.

What can I say, I love fake internet points!

The Bad

There is a lot to love about a Jekyll site running on GitHub Pages, but I ran into a few drawbacks too.

Customization takes time

First, switching does take time. The whole process took me several months of wall time, but it was a backburner project that I sometimes let sit for weeks or more. The most time consuming part was porting my existing WordPress theme to Jekyll instead of using one of the many off-the-shelf Jekyll Themes. This was a good way to learn liquid templating and level up my Sass skills though, so it was worth it. Plus I was able to keep the same look and feel.

Some features are not straightforward

Most of the features I used with WordPress can also be done with Jekyll, but often something that takes a few clicks in WordPress requires extensive customization to handle in Jekyll. One example is pinging aggregators when a new post is published. There is a way to do this, but it requires a custom Rakefile to generate the site instead of having GitHub do it.

Another drawback specific to GitHub Pages with custom domains is that it requires a bit of trickery and a third-party service like Cloudflare to use SSL. There is a good tutorial from Jonathan Petitcolas on setting that up. This is yet another item for my todo list.

Comments

The next big missing piece is a comment system. The most common solution is a JavaScript commenting system like Disqus, but the appearance and functionality isn’t customizable to the level I was looking for, and the import of existing comments was painful. Also having the site comments live on, and be controlled by, a third-party service didn’t sit right with me.

Enter Staticman. The comment form is set up to POST to the Staticman service, which will then open a pull request on GitHub with the content of the comment in a format that the template can display. This is very cool! But…I’m currently looking at 293 PRs for spam comments that I need to clean out. I tend to get a few of these a day, so it really needs a more robust solution. Staticman does have support for akismet, but unfortunately at the moment it requires running your own Staticman instance.

Migrating existing WordPress comments was possible after writing wordpress-to-staticman, a small conversion tool.

Staticman also supports more advanced features that I have yet to set up like threaded comments and email notifications for replys. Again, this requires some changes to my templates and configuration that take more time and technical fortitude to set up than they do with WordPress.

Was it worth it?

Definitely.

Pros:

Better uptime
More scalable
Cheaper
No more PHP (yay)
Everything in source control
Improves vanity metrics

Cons:

Switching takes time
Custom code or configuration required for many features
Comment spam

The improved performance, uptime, cost, and workflow far outweigh the cons. Plus I had a chance to learn a few things. GrokCode is happily a static Jekyll site hosted on GitHub pages.

Snakefooding Python Code For Complexity Visualization

Tue, 26 Aug 2014 13:51:03 +0000

Snakefood is a tool written by Martin Blais to create Python dependency graphs. Combined with GraphViz, snakefood can create beautiful visualizations of Python codebases. Here are graphs for some notable open source projects written in Python.

Python Web Frameworks

The different development philosophies of Bottle, Django, Flask, and Pyramid are apparent by looking at their snakefood graphs.

Bottle

Bottle is a fast and simple micro framework for Python web applications.

Bottle release 0.12.7

Django

Django is a batteries-included web framework for perfectionists with deadlines.

Django release 1.7c3

Flask

Flask is a microframework for Python.

Flask release 0.10.1

Pyramid

Pyramid is a small, fast, down-to-earth, open source Python web framework. It makes real-world web application development and deployment more fun, more predictable, and more productive.

Pyramid release 1.5.1

Queueing Implementations

Hat tip to Sylvain Zimmer for this deck on switching from Celery to RQ which was how I first found out about snakefood.

Celery

Celery is a complex feature-rich distributed queueing implementation.

Celery release 3.0.20

RQ

RQ is a simple Python library for queueing jobs and processing them in the background with workers.

RQ release 0.4.6

Other Python Codebases

For curiosity’s sake I ran snakefood on a few other notable Python codebases.

Twisted

Twisted is an event-based framework for internet applications.

Twisted release 14.0.0

Mercurial

Mercurial is a distributed source control management tool.

Mercurial release 3.1

Requests

Requests is HTTP requests for humans. It is frequently mentioned as an example of an elegant, Pythonic codebase.

Requests release 2.3.0

IPython

IPython is an alternative to the standard Python shell. If you aren’t using IPython, you should be. Here’s why IPython is awesome.

IPython release 2.2.0

What does this mean?

Snakefood graphs are helpful to visualize both the approximate size of a codebase and the level to which concerns are separated. As a measure of code complexity, snakefood graphs show more information than lines of code (which is not that great of a metric), but at the cost of introducing more room for interpretation than a strictly numeric metric.

The other problem with using snakefood graphs as a measure of code complexity is that it looks at code at the file level. A well-organized codebase containing many small files will look more complex than a codebase with a few very large but extremely long, complex files. Code with high testing coverage also tends to look more complex since the tests are dependent on so many parts of the code. Test and documentation modules were not included in any of the graphs above for this reason.

In cases where a more careful measure of code complexity is required, something like cyclomatic complexity is more appropriate. The graphs created by snakefood are actually similar to what would be used to compute cyclomatic complexity as applied to files instead of functions or classes which is the more usual way to do it.

Instead of using snakefood as a tool for visualizing complexity, the creator of snakefood uses it more to prevent unnecessary dependencies from creeping into the code:

Producing pretty graphs is fun, but I found the most leverage of it when I try to make my code simpler, I generate the graph and inspect unexpected dependencies and try to refactor my code to simply the dependency graph as much as possible (see furius.ca/beancount for a recent example). Enforcing rules can be done by writing a custom script against the snakefood output, as it’s very easy to accidentally introduce unwanted dependencies. I think the result is code that is better organized at a high level. –Martin Blais

So snakefood graphs are are (sometimes) beautiful. They are a fairly quick ad hoc way of visualizing code complexity, and the snakefood output can be used to help identify and enforce dependency relationships. All in all, snakefood is a pretty awesome tool to have in your Python toolbox.

How To Use Django’s Signing Functions For One Click Unsubscribes

Thu, 13 Feb 2014 16:04:27 +0000

Sometimes it is handy to allow users to change part of their profile without having to log in, as long as we can be reasonably sure of their identity. The most common use case is to unsubscribe from email newsletters, where it is annoying to have to login first.

The workflow we want to handle looks like this:

User clicks an unsubscribe link.
We verify the security token on the unsubscribe link.
The user gets a message saying they have been unsubscribed.

This workflow is pretty easy to set up using Django’s signing functions.

Creating the signed url

Let’s use Djanogo’s built in signing functions to create an unsubscribe link with a security token, and then create a way to verify the token. You can tuck the following code into the user profile.

from django.core.signing import TimestampSigner, BadSignature, SignatureExpired

def create_unsubscribe_link(self):
    username, token = self.make_token().split(":", 1)
    return reverse('user_signups.views.unsubscribe',
                   kwargs={'username': username, 'token': token,})

def make_token(self):
    return TimestampSigner().sign(self.user.username)

def check_token(self, token):
    try:
        key = '%s:%s' % (self.user.username, token)
        TimestampSigner().unsign(key, max_age=60 * 60 * 48) # Valid for 2 days
    except (BadSignature, SignatureExpired):
        return False
    return True

As long as the SECRET_KEY from django.conf.settings is kept private, it isn’t feasible for someone malicious to craft signed links.

While verifying the token in the unsubscribe link, we want to strike a good balance with user friendliness and security. In the common case, a user shouldn’t have to login to unsubscribe. But we also want to prevent users from being accidentally unsubscribed by others if they forward the email or otherwise make it public. Adding an expiry date to the token somewhat mitigates this risk. Above we use a 2 day expiry date.

The url pattern

Now hooking up the url pattern which accepts the special characters that may appear in the token:

url(r'^unsubscribe/(?P<username>[\w.@+-]+)/(?P<token>[\w.:\-_=]+)/$',
     'user_signups.views.unsubscribe'),

The view

def unsubscribe(request, username, token):
    """ 
    User is immediately unsubscribed if they are logged in as username, or
    if they came from an unexpired unsubscribe link. Otherwise, they are
    redirected to the login page and unsubscribed as soon as they log in.
    """

    user = get_object_or_404(User, username=username, is_active=True)

    if ( (request.user.is_authenticated() and request.user == user) or
         user.get_profile().check_token(token)):
       # unsubscribe them
       profile = user.get_profile()
       profile.newsletter = False
       profile.save()

       return render(request, 'registration/unsubscribe.html')
    
    # Otherwise redirect to login page
    next_url = reverse('user_signups.views.unsubscribe', 
                       kwargs={'username': username, 'token': token,})
    return HttpResponseRedirect('%s?next=%s' % (reverse('login'), next_url))

The templates

In the email templates, use the unsubscribe link {{user.get_profile.create_unsubscribe_link}}.

Then in registration/unsubscribe.html add a message confirming that the user has been unsubscribed.

Other notes

This type of signed link could also be used to improve the customer’s flow when upgrading SaaS plans, participating in a survey, or performing some other action that updates their profile when they click on a link in an email. Depending on what they are doing, you can add an additional security layer by checking that the ip address is one they have previously used while logging in.

You Should Change Your Python Shell

Mon, 16 Dec 2013 13:09:59 +0000

If you write Python code, switching to IPython is the number one thing you can do to immediately improve your productivity. Bold words, I know. Let’s look at how IPython can make you a more productive programmer.

If you are mostly a web developer or are new to Python, the IPython homepage does not make it obvious how awesome IPython is (even if you don’t do any scientific computing!). If you have never used IPython before, forget about notebooks, data visualization, embeddable interpreters, and parallel computing. It’s all great stuff. But just forget about it for now.

The absolute best thing about IPython is the shell. Here is a standard Python shell.

And below is IPython. They look pretty similar, but IPython has a whole bunch of extras that will save you a vast amount of time and give you an immediate bump in productivity.

OK so easy_install ipython[all] or pip install ipython to get started. If any issues crop up, refer to the official install guide.

Now let’s look at some of IPython’s most useful features that are applicable to everyone who uses a Python shell.

Persistent History

Command history is preserved when IPython is restarted, so you can use the up key to retrieve commands from the previous session. This will cut down on a lot of typing if your workflow looks like this:

Test something in the shell
Use an editor to make some code changes
Exit the python shell
Rerun the python shell so that the new code is loaded
Retype the test code
Test something in the shell again

But actually, if you are using IPython, you won’t need to restart the python shell either. You can just enable autoreload.

Automatic Module Reloading

IPython will automatically pull in changes to your code files. To enable it:

In [1]: %load_ext autoreload
In [2]: %autoreload 2

The workflow from above now looks like this:

Test something in the shell
Use an editor to make some code changes
Test something in the shell again

If you want autoreload enabled by default, you can edit your IPython profile to contain these lines:

c.InteractiveShellApp.exec_lines = []
c.InteractiveShellApp.exec_lines.append('%load_ext autoreload')
c.InteractiveShellApp.exec_lines.append('%autoreload 2')

Builtin Profiling Tools

IPython comes with some “magic functions” to profile execution time and memory.

%time times how long it takes to run a statement.

%timeit times how long it takes to run a statement, but will average over many runs. It is smart enough to use a lot of runs for statements that don’t take very long, but only a few runs when execution time is high.

%prun gives a breakdown of execution time per function.

%lprun gives a breakdown of execution time per line.

%mprun gives a breakdown of memory usage per line.

%memit gives a breakdown of memory usage averaged over many runs.

%mprun, %lprun, and %memit require some extra dependencies and a bit of configuration to enable, see details here.

Smart Copy and Paste

You can paste chunks of code directly into IPython and it will properly handle the indentation. Even if each line has leading indents. So, you can copy and paste a few lines from the inside of a function or loop and IPython will ignore the leading indents that aren’t meaningful, but preserve the ones that are. Use %paste and/or %cpaste.

Smart Indentation

IPython will start the next line in the right place when you do a carriage return after a function definition, loop, if statement, etc. This is a nice convenience that saves a few keystrokes.

Tab Completion

You can use tab completion to explore objects and autocomplete keywords.

Just type exit

In the standard Python shell, you have to type exit(), but in IPython if you forget the parens and just say exit, that’s fine too. It’s a small detail, but details matter.

IPython has loads of other features, some will likely be useful to you, others maybe not so much unless you do scientific computing. This article just touched the very tip of the iceberg. To learn more about what IPython can do, have a look at Learning IPython for Interactive Computing and Data Visualization and the IPython Cookbook.

Deploy with ‘Git Push’

Mon, 05 Nov 2012 07:08:29 +0000

Single command deploys are awesome. When setting up a testing or production server, I always setup a script to deploy in a single command because it makes life easier and encourages frequent deploys.

There are many different ways of setting up one click deploys. This tutorial covers how to setup a git postcommit hook that deploys on git push prod, which is pretty simple to get started with, but won’t scale well once you start running multiple app servers. For alternatives, you might have a look at Fabric or Capistrano.

Setup the git repo on the server

If you don’t already have git installed on the server, ssh in and do a

sudo apt-get install git

Now we want to create a git user:

sudo adduser \
    --system \
    --shell /bin/sh \
    --gecos 'git version control' \
    --group \
    --disabled-password \
    --home /var/git \
    git

And append your ssh public key to the git user’s list of authorized keys located at /var/git/.ssh/authorized_keys, creating the file if necessary.

Now let’s create the repository:

cd /var/git
sudo -u git mkdir <repo-name>.git
cd <repo-name>.git
sudo -u git git init --bare

The repository here is bare so that it can accept pushes. See this discussion of bare vs. non-bare repos if you are interested in learning more.

Create a hook

OK so now we want to create the script that will run after a git push is received.

Create a file called /var/git/<repo-name>.git/hooks/post-receive and give it these contents:

#!/bin/bash
export GIT_WORK_TREE=/var/www/<app-name>/project/
git checkout -f

The above assumes that your server is setup with /var/www/<app-name>/project/ as the location where your app server will be looking for your code files.

Add execute permissions and make sure it is owned by git:

sudo chmod u+x hooks/post-receive
sudo chown git:git hooks/post-receive

Handle permissions

The post receive script will be run as git, so we need to make sure that the git user has permissions to write to the GIT_WORK_TREE, and also that the server can read and write from those directories. To handle this, lets make git a member of the nginx group, and then have the post receive script touch up the file permissions after checkout.

sudo usermod -a -G nginx $USER

Then edit the script to look like this:

#!/bin/bash

# Checkout the repo.
export GIT_WORK_TREE=/var/www/<app-name>/project/
git checkout -f

# Fix up permissions.
cd /var/www/<app-name>;
chmod -R g+w project/*
chown -R git:nginx project/*

Prepare development box

Now on your development machine, add the new remote repository and push to it:

git remote add prod ssh://git@<server>/var/git/<repo-name>.git
git push prod master

Other deploy tasks

Often there will be other tasks that have to happen during the deploy. Here is an example of another post-receive hook script that I use for one of my Django projects. It collects the static files to prepare them to be served via nginx, runs the database migrations, and installs any new python requirements.

#!/bin/bash

# Checkout the repo.
export GIT_WORK_TREE=/var/www/<app-name>/project/
git checkout -f

# Fix up permissions.
cd $GIT_WORK_TREE
chmod -R g+w *py
chown -R git:nginx *py

source ../env/bin/activate

# Collect static files
./manage.py collectstatic --noinput -v 0

# Handle DB Migrations
./manage.py syncdb --migrate -v 0

# Update requirements
pip install -q -r requirements.txt

deactivate

So there it is – a quick way to setup one touch deploys with git.

How to Setup a Linux, Nginx, uWSGI, Python, Django Server

Wed, 25 Jul 2012 21:45:12 +0000

This is a tutorial for setting up a Linux, Nginx, uWSGI, Python, Django server with a PostgreSQL database. This is the easy, straightforward approach to server setup and deployment. The intended audience is developers or teams who need to get an application or two running in production without too much hassle.

If you need to oversee multiple servers or you are handling a large number of applications, the limitations of the setup described here will start to show through. Here is an example of a much more complex deployment workflow that is more appropriate reading for dedicated DevOps people.

General server setup

I’m using Linode running Ubuntu 12.04 LTS, but you should be able to adapt this tutorial to other hosts and distros as needed.

Before getting started, take steps to update and secure the server. Security is beyond the scope of this article, but if you don’t know where to start, have a look at this tutorial.

Install uWSGI

Let’s get started by installing uWSGI. ssh into the server and

sudo apt-get install uwsgi uwsgi-plugin-python

Create a user to run uWSGI:

sudo useradd -c 'uwsgi user' -g nginx --system \
--no-create-home --disabled-login --disabled-password uwsgi

Create an upstart configuration file at /etc/init/uwsgi.conf which looks like this:

description "uWSGI"
start on runlevel [2345]
stop on runlevel [06]

respawn

exec uwsgi --master --processes 4 --die-on-term --uid uwsgi --gid nginx \
--socket /tmp/uwsgi.sock --chmod-socket 660 --vhost --logto /var/log/uwsgi.log \
--plugins python

Install Nginx

The version of Nginx provided in the official Ubuntu repository is a bit outdated, so we are going to use the latest stable package from the Nginx repository. If you are on Ubuntu, just follow the instructions below, but if you are on another distro or run into trouble, have a look at the official install guide.

First we add the key

wget http://nginx.org/keys/nginx_signing.key
sudo apt-key add nginx_signing.key

Then open up the /etc/apt/sources.list file and append the following:

deb http://nginx.org/packages/ubuntu/ precise nginx
deb-src http://nginx.org/packages/ubuntu/ precise nginx

If you aren’t using Ubuntu 12.04, change precise to the the codename for your Ubuntu version.

Then install Nginx with:

sudo apt-get install nginx

Nginx is already running. To verify, open your server’s ip address in a browser, and you should see a ‘Welcome to nginx!’ screen.

Nginx config

The webapp is going to live at /var/www/<app-name>/. Here is what the directory tree will look like when we finish:

/var/www/<app-name>/
    |-- env/                    The virtual env
    |-- project/                The Django project files
        |-- <apps>/
        |-- <project>/
            |-- wsgi.py
        |-- requirements.txt    Pip project dependency list
    |-- static/                 Static files to be served by nginx     

So let’s setup that directory structure.

sudo mkdir /var/www/<app-name>
sudo mkdir /var/www/<app-name>/project
sudo mkdir /var/www/<app-name>/static

Now to setup the virtual environment for our application:

sudo apt-get install python-pip python-setuptools
pip install virtualenv
cd /var/www/<app-name>
virtualenv env

Now we need to fix some permissions so that uWSGI can read scripts and write the .pyc files:

sudo usermod -a -G nginx $USER
sudo chown -R $USER:nginx /var/www/<app-name>/
sudo chmod -R g+w /var/www/<app-name>/

Create the file /etc/nginx/conf.d/<app-name>.conf with these contents:

server {
    listen          80;
    server_name     $hostname;
    location /static {
        alias /var/www/<app-name>/static;
    }
    error_page   404              /404.html;
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
    location / {
        include         uwsgi_params;
        uwsgi_pass unix:/tmp/uwsgi.sock;
        uwsgi_param UWSGI_PYHOME /var/www/<app-name>/env;
        uwsgi_param UWSGI_CHDIR /var/www/<app-name>/project;
        uwsgi_param UWSGI_MODULE <project-name>.wsgi:application;
    }      
}

Don’t forget to swap out <app-name> for the name of your app in lines 5, 15, and 16, and use your own project name on line 17.

In this configuration, nginx will serve files in the static directory, but pass off everything else to uWSGI.

Now let’s move the default config out of the way, so the new config can take it’s place.

cd /etc/nginx/conf.d
sudo mv default.conf default.conf.bak

OK let’s start things up:

sudo service uwsgi restart
sudo service nginx restart

If everything has be setup properly so far, navigating to the IP address should now show a uWSGI error.

Install the Django app

Now install your Django app into /var/www/app-name/project/ For now, just get a clean copy out of source control and scp it to the server, or grab it directly from source control while on the server. Later, you should setup a deployment script to automate this, but for now let’s just get the app running.

In this case, I am keeping my code in a Bitbucket git repository, and I want to pull the code directly from Bitbucket to the server. So on the Bitbucket site, I create a new user with read only permissions on the code repository. Then I create a new ssh key on the server

ssh-keygen -t rsa -C "<email address>"

and add it to the Bitbucket account for the new user by uploading the id_rsa.pub file.

Now to install git

sudo apt-get install git

And pulling the source code out of the repository now looks like this:

cd /var/www/<app-name>/project/
git archive --format=tar --remote=ssh://git@bitbucket.org/<username>/<repo name>.git master | tar -xf -

Then fix up any permissions issues again:

sudo chown -R $USER:nginx /var/www/<app-name>/
sudo chmod -R g+w /var/www/<app-name>/

Install any packages required by your app or it’s dependencies. For example, psycopg2 and lxml respectively depend on the following packages:

sudo apt-get install python-dev libpq-dev
sudo apt-get install libxml2-dev libxslt-dev

Now install the project requirements into the virtual environment:

source /var/www/<app-name>/env/bin/activate
pip install -r /var/www/<app-name>/project/requirements.txt
deactivate

OK so now that the Django code files are on the server and the dependencies are satisfied, navigating to the IP address in your browser should show your app. Static files aren’t being served yet and the database isn’t installed, but we can see some progress.

If you haven’t got this far, the log files for upstart, nginx, and uwsgi are a good place to start troubleshooting.

In order to have Nginx serve the static files, they need to be collected to /var/www/<app-name>/static. In settings.py point your STATIC_ROOT to that static directory, then verify that STATIC_URL and STATICFILES_DIRS are properly set up. Get any changes from the repository, then:

source /var/www/<app-name>/env/bin/activate
cd /var/www/<app-name>/project/ 
python manage.py collectstatic

Test it out in your browser and you should now see css, javascript, and image files being loaded. However, without the database, your app probably isn’t doing much of anything. So let’s fix that.

Installing PostgreSQL

sudo apt-get install postgresql postgresql-contrib

Then setup passwords:

sudo passwd postgres
su postgres
createuser django --pwprompt
psql create database <dbname> owner django
exit
sudo /etc/init.d/postgresql restart

Now initialize the database:

cd /var/www/<app-name>
source env/bin/activate
cd project
./manage.py syncdb

If you use south for database migrations, run those now.

./manage.py migrate <name>

Your Django app should now be up and running.

Closing remarks

Now that the server is ready, setup one click deploys to speed up your code – test – deploy cycle.

Support the creation of tutorials like this one by using the following links to purchase Linode hosting or shop at Amazon.

A big thanks to Conrad Kramer, Collective Disorder, and Simon Westphahl for guiding the creation of this tutorial.

Dear Python, Why Are You So Ugly?

Thu, 26 Apr 2012 12:04:02 +0000

Dear Python, something has been bothering me for a while. Its just that, well, ummm…you’re kind of ugly. Look, you are beautiful inside: Python is a beautiful language and the Python community is open and welcoming. But Python resources are ugly enough to affect usability and adoption. This is damaging to the community.

Documentation and tutorials are often difficult to navigate. Products built with Python don’t put any thought into design. Blogging software written in Python helps create more ugly blogs about Python. Pythonistas just don’t care about presentation.

Let’s compare the usability and design of some different Python and Ruby sites. Ruby is a good reference point since it occupies a similar niche in the programming language ecosystem and is roughly the same age.

Above are online interactive tutorials for learning Python vs. Ruby. Learnpython.org isn’t ugly exactly, it’s just that Try Ruby is so much more polished. It’s obvious that a professional designer had a hand in Try Ruby, whereas the design of learnpython.org was thrown together by a programmer who later slapped a few ads on it and called it done. Which tutorial would you rather use?

Here are the homepages for Django and Ruby on Rails, popular web frameworks. Django’s homepage is poorly organized. The entire page is just a mess of links with no clear emphasis on anything. In contrast, the Ruby on Rails homepage does a good job of introducing people to Rails and pointing them to different areas of the site.

Homepage of Django co-creator Adrian Holovaty vs. Ruby on Rails creator David Heinemeier Hansson. Both are well organized, but it is obvious at a glance who places importance on professional design and who doesn’t.

Cloud hosting: Google App Engine vs. Heroku. Once again the Ruby side is sexier.

Python blogging software vs Ruby blogging software. The Python blog designs are uninspiring and unpolished, while the Ruby designs are striking. And who doesn’t love octopodes?

Some top Google search results for “python consulting” vs. “ruby consulting”. The Python results look painfully outdated.

Maybe Ruby has an advantage because it is more tightly focused on web programming? It seems likely that consultants for web apps would have nicer web pages than consultants working on, say, scientific problems. To eliminate any unfair advantage, below are some top Google search results for “django consulting” vs. “ruby on rails consulting”.

It doesn’t matter. Python is still the ugly duckling.

This wasn’t a scientific comparison – somebody could try show the opposite conclusion by cherry-picking a different set of sites – but I think the screenshots I used are pretty representative. I also think that most people who have spent a little bit of time in both communities will agree that Python sites are ugly compared to Ruby sites.

Why does this happen?

Either Pythonistas aren’t pairing with designers as often as Rubyists, Pythonistas have less design talent, or Pythonistas simply don’t care enough about design to spend the time to do it right. I’m not really sure which is the case.

Does it matter?

Yes! This stuff matters.

It’s not just that Python sites are ugly (even though they are). It’s that the uglyness makes sites hard to navigate and hard to use. It’s that nobody is inspired by uglyness and nobody wants to use ugly products when there are better options. Nobody wants to hire someone who builds ugly web apps. It is unprofessional and sloppy to be so ugly.

This damages the Python community.

How can we fix it?

If you are a Python coder (or any type of coder who doesn’t fully grok design), spend some time learning basic design principles. Try Design for Hackers, Bootstrapping Design, or Step by Step UI Design. [Edit: Also check out this lightening talk on attracting designers to your project.] Then make friends with designers and know when to call them.

Startup Sheep Vs. Non-Startup Goats (Or Transitioning From Coder to Founder)

Thu, 29 Mar 2012 12:23:32 +0000

There is some famous research, by Saeed Dehnadi and Richard Bornat, about “programming sheep and non-programming goats.” The gist is that educators find that there are two populations of students, those who can program, and those who can’t. Each population has it’s own independent bell curve. This “double hump” persists despite variations in programming language, application type, IDE, and student motivation.

So, there are natural born programmers. And there are natural born non-programmers.

Most natural born programmers notice this at some point (consciously or not), but often the conclusion they draw from it is overly general. Coders who can code may naively expect to found startups that, ermm, start. Or they may expect to be naturally good at other things that are only tangentially related to coding.

Lets take the example of a founder working on a software startup. Our founder is a programmer, at the top end of the programming bell curve. He is a really smart guy, he has done some reading on startups, but he hasn’t launched anything before, and he is entirely unaware that he has developed a severe case of shit’s easy syndrome regarding the non-programming tasks that need to get done before launch.

As programmers, we learn to be obsesively paranoid with our requirements analysis, timelines, and code otherwise the project will run off the rails. But we don’t apply that same rigor for business tasks, because well, that shit’s easy, right?

Turning a project into a product – that shit’s not easy

Packaging a programming project into a product that can be sold is a complex process. Assuming that the programming side is a beautiful finely-tuned instrument ticking along perfectly, there are still a behemothic number of things that need to be done to get it into the hands of paying customers, and each decision that needs to be made opens up into a fractally expanding rabbit hole leading to an infinite number of other decisions.

It starts off innocently enough: OK I need a website to sell this thing. How about a site with a free theme plus some marketing copy explaining the product tied into a simple shopping cart so customers can buy the product.

Which quickly turns into: How about a site with a free theme (which themes are optimized for SEO, which are optimized for conversions, how about some A/B testing, hey why is my site full of spam¹ plus some marketing copy explaining the product (what kind of copy sells, OK “benefits” are better than “features”, but now what are the product’s benefits…) tied into a simple shopping cart (secure digital delivery, payment processing, cart software, should I write some of this myself, will this stack work with an affiliate program…) so customers (who exactly are my customers, what “tribe” are they in, how do I find them, should I try advertising, where…) can buy the product. And spirals out of control from there.

It’s becoming obvious how “hello world hooked up to a random number generator” requires a super-smart, full-time guy doing the business tasks.²

Meanwhile, all of the startup guru expert guys are in my head screaming, “JUST LAUNCH! Iterate later,” but at this point I have lost all sense of what is absolutely necessary to prevent everything from blowing up in my face, what will just make me look stupid if it breaks, and what I can launch without and add later.

This launch paralysis is a problem for startups on down to micro-startups and coders that are just trying to make a bit of side money off of a weekend project. Programmers are probably more susceptible to launch paralysis because we are used to just naturally ‘getting it’ (remember, we have our own bell curve), and when confronted with mushy tasks like research or copywriting that don’t require the rigor of programming, we tend to fall into the shit’s easy trap.

A Tiny Example

My idea was to launch a weekend project, and use the experience to identify problems and iron out glitches in my workflow before tackling something bigger. I uncovered more than just glitches. Everything took drastically longer than expected, and some things I didn’t plan for at all.

The graph above is data from a tiny project that makes some head-smackingly simple changes to Amazon links so that WordPress blogs earn worldwide commissions instead of just US commissions. I assumed it would be easy to sell since the purchase is so easy to justify – once a blog has a certain threshold of earnings from amazon.com, the plugin is pretty much guaranteed to pay for itself. If you are interested, ~~click here to see how the plugin increases earnings~~. Now defunct.

This wasn’t a startup. Calling it a micro-startup would be generous. It’s a WordPress plugin. We’re talking 700 lines of code, tops. But it still took 5 months to launch it. I was only working on it part time, but still, the number of hours spent getting ready to launch such a simple project is staggering.

Much of that time was spent researching different tools, not actually integrating them. Luckily, most of that knowledge will transfer to future projects. Now I have a standard toolchain that handles design, digital delivery, payment processing, A/B testing, etc. I will be able to put this type of site together much faster in the future.

So, it finally launched. Sheep or goat?

Sales so far have been lower than expected, and certainly less than spectacular. At times it feels a lot like a big goat failure. But there are still some sheep to be found. After stepping back for a bit of perspective, I remember that this little failure created some big wins. First, all of that research gave me a standard toolchain for launching mini software startups (see my toolchain here). Next time things will move much faster. Second, even though the plugin hasn’t earned much through sales, having it installed on my own sites has given a nice bump to passive income.

The launch isn’t really the end of the story. After launch comes a whole new rabbit hole of customer acquisition and iterating the product. It’s a bit too early to make a success or failure call. Everyone says that an overnight success takes years.

Edit: This project now defunct. I still use the plugin on my own sites where it has increased earnings a fair amount, but the support and website maintenance overhead associated with selling the plugin to others was not worth my time.

Why you should never search for free WordPress themes ↩
This is in reference to Patrick McKenzie‘s Bingo Card Creator software, which is technically pretty unsophisticated itself, but is supported by complex mechanisms for A/B testing, scalable content generation, conversion tracking, and various other types of optimization. ↩

Launching Downloadable Products Quickly

Tue, 27 Mar 2012 15:35:33 +0000

My intent is to give recommendations that will be of use to other programmers who are trying to quickly turn a side project into a professional product that they can sell.

Assuming you already have a working side project or product (no small feat in itself, but not difficult for a coder to put together), the biggest missing piece is a website for getting that project into the hands of customers. It’s easy to underestimate how much work it takes to do this part well if you have never done it before – design, conversion optimization, payment processing, secure digital product delivery, analytics, affiliate programs…

Researching this stuff is a huge time suck. Here are the tools I recommend for getting a professional site done quickly. This setup has no monthly fees and low per transaction fees, but does require some glue code to get working.

ThemeForest is an online marketplace for site templates and themes.

Even if you plan on moving to a custom design later, using a pre-built theme at first will save time and let you launch sooner. Plus the initial feedback from customers who bought the product on the themed site will help to guide the custom design.

I prefer ThemeForest because all themes are manually reviewed by the ThemeForest staff before being added to the store – this means no secret spam links or malicious code embedded in your themes. (This is a huge problem with free themes.)

Professional looking themes typically sell for between $8 and $15, which is a pittance for the time it will save if you were to code it yourself. Browse themes here.

A stock theme will still need some customization. For custom graphics, try Fivvr.

I add a few extra pages to the theme to handle checkout: a payment page, a verify purchase page, and a thank-you or payment received page. These pages contain a bit of custom glue code to use my preferred payment processor, digital delivery method, and affiliate program tracker. The other benefit of handling these pages myself, instead of passing control off to an external cart or payment processor, is that it allows more control over the checkout process. This is valuable for gathering data about cart abandonment rates, providing opportunities for upselling, and optimizing these pages for conversion.

Previously, writing or hosting those pages yourself would be a huge security risk and a PCI compliance nightmare, but with Stripe to do the heavy lifting it’s a secure option.

Stripe makes it easy to accept credit cards on the web.

Payments take place through stripe.js, so your customers never leave your site during the checkout process, and you skip most PCI requirements since the transaction itself happens on Stripe’s servers, not yours.

The API is a joy to work with, and there are Stripe employees hanging out in the Campfire room at all hours of the day and night who can answer integration questions.

Cost is 2.9% plus $0.30 per successful charge. There are no hidden fees. Earnings are transferred to your account an a 7 day rolling basis.

Unlike most forms of accepting online payments, the sign up process is painless, and you can start with the test API right away. Sign up for Stripe.

Once payment has been processed, FetchApp provides secure delivery of digital files.

Each purchase triggers the creation of a unique download link. You can configure the length of time the link is valid and the number of times it can be used.

There is a simple online interface and an equally slick API to manage products, orders, and customers. The feature set includes the ability to send free trials and notify customers when a new version of a product is available.

FetchApp integrates with a number of shopping carts and payment providers out of the box, but my setup uses the API.

Pricing is based on amount of storage required, not number of sales. The free plan covers up to 1MB. FetchApp sign up page.

The above services mixed with some good marketing copy will get you up and running, but you will certainly want to set up a few extras like analytics, A/B testing, and possibly an affiliate program.

Google analytics is the standard choice for analytics – it is easy to setup and provides tons of useful information.

Google Website Optimizer provides A/B and Multivariate Testing.

There are a number of different options for this type of testing. I prefer GWO because it is easy to setup and the reports are simple to understand.

Another avenue for increasing sales is to set up an affiliate program, giving people who promote your product a cut of the sales. Companies like Commission Junction and ClickBank will handle this – they help affiliates find your program and track their sales.

However I don’t recommend either of those programs. Their fee structures are too complicated, and affiliates are often hit with hidden fees. Also, they use “hop links” to track purchases. These “hop links” don’t provide SEO value and look suspicious to customers (a typical link will look something like http://chdxsld.com/kdkd2002kdk before the customer is redirected to your site).

A self-hosted affiliate program will avoid those problems – you can customize the payout percent and affiliate links will look something like http://example.com/ref=ID, but the drawback is you need to publicize the program yourself and handle the affiliate payments.

I have yet to find a good affiliate solution, self-hosted or otherwise. Anyone have recommendations?

Bonus recommendation: if you intend to support a fledgling startup with contract work on the side, or if charging for support is part of your revenue model, Freshbooks is the best option I’ve seen for creating professional invoices.

The Freshbooks web app is an absolute pleasure to work with.

Pricing is based on the number of clients, with up to 3 clients on the free plan. Use this invite to sign up a free 30 day trial of Freshbooks.

You can see this software stack in action at ~~Geotargeter for WordPress – a plugin for Amazon Affiliates~~. Edit: Now defunct.