Prosthetic Conscience

Stockphoto 0.2.1 released

I’ve just released stockphoto 0.2.1. This is a bugfix release and contains no new features relative to 0.2. I would like to thank many people for bug reports on the previous version; plese see the credits in the README file.

I’ll be opening development on stockphoto 0.3 once I decide the best way to host a public version control repository (Google code vs. Savannah, vs self-hosting). The pre-0.3 branch will include new features, some of which are listed as to-dos in the current README.

[ Posted: 15:11] | [ Category: ] | Permalink | Comments: 0 ]

Yikes! Stockphoto on Django status update!

The last couple of days I have been seeing a renewed interest in Stockphoto, my basic, minimalist photo gallery application for Django. At the time, I wondered why, since I haven’t updated Stockphoto in the last year, nor made any announcements associated with it. Then, I saw that it had been mentioned in the Django status update on July 22.

This is just a little notice on stockphoto’s status. Stockphoto 0.2 has quite a few known bugs. I have to-do items for all of them in my personal organization system, most of them with contributed patches. I’ve just been too busy in my life to integrate all of these fixes and package them into a release. I want to do better and get two releases out the door soonish.

So here’s the release plan:

1. Stockphoto 0.2.1, soonish. Will fix all outstanding bugs that I know about.
2. Stockphoto 0.3, later. Will include changes to the models to allow slug-based rather than id-based URLs, and some conveniences for template designers.

Please bear with me – if you are in desperate need of a full-featured Django photogallery in the very near future, you will probably want to write it yourself, possibly looking at stockphoto’s code for examples if you are not familiar with Django’s file/image upload handling or with PIL.

Thanks for your patience.

[ Posted: 07:30] | [ Category: ] | Permalink | Comments: 0 ]

Web templating systems: be opinionated enough, but no more.

There’s some current discussion in the Django community on templating systems on Jacob Kaplan-Moss’s website and elsewhere.

I agree with the general argument. There is a rather fine line between putting too much power into the templating language, so that it either becomes a programming language in its own right, or simply a means of embedding primary-language code in the templates, and putting in too little, so that you can’t do more than variable substitution. That’s not a big insight in and of itself. I think almost everyone knows this by now.

What I think is significant is this:

1. Django’s templating system hits the sweet spot for web templating. Through a combination of good taste and (probably) luck, the Django developers have produced a templating system that strongly encourages web developers to put their controller logic and their view logic in the right place.
2. Django’s templating system is just tightly coupled enough to make people think twice before using something else. You certainly can use other view technologies, and it’s not really hard to do so (just do everything your other template system to produce a string or a file-like object representing the content of the response, and pass it as an argument to the Django response object you’re going to return).

   But neither the documentation nor the code for Django imply anywhere that other templating systems are just as good, just as suitable, or just as well-integrated into Django. The documentation mentions that you can use them, but only gives examples of the Django templating system. The code contains many shortcuts for using the Django templating system (e.g. render_to_response()) that can’t be used with other systems.

I think the latter point should really be emphasized more. The core Django developers like to talk a lot about how Django is loosely-coupled. And that’s sort-of true, and it’s a good thing. You don’t have to buy into everything to use Django. Django applications are just plain python code; they’re written not a special mini-language. But, Django is tightly coupled enough that there is only one obvious way to do it.

Here’s what I mean by that. In my work life, I’ve been learning a more corporate-friendly web development platform, the Spring framework for Java. Mostly Spring is a good thing. It manages to drag corporate web development halfway from J2EE-hell to Django in the same way that Java dragged corporate programmers halfway from C++ to Lisp. Once you get past the Enterprise Architecture Astronautics (not as bad in Spring as in other Java frameworks, because of the extensive use of interfaces rather than inheritance) and reams of XML configuration files (apparently significantly reduced in Spring relative to other Java frameworks!) it’s actually not bad, and developing for Spring MVC is very much like developing for Django while wearing mittens – you use the same patterns, but it takes twice as long to write.

But, the frustrating thing about Spring is that it is completely non-opinionated. How shall you do database access? They don’t care, and not only do they provide wrappers for the top 10 ORM systems as they’ll as plain JDBC, they support multiple completely different styles of using one of them, which require different layers of proxy objects and (of course) different XML configuration. And view technologies? They don’t much care which of those you use, either – pick one and use it. All this may make sense in an environment where the ORM layer and view layer can be mandated by corporate policy, but it’s no way to build a framework developers can be passionate about.

(Side note: for the templating language to use with Spring, I’ve settled on Apache Velocity. It is much like the Django templating language in scope, in that it interpolates variables and their properties, offers conditional constructs and iteration over collections, and so forth, but does not constitute an entire embedded programming language, or make it too easy to drop into the implementation language. Like Django’s templating language, it is a plain text language that is not tied to generating XML output. It doesn’t have two of the nicest features of Django’s template language, template inheritance and block substitution, but these are very easy to implement yourself by following certain conventions. Velocity is much closer to the templating sweet spot than JSP/JSTL.)

Django has been accused by passionate users of other Python web application frameworks of suffering from NIH Syndrome; or to put it another way, of being too opinionated. But the NIH components of Django (the ORM and the template system) provide very good implementations of that functionality for most of the target audience. If your needs are complex enough that you have to use something else, then you probably also have the skills to deal with the consequences (for example, SQLAlchemy can map existing databases that Django’s Models can’t – but if you use it, your app doesn’t get Django’s admin functionality, and probably can’t use generic views). So it goes. But the risks of being not opinionated enough are worse:

• The framework is harder to learn, because the documentation of different components may be spread across multiple projects, and new developers may have to learn multiple components in the same category in order to choose which best suits their project, instead of just diving in with a well-designed default.
• Different projects using the same framework will tend to be written in different dialects or subsets of the framework (e.g. Hibernate-Spring-Velocity vs. JPA-Spring-JSF).

So it’s not just that Django’s templating system hits the big bulgy middle of the bell curve, but that the project as a whole does. Opinionated, but not too opinionated. Enough salt to bring out the flavors, not enough to overwhelm them.

[ Posted: 09:56] | [ Category: ] | Permalink | Comments: 3 ]

Installing Django as CGI

When Django was first released, it was only straightforwardly possible to deploy it using mod_python, for which it was designed. However, it also soon included an adapter for WSGI, the Python standard for web application servers to interface with web servers. The WSGI interface was combined with a WSGI-fastcgi gateway called flup to make it possible to host Django applications on FastCGI, and this support eventually became fully integrated with Django. FastCGI is as fast as mod_python, and is somewhat more commonly deployed, especially on commodity shared web-hosting providers.

It is also capable of running the Django app under a different user identity from the web server, either through a suexec wrapper, or by being explicitly started as a server process by another user.

However, while FastCGI is widely deployed, it is not universally deployed, and many hosting providers that support it do so poorly. This article explains how to use the Django WSGI adapter to run Django as a pure CGI application. Performance with this method is terrible, but it might still be suitable for low volume sites on commodity hosting, and especially given adequate caching. It is also a convenient harness for testing, especially for sites to be deployed as FastCGI, and it provides the same separation of privileges from the web server that Fast CGI provides.

Installing a Django project as CGI is very similar to installing it as FastCGI. These instructions presume the use of Apache 2.x. Put this script in your cgi-bin directory. It is derived from the example cgi-wsgi gateway in PEP 333. There’s a similar implementation in Django’s ticket # 2407, but I didn’t know about this when I started writing this note. (I don’t claim any great originality for this method; I just threw together existing components and banged on them until they worked.) Adjust the paths to match where you have installed your project (outside your document root, hopefully!).

Then, you install an htaccess file like this into your DocumentRoot:

This will redirect all requests under your DocumentRoot except for cgi-bin, media, and files that actually already exist, to your django cgi script. Again, you will probably have to adjust these paths for your own circumstances, and you may even have to add a RewriteBase directive or something as well.

Now test your admin app. Everything should work correctly, but slowly.

[ Posted: 19:30] | [ Category: ] | Permalink | Comments: 2 ]

Stockphoto 0.2 released

I got impatient and decided to go ahead and release version 0.2 of stockphoto. This version introduces no new features, but it works with Django version 0.95 and other post-Magic-Removal versions. It also should be a little more tolerant of variation in settings variables, and so easier to install. Special thanks goes to William McVey for the initial work on porting stockphoto to MR, which I have finished and polished up for this release.

[ Posted: 15:00] | [ Category: ] | Permalink | Comments: 0 ]

Stockphoto 0.1 released

I’ve decided to release version 0.1 of stockphoto under the GPL. Stockphoto is a simple photo gallery application for django. There are better django photogalleries out there, so this is something of a wheel reinvention, but stockphoto is designed to be simple to understand and to install, with as few prerequisites as possible. Full instructions are included in the readme file.

[ Posted: 21:06] | [ Category: ] | Permalink | Comments: 1 ]

Some django gotchas

These are some django details I’ve found while working on my django demonstration site, things that are useful for making an integrated Django-powered site, but under-documented.

Make “View on site” work

If you give your model a “get_absolute_url” method, the admin view for each object in that model will have a “View on site” link. However, and this is the clever bit, that link will not work. I’m not exactly sure why, but instead of linking to what’s returned by “get_absolute_url”, it links to something starting with “/r/”. However, by adding this to your site’s urls.py, Django will redirect to the correct location.

    (r'^r/', include('django.conf.urls.shortcut')),

Use the admin app’s javascript widgets on your site

The admin app has a lot of neat convenience features, such as magical calendar links after each DateField. Here’s how you get those in your app.

Put this in your urls.py:

    (r'^jsi18n/$', 'django.views.i18n.javascript_catalog',
                   {'packages': 'django.conf'}), 

And put this in your base template, in the <head> section:

    <script type="text/javascript" src="/jsi18n/"></script> 
    <script type="text/javascript"
            src="/media/admin/js/core.js"></script><script
            type="text/javascript"
            src="/media/admin/js/admin/RelatedObjectLookups.js"></script><script
            type="text/javascript"
            src="/media/admin/js/calendar.js"></script><script
            type="text/javascript"
            src="/media/admin/js/admin/DateTimeShortcuts.js">
    </script>

You’ll need to add some CSS styles to make things look nice; look in the admin media css folder to find what you need.

Generic views for login and logout

The Django authentication documentation is good enough to allow you to write login and logout views for your site. But in fact, there are already generic views for this purpose, which are not discussed in the generic views documentation.

Add to your urls.py:

    (r'^accounts/login/', 'django.views.auth.login.login'),
    (r'^accounts/logout/', 'django.views.auth.login.logout'),

And write templates. Mine look like this:

[Template Dir]/registration/login.html:

{% extends "base" %}
{% block title %}Site : login{% endblock %}


{% block content %}
<h1>Login</h1>
<form action="/accounts/login/?next={{ request.GET.next }}"
      method="post">

  <p>
      {%if form.username.errors %}
      <span style="color: red;">
        {{ form.username.errors|join:", " }}
      </span><br/>
      {% endif %}

    <label class="fortextinput" for="id\_username">
      Username:
    </label>
    {{ form.username }}<br/>

    {%if form.password.errors %}
    <span style="color: red;">
      {{ form.password.errors|join:", " }}
      </span><br/>
    {% endif %}

    <label class="fortextinput" for="id\_password">
      Password:
    </label>
    {{ form.password }}<br/>

    <input type="submit" value="Login"/>
  </p>

and [Template Dir]/registration/logged_out.html :

{% extends "base" %}
{% block title %}site: logged out{% endblock %}


{% block content %}

<h1>Logged out</h1>
<p>You have logged out successfully.  This makes you special!</p>
{% endblock %}

This, with login/logout links (wrapped with if user.is_anonymous) in your templates, is enough to do basic login/logout for your entire site. Staff users will be able to use this login to get to the admin pages, as well.

[ Posted: 08:35] | [ Category: ] | Permalink | Comments: 1 ]

A very useful patch to django

The photo gallery application I’m working with allows you to upload batches of photographs as zip files, to populate a gallery all at once. As it stands in django, this zipfile is temporarily held entirely in memory. For substantially large zipfiles, as one might expect to upload to a photogallery, this will drive memory usage up quite a bit. An example file I was testing from here actually OOM-ed on the little server I’m running it on. Even for a file that does not OOM, it may easily cause the FastCGI connection to time-out (and I imagine the consequences could be even worse in mod_python).

While looking for a solution, I found this patch, which solves the issue. Moderately large files are processed reasonably quickly now, and server memory usage is much closer to what would normally be expected.

Consider this a vote for inclusion of this patch in trunk.

[ Posted: 15:43] | [ Category: ] | Permalink | Comments: 0 ]

Simpler Apache FCGI Django

Inspired by some tips I saw someplace else, and some fiddling on my own, a simpler Django setup on Apache/mod_fcgi:

Snippet from httpd.conf:

<VirtualHost *:80>
    ServerName mysite.dom
    DocumentRoot /home/httpd/html/mysite/

    AddHandler   fastcgi-script  fcg fcgi fpl fpy
    RewriteEngine on
    RewriteRule ^/media/.*$ - [L]
    RewriteRule ^(/.*)$ /mysite.fcgi$1 [L]

    SetEnv PYTHONPATH /home/httpd/html/mysite
    SuexecUserGroup   myuser mygroup

</VirtualHost>

And what mysite.fcgi looks like:

#!/usr/bin/python
from flup.server.fcgi_fork import WSGIServer
import os
from django.core.handlers.wsgi import WSGIHandler

os.environ[‘DJANGO_SETTINGS_MODULE’] = ‘mysite.settings’

handler = WSGIHandler()
WSGIServer(handler).run() 

You’ll have, of course, to change paths and userids to whatever you need. Features of this setup:

1. All requests are sent to Django, execpt for paths starting with /media/. This way you do not have to have RewriteRule lines for every application. This method assumes that your entire site except for media/images/etc. is served up with Django (using the flatpages application for everything that’s not an app view, for example).

2. mysite.fcgi is run as a dynamic FastCGI app. This means it will be started and stopped as necessary by mod_fastcgi process manager, and does not need to be individually added to the Apache config file (indeed, most of the above setup could be done from a .htaccess file assuming the VirtualHost was already set up and configured to accept .htaccess).

3. If the Apache server is set up with FastCGIWrapper, and Apache is set up correctly for suexec to work with virtual hosts, Django will run as myuser.mygroup, rather than as the Apache user. This is a great comfort in shared hosting environments.

Point 3 could also be accomplished by configuring your django runner script as a FastCGIExternalServer (in httpd.conf) and starting it manually as myuser.mygroup. This is a matter of personal preference. I prefer my way because httpd handles keeping one or more copies of my application running as needed; I don’t have to take further measures to make sure it gets restarted if it dies for some reason (crash, OOM kill, etc). I also like how it makes everything easier to configure (especially in a shared hosting environment, if everything is set up correctly beforehand). However, if you are allergic to Apache’s suexec, you may prefer the alternate method.

[ Posted: 14:15] | [ Category: ] | Permalink | Comments: 2 ]

Django alternate auth HOWTO

One of the most frequently-asked questions about Django is “How do I change the authentication system to (hook into my existing authentication system / use something other than the django database). In my first couple of posts in this section, I gave one answer. The method I gave works, and it has the advantage of working on a per-app basis (if that is what you need). However, it was also written before the Django authentication documentation was written. The method I posted uses anonymous sessions (from Django’s perspective) and does not really use the Django authentication framework. Most people will probably want to use as much of the Django authentication framework, however. As I’ve now figured it out (I’m sure I’m not the first, but I’ve never seen it comprehensively documented anywhere), I’m posting how to do this. Let me emphasize that I am not at all an expert on Django internals; I’m just posting my experiences and am open to correction on any points.

First, make an app to hold the new user model and (optionally) views for logging in and out.

In the model, you are going to be over-riding the standard django.models.auth.users model with one that subclasses it. So here is what you need.

from django.core import meta
from django.models import auth

# Create your models here.
class User(auth.User):

    location = meta.CharField(maxlength=100, blank=True)
    favourite_cheese = meta.CharField(maxlength=30, blank=True)

    class META:
        admin = meta.Admin(
            fields = (
                (None, {'fields': ('username', )}),
                ('Personal info', {'fields': ('first_name',
                                              'last_name', 'email')}),
                ('Extra info', {'fields': ('location', “favourite_cheese”,)}),
                ('Permissions', {'fields': ('is_staff', 'is_active',
                                            'is_superuser', 'user_permissions')}),
                ('Important dates', {'fields': ('last_login',
                                                'date_joined')}),
                ('Groups', {'fields': ('groups',)}),
                ),
            list_display = ('username', 'email', 'first_name',
                            'last_name', 'is_staff'),
            list_filter = ('is_staff', 'is_superuser'),
            search_fields = ('username', 'first_name', 'last_name',
                             'email'),
            remove_fields = ('password_md5',)
            )

        replaces_module = 'auth.users'

    def check_password(self, raw_password):
        '''Override the auth framework's password checking '''
        # Insert code for checking (e.g) ldap password here
        if external_password_check(self.username, raw_password):
            return True
        else:
            return False

    def set_password(self, raw_password):
        '''Override the auth framework's password setting '''
        # Insert code for setting password.
        external_password_set(self.username, raw_password)
        return True
# end

This model does not have to be installed using “django-admin.py install fancyauth”, and indeed, doing so will fail. What you need to do, instead, is modify the auth_users table in your project’s database to add any fields you have added. In the above model, that is ‘location’ and ‘favourite_cheese’.

The convenience function create_user(username, email, password) is also broken by this change. Writing a replacement is left as an exercise to the reader (if needed; I don't find it all that useful). You will also need to set/change passwords either through a python interactive session, or through a custom view – it can't be changed in the admin system, though you can create users there.

This is basically the minimum you could do to use custom authentication with the Django authentication framework. Other things you might want to do:

1. Make the User model function as a cache of external user information (e.g., fill out fields from GECOS data in LDAP or Unix password file).

2. Login view automatically creates a user record if needed before checking password.

3. Make everything completely transparent with regard to Django authentication API and admin framework.

One final note: this authentication system is project-wide. If you have an app that has significantly different authentication needs from the rest of your site, you will need either to give it its own authentication framework, or (more likely) write your views in such a way that just being authenticated doesn’t imply much about what a user is authorized to do.

[ Posted: 13:25] | [ Category: ] | Permalink | Comments: 3 ]

Redirects in Django

While working on the authentication bits for a feed-reader application I’m writing (more on that later), I noticed that polipo was cacheing my redirects. This is bad, because I’m using a decorator on most of my views that checks if a user is logged in, and, if not, redirects them to the login page. It looks like this, and is basically copied from the django admin code:

def login_required(view_func):
    “””
    Decorator for views that checks that the user is logged in, redirecting
    to the log-in page if necessary.
    “””
    def _checklogin(request, *args, **kwargs):
        try:
            user_id = request.session[‘user_id’]
            if user_id:
                return view_func(request, *args, **kwargs)
            else:
                return redirect_to_login(request.path)
        except KeyError:
            return redirect_to_login(request.path)
    return _checklogin

redirect_to_login is also basically copied from the django auth code, but with one significant change. Before making this change, after a successful login, you would be redirected back to the login page, rather than the index page. Also, because the redirect was being cached by the proxy, attempts to access the index page directly would be redirected to the login page. I would call this a proxy bug, but polipo is pretty anal about standards-compliance. So this is the modified redirect_to_login:

def redirect_to_login(next):
    “Redirects the user to the login page, passing the given ‘next’ page”
    response = HttpResponseRedirect(‘/engulf/accounts/login/?%s=%s’ %
                                (REDIRECT_FIELD_NAME, next))
    response[‘Pragma’] = “no cache”
    response[‘Cache-Control’] = “no-cache”
    return response

Basically it is the same as the original, but inserts cache-control headers for both http 1.0 and 1.1. This prevents the proxy from cacheing the redirect, and my login decorator now works. I may post the login view and manipulator code at some point, because I think it’s a pretty good example of using a custom manipulator with a form that doesn’t directly affect a model.

Edit: Apologies for the curly quotes in the code samples. It’s being done by the Smartypants plugin for my blogging software, and to get it to stop, I need to turn it off entirely. It shouldn’t affect the version in the RSS feeds, though.

[ Posted: 08:47] | [ Category: ] | Permalink | Comments: 0 ]