Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Keynote - Daniel Greenfeld
Django conferences have a tradition: there's an external luminary that gets to give a critical talk on Django. His talk won't be that. He's not external either: he wrote two scoops of Django together with Audrey (see also my review). Being critical is sometimes easy. Just bash class based views, for instance. Bashing is easy. A rant like Zed Shaw's is fun, but he's not asked because of his rants, but because of his contributions (like books). Similarly, Django delivers working stuff and that working stuff makes a lot of our work possible. So here are some good points about Django: Django is everywhere. So many people and companies use it. Django is powered by Python. Pep8, python is beautiful. And there's the import this zen of python that we use all the time to steer others and ourselves in the right direction. Django's API wins. It is understandable. No weird names: templates, views, logging, sessions. Django projects also have understandable structures. If there's no views.py or models.py or templates/ directory, you know someone messed something up. Fat models are great. Just put your business logic all on your models. They do get big this way, however. You can make ... -
Prehistorical Python: patterns past their prime - Lennart Regebro
Dicts This works now: >>> from collections import defaultdict >>> data = defaultdict(list) >>> data['key'].add(42) It was added in python 2.5. Previously you'd do a manual check whether the key exists and create it if it misses. Sets Sets are very useful. Sets contain unique values. Lookups are fast. Before you'd use a dictionary: >>> d = {} >>> for each in list_of_things: ... d[each] = None >>> list_of_things = d.keys() Now you'd use: >>> list_of_things = set(list_of_things) Sorting You don't need to turn a set into a list before sorting it. This works: >>> something = set(...) >>> nicely_sorted = sorted(something) Previously you'd do some_list.sort() and then turn it into a set. Sorting with cmp This one is old:: >>> def compare(x, y): ... return cmp(x.something, y.something) >>> sorted(xxxx, cmp=compare) New is to use a key. That gets you one call per item. The comparison function takes two items, so you get a whole lot of calls. Here's the new: >>> def get_key(x): ... return x.something >>> sorted(xxxx, key=get_key) Conditional expressions This one is very common! This old one is hard to debug if blank_choice also evaluates to None: >>> first_choice = include_blank and blank_choice or [] There's a ... -
Does your stuff scale? - Steven Holmes
They grew from a two-person company to a 70-person one in two years. Central to that growth were Django and google app engine. Scalability means both load scalability and functional scalability. You also have to deal with organizational scalability and geographical scalability if you want to grow your organization. 1: Running Django on app engine It is easy to get confused. Is app engine real? Is it a joke? How to run your django stuff on it? Their reasons to use it: Auto-scaling. They build high-profile stuff and it needs to scale. They had a valentine day site that got a lot of attention on that day and it automatically scaled up without a change in the app. The day after it scaled down automatically, too. Services and APIs. No sysadmin needed. Some caveats with app engine: it is a sandbox. You you cannot do "pip install". The filesystem isn't there in the traditional sense; there is a blob storage instead. And it is lock-in, mostly; portability is an issue. They could work arounds these issues and ended up with a better application as a result. There are three ways (that they use) of running Django on app engine: Django ... -
Splitting up settings in Django
Splitting up settings in Django By default all Django settings are in one monolithic settings.py file. A single big file is hard to read and hard to maintain. Django users have found many ways to split up the settings into multiple files, all of them with their pros and cons. In our latest projects, we have developed yet another way, which uses file inclusion instead of importing Python scripts. The main features of django-split-settings are: Settings can be split into files and directories. Files later in the list can modify configurations, for example add or remove apps from INSTALLED_APPS. The main settings file lists the files that make up the project’s settings. Files can be marked optional. Optional files can be used to override settings per instance. Wildcards can be used in file paths. Maintains support for Django’s runserver auto-reloading. Read more... -
Djangocon lightning talks day 2
Sorry if I mangled any of the names, that's the hardest part of blogging lightning talks. Many don't show their name long enough :-) Single page web apps with django and extjs - Michał Karzyński Single page apps: you're writing two apps. A front end one and a back end. The routing is done on the client side. The back end just spits out data (JSON api). ExtJS has a store that handles communication with the backend. So that talks to your JSON API. Plan that API carefully, try to keep it nicely RESTful. He showed a one minute demo. There is a longer one on his blog. Don't trust, check - Marcin Mincer & Tomek Kopczuk Check and question everything. Seek the best way. Not all good solutions are as good as they seem. They compared a standard view with a tastypie view and the regular view was much faster. They also checked, for their example, whether using jinja2 would be faster than django templates. Yes, it is faster. Despite what the two scoops book says. So: check everything for your usecase. Lessons learned - Tom Christie Tom maintains the Django rest framework project. He tells us a few ... -
Taming ajax and django - Marc Egli and Jérémie Blaser
Jérémie is a frontend developer and Marc does the backend. Address/state handling and content rendering are the two main challenges. Address and state handling Problems: Browser history. If you don't watch out, the back button won't be working. Deeplinking should stay possible. Crawler visibility: you want them to grab your entire site. But they don't use javascript. So you need a special URL for them Some solutions: A hash like http://yoursite.com/#/some/id. Javascript will need to handle everything behind the hash. Problem: without javascript it isn't visible. You're invisible to crawlers. It is easy to implement, though. A hashbang like http://yoursite.com/#!/some/id. The difference? Google and others replace the URL with http://yoursite.com/?_escaped_fragment_/some/id. You'll have to configure your website to support it. Deeplinks work this way and crawlers can access the site via links in a search engine sitemap. It works with almost all browsers. And it covers all three mentioned problems. You have multiple URLs, however. And you'll need to maintain legacy URLs. In django you could implement it with some middleware that detects the _escaped_frament_ GET parameter. Pushstate. The URL is a regular URL like http://yoursite.com/some/id. The best example is the github website. Pro: easier to implement on the backend, good ... -
Mock your database - Marc Tamlyn
Your database is slow. Especially for testing. Nobody likes waiting, so SPEED is essential. Speed of iteration. Faster unit tests mean more tests. No one likes waiting. Once you create a couple of model objects in your tests, you're going to hit your database. He did a quick test with some 8 database queries. 5 seconds for 1000 runs on postgresql. 4 seconds for in-memory sqlite, so that's a bit better. Why not use factory_boy? That could help, right? More efficient instantiation, perhaps, all in one go (as the test was with a bunch of related objects)? 6 seconds on postgres, sigh. Higher. He tried a dirty dirty dirty crazy hack, using Django internals. (I looked to my right and the #1 Django database core committer was grinning from ear to ear...). No database queries. 1 second. Better: mocking. So he did some still-quite-hacky mocking with the mock library. 1.2 seconds. Ok, but the code is a bit yucky. Summary till now: avoiding the database is quicker. The difference is bigger than the difference between postgresql and in-memory sqlite. So... write your code in such a way to make avoiding the database easier. But... is there a better way? He ... -
Getting past Django ORM limitations with Postgres - Craig Kerstiens
Tip: subscribe to the postgresql weekly newsletter that Craig makes. Why postgres? A colleague described it as "it is the emacs of databases". There's just so much available inside postgres. The problem is Django: it treats all databases the same. It doesn't prefer one over the other. It doesn't give special treatment. Look at all the types that Christophe mentioned yesterday: Django only supports a few of 'em. Likewise indexes. For instance postgresql's Array type. Django doesn't support it, but it'd be perfect for for instance a list of tags on a model. For many of these types, also for the Array type, you have django apps that add support for them. Great: hstore. NoSQL in your SQL. A key/value store in your SQL. They use it inside Heroku a lot: it scales fine and works fine. To use it in Django, use django-hstore. Add a data field as hstore to a model and suddenly you can do my_object.data = {'key': 'value', ...}! Queuing: most people use celery. Postgres is a great queue. There's a celery backend called trunk for it. Postgresql has great text search. You do need to do some setup in your models, but then it works ... -
Taming multiple databases with Django - Marek Stępniowski
Marek works at SetJam: "We came to Django for the views, but stayed for the ORM". Django's ORM is pretty much in the sweet spot. SQLalchemy in comparison is less nice, having to learn a non-sql, non-pythonic language. At SetJam, they have what they call a backend and frontend. The backend collects data and stores it in the database, the frontend spits it out, mostly via feeds. They started out with one single big database, but that was hard to optimize. Many backend servers would write to the same database and the frontend server would read from it. Hard to optimize. Next they added a database slave for reading. That was before Django's multi-db support, so they had if/elses in their settings files based on environment variables. After Django's multi-db support, they could really support two databases and refer to them in the code with 'DEFAULT' and 'SLAVE'. Later on they splitted up the database even more. What goes where is handled by two custom database routers: a "MasterSlaveRouter" for the master/slave distinction and an "AppRouter" for shuffling some apps' data to certain databases. Tip: look at https://github.com/jbalogh/django-multidb-router, especially for the handy decorators (@use_master, for instance) it provides. At a ... -
The web of stuff - Zack Voase
A plane flew over (noisily) at the start of his presentation. He put our work in perspective by saying that that was a 80 ton plane and that we're just building websites :-) Possibilities Computers used to take up whole rooms, now you have a smartphone. Big data is really big data now. Moore's lawworks both ways, though, so you have really small computers now. An arduino for instance. He often makes comparison to the human body. All over our body, sensors give off signals that go into the central nervous system. The brain processes it and gives signals back to muscles if necessary. Sensing, feedback, understanding, reaction. Stuff can talk to the cloud. Like a sensor in your body talks to your mind, stuff can treat the cloud as a brain. The cloud is what allows small tools to be smart. Stuff does often need a human to interact with it. Like a smartphone. There's all sorts of people thinking about how to "liberate the computers from their human overlords". Why cannot computers sense and act on their own account? So how do you bridge the gap betwen sensing and acting of stuff? How do you use Django for ... -
Apps for advanced plans, pricings, billings and payments - Krzysztof Dorosz
He runs multiple sites with a common business model: accounts with plan subscriptions. So there's an obvious need for a generic account billing application. The app should not be too specific, as that limits your business flexibility. Also it should not be too generic: you'll end up with an architecture from hell that way. And there's the billing as such: you need to pay close attention to security and so. Hard problem! What he's making is django-plans for keeping track of the billing data, the plans, etc. And django-getpaid as payment processing app. django-getpaid Some challenges for the actual payment integration: He wants it to be generic and lightweight. He doesn't want to pull in half of pypi for a payment processing app. He wants a single API so that he can switch payment brokers if needed. He wants it to be asynchronous. Synchronous processing blocks too long. Multiple currency support. None of the existing apps were good enough, so he made django-getpaid. It is stable and supports a lot of (Polish) payment systems and is pluggable if you need to add another one. Pluggability is achieved with special backends you can enable in your Django settings. This way you ... -
Growing open source seeds - Kenneth Reitz
He shows us three kinds of (more or less) open source projects. Type 1: public source Once upon a time there was an "open source project" called the facebook SDK. Basically it just stopped working one day and nobody could help, despite offers for help on the issue tracker. Hacker news got wind of it and it was on the front page for a while. Facebook's reaction? Disabling the issue tracker... (Later on they fixed it). That's not open source, that's public source. Often it is abandoned due to loack of interest, change of focus or so. The motivation for having it as open source simply is not clear. Type 2: shared investment A different example: gittip. They aim to be the world's first open company. There's a github issue for everything, even the company name. Major decisions are voted for on github. The code is open source, of course. All interviews with journalists are filmed and live-streamed. And all otherwise-often-backdoor-cooperation-agreements are fully open. Projects like gittip are shared investment projects. Shared ownership, extreme transparency. There is very little questioning of motivations. The motivation is clear and public. There's a documented process for new contributers. The advantage? It is low ... -
Play nice with others - Honza Král
Many people think that reusable apps don't work: there's always something you need to change or modify. Honza is going to talk about his experience with ella, a django CMS. He advocates using model inheritance. from ella.core.models import Publishable and then subclass your specific model (YoutubeVideo, for instance) from it. That Publishable has most of the basic CMS functionality. That way you get most of what the CMS needs for free and you still can extend it. Showing the new model? You can use different templates easily. render_to_string() and friends accept a list of templates. So you can give it ['publishable.html', 'youtubevideo.html'] and so, using templates named somewhat after the model. This way you can re-use basic templates, but modify them if you want, just by providing a specially-named template. No code changes necessary. They're using Redis to collect information from the Django database on publishables. This way you don't have any problem with Django's database's behaviour of focusing on a single kind of model at a time. They also use django-appdata for storing extra data on existing Django models. From the pypi page: extandable field and related tools that enable Django apps to extend your reusable app. Through a ... -
Advanced Python through Django: metaclasses - Peter Inglesby
Metaclasses are a handy feature of Python and Django makes good use of them. When you create certain kinds of classes in Django, a metaclass will do something to the class before it is created. For forms, the various attributes of the class are converted into a base_fields dictionary on the class. Similarly, a subclass of Model also fires up a metaclass that does some registering. A foreignkey to another model adds a relation back on that other model, for instance. As a recap, a class is something that can be instantiated into an object. It can have an __init__() method that does something upon instantiation. type(your_instance) will return the class. Did you know that you can create classes dynamically? See for yourself: >>> name = 'ExampleClass' >>> bases = (object,) >>> attrs = {'__init__': lambda self: print('Hello from __init__')} >>> ExampleClass = type(name, bases, attrs) >>> example = ExampleClass() Hello From __init__ >>> type(example) <class '__console__.ExampleClass'> So... we can actually control how classes are created! You could create a create_class() method that calls type but that modifies, for instance, the name. Or we could take all the attributes and add them to a base_fields dictionary on the instance. Hey, ... -
Fractal architectures - Laurens van Houtven
He worked twisted on. twisted And people tend to talk about subjects that are almost antithetical to how Django does things. The thing that he does different from Django is that he's not using a single data source... Once a database gets really really too big, putting multiple databaseservers next to eachother doesn't really work. You slowly start to get into expensive Oracle territory. How he set it up now is what he calls a fractal architecture. The whole accepts requests. The parts of the whole acccept requests. The parts of the parts accept requests. That's why he calls it fractal. You could also call it sharded, but that has a bad name: it is something you do when nothing else works. The way he looks at the architecture is SMTP. Email. Simple. He prefers SQLite. Simple and included in the python standard library. Sure, you can use postgres but you'll need a VM to re-create the same environment locally as on your production machine. SQLite is the same everywhere. In fact, he uses Axiom: an object store on top of SQLite. (Note: he is trying to write documentation for it at https://github.com/lvh/axiombook). Another advantage of sqlite: it is easy ... -
Bleed for speed - Rob Spectre
He started with a little history lesson. The sea battle of mobile bay. The admiral (Faragut) ordered the ships straight through the minefield (called "torpedoes" at the time). "Damn the torpedoes, full speed ahead". And it worked. What does this to have to do with Django? Well, "damn the torpedoes, full speed ahead" feels a bit like how rapid prototyping feels afterwards. He's often involved with hackathons. Lot of quick coding in limited time with a lot of people. He learned a lot about his tools that way (and he often used Django). There's a time to make a distinction between production and prototype. Sometimes it is better to just try something with a prototype. Throw-away code. Aaargh! Throw-away code?!? We never throw code away. But it is something we must learn. It is good to let go once in a while. Let your code go. It isn't yourself, it is just some code. The danger is that prototype code is put into action as production code. With some work, this danger can be prevented. What about Django? Django is the best for prototyping. For rapid prototyping, Django is better than micro-frameworks like Flask that might seem better at first ... -
Know Your Models
In web development, we have a unfortunate double meaning for the word “models” and as obvious as the separation of those two seems to seasoned developers, it shows again and again that it’s not as obvious to beginners. Anyone who took a class on object-oriented design in the last 20 years has inevitable heard about MVC. While there are recurring (and consistently fruitless) discussions about what exactly is what, what should be named how and how they should interact, the basic common understanding is more or less: You separate your application into models, views and controllers (or models, templates and views). And ideally, you should be able to tamper with/replace either mostly independently of the other two. This article concentrates on the models part which is the data the other components work with and display to the user. Ideally it should be a pleasant API that abstracts away the storage intricacies. However… There is more than one kind of models… Django calls its ORM classes ‘models’. Pylons/Pyramid calls the skeleton part for SQLAlchemy ‘models’. I’m going to claim that that’s rather misleading since unless you’re building a simplistic CRUD application, your application models are not the same as ORM models ... -
Migrating the future - Andrew Godwin
Andrew Godwin attempted to raise 2500 pounds for inclusion of south in Django core with kickstarter. It worked. In fact, he raised 17952 pounds! Why does South need to be replaced by a new version inside Django itself? It started 5 years ago, so there's 5 years of learning done in that period. Some things that made sense at the time aren't the best decision now. There's poor support for VCS branching. The migration files are huge. Migration sets get too large. There are projects with 1000 steps! The inside-django solution has two parts. The actual migration code and a separate backend. So if you want a different migration engine, you can probably reuse the backend code with its support for multiple types of databases. The new migration format is more declarative instead of imperative like it is now. This makes them smaller. It also allows you to compute the end result in memory and apply one single migration. Migrations will have a parent. So you won't have a problem with 0003_aaaa and 0003_bbbb migrations that halfway bite eachother. If a merge can be done automatically, fine, otherwise south/django will warn you. Squashing will be added. You can squash a ... -
Two scoops of Django book review
I took the train from Utrecht (NL) to Warsaw today. I only had to change in Amersfoort (NL) and Berlin (DE), so it was a pretty direct connection. 12 hours of train time (which I enjoy). So that's enough time to read through two scoops of Django, the Django book by Daniel Greenfeld and Audrey Roy! Here's my review. The summary: buy the book and learn a lot. For the longer version, I'll simply go though my notes I made for each chapter. Coding style The book starts off good, in my opinion, because it tells you to write good and neat code. PEP8. Good not-too-short variable names. And it got me thinking by advocating explicit relative imports ("from .models import SomeModel"). That's what's good about this book: Daniel and Audrey state preferences and tell you best practices and sometimes those best practices won't be your best practices. Or you didn't know something. Anyway, it gets you thinking; which is good and enjoyable. Virtualenv Hey, virtualenv in chapter two! Nice. Here, like in the rest of the book, I noticed they point a lot at existing documentation and don't provide much explanation. No virtualenv explanation here, for instance, just a ... -
Getting recommendations out of nothing - Ania Warzecha
Ania Warzecha researched recommendation systems. Recommendations means estimating ratings or preferences for items a user hasn't seen yet. For example books or movies you might also like based on earlier purchases. There are three kinds of recommendations. Collaborative recommendations. Mostly created based on actions from other users. Which books are often bought together, for instance. Simple to implement, but can be slow for big datasets. And doesn't work well on new items and/or new users Content-based recommendations. Looks for similar items. Fast and accurate, but tends towards over-specifications regarding needed data. Hybrid methods. Combining them. A case study: a Polish car parts website. You normally don't log in there, you just want a part. So older purchases aren't available. They did have a lot of parts and data, so they started with content-based recommendations. They mixed in some basic user actions. 0=didn't buy, 1=browsed, 2=bought. Later on more elaborate, like points for items found through searching or items placed on wishlists. They used Redis for its quick addition of user actions, simply pushing an additional score to an item which then gets added in the database. One thing they needed to do was to merge session keys after a user ... -
Having your pony and committing it too - Jacob Burch
Jacob Burch hopes you can learn from him if you're new at contributing to open source. He won't cover virtualenv, git, django's core code structure. And also not what to get involved in. What's this talk about? About you if you have something you want ("a pony") to get into Django core. You are initially probably going to be a bit afraid. Jacob showed a couple of quotes about people that were initially not quite sure/certain when committing to Django. Then he showed the names of the people those quotes came from: they're now all core committers :-) Two balances you have to keep in mind: You should be both pro-active and patient. This is a tough balance to strike. If you manage it, it helps a lot. You should be both confident and humble. Be humble, but be convinced of your idea. How to help here? The best thing is to run all the tests. It will give you confidence that your solution works (if it does). And it'll make you humble once you realize all the end cases that Django (and thus your fix) needs to support. There are three broad categories of contributions: Bug fixes Start with ... -
Combining Javascript and Django in a smart way - Przemek Lewandowski
Django is a javascript-agnostic web framework. Nothing is built-in so you can be up to date all the time. Javascript development moves very quickly. The basic approach is to include some custom inline javascript in the html pages. It quickly leads to illegible code that's hard to work on and hard to distribute. Javascript has frameworks, too. They give your application structure and take work off your hands. This is the advanced approach. It includes several parts: Communication with the server (REST api, websockets). Application building: combining and minimizing files. Static files management. Javascript improvements: coffeescript and so. What Przemek Lewandowski needed was a powerful javascript framework, coffeescript, testable code, js code minimization and fingerprinting for avoiding caches. And also rapid REST API development. Javascript framework They started with backbone, but it wasn't enough. They added marionette to backbone, but it still wasn't good enough. There's a lack of a binding mechanism; there are no reusable views; models are poor. AngularJS and Ember are better. Coffeescript It is controversial, but it helps to write code faster and use less code for it. It performs as well as javascript as it compiles to javascript. They used requireJS for painless coffeescript integration. ... -
Advanced PostgreSQL in Django - Christophe Pettus
(See also last year's talk) Database agnosticism: write once, run on any database. A critical selling point for Django: it runs on many databases. But for others, it is bad. You pay a performance hit for not using database-specific features. So once you have made your choice, really use that database. Here are some examples of good special things available in postgres. Custom types Custom types. If you like types, you'll love postgress. Many built-in types. And many are usable in Django by installing some small app. Do you do .lower() in python code or in your SQL? For an email address for instance? Why not use citext, a case insensitive text field provided by postgres. Often you want to add various key/value data to an object. Attributes. Extra table with a join? Add fields to the main table? Solution: hstore. Postgres has a built-in json type! No need for mongodb :-) It is validated going in. Postgres 9.3 will make it much faster. The UUID type is much more efficient than storing a long character string. IPv4 and IPv6 addresses. You can define your own! And it is easy to integrate into Python and Django: You adapt it into ... -
Thread profiling in Python - Amjith Ramanujam
Amjith Ramanujam recently wrote a thread profiler in Python and it was rediculously simple. He works for New Relic, which is all about performance and expecially performance measuring. Python comes with batteries included, so it also includes a C profiler that works pretty well. But it doesn't work nice for django because the output is so huge. If you use the GUI RunSnakeRun, it is more managable that way. Additional problem with cProfile: it has about 100% overhead, so you can't run it in production (which they need). You can do more targeted profiling. For instance in Django. The way a web framework processes requests is normally always in the same way. You can use that during profiling. There are two important stages: interrupt and inquire. Interrupt A statistical profiler looks how often a function is called and by who. For this it needs to interrupt the regular process. You could set an OS-level signal to call your profiler every x miliseconds so that it can do something. It only works in linux, btw. Another way is to create a python background thread that wakes up every x miliseconds. It is cross-platform and mod_wsgi compatible. It is less accurate for ... -
The imaginative programmer - Zed Shaw
His goal: teach programmers to be more creative. He's got a love/hate relationship with creativity. The first part of his talk was impossible to summarize. You'll have to watch the video later on :-) Artists tell him he's not artistic because he works on developing technical skills. Guitarists tell him he's not a real guitarist as he doesn't play in a band. And 'cause he builds his own guitars he's a programmer, not a Real Guitarist. Writers tell him he's not a writer because he writes technical books. Programmers tell him he's not a programmer because he doesn't work on their project. And by the way, he's a (technical) writer now, so he's not a programmer. He's not creative enough. Or so the others say. Or he's not acceptable. How to deal with creativity? In a way, you can re-phrase creativity. Programmers are always making something from nothing, right? Isn't that the pinnacle of creativity? Here are four hypothetical persons: Technique, no imagination: a stereotypical programmer. Imagination, no technique: stereotypical biz dude. No imagination, no technique. Probably doesn't exist. Both imagination and technique. Zed's goal. Zed's imaginative programmer process. Everyone has a process (if they're good), here's the one he ...
