Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Growing open source seeds - Kenneth Reitz
He shows us three kinds of (more or less) open source projects. Type 1: public source Once upon a time there was an "open source project" called the facebook SDK. Basically it just stopped working one day and nobody could help, despite offers for help on the issue tracker. Hacker news got wind of it and it was on the front page for a while. Facebook's reaction? Disabling the issue tracker... (Later on they fixed it). That's not open source, that's public source. Often it is abandoned due to loack of interest, change of focus or so. The motivation for having it as open source simply is not clear. Type 2: shared investment A different example: gittip. They aim to be the world's first open company. There's a github issue for everything, even the company name. Major decisions are voted for on github. The code is open source, of course. All interviews with journalists are filmed and live-streamed. And all otherwise-often-backdoor-cooperation-agreements are fully open. Projects like gittip are shared investment projects. Shared ownership, extreme transparency. There is very little questioning of motivations. The motivation is clear and public. There's a documented process for new contributers. The advantage? It is low ... -
Apps for advanced plans, pricings, billings and payments - Krzysztof Dorosz
He runs multiple sites with a common business model: accounts with plan subscriptions. So there's an obvious need for a generic account billing application. The app should not be too specific, as that limits your business flexibility. Also it should not be too generic: you'll end up with an architecture from hell that way. And there's the billing as such: you need to pay close attention to security and so. Hard problem! What he's making is django-plans for keeping track of the billing data, the plans, etc. And django-getpaid as payment processing app. django-getpaid Some challenges for the actual payment integration: He wants it to be generic and lightweight. He doesn't want to pull in half of pypi for a payment processing app. He wants a single API so that he can switch payment brokers if needed. He wants it to be asynchronous. Synchronous processing blocks too long. Multiple currency support. None of the existing apps were good enough, so he made django-getpaid. It is stable and supports a lot of (Polish) payment systems and is pluggable if you need to add another one. Pluggability is achieved with special backends you can enable in your Django settings. This way you ... -
Djangocon lightning talks day 2
Sorry if I mangled any of the names, that's the hardest part of blogging lightning talks. Many don't show their name long enough :-) Single page web apps with django and extjs - Michał Karzyński Single page apps: you're writing two apps. A front end one and a back end. The routing is done on the client side. The back end just spits out data (JSON api). ExtJS has a store that handles communication with the backend. So that talks to your JSON API. Plan that API carefully, try to keep it nicely RESTful. He showed a one minute demo. There is a longer one on his blog. Don't trust, check - Marcin Mincer & Tomek Kopczuk Check and question everything. Seek the best way. Not all good solutions are as good as they seem. They compared a standard view with a tastypie view and the regular view was much faster. They also checked, for their example, whether using jinja2 would be faster than django templates. Yes, it is faster. Despite what the two scoops book says. So: check everything for your usecase. Lessons learned - Tom Christie Tom maintains the Django rest framework project. He tells us a few ... -
Getting past Django ORM limitations with Postgres - Craig Kerstiens
Tip: subscribe to the postgresql weekly newsletter that Craig makes. Why postgres? A colleague described it as "it is the emacs of databases". There's just so much available inside postgres. The problem is Django: it treats all databases the same. It doesn't prefer one over the other. It doesn't give special treatment. Look at all the types that Christophe mentioned yesterday: Django only supports a few of 'em. Likewise indexes. For instance postgresql's Array type. Django doesn't support it, but it'd be perfect for for instance a list of tags on a model. For many of these types, also for the Array type, you have django apps that add support for them. Great: hstore. NoSQL in your SQL. A key/value store in your SQL. They use it inside Heroku a lot: it scales fine and works fine. To use it in Django, use django-hstore. Add a data field as hstore to a model and suddenly you can do my_object.data = {'key': 'value', ...}! Queuing: most people use celery. Postgres is a great queue. There's a celery backend called trunk for it. Postgresql has great text search. You do need to do some setup in your models, but then it works ... -
Fractal architectures - Laurens van Houtven
He worked twisted on. twisted And people tend to talk about subjects that are almost antithetical to how Django does things. The thing that he does different from Django is that he's not using a single data source... Once a database gets really really too big, putting multiple databaseservers next to eachother doesn't really work. You slowly start to get into expensive Oracle territory. How he set it up now is what he calls a fractal architecture. The whole accepts requests. The parts of the whole acccept requests. The parts of the parts accept requests. That's why he calls it fractal. You could also call it sharded, but that has a bad name: it is something you do when nothing else works. The way he looks at the architecture is SMTP. Email. Simple. He prefers SQLite. Simple and included in the python standard library. Sure, you can use postgres but you'll need a VM to re-create the same environment locally as on your production machine. SQLite is the same everywhere. In fact, he uses Axiom: an object store on top of SQLite. (Note: he is trying to write documentation for it at https://github.com/lvh/axiombook). Another advantage of sqlite: it is easy ... -
Taming multiple databases with Django - Marek Stępniowski
Marek works at SetJam: "We came to Django for the views, but stayed for the ORM". Django's ORM is pretty much in the sweet spot. SQLalchemy in comparison is less nice, having to learn a non-sql, non-pythonic language. At SetJam, they have what they call a backend and frontend. The backend collects data and stores it in the database, the frontend spits it out, mostly via feeds. They started out with one single big database, but that was hard to optimize. Many backend servers would write to the same database and the frontend server would read from it. Hard to optimize. Next they added a database slave for reading. That was before Django's multi-db support, so they had if/elses in their settings files based on environment variables. After Django's multi-db support, they could really support two databases and refer to them in the code with 'DEFAULT' and 'SLAVE'. Later on they splitted up the database even more. What goes where is handled by two custom database routers: a "MasterSlaveRouter" for the master/slave distinction and an "AppRouter" for shuffling some apps' data to certain databases. Tip: look at https://github.com/jbalogh/django-multidb-router, especially for the handy decorators (@use_master, for instance) it provides. At a ... -
Play nice with others - Honza Král
Many people think that reusable apps don't work: there's always something you need to change or modify. Honza is going to talk about his experience with ella, a django CMS. He advocates using model inheritance. from ella.core.models import Publishable and then subclass your specific model (YoutubeVideo, for instance) from it. That Publishable has most of the basic CMS functionality. That way you get most of what the CMS needs for free and you still can extend it. Showing the new model? You can use different templates easily. render_to_string() and friends accept a list of templates. So you can give it ['publishable.html', 'youtubevideo.html'] and so, using templates named somewhat after the model. This way you can re-use basic templates, but modify them if you want, just by providing a specially-named template. No code changes necessary. They're using Redis to collect information from the Django database on publishables. This way you don't have any problem with Django's database's behaviour of focusing on a single kind of model at a time. They also use django-appdata for storing extra data on existing Django models. From the pypi page: extandable field and related tools that enable Django apps to extend your reusable app. Through a ... -
Mock your database - Marc Tamlyn
Your database is slow. Especially for testing. Nobody likes waiting, so SPEED is essential. Speed of iteration. Faster unit tests mean more tests. No one likes waiting. Once you create a couple of model objects in your tests, you're going to hit your database. He did a quick test with some 8 database queries. 5 seconds for 1000 runs on postgresql. 4 seconds for in-memory sqlite, so that's a bit better. Why not use factory_boy? That could help, right? More efficient instantiation, perhaps, all in one go (as the test was with a bunch of related objects)? 6 seconds on postgres, sigh. Higher. He tried a dirty dirty dirty crazy hack, using Django internals. (I looked to my right and the #1 Django database core committer was grinning from ear to ear...). No database queries. 1 second. Better: mocking. So he did some still-quite-hacky mocking with the mock library. 1.2 seconds. Ok, but the code is a bit yucky. Summary till now: avoiding the database is quicker. The difference is bigger than the difference between postgresql and in-memory sqlite. So... write your code in such a way to make avoiding the database easier. But... is there a better way? He ... -
Bleed for speed - Rob Spectre
He started with a little history lesson. The sea battle of mobile bay. The admiral (Faragut) ordered the ships straight through the minefield (called "torpedoes" at the time). "Damn the torpedoes, full speed ahead". And it worked. What does this to have to do with Django? Well, "damn the torpedoes, full speed ahead" feels a bit like how rapid prototyping feels afterwards. He's often involved with hackathons. Lot of quick coding in limited time with a lot of people. He learned a lot about his tools that way (and he often used Django). There's a time to make a distinction between production and prototype. Sometimes it is better to just try something with a prototype. Throw-away code. Aaargh! Throw-away code?!? We never throw code away. But it is something we must learn. It is good to let go once in a while. Let your code go. It isn't yourself, it is just some code. The danger is that prototype code is put into action as production code. With some work, this danger can be prevented. What about Django? Django is the best for prototyping. For rapid prototyping, Django is better than micro-frameworks like Flask that might seem better at first ... -
The web of stuff - Zack Voase
A plane flew over (noisily) at the start of his presentation. He put our work in perspective by saying that that was a 80 ton plane and that we're just building websites :-) Possibilities Computers used to take up whole rooms, now you have a smartphone. Big data is really big data now. Moore's lawworks both ways, though, so you have really small computers now. An arduino for instance. He often makes comparison to the human body. All over our body, sensors give off signals that go into the central nervous system. The brain processes it and gives signals back to muscles if necessary. Sensing, feedback, understanding, reaction. Stuff can talk to the cloud. Like a sensor in your body talks to your mind, stuff can treat the cloud as a brain. The cloud is what allows small tools to be smart. Stuff does often need a human to interact with it. Like a smartphone. There's all sorts of people thinking about how to "liberate the computers from their human overlords". Why cannot computers sense and act on their own account? So how do you bridge the gap betwen sensing and acting of stuff? How do you use Django for ... -
Advanced Python through Django: metaclasses - Peter Inglesby
Metaclasses are a handy feature of Python and Django makes good use of them. When you create certain kinds of classes in Django, a metaclass will do something to the class before it is created. For forms, the various attributes of the class are converted into a base_fields dictionary on the class. Similarly, a subclass of Model also fires up a metaclass that does some registering. A foreignkey to another model adds a relation back on that other model, for instance. As a recap, a class is something that can be instantiated into an object. It can have an __init__() method that does something upon instantiation. type(your_instance) will return the class. Did you know that you can create classes dynamically? See for yourself: >>> name = 'ExampleClass' >>> bases = (object,) >>> attrs = {'__init__': lambda self: print('Hello from __init__')} >>> ExampleClass = type(name, bases, attrs) >>> example = ExampleClass() Hello From __init__ >>> type(example) <class '__console__.ExampleClass'> So... we can actually control how classes are created! You could create a create_class() method that calls type but that modifies, for instance, the name. Or we could take all the attributes and add them to a base_fields dictionary on the instance. Hey, ... -
Taming ajax and django - Marc Egli and Jérémie Blaser
Jérémie is a frontend developer and Marc does the backend. Address/state handling and content rendering are the two main challenges. Address and state handling Problems: Browser history. If you don't watch out, the back button won't be working. Deeplinking should stay possible. Crawler visibility: you want them to grab your entire site. But they don't use javascript. So you need a special URL for them Some solutions: A hash like http://yoursite.com/#/some/id. Javascript will need to handle everything behind the hash. Problem: without javascript it isn't visible. You're invisible to crawlers. It is easy to implement, though. A hashbang like http://yoursite.com/#!/some/id. The difference? Google and others replace the URL with http://yoursite.com/?_escaped_fragment_/some/id. You'll have to configure your website to support it. Deeplinks work this way and crawlers can access the site via links in a search engine sitemap. It works with almost all browsers. And it covers all three mentioned problems. You have multiple URLs, however. And you'll need to maintain legacy URLs. In django you could implement it with some middleware that detects the _escaped_frament_ GET parameter. Pushstate. The URL is a regular URL like http://yoursite.com/some/id. The best example is the github website. Pro: easier to implement on the backend, good ... -
Know Your Models
In web development, we have a unfortunate double meaning for the word “models” and as obvious as the separation of those two seems to seasoned developers, it shows again and again that it’s not as obvious to beginners. Anyone who took a class on object-oriented design in the last 20 years has inevitable heard about MVC. While there are recurring (and consistently fruitless) discussions about what exactly is what, what should be named how and how they should interact, the basic common understanding is more or less: You separate your application into models, views and controllers (or models, templates and views). And ideally, you should be able to tamper with/replace either mostly independently of the other two. This article concentrates on the models part which is the data the other components work with and display to the user. Ideally it should be a pleasant API that abstracts away the storage intricacies. However… There is more than one kind of models… Django calls its ORM classes ‘models’. Pylons/Pyramid calls the skeleton part for SQLAlchemy ‘models’. I’m going to claim that that’s rather misleading since unless you’re building a simplistic CRUD application, your application models are not the same as ORM models ... -
Copernicus, the great refactorer - Brandon Rhodes
Brandon Rhodes mentions Nicolai Kopernik, a famous Polish scientist. He "lifted Earth into Heaven" by his book where he put the sun in the center of our universe instead of Earth. We're no longer at the bottom. The near-earth environment was pretty well mapped out. 300BC, the size of the spherical earth was already known. Around 100BC the distance to the moon was known. But what about those planets? They were harder. Stars did move around the sky more or less linearly. But those planets. They seemed to move back and forth a bit. Did they need to have a different model? The sun-centric model was already known, but it didn't catch on. The reason? Medieval science was too emperic: the earth cannot be moving, it seems to stay in place. Throw a ball and it falls down again towards the earth, it doesn't career off into space. The church wasn't totally idiotic by later convicting Galileo: there was just no solid emperical evidence :-) One of the pieces of evidence that was missing was that there was no observable stellar parralax; visible movement between stars because of earth movement. It was only in 1838 that the instruments were good ... -
Combining Javascript and Django in a smart way - Przemek Lewandowski
Django is a javascript-agnostic web framework. Nothing is built-in so you can be up to date all the time. Javascript development moves very quickly. The basic approach is to include some custom inline javascript in the html pages. It quickly leads to illegible code that's hard to work on and hard to distribute. Javascript has frameworks, too. They give your application structure and take work off your hands. This is the advanced approach. It includes several parts: Communication with the server (REST api, websockets). Application building: combining and minimizing files. Static files management. Javascript improvements: coffeescript and so. What Przemek Lewandowski needed was a powerful javascript framework, coffeescript, testable code, js code minimization and fingerprinting for avoiding caches. And also rapid REST API development. Javascript framework They started with backbone, but it wasn't enough. They added marionette to backbone, but it still wasn't good enough. There's a lack of a binding mechanism; there are no reusable views; models are poor. AngularJS and Ember are better. Coffeescript It is controversial, but it helps to write code faster and use less code for it. It performs as well as javascript as it compiles to javascript. They used requireJS for painless coffeescript integration. ... -
Thread profiling in Python - Amjith Ramanujam
Amjith Ramanujam recently wrote a thread profiler in Python and it was rediculously simple. He works for New Relic, which is all about performance and expecially performance measuring. Python comes with batteries included, so it also includes a C profiler that works pretty well. But it doesn't work nice for django because the output is so huge. If you use the GUI RunSnakeRun, it is more managable that way. Additional problem with cProfile: it has about 100% overhead, so you can't run it in production (which they need). You can do more targeted profiling. For instance in Django. The way a web framework processes requests is normally always in the same way. You can use that during profiling. There are two important stages: interrupt and inquire. Interrupt A statistical profiler looks how often a function is called and by who. For this it needs to interrupt the regular process. You could set an OS-level signal to call your profiler every x miliseconds so that it can do something. It only works in linux, btw. Another way is to create a python background thread that wakes up every x miliseconds. It is cross-platform and mod_wsgi compatible. It is less accurate for ... -
Migrating the future - Andrew Godwin
Andrew Godwin attempted to raise 2500 pounds for inclusion of south in Django core with kickstarter. It worked. In fact, he raised 17952 pounds! Why does South need to be replaced by a new version inside Django itself? It started 5 years ago, so there's 5 years of learning done in that period. Some things that made sense at the time aren't the best decision now. There's poor support for VCS branching. The migration files are huge. Migration sets get too large. There are projects with 1000 steps! The inside-django solution has two parts. The actual migration code and a separate backend. So if you want a different migration engine, you can probably reuse the backend code with its support for multiple types of databases. The new migration format is more declarative instead of imperative like it is now. This makes them smaller. It also allows you to compute the end result in memory and apply one single migration. Migrations will have a parent. So you won't have a problem with 0003_aaaa and 0003_bbbb migrations that halfway bite eachother. If a merge can be done automatically, fine, otherwise south/django will warn you. Squashing will be added. You can squash a ... -
Djangocon lightning talks day 1
Sorry if I mangled any of the names, I took a photo of the lightning talk submission form and tried to decypher them :-) From carrots to Django - Kamila Stephiouska She tells about the Geek Girls Carrots community. A community for women interested in new technology. 11 cities, 4 special meetings, 1 sprint, 5 kinds of workshops. They like to promote women working in IT. The held a "django carrot" recently: 14 hours, 10 mentors, 23 participants. They try to get special guests. Last week Daniel and Audrey came (the writers of two scoops of Django). See http://django.carrots.pl They chose Django because of the community. Don't be afraid to commit - Daniele Procida Lots of people work with Django. Lots of people program with it. There are barriers to getting them to work on Django. They might not be effective. They might be afraid. They might not communicate effectively. You also need to manage your code and your environment. Virtualenv fixes the environment, but you need to learn that first. Version control helps with your code, but you first need to learn version control. Similarly, you need to learn documentation and tests. And you need to learn to have ... -
Having your pony and committing it too - Jacob Burch
Jacob Burch hopes you can learn from him if you're new at contributing to open source. He won't cover virtualenv, git, django's core code structure. And also not what to get involved in. What's this talk about? About you if you have something you want ("a pony") to get into Django core. You are initially probably going to be a bit afraid. Jacob showed a couple of quotes about people that were initially not quite sure/certain when committing to Django. Then he showed the names of the people those quotes came from: they're now all core committers :-) Two balances you have to keep in mind: You should be both pro-active and patient. This is a tough balance to strike. If you manage it, it helps a lot. You should be both confident and humble. Be humble, but be convinced of your idea. How to help here? The best thing is to run all the tests. It will give you confidence that your solution works (if it does). And it'll make you humble once you realize all the end cases that Django (and thus your fix) needs to support. There are three broad categories of contributions: Bug fixes Start with ... -
Processing payments for the paranoid - Andy McKay
Everyone should be paranoid when processing payments. The client, the programmer, everyone. He works on Firefox OS and more especially the marketplace ("don't call it an app store"). The marketplace is powered by Django. And of course it accepts payments. And of course it is open source (even the presentation is on github). Btw, they have a bug bounty in place. If you find a real bug, mail them and they'll pay you a bounty! The firefox add-on website already allows donations for firefox add-ons, handled through paypal. 500-2000 dollar per day. But the marketplace will process much larger amounts of money, so they needed to increase their paranoia level. For online payments, you need tokens and credentials. And they need to be stored somewhere. And suddenly you're a big fat juicy target just waiting to be hacked. XSS (cross site scripting) is an oft-occurring problem. Django has build-in protection for common cases. There's also content security policy that further limits it. They also started navigator.mozpay. Phishing. In-person tricks. For instance for getting your hand on a database for test usage. You do need something for debugging, so they now create an automatic anonymized debug database. SQL injection and so. ... -
The imaginative programmer - Zed Shaw
His goal: teach programmers to be more creative. He's got a love/hate relationship with creativity. The first part of his talk was impossible to summarize. You'll have to watch the video later on :-) Artists tell him he's not artistic because he works on developing technical skills. Guitarists tell him he's not a real guitarist as he doesn't play in a band. And 'cause he builds his own guitars he's a programmer, not a Real Guitarist. Writers tell him he's not a writer because he writes technical books. Programmers tell him he's not a programmer because he doesn't work on their project. And by the way, he's a (technical) writer now, so he's not a programmer. He's not creative enough. Or so the others say. Or he's not acceptable. How to deal with creativity? In a way, you can re-phrase creativity. Programmers are always making something from nothing, right? Isn't that the pinnacle of creativity? Here are four hypothetical persons: Technique, no imagination: a stereotypical programmer. Imagination, no technique: stereotypical biz dude. No imagination, no technique. Probably doesn't exist. Both imagination and technique. Zed's goal. Zed's imaginative programmer process. Everyone has a process (if they're good), here's the one he ... -
Advanced PostgreSQL in Django - Christophe Pettus
(See also last year's talk) Database agnosticism: write once, run on any database. A critical selling point for Django: it runs on many databases. But for others, it is bad. You pay a performance hit for not using database-specific features. So once you have made your choice, really use that database. Here are some examples of good special things available in postgres. Custom types Custom types. If you like types, you'll love postgress. Many built-in types. And many are usable in Django by installing some small app. Do you do .lower() in python code or in your SQL? For an email address for instance? Why not use citext, a case insensitive text field provided by postgres. Often you want to add various key/value data to an object. Attributes. Extra table with a join? Add fields to the main table? Solution: hstore. Postgres has a built-in json type! No need for mongodb :-) It is validated going in. Postgres 9.3 will make it much faster. The UUID type is much more efficient than storing a long character string. IPv4 and IPv6 addresses. You can define your own! And it is easy to integrate into Python and Django: You adapt it into ... -
Circus: process and socket manager - Tarek Ziadé
Tarek Ziadé can't believe he's giving a talk about his circus process manager in an actual circus tent :-) A typical deployment is with nginx and gunicorn or uwsgi. But you add more and more items right next to your django process(es). Celery or haystack for instance. So you add a supervisor that starts 'em all. An often-used one is supervisord. You could use a system-level tool like upstart, but you need root access for that. You don't need it for supervisord. Supervisord has some missing features like a powerful web console, clustering, realtime output, remote access and so. Supervisord has some of this, but not good enough. So they (mozilla) started with Circus. They used several existing libraries, like psutil, zeroMQ, socket.io. psutil is the core of the system. Very handy for interacting with processes. It was a bit slow, but together with the psutils author they managed to make it fast. ZeroMQ is an async library for message passing, so more or less a smart socket. They use message passing for making the various process data available to the circus tools, like 'circus-top' or 'circusd-stats'. And because everyting is nicely decoupled, it is possible to add your own ... -
Two scoops of Django book review
I took the train from Utrecht (NL) to Warsaw today. I only had to change in Amersfoort (NL) and Berlin (DE), so it was a pretty direct connection. 12 hours of train time (which I enjoy). So that's enough time to read through two scoops of Django, the Django book by Daniel Greenfeld and Audrey Roy! Here's my review. The summary: buy the book and learn a lot. For the longer version, I'll simply go though my notes I made for each chapter. Coding style The book starts off good, in my opinion, because it tells you to write good and neat code. PEP8. Good not-too-short variable names. And it got me thinking by advocating explicit relative imports ("from .models import SomeModel"). That's what's good about this book: Daniel and Audrey state preferences and tell you best practices and sometimes those best practices won't be your best practices. Or you didn't know something. Anyway, it gets you thinking; which is good and enjoyable. Virtualenv Hey, virtualenv in chapter two! Nice. Here, like in the rest of the book, I noticed they point a lot at existing documentation and don't provide much explanation. No virtualenv explanation here, for instance, just a ... -
Getting recommendations out of nothing - Ania Warzecha
Ania Warzecha researched recommendation systems. Recommendations means estimating ratings or preferences for items a user hasn't seen yet. For example books or movies you might also like based on earlier purchases. There are three kinds of recommendations. Collaborative recommendations. Mostly created based on actions from other users. Which books are often bought together, for instance. Simple to implement, but can be slow for big datasets. And doesn't work well on new items and/or new users Content-based recommendations. Looks for similar items. Fast and accurate, but tends towards over-specifications regarding needed data. Hybrid methods. Combining them. A case study: a Polish car parts website. You normally don't log in there, you just want a part. So older purchases aren't available. They did have a lot of parts and data, so they started with content-based recommendations. They mixed in some basic user actions. 0=didn't buy, 1=browsed, 2=bought. Later on more elaborate, like points for items found through searching or items placed on wishlists. They used Redis for its quick addition of user actions, simply pushing an additional score to an item which then gets added in the database. One thing they needed to do was to merge session keys after a user ...
