Django community: RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Dynamic models in Django - Juergen Schackmann
The classical approach in django is: Code development You create models. Deployment Tables and columns are created with syncdb. Runtime Models and db tables are populated. This means that models are pretty much static. There is no way to modify them at runtime based on user interactions. You can get something working with for instance hstore in postgresql (see the postgresql talk). His usecase is for medical forms. The contents of those forms should be able to be defined inside the system. There are strict processes for installing medical software, so you cannot just release a new version with a new field. So you must get it to work at runtime. The solution could be to use dynamic models, models created at runtime. Sometimes configuration by subject matter experts is better than code customization by developers. Also, dynamic models reduce the number of deployment cycles. He has some criteria: Performance. Querability, which means the standard django query stuff should work. Django standard tool integration (admin, cache, and so). Supported DB backends. If possible, support all django DB backends. Complexity/maintainability. There are a couple of possible solutions: Entity attribute value (EAV) Colums are stored in separate table rows. Instead of a … -
Does your stuff scale? - Steven Holmes
They grew from a two-person company to a 70-person one in two years. Central to that growth were Django and google app engine. Scalability means both load scalability and functional scalability. You also have to deal with organizational scalability and geographical scalability if you want to grow your organization. 1: Running Django on app engine It is easy to get confused. Is app engine real? Is it a joke? How to run your django stuff on it? Their reasons to use it: Auto-scaling. They build high-profile stuff and it needs to scale. They had a valentine day site that got a lot of attention on that day and it automatically scaled up without a change in the app. The day after it scaled down automatically, too. Services and APIs. No sysadmin needed. Some caveats with app engine: it is a sandbox. You you cannot do "pip install". The filesystem isn't there in the traditional sense; there is a blob storage instead. And it is lock-in, mostly; portability is an issue. They could work arounds these issues and ended up with a better application as a result. There are three ways (that they use) of running Django on app engine: Django … -
Lightning talks day 3 - Djangocon.eu
html5lib Browsers are terribly forgiving. Python's parsers don't deal with everything, even valid html5 docs. html5lib was a problem. Google code and so and not python 3 compatible. The new html5lib supports python 3. Github, readthedocs, works fine! See https://pypi.python.org/pypi/html5lib Real time web - Aymeric Augustin He looked at web sockets in django. He played with tulip, Guido's library for async python. He had 1000 processes calculating a 'game of life' screen and django connected with them just fine and pushed the result to the browser. PyWaw PyWaw is a python community in Warsaw. They have now had 24 meetings with about 55 attendees. At the last meeting they even had 100 people attending. They are not alone in Poland, there are other user groups. So... go back to your cities and start user groups! Scrapy Screen scraping is when you need to get structured information from the web, quickly and with no hassle. Scrapy takes the hassle out of screen scaping. It takes away the pain of parsing horrible html. It has perfect documentation and a helpful community. You can even scrape from amazon, even including logging in. What can you do? Convert SVG to VML. Stock checker for … -
Splitting up settings in Django
Splitting up settings in Django By default all Django settings are in one monolithic settings.py file. A single big file is hard to read and hard to maintain. Django users have found many ways to split up the settings into multiple files, all of them with their pros and cons. In our latest projects, we have developed yet another way, which uses file inclusion instead of importing Python scripts. The main features of django-split-settings are: Settings can be split into files and directories. Files later in the list can modify configurations, for example add or remove apps from INSTALLED_APPS. The main settings file lists the files that make up the project’s settings. Files can be marked optional. Optional files can be used to override settings per instance. Wildcards can be used in file paths. Maintains support for Django’s runserver auto-reloading. Read more... -
Apps for advanced plans, pricings, billings and payments - Krzysztof Dorosz
He runs multiple sites with a common business model: accounts with plan subscriptions. So there's an obvious need for a generic account billing application. The app should not be too specific, as that limits your business flexibility. Also it should not be too generic: you'll end up with an architecture from hell that way. And there's the billing as such: you need to pay close attention to security and so. Hard problem! What he's making is django-plans for keeping track of the billing data, the plans, etc. And django-getpaid as payment processing app. django-getpaid Some challenges for the actual payment integration: He wants it to be generic and lightweight. He doesn't want to pull in half of pypi for a payment processing app. He wants a single API so that he can switch payment brokers if needed. He wants it to be asynchronous. Synchronous processing blocks too long. Multiple currency support. None of the existing apps were good enough, so he made django-getpaid. It is stable and supports a lot of (Polish) payment systems and is pluggable if you need to add another one. Pluggability is achieved with special backends you can enable in your Django settings. This way you … -
Djangocon lightning talks day 2
Sorry if I mangled any of the names, that's the hardest part of blogging lightning talks. Many don't show their name long enough :-) Single page web apps with django and extjs - Michał Karzyński Single page apps: you're writing two apps. A front end one and a back end. The routing is done on the client side. The back end just spits out data (JSON api). ExtJS has a store that handles communication with the backend. So that talks to your JSON API. Plan that API carefully, try to keep it nicely RESTful. He showed a one minute demo. There is a longer one on his blog. Don't trust, check - Marcin Mincer & Tomek Kopczuk Check and question everything. Seek the best way. Not all good solutions are as good as they seem. They compared a standard view with a tastypie view and the regular view was much faster. They also checked, for their example, whether using jinja2 would be faster than django templates. Yes, it is faster. Despite what the two scoops book says. So: check everything for your usecase. Lessons learned - Tom Christie Tom maintains the Django rest framework project. He tells us a few … -
Fractal architectures - Laurens van Houtven
He worked twisted on. twisted And people tend to talk about subjects that are almost antithetical to how Django does things. The thing that he does different from Django is that he's not using a single data source... Once a database gets really really too big, putting multiple databaseservers next to eachother doesn't really work. You slowly start to get into expensive Oracle territory. How he set it up now is what he calls a fractal architecture. The whole accepts requests. The parts of the whole acccept requests. The parts of the parts accept requests. That's why he calls it fractal. You could also call it sharded, but that has a bad name: it is something you do when nothing else works. The way he looks at the architecture is SMTP. Email. Simple. He prefers SQLite. Simple and included in the python standard library. Sure, you can use postgres but you'll need a VM to re-create the same environment locally as on your production machine. SQLite is the same everywhere. In fact, he uses Axiom: an object store on top of SQLite. (Note: he is trying to write documentation for it at https://github.com/lvh/axiombook). Another advantage of sqlite: it is easy … -
Play nice with others - Honza Král
Many people think that reusable apps don't work: there's always something you need to change or modify. Honza is going to talk about his experience with ella, a django CMS. He advocates using model inheritance. from ella.core.models import Publishable and then subclass your specific model (YoutubeVideo, for instance) from it. That Publishable has most of the basic CMS functionality. That way you get most of what the CMS needs for free and you still can extend it. Showing the new model? You can use different templates easily. render_to_string() and friends accept a list of templates. So you can give it ['publishable.html', 'youtubevideo.html'] and so, using templates named somewhat after the model. This way you can re-use basic templates, but modify them if you want, just by providing a specially-named template. No code changes necessary. They're using Redis to collect information from the Django database on publishables. This way you don't have any problem with Django's database's behaviour of focusing on a single kind of model at a time. They also use django-appdata for storing extra data on existing Django models. From the pypi page: extandable field and related tools that enable Django apps to extend your reusable app. Through a … -
Taming multiple databases with Django - Marek Stępniowski
Marek works at SetJam: "We came to Django for the views, but stayed for the ORM". Django's ORM is pretty much in the sweet spot. SQLalchemy in comparison is less nice, having to learn a non-sql, non-pythonic language. At SetJam, they have what they call a backend and frontend. The backend collects data and stores it in the database, the frontend spits it out, mostly via feeds. They started out with one single big database, but that was hard to optimize. Many backend servers would write to the same database and the frontend server would read from it. Hard to optimize. Next they added a database slave for reading. That was before Django's multi-db support, so they had if/elses in their settings files based on environment variables. After Django's multi-db support, they could really support two databases and refer to them in the code with 'DEFAULT' and 'SLAVE'. Later on they splitted up the database even more. What goes where is handled by two custom database routers: a "MasterSlaveRouter" for the master/slave distinction and an "AppRouter" for shuffling some apps' data to certain databases. Tip: look at https://github.com/jbalogh/django-multidb-router, especially for the handy decorators (@use_master, for instance) it provides. At a … -
The web of stuff - Zack Voase
A plane flew over (noisily) at the start of his presentation. He put our work in perspective by saying that that was a 80 ton plane and that we're just building websites :-) Possibilities Computers used to take up whole rooms, now you have a smartphone. Big data is really big data now. Moore's lawworks both ways, though, so you have really small computers now. An arduino for instance. He often makes comparison to the human body. All over our body, sensors give off signals that go into the central nervous system. The brain processes it and gives signals back to muscles if necessary. Sensing, feedback, understanding, reaction. Stuff can talk to the cloud. Like a sensor in your body talks to your mind, stuff can treat the cloud as a brain. The cloud is what allows small tools to be smart. Stuff does often need a human to interact with it. Like a smartphone. There's all sorts of people thinking about how to "liberate the computers from their human overlords". Why cannot computers sense and act on their own account? So how do you bridge the gap betwen sensing and acting of stuff? How do you use Django for … -
Getting past Django ORM limitations with Postgres - Craig Kerstiens
Tip: subscribe to the postgresql weekly newsletter that Craig makes. Why postgres? A colleague described it as "it is the emacs of databases". There's just so much available inside postgres. The problem is Django: it treats all databases the same. It doesn't prefer one over the other. It doesn't give special treatment. Look at all the types that Christophe mentioned yesterday: Django only supports a few of 'em. Likewise indexes. For instance postgresql's Array type. Django doesn't support it, but it'd be perfect for for instance a list of tags on a model. For many of these types, also for the Array type, you have django apps that add support for them. Great: hstore. NoSQL in your SQL. A key/value store in your SQL. They use it inside Heroku a lot: it scales fine and works fine. To use it in Django, use django-hstore. Add a data field as hstore to a model and suddenly you can do my_object.data = {'key': 'value', ...}! Queuing: most people use celery. Postgres is a great queue. There's a celery backend called trunk for it. Postgresql has great text search. You do need to do some setup in your models, but then it works … -
Bleed for speed - Rob Spectre
He started with a little history lesson. The sea battle of mobile bay. The admiral (Faragut) ordered the ships straight through the minefield (called "torpedoes" at the time). "Damn the torpedoes, full speed ahead". And it worked. What does this to have to do with Django? Well, "damn the torpedoes, full speed ahead" feels a bit like how rapid prototyping feels afterwards. He's often involved with hackathons. Lot of quick coding in limited time with a lot of people. He learned a lot about his tools that way (and he often used Django). There's a time to make a distinction between production and prototype. Sometimes it is better to just try something with a prototype. Throw-away code. Aaargh! Throw-away code?!? We never throw code away. But it is something we must learn. It is good to let go once in a while. Let your code go. It isn't yourself, it is just some code. The danger is that prototype code is put into action as production code. With some work, this danger can be prevented. What about Django? Django is the best for prototyping. For rapid prototyping, Django is better than micro-frameworks like Flask that might seem better at first … -
Advanced Python through Django: metaclasses - Peter Inglesby
Metaclasses are a handy feature of Python and Django makes good use of them. When you create certain kinds of classes in Django, a metaclass will do something to the class before it is created. For forms, the various attributes of the class are converted into a base_fields dictionary on the class. Similarly, a subclass of Model also fires up a metaclass that does some registering. A foreignkey to another model adds a relation back on that other model, for instance. As a recap, a class is something that can be instantiated into an object. It can have an __init__() method that does something upon instantiation. type(your_instance) will return the class. Did you know that you can create classes dynamically? See for yourself: >>> name = 'ExampleClass' >>> bases = (object,) >>> attrs = {'__init__': lambda self: print('Hello from __init__')} >>> ExampleClass = type(name, bases, attrs) >>> example = ExampleClass() Hello From __init__ >>> type(example) <class '__console__.ExampleClass'> So... we can actually control how classes are created! You could create a create_class() method that calls type but that modifies, for instance, the name. Or we could take all the attributes and add them to a base_fields dictionary on the instance. Hey, … -
Growing open source seeds - Kenneth Reitz
He shows us three kinds of (more or less) open source projects. Type 1: public source Once upon a time there was an "open source project" called the facebook SDK. Basically it just stopped working one day and nobody could help, despite offers for help on the issue tracker. Hacker news got wind of it and it was on the front page for a while. Facebook's reaction? Disabling the issue tracker... (Later on they fixed it). That's not open source, that's public source. Often it is abandoned due to loack of interest, change of focus or so. The motivation for having it as open source simply is not clear. Type 2: shared investment A different example: gittip. They aim to be the world's first open company. There's a github issue for everything, even the company name. Major decisions are voted for on github. The code is open source, of course. All interviews with journalists are filmed and live-streamed. And all otherwise-often-backdoor-cooperation-agreements are fully open. Projects like gittip are shared investment projects. Shared ownership, extreme transparency. There is very little questioning of motivations. The motivation is clear and public. There's a documented process for new contributers. The advantage? It is low … -
Mock your database - Marc Tamlyn
Your database is slow. Especially for testing. Nobody likes waiting, so SPEED is essential. Speed of iteration. Faster unit tests mean more tests. No one likes waiting. Once you create a couple of model objects in your tests, you're going to hit your database. He did a quick test with some 8 database queries. 5 seconds for 1000 runs on postgresql. 4 seconds for in-memory sqlite, so that's a bit better. Why not use factory_boy? That could help, right? More efficient instantiation, perhaps, all in one go (as the test was with a bunch of related objects)? 6 seconds on postgres, sigh. Higher. He tried a dirty dirty dirty crazy hack, using Django internals. (I looked to my right and the #1 Django database core committer was grinning from ear to ear...). No database queries. 1 second. Better: mocking. So he did some still-quite-hacky mocking with the mock library. 1.2 seconds. Ok, but the code is a bit yucky. Summary till now: avoiding the database is quicker. The difference is bigger than the difference between postgresql and in-memory sqlite. So... write your code in such a way to make avoiding the database easier. But... is there a better way? He … -
Taming ajax and django - Marc Egli and Jérémie Blaser
Jérémie is a frontend developer and Marc does the backend. Address/state handling and content rendering are the two main challenges. Address and state handling Problems: Browser history. If you don't watch out, the back button won't be working. Deeplinking should stay possible. Crawler visibility: you want them to grab your entire site. But they don't use javascript. So you need a special URL for them Some solutions: A hash like http://yoursite.com/#/some/id. Javascript will need to handle everything behind the hash. Problem: without javascript it isn't visible. You're invisible to crawlers. It is easy to implement, though. A hashbang like http://yoursite.com/#!/some/id. The difference? Google and others replace the URL with http://yoursite.com/?_escaped_fragment_/some/id. You'll have to configure your website to support it. Deeplinks work this way and crawlers can access the site via links in a search engine sitemap. It works with almost all browsers. And it covers all three mentioned problems. You have multiple URLs, however. And you'll need to maintain legacy URLs. In django you could implement it with some middleware that detects the _escaped_frament_ GET parameter. Pushstate. The URL is a regular URL like http://yoursite.com/some/id. The best example is the github website. Pro: easier to implement on the backend, good … -
Know Your Models
In web development, we have a unfortunate double meaning for the word “models” and as obvious as the separation of those two seems to seasoned developers, it shows again and again that it’s not as obvious to beginners. Anyone who took a class on object-oriented design in the last 20 years has inevitable heard about MVC. While there are recurring (and consistently fruitless) discussions about what exactly is what, what should be named how and how they should interact, the basic common understanding is more or less: You separate your application into models, views and controllers (or models, templates and views). And ideally, you should be able to tamper with/replace either mostly independently of the other two. This article concentrates on the models part which is the data the other components work with and display to the user. Ideally it should be a pleasant API that abstracts away the storage intricacies. However… There is more than one kind of models… Django calls its ORM classes ‘models’. Pylons/Pyramid calls the skeleton part for SQLAlchemy ‘models’. I’m going to claim that that’s rather misleading since unless you’re building a simplistic CRUD application, your application models are not the same as ORM models … -
2013 EU Djangocon introduction
I'm at the 2013 European djangocon in Warsaw! Ready for three days of conferencing and, for me also, live blogging :-) Russell Keith Magee started off the conference. He remembered Malcolm Tredinnick, mentioning his code contributions, but especially his community involvement. Lots of mailinglist messages. Lots of personal involvement, too, as he visited many people and local communities. Not only Django: also chess, for instance. And he build a community here, too: working on the Australian chess community. He passed away unexpectedly a few months ago. Make the most of the time you have. It can be over quickly. And especially: be part of communities. Make communities work. And especially make this Django community work. Make friends. Enjoy our friendly community! -
Two scoops of Django book review
I took the train from Utrecht (NL) to Warsaw today. I only had to change in Amersfoort (NL) and Berlin (DE), so it was a pretty direct connection. 12 hours of train time (which I enjoy). So that's enough time to read through two scoops of Django, the Django book by Daniel Greenfeld and Audrey Roy! Here's my review. The summary: buy the book and learn a lot. For the longer version, I'll simply go though my notes I made for each chapter. Coding style The book starts off good, in my opinion, because it tells you to write good and neat code. PEP8. Good not-too-short variable names. And it got me thinking by advocating explicit relative imports ("from .models import SomeModel"). That's what's good about this book: Daniel and Audrey state preferences and tell you best practices and sometimes those best practices won't be your best practices. Or you didn't know something. Anyway, it gets you thinking; which is good and enjoyable. Virtualenv Hey, virtualenv in chapter two! Nice. Here, like in the rest of the book, I noticed they point a lot at existing documentation and don't provide much explanation. No virtualenv explanation here, for instance, just a … -
Circus: process and socket manager - Tarek Ziadé
Tarek Ziadé can't believe he's giving a talk about his circus process manager in an actual circus tent :-) A typical deployment is with nginx and gunicorn or uwsgi. But you add more and more items right next to your django process(es). Celery or haystack for instance. So you add a supervisor that starts 'em all. An often-used one is supervisord. You could use a system-level tool like upstart, but you need root access for that. You don't need it for supervisord. Supervisord has some missing features like a powerful web console, clustering, realtime output, remote access and so. Supervisord has some of this, but not good enough. So they (mozilla) started with Circus. They used several existing libraries, like psutil, zeroMQ, socket.io. psutil is the core of the system. Very handy for interacting with processes. It was a bit slow, but together with the psutils author they managed to make it fast. ZeroMQ is an async library for message passing, so more or less a smart socket. They use message passing for making the various process data available to the circus tools, like 'circus-top' or 'circusd-stats'. And because everyting is nicely decoupled, it is possible to add your own … -
The imaginative programmer - Zed Shaw
His goal: teach programmers to be more creative. He's got a love/hate relationship with creativity. The first part of his talk was impossible to summarize. You'll have to watch the video later on :-) Artists tell him he's not artistic because he works on developing technical skills. Guitarists tell him he's not a real guitarist as he doesn't play in a band. And 'cause he builds his own guitars he's a programmer, not a Real Guitarist. Writers tell him he's not a writer because he writes technical books. Programmers tell him he's not a programmer because he doesn't work on their project. And by the way, he's a (technical) writer now, so he's not a programmer. He's not creative enough. Or so the others say. Or he's not acceptable. How to deal with creativity? In a way, you can re-phrase creativity. Programmers are always making something from nothing, right? Isn't that the pinnacle of creativity? Here are four hypothetical persons: Technique, no imagination: a stereotypical programmer. Imagination, no technique: stereotypical biz dude. No imagination, no technique. Probably doesn't exist. Both imagination and technique. Zed's goal. Zed's imaginative programmer process. Everyone has a process (if they're good), here's the one he … -
Processing payments for the paranoid - Andy McKay
Everyone should be paranoid when processing payments. The client, the programmer, everyone. He works on Firefox OS and more especially the marketplace ("don't call it an app store"). The marketplace is powered by Django. And of course it accepts payments. And of course it is open source (even the presentation is on github). Btw, they have a bug bounty in place. If you find a real bug, mail them and they'll pay you a bounty! The firefox add-on website already allows donations for firefox add-ons, handled through paypal. 500-2000 dollar per day. But the marketplace will process much larger amounts of money, so they needed to increase their paranoia level. For online payments, you need tokens and credentials. And they need to be stored somewhere. And suddenly you're a big fat juicy target just waiting to be hacked. XSS (cross site scripting) is an oft-occurring problem. Django has build-in protection for common cases. There's also content security policy that further limits it. They also started navigator.mozpay. Phishing. In-person tricks. For instance for getting your hand on a database for test usage. You do need something for debugging, so they now create an automatic anonymized debug database. SQL injection and so. … -
Advanced PostgreSQL in Django - Christophe Pettus
(See also last year's talk) Database agnosticism: write once, run on any database. A critical selling point for Django: it runs on many databases. But for others, it is bad. You pay a performance hit for not using database-specific features. So once you have made your choice, really use that database. Here are some examples of good special things available in postgres. Custom types Custom types. If you like types, you'll love postgress. Many built-in types. And many are usable in Django by installing some small app. Do you do .lower() in python code or in your SQL? For an email address for instance? Why not use citext, a case insensitive text field provided by postgres. Often you want to add various key/value data to an object. Attributes. Extra table with a join? Add fields to the main table? Solution: hstore. Postgres has a built-in json type! No need for mongodb :-) It is validated going in. Postgres 9.3 will make it much faster. The UUID type is much more efficient than storing a long character string. IPv4 and IPv6 addresses. You can define your own! And it is easy to integrate into Python and Django: You adapt it into … -
Having your pony and committing it too - Jacob Burch
Jacob Burch hopes you can learn from him if you're new at contributing to open source. He won't cover virtualenv, git, django's core code structure. And also not what to get involved in. What's this talk about? About you if you have something you want ("a pony") to get into Django core. You are initially probably going to be a bit afraid. Jacob showed a couple of quotes about people that were initially not quite sure/certain when committing to Django. Then he showed the names of the people those quotes came from: they're now all core committers :-) Two balances you have to keep in mind: You should be both pro-active and patient. This is a tough balance to strike. If you manage it, it helps a lot. You should be both confident and humble. Be humble, but be convinced of your idea. How to help here? The best thing is to run all the tests. It will give you confidence that your solution works (if it does). And it'll make you humble once you realize all the end cases that Django (and thus your fix) needs to support. There are three broad categories of contributions: Bug fixes Start with … -
Getting recommendations out of nothing - Ania Warzecha
Ania Warzecha researched recommendation systems. Recommendations means estimating ratings or preferences for items a user hasn't seen yet. For example books or movies you might also like based on earlier purchases. There are three kinds of recommendations. Collaborative recommendations. Mostly created based on actions from other users. Which books are often bought together, for instance. Simple to implement, but can be slow for big datasets. And doesn't work well on new items and/or new users Content-based recommendations. Looks for similar items. Fast and accurate, but tends towards over-specifications regarding needed data. Hybrid methods. Combining them. A case study: a Polish car parts website. You normally don't log in there, you just want a part. So older purchases aren't available. They did have a lot of parts and data, so they started with content-based recommendations. They mixed in some basic user actions. 0=didn't buy, 1=browsed, 2=bought. Later on more elaborate, like points for items found through searching or items placed on wishlists. They used Redis for its quick addition of user actions, simply pushing an additional score to an item which then gets added in the database. One thing they needed to do was to merge session keys after a user …
