Django community: RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
PyGrunn: Processes, data and (the) rest - Henk Doornbos
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Henk Doornbos gave a talk in 2011 at PyGrunn about large codebases. Nice talk, that one, so I was looking forward to his talk this year. Programming is nice, especially with Python, but in the end the most important thing is to make sure you've handled all the processes and all the data in whatever you've got to build. Designing information systems. How do you do it? Talking to people, reading, brainstorming. It is hard to determine which data you have to work with. A big problem is that data is often stored in databases. Often legacy databases. And databases are much harder to refactor than code! So figuring out the data is very important. Getting the data structure out of written use cases is quite some work. You can look for nouns, for names: those often give important clues. But you'll still miss things. Trying to describe the process also gives clues. Often the process description language/diagram and the data language/diagram don't match. What he's looking for is a reliable, repeatable way to: Precisely describe a business process Find the complete and sufficient set … -
PyGrunn: Geoprocessing with Python - Greg Kowal
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Grek Kowal will talk about geoprocessing with GDAL/OGR. Geoprocessing is a lot about map projections. The earth is round, but not quite. And "round" needs to be mapped on a flat square screen, so you need to to reproject. Lots of ways to do that. What do people do with geoprocessing? Analyzing sensor data, calculating optimal traffic routes, flood risks, etc. Often you have to take multiple data sources and merge them somehow to end up with a proper map. What's there in python? Shapely is nice and pythonic, but it cannot handle many data sources. ArcPy is proprietary. GDAL/OGR is a real swiss army knife: useful like hell, but ugly. QGIS is a graphical swiss knife, very handy for visually toying with your data. GDAL is for rasters (bitmap), OGR for vectors (point, line, polygon). They're now distributed together. (Note: if you manage to install it on OSX you're a king: it is hard...). Why are they so interesting? OGR supports 78 formats from postgresql to the Czech cadastral exchange data format. GDAL in turn supports 133 formates from PNG and JPEG to a … -
PyGrunn: Modern authentication in python web apps - Arthur Barseghyan
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Arthur Barseghyan talks about SSO (single sign on) and two-factor authentication. Single sign on If you have multiple web frameworks and websites, normally every one of them would need a user database and its own authentication system. Without SSO, you could perhaps (bad idea) pick one of them and make that the leading one and hack the rest to support that one. Or you'd expect users to log in multiple times (also a bad idea). Or you could use a custom API to let the sites communicate their authentication data (also a bad idea). With single sign on you don't have many of these problems. As an example, he uses (JaSig) CAS , a java enterprise single sign-on solution. There are a whole lot of plugins. It is open source, scalable and well documented. It supports lots of backends. For logging in you need three parties: a web browser, the CAS server, your application server. Your application server functions as a CAS client. Pro: Centralised authentication for all frameworks and applications. No problem when one app is in Django and the other one in Flask … -
PyGrunn: Documentation is king - Kenneth Reitz
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Note: Kenneth Reitz gave another talk earlier during the day called "growing open source seeds". I was at a different talk, but the title is the same as a talk he gave at last year's EU djangocon. And I do have a summary of that one. A good talk! Note 2: can't get enough? He also compared Django and Flask at the 2013 EU djangocon. Recommended, mainly because of his suggestion to use open source all the things as your software architecture. "It is a helpful mindset to at least treat everything you make like it will be open sourced, even if you won’t actually do it." Anyway, on to his actual talk about documentation! He found a trend in all the stuff he does: the best things are simpler. For instance: prime lenses, handheld games, pen and paper, mechanical watch, a single carry-on bag. Constraints foster creativity. So something that constrains helps you get creative. Kenneth needs simple things to function well. The most well-known Python thing he's build is the requests library. Nice and simple and waaaaay more useful than the default python … -
PyGrunn: SSL, CAs and keeping your stuff safe - Armin Ronacher
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Armin often talks at PyGrunn. I've got three older summaries: the future of wsgi and python 3 (2010), the state of python and the web (2011), and I am doing http wrong (2012). The one I liked most was his 2011 talk at the EU djangocon about the impact of Django. He's proposing a new title for his talk: a capitalistic and system conformant talk about encryption. Well, no, it is just about SSL. He's working at "splash damage/fireteam": infrastructure for games. They want to keep someone from gaming the games, so they encrypt everything. (Note: I like one of their ipad games very much, rad soldiers). It is too easy to forget the bigger picture. He uses an analogy between bitcoin and credit cards. Bitcoins are completely encrypted and whatever. The credit card number, in contrast, is very insecure and unencrypted. But the whole credit card process is very secure. If someone steals your bitcoin key, you lose all the money, but you won't have the same problem with your credit card if it were stolen. So: think about the bigger picture! Regarding encryption: … -
PyGrunn: Writing idiomatic python - Jeff Knupp
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Jeff Knupp wrote an ebook about writing pythonic code. The subtitle of his talk is towards comprehensible and maintainable code. "Idiomatic python" doesn't mean "idiotic snake". It means "pythonic code". Code written in the way the Python community has agreed it should be written. Who decided this? Well, all the python developers through the code they write, share and criticize. The patterns you see there. Who really decides is sometimes the BDFL (Guido) or a PEP. Why would you? Three reasons: Readability. This helps people read your code. You keep the "cognitive burden" low. If I have to think during reading your code, reading your code is harder. I don't want to remember things if it isn't necessary. "Cognitive burden" is the best measure of readability. Obligatory Knuth quote, paraphrased: write code to explain to a human what we want the computer to do, don't write just for the computer. Maintainability. Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. Correctness. If you're the only one that can read your code, correctness … -
PyGrunn: gevent - Denis Bilenko
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). gevent gives you something that works pretty much like threads without many of the drawbacks. Denis Bilenko is the author of it. It works by using user-level event loops. Gevent mostly works at the level of IO, for instance for web connections, sockets, processes, etc. Gevent tries to be very stdlib compatible. Gevent modules are mostly drop-in replacements for python standard lib ones: just change the import and you use the gevent version. This helps a lot with the learning curve ("just the generic stdlib way of working") and it also helps by forcing backward compatibleness ("it has to work like the stdlib version"). Note: gunicorn, used a lot for Django, uses gevent behind the scenes. Gevent was written to avoid the complexity of event loops. Event loops mean you give up most of the control of your code to whatever is happening inside those loops. Regular python exception handling becomes almost impossible. And you have to give up context managers. And the common synchronous programming style. Giving all this up is not needed: use gevent. Gevent uses "greenlets" behind the scenes: so-called "stackful co-routines". … -
PyGrunn: Sphinx plus Robot framework - Pawel Lewicki
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Pawel Lewicki talks about documention as the result of functional testing. Sphinx is Python's standard way of creating documentation from rst files. The second technology he uses is the Robot framework, including the selenium2 plugin for headless browser testing. Robot framework test files are readable text files. Customers can read it pretty well, the tests contain regular English like "open browser to login page", "input text id_login demo-user", "click button css=.primary-action". Certain words in those text files are "keywords": words with special meaning to a test plugin. "Click button" is one of the selenium2 ones, for instance. There's also a selenium2screenshots plugin with keywords like "add pointy note". And especially "capture": capture a screenshot. This gets placed somewhere in the sphinx doc directory and can be included in the documentation. Screenshots that are always up to date!! He showed some examples. Robot framework code can be included in sphinx, with the proper plugins installed, with a .. code:: robotframework statement, followed by the robot code. A regular .. image:: instruction then includes the screenshot. Adjusting the screenshots regarding viewpoint height and width is possible. For … -
PyGrunn: Advanced continuous integration - Dirk Zittersteyn
(One of the summaries of the one-day 2014 PyGrunn conference in Groningen in the Netherlands). Dirk Zittersteyn introduces us to continuous integration. Always use version control. Even when you're programming on your own. Git, mercurial, subversion. Use branches. Keep the master/trunk branch working always. The master needs to be deliverable at all times. Changes and fixes and new features are made on branches and merged when ready. "Always be integrating". Modified quote from a movie. The mainline should always be "green". All the tests should run at all time. A problem is that agreements aren't always reality. Not everything is tested. You still can get broken code in. Just search github for the string "removed debug statement"... And two branches might be green, but combined they might still bring your mainline down. So... you must check the merge. And, really, you shouldn't merge manually. The mainline shouldn't be merged by mortals, a script should do it. The merge you're trying out should be, again, on a separate branch and only be merged into the mainline when green. Doing this manual isn't good, so they let Jenkins do it. The documentation is on https://github.com/paylogic/paylogic-jenkins-plugins . One of the things they use … -
Using PyCharm with a Django in a Buildout
To introduce a coworker to our project and Django in general, I suggested that he would try PyCharm, a Python IDE. One of the (many) nice things of PyCharm is that you can easily jump to the place where something is declared—ideal for exploring a project. When you use for instance a virtualenv, PyCharm will automatically detect which packages are installed. But PyCharm also supports buildout. As the documentation rightfully claims, PyCharm can automatically detect the use of buildout and enables support for it automatically. However, it defaults to using the path from the bin/buildout script. This usually only adds the setuptools and zc.buildout eggs, so it is of little use. Assuming you are using the djangorecipe buildout recipe, there is also a bin/django script available. And that script includes the paths to all the packages you have specified in your buildout. To do this, go to the settings, search for “buildout” and point PyCharm to the right script. Once you have done that, you can immediately see that code completion works, but CTRL+click also takes you to the declarations inside e.g. the Django package. -
Using PyCharm with Django in a Buildout
To introduce a coworker to our project and Django in general, I suggested that he would try PyCharm, a Python IDE. One of the (many) nice things of PyCharm is that you can easily jump to the place where something is declared—ideal for exploring a project. When you use for instance a virtualenv, PyCharm will automatically detect which packages are installed. But PyCharm also supports buildout. As the documentation rightfully claims, PyCharm can automatically detect the use of buildout and enables support for it automatically. However, it defaults to using the path from the bin/buildout script. This usually only adds the setuptools and zc.buildout eggs, so it is of little use. Assuming you are using the djangorecipe buildout recipe, there is also a bin/django script available. And that script includes the paths to all the packages you have specified in your buildout. To do this, go to the settings, search for “buildout” and point PyCharm to the right script. Once you have done that, you can immediately see that code completion works, but CTRL+click also takes you to the declarations inside e.g. the Django package. -
Using PyCharm with Django in a Buildout
To introduce a coworker to our project and Django in general, I suggested that he would try PyCharm, a Python IDE. One of the (many) nice things of PyCharm is that you can easily jump to the place where something is declared—ideal for exploring a project. When you use for instance a virtualenv, PyCharm will automatically detect which packages are installed. But PyCharm also supports buildout. As the documentation rightfully claims, PyCharm can automatically detect the use of buildout and enables support for it automatically. However, it defaults to using the path from the bin/buildout script. This usually only adds the setuptools and zc.buildout eggs, so it is of little use. Assuming you are using the djangorecipe buildout recipe, there is also a bin/django script available. And that script includes the paths to all the packages you have specified in your buildout. To do this, go to the settings, search for “buildout” and point PyCharm to the right script. Once you have done that, you can immediately see that code completion works, but CTRL+click also takes you to the declarations inside e.g. the Django package. -
Django1.6中自定义用户身份验证功能的教程[附源码]
本教程用于帮大家熟悉如何在Django中自定义用户身份验证功能。通过一个自定义的身份验证类，实现一个最简单的用 […] -
Custom queries II: SQLite
Let's see how to connect to an SQLite database from Python. The current version is SQLite3.Open a Python file, and import the library:import sqlite3Now, open a connection to the database. If the .db file already exists, connection to that file is opened, else a file of that name is created and connection is opened.conn = sqlite3.connect ("/foo/bar/filename.db")For executing queries, we need a cursor. So let's initialize that:cur = conn.cursor ()Now let's write a query.cur.execute ("select something from some_table")You can directly pass the string as a parameter, but for more customisations, you should initialize it separately and pass it as follows:query = "select foo from bar"cur.execute (query)If you want to insert data dynamically into your query, the following is the syntax:THIS DOES NOT WORK:query = "select empid from employee where name = '%s'" % namecur.execute (query)THIS SHOULD BE DONE:query = "select empid from employee where name = ?"data = (name,)cur.execute (query)Note that data has to be a list or tuple. The syntax shown forces a single element to be a tuple. However, if there are multiple elements, just list them out normally like this:query = "select empid, dept from employee where name = ? and salary = ?"data = (name, sal)cur.execute (query)No data … -
Custom queries I: Comparison between SQLite and MySQL
I've already said in earlier posts that you can use SQLite or MySQL as a database for your website. Recently when I was working on one I had to move to MySQL (I'll tell you why later in this article). So I thought I'll tell you guys how to transition.But first, let's compare.SQLite:It is a lightweight DBMSIt is mainly used for testing and for small websites that do not generate too much data.The biggest advantage is that it handles all the data through a .db file which can be transported elsewhere and used in the same wayIt supports up to 2^64 rows (wow!)On the downside, for the kind of data that giants like Facebook and Google generate, SQLite falls laughably shortIt also has low concurrency control, which means that it crashes if too many people are trying to access the same dataThe last point, the concurrency control, was the main reason I decided to switch to MySQL. The database of the website was crashing again and again. Let's look at MySQL now.MySQL:It is tough, inflexible and an extremely powerful DBMSIt is used by Facebook, Google, Twitter, pretty much all the web giants that exist todayIt has very high concurrency controlIt … -
Migrating Django projects with fixtures
When migrating a Django project it's often useful to work with the built-in dumpdata and loaddata management commands. However, this can easily lead to errors like django.db.utils.IntegrityError: Problem installing fixture Could not load foo.Bar(pk=ba): (1062, "Duplicate entry for key 'app_label'"). Such errors indictate that a similar object already exists in the database, and that you can't insert the duplicate from your fixture. There are basically two approaches to dump data in a way that solves this problem: Specify exactly which models to dump Exclude the conflicting models from the dump For the first approach you would use a command like: ./manage.py dumpdata --natural --indent=2 app1 app2 app3 auth.User > dump.json I include the auth data as well in this example, because it's what I usually do. You want to include all third party apps that create their own database tables and store user generated content. You want to avoid apps that auto-populate the database, like thumbnail generators, search indexers, etc. The second approach is to exclude models. You can do this like: ./manage.py dumpdata --natural --indent=2 --exclude=contenttypes --exclude=sessions.Session --exclude=south.Migrationhistory > dump.json If you googled the error django.db.utils.IntegrityError: Problem installing fixture Could not load contenttypes.ContentType(pk=X): (1062, "Duplicate entry for key 'app_label'"). … -
Migrating Django projects with fixtures
When migrating a Django project it's often useful to work with the built-in dumpdata and loaddata management commands. However, this can easily lead to errors like django.db.utils.IntegrityError: Problem installing fixture Could not load foo.Bar(pk=ba): (1062, "Duplicate entry for key 'app_label'"). Such errors indictate that a similar object already exists in the database, and that you can't insert the duplicate from your fixture. There are basically two approaches to dump data in a way that solves this problem: Specify exactly which models to dump Exclude the conflicting models from the dump For the first approach you would use a command like: ./manage.py dumpdata --natural --indent=2 app1 app2 app3 auth.User > dump.json I include the auth data as well in this example, because it's what I usually do. You want to include all third party apps that create their own database tables and store user generated content. You want to avoid apps that auto-populate the database, like thumbnail generators, search indexers, etc. The second approach is to exclude models. You can do this like: ./manage.py dumpdata --natural --indent=2 --exclude=contenttypes --exclude=sessions.Session --exclude=south.Migrationhistory > dump.json If you googled the error django.db.utils.IntegrityError: Problem installing fixture Could not load contenttypes.ContentType(pk=X): (1062, "Duplicate entry for key 'app_label'"). … -
Migrating Django projects with fixtures
When migrating a Django project it's often useful to work with the built-in dumpdata and loaddata management commands. However, this can easily lead to errors like django.db.utils.IntegrityError: Problem installing fixture Could not load foo.Bar(pk=ba): (1062, "Duplicate entry for key 'app_label'"). Such errors indictate that a similar object already exists in the database, and that you can't insert the duplicate from your fixture. There are basically two approaches to dump data in a way that solves this problem: Specify exactly which models to dump Exclude the conflicting models from the dump For the first approach you would use a command like: ./manage.py dumpdata --natural --indent=2 app1 app2 app3 auth.User > dump.json I include the auth data as well in this example, because it's what I usually do. You want to include all third party apps that create their own database tables and store user generated content. You want to avoid apps that auto-populate the database, like thumbnail generators, search indexers, etc. The second approach is to exclude models. You can do this like: ./manage.py dumpdata --natural --indent=2 --exclude=contenttypes --exclude=sessions.Session --exclude=south.Migrationhistory > dump.json If you googled the error django.db.utils.IntegrityError: Problem installing fixture Could not load contenttypes.ContentType(pk=X): (1062, "Duplicate entry for key 'app_label'"). … -
Sending emails
If you've noticed, on many sites, once you sign up, it sends you an automated confirmation email. This is another function your site should be able to perform. So today let's take a look at that.Let's assume you have a Gmail account (if you don't, create one today, it's great!). We'll be using Gmail's email host (wow that rhymed).Go to the settings.py file and add the following code anywhere in between.EMAIL_USE_TLS = TrueEMAIL_HOST = "smtp.gmail.com"EMAIL_HOST_USER = "your_name@example.com"EMAIL_HOST_PASSWORD = "your_password"EMAIL_PORT = 587The port number 587 is pretty standard. Also, it is very important to set TLS to True. We'll go into why later.You have to enter your Gmail password there, but I assure you it is secure. Only the programmer(s) can see it. You may also create another common account and use those credentials instead.Ok, now we'll need to write the Python code to send emails.Open the views.py file in the appropriate app and write the following code:At the top, import the Django's send_mail:from django.core.mail import send_mailAlso import settings.py:from django.conf import settingsLater, create this function:def foo (request): title = "some title" message = "some message" host = settings.EMAIL_HOST_USER receiver = ["someone@example.com"] send_mail (title, message, host, receiver, fail_silently = … -
Github
Hey guys! For those of you who don't know, Github is a code sharing site on which you can review and edit other people's code and post your own. I have shown you a lot of code on this blog. Some of the code like file upload, login sessions etc is the same and can be reused. So I'm going to post such pieces of code on Github for you guys to review and edit. Please do so, I would be very happy if someone improved upon the code I've written.Also, my favorite language is Python and I do a lot of other things like software development, hacking in it. I've posted repos for these also. Please feel free to review and edit those too.Here's the link:https://github.com/AgentK1729Cheers! -
Intermittent renewal demonstrated through github graph
I've got a number of books at home in the "personal development" genre. One of the more practical and better ones is The Power of Full Engagement: Managing Energy, Not Time, Is the Key to High Performance and Personal Renewal. Core idea 1 A core idea in this book is that you've got four kinds of energy that you need to watch: Physical, health. Emotional, happiness. Mental, focus. Spiritual, purpose. As an exercise, rate yourself on a scale of 1 (bad) to 10 (good) and multiply the numbers. Should be a pretty good measurement of how well you're doing! The wake-up call is that you shouldn't disregard any of the four. If you're physically drained because of 4 hours of sleep and a sedentary lifestyle, you can be focused all you want, but you won't get too much out of yourself. And when emotionally drained because of some family mishap or huge stress, sleeping 8 hours a night won't help all that much. So... try to get all of them to a reasonable level. Core idea 2 The second core idea is that each of these four is trainable. Just like you can train your body, you can train the … -
Making a specific Django app faster
This is just a simple tale of making a specific Django app faster. Just a tale of the steps I took to give you an idea of what you can do to make your app faster. This isn't the definitive guide, it just aims at giving ideas and giving a feel for the process. tl;dr summary: First measure, then optimize. Django's caching helps. There's a nice algorithm to speed up javascript graphs. Background The app is lizard-fewsjdbc. It is on github, though we're basically the only ones using it. The basic setup is that we've got an external special-kind-of-database that talks xml-rpc to the outside world. Over the xml-rpc we can send SQL queries to the database that in turn uses a jdbc connection to execute the query. Don't ask. So... we talk to some xml-rpc and have to return information, mostly as graphs. The big performance problem is that the data is time-based. A measurement every minute. And the customers wanted to see a year of data in a graph. The graph is a javascript flot chart because of it provides nice interaction like zooming in and out. And... javascript, or rather Flot, or rather the browser, doesn't take … -
Writing a README
Good documentation is important. I just heard two colleagues talking and what one of them said sounded like a README: - Install the shit. - Run it. - Look for error messages. It triggered me to write a blog post as I was just busy writing a README for a small script. pip install is currently compiling numpy and matplotlib, so time enough for me to gather my thoughts. "Good documentation is important", yes. But there are many things that we can all agree on "yeah, that's important, too". "Being important" isn't the same as "actually getting written". Many of the small tools and django apps that we write aren't actually going to see widespread use. Many are company-internal, effectively. Is read-the-docs level documentation useful/necessary? In some cases, yes. One of the company-internal django apps I wrote has extensive documentation and 100% test coverage. That's what you get with something named "lizard-security" which uses django's row-level security and some special settings and other tricks. Especially when it is used in almost all our sites. In most cases, nobody is going to bother looking up the official documentation, even if you wrote it. All they'll glance at is the README. And … -
Profiling Django apps and Python scripts with cProfile and Kcachegrind
Some time ago I wrote about profiling Django with hotshot with mod_python that is now dead. To profile execution time of Python scripts as well as Django views, tests, or management commands we use cProfile and its wrappers. There are also converters that provide data for KCachegrind. Let us see what we have available to use. -
Uploading files on Google Cloud Storage with Django
Intro On of the features of Django Appengine Toolkit is simplifying the configurations neede to make Google Cloud Storage a static files storage for Django applications running on Google App Engine. Infact all you have to do to use GCS as the destination of your uploads is writing something like this in your settings.py module: APPENGINE_TOOLKIT = { 'APP_YAML': os.path.join(BASE_DIR, 'app.yaml'), 'BUCKET_NAME': 'media-uploads', } DEFAULT_FILE_STORAGE = 'appengine_toolkit.storage.GoogleCloudStorage' STATICFILE_STORAGE = 'appengine_toolkit.storage.GoogleCloudStorage' A complete example This repo contains a minimalistic Django project implementing a file storage application that permits users to upload files, listing and delete them. The project has just one app implementing all the logic, defining the model and exposing the views. For detailed instructions on how to setup a Django project on App Engine with django-appengine-toolkit please check out this blog post before playing with this project. Now let’s take a look at the code. The Model class Document(models.Model): docfile = models.FileField(upload_to='documents/%Y/%m/%d') def delete(self, *args, **kwargs): storage, path = self.docfile.storage, self.docfile.path super(Document, self).delete(*args, **kwargs) storage.delete(path) Pretty easy, we have just one field containing the file. Notice the delete method we’re going to use so that once deleted an instance, the same will happen to corresponding file on Cloud Storage. …
