Django community: RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
How to Upload Files With Django
In this tutorial you will learn the concepts behind Django file upload and how to handle file upload using model forms. In the end of this post you will find the source code of the examples I used so you can try and explore. The Basics of File Upload With Django When files are submitted to the server, the file data ends up placed in request.FILES. It is mandatory for the HTML form to have the attribute enctype="multipart/form-data" set correctly. Otherwise the request.FILES will be empty. The form must be submitted using the POST method. Django have proper model fields to handle uploaded files: FileField and ImageField. The files uploaded to FileField or ImageField are not stored in the database but in the filesystem. FileField and ImageField are created as a string field in the database (usually VARCHAR), containing the reference to the actual file. If you delete a model instance containing FileField or ImageField, Django will not delete the physical file, but only the reference to the file. The request.FILES is a dictionary-like object. Each key in request.FILES is the name from the <input type="file" name="" />. Each value in request.FILES is an UploadedFile instance. I will talk more … -
Django + Elastic Search + haystack = Powering your website with search functionality – part 2
Search is one of the most important and powerful functionality in web applications these days. It helps users to easily find content, products etc on your website. Without search option, users will have to find their way out to reach the desired content which no one likes to do. Just imagine Amazon without search bar, you will have to navigate through various categories to find out the product, and it may take you forever to find it. Having said that, implementing search into your web app is not difficult thanks to lot of libraries built for this purpose. I have broken down this tutorial into two parts. First article explains how to setup elastic search and haystack with django, create search search indexes and how to query data. In this article, we will take one step forward and discuss about using autocomplete, spelling suggestions and creating custom backends. If you haven’t read part-1, I suggest you do it first and then jump to this article. Remember the SearchIndex we created in the part-1 of this tutorial, we are going to continue modifying the same in this article. import datetime from haystack import indexes from myapp.models import Blog class BlogIndex(indexes.SearchIndex, indexes.Indexable): … -
How to Export to Excel
Export data to excel is a common requirement on many web applications. Python makes everything easier. But, nevertheless, it is the kind of task I need to look for references whenever I have to implement. I will give you two options in this tutorial: (1) export data to a .csv file using Python’s csv module; (2) export data to a .xls file using a third-party module named xlwt. For both cases, consider we are exporting django.contrib.auth.models.User data. Export Data to CSV File Easiest way. No need to install dependencies, which is a great thing. Use it if you do not need any fancy formating. views.py import csv from django.http import HttpResponse from django.contrib.auth.models import User def export_users_csv(request): response = HttpResponse(content_type='text/csv') response['Content-Disposition'] = 'attachment; filename="users.csv"' writer = csv.writer(response) writer.writerow(['Username', 'First name', 'Last name', 'Email address']) users = User.objects.all().values_list('username', 'first_name', 'last_name', 'email') for user in users: writer.writerow(user) return response urls.py import views urlpatterns = [ ... url(r'^export/csv/$', views.export_users_csv, name='export_users_csv'), ] template.html <a href="{% url 'export_users_csv' %}">Export all users</a> In the example above I used the values_list in the QuerySet for two reasons: First to query only the fields I needed to export, and second because it will return the data in a … -
How to Export to Excel
Export data to excel is a common requirement on many web applications. Python makes everything easier. But, nevertheless, it is the kind of task I need to look for references whenever I have to implement. I will give you two options in this tutorial: (1) export data to a .csv file using Python’s csv module; (2) export data to a .xls file using a third-party module named xlwt. For both cases, consider we are exporting django.contrib.auth.models.User data. Export Data to CSV File Easiest way. No need to install dependencies, which is a great thing. Use it if you do not need any fancy formating. views.py import csv from django.http import HttpResponse from django.contrib.auth.models import User def export_users_csv(request): response = HttpResponse(content_type='text/csv') response['Content-Disposition'] = 'attachment; filename="users.csv"' writer = csv.writer(response) writer.writerow(['Username', 'First name', 'Last name', 'Email address']) users = User.objects.all().values_list('username', 'first_name', 'last_name', 'email') for user in users: writer.writerow(user) return response urls.py import views urlpatterns = [ ... url(r'^export/csv/$', views.export_users_csv, name='export_users_csv'), ] template.html <a href="{% url 'export_users_csv' %}">Export all users</a> In the example above I used the values_list in the QuerySet for two reasons: First to query only the fields I needed to export, and second because it will return the data in a … -
How to Create Django Signals
The Django Signals is a strategy to allow decoupled applications to get notified when certain events occur. Let’s say you want to invalidate a cached page everytime a given model instance is updated, but there are several places in your code base that this model can be updated. You can do that using signals, hooking some pieces of code to be executed everytime this specific model’s save method is trigged. Another common use case is when you have extended the Custom Django User by using the Profile strategy through a one-to-one relationship. What we usually do is use a “signal dispatcher” to listen for the User’s post_save event to also update the Profile instance as well. I’ve covered this case in another post, which you can read here: How to Extend Django User Model. In this tutorial I will present you the built-in signals and give you some general advices about the best practices. When Should I Use It? Before we move forward, know when you should use it: When many pieces of code may be interested in the same events; When you need to interact with a decoupled application, e.g.: A Django core model; A model defined by a … -
How to Create Django Signals
The Django Signals is a strategy to allow decoupled applications to get notified when certain events occur. Let’s say you want to invalidate a cached page everytime a given model instance is updated, but there are several places in your code base that this model can be updated. You can do that using signals, hooking some pieces of code to be executed everytime this specific model’s save method is trigged. Another common use case is when you have extended the Custom Django User by using the Profile strategy through a one-to-one relationship. What we usually do is use a “signal dispatcher” to listen for the User’s post_save event to also update the Profile instance as well. I’ve covered this case in another post, which you can read here: How to Extend Django User Model. In this tutorial I will present you the built-in signals and give you some general advices about the best practices. When Should I Use It? Before we move forward, know when you should use it: When many pieces of code may be interested in the same events; When you need to interact with a decoupled application, e.g.: A Django core model; A model defined by a … -
How to Return JSON-Encoded Response
Since version 1.7, Django counts with the built-in JsonResponse class, which is a subclass of HttpResponse. Its default Content-Type header is set to application/json, which is really convenient. It also comes with a JSON encoder, so you don’t need to serialize the data before returning the response object. See a minimal example below: from django.http import JsonResponse def profile(request): data = { 'name': 'Vitor', 'location': 'Finland', 'is_active': True, 'count': 28 } return JsonResponse(data) By default, the JsonResponse’s first parameter, data, should be a dict instance. To pass any other JSON-serializable object you must set the safe parameter to False. return JsonResponse([1, 2, 3, 4], safe=False) See below the class signature: class JsonResponse(data, encoder, safe, json_dumps_params, **kwargs) Defaults: data: (no default) encoder: django.core.serializers.json.DjangoJSONEncoder safe: True json_dumps_params: None Extra bits: If you want to return Django models as JSON, you may want to it this way: def get_users(request): users = User.objects.all().values('first_name', 'last_name') # or simply .values() to get all fields users_list = list(users) # important: convert the QuerySet to a list object return JsonResponse(users_list, safe=False) Troubleshooting Django 1.6 or below For older versions of Django, you must use an HttpResponse object. See an example below: import json from django.http import HttpResponse def … -
How to Return JSON-Encoded Response
Since version 1.7, Django counts with the built-in JsonResponse class, which is a subclass of HttpResponse. Its default Content-Type header is set to application/json, which is really convenient. It also comes with a JSON encoder, so you don’t need to serialize the data before returning the response object. See a minimal example below: from django.http import JsonResponse def profile(request): data = { 'name': 'Vitor', 'location': 'Finland', 'is_active': True, 'count': 28 } return JsonResponse(data) By default, the JsonResponse’s first parameter, data, should be a dict instance. To pass any other JSON-serializable object you must set the safe parameter to False. return JsonResponse([1, 2, 3, 4], safe=False) See below the class signature: class JsonResponse(data, encoder, safe, json_dumps_params, **kwargs) Defaults: data: (no default) encoder: django.core.serializers.json.DjangoJSONEncoder safe: True json_dumps_params: None Extra bits: If you want to return Django models as JSON, you may want to it this way: def get_users(request): users = User.objects.all().values('first_name', 'last_name') # or simply .values() to get all fields users_list = list(users) # important: convert the QuerySet to a list object return JsonResponse(users_list, safe=False) Troubleshooting Django 1.6 or below For older versions of Django, you must use an HttpResponse object. See an example below: import json from django.http import HttpResponse def … -
uWSGI Basic Django Setup
Here are two basic examples of almost the same uWSGI configuration to run a Django project; one is configured via an ini configuration file and the other is configured via a command line argument.This does not represent a production-ready example, but can be used as a starting point for the configuration.Setup for this example:# create dir for virtualenv and activatemkdir envs/virtualenv envs/runrun/. ./envs/runrun/bin/activate# create dir for project codebasemkdir proj/# install some django depspip install django uwsgi whitenose# create a new django projectcd proj/django-admin startproject runruncd runrun/# Add to or modify django settings.py to setup static file serving with Whitenoise.# Note: for prod environments, staticfiles can be served via Nginx.# settings.py MIDDLEWARE_CLASSES = [ .... 'whitenoise.middleware.WhiteNoiseMiddleware',] STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'Create the config file$ cat run.ini [uwsgi]module = runrun.wsgi:applicationvirtualenv = ../../envs/runrun/env = DJANGO_SETTINGS_MODULE=runrun.settings env = PYTHONPATH="$PYTHONPATH:./runrun/" master = trueprocesses = 5enable-threads = truemax-requests = 5000harakiri = 20vacuum = truehttp = :7777stats = 127.0.0.1:9191Execute configuwsgi --ini run.iniOr create a shell script$ cat run.sh uwsgi --module=runrun.wsgi:application \ -H ../../envs/runrun/ \ --env DJANGO_SETTINGS_MODULE=runrun.settings \ --env PYTHONPATH="$PYTHONPATH:./runrun/" \ --master \ --processes=5 \ --max-requests=5000 \ --harakiri=20 \ --vacuum \ … -
How to Reset Migrations
./manage.py migrate --fake <app-name> zero -
How to Reset Migrations
The Django migration system was developed and optmized to work with large number of migrations. Generally you shouldn’t mind to keep a big amount of models migrations in your code base. Even though sometimes it causes some undesired effects, like consuming much time while running the tests. But in scenarios like this you can easily disable the migrations (although there is no built-in option for that at the moment). Anyway, if you want to perform a clean-up, I will present a few options in this tutorial. Scenario 1: The project is still in the development environment and you want to perform a full clean up. You don’t mind throwing the whole database away. 1. Remove the all migrations files within your project Go through each of your projects apps migration folder and remove everything inside, except the __init__.py file. Or if you are using a unix-like OS you can run the following script (inside your project dir): find . -path "*/migrations/*.py" -not -name "__init__.py" -delete find . -path "*/migrations/*.pyc" -delete 2. Drop the current database, or delete the db.sqlite3 if it is your case. 3. Create the initial migrations and generate the database schema: python manage.py makemigrations python manage.py migrate … -
ShipIt Day Recap - July 2016
We finished up last week with another successful ShipIt Day. ShipIt Days are quarterly events where we put down client work for a little bit and focus on learning, stretching ourselves, and sharing. Everyone chooses to work together or individually on an itch or a project that has been on the back of their mind for the last few months. This time, we stretched ourselves by trying out new frameworks, languages, and pluggable apps. Here are some of the projects we worked on during ShipIt Day: -
SQLAlchemy and "Lost connection to MySQL server during query"
... a weird behaviour when pool_recycle is ignored Preface: This is quite a standard problem for apps/websites with low traffic or those using heavy caching and hitting the database quite seldom. Most of the articles you will find on the topic will tell you one thing - change the wait_timeout setting in the database. Unfortunately in some of the cases this disconnect occurs much earlier than the expected wait_timeout (default ot 8 hours). If you are in one of those cases keep reading. This issue haunted our team for weeks. When we first faced it the project that was raising it was still in dev phase so it wasn't critical but with getting closer to the release data we started to search for solution. We have read several articles and decided that pool_recycle is our solution. Adding pool_recycle: According to SQL Alchemy's documentation pool_recycle "causes the pool to recycle connections after the given number of seconds has passed". Nice, so if you recycle the connection in intervals smaller that the await_timeout the error above should not appear. Let's try it out: import time from sqlalchemy.engine import create_engine url = 'mysql+pymysql://user:pass@127.0.0.1:3306/db' engine = create_engine(url, pool_recycle=1).connect() query = 'SELECT NOW();' while True: … -
Django Tips #8 Blank or Null?
Django models API offers two similar options that usually cause confusion on many developers: null and blank. When I first started working with Django I couldn’t tell the difference and always ended up using both. Sometimes even using them improperly. Both do almost the same thing, as the name suggests, but here is the difference: Null: It is database-related. Defines if a given database column will accept null values or not. Blank: It is validation-related. It will be used during forms validation, when calling form.is_valid(). That being said, it is perfectly fine to have a field with null=True and blank=False. Meaning on the database level the field can be NULL, but in the application level it is a required field. Now, where most developers get it wrong: Defining null=True for string-based fields such as CharField and TextField. Avoid doing that. Otherwise, you will end up having two possible values for “no data”, that is: None and an empty string. Having two possible values for “no data” is redundant. The Django convention is to use the empty string, not NULL. So, if you want a string-based model field to be “nullable”, prefer doing that: class Person(models.Model): name = models.CharField(max_length=255) # Mandatory … -
Django Tips #8 Blank or Null?
Django models API offers two similar options that usually cause confusion on many developers: null and blank. When I first started working with Django I couldn’t tell the difference and always ended up using both. Sometimes even using them improperly. Both do almost the same thing, as the name suggests, but here is the difference: Null: It is database-related. Defines if a given database column will accept null values or not. Blank: It is validation-related. It will be used during forms validation, when calling form.is_valid(). That being said, it is perfectly fine to have a field with null=True and blank=False. Meaning on the database level the field can be NULL, but in the application level it is a required field. Now, where most developers get it wrong: Defining null=True for string-based fields such as CharField and TextField. Avoid doing that. Otherwise, you will end up having two possible values for “no data”, that is: None and an empty string. Having two possible values for “no data” is redundant. The Django convention is to use the empty string, not NULL. So, if you want a string-based model field to be “nullable”, prefer doing that: class Person(models.Model): name = models.CharField(max_length=255) # Mandatory … -
How to Extend Django User Model
The Django’s built-in authentication system is great. For the most part we can use it out-of-the-box, saving a lot of development and testing effort. It fits most of the use cases and is very safe. But sometimes we need to do some fine adjustment so to fit our Web application. Commonly we want to store a few more data related to our User. If your Web application have an social appeal, you might want to store a short bio, the location of the user, and other things like that. In this tutorial I will present the strategies you can use to simply extend the default Django User Model, so you don’t need to implement everything from scratch. Ways to Extend the Existing User Model Generally speaking, there are four different ways to extend the existing User model. Read below why and when to use them. Option 1: Using a Proxy Model What is a Proxy Model? It is a model inheritance without creating a new table in the database. It is used to change the behaviour of an existing model (e.g. default ordering, add new methods, etc.) without affecting the existing database schema. When should I use a Proxy Model? … -
How to Extend Django User Model
The Django’s built-in authentication system is great. For the most part we can use it out-of-the-box, saving a lot of development and testing effort. It fits most of the use cases and is very safe. But sometimes we need to do some fine adjustment so to fit our Web application. Commonly we want to store a few more data related to our User. If your Web application have an social appeal, you might want to store a short bio, the location of the user, and other things like that. In this tutorial I will present the strategies you can use to simply extend the default Django User Model, so you don’t need to implement everything from scratch. Ways to Extend the Existing User Model Generally speaking, there are four different ways to extend the existing User model. Read below why and when to use them. Option 1: Using a Proxy Model What is a Proxy Model? It is a model inheritance without creating a new table in the database. It is used to change the behaviour of an existing model (e.g. default ordering, add new methods, etc.) without affecting the existing database schema. When should I use a Proxy Model? … -
Django Tips #7 How to Get the Current URL Within a Django Template
Make sure you have django.template.context_processors.request listed in your context_processors. As of Django 1.9 version, it already comes configurated. The default TEMPLATES configuration looks like that: TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ] To get the current path: {{ request.path }} Current path with querystring: {{ request.get_full_path }} Domain, path and querystring: {{ request.build_absolute_uri }} Outputs Considering we are acessing the following URL: http://127.0.0.1:8000/home/?q=test Method Output request.path /home/ request.get_full_path /home/?q=test request.build_absolute_uri http://127.0.0.1:8000/home/?q=test Troubleshooting Django 1.7 or below If you are using an older version of Django (<= 1.7) where the TEMPLATES configuration is not available, you can include the context processor like this: settings.py from django.conf.global_settings import TEMPLATE_CONTEXT_PROCESSORS as TCP TEMPLATE_CONTEXT_PROCESSORS = TCP + ( 'django.core.context_processors.request', ) Notice the context processor was available inside the core module. Since version >= 1.8 it is available inside the template module. -
Django Tips #7 How to Get the Current URL Within a Django Template
Make sure you have django.template.context_processors.request listed in your context_processors. As of Django 1.9 version, it already comes configurated. The default TEMPLATES configuration looks like that: TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ] To get the current path: {{ request.path }} Current path with querystring: {{ request.get_full_path }} Domain, path and querystring: {{ request.build_absolute_uri }} Outputs Considering we are acessing the following URL: http://127.0.0.1:8000/home/?q=test Method Output request.path /home/ request.get_full_path /home/?q=test request.build_absolute_uri http://127.0.0.1:8000/home/?q=test Troubleshooting Django 1.7 or below If you are using an older version of Django (<= 1.7) where the TEMPLATES configuration is not available, you can include the context processor like this: settings.py from django.conf.global_settings import TEMPLATE_CONTEXT_PROCESSORS as TCP TEMPLATE_CONTEXT_PROCESSORS = TCP + ( 'django.core.context_processors.request', ) Notice the context processor was available inside the core module. Since version >= 1.8 it is available inside the template module. -
Sharing Media Files Without S3
Nowadays, it's common to deploy multiple application servers, but it poses a very common problem; How are these application servers going to share the media files contributed by the users? Cloud providers like Google, Rackspace or Amazon Web Services solve this problem by providing a cloud storage service. These services are comparable to an extremely slow hard disk that exposes an HTTP API. On the positive side, they are relatively cheap ($0.03/month per GB), provide redundant storage, and can be accessed from anywhere. Django’s storage abstraction layer, in combination with django-storages lets all your application servers manage files within the same bucket in the cloud. While emulating a server farm locally in containers, I looked into the options for sharing files without using a third-party service. NFS -- Network File System Network File System (NFS) is a distributed file system protocol originally developed by Sun Microsystems in 1984, allowing a user on a client computer to access files over the network much like local storage. This is the boring and battle-tested solution. In the simplest case, an NFS share can be created by installing the server from your package manager and adding a line to /etc/exports for each directory along … -
HTTP Status Codes Site
During the development of Simple Site Checker I realised that it would be useful for test purposes if there is a website returning all possible HTTP status codes. Thanks to Google App Engine and webapp2 framework building such website was a piece of cake. The site can be found at http://httpstatuscodes.appspot.com. The home page provides a list of all HTTP status codes and their names and if you want to get an HTTP response with a specific status code just add the code after the slash, example: http://httpstatuscodes.appspot.com/200 - returns 200 OK http://httpstatuscodes.appspot.com/500 - returns 500 Internal Server Error Also at the end of each page is located the URL of the HTTP protocol Status Codes Definitions with detailed explanation for each one of them. The website code is publicly available in github at HTTP Status Codes Site. If you find it useful feel free to comment and/or share it. -
Fabric & Django
Or how automate the creation of new projects with simple script Preface: Do you remember all this tiny little steps that you have to perform every time when you start new project - create virtual environment, install packages, start and setup Django project? Kind of annoying repetition, isn't it? How about to automate it a bit. Solution: Recently I started learning Fabric and thought "What better way to test it in practice than automating a simple, repetitive task?". So, lets mark the tasks that I want the script to perform: Create virtual environment with the project name Activate the virtual environment Download list of packages and install them Make 'src' directory where the project source will reside Create new Django project in source directory Update the settings Thanks to the local command the first one was easy. The problem was with the second one. Obviously each local command is run autonomously so I had to find some way have activated virtual environment for each task after this. Fortunately the prefix context manager works like a charm. I had some issues making it read and write in the paths I wants and voilà it was working exactly as I want. The … -
Python is not a Panacea ...
... neither is any other language or framework This post was inspired by the serial discussion on the topic "Python vs other language"(in the specific case the other one was PHP, and the question was asked in a Python group so you may guess whether there are any answers in favor of PHP). It is very simple, I believe that every Python developer will tell you that Python is the greatest language ever build, how easy is to learn it, how readable and flexible it is, how much fun it is to work with it and so on. They will tell you that you can do everything with it: web and desktop development, testing, automation, scientific simulations etc. But what most of them will forgot to tell you is that it is not a Panacea. In the matter of fact you can also build "ugly" and unstable applications in Python too. Most problems come not from the language or framework used, but from bad coding practices and bad understanding of the environment. Python will force you to write readable code but it wont solve all your problems. It is hard to make a complete list of what exactly you must … -
Django compressor and image preloading
Preface: Have you noticed how on some websites when you click on a link that opens a lightbox or any overlay for first time it takes some time to display the border/background/button images. Not quite fancy, right? This is because the load of this images starts at the moment the overlay is rendered on the screen. If this is your first load and these images are not in your browser cache it will take some time for the browser to retrieve them from the server. Solution: The solution for this is to preload the images i.e. to force the browser to request them from the server before they are actually used. With a simple javascript function and a list of the images URLs this is a piece of cake: $.preLoadImages = function() { var args_len = arguments.length; for (var i=0; i < args_len; i++) { var cacheImage = document.createElement('img'); cacheImage.src = arguments[i]; } } $.preLoadImages('/img/img1.png', '/img/img2.png') Please have in mind that the code above uses the jQuery library. Specialty: Pretty easy, but you have to hardcode the URLs of all images. Also if you are using Django compressor then probably you are aware that it adds extra hash to the … -
The road to hell is paved with regular expressions ...
... or what is the cost of using regular expressions for simple tasks Regular expressions are one of the most powerful tools in computing I have ever seen. My previous post about Django compressor and image preloading is a good example how useful they might be. The only limit of their use is your imagination. But "with great power, comes great responsibility" or in this case a great cost. Even the simplest expressions can be quite heavy compared with other methods. The reason to write about this is a question recently asked in a python group. It was about how to get the elements of a list that match specific string. My proposal was to use comprehension list and simple string comparison while other member proposed using a regular expression. I was pretty sure that the regular expression is slower but not sure exactly how much slower so I made a simple test to find out. import re import timeit my_list = ['abc-123', 'def-456', 'ghi-789', 'abc456', 'abc', 'abd'] def re_check(): return [i for i in my_list if re.match('^abc$', i)] t = timeit.Timer(re_check) print 're_check result >>', re_check() print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000) def simple_check(): return [i for i …
