Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Python in Azure Pipelines, Step by Step
Since the acquisition of Travis CI, the future of their free offering is unclear. Azure Pipelines has a generous free tier, but the examples I found are discouragingly complex and take advantage of features like templating that most projects don’t need. To close that gap, this article shows you how to move a Python project with simple CI needs from Travis CI to Azure Pipelines. -
Build an XML sitemap of XML sitemaps
Suppose that you have so many thousands of pages that you can't just create a single /sitemap.xml file that has all the URLs (aka <loc>) listed. Then you need to make a /sitemaps.xml that points to the other sitemap files. And if you're in the thousands, you'll need to gzip these files. The blog post demonstrates how Song Search generates a sitemap file that points to 63 sitemap-{M}-{N}.xml.gz files which spans about 1,000,000 URLs. The context here is Python and the getting of the data is from Django. Python is pretty key here but if you have something other than Django, you can squint and mentally replace that with your own data mapper. Generate the sitemap .xml.gz file(s) Here's the core of the work. A generator function that takes a Django QuerySet instance (that is ordered and filtered!) and then starts generating etree trees and dumps them to disk with gzip. import gzip from lxml import etree outfile = "sitemap-{start}-{end}.xml" batchsize = 40_000 def generate(self, qs, base_url, outfile, batchsize): # Use `.values` to make the query much faster qs = qs.values("name", "id", "artist_id", "language") def start(): return etree.Element( "urlset", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" ) def close(root, filename): with gzip.open(filename, "wb") as f: f.write(b'<?xml … -
Build an XML sitemap of XML sitemaps
Suppose that you have so many thousands of pages that you can't just create a single /sitemap.xml file that has all the URLs (aka <loc>) listed. Then you need to make a /sitemaps.xml that points to the other sitemap files. And if you're in the thousands, you'll need to gzip these files. The blog post demonstrates how Song Search generates a sitemap file that points to 63 sitemap-{M}-{N}.xml.gz files which spans about 1,000,000 URLs. The context here is Python and the getting of the data is from Django. Python is pretty key here but if you have something other than Django, you can squint and mentally replace that with your own data mapper. Generate the sitemap .xml.gz file(s) Here's the core of the work. A generator function that takes a Django QuerySet instance (that is ordered and filtered!) and then starts generating etree trees and dumps them to disk with gzip. import gzip from lxml import etree outfile = "sitemap-{start}-{end}.xml" batchsize = 40_000 def generate(self, qs, base_url, outfile, batchsize): # Use `.values` to make the query much faster qs = qs.values("name", "id", "artist_id", "language") def start(): return etree.Element( "urlset", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" ) def close(root, filename): with gzip.open(filename, "wb") as f: f.write(b'<?xml … -
Generate a random IP address in Python
A fun trick to generate random, but seeded, IPv4 addresses in a Django app. -
Generate a random IP address in Python
A fun trick to generate random, but seeded, IPv4 addresses in a Django app. -
Bitbucket Pipelines and Ansible: Continuous delivery for your Django project
In a previous post I described how to deploy a Django project using Ansible. Here I explain how to make a step further and deploy a Django project using Bitbucket Pipelines and Ansible. Bitbucket Pipelines are basically docker containers, hosted in the Bitbucket infrastructure, that you can run to build or deploy your code, attaching them to “events” happening in your repository. Usually a push on one branch. You can clone or fork this Bitbucket repository to follow along with this tutorial. First thing first, add a file named bitbucket-pipelines.yml to the root of your repository. The content of the file should be similar to this: # use the official Python 2.7.16 docker image image: python:2.7.16 pipelines: branches: # deploy only when pushing to the master branch master: - step: name: Deploy to prod # this is the name of the Bitbucket Deployment deployment: Production caches: # cache the Ansible installation - pip script: # install Ansible - pip install ansible==2.8.0 # go into the ansible directory # in this same repository - cd ansible # perform the actual deploy - ansible-playbook -i ./hosts deploy.yaml Let’s examine the configuration line by line: On line 2 you define the docker image … -
Celery Once
Within our audit logs, which happen at the database level, we also want to be able to show the user a representation of what the object "looks like": basically, what `str(instance)` is. This is shown as a supplementary thing, and often makes it easier to identify which object has been edited. Otherwise, the audit logs only show the things that have changed on the object in that event. However, because this happens in python/django, and not in the database, we need some mechanism for fetching audit logs that don't have this string representation, and then update those. This is the perfect use case for celery: we can have a periodic task that looks for audit logs missing this related object, and then creates them. There are a few things that can cause problems: * If we run the task infrequently, then there can at times be a large number of new audit logs, which can cause this task to take a long time to run: blocking other tasks, or perhaps even timing out. We should limit the number of objects that may be handled in a given task. * If we run the task infrequently, there can be a big … -
Django Tips: Recovering Gracefully From ORM Errors
Django views are the glue between your users and the underlying database. When a user visits an url Django can map that url with a view. And most of the times the view is also responsible for fetching some data from the database. Consider the following example. There is a model named Workshop which among the others has a slug field: """ models.py """ from django.db import models class Workshop(models.Model): """ ... some other field here """ slug = models.SlugField(max_length=20) """ ... some other field here """ def __str__(self): return f'{self.title}' You may want to use that slug for fetching the appropriate entity from the database when the user visits someurl/slugname/. That means you will have an url declaration like so: """ urls.py of your Django app """ from django.urls import path from .views import workshop_detail urlpatterns = [ path('someurl/<slug:slug>/', workshop_detail) ] At this point you're ready to create a Django view for displaying the model detail. I'll use a function view for keeping things easy. A naive implementation for the view could be the following: """ views.py of your Django app """ from django.shortcuts import render from .models import Workshop def workshop_detail(request, slug): workshop = Workshop.objects.get(slug=slug) context = { … -
The Simplest WSGI Middleware
My library apig-wsgi bridges between AWS API Gateway’s JSON format for HTTP requests and Python WSGI applications. Recently Théophile Chevalier opened an issue requesting the library add an extra WSGI environ variable. I closed it by pointing out that it’s not much code to add a custom WSGI middleware to do so (plus the exact key is a bit out of scope for the library). I guess most Python web developers don’t touch WSGI day to day, so I figured I’d write this short post to share the knowledge. A Quick Shot of WSGI The WSGI specification defines an application as a callable takes two positional parameters. These parameters are, with their conventional names: environ, a dict of variables that describe the HTTP request. start_response, a function to call to call to start generation of the HTTP response. The application is called for each request, should call start_response and then return an iterable representing the contents of the HTTP response body. There’s a lot of specification around this, but to implement the simplest middleware we don’t need to dive into that. We want to only change environ and pass everything on to the proxied application. If we have original_app pointing … -
Creating Evscaperoom, part 2
The Jester, your 'adversary'This is part two of my post-mortem dev-blog about Evscaperoom, the multiplayer, text-based 'escape room' I wrote in Python and Evennia. You can read the first part of the dev blog here. This was a game-jam entry I created in a month for the Mud Coder's guild's Game Jam. The theme was One Room. You can play the game for free in your browser or with a traditional MUD client. There are no spoilers in these blog posts. The first part dealt with the overall game-design aspects. This second, final part will go into details of the code and the systems I built to quickly create all the content. The code referenced here is released under the BSD license and is available on github.At the time of this post, players have played Evscaperoom for a little more than a week. At the end I'll share some observations and things I learned along the way.Ease of buildingOver the one-month game jam, I spent about four days making the game's 'engine' and toolset with Evennia. The rest of the time was spent using those tools to actually create game content (the story, puzzles etc).An important thing was that I … -
Postgres Generated Columns
A little while ago, I wrote about creating a nice way to have a [Django ComputedField](https://schinckel.net/2018/11/12/django-computedfield%28%29/). It is pretty neat, except it needs to do some black magic to sniff up the stack to work around a limitation in the way a Ref/Col works in Django. The way it works is that you define the expression in Python, and it evaluates it in the database, allowing you to query based on this, and have it automatically annotated on. What it _doesn't_ do, however, is actually store that value in the database. Indeed, if you are actually querying on this column, you'd probably want to have a functional index that uses the same expression, so that the database can do a reasonable job of improving query times on that column. New in Postgres 12 is a feature that really piqued my interest: [Generated Columns](https://www.postgresql.org/docs/12/ddl-generated-columns.html). These are basically what the ComputedField does, but at the database level. And, instead of it being an expression that is evaluated at query time, it is instead an expression that is evaluated at write time, and stored in an actual column (that could then have an index applied to it). Let's have a look at an … -
The Best Animation and Prototyping Tools for Designers in 2019
Prototyping is a great way to showcase your product before embarking on a long development process. It gives you the opportunity to make changes prior to development. It also saves you significant time and money. The post The Best Animation and Prototyping Tools for Designers in 2019 appeared first on Distillery. -
Creating Evscaperoom, part 1
Over the last month (April-May 2019) I have taken part in the Mud Coder's Guild Game Jam "Enter the (Multi-User) Dungeon". This year the theme for the jam was One Room.The result was Evscaperoom, an text-based multi-player "escape-room" written in Python using the Evennia MU* creation system. You can play it from that link in your browser or MU*-client of choice. If you are so inclined, you can also vote for it here in the jam (don't forget to check out the other entries while you're at it).This little series of (likely two) dev-blog entries will try to recount the planning and technical aspects of the Evscaperoom. This is also for myself - I'd better write stuff down now while it's still fresh in my mind!Inception When I first heard about the upcoming game-jam's theme of One Room, an 'escape room' was the first thing that came to mind, not the least because I just recently got to solve my my own first real-world escape-room as a gift on my birthday. If you are not familiar with escape-rooms, the premise is simple - you are locked into a room and have to figure out a way to get out of it by solving practical … -
New features planned for Python 4.0
With the release of Python 3.8 coming soon, the core development team has asked me to summarize our latest discussions on the new features planned for Python 4.0, codename "ouroboros: the snake will eat itself". This will be an exciting release and a significant milestone, many thanks to the hard work of over 100 contributors. After heated debate on the mailing list, the 79-character line limit prescribed by PEP8 will be updated. IDE users all over the world will now be able to take advantage of their 30" ultra-wide 4K monitors, as the recommended line length will be increased to 89.5 characters (this was a compromise with the 100-character lobby, the decision being to split the difference). All new libraries and standard lib modules must include the phrase "for humans" somewhere in their title, and have a splashy documentation page with lots of fonts typography and testimonials. Finally, a new string-type for the masses, Python 4.0 will feature "z-strings": C-style NULL terminated bytestrings. Just prefix your string with z'my string' and Python will automatically ensure it is NULL-terminated. Note: the new z-strings cannot be used with any of the existing APIs that take string arguments - they must first be … -
New features planned for Python 4.0
With the release of Python 3.8 coming soon, the core development team has asked me to summarize our latest discussions on the new features planned for Python 4.0, codename "ouroboros: the snake will eat itself". This will be an exciting release and a significant milestone, many thanks to the hard work of over 100 contributors. After heated debate on the mailing list, the 79-character line limit prescribed by PEP8 will be updated. IDE users all over the world will now be able to take advantage of their 30" ultra-wide 4K monitors, as the recommended line length will be increased to 89.5 characters (this was a compromise with the 100-character lobby, the decision being to split the difference). All new libraries and standard lib modules must include the phrase "for humans" somewhere in their title. Finally, a new string-type for the masses, Python 4.0 will feature "z-strings": C-style NULL terminated bytestrings. Just prefix your string like so, z'my string' and Python will automatically ensure it is NULL-terminated. Note: the new z-strings cannot be used with any of the existing APIs that take string arguments - they must first be decoded to unicode strings or cast to bytes. Type-hinting has been extended … -
<p>django-debug-toolbar 2.0a1 is now <a target="_blank" rel="nofollow" href="https://pypi.org/project/django-debug-toolbar/2.0a1/">available on PyPI</a>. Please help with testing or with success stori
django-debug-toolbar 2.0a1 is now available on PyPI. Please help with testing or with success stories or with bug reporting & squashing. -
The Price of the Hallway Track
There are many good reasons to not go to every talk possible when attending conferences. However increasingly it became hip to boast with not going to talks at all – encouraging others to follow suit. As a speaker, that rubs me the wrong way and I’ll try to explain why. -
5 Ways Software Outsourcing Helps FinTech Startups Compete and Win
As many FinTech startups have already learned, the right development partner is a massive competitive advantage. Don’t get left behind. Instead, let outsourcing help you accelerate. The post 5 Ways Software Outsourcing Helps FinTech Startups Compete and Win appeared first on Distillery. -
Django + Tailwind CSS = ❤️
If you use Tailwind CSS and want to integrate it easily with Django, check out a package I created about a month ago. Surprisingly it's called Django-Tailwind. You can find it here on GitHub: https://github.com/timonweb/django-tailwind. If you don't know what Tailwind CSS is, then ... Read now -
Query Zen is no queries at all
Performing no queries is always going to be faster than performing a query. Today I had two instances of the same problem: I have two tables, one of which essentially stores calculated data based on other data (and data in other tables, or involving a process that uses application code, and cannot be purely determined within the database). In one case, we have an audit logging table (which is purely handled within postgres) and another related table that stores a string representation of what the audited object looked like according to the application at that point in time, which needs to be calculated after the fact in Django. The other case stores some cached values that _can_ be calculated in the database: basically some metadata about a shift according to the location that the shift is at. Changes to the shift table will cause this value to automatically be updated, however we have several million shifts that do not currently have this value, but we need to create items for all shifts that currently don't have the annotation. In both cases, we have a celery task that will create a (relatively small, to prevent locks and other performance issues) number … -
Django Search Tutorial
Add basic search functionality to any Django website. -
Django Pony GIF's
I discovered GIFCities recently. It’s a project by the Internet Archive for browsing GIF’s extracted from their Geocities archive. I spent a bit of time searching it for pegasus gifs, to represent the unofficial Django mascot, the Django Pony. Here are the best I found! Running django-admin.py startproject Mentoring a New Django Developer Code Review Code Review Complete Your Pet Project When You Have One View Deploy Goes Perfectly Deploy Goes Bad and Roll Back Maximum Requests Per Second Serving a Slow Admin Page How You Describe Your Project How Your Project Actually Works Fin Enjoy the GIFt, —Adam -
PyGrunn: a day has only 24 ± 1 hours - Miroslav Šedivý
(One of my summaries of a talk at the 2019 PyGrunn conference). Time zones... If you do datatime.datetime.now() you'll get a date+time without timezone information. You can get different results on your laptop (set to local time) and a server (that might be set to UTC). You can use datetime.datetime.utcnow() that returns UTC time. But... without a timezone attached. Best is to request the time in a specific timezone. There are gotchas regarding time. Check your time only once in a calculation. If you call .utcnow() multiple times, you can get different dates when your code runs around 0:00. Same with time.time(): if the "ntp" daemon adjusts your system clock in the mean time you get weird results. For that, there is time.monotonic(). The original source for all time zone information is the time zone database (tzdata). You can download it and look at all the files per timezone. Interesting reading! Look at Istanbul's timezone. Daylight savings time being delayed by a day in a specific year because of a nationwide school exam. It was anounced a few weeks before. That's all in the time zone database. So if you make a Docker now and still use it in two … -
PyGrunn: testing your infrastructure code - Ruben Homs
(One of my summaries of a talk at the 2019 PyGrunn conference). Servers used to be managed by proper wizards. But even wizards can be killed by a balrog. So... what happens when your sysadmin leaves? The point of failure is the sysadmin. Knowledge about infrastructure is centralised. It is non-reproducible. A solution is configuration management. Chef, ansible, saltstack, puppet. Configuration that's in source control instead of information in a sysadmin's head. It is a reproducible way to build your infrastructure. Source code, so everyone can see how a system works. You can even version your infrastructure. He'll use saltstack as an example, that's what they're using in his company. It is a master/minion system. So a central master pushes out commands to the minion systems. For testing, he uses a tool called "kitchen", originally intended for puppet, which can however also be used with saltstack: https://kitchen.saltstack.com/ . He showed a demo where he created a couple of virtualbox machines and automatically ran the salt scripts on them. You can then ssh to those boxes and check if they're OK. But... that's manual work. So he started using testinfra and pytest. Testinfra helps you test infrastructure. There are build-in tests … -
PyGrunn: python as a scientist's playground - Peter Kroon
(One of my summaries of a talk at the 2019 PyGrunn conference). He's a scientist. Quite often, he searches for python packages. If you're writing python packages, you can learn how someone might search for your package. If you don't write python packages, you can learn how to investigate. Scientists try to solve unsolved problems. When doing it with computers, you basically do three things. Perform simulations. Set up simulations. Analyze results. Newton said something about "standing on the shoulders of giants". So basically he predicted the python package index! So much libraries to build upon! A problem is that there is so much software. There are multiple libraries that can handle graphs (directed graphs, not diagrams). He's going to use that as an example. Rule one: PR is important. If you don't know a package exists, it won't come on the list. Google, github discovery, stackoverflow, scientific liberature, friends, pygrunn talks, etc. A README is critical. Without a good readme: forget it. The five he found: graph-tool, networkx, igraph, python-graph scipy.sparse.csgraph. Rule two: documentation is very important. Docs should showcase the capabilities. This goes beyond explaining it, it should show it. I must be able to learn how to …