Django community: RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Adding charts to Django admin
## Introduction Django offers a functional admin UI out of the box with CRUD interface for db management. This covers most use cases for basic content and user management systems. However, it doesn't have exploratory views that show summaries or historical trends which is something you'd expect from an admin dashboard. Luckily, the django admin app is extensible and with a few tweaks we can add interactive Javascript charts to the admin. ## Problem I wanted to get a graph overview of the email subscribers over time on [findwork.dev](https://findwork.dev). Is the site growing or stagnating in terms of email subscribers? How many subscribers did we have last month? What week did we gain most subscribers? Are all subscribers verifying their emails? Using exploratory charts we can get a historical overview to how our site is performing. I initially explored the land of ready-made Django admin apps and dashboards. The requirements were that it included charting abilities, was well documented and looked good. While all of the apps I experimented with looked better that the default admin in terms of styling, they were either lacking in documentation or were unmaintained. - [xadmin](https://github.com/sshwsfc/xadmin) - no documentation in English - [django-jet](https://github.com/geex-arts/django-jet) - unmaintained … -
This Is How We Promote a Strong Culture With Distributed Teams
It’s no secret that building and maintaining a robust company culture is challenging for any organization. Is it any more difficult for distributed agile teams, in which an engineering organization is spread across multiple offices — or even across different countries and time zones? I’m Distillery’s General Manager. I’ve been with the company since 2012. As a fast-growing company with 150+ people distributed across 5 offices, 2 countries, and time zones 10 hours apart, I’ve got a strong perspective on this question.  What’s my answer? It’s not more difficult. It’s just different in certain ways. The bottom line is that there’s no “easy” way to build a strong company culture, whether your team is in one office or one hundred. Since it’s a consequence of how a company actually functions, it can’t be manufactured. It’s something you need to pay deliberate, daily attention to.  You can’t manufacture culture, but you can bring it to the surface. You can work to define it, demonstrate it, and protect it. By making our workplace culture a central focus for our people, we create a conversation and working environment that strengthens that culture. Here’s how we do it at Distillery.  Defining Our Company’s Culture  … -
How to Add Database Modifications Beyond Migrations to Your Django Project
On several Django projects I’ve worked on, there has been a requirement for performing database modifications beyond Django migrations. For example: Managing stored procedures Managing check constraints, which weren’t supported before Django 2.2 Importing static data from a file Recording migration operations in a log Let’s look at three approaches to extending Django to do this as neatly as possible. 1. Use Django migrations Often I find developers have only been taught how to use Django migrations for model operations. They might know some SQL, but since they haven’t used it within Django, they assume migrations can’t use SQL directly. They can! Many of the uses of custom migration-style SQL code that I’ve seen could be better implemented within migrations. Let’s take a look at an example we’ll use for the rest of the article. Imagine you’re running a Django version before 2.2 that doesn’t support database check constraints. You might add one to a database table with this SQL: ALTER TABLE myapp_book ADD CONSTRAINT percent_lovers_haters_sum CHECK ( (percent_lovers + percent_haters) = 100 ); This constraint will make the database raise an error for any rows added or updated in the myapp_book table that have a percentage of lovers and … -
API Calls within a Pandas Dataframe API using Pandas Apply
Whenever you're doing data ana... -
Feature-Policy updates - now required for an A+ on SecurityHeaders.com
In my blog post and DjangoCon Europe talk earlier this year How to Score A+ for Security Headers on Your Django Website, I covered that Feature-Policy was a “bonus header”. In a recent update, Scott Helme wrote that an A+ on SecurityHeaders.com now requires Feature-Policy. Also it no longer requires X-Xss-Protection (though it’s still a good idea). Chrome also has some Feature-Policy support enabled by default, so most users will be protected when it’s set. Previously it was hidden behind the “experimental web features” flag, but this is now only used for enabling Feature-Policy support for certain features. Opening the JavaScript console and querying for the list of features on Chrome 75 without the experimental flag on, I see 18 features allowed: > document.featurePolicy.allowedFeatures() < [ "accelerometer", "ambient-light-sensor", "autoplay", "camera", "encrypted-media", "focus-without-user-activation", "fullscreen", "geolocation", "gyroscope", "magnetometer", "microphone", "midi", "payment", "picture-in-picture", "speaker", "sync-xhr", "usb", "vr", ] I added some updates to my how-to on these changes. If you haven’t set the header previously, consider adding it for that sweet A+ score! You can add it on your Django apps with my django-feature-policy package. I updated django-feature-policy this morning to version 3.0.0 to include the latest set of features. Since I don’t … -
10 Important Cloud Migration Case Studies You Need to Know
For most businesses considering cloud migration, the move is filled with promise and potential. Scalability, flexibility, reliability, cost-effectiveness, improved performance and disaster recovery, and simpler, faster deployment — what’s not to like?  It’s important to understand that cloud platform benefits come alongside considerable challenges, including the need to improve availability and latency, auto-scale orchestration, manage tricky connections, scale the development process effectively, and address cloud security challenges. While advancements in virtualization and containerization (e.g., Docker, Kubernetes) are helping many businesses solve these challenges, cloud migration is no simple matter.  That’s why, when considering your organization’s cloud migration strategy, it’s beneficial to look at case studies and examples from other companies’ cloud migration experiences. Why did they do it? How did they go about it? What happened? What benefits did they see, and what are the advantages and disadvantages of cloud computing for these businesses? Most importantly, what lessons did they learn — and what can you learn from them?  With that in mind, Distillery has put together 10 cloud migration case studies your business can learn from. While most of the case studies feature companies moving from on-premise, bare metal data centers to cloud, we also look at companies moving … -
Subquery and Subclasses
Being able to use correlated subqueries in the Django ORM arrived in 1.11, and I also backported it to 1.8. Quite commonly, I am asked questions about how to use these, so here is an attempt to document them further. There are three classes that are supplied with Django, but it's easy to write extensions using subclassing. Let's first look at an example of how you might want to use the included classes. We'll consider a set of temperature sensors, each with a name and a code, both of which are unique. These sensors will log their current temperature at some sort of interval: maybe it's regular, maybe it varies between devices. We want to keep every reading, but want to only allow one reading for a given sensor+timestamp. {% highlight python %} class Sensor(models.Model): location = models.TextField(unique=True) code = models.TextField(unique=True) class Reading(models.Model): sensor = models.ForeignKey(Sensor, related_name='readings') timestamp = models.DateTimeField() temperature = models.DecimalField(max_digits=6, decimal_places=3) class Meta: unique_together = (('sensor', 'timestamp'),) {% endhighlight %} Some of the things we might want to do for a given sensor: * Get the most recent temperature * Get the average temperature over a given period * Get the maximum temperature over a given period … -
Django SASS/SCSS Tutorial Series
I will teach you how to use SASS/SCSS in your Django projects. -
How to use SCSS/SASS in your Django project (NPM Way)
In this blog post, I will talk about how to use SCSS/SASS in your Django project with NPM. -
Run Django tests in the main database
Recently I started working on a Django project, loaded the prod database into it, and ran the tests. Then the data was gone. I know some people intentionally want to run tests on the "main database", and there is a very simple way to achieve this. Simply inherit from a test class that doesn't wrap your db interactions in transactions or puts them on a different test database. Then the database in the settings will be used. Raw import unittest class YourDjangoTest(unittest.TestCase): pass -
Run Django tests in the main database
Recently I started working on a Django project, loaded the prod database into it, and ran the tests. Then the data was gone. I know some people intentionally want to run tests on the "main database", and there is a very simple way to achieve this. Simply inherit from a test class that doesn't wrap your db interactions in transactions or puts them on a different test database. Then the database in the settings will be used. Raw import unittest class YourDjangoTest(unittest.TestCase): pass -
The Right Agile Foundations Will Boost Your Success
Agile project management has strong foundations in open source software development and mobile application development for iOS & Android. In the course of the last decade, Agile project management techniques have grown in popularity. Agile methodologies first arose in the Web 2.0 era to speed up innovation in software development and get to market at lower costs, allowing startups and SMEs to better compete with enterprise corporations. In today’s Cloud era, Agile has become an industry standard and companies integrate it with DevOps techniques. Businesses of any size can use Agile techniques to implement their unique goals and requirements with cost-efficiency and innovation. To leverage Agile project management methodologies successfully, businesses must lay the proper foundations. In order to lay these foundations, it is important to understand the history and evolution of Agile project management. A Brief History of How Agile Project Management Evolved How Agile Evolved: The DotCom Era “The traditional Project Management (waterfall) approach is linear where all the phases of a process occur in sequence. Its concept depends on predictable tools and predictable experience. Each and every project follows the same life cycle which includes the stages such as feasibility, plan, design, build, test, production, (&) support…. … -
Web maps: the Mer et Demeures project
Since ancient times, maps have always allowed to seek and find information in “spatial mode”. Today it is very common for them to be inserted into websites, as we did in the case of Mer et Demeures. -
How Serverless Is Changing Security: The Good, Bad, Ugly, and How to Fix It
Serverless fundamentally changes security. Sometimes for the better. Sometimes for the worse. Sometimes a little in the middle. To keep data and applications secure in our increasingly serverless world, startups and enterprises of all sizes need to understand what’s different, why it matters, and what they should do to protect themselves. In his talk at O’Reilly’s 2017 Velocity Conference, “Serverless Security: What’s Left to Protect?”, Guy Podjarny, co-founder and CEO of open source security platform Snyk, effectively boiled down the basics. In this article, we’ve summarized his 40-minute presentation and bookended it with a bit of helpful context. (Podjarny’s full talk can be found here.) Podjarny focuses on the ways in which serverless is better, neutral, and worse for security. Where possible, he offers up tips and tricks. We’ve added background on what serverless is, why businesses are adopting it so quickly, and what you should do next. What Is Serverless? Serverless architectures are cloud-based and service-based. This means that, instead of provisioning and managing their own servers, organizations buy cloud computing services such as the following: Software as a Service, or SaaS, delivers centrally hosted and managed software on demand to end users over the internet (e.g., Gmail, Slack, … -
Django's Test Case Classes and a Three Times Speed-Up
This is a story about how I sped up a client’s Django test suite to be three times faster, through swapping the test case class in use. Speeding up test runs is rarely a bad thing. Even small teams can repeat their test run hundreds of times per week, so time saved there is time won back. This keeps developers fast, productive, and happy. Django’s Test Case Classes A quick refresher on how the Django’s three basic test case classes affect the database: SimpleTestCase is the simplest one. It provides the basic features of unittest.TestCase plus some Django extras. It blocks database access by default, because it doesn’t do anything to isolate changes you would make there. You should use it for testing components that don’t need the database. TransactionTestCase extends SimpleTestCase to allow database modifications. It resets the database at the end by removing all rows from all database tables. This is slow, but reliable. You should use it when you need the database, but you can’t use TestCase… TestCase is the class you should normally use. It extends TransactionTestCase and replaces its database reset process with one that uses transactions. This is much faster as the database only … -
SongSearch autocomplete rate now 2+ per second
By analyzing my Nginx logs, I've concluded that SongSearch's autocomplete JSON API now gets about 2.2 requests per second. I.e. these are XHR requests to /api/search/autocomplete?q=.... Roughly, 1.8 requests per second goes back to the Django/Elasticsearch backend. That's a hit ratio of 16%. These Django/Elasticsearch requests take roughly 200ms on average. I suspect about 150-180ms of that time is spent querying Elasticsearch, the rest being Python request/response and JSON "paperwork". Caching strategy Caching is hard because the queries are so vastly different over time. Had I put a Redis cache decorator on the autocomplete Django view function I'd quickly bloat Redis memory and cause lots of evictions. What I used to do was something like this: def search_autocomplete(request): q = request.GET.get('q') cache_key = None if len(q) < 10: cache_key = 'autocomplete:' + q results = cache.get(cache_key) if results is not None: return http.JsonResponse(results) results = _do_elastisearch_query(q) if cache_key: cache.set(cache_key, results, 60 * 60) return http.JsonResponse(results) However, after some simple benchmarking it was clear that using Nginx' uwsgi_cache it was much faster to let the cacheable queries terminate already at Nginx. So I changed the code to something like this: def search_autocomplete(request): q = request.GET.get('q') results = _do_elastisearch_query(q) response = http.JsonResponse(results) … -
SongSearch autocomplete rate now 2+ per second
By analyzing my Nginx logs, I've concluded that SongSearch's autocomplete JSON API now gets about 2.2 requests per second. I.e. these are XHR requests to /api/search/autocomplete?q=.... Roughly, 1.8 requests per second goes back to the Django/Elasticsearch backend. That's a hit ratio of 16%. These Django/Elasticsearch requests take roughly 200ms on average. I suspect about 150-180ms of that time is spent querying Elasticsearch, the rest being Python request/response and JSON "paperwork". Caching strategy Caching is hard because the queries are so vastly different over time. Had I put a Redis cache decorator on the autocomplete Django view function I'd quickly bloat Redis memory and cause lots of evictions. What I used to do was something like this: def search_autocomplete(request): q = request.GET.get('q') cache_key = None if len(q) < 10: cache_key = 'autocomplete:' + q results = cache.get(cache_key) if results is not None: return http.JsonResponse(results) results = _do_elastisearch_query(q) if cache_key: cache.set(cache_key, results, 60 * 60) return http.JsonResponse(results) However, after some simple benchmarking it was clear that using Nginx' uwsgi_cache it was much faster to let the cacheable queries terminate already at Nginx. So I changed the code to something like this: def search_autocomplete(request): q = request.GET.get('q') results = _do_elastisearch_query(q) response = http.JsonResponse(results) … -
Single-file Python/Django Deployments
This post covers portions of my talk, Containerless Django, from DjangoCon US 2018. Deploying Python has improved significantly since I started working with it over a decade ago. We have virtualenv, pip, wheels, package hash verification, and lock files. Despite all the improvements, it still feels harder than it needs to be. Installing a typical large project has many steps, each one easy to trip up on: Install Python Install build tools (pip/virtualenv, pipenv, poetry, etc.) Install build dependencies (C compiler, development libraries, etc.) Download the code Run the build tools If you're using Node to build client-side files for a website, repeat steps 1-5 for that It's normal for ops teams to repeat this process as part of the automated testing and then again on every server the project is deployed to. It's no wonder Docker has become so popular because of the ease in which you can build-once and deploy-everywhere. But Docker is a heavy-handed solution and doesn't fit for every project. I envy the simplicity of languages like Go where you can compile your project down to a single binary that runs without any external dependencies. Even Java's JAR file format which requires Java to be preinstalled, … -
Single-file Python/Django Deployments
This post covers portions of my talk, Containerless Django, from DjangoCon US 2018. Deploying Python has improved significantly since I started working with it over a decade ago. We have virtualenv, pip, wheels, package hash verification, and lock files. Despite all the improvements, it still feels harder than it needs to be. Installing a typical large project has many steps, each one easy to trip up on: Install Python Install build tools (pip/virtualenv, pipenv, poetry, etc.) Install build dependencies (C compiler, development libraries, etc.) Download the code Run the build tools If you're using Node to build client-side files for a website, repeat steps 1-5 for that It's normal for ops teams to repeat this process as part of the automated testing and then again on every server the project is deployed to. It's no wonder Docker has become so popular because of the ease in which you can build-once and deploy-everywhere. But Docker is a heavy-handed solution and doesn't fit for every project. I envy the simplicity of languages like Go where you can compile your project down to a single binary that runs without any external dependencies. Even Java's JAR file format which requires Java to be preinstalled, … -
WWDC 2019: How to Build the Best Product Strategy for iOS 13, watchOS 6, and iPadOS
While Apple’s annual Worldwide Developers Conference (WWDC) is always exciting and informative, the WWDC 2019 conference was particularly jam-packed with new updates, capabilities, and ideas worth knowing about. Of course, most product owners don’t have the time to sift through four days’ worth of video presentations, or the hundreds of reaction articles that come out in the weeks that follow. That’s exactly why Distillery has pulled together summaries of many of the most important updates from WWDC 2019. We’ve watched it and read it so you don’t have to. With that in mind, what do product owners really need to know about iOS 13, iPadOS, watchOS, and the other new capabilities unveiled at the Apple WWDC 2019? And what impacts and opportunities do all these changes mean for product strategy and development? Finally, what does your business need to do to be ready for iOS 13’s release this fall? Read on. Privacy Sign in with Apple What is it? This is Apple’s new single sign-on (SSO service) protected with biometric two-factor authentication, a feature that largely hasn’t been adopted by other SSO services. It offers users with Apple IDs two options: either log in using their actual email addresses, or … -
Graphs in Django and Postgres
I have written a [bunch of posts about dealing with trees](/tags/tree/) in [Postgres](https://www.postgresql.org/) and [Django](https://docs.djangoproject.com/), and Funkybob used some of this to start the package [django-closure-view](https://github.com/funkybob/django-closure-tree). Today, someone was looking for similar functionality, but for a graph. Specifically, a Directed Acyclic Graph. Now, not every graph, or even every DAG is a tree, but every tree is a DAG. So, the difference between a tree and a graph in this context is that a given node may have an arbitrary number of parents. But, and this is worth noting now, none of it's parents may also be dependencies. The first part of this tells us that we can no longer just use a simple self-relation in our model to store the relationship: because there could be multiple parents. Instead, we will need to have a many-to-many relation to store that. {% highlight python %} from django.db import models class Node(models.Model): node_id = models.AutoField(primary_key=True) name = models.TextField(unique=True) parents = models.ManyToManyField( 'self', related_name='children', symmetrical=False, ) {% endhighlight %} We can put some meaningful data into this graph to make it a little more obvious if our queries are sane: {% highlight python %} django, pytz, sqlparse, asgiref = Node.objects.bulk_create([ Node(name='django'), Node(name='pytz'), Node(name='sqlparse'), … -
Handling overlapping values
One of the things that I enjoy most about [Postgres](https://www.postgresql.org) are the rich types. Using these types can help reduce the amount of validation that the application needs to do. Take for instance anything which contains a start date and a finish date. If you model this using two fields, then you also need to include validation about `start <= finish` (or perhaps `start < finish`, depending upon your requirements). If you use a [date range](https://www.postgresql.org/docs/current/rangetypes.html) instead, then the database will do this validation for you. It is not possible to create a range value that is "backwards". Sure, you'll also need to do application-level (and probably client-side) validation, but there is something nice about having a reliable database that ensures you cannot possibly have invalid data. [Django](https://docs.djangoproject.com/) is able to make [good use of range types](https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#range-fields), and most of my new code seemingly has at least one range type: often a `valid_period`. So much so that I have a `Mixin` and a `QuerySet` that make dealing with these easier: {% highlight python %} class ValidPeriodMixin(models.Model): valid_period = DateRangeField() class Meta: abstract = True @property def start(self): if self.valid_period.lower_inc: return self.valid_period.lower elif self.valid_period.lower is not None: return self.valid_period.lower + datetime.timedelta(1) … -
Tuples versus Lists in Python
One thing I often ask for in code review is conversion of tuples to lists. For example, imagine we had this Django admin class: class BookAdmin(ModelAdmin): fieldsets = ( ( None, { "fields": ( "id", "name", "created_time", ) }, ), ) readonly_fields = ("created_time",) I’d prefer the tuples to all be lists, like: class BookAdmin(ModelAdmin): fieldsets = [ [ None, { "fields": [ "id", "name", "created_time", ] }, ], ] readonly_fields = ["created_time"] This is counter to the examples in the Django admin docs. So, why? Tuples use parentheses, and parentheses have several uses in Python. Therefore it’s easier to make typo mistakes with them. First, it’s easy to miss a single trailing comma and accidentally create a parenthesized string instead of a tuple. For example, if we missed it in the above code we would have: readonly_fields = ("created_time") This is the string "created_time", instead of a tuple containing that string. Woops. This often works with code you pass the variable to because strings are iterable character-by-character. (Django guards against this in the particular case of ModelAdmin.readonly_fields with its check admin.E034. But in general Python code doesn’t check for strings versus tuples/lists before iterating). There’s an argument Python shouldn’t … -
Fallback values in Django
It's not uncommon to have some type of cascading of values in a system. For instance, in our software, we allow a `Brand` to have some default settings, and then a `Location` may override some or all of these settings, or just fallback to the brand settings. I'm going to have a look at how this type of thing can be implemented using Django, and a way that this can be handled seamlessly. We'll start with our models: {% highlight python %} class Brand(models.Model): brand_id = models.AutoField(primary_key=True) name = models.TextField() class Location(models.Model): location_id = models.AutoField(primary_key=True) brand_id = models.ForeignKey(Brand, related_name='locations') name = models.TextField() WEEKDAYS = [ (1, _('Monday')), (2, _('Tuesday')), (3, _('Wednesday')), (4, _('Thursday')), (5, _('Friday')), (6, _('Saturday')), (7, _('Sunday')), ] class BrandSettings(models.Model): brand = models.OneToOneField(Brand, primary_key=True, related_name='settings') opening_time = models.TimeField() closing_time = models.TimeField() start_day = models.IntegerField(choices=WEEKDAYS) class LocationSettings(models.Model): location = models.OneToOneField(Location, primary_key=True, related_name='_raw_settings') opening_time = models.TimeField(null=True, blank=True) closing_time = models.TimeField(null=True, blank=True) start_day = models.IntegerField(choices=WEEKDAYS, null=True, blank=True) {% endhighlight %} We can't use an abstract base model here, because the `LocationSettings` values are _all_ optional, but the `BrandSettings` are not. We might have a look later at a way we can have a base model and inherit-and-change-null on the fields. In … -
Evennia 0.9 released
Last week we released Evennia 0.9, the next version of the open source Python MU* creation system.This release is the result of about 10 months of development, featuring 771 commits, 70 closed pull requests from the community and something like 80 issues and feature/requests closed. Thanks everyone!The main feature of Evennia 0.9 is that we have finally made the move to Python3. And we burn the bridges behind us; as announced in previous posts we completely drop Python2 support and move exclusively to only support the latest Python3.7.Overall the move to Python3 was not too bloody (and much work towards a never published py2+3 version was already done by Evennia contributors in a separate branch earlier). The main issues I ran into were mainly in the changes in how Python3 separates strings from bytes. This became crticial since Evennia implements several connection protocols; there were a lot of edge cases and weird errors appearing where data went to and from the wire.A regular user has it a lot easier though. So far people have not had too much trouble converting their games from 2.7 to 3.7. The biggest Linux distros don't all have Py3.7 out of the box though, so …
