Django community: Community blog posts RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
How Serverless Is Changing Security: The Good, Bad, Ugly, and How to Fix It
Serverless fundamentally changes security. Sometimes for the better. Sometimes for the worse. Sometimes a little in the middle. To keep data and applications secure in our increasingly serverless world, startups and enterprises of all sizes need to understand what’s different, why it matters, and what they should do to protect themselves. In his talk at O’Reilly’s 2017 Velocity Conference, “Serverless Security: What’s Left to Protect?”, Guy Podjarny, co-founder and CEO of open source security platform Snyk, effectively boiled down the basics. In this article, we’ve summarized his 40-minute presentation and bookended it with a bit of helpful context. (Podjarny’s full talk can be found here.) Podjarny focuses on the ways in which serverless is better, neutral, and worse for security. Where possible, he offers up tips and tricks. We’ve added background on what serverless is, why businesses are adopting it so quickly, and what you should do next. What Is Serverless? Serverless architectures are cloud-based and service-based. This means that, instead of provisioning and managing their own servers, organizations buy cloud computing services such as the following: Software as a Service, or SaaS, delivers centrally hosted and managed software on demand to end users over the internet (e.g., Gmail, Slack, … -
Django's Test Case Classes and a Three Times Speed-Up
This is a story about how I sped up a client’s Django test suite to be three times faster, through swapping the test case class in use. Speeding up test runs is rarely a bad thing. Even small teams can repeat their test run hundreds of times per week, so time saved there is time won back. This keeps developers fast, productive, and happy. Django’s Test Case Classes A quick refresher on how the Django’s three basic test case classes affect the database: SimpleTestCase is the simplest one. It provides the basic features of unittest.TestCase plus some Django extras. It blocks database access by default, because it doesn’t do anything to isolate changes you would make there. You should use it for testing components that don’t need the database. TransactionTestCase extends SimpleTestCase to allow database modifications. It resets the database at the end by removing all rows from all database tables. This is slow, but reliable. You should use it when you need the database, but you can’t use TestCase… TestCase is the class you should normally use. It extends TransactionTestCase and replaces its database reset process with one that uses transactions. This is much faster as the database only … -
SongSearch autocomplete rate now 2+ per second
By analyzing my Nginx logs, I've concluded that SongSearch's autocomplete JSON API now gets about 2.2 requests per second. I.e. these are XHR requests to /api/search/autocomplete?q=.... Roughly, 1.8 requests per second goes back to the Django/Elasticsearch backend. That's a hit ratio of 16%. These Django/Elasticsearch requests take roughly 200ms on average. I suspect about 150-180ms of that time is spent querying Elasticsearch, the rest being Python request/response and JSON "paperwork". Caching strategy Caching is hard because the queries are so vastly different over time. Had I put a Redis cache decorator on the autocomplete Django view function I'd quickly bloat Redis memory and cause lots of evictions. What I used to do was something like this: def search_autocomplete(request): q = request.GET.get('q') cache_key = None if len(q) < 10: cache_key = 'autocomplete:' + q results = cache.get(cache_key) if results is not None: return http.JsonResponse(results) results = _do_elastisearch_query(q) if cache_key: cache.set(cache_key, results, 60 * 60) return http.JsonResponse(results) However, after some simple benchmarking it was clear that using Nginx' uwsgi_cache it was much faster to let the cacheable queries terminate already at Nginx. So I changed the code to something like this: def search_autocomplete(request): q = request.GET.get('q') results = _do_elastisearch_query(q) response = http.JsonResponse(results) … -
Single-file Python/Django Deployments
This post covers portions of my talk, Containerless Django, from DjangoCon US 2018. Deploying Python has improved significantly since I started working with it over a decade ago. We have virtualenv, pip, wheels, package hash verification, and lock files. Despite all the improvements, it still feels harder than it needs to be. Installing a typical large project has many steps, each one easy to trip up on: Install Python Install build tools (pip/virtualenv, pipenv, poetry, etc.) Install build dependencies (C compiler, development libraries, etc.) Download the code Run the build tools If you're using Node to build client-side files for a website, repeat steps 1-5 for that It's normal for ops teams to repeat this process as part of the automated testing and then again on every server the project is deployed to. It's no wonder Docker has become so popular because of the ease in which you can build-once and deploy-everywhere. But Docker is a heavy-handed solution and doesn't fit for every project. I envy the simplicity of languages like Go where you can compile your project down to a single binary that runs without any external dependencies. Even Java's JAR file format which requires Java to be preinstalled, … -
WWDC 2019: How to Build the Best Product Strategy for iOS 13, watchOS 6, and iPadOS
While Apple’s annual Worldwide Developers Conference (WWDC) is always exciting and informative, the WWDC 2019 conference was particularly jam-packed with new updates, capabilities, and ideas worth knowing about. Of course, most product owners don’t have the time to sift through four days’ worth of video presentations, or the hundreds of reaction articles that come out in the weeks that follow. That’s exactly why Distillery has pulled together summaries of many of the most important updates from WWDC 2019. We’ve watched it and read it so you don’t have to. With that in mind, what do product owners really need to know about iOS 13, iPadOS, watchOS, and the other new capabilities unveiled at the Apple WWDC 2019? And what impacts and opportunities do all these changes mean for product strategy and development? Finally, what does your business need to do to be ready for iOS 13’s release this fall? Read on. Privacy Sign in with Apple What is it? This is Apple’s new single sign-on (SSO service) protected with biometric two-factor authentication, a feature that largely hasn’t been adopted by other SSO services. It offers users with Apple IDs two options: either log in using their actual email addresses, or … -
Tuples versus Lists in Python
One thing I often ask for in code review is conversion of tuples to lists. For example, imagine we had this Django admin class: class BookAdmin(ModelAdmin): fieldsets = ( ( None, { "fields": ( "id", "name", "created_time", ) }, ), ) readonly_fields = ("created_time",) I’d prefer the tuples to all be lists, like: class BookAdmin(ModelAdmin): fieldsets = [ [ None, { "fields": [ "id", "name", "created_time", ] }, ], ] readonly_fields = ["created_time"] This is counter to the examples in the Django admin docs. So, why? Tuples use parentheses, and parentheses have several uses in Python. Therefore it’s easier to make typo mistakes with them. First, it’s easy to miss a single trailing comma and accidentally create a parenthesized string instead of a tuple. For example, if we missed it in the above code we would have: readonly_fields = ("created_time") This is the string "created_time", instead of a tuple containing that string. Woops. This often works with code you pass the variable to because strings are iterable character-by-character. (Django guards against this in the particular case of ModelAdmin.readonly_fields with its check admin.E034. But in general Python code doesn’t check for strings versus tuples/lists before iterating). There’s an argument Python shouldn’t … -
Tuples versus Lists in Python
One thing I often ask for in code review is conversion of tuples to lists. For example, imagine we had this Django admin class: class BookAdmin(ModelAdmin): fieldsets = ( ( None, { "fields": ( "id", "name", "created_time", ) }, ), ) readonly_fields = ("created_time",) I’d prefer the tuples to all be lists, like: class BookAdmin(ModelAdmin): fieldsets = [ [ None, { "fields": [ "id", "name", "created_time", ] }, ], ] readonly_fields = ["created_time"] This is counter to the examples in the Django admin docs. So, why? Tuples use parentheses, and parentheses have several uses in Python. Therefore it’s easier to make typo mistakes with them. First, it’s easy to miss a single trailing comma and accidentally create a parenthesized string instead of a tuple. For example, if we missed it in the above code we would have: readonly_fields = ("created_time") This is the string "created_time", instead of a tuple containing that string. Woops. This often works with code you pass the variable to because strings are iterable character-by-character. (Django guards against this in the particular case of ModelAdmin.readonly_fields with its check admin.E034. But in general Python code doesn’t check for strings versus tuples/lists before iterating). There’s an argument Python shouldn’t … -
Evennia 0.9 released
Last week we released Evennia 0.9, the next version of the open source Python MU* creation system.This release is the result of about 10 months of development, featuring 771 commits, 70 closed pull requests from the community and something like 80 issues and feature/requests closed. Thanks everyone!The main feature of Evennia 0.9 is that we have finally made the move to Python3. And we burn the bridges behind us; as announced in previous posts we completely drop Python2 support and move exclusively to only support the latest Python3.7.Overall the move to Python3 was not too bloody (and much work towards a never published py2+3 version was already done by Evennia contributors in a separate branch earlier). The main issues I ran into were mainly in the changes in how Python3 separates strings from bytes. This became crticial since Evennia implements several connection protocols; there were a lot of edge cases and weird errors appearing where data went to and from the wire.A regular user has it a lot easier though. So far people have not had too much trouble converting their games from 2.7 to 3.7. The biggest Linux distros don't all have Py3.7 out of the box though, so … -
What Is DevOps? Here Are the Core Concepts You Actually Need to Know
Over the past few years, DevOps has rapidly gained in popularity. We hear news near-constantly about how organizations worldwide are being transformed with the help of DevOps, and how engineers are mastering the new profession of DevOps. But what actually is the definition of DevOps? Is it a methodology, a profession, or both at once? What’s the difference between CI and CD, or between IaaS, PaaS, and SaaS? How do DevOps and SRE relate to each other? And what other terms and concepts do you need to know to understand the language and meaning of DevOps? DevOps Concepts You Need to Know — A DevOps Glossary To answer the question, “What is DevOps?”, Dmitry Stepanenko, Distillery’s Head of DevOps, has prepared a glossary of core concepts and definitions you need to know. Our DevOps glossary cuts through the clutter, defining and explaining only the terms you really need to know, including: A Definition of DevOps CI and CD: The Backbone of DevOps TDD and Microservices: The Development Side of DevOps Virtual Machines and Cloud Computing IaaS, PaaS, and SaaS Containers and Kubernetes Infrastructure as Code — the Confluence of “Dev” and “Ops” What Is the Definition of DevOps? DevOps — … -
Celery, Rabbits, and Warrens
Every time I pick up the Python job queue Celery after not using it for a while, I find I’ve forgotten exactly how RabbitMQ works. I find the Advanced Message Queuing Protocol (AMQP) concepts drop out of my head pretty quickly: Exchanges, Routing Keys, Bindings, Queues, Virtual Hosts… I’ve repeatedly returned to the blog post “Rabbits and Warrens” by Jason J. W. Williams to refresh my mind on the concepts. I saved it into my Evernote on December 2014, but unfortunately it has gone offline since. Luckily it’s saved in the Web Archive. If you use Celery with an AMQP backend, or RabbitMQ, it’s worth reading. My Favourite Bit Here’s the the extract that helps clarify things the most for me. First, the basic AMQP building blocks: There are four building blocks you really care about in AMQP: virtual hosts, exchanges, queues and bindings. A virtual host holds a bundle of exchanges, queues and bindings. Why would you want multiple virtual hosts? Easy. A username in RabbitMQ grants you access to a virtual host…in its entirety. So the only way to keep group A from accessing group B’s exchanges/queues/bindings/etc. is to create a virtual host for A and one for … -
Celery, Rabbits, and Warrens
Every time I pick up the Python job queue Celery after not using it for a while, I find I’ve forgotten exactly how RabbitMQ works. I find the Advanced Message Queuing Protocol (AMQP) concepts drop out of my head pretty quickly: Exchanges, Routing Keys, Bindings, Queues, Virtual Hosts… I’ve repeatedly returned to the blog post “Rabbits and Warrens” by Jason J. W. Williams to refresh my mind on the concepts. I saved it into my Evernote on December 2014, but unfortunately it has gone offline since. Luckily it’s saved in the Web Archive. If you use Celery with an AMQP backend, or RabbitMQ, it’s worth reading. My Favourite Bit Here’s the the extract that helps clarify things the most for me. First, the basic AMQP building blocks: There are four building blocks you really care about in AMQP: virtual hosts, exchanges, queues and bindings. A virtual host holds a bundle of exchanges, queues and bindings. Why would you want multiple virtual hosts? Easy. A username in RabbitMQ grants you access to a virtual host…in its entirety. So the only way to keep group A from accessing group B’s exchanges/queues/bindings/etc. is to create a virtual host for A and one for … -
Python 3 GUI: wxPython 4 Tutorial - Urllib & JSON Example
In this tutorial, we'll learn to build a Python 3 GUI app from scratch using wxPython and Urllib. We'll be consuming a third-party news REST API available from newsapi.org which provides breaking news headlines, and allows you to search for articles from over 30,000 news sources and blogs worldwide. We'll use Urllib for sending HTTP requests to the REST API and the json module to parse the response. Throughout you'll understand how to create desktop user interfaces in Python 3, including adding widgets, and managing data. In more details, you'll see: How to use Urllib to send HTTP requests to fetch JSON data from a third-party REST API. How to use the json module to parse JSON data into Python 3 dictionaries. How to use the webbrowser module to open URLs in your default web browser. First of all, head over to the registration page and create a new account then take note of the provided API key which will be using later to access the news data. What is wxPython wxPython is a Python wrapper around wxWidgets - the cross platform C++ library for building desktop apps for macOS, Linux and Windows. wxPython was created by Robin Dunn. Prerequisites … -
Simple bash deployment script for Django
#!/bin/bash remote=$1 cmd=$2 if [[ ! "${remote}" ]]; then echo No remote given, aborting, try username@host exit 1 fi if [[ ! "${cmd}" ]]; then echo No command given, aborting, try switch or nop exit 1 fi python=/usr/bin/python3.5 commit=$(git rev-parse HEAD) date=$(date +%Y%m%d_%H%M%S) name="${date}_${commit}" src="${name}/git" settings="${src}/src/conf/settings/" venv="${name}/virtualenv" archive="${name}.tar.gz" previous="previous" latest="latest" set -e echo "Transfer archive..." git archive --format tar.gz -o "${archive}" "${commit}" scp "${archive}" ${remote}: rm -f "${archive}" echo "Set up remote host..." ssh "${remote}" mkdir -p "${src}" ssh "${remote}" tar xzf "${archive}" -C "${src}" ssh "${remote}" virtualenv --quiet "${venv}" -p ${python} ssh "${remote}" "${venv}/bin/pip" install --quiet --upgrade pip setuptools ssh "${remote}" "${venv}/bin/pip" install --quiet -r "${src}/requirements.txt" echo "Set up django..." ssh "${remote}" "${venv}/bin/python ${src}/src/manage.py check" ssh "${remote}" "${venv}/bin/python ${src}/src/manage.py migrate --noinput" ssh "${remote}" "${venv}/bin/python ${src}/src/manage.py collectstatic --noinput" if [[ "$cmd" == "switch" ]]; then echo "Switching to new install..." ssh "${remote}" rm -f "${previous}" ssh "${remote}" mv "${latest}" "${previous}" ssh "${remote}" ln -s ${name} "${latest}" ssh "${remote}" 'kill -15 $(cat previous/git/src/gunicorn.pid)' echo "Deleting obsolete deploys" ssh "${remote}" /usr/bin/find . -maxdepth 1 -type d -name "2*" \ grep -v $(basename $(readlink latest)) | \ grep -v $(basename $(readlink previous )) \ | /usr/bin/xargs /bin/rm -rfv fi echo "Cleaning up..." ssh … -
Testing email in Django
Sending email from a web app often seems like throwing stones into a black hole. You create a message, pass it to a mail send function and hope for the best. You don't control an inbox and mail server, so this whole process happens somewhere in between, and you ... Read now -
Series A Funding: Why Outsourcing Should Be Part of Your Startup Growth Strategy
You’ve raised your seed funding round and are on the path to Series A — Your startup shows promise! Investors have taken note. That’s fantastic, and we’re excited for you. But we also want to make it clear: Nowadays, to be a successful startup, you need to get right back to thinking about how you can spend that seed funding as wisely as possible. And you need to start thinking ASAP about how you’ll ensure your startup is deemed worthy of any additional funding. Why the super-practical downer advice? While more startups than ever are getting seed money, fewer than ever are getting follow-on funding. So that Series A funding you’re hoping for? It’s not a given. Also, as accelerator Y Combinator always advises its startups, it’s not your money. Getting more money is dependent on showing you can be a smart spender with good margins and a sustainable business model. Let me explain. What’s the “Series A Crunch”? Analyzing Seed vs. Series A Funding The venture capital community coined the term “Series A Crunch” to describe a trend they saw in Series A funding. Essentially, they noted that, while huge numbers of startups were easily raising large sums in … -
Responses versus Requests-Mock
Last October, I swapped out using Responses for Requests-Mock in my Python projects. Both are libraries for achieving the same goal of mocking out HTTP requests made with the popular Requests library. I swapped on my private and open source projects. For example of the transition, see my PR on ec2-metadata. Responses is still vastly more popular than Requests-Mock. At time of writing, Responses has 2,376 GitHub stars to Requests-Mock’s 257 (92 + 165 from its original repository). I’m writing this post as a bit of promotion for such a useful library, and also to give an example of my reasoning behind choosing one open source library over another. This is not to disparage Responses - it works really well, I’ve enjoyed using it, and have even contributed to it several times. Project Health The more people use a project, the more likely it solves the problem well, its “sharp corners” have been “sanded off”, and bugs have been fixed. We also want to see this continuing over time to know that we’re likely to be supported going forwards. Here are the statistics on each project: responses https://github.com/getsentry/responses/commits/master 2070 stars, 57 contributors. With this much support, unlikely to have many … -
How to Set Up a Centralized Log Server with rsyslog
For many years, we've been running an ELK (Elasticsearch, Logstash, Kibana) stack for centralized logging. We have a specific project that requires on-premise infrastructure, so sending logs off-site to a hosted solution was not an option. Over time, however, the maintenance requirements of this self-maintained ELK stack were staggering. Filebeat, for example, filled up all the disks on all the servers in a matter of hours, not once, but twice (and for different reasons) when it could not reach its Logstash/Elasticsearch endpoint. Metricbeat suffered from a similar issue: It used far too much disk space relative to the value provided in its Elasticsearch indices. And while provisioning a self-hosted ELK stack has gotten easier over the years, it's still a lengthy process, which requires extra care anytime an upgrade is needed. Are these problems solvable? Yes. But for our needs, a simpler solution was needed. Enter rsyslog. rsyslog has been around since 2004. It's an alternative to syslog and syslog-ng. It's fast. And relative to an ELK stack, its RAM and CPU requirements are negligible. This idea started as a proof-of-concept, and quickly turned into a production-ready centralized logging service. Our goals are as follows: Set up a single VM … -
All Is Turned to Black
If the title worried you about my mental state, you can relax. It’s about the Python code formatter Black! Yesterday I finished converting the open source Python projects I maintain to use it. I’ve followed the project from a distance for a while, and figured now is a my time to try it out. It’s not out of beta yet, but it’s stable and gaining traction: My most recent client ev.energy decided to use it on their codebase. On Django we recently agreed on Django Enhancement Proposal (DEP) 8, which states Django will convert to Black when it leaves beta. It’s an unofficially official Python project, living on the Python GitHub Organization. The original author is Łukasz Langa, the Python 3.8 release manager, whom I saw speak last weekend at PyLondinium. Is the name some kind of “Black humo(u)r”? The name confuses some people. It’s a reference to Henry Ford, who quipped about the single colo(u)r option on his Model T: Any customer can have a car painted any color that he wants so long as it is black. Hence the project’s logo bears similarity to the Ford motors one: Black similarly tries to be “one style fits all” with … -
All Is Turned to Black
If the title worried you about my mental state, you can relax. It’s about the Python code formatter Black! Yesterday I finished converting the open source Python projects I maintain to use it. I’ve followed the project from a distance for a while, and figured now is a my time to try it out. It’s not out of beta yet, but it’s stable and gaining traction: My most recent client decided to use it on their codebase On Django we recently agreed on Django Enhancement Proposal (DEP) 8, which states Django will convert to Black when it leaves beta. It’s an unofficially official Python project, living on the Python GitHub Organization. The original author is Łukasz Langa, the Python 3.8 release manager, whom I saw speak last weekend at PyLondinium. Is the name some kind of “Black humo(u)r”? The name confuses some people. It’s a reference to Henry Ford, who quipped about the single colo(u)r option on his Model T: Any customer can have a car painted any color that he wants so long as it is black. Hence the project’s logo bears similarity to the Ford motors one: Black similarly tries to be “one style fits all” with the … -
5 Communication Mistakes That Are Hurting Your IT Outsourcing Relationship
When done right, IT outsourcing can provide businesses with a massive competitive advantage, powering innovation, sustainable scaling, cost savings, budget/staffing flexibility, and countless other benefits. It makes IT leaders’ lives easier, making them look good while driving value for the business. Again, however, that’s when it’s done right. So how can you avoid doing it wrong? Much of the success of any IT outsourcing relationship boils down to a single factor: effective communication. Communication mistakes are incredibly common in outsourcing. Below, we’ve broken down five of the most common mistakes, explaining the problems, their potential impact, and how you can avoid them. Communication Mistake #1: Micromanaging The Problem It’s your business and therefore your baby. It’s not surprising that it can be difficult to give up control over how things get done. So you become overly prescriptive in telling your IT outsourcing provider how they should do your work. After all, you don’t want to risk they’ll do things wrong. The Damage Reduced productivity, efficiency, and quality. Your IT outsourcing provider is increasingly hamstrung in how they can serve you. You’ve narrowed the definitions of what’s acceptable so much that it’s causing an entirely new set of problems. As they … -
Django Favicon Tutorial
How to add a favicon to any Django website. -
Types of Outsourcing: How to Choose the Best Solution for Your Business
To stay competitive, more businesses than ever are turning to outsourcing. These businesses know that leveraging one or more types of outsourcing helps them: Execute and innovate faster. From 2016 to 2018, Deloitte saw an increase from 20% to 49% in the number of organizations moving to outsourced providers as they innovate. Source top talent with the right expertise. Gartner’s 2017 and 2016 CIO Agenda Reports identified talent as the “single biggest issue standing in the way of CIOs achieving their objectives.” Make smarter use of their resources. As the Harvard Business Review states, “Full-time employees are the most expensive and least flexible source of labor.” That’s why smart businesses are making strategic use of outsourced, contract workers to fill gaps in expertise, meet fluctuating demands, and manage certain projects. Did you know that Microsoft and Google both have contract workforces that either equal or exceed their in-house headcounts? Done correctly, outsourcing can be your secret weapon to ongoing innovation, rapid releases, and sustainable business success. This guide will help you gain a stronger grasp on how different types of outsourcing models really work, so you can accurately assess which model is the right fit for your needs. We’ll start … -
Python in Azure Pipelines, Step by Step
Since the acquisition of Travis CI, the future of their free offering is unclear. Azure Pipelines has a generous free tier, but the examples I found are discouragingly complex and take advantage of features like templating that most projects don’t need. To close that gap, this article shows you how to move a Python project with simple CI needs from Travis CI to Azure Pipelines. -
Build an XML sitemap of XML sitemaps
Suppose that you have so many thousands of pages that you can't just create a single /sitemap.xml file that has all the URLs (aka <loc>) listed. Then you need to make a /sitemaps.xml that points to the other sitemap files. And if you're in the thousands, you'll need to gzip these files. The blog post demonstrates how Song Search generates a sitemap file that points to 63 sitemap-{M}-{N}.xml.gz files which spans about 1,000,000 URLs. The context here is Python and the getting of the data is from Django. Python is pretty key here but if you have something other than Django, you can squint and mentally replace that with your own data mapper. Generate the sitemap .xml.gz file(s) Here's the core of the work. A generator function that takes a Django QuerySet instance (that is ordered and filtered!) and then starts generating etree trees and dumps them to disk with gzip. import gzip from lxml import etree outfile = "sitemap-{start}-{end}.xml" batchsize = 40_000 def generate(self, qs, base_url, outfile, batchsize): # Use `.values` to make the query much faster qs = qs.values("name", "id", "artist_id", "language") def start(): return etree.Element( "urlset", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" ) def close(root, filename): with gzip.open(filename, "wb") as f: f.write(b'<?xml … -
Generate a random IP address in Python
A fun trick to generate random, but seeded, IPv4 addresses in a Django app.
