Django community: RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Pygrunn: django channels - Bram Noordzij/Bob Voorneveld
(One of my summaries of the one-day 2016 PyGrunn conference). Django channels is a project to make Django to handle more than "only" plain http requests. So: websockets, http2, etc. Regular http is the normal request/response cycle. Websockets is a connection that stays open, for bi-directional communication. Websockets are technically an ordered first-in first-out queue with message expiry and at-most-once delivery to only one listener at the time. "Django channels" is an easy-to-understand extension of the Django view mechanism. Easy to integrate and deploy. Installing django channels is quick. Just add the application to your INSTALLED_APPS list. That's it. The complexity happens when deploying it as it is not a regular WSGI deployment. It uses a new standard called ASGI (a = asynchronous). Currently there's a "worker service" called daphne (build in parallel to django channels) that implements ASGI. You need to configure a "backing service". Simplified: a queue. They showed a demo where everybody in the room could move markers over a map. Worked like a charm. How it works behind the scenes is that you define "channels". Channels can recieve messages and can send messages to other channels. So you can have channel for reading incoming messages, do … -
Pygrunn: Micropython, internet of pythonic things - Lars de Ridder
(One of my summaries of the one-day 2016 PyGrunn conference). micropython is a project that wants to bring python to the world of microprocessors. Micropython is a lean and fast implementation of python 3 for microprocessors. It was funded in 2013 on kickstarter. Originally it only ran on a special "pyboard", but it has now been ported to various other microprocessors. Why use micropython? Easy to learn, with powerful features. Native bitwise operations. Ideal for rapid prototyping. (You cannot use cpython, mainly due to RAM usage.) It is not a full python, of course, they had to strip things out. "functools" and "this" are out, for instance. Extra included are libraries for the specific boards. There are lots of memory optimizations. Nothing fancy, most of the tricks are directly from compiler textbooks, but it is nice to see it all implemented in a real project. Some of the supported boards: Pyboard The "BBC micro:bit" which is supplied to 1 million school children! Wipy. More of a professional-grade board. LoPy. a board which supports LoRa, an open network to connect internet-of-things chips. Development: there is one full time developer (funded by the ESA) and two core contributors. It is stable and … -
Caktus CTO Colin Copeland Invited to the White House Open Police Data Initiative
We at Caktus were incredibly proud when the White House Police Data Initiative invited CTO Colin Copeland to celebrate their first year accomplishments. While at the White House, Colin also joined private breakout sessions to share ideas with law enforcement officials, city staff, and other civic technologists from across the country. Colin is the co-founder of Code for Durham and served as lead developer for OpenDataPolicingNC.com. OpenDataPolicingNC.com, a site built for the Southern Coalition for Social Justice, displays North Carolina police stop data. -
Why using factories in Django
From the very beginning of a project, you need some data. You need data in your development database and you need data for your automated tests. The instinctive solution is to manually enter a set of data via the Django admin. The official way is to enter data via Django fixtures file(s). Using factories will make it easier and better; here is why. -
What We’re Clicking - April Link Roundup
It’s time for this month’s roundup of articles and posts shared by Cakti that drew the most attention on Twitter. The list highlights new work in civic tech and international development as well as reasons for the increasing popularity of Python and open source development. -
How to track Google Analytics pageviews on non-web requests (with Python)
tl;dr; Use raven's ThreadedRequestsHTTPTransport transport class to send Google Analytics pageview trackings asynchronously to Google Analytics to collect pageviews that aren't actually browser pages. We have an API on our Django site that was not designed from the ground up. We had a bunch of internal endpoints that were used by the website. So we simply exposed those as API endpoints that anybody can query. All we did was wrap certain parts carefully as to not expose private stuff and we wrote a simple web page where you can see a list of all the endpoints and what parameters are needed. Later we added auth-by-token. Now the problem we have is that we don't know which endpoints people use and, as equally important, which ones people don't use. If we had more stats we'd be able to confidently deprecate some (for easier maintanenace) and optimize some (to avoid resource overuse). Our first attempt was to use statsd to collect metrics and display those with graphite. But it just didn't work out. There are just too many different "keys". Basically, each endpoint (aka URL, aka URI) is a key. And if you include the query string parameters, the number of keys … -
Florida Open Debate Platform Receives National Attention (The Atlantic, USA Today, Engadget)
Several national publications have featured the Florida Open Debate platform, including USA Today, Engadget, and The Atlantic. Caktus helped develop the Django-based platform on behalf of the Open Debate Coalition (ODC) in advance of the nation’s first-ever open Senate debate held in Florida on April 25th. The site enabled citizens to submit debate questions as well as vote on which questions mattered most to them. Moderators then used the thirty most popular questions from the site to structure the debate between Florida Senate candidates David Jolly (R) and Alan Grayson (D). According to *The Atlantic, *more than 400,000 votes were submitted by users on the site, including more than 84,000 from Florida voters. -
ES6 For Django Lovers
ES6 for Django Lovers! The Django community is not one to fall to bitrot. Django supports every new release of Python at an impressive pace. Active Django websites are commonly updated to new releases quickly and we take pride in providing stable, predictable upgrade paths. -
Multi-table Inheritance and the Django Admin
Django's admin interface is a great way to be able to interact with your models without having to write any view code, and, within limits, it's useful in production too. However, it can quickly get very crowded when you register lots of models. Consider the situation where you are using Django's multi-table inheritance: {% highlight python %} from django.db import models from model_utils.managers import InheritanceManager class Sheep(models.Model): sheep_id = models.AutoField(primary_key=True) tag_id = models.CharField(max_length=32) date_of_birth = models.DateField() sire = models.ForeignKey('sheep.Ram', blank=True, null=True, related_name='progeny') dam = models.ForeignKey('sheep.Ewe', blank=True, null=True, related_name='progeny') objects = InheritanceManager() class Meta: verbose_name_plural = 'sheep' def __str__(self): return '{}: {}'.format(self._meta.verbose_name, self.tag_id) class Ram(Sheep): sheep = models.OneToOneField(parent_link=True) class Meta: verbose_name = 'ram' verbose_name_plural = 'rams' class Ewe(Sheep): sheep = models.OneToOneField(parent_link=True) class Meta: verbose_name = 'ewe' verbose_name_plural = 'ewes' {% endhighlight %} Ignore the fact there is no specialisation on those child models: in practice you'd normally have some. Also note that I've manually included the primary key, and the parent link fields. This has been done so that the actual columns in the database match, and in this case will all be `sheep_id`. This will make writing joins slightly simpler, and avoids the (not specific to Django) ORM anti-pattern of … -
(Directly) Testing Django Formsets
Django Forms are excellent: they offer a really nice API for validating user input. You can quite easily pass a dict of data instead of a `QueryDict`, which is what the request handling mechanism provides. This makes it trivial to write tests that exercise a given Form's validation directly. For instance: {% highlight python %} def test_my_form(self): form = MyForm({ 'foo': 'bar', 'baz': 'qux' }) self.assertFalse(form.is_valid()) self.assertTrue('foo' in form.errors) {% endhighlight %} Formsets are also really nice: they expose a neat way to update a group of homogenous objects. It's possible to pass a list of dicts to the formset for the `initial` argument, but, alas, you may not do the same for passing data. Instead, it needs to be structured as the `QueryDict` would be: {% highlight python %} def test_my_formset(self): formset = MyFormSet({ 'formset-INITIAL_FORMS': '0', 'formset-TOTAL_FORMS': '2', 'formset-0-foo': 'bar1', 'formset-0-baz': 'qux1', 'formset-1-foo': 'spam', 'formset-1-baz': 'eggs' }) self.assertTrue(formset.is_valid()) {% endhighlight %} This is fine if you only have a couple of forms in your formset, but it's a bit tiresome to have to put all of the prefixes, and is far noisier. Here's a nice little helper, that takes a `FormSet` class, and a list (of dicts), and instantiates … -
2016 DBIR Highlights
The 2016 edition of Verizon’s Data Breach Investigations Report is out, and as usual it’s compelling reading. The DBIR is one of the only sources of hard data about information security, which makes it a must-read for anyone trying to run a security program in a data-driven manner. What follows are the bits that I found especially interesting, and a bit of my own commentary. Internal threats are rare [T]he Actors in breaches are predominantly external. -
Florida Open Debate Site Powers First-Ever Crowd-Sourced Open Senate Debate
Florida Open Debate launched ahead of the upcoming, bi-partisan debate between candidates for the Florida Senate. The site, which crowdsources debate questions from the general public, was met with national acclaim. Citizens can not only submit questions, but also vote on which ones matter most. Caktus helped develop the tool on behalf of the Open Debate Coalition (ODC), a non-partisan organization dedicated to supporting participatory democracy through the use of civic tech. -
Django: Running management commands inside a Docker container
Okay, so we have dockerized our django app and we need to run a manage.py command for some task. How do we do that? Simple, we have to locate the container that runs the django app, login and then run the command. Locate The Container It’s very likely that our app uses multiple containers to compose the entire system. For exmaple, I have one container running MySQL, one container running Redis and another running the actual Django app. If we want to run manage.py commands, we have to login to the one that runs Django. While our app is running, we can find the running docker containers using the docker ps command like this: $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 308f40bba888 crawler_testscript "/sbin/my_init" 31 hours ago Up 3 seconds 5000/tcp crawler_testscript_1 3a5ccc872215 crawler_web "bash run_web.sh" 31 hours ago Up 4 seconds 0.0.0.0:8000->8000/tcp crawler_web_1 14f0e260fb2c redis:latest "/entrypoint.sh redis" 31 hours ago Up 4 seconds 0.0.0.0:6379->6379/tcp crawler_redis_1 252a7092870d mysql:latest "/entrypoint.sh mysql" 31 hours ago Up 4 seconds 0.0.0.0:3306->3306/tcp crawler_mysql_1 In my case, I am using Docker Compose and I know my Django app runs using the crawler_web image. So we note the name of the container. In … -
Autocomplete with Django-Haystack and Elasticsearch with single letter querying.
Django's haystack provides autocomplete functionality. To do autocomplete effectively, the search backend(elasticsearch in this case) uses n-grams (essentially a small window passed over the string). Because this alters the way your data needs to be stored. We had two choices: NgramField and EdgeNgramField used as n-grams in search backend. The major drawback of the n-grams is that they take minimum of 3 letters in the search query. Still EdgeNgramField or NgramField fields won't produce consistent results(For Eg: . By customising the Haystack's built in elastcisearch engine backend we can achieve single letter query with Charfield itself. In your search_indexes.py from haystack import indexes from myapp.models import Book class BookIndex(indexes.SearchIndex, indexes.Indexable): text = indexes.CharField(document=True, use_template=True) title = indexes.CharField(model_attr='title') pub_date = indexes.DateTimeField(model_attr='pub_date') def get_model(self): return Book Create backends.py in your app. And this file will contain the actual customized elasticsearch search engine backend. from django.conf import settings from haystack.backends.elasticsearch_backend import ElasticsearchSearchBackend from haystack.backends.elasticsearch_backend import ElasticsearchSearchEngine from haystack.backends.elasticsearch_backend import ElasticsearchSearchQuery from haystack.constants import VALID_FILTERS, FILTER_SEPARATOR, DEFAULT_ALIAS from haystack.inputs import Clean, Exact, PythonData, Raw from django.utils import six class CustomElasticsearchSearchQuery(ElasticsearchSearchQuery): def __init__(self, using=DEFAULT_ALIAS): super(CustomElasticsearchSearchQuery, self).__init__(using=DEFAULT_ALIAS) def build_query_fragment(self, field, filter_type, value): from haystack import connections query_frag = '' if not hasattr(value, 'input_type_name'): # Handle when we've got a ``ValuesListQuerySet``... if … -
How to create Custom User Model in Django?
Django provides built-in authentication which is good for most of the cases, but you may have needs that are being served by the existing system. For Ex: You want 'email' for authentication purpose rather than Django's username field and you want an extra field called 'display_name' as a full name for the logged in User. To meet the above requirements, we need to customize Django's built-in user model or substitute a completely customized model. In this blog post, we'll learn how to customize Django's built-in User model. Its good idea to extend Django User Model rather than writing whole new user model and there are a lot of ways but one good way is to extend it from Django itself. We get all features and properties of Django User and our own custom features on top of it. Add the following to your app that holds custom user model. In models.py from django.db import models from django.contrib.auth.models import AbstractUser class User(AbstractUser): name = models.CharField(max_length=100, blank=True, null=True) and specify your custom user model in settings.py AUTH_USER_MODEL = ‘your_app.User' If you want to use your custom user model in models.py, you can get that model with `settings.AUTH_USER_MODEL` or `get_user_model()` of Django's … -
A reading list for InfoSec engineers
I’ve started a curated reading list for InfoSec engineers. I was inspired by Mark McGranaghan’s Services Engineering reading list. I really enjoy these kinds of personal, highly-curated reading lists, and for some time I’ve wanted to pull together one of my own. This is my list, not a definitive one — that is, these are resources I’ve found useful. As such it has some biases: It’s oriented towards providers of Software-, Platform-, and Infrastructure-as-a-Service. -
Django REST Framework: Remember to disable Web Browsable API in Production
So this is what happened – I built an url shortening service at work for internal use. It’s a very basic app – shortens urls and tracks clicks. Two models – URL and URLVisit. URL model contains the full url, slug for the short url, created time etc. URLVisit has information related to the click, like user IP, browser data, click time etc and a ForeignKey to URL as expected. Two different apps were using this service, one from me, another from a different team. So I kept the Web Browsable API so the developers from other teams can try it out easily and they were very happy about it. The only job of this app was url shortening so I didn’t bother building a different home page. When people requested the / page on the domain, I would redirect them directly to /api/. Things were going really great initially. There was not very heavy load on the service. Roughly 50-100 requests per second. I would call that minimal load. The server also had decent hardware and was running on an EC2 instance from AWS. nginx was on the front while the app was run with uwsgi. Everything was so … -
Psychological safety in the InfoSec industry
My co-worker Eric Mill recently brought up the topic of psychological safety. Referencing a study by Google that points to psychological safety as a key factor in successful teams, Eric wrote: Maybe these situations sounds familiar to others (they definitely both are to me): Did you feel like you could ask what the goal was without the risk of sounding like you’re the only one out of the loop? -
Factory Boy - An alternative for Fixtures
Factory Boy is a fixtures replacement tool. It allows you to use objects customized for the current test, while only declaring the test-specific fields. Setting up your factories: For testing the Django app, create a root directory named tests. Then create __init__.py, tests.py (to write your tests), factories.py (to write your model factories) in that directory. So, your app layout looks like: myapp |- __init__.py |- admin.py |- forms.py |- models.py |- tests |- __init__.py |- factories.py |- tests.py |- urls.py |- views.py Features of Factory Boy: It allows you to automate much of the testing scenarios. Some examples are: 1. Use a 'Sequence' object using a lambda function to dynamically create unique field values: EX: username = factory.Sequence(lambda n: 'user_%d' % n) 2. Use a LazyAttribute object to pick from a range of choices for a defined field EX: choice = factory.LazyAttribute(lambda s: random.choice(dict(MyModel.CHOICES).keys())) A basic factory: Factories declare a set of attributes used to instantiate an object. Here's a basic factory that … -
Django Generic many to many field implementation
Django application has a GM2MField that combines the features of the standard Django "ManyToManyField" and "GenericForeignKey". Installation: pip install django-gm2m Django’s contenttype framework must be (django.contrib.contenttypes) mentionned in the INSTALLED_APPS. INSTALLED_APPS = [ ... 'django.contrib.contenttypes', ... 'gm2m', ] Uses Of django-gm2m: 1. reverse relations 2. prefetching 3. Allows you to customize the deletion behaviour Example Models: from django.db import models class Videos(models.Model): name = models.CharField(maxlength=100) class Articles(models.Model): name = models.CharField(maxlength=100) If you want to have a field for the preferred videos and articles of a User, you need to add GM2MField to the User model. from gm2m import GM2MField class User(models.Model): email = models.CharField(maxlength=100) username = models.CharField(maxlength=100) related = models.GM2MField() Now you can add videos, articles to the related set. Creating User object: user = User.objects.create(email="mp@mp.com", username="mp@mp.com") video = Videos.objects.create(name="video") article = Articles.objects.create(name="sample document") user.related.add(video) user.related.add(article) or you can add both videos, articles instances. user.related = [video, article] From User instance, you can fetch all related videos, articles set or you can filter model using the "Model" or "Model__in" keywords. list(user.related) # This provides list of model objects list(user.related.filter(Model=Videos)) # This … -
How to Create initial django migrations for existing DB schema.
Django provides the comfort database migrations from its version 1.8, with which we can avoid the usage of third party packages like the south. Adding migrations to new apps is straightforward - they come preconfigured to accept migrations, and so just run make migrations once you’ve made some changes. But if your app already has models and database tables, and doesn’t have migrations yet (for example, you created it against a previous Django version) or you got your migrations messed up, you’ll need to convert your app to use migrations. Following are the steps to create initial migrations to your app: Step1: Empty the django_migrations table: Just go to your corresponding database terminals and delete all the records from you django_migrations table with delete from django_migrations; Step2: Remove all the files in migrations folders in each and every app of your project. Go to terminal and run remove all files in migrations folder with rm -rf <app>/migrations/ Step3: Reset the migrations for the "built-in" apps: Reset all the migrations of the Django's built-in apps like admin with the command python manage.py migrate --fake Step4: Create initial migrations for each and every app: For each app run: python manage.py makemigrations <app>. … -
Celery Flower to monitor task queue
Celery is a task queue that is to built an asynchronous message passing system. It can be used as a bucket where programming tasks can be dumped. The program that passed the task can continue to execute and function responsively. To monitor the status of these celery tasks we use celery flower The Celery Flower is a tool for monitoring your celery tasks and workers. It's web based and allows you to see task progress, details, worker status. Install flower with pip pip install flower Install RabbitMQ Celery requires a messaging agent in order to handle requests from an external source. This agent is referred to as a "broker". RabbitMQ is the messaging server which powers this service. RabbitMQ does the following: Listens for messages sent via the AMQP protocol. Stores these messages in one or more queues. Releases these messages to workers for consumption and processing. RabbitMQ does not actually execute tasks. Rather, messages will be sent and stored in queue untill executed. Install rabbitmq using sudo apt-get install rabbitmq-server Add broker_api url in your settings.py RabbitMQ management api broker_api = 'http://guest:guest@localhost:15672/api/' # Enable debug logging logging = 'DEBUG' We need to configure RabbitMQ for message broker services before … -
Check test coverage in Django code with Coveralls
Coverage: It is a tool used for showing the percentage of your codebase covered by tests. Test Coverage is an important indicator of software quality and an essential part of software maintenance. It helps in evaluating the test cases by providing the data on different coverage items. It is a useful tool for finding untested part of a code. Test coverage can help in monitoring the quality of testing and assist in directing the test generators to create test cases that cover areas that have not been tested. It helps to measure the quality and identifies unnecessary test cases which will not increase coverage. Benefits of Test Coverage: Defect prevention in the project. It creates additional test cases to increase coverage. It helps in determining a quantitative measure of code coverage, which indirectly measures the quality of the application or product. Installation: Pip install coverage Use "coverage run" to run your program and gather data: coverage run manage.py test coverage report -m # provides the report for the tests Individual app test can be run using the command coverage run --source=app1,app2 manage.py test -
From Intern to Professional Developer: Advice on a Mid-Career Pivot
A few weeks ago, Rebecca Conley attended DjangoCon Europe 2016 in Budapest, Hungary. The event is a five-day conference that brings together Django lovers from all over the world to learn about and share each other’s experiences with Django. -
Adopting Scrum in a Client-services, Multi-project Organization
Caktus began the process of adopting Scrum mid-November 2015 with two days of onsite Scrum training and fully transitioned to a Scrum environment in January 2016. From our original epiphany of “Yes! We want Scrum!” to the beginning of our first sprint, it took us six weeks to design and execute a process and transition plan. This is how we did it:
