Django community: RSS
This page, updated regularly, aggregates Community blog posts from the Django community.
-
Encoding Problem
It was pointed out today that the euro sign was not displaying properly on the news page. I had a look and I saw that the problem is that the database encoding is set to Latin-1 instead of UTF-8. It turns out that was the default on the version Debian I started with and as I've dumped and restored the database the encoding has remained the same. The solution is relatively simple which is to dump the database and reload with the correct encoding. This could be done during a short maintenance window in the middle of the night. But no! As the sun no longer sets on the Fine Coffee Club empire there is no particularly good time when we can shut down for maintenance. Update: It turned out to only require a couple of minutes of outage. However the point remains. -
Currency Refactor
This is technical notes just for me on the currency refactor. The background is that currency is presently on the presentation layer in the templates. However as I'm expanding currencies this is getting more complex and there is also a problem that invoices are generated as a PDF so I have to do the currency code twice: once for the templates and once for the PDF which break the DRY (Do not Repeat Yourself) rule. The code has to deal with the following: Changing currency symbol, e.g. £1.00 and $1.00.Changing decimal separator, e.g. €1.00 and €1,00Changing currency symbol position, e.g. €1.00 and 1,00€Different currencies with the same symbol e.g. Canadian dollars (CAD) and US dollars (USD). An additional requirement is that I want to be able to add currencies relatively frequently without changing too much code. Pushing this down into the model layer should make things a bit easier but there is quite a lot of complexity that my design has to cope with. Products and Shipment Methods have multiple prices, one for each currency, chosen by settings.pyBaskets have one currency for multiple totals depending on settings.pyShipment and Order have the currency in the model and multiple totalsAdmin interface reports … -
Testing Update
I'm off on holiday on Friday for a week without much in the way of internet access so I'm going to have a code freeze as of now. The final code I committed was i18n for the French launch. Unfortunately there was a bug in one of the live site settings files which caused some problems on the Irish site. I fixed it and wrote a test which checks all the major settings across all the countries. I also ran coverage.py on the codebase which shows the percent of the code covered by tests and can generate a nice HTML report. At the moment it is 61%. During the code freeze I am going to focus on adding unit tests to try get the coverage over 70% this week. In other news, outsourcing of fulfilment is going well with Seko chosen who have warehousing in the USA, UK, Europe, Australia and other locations giving us a complete global presence. They are really a bit big for us but that means we can stick with them long term. There's going to be quite a lot of work in integrating into their API but it is very well documented. It also means … -
Django / ZenDesk API
Anna was off and I was left doing all of the email support which we do through the excellent Zendesk. After about 10 minutes I has deeply frustrated with the process of finding customer orders on our system. So I've written a simple application which links straight from a ticket to the relevant orders with just a couple of clicks. It has saved an enormous amount of time. I'm going to generalise this slightly by getting it to bring up the Django user and then I'll publish it as the first bit of code to be released. -
Django Errors Going to Spam
Like many sites I've setup Django to email me when there are errors. Unfortunately the errors were all ending up in my spam. I had a look at the mail system and there were no problems there. The first issue was very simple. It'd not set SERVER_EMAIL in settings.py which meant that the emails were being sent from 'root@localhost' which spam filters are not going to like. The second problem was the very large number of emails which was also a problem for me in that I stopped taking the emails seriously. The Django ALLOWED_HOSTS setting improves security but generates a lot of errors (this has been fixed in the development version of Django but for now it fills up the log). I found an excellent article Prevent email notification on SuspiciousOperation with detailed code which I've implemented and it's made a huge difference. -
Blog Move
I've decided to move my blog from WordPress.com onto Django on the Fine Coffee Club site. I put together a simple blogging app in about 100 lines of code. I'm struggling to write a proper justification for this, WordPress worked perfectly well and the last thing the world needed was another blogging product. The best I can come up with is that I wanted to play with the syndication framework and it gave me the chance to play around with some other odds and ends in the framework that I don't need in my day to day e-commerce. It is also easier to customise when it is on your own server and I would rather not install WordPress as that would also required PHP and MySQL to be installed which are a little heavy weight to be on my main server. It's a very weak justification I know! R6J9WMEMPRC4 -
Reverse Is A Good Idea
Up to know the Fine Coffee Club websites have each had their domain like this: finecoffeeclub.comfinecoffeeclub.co.ukfinecoffeeclub.iefinecoffeeclub.ft However for Australia and Canada we are not going to give them their own domain names for various good reasons, so these addresses will be: finecoffeeclub.com/aufinecoffeeclub.com/ca I thought that would be an easy change to make the change but it turned out to be quite tricky as all of the links in the templates and more than 100 redirects in the Python code will all have to change. It would also be a bit of a pain to work out exactly what the URL's as some of them will be prefixed with the country code and some of them not. I am now porting everything over to using reverse resolution and I wish I had done this from the start. Up till now I hadn't really appreciated how useful an idea it is. With the url tag I can now leave the whole thing up to Django. -
Converting our multi-page Django app to use AMD
Introduction The application which powers 2degrees has, to date, mainly been driven by a Django-powered back-end. Django is agnostic about any front-end stack and this has certainly contributed to a fairly ad-hoc approach to javascript. With an increasing demand for a more interactive experience, we have been adding more and more javascript, experimenting with AngularJS to power our pinboard, and looking at ways to use/build frameworks to play nicely with the back-end. As a result of this work, we have found issues of manageability of the javascript code. Manageability issues The issues we had fell into the following categories: Risk of missing dependencies when inheriting complex pages with lots of javascript. Dependencies loading out-of-order in some cases. The code becoming increasingly complex to manage (poor separation of concerns in the code, etc.) AMD to the rescue? We hoped that a modular approach to javascript would address all of these issues and more. After some research about various module patterns and loaders, we decided to use RequireJS. This decision was based mainly on API features and how well-maintained and well-documented the library is. Porting the codebase to using RequireJS After performing a spike to asses the complexity of the task, we … -
使用 uncss 找出未使用的 CSS 样式
我们知道, 保持代码的经凑干净对于代码的可阅读性至关重要, 之前的博文中也讲过这点. 但在修改CSS时, 我们经常会因为大量的CSS代码和CSS自身的可覆盖性而只增加不删减. 本篇中我们就介绍一下uncss, 一个可以用于找出未使用的CSS代码的工具, 方便我们精简代码. uncss最基本的用法是命令行: uncss http://example.com > styles.css 输出的结果就是在使用的CSS代码, 而未使用的CSS代码已经被自动删除掉. uncss是如何工作的呢? HTML文件通过PhantomJS载入, 相关JavaScript代码被运行 在使用的CSS代码从HTML文件中抽离 CSS代码通过css-parse重新组合起来 document.querySelector将HTML中未找到的selector分离出去 剩余的CSS代码重新加入组合 就像每个NodeJS工具一样, 我们能使用其JavaScript API: var uncss = require('uncss'); var files = ['my', 'array', 'of', 'HTML', 'files'], options = { ignore : ['#added_at_runtime', /test\-[0-9]+/], media : ['(min-width: 700px) handheld and (orientation: landscape)'], csspath : '../public/css/', raw : 'h1 { color: green }', stylesheets : ['lib/bootstrap/dist/css/bootstrap.css', 'src/public/css/main.css'], ignoreSheets : [/fonts.googleapis/], urls : ['http://localhost:3000/mypage', '...'], // Deprecated timeout : 1000, htmlroot : 'public' }; uncss(files, options, function (error, output) { console.log(output); }); /* Look Ma, no options! */ uncss(files, function (error, output) { console.log(output); }); /* Specifying raw HTML */ var raw_html = '...'; uncss(raw_html, options, function (error, output) { console.log(output); }); -
Extending Django's QuerySet to return approximate COUNTs
UPDATE: I’ve re-written and open-sourced a better way of doing the below as part of my library django-mysql. The docs there on approximate counting are just as good a read as the below, and you can pip install the solution. I was looking through the MySQL slow_log for YPlan and discovered that there were a lot of SELECT COUNT(*) queries going on, which take a long time because they require a full table scan. These were coming from the Django admin, which displays the total count on every page. “Why is SELECT COUNT(*) such a slow query?” you might think, “surely MySQL could just keep a number in the table metadata and update it on INSERT/DELETE.” Aha! You are totally right - for the MyISAM storage engine. But Innodb, which you shoudl be using, provides transacational support and other niceties, at the cost of making such a metadata count impossible. Each transaction must be isolated from the others until it commits or rolls back, so a single ‘accurate’ COUNT(*) value per table is impossible. It would also be a point of contention from locking, which MyISAM doesn’t care about anyway because it locks the whole table for any write. Hence, … -
Extending Django's QuerySet to return approximate COUNTs
UPDATE: I've re-written and open-sourced a better way of doing the below as part of my library django-mysql. The docs there on approximate counting are just as good a read as the below, and you can pip install the solution. I was looking through the MySQL slow_log for YPlan and discovered that there were a lot of SELECT COUNT(*) queries going on, which take a long time because they require a full table scan. These were coming from the Django admin, which displays the total count on every page. "Why is SELECT COUNT(*) such a slow query?" you might think, "surely MySQL could just keep a number in the table metadata and update it on INSERT/DELETE." Aha! You are totally right - for the MyISAM storage engine. But Innodb, which you shoudl be using, provides transacational support and other niceties, at the cost of making such a metadata count impossible. Each transaction must be isolated from the others until it commits or rolls back, so a single 'accurate' COUNT(*) value per table is impossible. It would also be a point of contention from locking, which MyISAM doesn't care about anyway because it locks the whole table for any write. Hence, … -
Extending Django's QuerySet to return approximate COUNTs
UPDATE: I've re-written and open-sourced a better way of doing the below as part of my library django-mysql. The docs there on approximate counting are just as good a read as the below, and you can pip install the solution. I was looking through the MySQL slow_log for YPlan and discovered that there were a lot of SELECT COUNT(*) queries going on, which take a long time because they require a full table scan. These were coming from the Django admin, which displays the total count on every page. "Why is SELECT COUNT(*) such a slow query?" you might think, "surely MySQL could just keep a number in the table metadata and update it on INSERT/DELETE." Aha! You are totally right - for the MyISAM storage engine. But Innodb, which you shoudl be using, provides transacational support and other niceties, at the cost of making such a metadata count impossible. Each transaction must be isolated from the others until it commits or rolls back, so a single 'accurate' COUNT(*) value per table is impossible. It would also be a point of contention from locking, which MyISAM doesn't care about anyway because it locks the whole table for any write. Hence, … -
Python中使用Faker创建虚拟数据
在测试数据库时, 我们经常会需要用到假数据来支持代码的运行. 本篇就介绍一下Faker, Faker的唯一功能就是生成半随机的虚假数据, 例如名字, 地址, 域名, 段落等. 创建virtualenv, 并安装Faker: mkvirtualenv test pip install fake-factory 创建假名字: from faker import Factory #---------------------------------------------------------------------- def create_names(fake): """""" for i in range(10): print fake.name() if __name__ == "__main__": fake = Factory.create() create_names(fake) 得到不同的名字, (你得到的应该与我不同): Mrs. Terese Walter MD Jess Mayert Ms. Katerina Fisher PhD Mrs. Senora Purdy PhD Gretchen Tromp Winnie Goodwin Yuridia McGlynn MD Betty Kub Nolen Koelpin Adilene Jerde 不希望名字中含有铅坠河后缀: from faker import Factory #---------------------------------------------------------------------- def create_names2(fake): """""" for i in range(10): name = "%s %s" % (fake.first_name(), fake.last_name()) print name if __name__ == "__main__": fake = Factory.create() create_names2(fake) 接下来我们看如何生成其他虚假数据: from faker import Factory #---------------------------------------------------------------------- def create_fake_stuff(fake): """""" stuff = ["email", "bs", "address", "city", "state", "paragraph"] for item in stuff: print "%s = %s" % (item, getattr(fake, item)()) if __name__ == "__main__": fake = Factory.create() create_fake_stuff(fake) 你可能会得到以下信息: email = pacocha.aria@kris.com bs = reinvent collaborative systems address = 57188 Leuschke Mission Lake Jaceystad, KY 46291 city = West Luvinialand state = Oregon paragraph = Possimus nostrum exercitationem harum eum in. Dicta aut off Faker中还有许多其他方法没有在这里提及到, 你可以自行查看其文档. -
Tinkering with Django and Ember.js
Hi everyone. Recently I have been looking around at the various JavaScript frameworks that have been blooming out recently. Three came in front of the others, namely BackboneJs, AngularJs and EmberJs. After looking at all 3 of them, EmberJs is the one which appeals to me the most. But if you hadn't guessed from the tutorials on this blog, I also like Django a lot and most tutorials out there talking about EmberJs and actually using a data backend are using Ruby on Rails and the few that I could find that were talking about Django were based on TastyPie while I like Django-REST-Framework better... So I decided to dive in and go full speed ahead and learn the hard way. The result of which is Djember-CMS. -
Python中如何重新引入被覆盖的自带function
最近在写python应用时遇到一个问题: 引入某个模块时会自动引入自定义的int到python的namespace中, 从而覆盖了python自带的int function. 因为我们需要使用python的int, 所以不得不找到重新引入这int的方法: 幸运的是, 这一问题还是很容易解决的, 我们只需要使用__builtins__: from __builtins__ import int as py_int 这样一来我们又可以重新使用python的int了, 但在此时叫做py_int. 一个function或变量的被覆盖最常见的原因是在引用时使用了"*": from something import * 当这样使用import时, 我们无法明确的指导究竟引入了哪些变量或function, 也无法知道这些变量或function是否会覆盖原来的变量或function. 所以这也是在使用import时不推荐使用"*"的主要原因之一. 在python 3中, 可以使用builtins代替__builtins__. -
July 2014 ShipIt Day Recap
This past Friday we celebrated another ShipIt day at Caktus. There was a lot of open source contribution, exploring, and learning happening in the office. The projects ranged from native mobile Firefox OS apps, to development on our automated server provisioning templates via Salt, to front-end apps aimed at using web technology to create interfaces where composing new music or performing Frozen’s Let It Go is so easy a anyone can do it. Here is a quick summary of the projects that folks worked on: Calvin worked on updating our own minimal CMS component for editing content on a site, django-pagelets, to work nicely with Django 1.7. He also is interested in adding TinyMCE support and making it easy to upload images and reference them in the block. If you have any other suggestions for pagelets, get in touch with Calvin. Philip worked on a code to tag words in a text with basic information about their etymologies. He was interested in exploring words with dual French and Anglo-Saxon variations eg “Uncouth” and “Rude”. These words have evolved from different origins to have similar meanings in modern English and it turns out that people often perceive the French or Latin … -
July 2014 ShipIt Day Recap
This past Friday we celebrated another ShipIt day at Caktus. There was a lot of open source contribution, exploring, and learning happening in the office. The projects ranged from native mobile Firefox OS apps, to development on our automated server provisioning templates via Salt, to front-end apps aimed at using web technology to create interfaces where composing new music or performing Frozen’s Let It Go is so easy anyone can do it. -
Django访问多个PostgreSQL Schema
django缺少对PostgreSQL的多schema支持, 之前我们尝试了多种方法访问除public schema之外的schemas, 但这些方式都难以维护. 然而, 最近我们发现这一问题可以使用PostgreSQL的search_path参数轻松地解决. 一个简单的例子 假设一个django项目中所有的表都创建在django schema中, 并且我们的项目需要用到legacy schema中几个表. 我们可以通过PostgreSQL数据库映射, 然后在django中使用DATABASE设置实现, 让我们展示一下其他两种不同的设置方法: 在连接时设置search_path 假设django和legacy schema已经存在, 且django项目具有数据库的访问权. 在django设置中, 我们在options中设置search_path: # settings.py DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'OPTIONS': { 'options': '-c search_path=django,public' }, 'NAME': 'multi_schema_db', 'USER': 'appuser', 'PASSWORD': 'secret', }, 'legacy': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'OPTIONS': { 'options': '-c search_path=legacy,public' }, 'NAME': 'multi_schema_db', 'USER': 'appuser', 'PASSWORD': 'secret', }, } 这一设置方式对于已有的项目而言需要修改的地方最少. 设置不同的数据库用户 对于以上设置而言, 最大的问题可能是set search_path参数在每次django项目与PostgreSQL数据库进行连接时都需要传输. 为了节约这一时间, 我们可以事先为每个用户设定search_path: 用postgres用户登入psql shell: -- user accessing django schema... CREATE USER django_user LOGIN PASSWORD 'secret'; GRANT appuser TO django_user; ALTER ROLE django_user SET search_path TO django, public; -- user accessing legacy schema... CREATE USER legacy_user LOGIN PASSWORD 'secret'; GRANT appuser TO legacy_user; ALTER ROLE legacy_user SET search_path TO legacy, public; django项目中settings.py: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'multi_schema_db', 'USER': 'django_user', 'PASSWORD': 'secret', }, 'legacy': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'multi_schema_db', 'USER': 'legacy_user', 'PASSWORD': 'secret', }, } 以上便是两种不同的使用search_path的方式. 自定义django database router也许也能做到自动选择正确的schema. -
Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis
Setting up celery with Django can be a pain, but it doesn't have to be. In this video learn what it takes to setup Celery for deferred tasks, and as your cron replacement. We will use Celery 3.1 and Django 1.7 both introduce changes you need to be aware of.Watch Now... -
lxml的元素构建器和CDATA对象
lxml有着非常好用的元素构建器, 但对于CDATA对象似乎没这么给力: >>> from lxml.builder import E >>> from lxml.etree import CDATA >>> E.stuff(CDATA('Some stuff that needs to be in a CDATA section')) Traceback (most recent call last): File "<ipython-input-4-40103024e8d8>", line 1, in <module> E.stuff(CDATA('Some stuff that needs to be in a CDATA section')) File "/usr/lib/python2.7/dist-packages/lxml/builder.py", line 220, in __call__ raise TypeError("bad argument type: %r" % item) TypeError: bad argument type: <lxml.etree.CDATA object at 0x238a130> 可以使用以下方式修复: from lxml.builder import ElementMaker from lxml.etree import CDATA def add_cdata(element, cdata): assert not element.text, "Can't add a CDATA section. Element already has some text: %r" % element.text element.text = cdata E = ElementMaker(typemap={ CDATA: add_cdata }) 然后就可以正常使用了: >>> from lxml import etree >>> etree.tostring(E.stuff(CDATA('Some stuff that needs to be in a CDATA section'))) '<stuff><![CDATA[Some stuff that needs to be in a CDATA section]]></stuff>' -
如何在Python中发送邮件
使用python发送邮件有以下几种情况: 纯文本的邮件 带附件的邮件 其他邮件 首先我们使用virtualenv创建环境: $ virtualenv env $ env/bin/pip install wheezy.core 纯文本邮件 直接上代码: # plain.py from wheezy.core.mail import MailMessage from wheezy.core.mail import SMTPClient mail = MailMessage( subject='Welcome to Python', content='Hello World!', from_addr='someone@dev.local', to_addrs=['you@dev.local']) client = SMTPClient() client.send(mail) 然后使用以下命令发送: python plain.py 如果需要发送HTML的邮件的话, 只需要将content换成HTML, 将content_type设置成'text/html', 设置charset即可: from wheezy.core.mail import MailMessage from wheezy.core.mail import SMTPClient content = """\ <html><body> <h1>Hello World!</h1> </body></html>""" mail = MailMessage( subject='Welcome to Python', content=content, content_type='text/html', charset='utf-8', from_addr='someone@dev.local', to_addrs=['you@dev.local']) client = SMTPClient() client.send(mail) 另外, 还可以设置SMTPClient的host, port, tls, credentials等. 带附件的邮件 # attachment.py from wheezy.core.mail import Attachment from wheezy.core.mail import MailMessage from wheezy.core.mail import SMTPClient mail = MailMessage( subject='Welcome to Python', content='Hello World!', from_addr='someone@dev.local', to_addrs=['you@dev.local']) mail.attachments.append(Attachment( name='welcome.txt', content='Hello World!')) client = SMTPClient() client.send(mail) 可以使用factory方法Attachment.from_file从本地文件创建一个附件 其他邮件 我们下载python的logo作为邮件: import os.path from wheezy.core.mail import Alternative from wheezy.core.mail import MailMessage from wheezy.core.mail import Related from wheezy.core.mail import SMTPClient mail = MailMessage( subject='Welcome to Python', content='Hello World!', from_addr='someone@dev.local', to_addrs=['you@dev.local']) alt = Alternative("""\ <html><body> <h1>Hello World!</h1> <p><img src="cid:python-logo.gif" /></p> </body></html>""", content_type='text/html') curdir = os.path.dirname(__file__) path = os.path.join(curdir, 'python-logo.gif') alt.related.append(Related.from_file(path)) mail.alternatives.append(alt) client = SMTPClient() client.send(mail) -
Python Dev Tip: DRY your shell with PYTHONSTARTUP
Python Dev Tip: DRY your shell with PYTHONSTARTUP -
heya-unify: back to JS
As programmers we rarely have a luxury to write a project from scratch. Usually we have to retrofit existing projects with all cool things we need. If a new component, or a library we want to use introduces new concepts that bleed outside its boundary, we have a “culture clash”, when old code is unaware about new concepts have to work with it anyhow. Sometimes the clash is so bad that we have to give up on using shiny new things, or have to significantly rework their code, which requires time and efforts we cannot afford. -
可重复使用的 juju charm: ansible role
我们时常会使用juju charms来自动化部署许多不同的paas, 大多数是wsgi应用. 我们通常使用ansible使juju charms自动化部署更加轻松, 但是每个wsgi charm还是得重复坐以下相同的事情: 设置特定的用户 安装build代码到特定目录 安装依赖包 与后台连接(postesql, elasticsearch等) 生成设置 设置wsgi服务 设置log 支持更新代码而不必升级charm 支持不断地更新代码 其中只有三项是会随着paas不同而存在略微差别的: 依赖包, 生成设置和后台连接. 在尝试创建一个可重复使用的wsgi charm后, 我们借助ansible内置的对可重复利用的roles的支持创建了charm-bootstrap-wsgi, 其中包含了以上所有需求. 其中charm非常简单, 重新是使用wsgi-app的role: roles: - role: wsgi-app listen_port: 8080 wsgi_application: example_wsgi:application code_archive: "{{ build_label }}/example-wsgi-app.tar.bzip2" when: build_label != '' 我们只需要做两件事情: tasks: - name: Install any required packages for your app. apt: pkg={{ item }} state=latest update_cache=yes with_items: - python-django - python-django-celery tags: - install - upgrade-charm - name: Write any custom configuration files debug: msg="You'd write any custom config files here, then notify the 'Restart wsgi' handler." tags: - config-changed # Also any backend relation-changed hooks for databases etc. notify: - Restart wsgi 其他则是由reusable wsgi-app role提供支持. -
From LIKE to Full-Text Search (part II)
If you missed it, read the first post of this series What do you do when you need to filter a long list of records for your users? That was the question we set to answer in a previous post. We saw that, for simple queries, built-in filtering provided by your framework of choice (think Django) is just fine. Most of the time, though, you'll need something more powerful. This is where PostgreSQL's full text search facilities comes in handy. We also saw that just using to_tsvector and to_tsquery functions goes a long way filtering your records. But what about documents that contain accented characters? What can we do to optimize performance? How do we integrate this with Django? Hola, Mundo! We have found that the need to search documents in multiple languages is fairly common. You can query your data using to_tsquery without passing a language configuration name but remember that, under the hood, the text search functions always use one. The default language is english, but you have to use the right language stemmer according to your document language or you might not get any matches. If, for example, we search for física in spanish documents that have …
