Two Approaches to Concurrent Write-Safety in Django

Two Approaches to Concurrent Write-Safety in Django - HedgeDoc

<center> # Two Approaches to Concurrent Write-Safety in Django *Originally published 2016-08-03 on [InvalidPatent.Wordpress.com](https://invalidpatent.wordpress.com/2016/08/03/two-approaches-to-concurrent-write-safety-in-django/).* For a more in-depth overview, see: <a href="https://docs.sweeting.me/s/HJdHDKhjz">Architecting a Banking service for Real-Time Gaming at OddSlingers</a> </center> --- Sometimes when dealing with Django models accessed by multiple people, you want a way to make sure two requests don't perform writes at the same time. For example, lets say you run a poker site, and you have 3 users playing a poker game together in the browser. <img class=" wp-image-341 aligncenter" src="https://invalidpatent.files.wordpress.com/2016/08/poker_game.png" alt="poker_game" width="562" height="449" /> You want only one active player to be able to perform game actions, and you want only one action accepted at a time. Imagine the catastrophe if your active player could open the game in 2 tabs, fold his hand in one, and bet in the other simultaneously! Here are two easy solutions to create concurrent-write-safe code (to prevent your hypothetical poker site from losing millions of dollars to devious 2-tab players): <ul> <li>atomic database transactions, using either version numbers or timestamps to check that the version you're writing is the same as the version on disk before committing a transaction (aka <a href="https://en.wikipedia.org/wiki/Optimistic_concurrency_control">Optimistic concurrency control/optimistic locking</a>)</li> <li>locking during the entire operation using a semaphore, to ensure only one person can modify the models at a time (aka <a href="https://en.wikipedia.org/wiki/Lock_(database)">normal locking/pessimistic locking</a>)</li> </ul> The first solution is commonly used when doing lock-free programming, but is difficult to implement properly. The second solution is simpler to implement, but can cause deadlocks if your code throws an exception and you leave models locked. Here I illustrate how to do both. The code examples use Django, but these principles are generalizable to almost any web backend that mutates shared data. <h2>Solution 1: Atomic Transactions with Check-Before-Write</h2> <pre class="lang-py prettyprint prettyprinted"><code>def save(self, *args, **kwargs): if self.id: on_disk = Game.objects.get(pk=self.id) if on_disk.modified > self.modified: raise StaleWriteError('Tried to save outdated Game') super(Game, self).save(*args, **kwargs) </code></pre> However, THIS CODE IS NOT CORRECT, this naive solution is not good enough to guarantee concurrent-write safety. Look over the code and think about it for a minute before reading on to find out why. ... The reason is: a race condition can occur between the timestamp if-check and the SQL update query. The proper way to do this is make the condition part of the update query (which is atomic). <pre class="lang-py prettyprint prettyprinted"><code>updated = Game.objects.filter(Q(id=game.id) && Q(version=game.version))\ .update(field_name=new_field_value, version=game.version + 1) if not updated: raise StaleWriteError('Tried to save outdated Game')</code></pre> Source: <a href="http://stackoverflow.com/questions/320096/django-how-can-i-protect-against-concurrent-modification-of-database-entries" target="_blank">this Stack Overflow question</a> Alternative implementations use a last-modified timestamp instead of a version number, or even the whole state of the database row (which is slow but avoids requiring a separate version column) to check before writing. If your database supports fast row-hashing, you can also compare row hashes before writing. <h2>Solution 2: Row Locking With a Redis Lock-Table</h2> Think carefully about whether you trust redis to be a safe enough backend for your locks. In it's default configuration, redis will lose data if the machine it's running on is suddenly powered off! ```python import redis from django.db import models lock_table = redis.StrictRedis(host='localhost', port=6379) class ConcurrentModificationError(ValueError): """Base error class for write concurrency errors""" pass class StaleWriteError(ConcurrentModificationError): """Tried to write a version of a model that is older than the current version in the database""" pass class AlreadyLockedError(ConcurrentModificationError): """Tried to aquire a lock on a row that is already locked""" pass class WriteWithoutLockError(ConcurrentModificationError): """Tried to save a lock-required model row without locking it first""" pass class LockedModel: """Add row-level locking backed by redis, set lock_required=True to require a lock on .save()""" lock_required = False # whether a lock is required to call .save() on this model @property def _lock_key(self): model_name = self.__class__.__name__ return '{0}__locked:{1}'.format(model_name, self.id) def is_locked(self): return lock_table.get(self._lock_key) == b'1' def lock(self): if self.is_locked(): raise AlreadyLockedError('Tried to lock an already-locked row.') lock_table.set(self._lock_key, b'1') def unlock(self): lock_table.set(self._lock_key, b'0') def save(self, *args, **kwargs): if self.lock_required and not self.is_locked(): raise WriteWithoutLockError('Tried to save a lock-required model row without locking it first') super(LockedModel, self).save(*args, **kwargs) # example usage to require locking on a model when calling .save(): class Game(models.Model, LockedModel): lock_required = True players = models.ManyToManyField(Player) ``` Locking is a generic pattern than can be used for more complex operations than just single-row locking. You can always manually use the lock_table to create and hold locks over whole blocks of code that aren't linked to a specific DB row. Here's how you'd use the above row-locking feature in a real use case, (e.g. calling perform_game_action from inside a view). ```python from django.db import IntegrityError, transaction from .models import Game, Player def perform_game_action(game: Game, new_player: Player): # acquire redis write-lock on db objects game.lock() try: with transaction.atomic(): # modify your database object here game.players.add(new_player) # save all modified state to database game.save() except ConcurrentModificationError, IntegrityError: # handle write integrity errors/lock contention cases here print('Game transaction failed!') finally: # release redis write-lock on table object game.unlock() ``` <h2>Alternative Solution 2: Use Django's built-in locking</h2> I've recently been informed of a third solution by the kind people of the internet and this <a href="http://stackoverflow.com/questions/1123200/how-to-lock-a-critical-section-in-django">Stack Overflow question</a>. This is the canonical "Django Solution", but my version above with Redis locking gives a little bit more exception granularity, allowing you to handle different contention cases separately. The Django method just throws a generic <a class="reference internal" title="django.db.DatabaseError" href="https://docs.djangoproject.com/en/1.9/ref/exceptions/#django.db.DatabaseError"><code class="xref py py-exc docutils literal">DatabaseError</code></a> regardless of the situation. Quoting the Django Docs, <a href="https://docs.djangoproject.com/en/1.9/ref/models/querysets/#django.db.models.query.QuerySet.select_for_update">select_or_update()</a>: Returns a queryset that will lock rows until the end of the transaction, generating a <code class="docutils literal">SELECT ... FOR UPDATE</code> SQL statement on supported databases. <pre>games = Game.objects.select_for_update().filter(active=True)</pre> All matched games will be locked until the end of the transaction block, meaning that other transactions will be prevented from changing or acquiring locks on them. Usually, if another transaction has already acquired a lock on one of the selected rows, the query will block until the lock is released. If this is not the behavior you want, call <code class="docutils literal">select_for_update(nowait=True)</code>. This will make the call non-blocking. If a conflicting lock is already acquired by another transaction, <a class="reference internal" title="django.db.DatabaseError" href="https://docs.djangoproject.com/en/1.9/ref/exceptions/#django.db.DatabaseError"><code class="xref py py-exc docutils literal">DatabaseError</code></a> will be raised when the queryset is evaluated. <hr /> Which approach you choose is up to you, just remember to test your code with a variety of write conditions and load levels. Prefer atomic operations over non-atomic, and test every lock contention edge-case before deploying your shiny new "concurrent-write-safe" code to production. Beware of <a href="https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use">TOCTTOU</a> bugs! Remember, a single edge case that happens even 1 out of every 10,000 requests can be exploited by a devious user, and could potentially cost your tiny startup lots of money! (follow-up post on rate-limiting in Django coming soon) Please leave comments if you have other solutions to share, or if you find any errors in my implementations! <hr /> <h3>Related reading:</h3> <ul> <li>Django Docs on the <a href="https://docs.djangoproject.com/en/1.10/topics/db/queries/#updating-multiple-objects-at-once">update() method</a></li> <li>Django Docs on the <a href="https://docs.djangoproject.com/en/1.9/ref/models/querysets/#django.db.models.query.QuerySet.select_for_update">select_for_update() method</a></li> <li>Django Docs on <a href="https://docs.djangoproject.com/en/1.10/topics/db/transactions/#controlling-transactions-explicitly">atomic database transactions</a></li> <li>Top <a href="http://stackoverflow.com/questions/320096/django-how-can-i-protect-against-concurrent-modification-of-database-entries">Stack Overflow question</a> on Django concurrent-write protection</li> <li>If you're into the history of this feature, check out the <a href="https://code.djangoproject.com/ticket/2705">original Django ticket</a> that added update() support</li> <li><a href="http://stackoverflow.com/questions/1123200/how-to-lock-a-critical-section-in-django">Stack Overflow question</a> on locking critical sections in Django</li> <li><a href="http://stackoverflow.com/questions/129329/optimistic-vs-pessimistic-locking">Stack Overflow question</a> explaining the difference between optimistic and pessimistic locking</li> <li>Wikipedia pages on <a href="https://en.wikipedia.org/wiki/Optimistic_concurrency_control">Optimistic Concurrency Control</a>, and <a href="https://en.wikipedia.org/wiki/Lock_(database)">Locking</a></li> </ul> P.S. Sorry about the ads and lack of syntax highlighting on my blog! I'll work on migrating it to Jekyll eventually...

Nick Sweeting

Monadical Co-Founder @theSquashSH

Published on 2016-08-03