transaction.atomic good practices

When using transaction.atomic() we should remember that it can commit or rollback just the db related calls inside the atomic() block. Itโ€™s not able to rewind a HTTP call, a call to redis or putting a payload on any queueing system. This means in case of a rollback weโ€™d emit an event or call an external API to inform it about an object which effectively was not saved to the database.

In the project Iโ€™m working on now a real life scenario would e.g. end up sending an e-mail or an SMS to a person when their order was not created and customer support would need to deal with this.

A good practice is tho have the atomic() block the minimal set of db calls which have to be run together. Everything else should ideally be moved out.

๐Ÿ“– Celery and transaction.atomic Stack Overflow thread

๐Ÿ“– Django docs on transaction.atomic

๐Ÿ“– Database transactions - How do they work?

utils.py ๐Ÿ™ˆ

I have written the following statements on slack today in a more project-related manner and thought this could make a small dev-tip post as well. Here goes ๐Ÿ’ช You basically donโ€™t ever want to create a file called utils.py, or helpers.py (you also donโ€™t want core.py or common.py but letโ€™s focus on utils for now). There are a few reasons to that: ๐Ÿ›  putting things into utils.py makes it a bag full of everything from simple datetime calculations to sophisticated validators and Class mixins ๐Ÿ›  utils is a nondescriptive name so nobody including you will know what has been put there in two weeks and more importantly where to look if someone has already written that util you are looking for (because they might put it in itโ€™s proper place) ๐Ÿ›  there is always a better place with a better name than utils (examples below)

How to approach the topic in an already operating project:

๐Ÿ›  itโ€™s good to move functions out from utils.py to their corresponding utils/proper_module.py each time you find an example and talk with your team about the whole purpose of moving that out

๐Ÿ›  if something is more generic than a simple module in your Django app / python module, you can move it higher in directory structure to the more generic utils/proper_module.py file. So generally speaking if you want utils in your project, just make it more structured. I prefer the name helpers more than utils, but no strings attached here - itโ€™s more important to keep consistency within the project, not looking at your own personal preferences.

Some examples of moves from utils.py:

๐ŸŒŸ amount_of_cat_food_validator โ†’ helpers/validators

๐ŸŒŸ CatPurrAwareMixin โ†’ helpers/mixins

๐ŸŒŸ calculate_days_until_christmas โ†’ helpers/datetime

๐ŸŒŸ a few topic related files can reside in their own directory, so form related modules (like new field types) will reside in helpers/forms/

๐Ÿ“– http://deviq.com/naming-things/

๐Ÿ“– https://github.com/erikras/react-redux-universal-hot-example/issues/808

๐Ÿ“– https://softwareengineering.stackexchange.com/a/175190

Moving from Solr 3.x to Solr 5.5.x

I have spent a few days trying to translate the configs from a Solr 3.x instance to Solr 5.5.x. The process was a bit devastating and consumed a lot of time. I did a lot of googling and browsing the documentation but I havenโ€™t found one place with all the information essential to get things going. This is a summary of what I have done and what might be helpful for others. Letโ€™s go!

image

๐Ÿš€ solr.xml

TL;DR Remove everything and use the default solr.xml found in $SOLR_HOME for the new version. There are multiple changes to the solr.xml file. Solr has introduced cores autodiscovery so all <core> definitions can be removed. More on cores autodiscovery in the next step. The server persistency flag persistent=โ€œ(true|false)โ€ has been deprecated and have to be removed. Cores with proper config files will be persisten by default.

๐Ÿš€ Cores

Copy old core(s) configuration directories into $SOLR_HOME. By default this will be the directory where solr.xml resides. For each core configuration copy the old directories to the $SOLR_HOME. The database data folder can stay the way they were in the old version, you will specify itโ€™s folder later on. Create an empty core.properties file in the main core directory. By default solr will only autodiscover the cores which have the file present. The file has to exist and can be empty. The created core will have the same name as the directory the file is in. If you want to change itโ€™s name or add other configuration options place them separately in each line in a simple key=value manner. It is explained quite well in the solr documentation. This is the place you should define the data directory for each core if it is not inside the core configuration directory. E.g. config with a core name and data directory defined:

    name=tobacco
    dataDir=/var/solr/db/core_name

In the conf/solr.xml file remove all occurances of enablePositionIncrements with arguments. The parameter has been deprecated in Solr 4 and canโ€™t be used. If you happen to need this setting you should take a look at this thread on Solr jira. Remove all definitions of fields for older versions of Solr. The fields are usually documented with comments about consistency with older versions. I had to remove all this from my schema.xml to get my instance working:

    <fieldType name="pint" class="solr.IntField"/>
    <fieldType name="plong" class="solr.LongField"/>
    <fieldType name="pfloat" class="solr.FloatField"/>
    <fieldType name="pdouble" class="solr.DoubleField"/>
    <fieldType name="pdate" class="solr.DateField" sortMissingLast="true"/>

    <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/>
    <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/>
    <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/>
    <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/>

Change luceneMatchVersion in solrconfig.xml file to the solr version you have installed. The core wonโ€™t work if the luceneMatchVersion wonโ€™t be changed toโ€ฆ match the current solr version. So for me it was 5.5.0. Change directory ownership. Check if all the cores directories belong to the solr user and change the ownership with chown -R solr: $SOLR_HOME

๐Ÿš€ First run & debugging

After making all these changes run your newly created instance or restart it if it was already running and go to admin to check that your cores were loaded. If they were not loaded and you canโ€™t seem to find any information on what has happened try loading the core manually from the admin. It should show you the errors with core initialization with the erroring files specified.

๐Ÿš€ HTTP requests to the Solr server

In the code making requests to Solr you will need to remove all ContentType definitions. Solr 5 doesnโ€™t accept passing ContentType in GET as well as POST requests.

๐ŸŽ‰ All set!

This is it, let me know if you had any other problems and I will happily add more information. It seems Solr is not a pleasant piece of software at times ๐Ÿ˜Ž