[Django]-Django "migrate" consuming too much CPU

by

3👍

✅

Are you depending on migrate to run regularly? Because once the project is nearing and then entering production state, there shouldn’t be many migrations to run. Or do you mean that migrate takes this long, even if migrate --list shows that there is nothing to migrate?

Also, to know what Postgres is doing, you should set up logging of queries including their time. You can filter to log only longer running queries:
http://www.postgresql.org/docs/9.5/static/runtime-config-logging.html

Run those queries through the explain analyze sql command:

psql> EXPLAIN ANALYZE <complete query>;

http://www.postgresql.org/docs/9.5/static/using-explain.html

You need to provide the information you get from explain to get further help.

EDIT:

Also, you could try to squash migrations if you have a lot of migration files. I could imagine that Django works itself through all of them, one by one. So if you have many apps with many files depending on each other, you can imagine what happens.
https://docs.djangoproject.com/en/1.9/topics/migrations/#squashing-migrations

EDIT 2:

Moving this from the comment into the answer:

Does migrate --list also consume that much CPU? If not, then you could run it first, see whether there really is a need to migrate and only run migrate on those apps that have open migrations.

I think this would be the best. If you can profile in more detail, you might actually address the Django community for help. I could imagine that you have an interesting setup with which to find out how to tune the Django migrations to do less (actually unnecessary) work. But I don’t know the migrations code too much so I cannot tell.
But this also depends on how many apps we are talking about, and how many migration files. If you have less than 30 apps (including 3rd party), I think it should work fine and there is something else wrong (IMHO!).

Also, you have not shown the resource usage of your server. If the slowness is due to swapping/too much RAM usage you really might be able to boost it by supplying more RAM (to the process).

👤Risadinha

2👍

I believe migrations consume a lot, specially when having many models and many apps, more apps more dependencies more migrations complexity.

I would recommend starting a new instance which only run migration and shutdown after this. This way you web server could be reachable.

👤Mounir

0👍

This does not address the problem statement exactly but a part of it. I went through the documentation of AWS t2.micro and found that T2.Micro instances are designed to handle the CPU Burst of short intervals(~1 min) happening after reasonable long intervals. From the t2.micro documentation:

A CPU Credit provides the performance of a full CPU core for one minute. Traditional Amazon EC2 instance types provide fixed performance, while T2 instances provide a baseline level of CPU performance with the ability to burst above that baseline level. The baseline performance and ability to burst are governed by CPU credits.

Running migration’s shouldn’t be an issue given this ^ even if it is consuming 100% of the CPU. We investigated more and found that there were crons running on the server which were not supposed to be.

👤Shubham

Leave a comment