[Django]-Sharding a Django Project

In the past I’ve done something similar using Postgresql Table Partitioning, however this merely splits a table up in the same DB. This is helpful in reducing table search time. This is also nice because you don’t need to modify your django code much. (Make sure you perform queries with the fields you’re using for constraints).

But it’s not sharding.

If you haven’t seen it yet, you should check out Sharding Postgres with Instagram.

I agree with @DanielRoseman. Also, how many is too many rows. If you are careful with indexing, you can handle a lot of rows with no performance problems. Keep your indexed values small (ints). I’ve got tables in excess of 400 million rows that produce sub-second responses even when joining with other many million row tables.

It might make more sense to break user up into multiple tables so that the user object has a core of commonly used things and then the “profile” info lives elsewhere (std Django setup). Copies would be a small table referencing books which has the bulk of the data. Considering how much ram you can put into a DB server these days, sharding before you have too seems wrong.