[Answer]-How to implement composition/agregation with NDB on GAE

0👍

It really does depend on the use case. For small numbers of items StructuredProperty and repeated properties may well be the best fit.

For large numbers of entities you will then look at setting the parent/ancestor in the Key for composition, and have a KeyProperty pointing to the primary entity in a many to one aggregation.

However the choice will also depend heavily on the actual use pattern as well. Then considerations of efficiency kick in.

The best I can suggest is consider carefully how you plan to use these relationships, how active are they (ie are they constantly changing, adding, deleting), do you need to see all members of the relation most of the time, or just subsets. These consideration may well require adjustments to the approach.

1👍

An alternative would be to use BigQuery. At first we used NDB, with a RawModel which stores individual, non-aggregated records, and an AggregateModel, which a stores the aggregate values.

The AggregateModel was updated every time a RawModel was created, which caused some inconsistency issues. In hindsight, properly using parent/ancestor keys as Tim suggested would’ve worked, but in the end we found BigQuery much more pleasant and intuitive to work with.

We just have cronjobs that run everyday to push RawModel to BigQuery and another to create the AggregateModel records with data fetched from BigQuery.

(Of course, this is only effective if you have lots of data to aggregate)

👤john2x

Leave a comment