[Answer]-Insert and categorize a Numpy array into a Django modelled database EAV schema

by

1👍

✅

Anzel answered the iterating problem here – use the Pandas to_csv() function. My dictionary syntax was also wrong. My final solution was therefore:

# populate a Dictionary to find dimension_ids for category labels
parent_dimension_age = Dimension.objects.get(name='Age')
parent_dimension_income = Dimension.objects.get(name='Income')
dims_age = dict([ (d.name, d.id) for d in Dimension.objects.filter(parent_id=parent_dimension_age.id) ])
dims_income = dict([ (d.name, d.id) for d in Dimension.objects.filter(parent_id=parent_dimension_income.id) ])

# Retrieves a row at a time into a comma delimited string
for line in pandas_pivottable.to_csv(header=False, index=True, sep='\t').split('\n'):
    if line:
        # row[0] = income category, row[1] = age category, row[2] = age, row[3] = income
        row = line.split('\t')
        entity = Entity(name='data pivot row', dataset_id=dataset.id)
        entity.save()
        # dims_age.get(row[1]) gets the ID for the category whose name matches the contents of row[1]
        age_val = FloatValue(value=row[2], entity_id=entity.id, attribute_id=attrib_age.id, dimension_id=dims_age.get(row[1]))
        age_val.save()
        income_val = FloatValue(value=row[3], entity_id=entity.id, attribute_id=attrib_income.id, dimension_id=dims_income.get(row[0]))
        income_val.save()

For more on the Entity-Attribute-Value (EAV) schema see the Wikipedia page, (if you are considering it see the Django-EAV extension). In the next iteration of this project however, I will be replacing it with postgresql’s new JSONB type. This promises to make the data more legible and perform equally or better.

👤Chris

Leave a comment