[Django]-Django-way for building a "News Feed" / "Status update" / "Activity Stream"

by

9👍

✅

I can think in two ways:

First, maybe you could make feeds for your models Assigments, PdfFiles, and Youtube link, and use the library feedparser to embed it in your news views, this is the easy way because you can define in the templates, the code for each kind of new activity.

The second thing I can think of is to make a class Activity:

class Activity(models.Model):
    date = models.DateTimeField(auto_now_add = True)
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey('content_type', 'object_id')

And through the signals make a new instance of Activity every time you have a new assigment or pdf upload or youtube link, and for each class make a method like render_to_html, in this way in your view, you can make a for over Activities and call the method render_to_html

👤diegueus9

17👍

Python is actually a great language for building Activity Streams and Newsfeeds. Tommaso and I have written the Stream Framework package.
https://github.com/tschellenbach/stream-framework
It is currently the most used Python solution for building newsfeeds. We’re also offering a hosted solution at https://getstream.io. The Django client is by far the easiest to get started with:
https://github.com/GetStream/stream-django and python can be found here (https://github.com/getstream/stream-python)

The templating part works like this

{% load stream_django %}

{% for activity in activities %}
    {% render_activity activity %}
{% endfor %}

This will render a template located in activity/tweet.html with the activity as context. For instance

{{ activity.actor.username }} said "{{ activity.object.body }} {{ activity.created_at|timesince }} ago"

The full docs are here:
https://github.com/GetStream/stream-django#templating

The Stream Framework allows you to build any type of newsfeed using either Redis or Cassandra. It’s build to scale and creates the individual newsfeeds using a fanout process.

Besides the Stream Framework (which I obviously prefer) there are many other solutions out there. A full list is available on django packages:
https://www.djangopackages.com/grids/g/activities/

Note that with newsfeeds there are a few scaling issues to keep in mind.
In general there are 3 common approaches:

Denormalization strategies

Pull
Most users start out this way. When you open the feed page you simply query the feeds from all the users you follow. If the users feeds are stored in memory this will keep on working for quite some time. Eventually it is quite hard to keep on using such as strategy though as you often have to query most of the nodes storing your user’s feeds.

Push
The push approach writes your activity to all of your followers feeds. Of course this means you’re wasting a ton of resources, but the end result is a pre computed feed per user. This approach (though initially not very efficient) scales nicely.

Combination
Some optimized systems use a combination of these two approaches. Also see the Yahoo paper on this topic.

Storage options

In terms of storing all this data the most common options are Redis, Cassandra and MongoDB.
Let’s quickly compare these:

Redis
Redis is Extremely easy to setup and maintain. It however only stores data in memory. This means you’ll have to optimize how you serialize data and maybe fallback to the database for less frequently queried data. Another problem is that it’s not trivial to add machines to your Redis cluster.

MongoDB
Mongo DB is used primarily by a few ruby projects and it’s also available as a backend for pump.io by e14n. I’ve personally never ran it in production so I can’t properly evaluate this option. There are however a lot of blogposts covering issues with the performance, scalability and maintainability of mongo.

Cassandra
Fashiolista, Instagram and Spotify are all using Cassandra. Our hosted solution also uses Cassandra as a backend. It’s extremely cost effective to operate and you can add more nodes with ease. The only problem is that it’s hard to setup and maintain.

Articles

In addition have a look at this high scalability post were we explain some of the design decisions involved:
http://highscalability.com/blog/2013/10/28/design-decisions-for-scaling-your-high-traffic-feeds.html

To learn more about feed design I highly recommend reading some of the articles which we based Feedly on:

👤Thierry

5👍

After more googling and one helpful keyword(“Activity“) that diegueus9 mentioned and that I haven’t thought before, I was able to find more relevant material.

First, two blog posts on how to build a tumbleblog using django using the ContentType framework:

After that, another post that gives suggestions on how to reduce the (1 + n) queries problem (which was one of my concerns initially, but I didn’t mention to avoid cluttering the question).

And finally an reusable Django app that has some of the feature that I needed and can be useful for further reference:

👤Tiago

3👍

Generic relations would be the way to go here. Just make sure to resolve the model yourself instead of joining against the update table.

👤Ignacio Vazquez-Abrams

Leave a comment