[Fixed]-Building an API

Easiest way to do in Django is using Django Endless Pagination

I think the better way to apply it is setting limit in your query. For example, If you have 1000 of records in your database, then retrieving all data at once takes time. So, if a user search a word ‘apple’, you initially send the database request with limit 10. And, you can set pagination or scroll feature at your front-end. If the user click next page or scroll your page, you can again send the database request with another limit 10 so that the database read action will not take more time to read the limited data.

From your explanation

We’re pulling our data from multiple sources with each user search.
Being directly connected to the scrapers for those sources, we display
the content as each scraper completes content retrieval. I was
originally looking to mimic this in the API, which is obviously quite
different from traditional pagination – hope this clarifies.

So you in your API, you want to

1. take query from user
2. initiate live scrapers
3. get back the data to the user when scrapers finish the job !
   (correct me if im wrong)

My Answer

This might feel little complicated, but this is the best one I can think of.

1. When user submits the query:
1. Initiate the live scrapers in to celery queue (take care of the priority).
2. Once the queue is finished, get back to the user with the information you have via sockets(this is how facebook or any website sends users notifications`. But in your case you will send the results html data in the socket.
3. Since you will have the data already, moved into the db as you scraped, you will can paginate it like normal db.

But this approach gives you a lag of a few seconds or a minute to reply back to the user, meanwhile you keep theuser busy with something on the UI front.