Paginate/limit comments from /find endpoint

[CodeWithOz]

Is there a built-in way to paginate when fetching comments from the /find endpoint? For instance if a particular thread has 1000 replies and I only want to get them in batches of 50 each time. Do I need my own custom code in the backend to implement this?

I can see the since param but ideally I'd like to also specify a maximum number of replies to include in the response.

Thanks!

[umputun]

As of now sinceis the only way to limit the response.

[ingria]

Would've been a great feature. I have some threads with over 10k comments, and there's no way to load them page by page or with infinite loader. Loading everything at once is not an option, because of a) large amount of traffic b) heavy cpu load while parsing JSON.

Example implementation: Disqus

[paskal]

I'll look into it at some point, as it seems like a must-have for bigger installations.

[JayDi85]

It's a critical feature 💪

[paskal]

I just wanted to let you know that I plan to work on this issue shortly. Current endpoint:

// GET /find?site=siteID&url=post-url&format=[tree|plain]&sort=[+/-time|+/-score|+/-controversy]&view=[user|all]&since=unix_ts_msec
// find comments for the given post. Returns in tree or plain formats, sorted

What I plan to add:

// GET /find?<...>&limit=100&offset_id={id}
// When `limit` is set, first {limit} comments are returned. When `offset_id` is set, comments are returned starting
// after the comment with the given id.

We will unlikely have support for this feature in the frontend right away, but it will at least unblock anyone savvy enough to alter their frontend or willing to spend time updating the upstream for everyone.

@ingria, how long does it take for Remark42 to generate an answer for a post with 10,000 comments? If the time is significant, I will return the first N comments as-is. Suppose the generation time is not that big. In that case, I might think about returning comments level by level, Reddit-style, so that users would be able to see all the top-level conversations and further comments will be available by clicking "show more", but that approach would require getting them all into memory and filtering then instead of just returning whatever DB gave us.

edit: changed limit_start to offset_id.

[ingria]

how long does it take for Remark42 to generate an answer for a post with 10,000 comments

I don’t actually remember, that was quite a long time ago.

My main concern was not about the remark42 speed – getting 10k rows is not a problem for a modern DB on a modern hardware. The problem here is unnecessary network traffic (which, by the way, is uncachable), along with cpu and memory consumption (parsing json and creating dom nodes) on user device.

I might think about returning comments level by level, Reddit-style

That would be perfect. I think Reddit comments have the best possible UX.

// GET /find?<...>&limit=100&limit_start={id}

I suggest calling the paramoffset_id instead of limit_start, that would be more intuitive IMO

[paskal]

#1699 adds backend support for solving that issue. If someone here has the skills to do it, feel free to review and leave your comments.