How do you make a feed index posts made prior to the creation of the feed?

[venteto]

Curated user lists on Bluesky, Lists on Twitter, and even the SkyFeed tool all seem to be able to index posts that were made long before a list or custom feed were created, but thusfar my custom feeds made without SkyFeed only start indexing posts that are made after I launch my feed generator app.

[emilyhunt]

The relay - which feeds connect to to get their posts - is only really made for tracking live posts. I think that there's only about 3 days of post history on the relay, so the most you can do is set the cursor as old as you can - I assume if you set the cursor to a value of 0 on start up, then it should start at the oldest post on the relay and work its way through.

If you still want to include the old posts, then you could use the API to search for old posts by the accounts in your feed (it's just three, right? So should be easy) and add those posts to your database of posts manually.

[bnewbold]

We call this need "backfill". As @emilyhunt notes, the firehose only has a limited window of availability. Separately, each individual repository can be fetched, eg from the original PDS. The process of doing a full backfill is a careful dance of consuming from the firehose (to get recent content), enumerating all existing known accounts (eg, from the PLC registry, or from Relay), and fetching the full repo history from PDS instances. This process can take a day or more, and the end state is a complete network snapshot getting constant updates via the firehose.

This requires some non-trivial compute resources (mostly bandwidth and fast disk I/O), and many feed generators are only interested in the past couple days of content, so the feed gen template does not implement this backfill pattern. We have some golang packages (in the indigo repo) which are helpful for this process; I don't think we have TypeScript packages to help with this. There will be some short docs on this coming soon, but the current state is that it requires some non-trivial engineering work.