Native curated user list vs. custom feed performance disparity

[venteto]

I've noticed in just a very basic test that if I follow the same three "art bots" using a curated user list created directory in the Bluesky client web app, and then attempting to follow the same three accounts in custom feeds using both the SkyFeed third-party tool and using the AT Proto Python SDK, both of the custom feeds occasionally miss a few (or several) of the posts that how up in the curated user list.

I figure my Python attempt is crappy for sure, but it was surprising that the SkyFeed version was also missing posts.

Is there anything beyond crappy code by people trying to implement custom feeds that might account for such disparities? Do the "native" curated user lists have any sort of inherent accuracy / performance advantage, something that ensures they do not miss a sngle relevant post?

Even if I could do more filtering with a custom feed, beyond just constraining output to a list of accounts, it kind of discourages me from pursuing custom feeds any further if I think I could be missing posts without even being aware of it, particularly if I don't have a relevant user list to compare results with (in a case where constraining output to a particular set of accounts would not be desirable for a feed).

[emilyhunt]

Do you have a public copy of the Python code your feed is currently using? Did you have any luck with these suggestions?

I haven't found that the Astronomy feed using the current implementation relying on the ATProto Python SDK misses any posts. In theory, there shouldn't be any reason why it would miss any posts, as feeds by definition can check every post/event on the network.

[venteto]

I haven't made any progress, I gave up on it for awhile. I can't yet figure out how to add multiprocessing to my indexer (or if it would even work with the free tier shared virtual CPU on Fly.io), or whether it might be useful to possibly add async features.

All I've done so far is just try to remove the firehose indexing bits from the Flask feed server. This is the current repo.

I've tried running my separated indexer only locally, with both sqlite and postgres (was worried sqlite could be a bottleneck, but I also don't want to pay for postgres on the remote host either ... if sqlite could be a bottleneck, somehow I'd like to just stage any posts which need to be written to the database in some sort of queue until sqlite is ready to write the next one).

I've left the indexer running and then come back to look in the database, then look at the curated user lists, and have seen where posts that should've been added to the database were not, as expected, since I've not yet tried to do anything with multiprocessing, async, etc.

Moreover, the indexer has stopped working on different occasions, anywhere from 6 minutes after starting, to presumably a couple of hours after starting. The logger output in the terminal just stops, but the bash script starting the python module does not exit. The memory, cpu and time metrics in htop for 3 separate python processes stop changing. It's like it is just hanging, so I don't know how I would keep the indexer alive on the remote server at this point. I'm not sure if it would be a good idea to try to start it with gunicorn too, but in any event trying to start it with gunicorn on my laptop doesn't seem to do anything (but there are no gunicorn errors, and gunicorn does not die).

I went ahead and deployed to Fly again just to make sure the feeds server (in the responder directory in the repo) was still working, just sending the the sqlite database along with it, containing more than 4 records. However, while the feeds are visible in the Bluesky client, not every record in the database is being shown at either of the two feeds. That's a new wrinkle. I ssh into the remote container and confirm the records are in the database, I even rsync an updated copy into the remote container, but the feeds in the Bluesky client only display four of the records in total (3 for the museums feed, 1 for the artists feed).

[venteto]

I am closing this for now as it seems my Bluesky (bsky.app) client language settings are causing this problem