Fix initial cursor

.env.example

@@ -0,0 +1,25 @@
+# Whichever port you want to run this on 
+FEEDGEN_PORT=3000
+
+# Change this to use a different bind address
+FEEDGEN_LISTENHOST="localhost"
+
+# Set to something like db.sqlite to store persistently
+FEEDGEN_SQLITE_LOCATION=":memory:"
+
+# Don't change unless you're working in a different environment than the primary Bluesky network
+FEEDGEN_SUBSCRIPTION_ENDPOINT="wss://bsky.social"
+
+# Set this to the hostname that you intend to run the service at
+FEEDGEN_HOSTNAME="example.com"
+
+# Set this to the DID of the account you'll use to publish the feed
+# You can find your accounts DID by going to
+# https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=${YOUR_HANDLE}
+FEEDGEN_PUBLISHER_DID="did:plc:abcde...."
+
+# Only use this if you want a service did different from did:web
+# FEEDGEN_SERVICE_DID="did:plc:abcde..."
+
+# Delay between reconnect attempts to the firehose subscription endpoint (in milliseconds)
+FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY=3000

.gitignore

@@ -1 +1,130 @@
-node_modules
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+.pnpm-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+*.lcov
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# Snowpack dependency directory (https://snowpack.dev/)
+web_modules/
+
+# TypeScript cache
+*.tsbuildinfo
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional stylelint cache
+.stylelintcache
+
+# Microbundle cache
+.rpt2_cache/
+.rts2_cache_cjs/
+.rts2_cache_es/
+.rts2_cache_umd/
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variable files
+.env
+.env.development.local
+.env.test.local
+.env.production.local
+.env.local
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+.parcel-cache
+
+# Next.js build output
+.next
+out
+
+# Nuxt.js build / generate output
+.nuxt
+dist
+
+# Gatsby files
+.cache/
+# Comment in the public line in if your project uses Gatsby and not Next.js
+# https://nextjs.org/blog/next-9-1#public-directory-support
+# public
+
+# vuepress build output
+.vuepress/dist
+
+# vuepress v2.x temp and cache directory
+.temp
+.cache
+
+# Docusaurus cache and generated files
+.docusaurus
+
+# Serverless directories
+.serverless/
+
+# FuseBox cache
+.fusebox/
+
+# DynamoDB Local files
+.dynamodb/
+
+# TernJS port file
+.tern-port
+
+# Stores VSCode versions used for testing VSCode extensions
+.vscode-test
+
+# yarn v2
+.yarn/cache
+.yarn/unplugged
+.yarn/build-state.yml
+.yarn/install-state.gz
+.pnp.*

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Bluesky PBLLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,56 +1,67 @@
 # ATProto Feed Generator
 
-🚧 Work in Progress 🚧 
-
-We are actively developing Feed Generator integration into the Bluesky PDS. Though we are reasonably confident about the general shape and interfaces laid out here, these interfaces and implementation details _are_ subject to change. 
-
-In the meantime, we've put together this starter kit for devs. It doesn't do everything, but it should be enough to get you familiar with the system & started building!
+This is a starter kit for creating ATProto Feed Generators. It's not feature complete, but should give you a good starting ground off of which to build and deploy a feed.
 
 ## Overview
 
-Feed Generators are services that provide custom algorithms to users through the AT protocol.
+Feed Generators are services that provide custom algorithms to users through the AT Protocol.
 
-They work very simply: the server receives a request from a user's server and returns a list of [post URIs](https://atproto.com/specs/at-uri-scheme) with some optional metadata attached. Those posts are then hydrated into full views by the requesting server and sent back to the client. This route is described in the [`com.atproto.feed.getFeedSkeleton` lexicon](https://github.com/bluesky-social/atproto/blob/custom-feeds/lexicons/app/bsky/feed/getFeedSkeleton.json).
+They work very simply: the server receives a request from a user's server and returns a list of [post URIs](https://atproto.com/specs/at-uri-scheme) with some optional metadata attached. Those posts are then hydrated into full views by the requesting server and sent back to the client. This route is described in the [`app.bsky.feed.getFeedSkeleton` lexicon](https://atproto.com/lexicons/app-bsky-feed#appbskyfeedgetfeedskeleton).
 
-A Feed Generator service can host one or more algorithms. The service itself is identified by DID, however each algorithm that it hosts is declared by a record in the repo of the account that created it. For instance feeds offered by Bluesky will likely be declared in `@bsky.app`'s repo. Therefore, a given algorithm is identified by the at-uri of the declaration record. This declaration record includes a pointer to the service's DID along with some profile information for the feed.
+A Feed Generator service can host one or more algorithms. The service itself is identified by DID, while each algorithm that it hosts is declared by a record in the repo of the account that created it. For instance, feeds offered by Bluesky will likely be declared in `@bsky.app`'s repo. Therefore, a given algorithm is identified by the at-uri of the declaration record. This declaration record includes a pointer to the service's DID along with some profile information for the feed.
 
 The general flow of providing a custom algorithm to a user is as follows:
 - A user requests a feed from their server (PDS) using the at-uri of the declared feed
 - The PDS resolves the at-uri and finds the DID doc of the Feed Generator
 - The PDS sends a `getFeedSkeleton` request to the service endpoint declared in the Feed Generator's DID doc
   - This request is authenticated by a JWT signed by the user's repo signing key
 - The Feed Generator returns a skeleton of the feed to the user's PDS
-- The PDS hydrates the feed (user info, post contents, aggregates, etc)
-  - In the future, the PDS will hydrate the feed with the help of an App View, but for now the PDS handles hydration itself
+- The PDS hydrates the feed (user info, post contents, aggregates, etc.)
+  - In the future, the PDS will hydrate the feed with the help of an App View, but for now, the PDS handles hydration itself
 - The PDS returns the hydrated feed to the user
 
-To the user this should feel like visiting a page in the app. Once they subscribe, it will appear in their home interface as one of their available feeds.
+For users, this should feel like visiting a page in the app. Once they subscribe to a custom algorithm, it will appear in their home interface as one of their available feeds.
 
 ## Getting Started
 
-We've setup this simple server with sqlite to store & query data. Feel free to switch this out for whichever database you prefer.
+We've set up this simple server with SQLite to store and query data. Feel free to switch this out for whichever database you prefer.
 
-Next you will need to do two things:
+Next, you will need to do two things:
 
-- Implement indexing logic in `src/subscription.ts`. 
+1. Implement indexing logic in `src/subscription.ts`. 
+
+   This will subscribe to the repo subscription stream on startup, parse events and index them according to your provided logic.
 
-This will subscribe to the repo subscription stream on startup, parse events & index them according to your provided logic.
+2. Implement feed generation logic in `src/algos`
 
-- Implement feed generation logic in `src/feed-generation.ts`
+   For inspiration, we've provided a very simple feed algorithm (`whats-alf`) that returns all posts related to the titular character of the TV show ALF. 
 
-The types are in place and you will just need to return something that satisfies the `SkeletonFeedPost[]` type
-
-For inspiration, we've provided a very simple feed algorithm ("whats alf") that returns all posts related to the titular character of the TV show ALF.
+   You can either edit it or add another algorithm alongside it. The types are in place, and you will just need to return something that satisfies the `SkeletonFeedPost[]` type.
 
 We've taken care of setting this server up with a did:web. However, you're free to switch this out for did:plc if you like - you may want to if you expect this Feed Generator to be long-standing and possibly migrating domains.
 
-Once the custom algorithms feature launches, you'll be able to publish your feed in-app by providing the DID of your service.
+### Deploying your feed
+Your feed will need to be accessible at the value supplied to the `FEEDGEN_HOSTNAME` environment variable.
+
+The service must be set up to respond to HTTPS queries over port 443.
+
+### Publishing your feed
+
+To publish your feed, go to the script at `scripts/publishFeedGen.ts` and fill in the variables at the top. Examples are included, and some are optional. To publish your feed generator, simply run `yarn publishFeed`.
+
+To update your feed's display data (name, avatar, description, etc.), just update the relevant variables and re-run the script.
+
+After successfully running the script, you should be able to see your feed from within the app, as well as share it by embedding a link in a post (similar to a quote post).
+
+## Running the Server
+
+Install dependencies with `yarn` and then run the server with `yarn start`. This will start the server on port 3000, or what's defined in `.env`. You can then watch the firehose output in the console and access the output of the default custom ALF feed at [http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf](http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf).
 
 ## Some Details
 
 ### Skeleton Metadata
 
-The skeleton that a Feed Generator puts together is, in its simplest form, a list of post uris.
+The skeleton that a Feed Generator puts together is, in its simplest form, a list of post URIs.
 
 ```ts
 [
@@ -60,18 +71,12 @@ The skeleton that a Feed Generator puts together is, in its simplest form, a lis
 ]
 ```
 
-However, we include two locations to attach some additional context. Here is the full schema:
+However, we include an additional location to attach some context. Here is the full schema:
 
 ```ts
 type SkeletonItem = {
   post: string // post URI
-
-  // optional metadata about the thread that this post is in reply to
-  replyTo?: {
-    root: string, // reply root URI
-    parent: string, // reply parent URI
-  }
-
+
   // optional reason for inclusion in the feed
   // (generally to be displayed in client)
   reason?: Reason
@@ -81,9 +86,8 @@ type SkeletonItem = {
 type Reason = ReasonRepost
 
 type ReasonRepost = {
-  $type: @TODO
-  by: string // the did of the reposting user
-  indexedAt: string // the time that the repost took place
+  $type: 'app.bsky.feed.defs#skeletonReasonRepost'
+  repost: string // repost URI
 }
 ```
 
@@ -111,12 +115,12 @@ const payload = {
 }
 ```
 
-We provide utilities for verifying user JWTs in the `@atproto/xrpc-server` package.
+We provide utilities for verifying user JWTs in the `@atproto/xrpc-server` package, and you can see them in action in `src/auth.ts`.
 
 ### Pagination
-You'll notice that the `getFeedSkeleton` method returns a `cursor` in its response & takes a `cursor` param as input.
+You'll notice that the `getFeedSkeleton` method returns a `cursor` in its response and takes a `cursor` param as input.
 
-This cursor is treated as an opaque value & fully at the Feed Generator's discretion. It is simply pased through he PDS directly to & from the client.
+This cursor is treated as an opaque value and fully at the Feed Generator's discretion. It is simply passed through the PDS directly to and from the client.
 
 We strongly encourage that the cursor be _unique per feed item_ to prevent unexpected behavior in pagination.
 
@@ -127,17 +131,22 @@ We recommend, for instance, a compound cursor with a timestamp + a CID:
 
 How a feed generator fulfills the `getFeedSkeleton` request is completely at their discretion. At the simplest end, a Feed Generator could supply a "feed" that only contains some hardcoded posts.
 
-For most usecases, we recommend subscribing to the firehose at `com.atproto.sync.subscribeRepos`. This websocket will send you every record that is published on the network. Since Feed Generators do not need to provide hydrated posts, you can index as much or as little of the firehose as necessary.
+For most use cases, we recommend subscribing to the firehose at `com.atproto.sync.subscribeRepos`. This websocket will send you every record that is published on the network. Since Feed Generators do not need to provide hydrated posts, you can index as much or as little of the firehose as necessary.
 
 Depending on your algorithm, you likely do not need to keep posts around for long. Unless your algorithm is intended to provide "posts you missed" or something similar, you can likely garbage collect any data that is older than 48 hours.
 
 Some examples:
 
 ### Reimplementing What's Hot
-To reimplement "What's Hot", you may subscribe to the firehose & filter for all posts & likes (ignoring profiles/reposts/follows/etc). You would keep a running tally of likes per post & when a PDS requests a feed, you would send the most recent posts that pass some threshold of likes.
+To reimplement "What's Hot", you may subscribe to the firehose and filter for all posts and likes (ignoring profiles/reposts/follows/etc.). You would keep a running tally of likes per post and when a PDS requests a feed, you would send the most recent posts that pass some threshold of likes.
 
 ### A Community Feed
-You might create a feed for a given community by compiling a list of DIDs within that community & filtering the firehose for all posts from users within that list.
+You might create a feed for a given community by compiling a list of DIDs within that community and filtering the firehose for all posts from users within that list.
 
 ### A Topical Feed
-To implement a topical feed, you might filter the algorithm for posts and pass the post text through some filtering mechanism (an LLM, a keyword matcher, etc) that filters for the topic of your choice.
+To implement a topical feed, you might filter the algorithm for posts and pass the post text through some filtering mechanism (an LLM, a keyword matcher, etc.) that filters for the topic of your choice.
+
+## Community Feed Generator Templates
+
+- [Python](https://github.com/MarshalX/bluesky-feed-generator) - [@MarshalX](https://github.com/MarshalX)
+- [Ruby](https://github.com/mackuba/bluesky-feeds-rb) - [@mackuba](https://github.com/mackuba)

package.json

@@ -7,20 +7,27 @@
   "author": "dholms <dtholmgren@gmail.com>",
   "license": "MIT",
   "scripts": {
-    "start": "ts-node src/index.ts"
+    "publishFeed": "ts-node scripts/publishFeedGen.ts",
+    "start": "ts-node src/index.ts",
+    "build": "tsc"
   },
   "dependencies": {
+    "@atproto/api": "^0.3.7",
+    "@atproto/did-resolver": "^0.1.0",
+    "@atproto/lexicon": "^0.1.0",
     "@atproto/repo": "^0.1.0",
     "@atproto/uri": "^0.0.2",
-    "@atproto/xrpc-server": "^0.1.0",
+    "@atproto/xrpc-server": "^0.2.0",
     "better-sqlite3": "^8.3.0",
+    "dotenv": "^16.0.3",
     "express": "^4.18.2",
-    "kysely": "^0.22.0"
+    "kysely": "^0.22.0",
+    "multiformats": "^9.9.0"
   },
   "devDependencies": {
     "@types/better-sqlite3": "^7.6.4",
     "@types/express": "^4.17.17",
-    "@types/node": "^20.1.1",
+    "@types/node": "^20.1.2",
     "ts-node": "^10.9.1",
     "typescript": "^5.0.4"
   }