DIY Deno-Based Bluesky Thread Unroller

The Nazistack reimbursements are still in limbo (I’m going to try to talk to or at least tele-chat with an actual Substack proto-human this week to get this resolved), so everybody gets a Bonus Drop until that’s fixed.

Bluesky is going GA “this month” (I have two invite codes at the end of this Drop if you want to jump the line), and in preparation for the “big time”, they’ve also made it possible to view individual posts/threads sans-authentication.

So, today, we’re going to write some Deno-flavoured JavaScript and create both a CLI thread unroller as well as stand up a free thread unroller REST API (using virtually the same code).

Bluesky Threads & JSON

If you hit a given Bluesky post/thread, such as this one with Developer Tools open in an inconito session you can see a few useful items:

• The “view source” version of the HTML has a handy <noscript> section with the bare minimum post and author information.
• One XHR request for the JSON of the author info: <https://public.api.bsky.app/xrpc/com.atproto.identity.resolveHandle?handle=gregsargent.bsky.social>
• Another XHR request for the JSON of the entire thread <https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread?uri=at%3A%2F%2Fdid%3Aplc%3A2amnkge5a6hplfwyesmxqkfl%2Fapp.bsky.feed.post%2F3khjvpz5cfc2e>

Please check those out before continuing.

“Um…hrbrmstr…that thread JSON is complete. Why do we need an unroller?”

Glad you asked!

That thread JSON is absolutely complete. Too complete. It has the author’s main thread, then all the nested and layered-in replies from the author and folks engaging with the content. The majority of references are also in “raw” Blueskly/ATProto form, meaning we need to do some work to turn any attached JPEG identifiers into fully href’d <img src=…> equivalents, and fish out any truncated URLs.

We’ll get to some of that in a moment.

Why Deno?

A thread unroller will be useful in CLI form, and we can turn Deno scripts into executables with a single command. An unroller API endpoint would also be pretty handy. Both idioms could use with a bit of caching, and Deno (unstable) has a built-in key/value store that “scales to the cloud” without you having to know the gory details.

I’m going to ask you to head to Deno Deploy and create a free account. You will not need to spend any coin to work with the resources in this Drop and deploy an API service to Deno. If you recall a previous Drop where we created our own edge service, the code will work there, too, minus the Deno key/value caching part (I need to modify the Supabase Docker setup to run Deno in “unstable” mode to make that work and just haven’t had time).

CLI Time

You can find the code for the CLI version of the Bluesky thread unroller on Codeberg. It has JDDoc comments and is also lightly annotated, so we won’t spend a ton of time working through some basic HTML and JSON parsing. We will focus (a bit) on this function which I’ve also put below:

function extractReplies(thread, authorDid) {
  const replies = [];
  const cid = thread.post.cid

  function traverseReplies(replyArray, pcid) {
    if (!Array.isArray(replyArray)) {
      return;
    }
    for (const reply of replyArray) {

      if (reply.post.author.did === authorDid) {
        if (reply.post.record.reply.root.cid == cid) {
          if (reply.post.record.reply.parent.cid == pcid) {
            replies.push({
              uri: reply.post.uri,
              text: reply.post.record.text,
              embed: extractEmbeds(reply.post.record.embed, authorDid),
              facets: extractFacets(reply.post.record.facets)
            });
            if (reply.replies && reply.replies.length > 0) {
              traverseReplies(reply.replies, reply.post.cid);
            }
          }
        }
      }
    }
  }

  traverseReplies(thread.replies, thread.post.cid);
  return replies;
}

This function traverses the nested replies structures of the thread and uses the metadata of the original post ID and author ID, plus subsequent direct reply IDs, to ensure we’re just seeing what the author has directly contributed. If you need more, then the full, ugly, nested JSON is what you should work with.

As we traverse the replies, we extract the “embeds” — which are, for now, just images — and facets, which can be hashtags or URLs (we only focus on URLs).

Search the file for kv to see three touch points required to add a caching layer (which will speed-up subsequent requests for the same thread).

Take a peek at the included justfile to see how to call it on the CLI

REST API Time

The “serverless” REST API version is mostly the same, except that we’re using Hono to deal with the HTTP request handling. In the CLI we just used Deno’s access to the CLI parameters. In this server version, we setup the Hono routing and wire it up to Deno.

You can run/test it locally via (Deno will choose 8000 or an unused port, so sub that out if necessary):

$ deno run --allow-all --unstable index.mjs  
$ curl "http://localhost:8000/bskyunroll?postURL=https://bsky.app/profile/gregsargent.bsky.social/post/3khjvpz5cfc2e"

Deno Deploy

I’m making the assumption you followed through on the “create a Deno account and perform a test deploy”. If so, you can do:

$ deployctl deploy -p SOMEUSEFULHOSTPREFIX --entrypoint=index.mjs

and Deno will let you know where to go next. You can test mine out via:

$ curl "https://bskyunroll.deno.dev/bskyunroll?postURL=https://bsky.app/profile/gregsargent.bsky.social/post/3khjvpz5cfc2e"

In the dashboard for that project, you can find instructions on how to wire-up the global key/value store (if you want to).

Making It Real

The new “unrolled” JSON is still “just JSON”, so I whipped up an Observable Notebook that shows one way to visualize the thread.

Make sure to delete the project if you do not want to accidentally move from free to “paid” due to this project.

I have an alternate version of this running on my self-hosted “edge” service, since I’ll likely shut down the Deno Deploy one if it starts to cost real money:

$ curl "https://edge.hrbrmstr.dev/bskyunroll?postURL=https://bsky.app/profile/gregsargent.bsky.social/post/3khjvpz5cfc2e"

FIN

Here are the aforementioned Bluesky codes (first-come/first-served):

bsky-social-zurbs-csz73
bsky-social-r737p-bkppg

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev ☮️