SummarAIse Proust Socials

Thanks to (*gestures hands opening wide in front of himself*), the state of modern discourse on both Bluesky and Mastodon is definitely trending “ugh”. Most recently, it’s been the SCOTUS disaster, POTUS race, the climate crisis we’re in, and a smattering of other topics that all trigger 🤯, with there also being very little practical things an individual can do in the moment to change anything.

Rather than try to manage lists, micromanage filters, shunt the “must reads” to RSS, or just ignore socmed outright, I decided to do something useful and have our AI overlords give me a summary of that I “missed” while not doomscrolling.

I’ll show both how I did this with Perplexity, and show a bit of a start on hacking on a local LLM/GPT — Gemma v2 27b Instruct to do something similar.

SummarAIse With Perplexity

I’m only covering Mastodon in today’s post, and to get the home timeline I’m using madonctl. We’re using “plain” mode for the output as there’s a nasty bug in this semi-old CLI that borks the JSON and YAML output (it messes up post author attribution).

This is the CLI command I’m using:

$ madonctl timeline -k 300 -l 300 -o plain home > status.txt

and, this is the prompt:

The attached input file is a series of Mastodon posts.

Each post from the input has this format:

- Status ID: `status_post_id_number`
 From: `user@instance` (`user_handle`)
 Timestamp: `status_post_iso_timestamp`
 Contents: `contents_of_status_post`
NOTE that the status post `contents_of_status_post` may continue on multiple lines without spaces in front
 URL: `url_of_status_post`

Create a detailed summary of what I missed, especially >if more than one Status talks about a similar topic.

Then, pick at least one or two profound or >well-articulated posts and put them in the output with `user_handle` and `contents_of_status_post`.

Then, create a separate summary of posts with `#RStats`, `#rstats`, “#Golang”, “#golang”, “#DataBreach”, or “#databreach” hashtags.

Finally, include a sorted bullet list of all the hashtags used across all the posts in the attached input file.

After pasting the prompt into Perplexity, I attach status.txt to the question and have it do some magic. You can see one incantation of that here.

Perplexity automates the painful work of chunking the input to fit the context window and has — very obviously — been configured quite well to deal with topic summarisation.

Note that it also generated a couple Python scripts to deal with some of my “then“s.

It definitely impressed me, and I’ve run it quite a bit and verified that the summaries are accurate. While it does miss some things I might have wanted to look at, summarising is — essentially — editing, and some things will remain on the cutting room floor.

If you’re curious how ChatGPT fared: here’s the output for the same input file/prompt.

Local SummarAIsing With llamafile

Google recently released their Gemma v2 models, and Justine was kind enough to make llamafiles for them. Both versions of the model run stupid fast on macOS with Apple Silicon (I did all my hacking on an Apple M1 Max laptop). I usually cannot tolerate using local LLMs due to the speed issue.

I’m not about to try to replicate all of what Perplexity did for this post (I’m hacking on the local version over time), so I settled on a per-user summary.

I started up the Gemma llamafile in server mode with:

$ ./gemma --temp 0 -c 0 --threads 10 --host 127.0.0.1 --port 4141

The, use the following R code to pull the last ~300 posts from my home feed, filter out ones with little or no content, grouped the content text by user, and fed it to the completion API endpoint:

library(httr)
library(stringi)
library(tidyverse)

home <- vector("list", 8)

next_url <- "https://mastodon.social/api/v1/timelines/home?limit=40"

for (i in 1:8) {

  writeLines(as.character(i))

  httr::GET(
    url = next_url,
    httr::add_headers(
      `Authorization` = sprintf("Bearer %s", Sys.getenv("MASTODON_TOKEN"))
    )
  ) -> res 
  
  res |> 
    httr::content(as = "text") |> 
    jsonlite::fromJSON() -> posts

  home[i] <- list(posts)
  
  res$headers$link |> 
    stri_split_fixed(">") |> 
    unlist() |> 
    stri_replace_first_fixed("<", "") |> 
    getElement(1) -> next_url
  
}

home |> 
  bind_rows() |> 
  filter(
    nchar(content) >= 30
  ) -> xdf

data.frame(
  username = xdf$account$username,
  content = map_chr(xdf$content, \(.x) {
    read_html(.x) |> 
    html_text() |> 
    paste0(collapse = " ")
  })
) |> 
  arrange(username) -> user_posts

prompt_tmpl <- r"(<start_of_turn>User
The input at the end is a series of Mastodon posts from user %s, with each post on a separate line. Give me a very high level overview of topics they posted about while I was away.

Use the following format — it is very important to include the `###` and username — and do not provide extra commentary:

### %s

- summary bullet (if more than one topic, add more bullets)

%s<end_of_turn>
<start_of_turn>Llama 
)"

users <- unique(user_posts$username)

users |> 
  map(\(.user) {
    writeLines(.user)
  
    httr::POST(
      url = "http://127.0.0.1:4141/completion",
      encode = "json",
      body = list(
        cache_prompt = TRUE,
        stop = c("</s>", "Llama", "User"),
        temperature = 0.7,
        prompt = sprintf(
          fmt = prompt_tmpl,
          .user,
          .user,
          user_posts |> 
            filter(
              username == .user
            ) |> 
            pull(content) |> 
            paste0(collapse = "\n")
        )
      )
    ) -> res
    
    httr::content(res) |> 
      getElement("content")

}, .progress = TRUE) -> out

unlist(out) |> 
  stri_replace_all_fixed("<end_of_turn>", "") |> 
  cat(
    sep = "\n------\n\n",
    file = "/tmp/briefing.md"
  )

While it is noticeably slower than Perplexity, it’s just fast enough for this task to make using it tolerable.

The prompt definitely needs some tweaking, but you can see markdown output rendered to HTML over here.

FIN

I’ll be migrating the local version to behave more like the Perplexity version, and will post and update when I do.

If you made it this far (thanks!), why did a Very British TV Show spell summarise with a Zed?

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev ☮️