Metawarc 📇; Squares 🟦; xdg-ninja 🥷🏼
WARC is back on the menu (as promised) as well as some fun-and-games, and something to run over the holidays.
Also, there’s an extra longform resource over at the resurrected main blog that can help you talk to friends & fam about Meta’s Muse over the coming holidays. If you don’t know what Muse is or why it’s not great, I highly suggest giving it a look.
Metawarc

Metawarc (GH) began life in 2020 as a small, focused answer to an systemic problem in the web-archiving world: WARC files are a ace archival format, but they’re miserable to interrogate. Each WARC is essentially a giant sequential container of HTTP request/response pairs, so asking simple questions like “how many PDFs are in this collection?” or “give me every image capture that mentions ‘maine’” traditionally meant writing lovingly hand-crafted parser scripts. Metawarc’s original move was to be metadata-first: one pass through your WARCs, and you get a SQL database where the archive becomes queryable with plain SQL. That idea remains at the heart of the project, but the surrounding tools have grown up considerably.
In its current form, Metawarc indexes collections into a DuckDB catalog paired with Parquet sidecars. That’s a serious departure from the original SQLite-era design, and it’s a solid one for this workload: DuckDB is analytical, column-oriented, and fast at the aggregation-heavy questions folks actually ask of archives (“content types over time”, “domains by crawl”), and Parquet sidecars keep the heavy data (payloads, extracted text) in a portable format that plays well with the wider data-engineering ecosystem. The philosophy underneath stays the same as day one: source WARCs are immutable. Everything Metawarc builds is a derived index, so you can always blow the workspace away and rebuild from the originals.
The release that shipped in August, was a proper rewrite rather than a tiny version bump. Out went fragile filename-derived table names and whole-archive buffering; in came stable archive IDs, explicit workspace metadata, batched writes, atomic publication (so an interrupted index run can’t leave you with a half-written catalog), and a typed query layer shared by every interface. The CLI, the REST API, and the MCP server all speak the same query model, so behavior can’t drift between how a human and how an agent interrogate the same collection.
The CLI is also deliberately flat: index and ingest build the workspace, list-files/dump get payloads out, analyze does trend and distribution work, and serve/replay bring archives back to life as browsable websites. Beyond that there are three optional interfaces: a read-only REST API (API writes deliberately excluded), the local website replay mode, and, notably, said MCP server for LLM agents. Like it or not, we’re now a decade into an era where “query an archive with SQL” is being supplemented by “let an agent go explore the archive for me.” The docs even include a threat model for hostile archives (which I super appreciate). The implied paranoia is warranted as WARC files are 100% untrusted input that gets parsed at scale, and the team has spent recent releases hardening network binding and release dependencies accordingly.
Metawarc documentation lives in a Docusaurus site published at ruarxive.org/metawarc, with a task-oriented cookbook, full CLI reference, and an architecture section covering workspace schema and query model. Install via pip install metawarc on Python 3.11+, with metawarc[api], [replay], and [mcp] extras (or [all] if you want everything). Extra points if you use uv and virtual environs.
I made a small WARC file + CDX file that you can use to work with in metawarc. Running metawarc index 'warcs/dd2.warc.gz' --dbfile collection.db should result in 63 records, revision 1. The MIME mix should look like: 31 PNG, 8 woff2, 11 JS, 7 CSS, 2 HTML. metawarc serve --dbfile collection.db --port 8765 will expose the replay on localhost, which you can see from the screenshot in the section header.
Squares

To understand Squares, you have to understand the corner of the word-game universe it occupies. In early 2022, Wordle’s explosion sent everyone hunting for the next daily ritual, and one of the most successful second-generation heirs was a game called Squaredle, a browser game by Michael Giuffrida that fused Wordle’s daily-puzzle cadence with Boggle’s mechanics: a 4×4 letter grid where you trace connected paths (horizontal, vertical, diagonal) to spell words of four letters or more, no square reused within a word. Squaredle’s clever twist was making the puzzle completabl* (i.e., every grid hides a fixed list of common words, and you’re done when you’ve found them all, with progress hints, streaks, and puzzles that get harder across the week).
That formula proved pretty durable and, inevitably, portable/copyable. A cluster of “Squares”-branded daily word-hunt games now exists (squares.org, squaresgame.app, spellby’s version, and so on), all playing the same basic game: a grid of letters, a swipe-trace to spell connected words, a daily reset, streaks and hint systems.
The topic of today’s Drop presents itself as “a daily word game where you connect letters in a grid to form words,” one new puzzle per day, playable free in the browser (it has app versions as well). It has both a dark and light-themed UI, a mode dropdown set to “Classic” at puzzle #999, a found-words tracker (e.g., “Words found 0 / 32”), tabs to organize the hidden word list by order or by length, and native apps for both the App Store and Google Play.
From the site’s own “what’s new”:
- 19 languages, which pushes it well beyond the English-only default of most Wordle descendants
- Large mode, a 5×5 grid where words must be at least five letters (your screenshot shows exactly this board)
- Shapes, a new board shape every day, which breaks the genre’s rectangle monotony
- Practice mode, unlimited extra puzzles for folks who don’t want to wait for tomorrow
- Fairer streaks, where each mode keeps its own streak rather than one streak ruling them all
It’s more fun to play than I thought it would be, and I appreciate the challenge of having to make twisty paths to get to the longer words. I did a quick 3×3, today, to make the section heaer.
xdg-ninja

If you are even remotely like me, your home directory is a REDACTED mess. The XDG Base Directory Specification was supposed to fix this ages ago: dotfiles and program state belong in ~/.config, ~/.cache, ~/.local/share, and friends, not scattered willy-nilly across $HOME. But the spec is voluntary, and decades of software, from serious tools to weekend projects, do not necessarily respect any of it. The result is the tragicomedy every ls -la ~ reveals: .npm, .dbus, .gitconfig, plus a dozen mystery directories whose origin not a single one of us remembers.
xdg-ninja does one thing, and only one thing, super well: a POSIX shell script (bash, zsh, dash, whatever you like) that scans your $HOME against a large crowd-maintained knowledge base, and for every offending file or directory it recognizes, tells you two things: whether something can be moved somewhere spec-compliant, and precisely how, in rendered Markdown instructions. This knowledge base is built from three streams: the Arch Wiki’s exhaustive XDG_BASE_DIR page, the now-dormant antidot project, and years of crowdsourced contributions. The metadata is, essentially, the product, with the script being the helpful delivery mechanism.
The design splits cleanly into data and code:
- The knowledge base lives in
programs/, a directory of JSON files (JSON-schema’d since early 2024), one per program, each entry saying where a file lands and whether/how to relocate it. Anyone can write one in a text editor, and PRs back to the repo are encouraged. - The checker is
xdg-ninja.shitself, works great and is very grokable, requing only a POSIX shell plusjqandfind. No language runtime and no dependencies y’all don’t already have. The recent “performance improvements” merge even rewrote string substitution as POSIX-compliant shell, tightening the script to run super fast on big home directories. - OK, there is one extra complex bit: a Haskell CLI,
xdgnj, which exists purely to maintain the JSON configs (add/edit/preview/run). It’s why GitHub’s daft language bar shows Haskell as the majority language for what is functionally a shell-script project.
This might be something to run over the coming holiday breaks to get your home dir in order.
FIN
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:
- Mastodon via
@dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev - Bluesky via
<https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy>
☮️
Leave a Reply