Honey, I Shrunk The MMDBs; Font Revival;🔒🦆
I’m in travel mode this week, and very much so today (Monday). I’m also unusually awake on a plan and have ~four more hours to kill and oddly decent in-flight Wi-Fi, so we have time for a Drop!
No theme, but I think Drop readers will like all three of today’s resources.
Honey, I Shrunk The MMDBs
An MMDB file (used in IP geo/ASN mappings/lookups) is composed of two parts: a binary trie over the bits of an IP address (left on 0 and right on 1) whose leaf holds a pointer; and a data section where the records (geo/ASN/etc.) live. The format’s designers gave the data section a pointer type, so a repeated record is written once and referenced from many leaves.
Sadly, the trie bit did not receive the same love and attention. Every node is written out as two records, one per child, and a standard writer emits every node it visits without asking whether it already wrote an identical one. So, if two /24s in different parts of the address space that return the same answer for all 256 addresses have byte-identical subtrees beneath them, they both get stored. As you might imagine, this is a tad wasteful, which is precisely what the keen folks over at IPInfo thought.
Their fix was to process the tree from the bottom-up. To grok why, imagine you are organizing a library. If you follow the OG MMDB idiom’ (top-down) you build a shelf (Parent), give it a label “Science” (Key), and then put books (Children) on it. However, if you use a bottom-up approach, you’d gather all the books and make sure they are the correct (authoritative/canonical) editions, then you build the shelf and give it a label (keyed) based on which books you ended up with.
In the case of MMDBs, a smarter way to work is to process the tree from the leaves up, so children are canonical before their parents get keyed; then key each node on the ordered pair of canonical child references, and compare the full pair rather than the hash, so a collision can never merge two different subtrees. Reuse the existing node or add a new one. There is even an accepted technical term for this: hash-consing. When the process is done, we end up with a directed acyclic graph where several parents point at one stored subtree, written back with the root at node 0, every reference renumbered, and node_count updated. The file format does not change. Each node’s two records can name another node, a data record, or an empty result, and the specification already describes this sharing for IPv4 aliases.
Across the 89 databases IPinfo builds daily, the catalogue went from 91.2 GB to 62.8 GB, 31% off the total. ipinfo_privacy lost the most per file, 789 MB down to 155 MB (80%); core 1.5 GB to 728 MB (53%); lite 38 MB to 25 MB (34%). The worked example on Lite explains the mechanism in one block: 37.76 MB to 25.08 MB in 452 ms, nodes: 3,790,310 to 2,205,358 (-41.82%), tree section: 30.32 MB to 17.64 MB, and the data section at 7.44 MB both before and after. Nothing was “compressed”, no reader changed, and because MMDB files are memory-mapped the file size is the memory footprint — resident memory on that example fell from 29.34 MiB to 27.58 MiB.
The tool is Apache 2.0 and I’m going to try it on the excellent Geolocus MMDB provided by the equally excellent folks from ONYPHE to see if I can shave some MBs of the fairly hefty downloads.
Font Revival
⚠️ AI: the site admits to using LLMs to help with the revivals.
Font Revival is an open set of historical typefaces, redrawn from old type-founder specimen books and released for anyone to use. The site’s tagline says it plainly: “Old forms. New life.”
The collection consists of 32 revivals and the dates in the names are the specimen dates: Baltimore foundry 1832, Boston and Central 1889, Marder, Luse 1894, Dickinson 1894, etc. Each entry lists its source foundry, a one-line description of the letterforms, a character count (58 to 302), a version number, and download links.
Every face ships as OTF, TTF, and WOFF2, plus a printable specimen PDF and a link into the GitHub repo where the outlines and build live.
Before you get all excited and dig in, many faces are capitals-only, and the smallest has 58 glyphs, which may be fine for display and titling work, but is pretty thin for anything needing figures or a full lowercase.
The site ships with a browser editor that lets you arrange ornaments, pick inks, set your own words, and export to PDF, Word, HTML, or an image. Your draft stays in the browser, and a live text box and size slider on the collection page let you test any face first.
The section header was composed with this site.
🔒🦆
I’m a sucker for anything that helps folks run their kit even a wee bit safer. So, imagine the glee when I discovered Securing a DuckDB server in production hot on the heels of the GA release of DuckDB 2.0.0, which ships with a robust server-mode (Quack) that does require you to be a bit more careful how you sling SQL around. But the server mode isn’t the only place you need to be careful.
DuckDB’s defaults assume the person writing SQL is the OS user that owns the process. That’s fine on a laptop. It breaks once DuckDB sits behind a shared server (Quack, Flight SQL, HTTP, MCP). Starlake’s Quack on Demand team audited this during their move to DuckDB 2.0 and found eight ways a SQL user can reach things you the operator didn’t mean to share. Full disclosure: the post ends as a pitch for their gateway, but the technical content is legit.
The eight exposures, risks, and mitigations are below and are deliberately succinct. Folks steeped in DuckDB and some form of linux/bsd should be able to grok the technical bits. Everyone else (perhaps also you DuckDB/linux/bsd nerds) should hit up Starlake’s post as it goes into detail (and they def deserve some 👀).
| Exposure | The Risk (Default Behavior) | The Mitigation (Production Fix) |
|---|---|---|
| 1. Credentials | getenv(), CLI history, and ATTACH strings can leak passwords/keys. | Use temporary secrets; drop them immediately after the ATTACH command. Use -init (FIFO) instead of stdin. |
| 2. Persistent Secrets | Automatically loads secrets from ~/.duckdb/stored_secrets. | Set secret_directory to an empty, read-only directory containing a file marker to prevent additions. |
| 3. Filesystem | SQL can read local files via paths or by treating a filename as a table name. | Run process in a private empty working directory. Ban path-based reads for non-admins. |
| 4. Global Settings | SET commands for memory, threads, or parsing affect all users on the node. | Lock global settings at startup; only allow session-level settings (e.g., TimeZone). |
| 5. Introspection | duckdb_secrets() and duckdb_logs reveal other users’ data/queries. | Reserve all introspection functions and system tables for administrators only. |
| 6. Dynamic SQL | query() or table macros can execute unvetted strings to bypass grants. | Refuse run-time SQL carriers or strictly validate arguments to be constants. |
| 7. Raw Passthrough | postgres_execute can bypass READ_ONLY attachments to write to remote DBs. | Use a low-privilege remote role (no write access) on the target database. |
| 8. CLI as Daemon | stdin allows .shell commands; .duckdbrc runs OS-user scripts. | Close stdin, use -no-agent, and avoid “keep-alive” queries that pin transactions. |
As noted, Starlake’s post is also an advert for their SQL gateway (i.e., “firewall”) which handles a fair chunk of these safety concerns. And, they’re not wrong that if you’re going to institute some security checks into your DuckDB machinations, those checks must read SQL exactly how DuckDB reads it.
Because DuckDB 2.0 supports backticks, string folding, and complex CTEs, a simple regex or basic parser in a security gateway can be bypassed. To solve this, you’d need to use a gateway that uses DuckDB’s own tokenizer. This ensures that the security layer and the database engine see the exact same tokens, preventing “obfuscation” attacks.
This sounds like an excellent candidate for an open-source project (for y’all…I’ve got plenty of side projects already).
We’ll close with a DuckDB production hardening checklist that you will definitely want to use if you do more with DuckDB than tinker/play:
- Credentials: No secrets in env vars or command lines; use temporary secrets.
- Storage:
secret_directoryis locked and read-only. - Isolation: Each process has a private working and spill directory.
- Access Control: File path access is disabled for general users.
- Settings: Node-wide settings are frozen after startup.
- Privacy: Metadata/Log tables are restricted to admins.
- Validation: Dynamic SQL and macros are blocked or strictly audited.
- Remote Security: Remote DBs are connected via read-only roles.
- Process Mgmt: stdin closed;
-initused for setup.
Build a better one from Starlake’s excellent guide.
FIN
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:
- Mastodon via
@dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev - Bluesky via
<https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy>
☮️
Leave a Reply