hrbrmstr's Daily Drop

Drop #713 (2025-09-19): AVAST ME HEARTIES!

hrbrmstr

Pirating Pirate Data; Ducking Around With Pirate Data;

(I’ll refrain from more pirate-speak for the remainder of the post. Also, no TL;DR.)

Here I was all set to show off some Observable Notebook 2.0 features using the U.S. Gov Maritime Safety Information (MSI), Anti-Shipping Activity Messages when I noticed my dev-only R {asam} pacakge, only to discover:

> asam::read_asam()
Reading ASAM database (this may take a while)
Error in (function (quiet = FALSE)  : Not Found (HTTP 404).

I assumed the MSI just did a site re-vamp,gave it a visit in the browser, saw a new link to “Download ASAM Geospatial Files” which included this geodatabase, only to discover:

🦆>FROM st_read('Asam_data_download.gdb') SELECT min(dateofocc), max(dateofocc);
┌─────────────────────┬─────────────────────┐
│   min(dateofocc)    │   max(dateofocc)    │
│    timestamp_ms     │    timestamp_ms     │
├─────────────────────┼─────────────────────┤
│ 1978-05-01 00:00:00 │ 2024-06-25 17:00:54 │
└─────────────────────┴─────────────────────┘

that the data ends in mid-2024.

O_O

I did a massive amount of Kagi-ing, and can report I have new/fresh data!

Read the first section for how to hack JWTs to access it, and the second section to see why the above example is in DuckDB.

Pirating Pirate Data

Given that it looks like the U.S. Government doesn’t care about piracy anymore (well, there were those Signal chats earlier this year, I guess), I figured this data had to be published somewhere.

It turns out Maritime Optima does publish Anti Shipping Activity Messengers (Piracy maps) | ShipAtlas User Guides in their Ship Atlas webapp/portal. You will need to sign up for a free account to access the portal, and — thankfully — the piracy data is one of the datasets offered in the free plan.

The section header is a BurpSuite screen capture. I used the embedded browser in it to login and spelunk the responses. It’s manual work, but it also didn’t take too long. I just patiently followed the login flow to see where the JWT operations were, and did some response and request surgery (as you’ll see, below) to automate the downloads.

NOTE: MAKE SURE TO USE USERNAME+PASSWORD vs OAuth, since you’re going to need credentials to get programmatically download the data (datasci friends don’t let datasci friends rely on manual download idioms).

We’ll be using R since the old-school {httr} library is expressive + concise, and makes it easy to login and get the data. Store your username and password in the env vars seen below and run the code:

library(httr)

# GET JWT TOKEN

c(
  Host = "api.maritimeoptima.com",
  `Content-Length` = "311",
  `Sec-Ch-Ua-Platform` = '"macOS"',
  `Local-Utc-Epoch-Time` = as.numeric(Sys.time()),
  `Sec-Ch-Ua` = '"Not?A_Brand";v="99", "Chromium";v="130"',
  `Accept-Language` = "en-US,en;q=0.9",
  `Sec-Ch-Ua-Mobile` = "?0",
  `App-Version` = "v2.227.1-master-frontend",
  `App-Context` = "vessel-tracker",
  `Device-Id` = "Chrome-7e87e680-f565-43e3-ade9-bb531d9debb9",
  Accept = "*/*",
  `Content-Type` = "application/json",
  `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.70 Safari/537.36",
  Origin = "https://app.maritimeoptima.com",
  `Sec-Fetch-Site` = "same-site",
  `Sec-Fetch-Mode` = "cors",
  `Sec-Fetch-Dest` = "empty",
  Referer = "https://app.maritimeoptima.com/",
  `Accept-Encoding` = "gzip, deflate, br",
  Priority = "u=1, i"
) -> login_headers

sprintf(
  fmt = '{"operationName":"Login","variables":{"email":"%s","password":"%s"},"query":"mutation Login($email: String!, $password: String!, $utm: UTMInput) {\\n  login(email: $email, password: $password, platform: WEB, utm: $utm) {\\n    jwt\\n    refresh\\n    numLoggedOut\\n    __typename\\n  }\\n}"}',
  Sys.getenv("MARITIME_OPTIMA_EMAIL"),
  Sys.getenv("MARITIME_OPTIMA_PASSWORD")
) -> login_data

httr::POST(
  url = "https://api.maritimeoptima.com/graphql-public", 
  httr::add_headers(
    Host = "api.maritimeoptima.com",
    `Content-Length` = "311",
    `Sec-Ch-Ua-Platform` = '"macOS"',
    `Local-Utc-Epoch-Time` = as.numeric(Sys.time()),
    `Sec-Ch-Ua` = '"Not?A_Brand";v="99", "Chromium";v="130"',
    `Accept-Language` = "en-US,en;q=0.9",
    `Sec-Ch-Ua-Mobile` = "?0",
    `App-Version` = "v2.227.1-master-frontend",
    `App-Context` = "vessel-tracker",
    `Device-Id` = "Chrome-7e87e680-f565-43e3-ade9-bb531d9debb9",
    Accept = "*/*",
    `Content-Type` = "application/json",
    `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.70 Safari/537.36",
    Origin = "https://app.maritimeoptima.com",
    `Sec-Fetch-Site` = "same-site",
    `Sec-Fetch-Mode` = "cors",
    `Sec-Fetch-Dest` = "empty",
    Referer = "https://app.maritimeoptima.com/",
    `Accept-Encoding` = "gzip, deflate, br",
    Priority = "u=1, i"
  ), 
  body = login_data
) |> 
  httr::content(as = "text") |> 
  jsonlite::fromJSON() -> login_jwt

# USE JWT TOKEN TO GET TO THE PIRATE JSON

httr::GET(
  url = "https://api.maritimeoptima.com/mapdata/piracy-data.geojson", 
  httr::add_headers(
    Host = "api.maritimeoptima.com",
    Authorization = sprintf("Bearer %s", login_jwt$data$login$jwt),
    `X-Team` = "587718",
    `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.70 Safari/537.36",
    Accept = "application/json",
    Origin = "https://app.maritimeoptima.com",
    `Sec-Fetch-Site` = "same-site",
    `Sec-Fetch-Mode` = "cors",
    `Sec-Fetch-Dest` = "empty",
    Referer = "https://app.maritimeoptima.com/",
    `Accept-Encoding` = "gzip, deflate, br",
    `Accept-Language` = "en-US,en;q=0.9",
    Priority = "u=1, i"
  )
) |> 
  httr::content(as = "text") |> 
  writeLines("piracy-data.geojson")

Now, I’m not thrilled we have to rely on the good graces of a commercial entity to gain access to critical safety information, but that’s how 2025+ seems to be going all ’round.

There’s still time to play with that data on International TLAP Day. If you don’t feel like “hacking”, you can get the download I did in the AM on the 19th here: https://rud.is/dl/2025-09-19-piracy-data.geojson.

Ducking Around With Pirate Data

DuckDB has great support for spatial data, including GeoJSON.

If you’ve ever played with the ASAM data, you may remember that it wasn’t exactly, er, consistently formatted. You’l be glad to know that not much has changed in the GeoJSON format.

This will get you started with clean-ish data:

🦆>INSTALL spatial;
🦆>LOAD spatial;
🦆>
🦆>CREATE OR REPLACE TABLE asam AS (
     FROM st_read('piracy-data.geojson')
     SELECT
       TO_TIMESTAMP(date) AS ts,
       attack_number,
       attack_type,
       regexp_replace(description, '^.*Posn: [^EW]+[EW][^\w\s]+ ', '') AS description,
       geom
     ORDER BY ts
   );
🦆>
🦆>FROM asam LIMIT 10;
┌──────────────────────┬───────────────┬─────────────┬──────────────────────┬───────────────────────────┐
│          ts          │ attack_number │ attack_type │     description      │           geom            │
│ timestamp with tim…  │    varchar    │   varchar   │       varchar        │         geometry          │
├──────────────────────┼───────────────┼─────────────┼──────────────────────┼───────────────────────────┤
│ 2024-09-22 07:00:0…  │ 078-24        │ Hijacked    │ Around 47nm South …  │ POINT (113.833611111111…  │
│ 2024-09-29 13:02:0…  │ 082-24        │ Boarded     │ Singapore Straits.…  │ POINT (103.7283333333 1…  │
│ 2024-10-01 12:01:0…  │ 081-24        │ Boarded     │ Singapore Straits.…  │ POINT (103.7166666667 1…  │
│ 2024-10-12 15:08:0…  │ 083-24        │ Boarded     │ Kutubdia Anchorage…  │ POINT (91.785 21.811666…  │
│ 2024-10-15 17:00:0…  │ 090-24        │ Boarded     │ Panjang Anchorage,…  │ POINT (105.2875 -5.5008…  │
│ 2024-10-16 15:05:0…  │ 104-24        │ Boarded     │ Belawan Anchorage,…  │ POINT (98.8300715605417…  │
│ 2024-10-17 12:05:0…  │ 086-24        │ Boarded     │ Singapore Straits.…  │ POINT (103.49 1.13)       │
│ 2024-10-17 14:00:0…  │ 085-24        │ Boarded     │ Singapore Straits.…  │ POINT (103.513333333333…  │
│ 2024-10-17 14:04:0…  │ 088-24        │ Boarded     │ Singapore Straits.…  │ POINT (103.7175 1.09083…  │
│ 2024-10-18 18:09:0…  │ 084-24        │ Boarded     │ Takoradi Anchorage…  │ POINT (-1.6865 4.878333…  │
├──────────────────────┴───────────────┴─────────────┴──────────────────────┴───────────────────────────┤
│ 10 rows                                                                                     5 columns │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┘

Let’s check on anti-shipping activity for this year:

🦆>FROM asam
   SELECT
     EXTRACT(MONTH FROM ts) AS month,
     MONTHNAME(ts) AS month_name,
     attack_type,
     COUNT(*) AS total_attacks
   WHERE EXTRACT(YEAR FROM ts) = 2025
   GROUP BY EXTRACT(MONTH FROM ts), MONTHNAME(ts), attack_type
   ORDER BY month, attack_type;
┌───────┬────────────┬──────────────────┬───────────────┐
│ month │ month_name │   attack_type    │ total_attacks │
│ int64 │  varchar   │     varchar      │     int64     │
├───────┼────────────┼──────────────────┼───────────────┤
│     1 │ January    │ Boarded          │            16 │
│     2 │ February   │ Boarded          │            12 │
│     2 │ February   │ Hijacked         │             2 │
│     3 │ March      │ Attempted Attack │             4 │
│     3 │ March      │ Boarded          │            15 │
│     3 │ March      │ Hijacked         │             2 │
│     4 │ April      │ Attempted Attack │             2 │
│     4 │ April      │ Boarded          │            10 │
│     5 │ May        │ Boarded          │            16 │
│     5 │ May        │ Fired Upon       │             1 │
│     6 │ June       │ Attempted Attack │             1 │
│     6 │ June       │ Boarded          │            17 │
│     7 │ July       │ Attempted Attack │             1 │
│     7 │ July       │ Boarded          │             9 │
│     8 │ August     │ Boarded          │             3 │
│     9 │ September  │ Boarded          │             1 │
├───────┴────────────┴──────────────────┴───────────────┤
│ 16 rows                                     4 columns │
└───────────────────────────────────────────────────────┘

The old data from the GDB the U.S. Gov still provides does not have the same fields (so, I guess there was some cleanup), but it looks like piracy is either down, or reporting is “meh”:

🦆>CREATE OR REPLACE TABLE asam AS (
     FROM st_read('Asam_data_download.gdb')
     SELECT
       dateofocc::DATE AS ts,
       hostilitytype_l
     ORDER BY ts
   );
🦆>
🦆>FROM asam
      SELECT
        EXTRACT(MONTH FROM ts) AS month,
        MONTHNAME(ts) AS month_name,
        hostilitytype_l,
        COUNT(*) AS total_attacks
      WHERE EXTRACT(YEAR FROM ts) = 2024
      GROUP BY EXTRACT(MONTH FROM ts), MONTHNAME(ts), hostilitytype_l
      ORDER BY month, hostilitytype_l;
┌───────┬────────────┬─────────────────┬───────────────┐
│ month │ month_name │ hostilitytype_l │ total_attacks │
│ int64 │  varchar   │      int16      │     int64     │
├───────┼────────────┼─────────────────┼───────────────┤
│     1 │ January    │               2 │            24 │
│     1 │ January    │               3 │            13 │
│     1 │ January    │               4 │             1 │
│     1 │ January    │               6 │             6 │
│     1 │ January    │               7 │             5 │
│     1 │ January    │               9 │             1 │
│     1 │ January    │              11 │            13 │
│     2 │ February   │               2 │            20 │
│     2 │ February   │               3 │             5 │
│     2 │ February   │               6 │             3 │
│     2 │ February   │               9 │             2 │
│     2 │ February   │              11 │             9 │
│     3 │ March      │               1 │             2 │
│     3 │ March      │               2 │            12 │
│     3 │ March      │               3 │             4 │
│     3 │ March      │               6 │             5 │
│     3 │ March      │               7 │             3 │
│     3 │ March      │              11 │             4 │
│     4 │ April      │               2 │             9 │
│     4 │ April      │               3 │             4 │
│     4 │ April      │               4 │             1 │
│     4 │ April      │               6 │             4 │
│     4 │ April      │               7 │             1 │
│     4 │ April      │              10 │             1 │
│     4 │ April      │              11 │             4 │
│     5 │ May        │               1 │             1 │
│     5 │ May        │               2 │             4 │
│     5 │ May        │               3 │             4 │
│     5 │ May        │               4 │             1 │
│     5 │ May        │               6 │             4 │
│     5 │ May        │               7 │             2 │
│     5 │ May        │              11 │            10 │
│     6 │ June       │               2 │            14 │
│     6 │ June       │               3 │             3 │
│     6 │ June       │               4 │             1 │
│     6 │ June       │               6 │             1 │
│     6 │ June       │              11 │             5 │
├───────┴────────────┴─────────────────┴───────────────┤

There’a bit of a time gap between the two data sets, and the data fields aren’t uniform, but they’re both usable.

Maritime Optima only serves up ASAM data for a bout a ~year, so you’ll need to set up a systemd timer/cron job to pull the JSON on a regular (I’d suggest daily) basis if you do want to work with a more complete dataset next year.

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:

  • 🐘 Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev
  • 🦋 Bluesky via https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy

☮️

, , , , ,

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.