xh; xq; lowcharts

Dealing with annual U.S. Federal Government citizen fiduciary mandate sapped all the will out of me over the weekend, so we’ll try to get a full week of Drops in to make up for no Bonus Drop.

xh

xh is a modern, friendly command-line tool designed for sending HTTP requests from the CLI, which is something I believe all readers have wont to do every now and again. It is written in Rust, and aims to re-implement much of HTTPie‘s excellent design with a focus on improved performance.

Like HTTPie, xh has expressive and intuitive syntax, which makes it easier-ish for folks to use than, say, bare curl. For example, sending a POST request with JSON data can be as simple as xh http://example.com/api post key=value more=data (super-helpful for testing/exploring APIs).

xh also has formatted and colorized terminal output, which enhances readability and makes it easier to understand the structure of responses, especially when dealing with JSON data. This feature, along with built-in JSON support, also gives it an edge over plain curl, and forgoes the need for jq (at least for a basic use-case).

For usage examples, xh‘s README provides a range of scenarios demonstrating how to use the tool for common tasks such as sending multipart requests, working with headers, and handling cookies. So, there’s plenty of bootstrap code to help get you started with xh and to explore its capabilities.

A fun set of CLI options are --curl and --curl-long which will output the equivalent curl commands for the simplified xh requests. This can be used as a tutorial for curl commands or for integrating new requests into curl-centric workflows.

We’ll close with a proper example usage of pulling headers from one of my sites:

$ xh --headers --json https://rud.is/b/
HTTP/2.0 200 OK
accept-ch: viewport-width, dpr, device-memory, rtt, downlink, ect, ua, platform, arch, model, mobile
cache-control: no-store, no-cache, must-revalidate
content-encoding: gzip
content-type: text/html; charset=UTF-8
date: Mon, 15 Apr 2024 17:00:57 GMT
expires: Thu, 19 Nov 1981 08:52:00 GMT
feature-policy: geolocation 'none';midi 'none';sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';fullscreen 'self';payment 'none';
link: <https://rud.is/b/wp-json/friends/v1>; rel="friends-base-url"
link: <https://rud.is/b/wp-json/>; rel="https://api.w.org/"
link: <https://wp.me/23idr>; rel=shortlink
pragma: no-cache
referrer-policy: origin
server: nginx
set-cookie: PHPSESSID=kp3v141b5393mdbohe0d5rnk5o; path=/
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: Accept-Encoding
vary: accept, content-type
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
x-powered-by: ❤
x-xss-protection: 1; mode=block

xq

I, somehow, missed covering xq when we covered various XML/HTML processing tools a while back. It’s a blazingly fast and capable Golang-based beautifier and content extractor for all XML/HTML content. It features:

• syntax highlighting
• automatic indentation and formatting
• automatic pagination
• node content extraction

Let’s say we want to find all the malware families the Lazarus Group uses (courtesy of Malopedia). It’s pretty straightforward, especially since xq supports CSS selectors:

$ xh https://malpedia.caad.fkie.fraunhofer.de/actor/lazarus_group | xq --query "span.badge-custom"
win.unidentified_101
win.unidentified_090
aix.fastcash
apk.badcall
apk.hardrain
elf.badcall
elf.simpletea
elf.spectral_blur
js.quickcafe
osx.3cx_backdoor
osx.applejeus
osx.casso
osx.dacls
osx.hloader
osx.interception
osx.kandykorn
osx.manuscrypt
osx.poolrat
osx.rustbucket
osx.simpletea
osx.spectral_blur
osx.sugarloader
osx.unidentified_001
osx.watchcat
osx.yort
...

Unfortunately, you kind of have to use that approach for most HTML content on the web since most HTML content on the web is malformed XML and this tool does not like that. If you do need to use XPath on gnarly HTML, consider passing the content through tidy-html5.

It’s nice to have another tool in the *ML processing toolbox.

lowcharts

I’ve been poking around at CLI graphing alternatives for output from DuckDB so I can dedicate a chapter to them in my Cooking With DuckDB e-book, and came across lowcharts during this quest. It’s not a great fit for DuckDB but is a pretty neat fit-for-purpose CLI charting tool.

In their own words:

lowcharts is meant to be used in those scenarios where we have numerical data in text files that we want to display in the terminal to do a basic analysis.

An example would be the logs of a service (webserver, database, proxy, container orchestration, etc.) where times (or sizes) of requests are logged. In an ideal world you would have those logs accessible via a kibana (or similar) or those metrics exposed to a prometheus (or similar) and graphed in a grafana dashboard (or similar). But sometimes we need to cope with non ideal worlds, and troubleshoot a service with nothing more of what we can muster in a shell terminal.

The section header shows off a pretty solid use-case: time-histogram of 200 hits to my main site’s access log, today. More interesting (and, one that fits in text), 5xx hits:

lowcharts timehist --format '%d/%b/%Y:%H:%M:%S' --regex ' 5\d\d ' rud.is.access.log
Matches: 56.
Each ∎ represents a count of 1

[02:51:13] [ 2] ∎∎
[03:16:34] [ 1] ∎
[03:41:55] [ 0]
[04:07:16] [ 0]
[04:32:37] [ 1] ∎
[04:57:58] [ 2] ∎∎
[05:23:19] [ 2] ∎∎
[05:48:40] [ 0]
[06:14:01] [ 0]
[06:39:22] [ 2] ∎∎
[07:04:44] [39] ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
[07:30:05] [ 0]
[07:55:26] [ 0]
[08:20:47] [ 0]
[08:46:08] [ 3] ∎∎∎
[09:11:29] [ 0]
[09:36:50] [ 0]
[10:02:11] [ 1] ∎
[10:27:32] [ 2] ∎∎
[10:52:53] [ 1] ∎

It has many options, too:

• common-terms: plot histogram with most common terms in input lines
• hist: plot a histogram from input values
• matches: plot barchat with counts of occurrences of matches parameters
• plot: plot a 2D x-y graph where y-values are averages of input values
• split-timehist: plot histogram of with number of matches over time, split per match type
• timehist: plot histogram with number of matches over time

each of those sub-commands has other options as well.

This is something I’ll be keeping in my cloud-init default install configs.

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev ☮️