moreutils; Rats

Yep, we’ve got just two sections for this Friday edition of the Drop. The first is a collection of additional Unix utilities that augment the standard set of Unix tools. These utilities are designed to perform common tasks not covered by the basic tools, providing a high-utility command-line environment for users.

The second is more to just freak folks out and provide some data to folks participating in the 30-Day Chart Challenge (and, beyond).

moreutils

Each of the CLI tools in the moreutils project embraces the Unix philosophy of being small, modular, and doing one thing well:

• chronic: Runs a command quietly unless it fails (exits non-zero), which is useful for running commands in scripts without generating output unless there is an error.
• combine: Combines lines from two files using boolean operations, allowing for complex text processing tasks.
• errno: Looks up errno names and descriptions, providing a quick reference for system error codes.
• ifdata: Retrieves network interface information without the need for parsing the output of ifconfig, simplifying network management tasks.
• ifne: (If Not Empty) Runs a program if the standard input is not empty, which can be used to conditionally execute commands based on the presence of input.
• isutf8: Checks if a file or standard input is utf-8 encoded, ensuring text encoding compatibility.
• lckdo: Executes a program with a lock held, which is useful for preventing race conditions in scripts.
• mispipe: Pipes two commands, returning the exit status of the first, which can be used for error checking in command pipelines.
• parallel: Runs multiple jobs at once, taking advantage of multi-core processors to improve efficiency.
• pee: Tees standard input to pipes, similar to tee but for pipes, allowing the output of a command to be sent to multiple commands in parallel.
• sponge: Soaks up standard input and writes to a file, enabling in-place file editing without the need for temporary files.
• ts: Adds or converts timestamps to standard input, which can be used for logging and monitoring purposes.
• vidir: Allows editing of a directory in a text editor, providing a convenient way to batch rename or delete files.
• vipe: Inserts a text editor into a pipe, allowing for interactive editing of a pipeline’s data stream before it is passed to the next command.
• zrun: Automatically uncompresses arguments to a command, facilitating the use of compressed files with commands that do not natively support them.

The moreutils origin story begins with its creation by Joey Hess in 2006. The project was motivated by the recognition of gaps in the Unix toolset, where certain useful commands were missing. There is a vetting process for the inclusion of new tools; and, over the years, it has become a valuable addition to the Unix/Linux ecosystem.

We won’t cover every utility, but I’ll share some that I find useful.

I believe all Drop readers know about the GNU parallel tool that is available as a standalone CLI package. A different flavor of it comes along for the ride with moreutils. Both parallels let us execute jobs in parallel rather than sequentially. This can significantly speed up tasks that are CPU-bound or waiting on I/O when working with multiple files or data streams. The moreutils version of parallel is simpler and more focused on the execution of commands across multiple processors without the extensive feature set of its GNU counterpart.

The vidir CLI offers a quick way to edit directories or sets of files directly from your text editor. It presents the contents of a directory as a text file where each line represents a file or directory. We can then delete lines to remove files, or edit lines to rename files. This approach provides a clean, intuitive interface for batch file operations, and leverages whatever text editor you run with.

When reading data from UDP broadcast sources or from any source that is line-oriented but does not include timestamps, ts is a much quicker way of getting timestamps included with the output (literally just pipe stdin to ts).

I think my fav one is ifdata since it provides a quick way to retrieve network interface information without needing to parse the output of ifconfig. So:

$ ifdata -p en1
10.1.10.208 255.255.255.0 10.1.10.255 1500

Gets you easily parsable IP, mask, etc.

There’s tons more you can do with it. Everyone can:

• -e: Test to see if the interface exists, exit nonzero if it does not.
• -p: Prints out the whole configuration of the interface.
• -pe: Prints “yes” or “no” if the interface exists or not.
• -pa: Prints the IPv4 address of the interface.
• -pn: Prints the netmask of the interface.
• -pN: Prints the network address of the interface.
• -pb: Prints the broadcast address of the interface.
• -pm: Prints the MTU of the interface.

and, Linux folks have even more functionality:

• -ph: Prints the hardware address of the interface. Exit with a failure exit code if there is no hardware address for the: given network interface.
• -pf: Prints the flags of the interface.
• -si: Prints out all the input statistics of the interface.
• -sip: Prints the number of input packets.
• -sib: Prints the number of input bytes.
• -sie: Prints the number of input errors.
• -sid: Prints the number of dropped input packets.
• -sif: Prints the number of input fifo overruns.
• -sic: Print the number of compressed input packets.
• -sim: Prints the number of input multicast packets.
• -so: Prints out all the output statistics of the interface.
• -sop: Prints the number of output packets.
• -sob: Prints the number of output bytes.
• -soe: Prints the number of output errors.
• -sod: Prints the number of dropped output packets.
• -sof: Prints the number of output fifo overruns.
• -sox: Print the number of output collisions.
• -soc: Prints the number of output carrier losses.
• -som: Prints the number of output multicast packets.
• -bips: Prints the number of bytes of incoming traffic measured in one second.
• -bops: Prints the number of bytes of outgoing traffic measured in one second.

And, since I deal quite a bit with processing stdin, ifne is pretty cool, too. Their baked-in example shows why pretty well:

$ find . -name core | ifne mail -E 'set escape' -s "Core files found" root

Grab it and start relying on this extended functionality, and watch this space for updates.

Rats in Realtime

This is not an acronym or CLI utility. It’s literally data and a dashboard on the rat sightings of NYC.

Truth be told, it’s also an introduction to Realtime, a collection of data feeds that the Realtime platform generates their newsletters, reports, and front page stories.

Realtime continuously tracks and detects key changes within public data on a range of topics: economics, finance, industry, politics, sports, environment, media, local government activities, and others. It then uses “AI” to generate live updating visualizations and reports that combine observations about the data with news reports that may give context for what is being observed.

It also aims to serve as a counterweight to the emotional appeals that often define media and political discourse by anchoring public narratives in empirical data. This information is usually scattered across the internets, and Realtime centralizes it and makes it searchable.

It also provides better access to high-frequency data sources, like prediction markets and economic forecasts, that are typically only available to large institutions.

Make sure to subscribe to the free newsletter feed while the site is in beta; and, Developer Tools is your BFF (for data).

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev ☮️