Nushell 0.91.0

Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.91.0 of Nu. This release adds changes to globing, an overhaul to the plugin protocol, support for piping command stderr, and new commands!

Where to get it

Nu 0.91.0 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install nu.

Note

The optional dataframe functionality is available by cargo install nu --features=dataframe.

As part of this release, we also publish a set of optional plugins you can install and use with Nu. To install, use cargo install nu_plugin_<plugin name>.

Table of content

Themes of this release / New features
Breaking changes
Full changelog

Themes of this release / New features [toc]

Handling globs for variables [toc]

Breaking change

See a full overview of the breaking changes

From this release, if you pass a string variable to commands that support glob patterns, then Nushell won't auto-expand the glob pattern (#11886, #11946).

For example, given let f = "a[ab]c.txt", then ls $f will only list a file named a[ab]c.txt. But if you want to auto-expand the glob pattern on variables, there are 3 ways:

Use the glob command with spreading:
```
let f = "a*c.txt"
rm ...(glob $f)
```
Use the into glob command:
```
let f = "a*c.txt"
rm ($f | into glob)
```
Annotate the variable with glob type:
```
let f: glob = "a*c.txt"
rm $f
```

In that case, str escape-glob command is no longer useful, and it has been deprecated (#12018).

You can check the guide and book for more information.

Plugin protocol overhaul [toc]

Breaking change

See a full overview of the breaking changes

The plugin protocol has been redesigned to support plugins that operate on streams (#11911). A new StreamingPlugin trait is provided for plugins that want to take advantage of this functionality.

The API for existing plugins written in Rust has not changed, but they do need to be recompiled with the latest version of nu-plugin to use the new protocol.

As part of this effort, several other aspects of the protocol have been improved:

Plugins will now automatically refuse to run with an incompatible version of Nushell. This should greatly improve error messages in that case.
Plugin custom values are now accepted nested within other values and as arguments. The previous protocol treated them as a special case, and they could only be used as input or output from a plugin command, and could not be nested within other values.
Plugins are now expected to keep running until Nushell tells them to stop. Currently, we still run a new plugin executable for each command, but this opens the door for that to change in the future.
The bidirectional communication is abstract enough to allow for plugins to have much tighter integration with the engine. Expect further improvements to plugins in future releases!

For details about how the new protocol works, please refer to the updated plugins section of the contributor book, as well as the new plugin protocol reference.

As a great example of a streaming plugin in the wild, @cablehead has already created the nu_plugin_from_sse plugin for parsing server-sent events in realtime from an http get call. Kudos!

Stderr pipeling support [toc]

Nushell now supports piping an external command's stderr to another command (#11708).

Let's say you want to pass stderr output to the less command, you can do this:

cmd e>| less

In case you want to combine both stdout and stderr output to less, you can do this:

cmd o+e>| less

For more information, you can check the guide.

REPL stability and panic recovery [toc]

Thanks to the work of @ayax79 in #11860, #11935, and #11953, the Nushell REPL should no longer crash if a panic occurs and should no longer exit if some other error is encountered. Besides being very convenient, this should also make Nushell safer to use a login shell, since panics would previously cause Nushell to crash on an attempted login. Similarly, if a panic was triggered when loading a config file, then this used to prevent the REPL from starting. Now, the REPL falls back to loading the default config files in this case.

Note that panics are still important errors/bugs, so please open issues and bug reports if you encounter any!

Hall of fame [toc]

Bug fixes [toc]

Thanks to all the contributors below for helping us solve issues and bugs 🙏

author	title	PR
@PanGan21	fix: process empty headers in `to md` command	#12012
@devyn	Add Goodbye message to ensure plugins exit when they are no longer needed	#12014
@zhiburt	nu-table: Improve `table -a`	#11905
@ayax79	wrapping run_repl with catch_unwind and restarting the repl on panic	#11860
@WindSoilder	make stderr works for failed external command	#11914
@kit494way	separate `commandline` into subcommands	#11877
@kit494way	Fix panic in `seq date`	#11871
@kit494way	Fix commandline --cursor to return int	#11864
@dannou812	Fixed to/from toml date format	#11846
@IanManske	Prevent duplicate keys for `lazy make`	#11808
@IanManske	Prevent duplicate records keys when decoding from nuon	#11807
@kit494way	Allow comments in match blocks	#11717
@WindSoilder	Fix file completions which contains glob pattern	#11766
@TrMen	Enforce call stack depth limit for all calls	#11729

Our set of commands is evolving [toc]

New commands [toc]

Thanks to the work of @devyn, this release adds two new commands related to streaming!

`tee`

Inspired by the Unix tee command, this command allows you to make a copy of a stream to a closure in the middle of your pipeline (#11928).

Examples:

# Get the sum of numbers from 1 to 100, but also save those numbers to a text file
seq 1 100 | tee { save numbers.txt } | math sum
# The exact opposite: keep the numbers, but save the sum to a file
seq 1 100 | tee { math sum | save sum.txt }
# Run an external command, and save a copy of its log output on stderr
do { cargo run } | tee --stderr { save err.txt }
# Filter the log output before saving it
do { cargo run } | tee --stderr { lines | find WARN | save warnings.txt }

The closure will run in a background thread in parallel, and it will only get a copy of values that actually make their way to the end of the pipeline. For example, if you cut the stream short:

seq 1 100 | tee { save numbers.txt } | first 5

then "numbers.txt" will only contain the first 5 numbers as well.

`interleave`

This command supports consuming multiple streams in parallel, and combining the streams into a single stream (#11955).

In contrast to zip, the values are added to the final stream as soon as they're ready, without any regard for fairness. There is no way to know which stream the values came from unless that information is embedded into the values by the closures being executed.

For example, the following zip-based pipeline will always produce output that looks like this:

> seq 1 50 | wrap a | zip { seq 1 50 | wrap b } | flatten
╭────┬────┬────╮
│  # │ a  │ b  │
├────┼────┼────┤
│  0 │  1 │ ❎ │
│  1 │ ❎ │  1 │
│  2 │  2 │ ❎ │
│  3 │ ❎ │  2 │
│  4 │  3 │ ❎ │
│  5 │ ❎ │  3 │
...

Each number from a is always paired with a number from b. However, if interleave is used instead, it is not predictable in which order a values will appear with respect to b:

> seq 1 50 | wrap a | interleave { seq 1 50 | wrap b }
╭────┬────┬────╮
│  # │ b  │ a  │
├────┼────┼────┤
│  0 │  1 │ ❎ │
│  1 │ ❎ │  1 │
│  2 │  2 │ ❎ │
│  3 │  3 │ ❎ │
│  4 │ ❎ │  2 │
│  5 │ ❎ │  3 │
│  6 │  4 │ ❎ │
│  7 │  5 │ ❎ │
│  8 │  6 │ ❎ │
│  9 │  7 │ ❎ │
│ 10 │ ❎ │  4 │
...

One advantage of this is that it is not necessary for both streams to produce the same amount of output, and whatever output is produced will be immediately available. This is particularly helpful for running external commands and interleaving lines from all of them:

interleave ...(
  (ls projects).name | each { |project|
    {
      cd $project
      make | lines | each { |line| {project: $project, out: $line} }
    }
  }
)

This example would run the make command in every subdirectory of the "projects" directory, and gather the output lines in parallel.

`is-not-empty`

As a quality of life improvement, we have added the is-not-empty command in #11991. It is the same as the is-empty command but negates the result. This should hopefully eliminate the need for users to hand-roll their own is-not-empty command.

`commandline`

In #11877, @kit494way improved the signatures for the commandline command. Instead of using flags to perform different operations, commandline now has subcommands:

commandline edit: to append, insert, or replace the cursor line (returns nothing)
commandline get-cursor: to get the cursor position (returns an integer)
commandline set-cursor: to set the cursor position (returns nothing)

These subcommands make certain flags unnecessary, and so these have been marked as deprecated:

--cursor
--cursor-end
--append
--insert
--replace

Changes to existing commands [toc]

`zip` supports closures

When zip is passed a closure now, it will run that closure and zip its output as a stream (#11924). With this change, it is no longer necessary for the argument to zip to complete before the result is available:

seq 1 10 | zip { 1.. | each { $in * 2 } }

This would not have completed before: the infinite stream would have had to be on the input side of zip, because a subexpression would have been (attempted to be) fully consumed:

# never completes, and probably eats up all of your memory!
seq 1 10 | zip (1.. | each { $in * 2 })
# works fine
1.. | each { $in * 2 } | zip (seq 1 10)

Migrating more commands to use `uutils`

Continuing the integration with uutils, this release migrates the mv and mkdir commands. In #12022, we renamed the umv command to mv, removing the old implementation of mv. Similarly, we removed the old mkdir command in #12007 and renamed the umkdir command to take its place.

`bits` supports binary values

Despite their name, the bits commands used to only support integers and did not work with binary values. @astral-l has remedied this in #11854, and binary values now work with the bits family of commands. Where appropriate, some of these commands (bits and, bits or, and bits xor) now have an --endian flag to specify the endianness when operating on binary values of different lengths.

`into int --signed`

Thanks to @astral-l's work in #11902, the into int command now has a --signed flag for interpreting binary values as signed integers. (The default behavior when converting binary values is to zero-extend.) Additionally, binary values longer than 8 bytes will now error, as this could overflow a 64-bit integer which is what Nushell uses internally for integers.

let cols = [name size]
ls | select $cols

In #11859, this was removed in favor of the spread operator:

ls | select ...$cols

Or, one can still provide the cell paths directly:

ls | select name size

Deprecated commands [toc]

str escape-glob has been deprecated in #12018.
Certain flags for commandline were deprecated in #11877.

Removed commands [toc]

umv has been renamed to mv in #12022.
umkdir has been renamed to mkdir in #12007.

Breaking changes [toc]

#11902 add --signed flag for binary into int conversions
#11911 Bidirectional communication and streams for plugins
#11972 Rename spans in the serialized form of Value
#11886 open, rm, umv, cp, rm and du: Don't globs if inputs are variables or string interpolation
#11877 separate commandline into subcommands
#11864 Fix commandline --cursor to return int
#11859 Remove list of cell path support for select and reject
#11857 Disallow spreading lists automatically when calling externals