Nushell 0.94.0

Nushell, or Nu for short, is a new shell that takes a modern, structured approach to your command line. It works seamlessly with the data from your filesystem, operating system, and a growing number of file formats to make it easy to build powerful command line pipelines.

Today, we're releasing version 0.94.0 of Nu. This release adds case-preserving environment, changes to path handling, raw string literals, and improvements to streaming!

Where to get it

Nu 0.94.0 is available as pre-built binaries or from crates.io. If you have Rust installed you can install it using cargo install nu.

As part of this release, we also publish a set of optional plugins you can install and use with Nu. To install, use cargo install nu_plugin_<plugin name>.

Warning

The dataframes cargo feature has been removed in this release, after being deprecated in 0.93.0. You may need to modify your build scripts if you have been building Nushell from source.

Table of content

Themes of this release / New features [toc]

Fixing path and PWD handling [toc]

One of our contributors, @YizhePKU, has started an effort to refactor how Nushell internally handles the current working directory and paths in general. Namely, Nushell aggressively canonicalizes paths instead of using logical paths which can lead to unintuitive or annoying behavior (#2175). Thanks to their work, $env.PWD and the pwd command now support logical paths. With time, we intend to make similar to changes to other commands to make them more intuitive and consistent with the rest of Nushell. Another goal with these changes is to eliminate bugs in commands, as some commands incorrectly use the Nushell process's current working directory instead of $env.PWD. So far, the grid, path type, and touch --reference commands have been fixed, and we are aiming to bring more fixes in the next release. We kindly ask for your patience as we rework this part of Nushell, and we would appreciate if you would report any issues you encounter!

Case-preserving environment [toc]

Breaking change

See a full overview of the breaking changes

In an effort to allow more ergonomic cross-platform scripting, we have changed the environment in Nushell to be case-preserving (#12701). That is, environment variables keep their initial casing but can be accessed in a case-insensitive manner within Nushell. This means that accessing $env.PATH or $env.Path will now work on both unix-based systems and Windows systems. To get environment variables in a case-sensitive manner, you can use get -s.

$env | get -s PATH

Please note that there are still a few things that have to be ironed out with this new behavior. For example, it is currently unclear what load-env { FOO: BAR, foo: bar } should do. So, there may be more changes in the near future.

Streaming all the things [toc]

Another major effort with this release was to make streaming more consistent and widespread (#12774, #12897, #12918, and more). For example, Nushell now passes operating system pipe descriptors directly to external commands, meaning there should be an improvement to throughput for some pipelines. Additionally, several commands were reworked so that they are now able to stream their output in more cases instead of having to collect it all in memory beforehand. The commands in question include from csv, to csv, from tsv, to tsv, from json --objects, take, skip, first, last, bytes ends-with, bytes collect, str join, into string, and into binary. Of course, the plan is to add more commands to this list in the future, so this is only the beginning!

New language feature: raw strings [toc]

This release adds a new type of string literal to the language: raw strings (#9956). These are just like single quoted strings, except that raw strings may also contain single quotes. This is possible, since raw strings are enclosed by a starting r#' and an ending '#. This syntax should look familiar to users of Rust.

r#'some text'# == 'some text' # true

r#'contains 'quoted' text'# == "contains 'quoted' text"

Additional # symbols can be added to the start and end of the raw string to enclose one less than the same number of # symbols next to a ' symbol in the string.

r###'this text has multiple '## symbols'###

Removal of deprecated features [toc]

Breaking change

See a full overview of the breaking changes

Last release, we deprecated the dataframe commands and feature in favor of the new polars plugin. See the previous release notes for more information and migration steps. In this release, the dataframe commands have been removed as planned alongside the dataframes cargo feature. Similarly, this release removes lazy records from the language following their deprecation. This includes the removal of the lazy make command and the --collect-lazyrecords flag on describe. One consequence of this change is that the sys command will now return a regular record and will take a minimum of 400ms to complete (to sample CPU usage). As such, the sys command has been deprecated, and new subcommands have been added in its place (sys mem, sys cpu, sys host, sys disks, sys net, sys temp, and sys users).

Shell integration config [toc]

Breaking change

See a full overview of the breaking changes

With this release, we have changed the shell_integration setting in the config to be a record of multiple settings (#12629):

shell_integration: {
  # osc2 abbreviates the path if in the home_dir, sets the tab/window title, shows the running command in the tab/window title
  osc2: true
  # osc7 is a way to communicate the path to the terminal, this is helpful for spawning new tabs in the same directory
  osc7: true
  # osc8 is also implemented as the deprecated setting ls.show_clickable_links, it shows clickable links in ls output if your terminal supports it
  osc8: true
  # osc9_9 is from ConEmu and is starting to get wider support. It's similar to osc7 in that it communicates the path to the terminal
  osc9_9: false
  # osc133 is several escapes invented by Final Term which include the supported ones below.
  # 133;A - Mark prompt start
  # 133;B - Mark prompt end
  # 133;C - Mark pre-execution
  # 133;D;exit - Mark execution finished with exit code
  # This is used to enable terminals to know where the prompt is, the command is, where the command finishes, and where the output of the command is
  osc133: true
  # osc633 is closely related to osc133 but only exists in visual studio code (vscode) and supports their shell integration features
  # 633;A - Mark prompt start
  # 633;B - Mark prompt end
  # 633;C - Mark pre-execution
  # 633;D;exit - Mark execution finished with exit code
  # 633;E - NOT IMPLEMENTED - Explicitly set the command line with an optional nonce
  # 633;P;Cwd=<path> - Mark the current working directory and communicate it to the terminal
  # and also helps with the run recent menu in vscode
  osc633: true
  # reset_application_mode is escape \x1b[?1l and was added to help ssh work better
  reset_application_mode: true
}

This change provides greater granularity, allowing you to disable any integrations that may cause issues in certain terminals.

Hall of fame [toc]

Bug fixes [toc]

Thanks to all the contributors below for helping us solve issues and bugs 🙏

authortitleurl
@IanManskeFix panic when redirecting nothing#12970
@YizhePKUFix touch --reference using PWD from the environment#12976
@YizhePKUFix path type using PWD from the environment#12975
@IanManskeClear environment for child Commands#12901
@IanManskePreserve metadata in more places#12848
@WindSoilderallow define it as a variable inside closure#12888
@IanManskeReplace ExternalStream with new ByteStream type#12774
@WindSoilderallow passing float value to custom command#12879
@IanManskeFix sys panic#12846
@IanManskeFix char panic#12867
@ExaltedBagelFix panic when exploring empty dictionary#12860
@ExaltedBagelFix improperly escaped strings in stor insert#12820
@IanManskeFix custom converters with save#12833
@IanManskeFix pipe redirection into complete#12818
@IanManskeFix syntax highlighting for not#12815
@IanManskeFix list spread syntax highlighting#12793
@WindSoilderallow raw string to be used inside subexpression, list, and closure#12776
@WindSoilderAllow ls works inside dir with [] brackets#12625
@ExaltedBagelEnable columns with spaces for into_sqlite by adding quotes to column names#12759
@WindSoilderallow raw string to be used inside subexpression, list, and closure#12776
@IanManskeBump base64 to 0.22.1#12757
@devynFix trailing slash in PWD set by cd#12760
@lavafrothfix: prevent relative directory traversal from crashing#12438
@merelymyselfPrevent each from swallowing errors when eval_block returns a ListStream#12412

Enhancing the documentation [toc]

Thanks to all the contributors below for helping us making the documentation of Nushell commands better 🙏

authortitleurl
@NotTheDr01dsFixed small error in the help-examples for the get command#12877
@NotTheDr01dsFixed a nitpick usage-help error - closure v. block#12876
@fdncredmake it clearer what is being loaded with --log-level info#12875
@WindSoildermake better messages for incomplete string#12868
@NotTheDr01dsSearch terms for compact command#12864
@NotTheDr01dsAdd example and search term for 'repeat' to the fill command#12844
@devynMake the message when running a plugin exe directly clearer#12806
@amtoineimprove NUON documentation#12717
@sholderbachUpdate PLATFORM_SUPPORT regarding feature flags#12741

Our set of commands is evolving [toc]

New commands [toc]

sys subcommands [toc]

This release adds new sys subcommands corresponding to each column of the record returned by sys (#12747):

  • sys host
  • sys cpu
  • sys mem
  • sys net
  • sys disks
  • sys temp
  • sys users (new!)

The output of these commands should be the same as before as the corresponding sys column except for a few differences:

  • sys host does not have a sessions column. This table has been moved to the new sys users command (#12787). (The old sys command still has the sessions column.)
  • The boot_time column in sys host and sys | get host is now a date value instead of a formatted string (#12846).

ps support on BSD systems [toc]

In #12892, @devyn has added support for the ps command on FreeBSD, NetBSD, and OpenBSD!

debug profile --lines [toc]

The debug profile command now has a --lines flag with #12930. Providing this will flag will add a file and line column to the output of debug profile.

Changes to existing commands [toc]

Making range semantics consistent [toc]

Breaking change

See a full overview of the breaking changes

Ranges are inclusive in the upper bound by default in Nushell (e.g., 1..2 or 1..=2 gives [1, 2]). However, some commands treated the upperbound as exclusive. With #12894 this has been fixed for the following commands:

  • str substring
  • str index-of
  • detect columns

To specify an exclusive upperbound, use the ..< form for ranges (e.g., 1..<2).

parse [toc]

Breaking change

See a full overview of the breaking changes

To support streaming output, parse now operates/matches on a single line at a time only if provided with the streaming output of an external command, file, or raw byte stream. For example, these snippets will try to match each line separately:

^cat file.txt | parse -r "some regex"
^cat file.txt | take 1024 | parse -r "some regex"

open file.txt | parse -r "some regex"
open --raw file.txt | parse -r "some regex"
open --raw file.txt | skip 1024 | parse -r "some regex"

The old behavior was to collect all of the output the external command or byte stream and then run the regex across the whole string. To mimic this behavior, you can use the collect command or store the output in a variable:

^cat file.txt | collect | parse -r "some regex"

let text = open file.txt
$text | parse -r "some regex"

Note that this change does not affect normal value streams like:

[(open foo.txt) (open bar.txt)] | parse -r "some regex"

In this case, the regex is run separately over each string value in the stream. With the proper regex flags, the regex is able to match across multiple lines within the same string value.

Note that parse may see more breaking changes in the next release to simplify these behaviors.

scope commands [toc]

Breaking change

See a full overview of the breaking changes

After #12832, several columns from the output of scope commands have been removed:

  • is_builtin
  • is_plugin
  • is_custom
  • is_keyword
  • is_extern

All of these columns are mutually exclusive (or one column always implies another), so these columns have instead been replaced with a single type column. It will contain one of the following string values:

  • built-in
  • plugin
  • custom
  • keyword
  • external
  • alias

which [toc]

Breaking change

See a full overview of the breaking changes

Related to the changes for scope commands, the type column reported by which can now be:

  • plugin or keyword instead of built-in
  • external instead of custom for declared extern commands

describe [toc]

Breaking change

See a full overview of the breaking changes

With the --detailed flag, the stdout, stderr, and exit_code columns have been removed for external streams / byte streams. Instead, the origin column now reports unknown, file, or external for byte streams, and the type column will be byte stream, string (stream), or binary (stream).

Streaming more commands [toc]

This release changes a bunch of commands to stream where possible (#12897):

  • first and last now support byte stream input and will return the first or last specified number of bytes (without collecting the whole stream into memory at once).
  • take and skip also now support byte streams input and will return new byte streams.
  • str join will now stream string output if it is given a stream of string values.
  • bytes collect can also stream its output now.
  • bytes ends-with now supports byte streams and does not need to collect the entire stream output in memory at once (#12887).
  • from json --objects now streams its output if its input is a stream (#12949).
  • into binary and into string will now apply a type hint to byte streams and will simply return the input stream without consuming or draining it.

cd and pwd [toc]

Thanks to @YizhePKU in #12603, the cd command now sets $env.PWD to a logical path by default. Instead, there is now a --physical flag for cd which will canonicalize the path before setting it as the PWD. Similarly, pwd now has a --physical flag to resolve symlinks before reporting the current working directory.

collect [toc]

After #12788, the closure parameter for collect is now optional. If no closure is provided, the collected pipeline input is simply returned as a value.

complete [toc]

Previously, if an external command was terminated by a signal, the exit code returned from complete would be -1. Also, if the external command core dumped, then an error value would be present in the exit_code column. Now with #12774, if an external command was terminated by a signal (this includes core dumps), then the exit_code column will contain the negation of the signal number (to differentiate it from a regular exit code).

Also, the o+e>| combined pipe redirection now works properly with complete. That is, the stdout and stderr of the preceding external command are merged prior to be passed to complete, and the combined output will be present in the stdout column of the record returned by complete (#12818).

each [toc]

Thanks to @merelymyself, the each command now bubbles up errors in certain cases where it would previously ignore them (#12412).

ls [toc]

In a similar change, the ls command will now return errors instead of empty entries if it fails to list a path (#12033). Additionally, ls has received a bug fix with #12625 so that it works in a directory with [ or ] in its path.

bytes build [toc]

With #12685, the bytes build command now accepts integers as rest arguments (each integer number is treated as a byte).

from nuon and to nuon [toc]

The nuon commands can now serialize and deserialize cell path values after #12718.

save [toc]

With #12833, custom converters now work again for the save command. E.g., save output.custom now uses the to custom command if it exists (note that this is currently not done for singular string value inputs).

hide-env [toc]

A long-standing bug with hide-env is that it would not hide environment variables from external commands / child processes in certain cases. With #12901, this issue has finally been fixed.

PWD fixes [toc]

Some commands are incorrectly using the current working directory of the shell process, instead of the internal $env.PWD tracked in the engine state. This release fixes a few of these commands:

nu-highlight [toc]

nu-highlight has received two bug fixes. Namely, highlighting not has been fixed (#12815) and the extra output present when highlighting list spreads has been fixed (#12793).

into sqlite [toc]

Columns with spaces are now supported by into sqlite, since it will wrap the columns with backticks (#12759).

stor insert [toc]

Thanks to @ExaltedBagel in #12820, stor insert now escapes single quotes.

decode base64 and encode base64 [toc]

The alphabet used for --character-set binhex has been corrected in #12757.

char [toc]

In #12867, an issue with char was fixed where it can panic if a list is spread after the --integer or --unicode flags.

Deprecated commands [toc]

sys [toc]

With the removal of lazy records, the sys command will now return a regular record. This means it has to compute all of its columns before returning. Since the sys command samples CPU usage over a 400ms period, then this would be unideal if the only information you needed from the command was, e.g., the host information. Because of this, the command has been deprecated in favor of the new sys subcommands as listed in new commands.

str contains --not [toc]

This release deprecates the --not flag for str contains, since none of the other string commands have a similar flag (#12837). Instead of the --not flag, you can use the not operator in one of the following ways:

# using parentheses
not ("foobar" | str contains foo)

# using a pipe with the `$in` variable
"foobar" | str contains foo | not $in

Removed commands [toc]

lazy make [toc]

The lazy make command has been removed in #12682. See removal of deprecated features for more information.

describe --collect-lazyrecords [toc]

The --collect-lazyrecords flag has been removed from describe, since lazy records have been removed from the language (#12682).

List of environment variables support in with-env [toc]

This release removes support for the list of environment variable form of with-env that was deprecated in the previous release. Going forward, only the record form is supported.

For plugin developers [toc]

Breaking API changes [toc]

  • The ExternalStream type has been replaced entirely with ByteStream, which represents a single (possibly typed) stream of bytes. Through the plugin layer, only the Read variant is used. It is no longer possible to get stderr or exit_code from an external command by piping it into a plugin command directly.
  • The ListStream type now requires a Span.
  • Config's shell_integration field has been modified and split into several fields. It was previously a single bool.
  • LazyRecord has been removed, but this was not allowed in plugins anyway.

Breaking protocol changes [toc]

Breaking changes [toc]

  • #12939 Remove list support in with-env
  • #12894 fix range semantic in detect_columns, str substring, str index-of
  • #12930 Small improvements to debug profile
  • #12889 Remove dataframes crate and feature
  • #12897 Add string/binary type color to ByteStream
  • #12832 Use CommandType in more places
  • #12774 Replace ExternalStream with new ByteStream type
  • #12846 Fix sys panic
  • #12837 remove --not flag for 'str contains'
  • #12747 Add sys subcommands
  • #12669 Polars lazy refactor
  • #12524 ListStream touchup
  • #12682 Remove lazy records
  • #12629 overhaul shell_integration to enable individual control over ansi escape sequences

Full changelog [toc]