0.1.1 Newshell Changelog

[caribpa]

@ret2libc mentioned in Mattermost that the following wizardry works thanks to the newshell changes: (_;?en Function: ;fd.;pi 1 @@b;?e)() @@F:fcn.000182* (command chaining works).

While that seems like what every console hacker would dream of, I'm curious to learn what other magic can be performed with the new changes (and vs oldshell).

This said, I think this could be the place to start recording the new functionality so that the 0.1.1 newshell release notes (and later the Rizin's book newshell chapter) won't be so boring to write 🙂

[ret2libc]

User changes:

• escaping special chars like: ?e Hello\@World prints out Hello@World or $foo=\#!pipe echo hello (usually #is for comments)
• afvn "My Argument Wrapped in quotes" old_name, afvn 'my@arg;with>whatever' old_name
• proper wrapping of arguments, like: w "Hello World " writes the string Hello World (spaces included), while with oldshell you'd have to do "w Hello World ". Related: w "Snoo\"ping as\" usual," while in oldshell w Snoo"ping as" usual,
• proper command substitution, even nested as one would expect: pd 1$(?e 1$(?e -10))
• w "Hello $(?e World)" substitution works like in bash, while w 'Hello $(?e World)' substitution does not work like in bash. Note that there are no way in oldshell to do substitution and at the same time considering spaces because "w Hello $(?e WOrld) " would print everything "as-is" (actually $( are converted to another char)

Dev changes:

• a grammar file shlr/rizin-shell-parser/grammar.js that describes the entire grammar supported. This was a huge step forward, considering that there was no single place where all things accepted by the shell were described.
• more "structured" way of dealing with complex statements. For example, if you look at cmd.c you clearly see how iter_flags_command (the code for @@f:) works and you can see that it just iterates over the flags and it executes whatever command is being passed on the left of @@f:

The above points come mainly from having moved to tree-sitter and a proper grammar (which is still not perfect, but it can be easily improved and there are tests even at the grammar level, see shlr/rizin-shell-parser/corpus). Also, some things, in particular with regards to escaping, work only if the command was converted to newshell (this is because the oldshell command handlers would parse the input themselves, so it's practically impossible to change their arguments without breaking some commands functions).

When we talk about "newshell" though, we talk also about the new way commands are registered to the system. In oldshell the input is parsed char-by-char with nested switch-case nested in other switch-cases until the proper code is reached. That code usually has to handle itself the arguments, split them, escape/unescape, check if there are the right number of arguments, etc. Moreover, you could never really know, in the code, whether a command exists or not.

User changes:

• (almost) uniform way of displaying the help for each command (see existing issues about dealing with sub-modes, subcommands, short/long help, etc.)
• automatic way of constructing commands-tree help, based on the YAML file which is also used to define the commands handler. This means that if a command can be executed, it is also listed in the help. If it is not listed in the help, it does not exist. Right now, instead, you'd have to make sure that the help strings are in sync with the code and many times they are not.
• automatic way of constructing arguments when showing help, based on the YAML. In oldshell you never know whether a command accepts 1,2,3,n arguments and if they are optional or not. With newshell the arguments strings (e.g. zfd <filename>) are created automatically based on the YAML description. If you pass less/more arguments than expected to a command, the command handler is not called and the command help is shown. Sure, you could still have the C code and the YAML file not synced (e.g. the YAML says one argument is optional, but the C code expects it to be always present), but there's not much we can do about it.
• provide a longer description with examples for each command, described in the YAML file. As the short summary usually printed near the command name is not always enough to understand what a command does, we chose to provide longer descriptions (currently accessed by appending ?? to a command).
• autocompletion based on the YAML file. In oldshell you had to list (in core.c) what type of argument a command accepted. This was yet another file to modify and it often resulted in commands being forgotten there. Also, autocompletion was usually done only on the first argument. With newshell autocompletion can be done based on the YAML file, which also has information about the type of each argument accepted by a command. Based on the YAML info and the fact that tree-sitter can easily split the input commands in arguments (considering also quotes, etc.), we can now autocomplete without any effort, since when adding a command you are forced to specify its arguments and their types.

Developer changes:

• much easier to write commands, because they have a prototype very similar to a main with argc/argv already split.
• much easier to read commands handlers, because of reduced indentation levels caused by switch-cases nested in switch-cases

This is what I can think about right now.

[caribpa]

* `w "Hello $(?e World)"` substitution works like in bash, while `w 'Hello $(?e World)'` substitution does not work like in bash.

With this you mean that shell expansion is also performed when using single quotes, right?

[ret2libc]

No. I mean that command substitution works within double quotes, but it doesn't work within single quotes. The behaviour should be the same as bash, right?

[caribpa]

Ah, OK, I read it like "substitution does not work like in bash", instead of "substitution does not work, like in bash", my bad 👍

[ret2libc]

Ah, as a side effect we also have the possibility for RzCorePlugins to actually register new commands when they are loaded, which allows us to list those commands in the same way as all the other commands, provide autocompletion for them easily and all the benefits of the more rigid parser (e.g. quoting, etc.).

Before, a core plugin had a call method that was called every time a command was executed (no matter what the input was, even wx 90 would be passed to all the core plugins registered), to check if the plugin was able to deal with that input, somehow. The core plugin had to split the arguments and do the usual parsing itself. Moreover, it was impossible for the core to know which commands were actually registered, so there was no good way to show the available commands.

For example, see the java core plugin that is being moved to newshell in #671 . java command was not visible before in the help, while it is once moved to newshell, because the core is aware of the registered commands. This is a big improvement for plugins development IMO and it improves commands discoverability and usability. Also, autocompletion is automatically supported because it is based on the types of arguments provided by the plugin itself. With oldshell you can't autocomplete these things, unless you assume to know all plugins that are going to be loaded beforehand.

[caribpa]

as a side effect we also have the possibility for RzCorePlugins to actually register new commands when they are loaded, which allows us to list those commands in the same way as all the other commands

Was it the intended behavior, or an accident that will become a feature?

What happens if a plugin overrides core commands (or two plugins define the same keybinding), is a warning/error issued? which one has preference?

[ret2libc]

Was it the intended behavior, or an accident that will become a feature?

It was not an accident, just the result of good design imo :)

What happens if a plugin overrides core commands (or two plugins define the same keybinding), is a warning/error issued? which one has preference?

It shouldn't be able to override core commands (there is actually a bug listed at #701 (comment)). The first that registers a command wins. It is of course possible, in theory, for a plugin to delete an already registered command, but.... Well, we don't think it's something that should be done. I mean, if a C plugin wants to mess something, it can do it anyway.

[caribpa]

Good to know 😄

Talking about plugins, there's currently no documentation regarding how to write a plugin (like radare2 has). It would be really helpful to have a tutorial regarding how to create a real plugin, for example adding BOF analysis (just wrapping exploit-bad-characters and calling expdevBadChars for memory comparison, as layering mona.py will be a lot of work).

[ret2libc]

Well... it is essentially the same right now :)

[ret2libc]

Actually... my bad I did not notice you just mentioned changes for 0.1.1 :( :( I haven't checked yet but I don't think there are major changes compared to 0.1.0

[caribpa]

So, converting to newshell is a priority for 0.2.0, right?

[ret2libc]

Yes, but if we wait until all commands are converted to newshell we will release in... dunno, a lot of time likely :)

[caribpa]

Ok, this is why setting milestones for converting each "command group" would be beneficial to track the process as well as providing regular "stable" releases, helping the project gain visibility, and distro packers distributing it.

[caribpa]

Let's say that "a lot of time for the commands to be converted" means 1.0.0, then what are the most important/common group that should be converted for 0.2.0? I'm asking about only 1 group/sub-group/sub-command.

[ret2libc]

The most used commands are probably i and p. But p is too big/too messy right now to expect it to be done in one shot, so we should probably split it and start converting some parts of it. Probably the most used p sub-group used is pd.

[XVilka]

Note, that 0.1.1 has been released: https://github.com/rizinorg/rizin/releases/tag/v0.1.1

[ret2libc]

Is it possible to "close a discussion"?

[XVilka]

@ret2libc looks like impossible for now:

• Close discussions without deleting them community/community#3136
• Close discussions community/community#3097

[caribpa]

The workarounds suggested in community/community#3097 are:

• Close discussions community/community#3097 (comment) ➡️ Create an Archive category and move the discussion there
• Close discussions community/community#3097 (comment) ➡️ Lock the discussion and use the is:unlocked filter when searching discussions

I don't have the necessary permissions to perform any of the mentioned workarounds (I can move the discussion to a different category but I cannot create new categories) so feel free to lock and/or move the discussion (or notify me if for some reason you cannot move it) 🙂