Nice to see that ksh is still alive

[kfl02]

I am the author of the bash style history expansion (src/cmd/ksh83/edit/hexpand.c), which I wrote for dgk in 2002 because I was bored a bit and he had it on his to-do list. It was tied to SHOPT_BASH I think, but this seems to be gone now, which might be a good sign.

I wrote some more small bash compatibility stuff about 12 years ago, but the source code is lost and I never got it to dgk. Honestly, I don't remember what I did.

Keep up the good work!

Cheers,
Karsten

[McDutchie]

Welcome Karsten! Thanks for stopping by, I hope you stick around.

History expansion itself was not tied to the bash mode, but the histreedit and histverify options were -- I've made those available using the regular 'set' command.

Relevant commits: 7b25709, 0761824, f792c4b, 89184ea, 893df4d, 0a71a8a, d507847, f1e5a19

On my wishlist: allow multibyte histchars, so I could use, e.g., § instead of ! (my Mac has a § key that rarely gets used and would be great for that purpose). I might work on that sometime, but if you'd like to and have the time, please feel free to go right ahead.

[kfl02]

Hi Martijn,

I'm pretty sure that in my initial code history expansion was tied to bash mode. Probably dgk moved it at some point of time.
So the bash mode is completely gone now, and with it the shopt builtin? Because in bash histreedit and histverify are shopt options, not set -o options.

Honestly, I never used history expansion, neither with ksh nor with bash. To me it's a rather annoying feature. How many times did bash f**k up my command line, because I accidentally typed an unquoted ! somewhere? Countless times. So I always turn it off.

Regarding multibyte characters, I think my implementation is completely broken anyways. I have no idea how ksh handles multibyte characters, but I would look into how it's done in edit/completion.c. There are many calls to mbchar(). I guess every string pointer increment in hexpand.c would have to be replaced with that.

Remember that the characters for history expansion are taken from the environment variable histchars, when set (see hist_setchars() in hexpand.c). § is still in ASCII range, so you could probably just get away with simply setting histchars to "§". You can set up to three characters there. If there are none or less than three are set, the defaults are '!', '^' and '#'. For fully enabling multibyte characters you would have to treat the histchars variable accordingly, that means changing the hist_setchars() function, any access to hc[3] array in hist_expand and in edit/completion.c, fmtx() as well.

I hope that helps a little.

I don't know if I will have time for looking into it, as I'm just applying for a new job. I stumbled in here, because I was looking for something interesting to put into my CV, and having had the honor to work with dgk and gsf surely was interesting :)

[McDutchie]

So the bash mode is completely gone now, and with it the shopt builtin? Because in bash histreedit and histverify are shopt options, not set -o options.

Yep, it's been gone for about four years, see #9. In the 93u+ 2012-08-01 release, it didn't even compile; it had a typo in the code, and after fixing the typo, the compiled code failed to link. Clearly, dkg et al had stopped caring (and this is far from the only way in which the code base was a mess, I'm afraid).

I didn't want to spend my own limited time fixing and maintaining the bash mode -- what's the point of it when the real bash is free (in both senses) and universally available? One or two people said something about not throwing things away, so I called for volunteers to fix it and waited a few months. No one came forward, so I got rid.

However, a few things from it survived, that I made available as regular ksh 93u+m features. These include the &> redirection shorthand, the idea of a posix option, and (as of 7b25709) the histreedit and histverify options, now as regular set options.

(The fact that bash has two arbitrarily separated commands/namespaces for options grates me to no end, by the way. I often fail to remember which option belongs to which command. But I believe in eating my own dog food, so I've been using ksh full time for the last four years -- problem solved ;-))

Honestly, I never used history expansion, neither with ksh nor with bash. To me it's a rather annoying feature. How many times did bash f**k up my command line, because I accidentally typed an unquoted ! somewhere? Countless times. So I always turn it off.

Ha! Same here. It's funny you would feel that way too, since you wrote it :-)

At least it's not enabled by default on ksh, and you'll be happy to hear I'm not changing that. But many people like and use it, so it's staying available. However, I find that enabling both histreedit and histverify makes it tolerable to use; you don't lose your command line if a history expansion fails, and you get to verify it before executing.

I've also fixed some issues, the latest of which is that a ! right after ${! now gets ignored by history expansion so that you can use ksh indirection expansions like ${!varname} and ${!prefix@}.

History expansion is now also fully documented in the manual page, by the way. I took the section from the tcsh(1) page and adapted it.

I have no idea how ksh handles multibyte characters, but I would look into how it's done in edit/completion.c. There are many calls to mbchar(). I guess every string pointer increment in hexpand.c would have to be replaced with that.

Yes, that's right. Those are macro expansions, not function calls (although some of the macros call internal functions). Note that mbchar(cp) yields the current multibyte character and increases the cp pointer, which confused the heck out of me for a good long while at first.

The AT&T folks generally really seemed to like their unsafe function-like macros -- there are many more besides the multibyte ones. It is what it is, and I can't be bothered to redesign all that; it's livable-with if you're aware of it and avoid passing arguments with side effects.

I fixed a nasty bug in multibyte handling early on, see #5. I've since refactored them somewhat and fixed some more issues with them. If you don't need multibyte, you can now globally disable it for both libast and ksh by adding -DAST_NOMULTIBYTE to your CCFLAGS.

§ is still in ASCII range, so you could probably just get away with simply setting histchars to "§".

Nope, that's not ASCII. I use a UTF-8 locale, so § is encoded in two bytes and won't work. If you use an ISO-8859 locale, it will have that character as a single byte with the high bit set, and that should work fine.

I'm aware of the histchars variable and the hc array -- in d507847, we separated out setting hc[] in a function and made that available to completion.c so that tab completion could properly escape the current history expansion characters.

I appreciate your hints and your kindness, though. Again, thanks for stopping by. Good luck with the new job! I hope you may find some time to give the new ksh version a try and let me know how you get on with it.

[kfl02]

Thanks to git, nothing is really thrown away. Or probably, it is, since the ast developers had been using their own versioning system (based on 3dfs, I believe).

I briefly looked into the discussion on the bash mode removal, especially https://github.com/ksh93/ksh/issues/9#issuecomment-643702885. I have the vague recollection that those undefined symbols came from me :)
The bash_rc stuff was about the different start up files (.bash_profile, .bash_login, .bashrc etc), sh_bash1 was initialization stuff that ran before ksh initialization, sh_bash2 was called after that.

I found an old version of ast on my Google drive. It might even contain the stuff I mentioned in the first post, that I considered lost. I will look into it and see what could still be useful. Maybe there are ways to get some things into ksh without a dedicated bash mode.

I've been rolling my own builds of ksh for many years and preferred using it over bash. Ksh is superior to bash in many ways: floating point variables, discipline functions and compound variables come to my mind. And it's way faster (bash's crappy hash algorithm for associative arrays can grind down the speed of your script massively if you have to process a only few thousand items).
OTOH, bash has some great features, like the programmable completion, that are missing in ksh.

The macros, yeah, I know. Sometimes macros expand to a simple function call, sometimes they do a hell lot of things, often depending on the output of iffe. The goal for the ast library was to run and build on every UNIX system out there. In many cases those macros could be replaced by inline functions. But that would require at least C99. When I worked with dgk and gsf, not every C compiler had C99 implemented.
The real master of macros was gsf. When I helped him to port ast to Cygwin, he found so many tiny little bugs and subtleties inside Cygwin he had to work around, so that the number of macros grew massively. (Many of those bugs have been fixed though, after I reported gsf's findings to the cygwin devs.)

[kfl02]

Just downloaded the old ast stuff.

I was wrong about sh_bash1 and sh_bash2 stuff. These were just documentation strings, but they are present in the old archive.
e_bash_rc was just the string "$HOME/.bashrc".

At least I seem to be having a complete version, so I can dig around a bit. I'm certainly not going to reimplement the bash mode.

[McDutchie]

Thanks to git, nothing is really thrown away. Or probably, it is, since the ast developers had been using their own versioning system

Yeah. Unfortunately, the AT&T RELEASE files are so sloppy, incomplete and inaccurate, they're little use.

Thankfully, @JohnoKing restored as much history from published AST tarballs as possible, in the ast-open-archive and ksh93-history repos (the latter being the former without most of the non-ksh AST stuff). He was even able to make most of them buildable on modern Linux systems. So I've got a stash of 198 compiled historic versions on a Linux VM, from 2004-10-11 to 2016-01-10. In many cases, this work has made it possible for us to do the archaeological research necessary to isolate the changes that introduced bugs, so we could fix them reliably -- and to backport a considerable number of bug fixes from the latter-day unreleased 93v- beta versions as well. (The AT&T ksh/AST team were certainly introducing bugs faster than they were fixing them in their latter years, which is why we forked 93u+ and not 93v-, but of course I'll still use their bug fixes! Also, @JohnoKing backported the new and much faster ${.sh.match} mechanism from 93v-, which is very nice.)

As for ksh 93u+m, if you look through this repo's commit history, you'll notice I am obsessive about documenting each and every significant change and the reasons for them -- and I expect the same of others. I spend about as much time documenting the changes as I do coding them. This is what I always do anyway, because my own memory is like a sieve, and I really need my own documentation a week after committing something. But of course it also allows contributors to actually understand each others' changes without doing advanced detective work.

(based on 3dfs, I believe).

(I removed 3dfs and libcoshell support early on because no one outside of AT&T was using these, and they were obsolete, buggy and racy. There have been zero complaints about this.)

(bash's crappy hash algorithm for associative arrays can grind down the speed of your script massively if you have to process a only few thousand items).

(On the other hand, ksh's associative arrays leak memory like crazy and I've not been able to fix it so far; see #94.)

I've been rolling my own builds of ksh for many years and preferred using it over bash. Ksh is superior to bash in many ways: floating point variables, discipline functions and compound variables come to my mind.

Yes. However, there is a serious arithmetic problem on 64-bit systems, especially ARM, and it's because of a fundamental design flaw in the arithmetic subsystem: all arithmetic is internally done in floating point, even for integer values – but not all 64-bit long integer values can be represented exactly in even the largest floating point types, especially on ARM systems which don't support long double, but also (to a lesser degree) on x86_64. The results are about as disastrous as you might imagine. See #771 (amazingly, no one had ever reported it – I discovered while fixing #770!). Until we can fix this (if ever), ksh should not be used for serious arithmetic, or any that might involve large integer values. Not that people are doing that, or this would have been reported long ago.

Hey, would fixing #771 look good on your CV? ;-)

The goal for the ast library was to run and build on every UNIX system out there.

It still is. That's one aspect I kept. This code still builds and runs on all sorts of obscure and old systems and I'm proud of that. The build system still depends only on a standard POSIX shell, utilities and compiler environment (with tolerance for shell bugs) -- it doesn't need make or anything, it does it all itself.

In many cases those macros could be replaced by inline functions. But that would require at least C99.

Yeah. I already got rid of huge amounts of K&R and C++ compatibility cruft (see a34e831, 1064933 and the commits that reference these), so we require C90 now. But I will move to requiring C99 pretty soon. That's 25 years old now, so it's time, even given my goal of staying extremely compatible with old and obscure systems.

There are other problematic obsolete things in the code, mainly the struct hack, which is undefined behaviour, and it's used a lot in libast and ksh. It works in practice because C90 had no alternative and compiler implementers just made it work regardless of what the standard said. But C99 does have a good construct for exactly that use case (flexible array members), so we should use it. There is no guarantee the struct hack will still work in practice 25 years from now.

The real master of macros was gsf.

I have certainly noticed this. :-)

I've found Exuberant Ctags invaluable. Given an up-to-date tags file, it will very quickly find me a symbol's definition so I can see whether it's a function or macro, and what it does. I spent the first few years grepping the source code and getting very frustrated, and then I discovered Ctags, and whoa boy, did my life improve then.

[kfl02]

I'm going to be a bit off topic now - just sharing memories,.

The AT&T or Bell guys had great ideas, without them, we wouldn't have UNIX at all.
A good read is the book "Practical Reusable UNIX Software" (ISBN-13 ‏ : ‎ 978-0471058076). It it pretty much outdated, but captures their ideas in a good way (I own a copy signed by dgk, gsf and kpv, see the attached pictures).
I know personally that dgk, gsf and kpv were eager to release their software publically under an absolutely free license, so that everyone could participate, but they weren't allowed - AT&T insisted on everything being covered by licenses that where not compatible with any common open source license.

I worked with UWIN in the early 2000s. UWIN was absolutely not free, a commercial product.
I discovered many bugs in UWIN, but I had no access to the source code. dgk broke the code of conduct many times and sent me pieces of the source code, so that I could investigate, because he could not reproduce the bugs I found.

I visited the labs in 2002 when I was on vacation in NYC. dgk and gsf invited me to have talk there about my work with UWIN. Of course I did this. It was really inspiring day, especially seeing gsf wearing shorts and toesies and discovering that dgk's office was even more messy than mine. And they showed me the office space where Bjarne Stroustrup invented C++.

dgk then asked me if I had some spare time. It was about the bash compatibility mode. His idea was to have ksh emulate any Bourne shell descendant shell out there, the most prominent being bash and zsh. Of course I opted in. Most of the stuff that is now deleted came from me. I wrote it over a few weeks, history expansion took me a day or two.

Bash mode was clearly dgk's idea, I only contributed as much as I could in my spare time. I feel no anger that my code has been removed. At least history expansion is still there, which I would not recommend to use anyways :)

[McDutchie]

Thanks for the memories!

I picked up a second-hand copy of that book a couple of years ago at the recommendation of Siteshwar Vashisht (who was also generous enough to hand this “ksh93” GitHub organisation account over to me). My copy seems to be signed by the editor, Balachander Krishnamurthy, but I just picked it up from a random second-hand bookseller via Amazon.

I definitely try to operate in the spirit of that book, and keep all the good ideas. iffe, for instance, is a brilliant concept (so much nicer than autoconf) and, since it's just a shell script, we've been able to keep, maintain and improve it. nmake, on the other hand, proved very brittle, and too complicated to maintain (it seems as complex as ksh itself). No one has time or resources for that now.

So I improved the very small mamake fallback program instead, giving us much of the flexibility back that nmake was supposed to provide. I added a few features (e.g., automatic variables and a simple loop construct) plus some helper scripts called from shell actions to replace code generated by custom Nmake rules. It's working very nicely now. I find the new-style Mamfiles actually easier to work with than Makefiles. As of a few days ago, mamake can build the code in parallel. But it is still a small and simple program. Working on mamake has been a breeze, a welcome break from the inscrutable complexity of ksh. "Do one thing, and do it well": the original UNIX spirit.

In my view, the AST team had well and truly lost track of that spirit. Instead, it became more like "Do everything, not very well". Korn wanted ksh to emulate every Bourne-derived shell, you say? Well, that neatly illustrates the problem right there. The AT&T ksh code base is full of evidence of this sort of mindset, and I'm afraid it suffered badly for it.

Don't get me wrong though, Korn was also a genius and his work deserves respect and use. Many of the ideas and some of their execution are brilliant. That's why this project is so frustrating sometimes. Because I also see evidence that he was capable of doing a lot better. So I treat the code with respect, but not deference.

UWIN seems interesting, a lot nicer than Cygwin. Some years ago, I had a chance to try out a binary distribution that was bundled with something else, I can't remember now. I liked how it exposed the Windows registry as UNIX files. But we've been unable to figure out how to build the code at https://github.com/att/uwin. It seems like it needs UWIN with a compiler to build, and that just does not seem to be available anywhere. Given that and the clear lack of general interest in UWIN, unfortunately we've had to remove UWIN support from the ast/ksh code base to avoid excessive bit rot. There were thousands of lines of UWIN-specific code -- they clearly cared about it a lot.

David Korn and Glenn Fowler seem like nice people IRL. Thanks for sharing those memories. I don't know what I'd do if I ever met them. I doubt they would like very much what I have to say.

As for history expansion, at some point I'll fix it enough to want to use it. ;-) Thanks for being gracious about the bash emulation removal.