FR: better error for ssh hostname changes

[fowles]

I recently tried to use jj to push to github. The error I got was very perplexing to me. In particular I saw this.

[sonmi]~/dev/protobuf$ jj git push
Branch changes to push to origin:
  Add branch test_jj to 5a9dbd9a4edc
Error: invalid or unknown remote ssh hostkey; class=Ssh (23); code=Certificate (-17)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

I eventually traced this down to the recent github host changing

[sonmi]~/dev$ git clone [email protected]:fowles/abseil-hello.git
Cloning into 'abseil-hello'...
The authenticity of host 'github.com (140.82.114.3)' can't be established.
ED25519 key fingerprint is XXXXXXXXXXXXX
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Warning: the ED25519 host key for 'github.com' differs from the key for the IP address '140.82.114.3'
Offending key for IP in /usr/local/google/home/kfm/.ssh/known_hosts:6
Are you sure you want to continue connecting (yes/no)? yes

so this fixed it, but an error message that told me something more (or better yet gave me the prompt that git does) would be awesome.

[necauqua]

While this is true and we need a better human error, the ssh -F /dev/null hint was added because of this - did it not help you trace it down to the github thing right away?

ssh [email protected] would fail with a giant MITM error, prompting you to do ssh-key -R github.com and then ssh [email protected] again to get the new fingerprint.

[fowles]

truthfully I did not try ssh [email protected] because I just assumed there is no possible way that github left the default port for ssh open in that way. I am used to thinking of ssh solely in terms of machines I have remote terminal access to and I don't have remote terminal access to github. I suspect that if the suggestion had said to try ssh -F /dev/null [email protected] I would have tried that.

The end result is that I actually spent about 45 minutes trying various stupid things before figuring out the workaround I did.

[necauqua]

default port for ssh open in that way

Well what port would the git clone over ssh use then? :)

I suspect that if the suggestion had said to try ssh -F /dev/null [email protected] I would have tried that.

That should be an easy partial fix for the hint then, thanks, makes total sense that people don't think about ssh-ing to github.com like that.

Ideally ofc the hint would be specifically for when host identification changes like it happened with github, libssh2 errors dont give too much information though it seems

[fowles]

Well what port would the git clone over ssh use then? :)

Honestly, I had never thought about it. It is funny the abstractions we just accept :)

[martinvonz]

I suspect we should add a certificate_check callback and check known_hosts there but that's just a guess.

[clintonc]

Relatedly, when I attempted jj git clone [email protected]:..., I got the following error both before and after removing github.com from my known hosts:

Error: Fetch failed: invalid or unknown remote ssh hostkey; class=Ssh (23); code=Certificate (-17)

When I did a normal git clone with the same argument, I got the normal new host flow:

The authenticity of host 'github.com (192.30.255.113)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com,192.30.255.113' (ECDSA) to the list of known hosts.

In this case, there was no error message to help me figure out (or remediate) the problem.

[martinvonz]

If you install jj from head, you should at least get a hint there:

jj/src/commands/git.rs

Lines 165 to 181 in 0c36b1a

	fn map_git_error(err: git2::Error) -> CommandError {
	if err.class() == git2::ErrorClass::Ssh {
	let hint =
	if err.code() == git2::ErrorCode::Certificate && std::env::var_os("HOME").is_none() {
	"The HOME environment variable is not set, and might be required for Git to \
	successfully load certificates. Try setting it to the path of a directory that \
	contains a `.ssh` directory."
	} else {
	"Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F \
	/dev/null` to the host work?"
	};
	user_error_with_hint(err.to_string(), hint)
	} else {
	user_error(err.to_string())
	}
	}

I'm guessing jj git clone works now after the successful git clone?

[clintonc]

I installed jj from head (using the cargo command above), and a hint does show then. Cloning does not work afterwards (just hangs; verbose logs just shows a lot of lines like INFO fetch{remote_name="origin" branch_name_globs=None git_settings=GitSettings { auto_local_branch: true }}: jujutsu_lib::git: using ssh_key_from_agent username="git", eventually failing with Error: remote rejected authentication: Failed getting response; class=Ssh (23); code=Auth (-16)) but that's a separate issue.

[zzhaolei]

(I turned on VPN (quantumult x).Yes, I have to turn it on.)

I encountered a similar situation
https is ok, but ssh failed:

11:37 ❯ jj git clone https://github.com/zzhaolei/jj-demo.git
Fetching into new repo in "/Users/zhaolei/Study/jj-demo"
branch: main@origin [new] untracked
Setting the revset alias "trunk()" to "main@origin"
Working copy now at: srltlnwl d6627064 (empty) (no description set)
Parent commit      : luoyttxn 58cd234f main | add gitignore
Added 2 files, modified 0 files, removed 0 files
11:37 ❯ rm -rf jj-demo/
11:37 ❯ jj git clone [email protected]:zzhaolei/jj-demo.git
Fetching into new repo in "/Users/zhaolei/Study/jj-demo"
Error: failed to start SSH session: Failed getting banner; class=Ssh (23)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

11:39 ❯ ssh -F /dev/null [email protected]
Connection closed by 198.18.1.134 port 22

git erverything is ok:

11:40 ❯ git clone https://github.com/zzhaolei/jj-demo.git
...
11:40 ❯ rm -rf jj-demo/
11:40 ❯ git clone [email protected]:zzhaolei/jj-demo.git
...

11:42 ❯ ssh -T [email protected]
Hi zzhaolei! You've successfully authenticated, but GitHub does not provide shell access.

~/.ssh/config and ~.gitconfig has no configuration for github.com

[yuchenshi]

I'm guessing jj git clone works now after the successful git clone?

@martinvonz I just ran into this when cloning for the first time on a new device. The ssh -F /dev/null hint was helpful and prompted me to accept the public key for github.com for the first time. And then it worked perfectly fine. Does libssh2 have a way to add the key to known hosts (after user confirmation)?

[martinvonz]

Does libssh2 have a way to add the key to known hosts (after user confirmation)?

Yes, looks like it: https://libssh2.org/libssh2_knownhost_addc.html There's also #3191 for switching from libssh2 to OpenSSH, which should solve lots of problems related to ssh config.