I am using libssh2, where my flow ends up calling _libssh2_userauth_publickey, which itself calls _libssh2_ntohu32(pubkeydata). My understanding is that it should return a small number (e.g. 7 for "ssh-rsa", but in my case it returns garbage (e.g. 813826570).

My guess is that the pubkeydata I am giving is in the wrong format (it should start with something like 00 00 00 07 for "ssh-rsa", and obviously it doesn't).

The bytes I am passing as pubkeydata are my id_rsa.pub converted to PEM and then to bytes, using e.g.:

ssh-keygen -e -m pem -f id_rsa.pub | openssl asn1parse -dump -out pubkeydata.bin

But this doesn't start with 00 00 00 07, which makes me believe this is not what _libssh2_userauth_publickey expects.

What format does _libssh2_userauth_publickey expect, and how do I convert my id_rsa.pub to it?

I am using libssh2, where my flow ends up calling _libssh2_userauth_publickey, which itself calls _libssh2_ntohu32(pubkeydata). My understanding is that it should return a small number (e.g. 7 for "ssh-rsa", but in my case it returns garbage (e.g. 813826570).

My guess is that the pubkeydata I am giving is in the wrong format (it should start with something like 00 00 00 07 for "ssh-rsa", and obviously it doesn't).

The bytes I am passing as pubkeydata are my id_rsa.pub converted to PEM and then to bytes, using e.g.:

ssh-keygen -e -m pem -f id_rsa.pub | openssl asn1parse -dump -out pubkeydata.bin

But this doesn't start with 00 00 00 07, which makes me believe this is not what _libssh2_userauth_publickey expects.

What format does _libssh2_userauth_publickey expect, and how do I convert my id_rsa.pub to it?

• ssh-keys
• libssh2

• Someone voted to close this. May I know why? The question is about code: libssh2 is a C library, and I am not clear about the argument I should be passing to one of its functions. – JonasVautherin Commented Mar 27 at 15:43
• Given it puts this value in the packet, it clearly wants the wire format defined in RFC4253 section 6.6. Looking around in that source file, file_read_publickey reads a modern-OpenSSH pubkey file like your id_rsa.pub and returns exactly what you need -- and userauth_publickey_fromfile (which looks like something you were supposed to call) in one case actually uses file_read_publickey to get the 'pubkey' piece (and stash the 'method') and immediately passes exactly that to _libssh2_userauth_publickey – dave_thompson_085 Commented Mar 28 at 3:15
• Oh thanks, that's helpful! I can't call those functions (they are no exposed), but that's helpful to understand the format. Doesn't the wire format described in RFC4253 section 6.6 have a name? That would help, too :-) – JonasVautherin Commented Mar 28 at 17:48

1 Answer 1

It seems like pubkeydata is "simply" the decoded value of the base64-encoded part of the .pub file.

Say id_rsa.pub looks like this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQClFuoMbXDHrs0EWyMvWueEGz9pJjsszh8bcPGFzSmux2bFmM3G0nCTSRc+svifaTPMO+WB865tPpkpEWNNUsKPmzlKHy1zPRo5FJMmfulExJCQGKPF6B9rqJHnjILsVvAAtarDyPag/ [...] PBcEhS8YA7xn60zhXzV+0F6i6DVo6hN2TfoK85zGG59cf1lXwrUjS4hXu1XjjDavZOBBCk1Q5aIA+cVuUJWg3NE9mQYRpPSyn9L+rs+SpoS2YX/oRBD2wnlu7czU9jY6NwCedIeJI+D/aLDLqW9GNp2B4XVjvrMWa+Q==

The base64-encoded value is AAAAB3Nza...a+Q==. Decode this and pass it as pubkeydata.