Public key authentication error -- Unknown file format

[dverbern]

I admit to being pretty stressed and confused at the moment, so apologies.

I'm using VSFTP (VShell) over the command-line. I'm providing options like "-i" and specifying a path in double-quotes to the private key file, I'm specifying "-v" for verbose logging, followed by the remote username the 3rd party has set up for us, "@" the remote host server, then ":" and port 22.

This results in some verbose logging, which ends with:

Public key authentication error.
Could not load the public key from the private key file:
C:\Program Files\VanDyke Software\VShell\PublicKey\{SomeThirdParty}\{SomePrivateKeyName}

Unknown file format.

I've looked this up online and found notes from Brenda and others suggesting that VSFTP employs matching of file names so that the private key could be {Filename} without an extension, so public file should exist in same folder and be {Filename}.pub.

I've followed that guideline, but still fails to improve the situation. Do I need to massage the original private key into some format that VSFTP wants?

The original private key provided to me from vendor has extension ".ppk" and starts with the following lines:

PuTTY-User-Key-File-2: ssh-rsa
Encryption: aes256-cbc
Comment: rsa-key-20190618
Public-Lines: 6

The file goes onto include a "Private-Lines" and "Private-MAC" section.
Most of our other public keys from 3rd parties are much more compact and just start with generic text like "---- BEGIN SSH2 PUBLIC KEY ----
Comment: "rsa-key-20180301""

I'm sorry I am clearly not understanding something. I've got access to the PuTTY Key Generator utility and I've played around with loading an existing private key and generating a new pair, or just saving the public key text to a file, but ultimately I'm not sure what I'm doing or how to progress this issue.

Any help appreciated.

[bgagnon]

Hi dverbern,

The vsftp version that corresponds to VShell 4.0.4 is version 7.2.6, which did not include support for putty private keys. That support was added in ClientPack (vsftp) v8.0 (VShell v4.3.x):

Changes in the VanDyke ClientPack 8.0 (Beta 1) -- January
------------------------------------------------------------------

New features:

• vsh/vcp/VRALib: PuTTY PPK keys can be used for public-key authentication.

You will need to save the private key as OpenSSH format.

[dverbern]

Thank you Brenda for your prompt response.
Good to know the newer versions supports PuTTY format key.

I've loaded the PuTTY format private key in PuTTY Key Generator, entered the key's passphrase the vendor provided to me, chose RSA, 2048 bits then clicked in the Conversions menu and chose "Export OpenSSH key (Force new file format)" and saved file, again with *.ppk extension.

I updated my config file for the software that calls SFTP, telling it the path to the private key file, but the service just pumps out the same event:

Starting: Process incoming: Failed: Unknown file format.
at vralib.IConnection.SetPrivateKeyFile(String Filename, String Passphrase)
at Hedwig.HedwigSvc.ProcessIncoming(Boolean isStartupRun)

If I use Notepad to open the converted private key, it reads as:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktd ......

It certainly seems like it's been converted.
I don't know what I'm doing wrong.

[dverbern]

Sorry, that event viewer error was from our custom software, the native VSFTP error seems to be the same as before:

Public key authentication error.
Could not load the public key from the private key file:
C:\Program Files\VanDyke Software\VShell\PublicKey\{SomeVendor}\{SomeCertificateName}.ppk

Unknown file format.

[dverbern]

I think I have resolved my own issue.
I did the following:

1. Took the original PuTTY-format key file that vendor provided to me and loaded it in PuTTY Key Generator. Entered passphrase for the key that vendor had provided to me.

2. I chose Conversion menu and chose Export OpenSSH key and chose to save in a particular folder and gave the file a name that made it clear it was in OpenSSH format and then I intentionally left off the file extension.

2. Back in PuTTY Key Generator, I copied the public key text in that window to my clipboard and pasted into a new text file in the same folder as the private key I'd just created and gave it a file name same as the newly-created file, except I added on a file extension of ".pub" for this new file.

3. I updated my configurations and scripts and confirmed connection working!

This ticket can be resolved.

[bgagnon]

Hi dverbern,

Thanks for the update. I am glad to hear the issue is resolved.

As a note, vsftp does have its own command-line syntax. When you use a third-party software (that then runs the vsftp command-line), we still need to troubleshoot from the vsftp perspective.