VSFTP - Cannot "see" files in remote folder, not even sure if accessing remote folder

[dverbern]

I'm a SFTP newbie.
I'm trying to use VSFTP (VShell 4.0.4 (x64 build 636) to connect to a 3rd party SFTP, change to a particular remote folder and then list files and if it matches a regular expression, download those files.

This process occurs perfectly if I test (using the same SFTP server) and WinSCP, however with VSFTP (which I'm not using in isolation but by way of a suite called Hedwig which allows us to control FTP jobs and so forth) it connects to the 3rd party, shows attempts to change folder, but then moments later ends the connection, having detected no files and therefore matching no files.

I'll say at this point that we have 3rd party SFTP exchanges with many other parties successfully - both internal servers of our own and with 3rd parties. This is a NEW partner I'm talking about and i'm very new to the technology itself.

Looking at the verbose logs my suite generates, I see what looks like a successfully-authenticated session, a "CHANGE-PATH" statement attempting to change to the local subfolder called "in" on the remote server, but seemingly nothing else before the session is closed.

Here is an excerpt of some of the verbose logging the SFTP suite I'm using generates, I'm not sure if this is native VSFTP logging or not. I've bolded some lines I think are relevant:

1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : AUTH_SUCCESS
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV[0]: SSH_MSG_CHANNEL_OPEN
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : Server Sftp Version: 6
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SERVER-SUPPORTED-VERSIONS: 1,2,3,4,5,6
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : VENDOR-NAME: Extenua
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : PRODUCT-NAME: SilverSHielD SSH2/SFTP Server
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : PRODUCT-VERSION: 6.0.0
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : PRODUCT-BUILD: 0000000000000000
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SEND : fs-multiple-roots-supported request[Off]
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SEND : vendor-id request
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : Status 8, Text: Unsupported extension
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : fs-multiple-roots-supported reply: 8
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : Status 0, Text:
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SEND : RealPath, base=.
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SEND : change-path=/in
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : Names, count 1
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SEND : OpenDir /
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : Handle[1b5b6d70]
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SEND : Readdir[1b5b6d70]
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : Names, count 4
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : SEND : Close[1b5b6d70]
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : RECV : Status 0, Text:
1/07/2019 3:23:05 PM,2004,25,[LOCAL] : Changing state from STATE_CONNECTION to STATE_TRANSPORT_STOPPING

Trying the same process but using WinSCP, which we have purely for testing purposes, it generates a far different log, but shows events like:

! 2019-07-01 15:19:22.569 Using username "SomeUserName".
. 2019-07-01 15:19:22.584 Server offered these authentication methods: password,publickey
. 2019-07-01 15:19:22.584 Offered public key
. 2019-07-01 15:19:22.834 Offer of public key accepted
! 2019-07-01 15:19:22.834 Authenticating with public key "imported-openssh-key"
. 2019-07-01 15:19:22.896 Prompt (passphrase, "SSH key passphrase", <no instructions>, "Passphrase for key "imported-openssh-key": ")
. 2019-07-01 15:19:26.344 Sent public key signature
. 2019-07-01 15:19:26.578 Access granted

...

. 2019-07-01 15:19:26.718 Using SFTP protocol.
. 2019-07-01 15:19:26.718 Doing startup conversation with host.

...

. 2019-07-01 15:19:26.796 SFTP versions supported by the server: 1,2,3,4,5,6
. 2019-07-01 15:19:26.796 Server software: SilverSHielD SSH2/SFTP Server 6.0.0 (0) by Extenua
> 2019-07-01 15:19:26.796 Type: SSH_FXP_EXTENDED, Size: 63, Number: 200
< 2019-07-01 15:19:26.796 Type: SSH_FXP_STATUS, Size: 17, Number: 200
. 2019-07-01 15:19:26.796 We will use UTF-8 strings as it is mandatory with SFTP version 4 and newer
. 2019-07-01 15:19:26.796 Getting current directory name.
. 2019-07-01 15:19:26.812 Getting real path for '.'
> 2019-07-01 15:19:26.812 Type: SSH_FXP_REALPATH, Size: 11, Number: 272
< 2019-07-01 15:19:26.828 Type: SSH_FXP_NAME, Size: 91, Number: 272
. 2019-07-01 15:19:26.828 Real path is '/'
. 2019-07-01 15:19:27.030 Listing directory "/".
> 2019-07-01 15:19:27.030 Type: SSH_FXP_OPENDIR, Size: 10, Number: 523
< 2019-07-01 15:19:27.046 Type: SSH_FXP_HANDLE, Size: 13, Number: 523
> 2019-07-01 15:19:27.046 Type: SSH_FXP_READDIR, Size: 13, Number: 780
< 2019-07-01 15:19:27.062 Type: SSH_FXP_NAME, Size: 332, Number: 780
> 2019-07-01 15:19:27.062 Type: SSH_FXP_READDIR, Size: 13, Number: 1036
> 2019-07-01 15:19:27.077 Type: SSH_FXP_CLOSE, Size: 13, Number: 1284
. 2019-07-01 15:19:27.077 archive;D;0;2019-05-21T05:56:46.000Z;3;"user" [0];"group" [0];r--r--r--;0
. 2019-07-01 15:19:27.077 in;D;0;2019-06-30T18:27:10.000Z;3;"user" [0];"group" [0];r--r--r--;0
. 2019-07-01 15:19:27.077 out;D;0;2019-06-27T02:29:13.000Z;3;"user" [0];"group" [0];r--r--r--;0
. 2019-07-01 15:19:27.077 ..;D;0;1899-12-30T11:00:00.000Z;0;"" [0];"" [0];---------;0
. 2019-07-01 15:19:27.155 Startup conversation with host finished.
. 2019-07-01 15:19:29.963 Cached directory change via "in" to "/in".
. 2019-07-01 15:19:29.963 Getting current directory name.
. 2019-07-01 15:19:29.963 Listing directory "/in".
> 2019-07-01 15:19:29.963 Type: SSH_FXP_OPENDIR, Size: 12, Number: 1547
< 2019-07-01 15:19:29.963 Type: SSH_FXP_STATUS, Size: 25, Number: 1036
. 2019-07-01 15:19:29.963 Discarding reserved response
< 2019-07-01 15:19:29.963 Type: SSH_FXP_STATUS, Size: 17, Number: 1284
. 2019-07-01 15:19:29.963 Discarding reserved response
< 2019-07-01 15:19:29.979 Type: SSH_FXP_HANDLE, Size: 13, Number: 1547
> 2019-07-01 15:19:29.979 Type: SSH_FXP_READDIR, Size: 13, Number: 1804
< 2019-07-01 15:19:29.994 Type: SSH_FXP_NAME, Size: 2818, Number: 1804
> 2019-07-01 15:19:29.994 Type: SSH_FXP_READDIR, Size: 13, Number: 2060
> 2019-07-01 15:19:29.994 Type: SSH_FXP_CLOSE, Size: 13, Number: 2308
. 2019-07-01 15:19:29.994 ..;D;0;2019-06-30T18:27:10.000Z;3;"user" [0];"group" [0];r--r--r--;0
. 2019-07-01 15:19:29.994 SomeFiles.csv;-;4;2019-06-27T02:06:54.000Z;3;"user" [0];"group" [0];rw-rw-rw-;1
. 2019-07-01 15:19:33.567 Copying 1 files/directories to local directory "SomeDirectory"

Note: Obviously I've changed some lines for privacy.

Note, the SFTP automation suite we use, which utilises VSFTP and has done so for several years very successfully, involves me providing a regular expression as a means of matching with remote files. I've experimented with my regular expression, running it past other tech support staff, successfully testing it on one of the many regular expression testing websites, I've even stripped my regular expression back to very generalised search. Whilst I realise a regular expression is a place where this process could fail, I don't think it is getting as far as matching or not matching my regex at all, it's like the attempts I'm making to switch to the '/in' remote folder are itself not being processed and therefore I'm not getting a list of files at all.

I realise this is a lot of information, but any information or tips much appreciated.

[bgagnon]

Hi dverbern,

What is the vsftp command-line you are using?

If this contains sensitive data, please send email to support@vandyke.com and include Attn Brenda - Forum Thread #13609 in the subject line.

[dverbern]

Hi Brenda,

Thanks for your prompt response.
I don't actually know the exact way VSFTP is being called, because I'm using it as part of a larger SFTP automation toolkit called Hedwig, but when I drop down to command prompt to test, I've been using this synta, which I've redacted a bit:

vsftp -i "C:\Program Files\VanDyke Software\VShell\PublicKey\{SomeVendor}\{SomeVendorsCertificate}_openssh_format.ppk" -v {RemoteUserName}@{SomeFTPServer}:22

[dverbern]

I've made some progress, I've converted the vendor-supplied public key from PuTTY format to OpenSSH format to play nicely with VSFTP and I've now got a command-line syntax working and connecting to the vendor's SFTP server.

However, the underlying issue remains - I can get a listing of remote folders, but I cannot change to any of those folders. Here is an excerpt from my log of successfully connecting to the vendor's server:

[LOCAL] : SEND: SERVICE_REQUEST[ssh-userauth]
[LOCAL] : RECV: SERVICE_ACCEPT[ssh-userauth] -- OK
[LOCAL] : SENT : USERAUTH_REQUEST [none]
[LOCAL] : Authenticating as user {SomeUser}
[LOCAL] : RECV : USERAUTH_FAILURE, continuations [password,publickey]
[LOCAL] : SENT : USERAUTH_REQUEST [publickey (ssh-rsa) - unsigned,fingerprint (S
HA-2 hash): 4d:cf:49:cc:7a:e5:7d:62:af:83:34:e0:c6:78:29:6e:6f:f7:f0:54:72:63:75
:61:73:14:27:6a:42:a1:af:58]
[LOCAL] : SENT : USERAUTH_REQUEST [publickey (ssh-rsa) - unsigned,fingerprint (S
HA-1 hash): 82:7e:38:9c:d5:aa:24:70:5c:1a:98:f8:e0:cb:32:c7:82:74:a9:06]
[LOCAL] : SENT : USERAUTH_REQUEST [publickey (ssh-rsa) - unsigned,fingerprint (M
D5 hash): c6:34:12:75:7b:de:fa:27:88:75:da:51:92:63:0e:a2]
[LOCAL] : SENT : USERAUTH_REQUEST [publickey (ssh-rsa) - signed,May 2000 Standard
d]
[LOCAL] : RECV : AUTH_SUCCESS
[LOCAL] : RECV : Server Sftp Version: 6
[LOCAL] : SERVER-SUPPORTED-VERSIONS: 1,2,3,4,5,6
[LOCAL] : VENDOR-NAME: Extenua
[LOCAL] : PRODUCT-NAME: SilverSHielD SSH2/SFTP Server
[LOCAL] : PRODUCT-VERSION: 6.0.0
[LOCAL] : PRODUCT-BUILD: 0000000000000000
[LOCAL] : SEND : fs-multiple-roots-supported request[Off]
[LOCAL] : SEND : vendor-id request
[LOCAL] : RECV : fs-multiple-roots-supported reply: 8
[LOCAL] : SEND : RealPath, base=.
[LOCAL] : SEND : change-path=.
vsftp> ls -a <-- This is me checking my whereabouts and subfolders
[LOCAL] : SEND : RealPath, base=/
[LOCAL] : SEND : change-path=.
. archive in out <-- Okay, this matches what I expect, subfolders "in", "out" and "archive".
vsftp> cd in <-- Issuing "help" on this remote server has told me that this syntax should work to change remote path.
[LOCAL] : SEND : RealPath, base=/
[LOCAL] : SEND : change-path=in <-- Server seems to be following my request...
vsftp> pwd <-- This command should verify my new remote working directory.
/ <-- Darn, I'm still in the root path of the remote server. My root problem remains, I can't seem to be able to navigate to other folders.
vsftp>

Using WinSCP on the same server, reaching out to the same remote server, with the same username (admittedly using a PuTTY format key instead of OpenSSH format key), I can not only see the same folders, I can traverse them by double-clicking on them and entering them and seeing files waiting for me to download, which I can without issues.

I've just now tried using our backup VSFTP server, which is running a newer version of VShell (Version 4.4.1 (x64 build 1816), but I'm getting same results as when I try on our primary server, which is still using VShell 4.0.4.

Any more tips much appreciated.

[bgagnon]

Hi dverbern,

It looks like you are doing everything right. I did find an investigation that seems to indicate when sftp v6 is negotiated with this server:

[LOCAL] : RECV : Server Sftp Version: 6
[LOCAL] : SERVER-SUPPORTED-VERSIONS: 1,2,3,4,5,6
[LOCAL] : VENDOR-NAME: Extenua
[LOCAL] : PRODUCT-NAME: SilverSHielD SSH2/SFTP Server

There is a problem navigating the directories.

I'm guessing WinSCP is negotiating the more stable/common sftp v3.

If you add this option (--sftp-version 3) to your vsftp command-line, what are the results?

vsftp --sftp-version 3 -i "C:\Program Files\VanDyke Software\VShell\PublicKey\{SomeVendor}\{SomeVendorsCertificate}_openssh_format.ppk" -v {RemoteUserName}@{SomeFTPServer}:22

[dverbern]

Hi Brenda, thanks for your reply, I'll try adding that version switch and get back to you with results!

[dverbern]

EXCELLENT! Brenda! With that switch added on, "--sftp-version 3", I am now able to navigate folders on the remote host! Fantastic!

Now the trick will be how I can fold this explicit switch into the broader "Hedwig" SFTP automation and job management suite I'm using, but that's a matter for me I'd imagine.

I'm very happy with your support, thank you!

[dverbern]

Brenda, I've checked with the 3rd party whose SFTP server I was connecting to - they confirm their environment supports Sftp Versions 1 through 7, but have no means to make versioning exceptions per incoming client.

As I'm using VSFTP as part of a larger product I'm now trying to figure out how I might "tell" VSFTP to use this extra switch, because buried somewhere in my existing bundle of software, presumably it's passing a bunch of arguments to VSFTP and the version switch 'aint' one of them!

If you have any means at your end to recompile VSFTP to force a particular switch or anything fancy like that, that would be great to know as I might then be able to use that bespoke version of the vsftp.exe to get over our issue, but I'm assuming that's not really possible.

[bgagnon]

Hi dverbern,

There are already [options] being included in the vsftp command (ie: -i -v). It's not clear to me why one more option cannot be included.

The info I found about making this option necessary is that it is a bug in the server software so you may want to contact them.

[dverbern]

Hi Brenda,

Normally I'd agree, that I should just find how the VSFTP is being called from my software and make sure I can introduce another parameter.

The problem is, I didn't write the software I'm using, I don't have the source code and all the parameters I would use to call VSFTP are contained in an XML file and that XML has no parameter or ability to switch the SFTP Version like we have done.

I'm going to make a deeper dive into that software today and see if I can get under the hood a little, but for now I am thankful that your suggestion has got me mostly across the line. I am happy for this 'thread' to be considered solved.

Daniel -- Melbourne, Australia