Welcome to the VanDyke Software Forums

Join the discussion today!


Go Back   VanDyke Software Forums > File Transfer

Reply
 
Thread Tools Display Modes
  #1  
Old 08-24-2007, 08:42 AM
GlennStanton GlennStanton is offline
Registered User
 
Join Date: Aug 2007
Location: Old Tappan, NJ USA
Posts: 4
Optimize VShell/SecureFX SFTP transfer

we're trying to transfer a very large (46GB) compressed file over a wide area network using SFTP. so far, we're seeing about 32kbs transfer rate whereas we have a 1Mbs quota available. network is a D3 class without encryption or compression and has a pair of firewalls between the sites.

The server is VShell 2.6 and the client is SecureFX 3.1 or 4.0.

The network folks recommend trying to set the client and/or server to use more threads if possible to ensure we consume the available bandwidth (1Mbs). Is there a setting/registry entry we can modify?

On the global and session options, there is a compression setting under the SSH2 -> advanced options. should we set this to 0 since the file is already compressed heavily using Zip?

In the global settings, the send-receive buffers are 8192 bytes, should we increase that?

settings like keep-alive, PASV, etc? do these matter?

there is very little activity on the servers (both ends are W3K SP1 with 4 dual-core CPU w/ 16GB RAM) other than SQL Server - but the testing we're doing is a test box so nothing is really happening...

any other suggestions on how to optimize the VShell/SecureFX settings to maximize the transfer rate would be a great help. Thanks!
__________________
Glenn
Reply With Quote
  #2  
Old 08-24-2007, 11:00 AM
tnygren's Avatar
tnygren tnygren is offline
Registered User
 
Join Date: May 2005
Posts: 1,408
Hi Glenn,

Getting consistent sftp performance has been a problem since the beginning of sftp. Things have improved, but there are a number of variables:

- SSH2 Window size
- SFTP packet size
- number of parallel sftp requests

The best performance is achieved when the client and the server have made similar assumptions. SecureFX and VShell do make similar assumptions for these settings.

There are other options that can be change that may affect the speed of the connection.

Setting the compression to 'None' can speed up the connection along with trying different Ciphers for the connection. In SecureFX 4.5, allowed Ciphers can be changed in the 'Advanced' sub-category under 'SSH2' in the 'Session Options'.

Does changing any of these options help to improve the speed of the connection?

Quote:
settings like keep-alive, PASV, etc? do these matter?
PASV is only used in FTP connections. The keep alives will not affect the connection speed but can keep the session active if there is no activity in the session.

Quote:
In the global settings, the send-receive buffers are 8192 bytes, should we increase that?
These also only affect FTP connections.
__________________
Thanks,

Teresa

Teresa Nygren
Reply With Quote
  #3  
Old 08-24-2007, 11:47 AM
GlennStanton GlennStanton is offline
Registered User
 
Join Date: Aug 2007
Location: Old Tappan, NJ USA
Posts: 4
ok, i tried the compression off - its about a 30% increase (from 32Kbs to 45Kbs). ciphers didn't seem to make a difference, nor did hash settings. next step would be to find a way to get the client threads increased or some other setting to consume more of the available bandwidth.
__________________
Glenn

Last edited by GlennStanton; 08-24-2007 at 12:22 PM.
Reply With Quote
  #4  
Old 08-31-2007, 08:19 AM
tnygren's Avatar
tnygren tnygren is offline
Registered User
 
Join Date: May 2005
Posts: 1,408
Hi Glenn,

I'm glad to hear that the speed increased somewhat without compression enabled!

As for the multiple threads for the transfer, this is not currently an option in SecureFX.

I have added a request to be able to open multiple threads to the server during the file transfer to our features database.

If this options is added to a future version of SecureFX, a post will be made here.

If you would prefer to be contacted via email, please send me a message and mention this forum thread (2437).
__________________
Thanks,

Teresa

Teresa Nygren
Reply With Quote
  #5  
Old 09-05-2007, 10:49 AM
tnygren's Avatar
tnygren tnygren is offline
Registered User
 
Join Date: May 2005
Posts: 1,408
Hi Glenn,

In looking further into this issue, there may be something else slowing down this connection also.

I would like to get a little more information about the connection between SecureFX and VShell.

What transfer speeds are you seeing if non-encrypted FTP is used between SecureFX and a FTP server on the machine that VShell is installed on?

How high/low is the latency of the network?

For example, do ping commands have a fairly long delay in getting the response or does a trace route show a delay?

Are there any firewalls on the machines that SecureFX and VShell are installed on or firewalls between the 2 machines?

If so, do any of these firewalls attempt to do deep packet inspections?
__________________
Thanks,

Teresa

Teresa Nygren
Reply With Quote
  #6  
Old 09-12-2007, 12:13 PM
GlennStanton GlennStanton is offline
Registered User
 
Join Date: Aug 2007
Location: Old Tappan, NJ USA
Posts: 4
Hi Teresa,

there are firewalls between the 2 data centers - i'm not sure what they're settings are exactly but in testing some other SFTP clients with multi-threaded connections we see transfer rates of 400KBs-1MBs (our target) so presumably we're experiencing some overhead with the SSH/SFTP protocols and something on the network (probably firewalls) contraining us to 40KBs per connection.
__________________
Glenn
Reply With Quote
  #7  
Old 09-13-2007, 10:09 AM
tnygren's Avatar
tnygren tnygren is offline
Registered User
 
Join Date: May 2005
Posts: 1,408
Hi Glenn,

I had one question.

What transfer speeds are occurring with other non-multi threaded clients?

As SecureFX is not Multi-threaded, I would be curious to know how it compares to other non-multi threaded SFTP clients on your network.
__________________
Thanks,

Teresa

Teresa Nygren
Reply With Quote
  #8  
Old 04-08-2008, 03:15 AM
garp99 garp99 is offline
Registered User
 
Join Date: Apr 2008
Posts: 1
Current status

What is the current status on this?

When will SecureFX include support for multi-threaded transfer
so that it can use as much bandwidth as is available (or max
set by user config options)?

Without this feature (similar to downloading from newsgroups),
SecureFX will become obsolete compared to other secure FTP clients.

This is not a wish item anymore, it's a must have!
Reply With Quote
  #9  
Old 04-08-2008, 08:48 AM
kbarnette kbarnette is offline
VanDyke Technical Support
 
Join Date: Aug 2007
Posts: 587
Hi garp99,

Thanks for your comments.

This feature is still being considered for inclusion in a future release of SecureFX.

I have added your follow-up comments to the feature request entry.

As Teresa mentioned previously, we will post here in the event that this feature is implemented.

You can also opt to be contacted personally by contacting us with your request as well as a reference to this forum thread (2437).
Reply With Quote
  #10  
Old 09-01-2009, 05:25 PM
Bigwebmaster Bigwebmaster is offline
Registered User
 
Join Date: Sep 2009
Posts: 3
I was wondering if this feature has been included yet. I have noticed my upload/download speeds have been below par as well. Today I finally looked into it and soon realized that its the SecureFX client software. My other tests with the server show that my server speeds are fine. Initially I thought the network could have been at fault until I pinpointed the problem. From my location to my server I normally get download speeds around 10mbps, and upload speeds around 2mbps, and while using SecureFX I am roughly downloading/uploading around 1mbps. The testing today confirmed this while using other software.

I would really like this feature if it is not available yet.
Reply With Quote
  #11  
Old 09-01-2009, 05:41 PM
miked's Avatar
miked miked is offline
Registered User
 
Join Date: Feb 2004
Posts: 2,040
Hello,

We have not added the ability to split transfers over multiple connections yet, but I've updated our request database and let the product manager know. We'll post a follow up message to this forum thread if we implement the feature, and if you would like e-mail notification in addition, please let us know.
__________________
Mike
VanDyke Software
Technical Support
[http://www.vandyke.com/support]
Reply With Quote
  #12  
Old 09-10-2009, 09:30 AM
Olaf van der Spek Olaf van der Spek is offline
Registered User
 
Join Date: Jul 2004
Posts: 178
The Debian sftp app manages to transfer a file at full speed (1.3 mbyte/s) whereas SFX gets stuck at about 0.4 mbyte/s. I don't think that app uses multiple connections, so there's probably a bottleneck elsewhere. RTT is about 200 - 400 ms.
Reply With Quote
  #13  
Old 09-10-2009, 11:06 AM
miked's Avatar
miked miked is offline
Registered User
 
Join Date: Feb 2004
Posts: 2,040
Hi Olaf,

Thanks for your insight. Can you let us know what the Debian SFTP application name is?

What is the difference between your network connections on Debian where you're running this application and the Windows machine where you're running SecureFX?

Is "RTT" short for round trip time? If so, how are you measuring this time (ping, etc.)?
__________________
Mike
VanDyke Software
Technical Support
[http://www.vandyke.com/support]
Reply With Quote
  #14  
Old 09-11-2009, 10:57 AM
Olaf van der Spek Olaf van der Spek is offline
Registered User
 
Join Date: Jul 2004
Posts: 178
It's "sftp", the command line app that's installed by default. From it's man page:
BSD General Commands Manual
sftp - secure file transfer program

Debian is actually running in a virtual machine (VMware Workstation) on a Windows host. So the network connection is kinda the same.

RTT has been measured with ping.
Reply With Quote
  #15  
Old 09-11-2009, 11:19 AM
miked's Avatar
miked miked is offline
Registered User
 
Join Date: Feb 2004
Posts: 2,040
Hi Olaf,

This forum thread seems to have permuted a little bit, however it sounds like you're raising a point about a problem that you'd like to see fixed. We'll need to get log files in order to address your concern.

Can you send log files from your sftp command, SecureFX transfer and SFTP server?

For the SFTP command, use the command you usually use but add "-vv" after typing "sftp", like this:
sftp -vv user@servername
Also, so that we can timestamps, please add the "date" command before and after the command:
date; sftp -vv user@servername; date
To configure SecureFX to generate a log file:
1) Open SecureFX and choose Options / Global Options.

2) Browse to the File Transfer / Options / Advanced
category in the "Global Options" dialog and enable
the "Enable log output to file" option.

Note: If you're using a version of SecureFX earlier
than version 3.0, this option is located in the
Options / Advanced category of the Global
Options dialog.

3) Specify a filename in the "Log filename" field. For example:

C:\Documents and Settings\<your_username>\My Documents\SecureFX.log

4) Press the "OK" button to close the Global Options dialog.

5) Attempt the connection to the remote machine and
perform the steps needed to replicate the problem.

6) Once the problem occurs, browse to the location of
the SecureFX log file and send it to us by means of
an e-mail attachment.
The server log will also be helpful for determining if there's any problem the server sees - this could be pretty important. Please enable debug logging on the server and perform a file transfer of the same file using the sftp command and SecureFX, then send the server log file.

You can send the files to support@vandyke.com with a subject of "Forum thread 2437 attn: Mike".
__________________
Mike
VanDyke Software
Technical Support
[http://www.vandyke.com/support]

Last edited by miked; 09-11-2009 at 11:32 AM.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -6. The time now is 11:28 AM.