Synchronous behaviour not experienced when using SSH on jumpserver

[umbongo]

I have been trying to write a script that examines the software type on various routers and have come across a discrepancy in expected behaviour between telnet and SSH connections. The synchronous property does not seem to have an affect on SSH sessions while it does for telnet sessions. The stranger part of this problem is that I am using a jump server so I would have expected SecureCRT to be agnostic as to what protocol is being used.
I have enabled Synchronous as true on the screen object.
I have thoroughly read through the published scripting guide, which even mentions SSH as part of the example.

Here is a cut-down version of the script I'm using the exhibits the behaviour I'm describing:

Sub Main()
	Dim objTab
	Dim UID
	Dim Router
	Dim protocol
	Dim Password
	
	Router = "router"
	protocol = "SSH"
	Password = "password"
	UID = "user"
	Set objTab = crt.GetScriptTab
	objTab.Screen.Synchronous = True
	vPossiblePrompts = Array(_
		Router + "#",_
		Router + "-re0>",_
		Router + "-re1>")
	
	Select Case protocol
		Case "Telnet"
			cnxnString = "telnet " & Router
		Case "SSH"
			cnxnString = "ssh " & "-P " & Password & " " & UID & "@" & Router 
	End Select 

	objTab.Screen.Send cnxnString & chr(13)
	If protocol = "Telnet" then
		objTab.Screen.WaitForStrings "sername:","ogin:"
		objTab.Screen.Send UID & chr(13)
		objTab.Screen.WaitForString "assword:"
		objTab.Screen.Send Password & chr(13)
	End If
	objTab.Screen.WaitForStrings vPossiblePrompts
	objTab.Screen.Send "sh ver " + chr(124) + " i Cisco.*[I-]OS" + chr(13)
	objTab.Screen.WaitForStrings vPossiblePrompts
	screenrow = objTab.Screen.CurrentRow
	screenrow = screenrow - 1
	tempDevice = objTab.Screen.Get(screenrow, 1, screenrow, 80 )
	DeviceType = "undefinied"
	
	If instr(tempDevice,"IOS") > 0 then
		DeviceType = "IOS"
	End If		
	If instr(tempDevice,"IOS-XE") > 0 then
		DeviceType = "IOS-XE"
	End If		
	If instr(tempDevice,"Nexus") > 0 then
		DeviceType = "NX-OS"
	End If
	
	MsgBox "tempDevice is " & tempDevice
	MsgBox "DeviceType is " & DeviceType
	objTab.Screen.Send "exit" & chr(13)
End Sub

When connecting using the Telnet protocol everything works as expected. When using SSH the 'tempDevice' variable returns with the line directly above the command prompt immediately after logging in to the device. I have checked the output difference between telnet and SSH and the SSH output contains no unexpected matches within the "vPossiblePrompts" variable that could be causing the WaitForStrings to continue unexpectedly. When the message boxes are dismissed the commands run as expected.

The output of the SSH session is as follows:

$ ssh -P password user@router

********************************************************************************
* *
* You are authorized to use this System for approved business purposes only. *
* Use for any other purpose is prohibited. All transactional records, reports, *
* e-mail, software, and other data generated by or residing upon this System *
* are the property of the Company and may be used by the Company for any *
* purpose. Authorized and unauthorized activities may be monitored. *
* *
********************************************************************************

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2015, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source. This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0 or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
router# sh ver | i Cisco.*[I-]OS <------ Text boxes display at this line before the command is typed
Cisco Nexus Operating System (NX-OS) Software
router#
router# exit
$

SecureCRT 7.2.3 build 500
Windows 7


Thanks in advance

[jdev]

Before going too far down a diagnostic approach, you made a statement that I can interpret different ways, so I thought I'd ask for clarification:

> Here is a cut-down version of the script I'm using the
> exhibits the behaviour I'm describing:

Does this cut-down version exhibit the problem?

In other words, if you're already connected with SecureCRT to your jump host, and you're sitting there at the jump host's shell prompt and you run this cut-down script, do you see the problem occur?

BTW, the 'ssh' and 'telnet' you're sending as commands to your jump host should not be considered as relevant to SecureCRT's synchronous property... the only protocol that matters to SecureCRT is the initial protocol you use in SecureCRT's session configuration to connect to your jump host. After that, anything you do is simply text being sent to the remote, and text appearing on the screen to SecureCRT. This means that the 'ssh' and 'telnet' commands you're running in the remote shell don't have much to do at all with SecureCRT's "synchronous" attribute. Once Synchronous is set to true on the SecureCRT Screen object, it remains true until SecureCRT itself is disconnected, or until your script stops running, or until you set Synchronous back to True with a line of code).

If in your script you're running a loop issuing 'telnet' and 'ssh' commands on your jump host to multiple hosts, you will want to make sure that you perform a WaitFor...() operation on ALL commands you send (even the 'exit' at the end). Otherwise, things could get out of sync.

The way to handle this specifically is to know what the prompt will always be for your jump host, and set that as a special one that you wait for after each "exit" command you use to "disconnect" the jump host from the secondary host.

Another possibility is that the remote system might be sending an escape sequence that contains text that could be found that doesn't appear on the screen. For example, some UNIX shell prompts include escape sequences that set the "title" of the window; such titles can include the prompt text, which can result in WaitForString*() returning unexpectedly. You can set Screen.IgnoreEscape = True, to make sure that escape sequences don't cause problems with your WaitFor()s.

To have an idea of what's actually being sent to SecureCRT, you can turn on "Raw" logging (File -> Raw Log Session), run your script, and then go take a look at the raw log file to see what the remote is sending to SecureCRT (some of which doesn't get displayed to the terminal screen because it's part of an escape sequence).

--Jake

[umbongo]

Does this cut-down version exhibit the problem?

In other words, if you're already connected with SecureCRT to your jump host, and you're sitting there at the jump host's shell prompt and you run this cut-down script, do you see the problem occur?

Yes, of course! I tested it before posting. It was actually far down the path of my non-programmer's diagnostic approach. I wanted to make sure it wasn't another part of the script that was causing an issue. The script is rather lengthy so didn't want to post the whole thing up. To re-iterate: the script I posted experiences the problem when initiated from the script menu once connected to the jump server.

BTW, the 'ssh' and 'telnet' you're sending as commands to your jump host should not be considered as relevant to SecureCRT's synchronous property...

Yes, I know... that's what makes the problem even more odd! I understand the concept of synchronous and that it applies to the connection to the jumpserver.

If in your script you're running a loop issuing 'telnet' and 'ssh' commands on your jump host to multiple hosts, you will want to make sure that you perform a WaitFor...() operation on ALL commands you send (even the 'exit' at the end). Otherwise, things could get out of sync.

It is running in a loop in th production script, however the example is not. Also if everything is connecting using telnet then everything works perfect. It is the switch to SSH that breaks it, so I would think that proves the waitFor and sends are OK. The only ifference really is that telnet login is interactive so has to be automated. But to my untrained eye the logic seems OK there.

The way to handle this specifically is to know what the prompt will always be for your jump host, and set that as a special one that you wait for after each "exit" command you use to "disconnect" the jump host from the secondary host.

On the host I'm testing SSH on I know exactly what the prompt is and I did set it statically as part of troubleshooting using "objTab.Screen.WaitForString "router#"... but it made no difference.

Another possibility is that the remote system might be sending an escape sequence that contains text that could be found that doesn't appear on the screen. For example, some UNIX shell prompts include escape sequences that set the "title" of the window; such titles can include the prompt text, which can result in WaitForString*() returning unexpectedly. You can set Screen.IgnoreEscape = True, to make sure that escape sequences don't cause problems with your WaitFor()s.

Excellent! I'll try that and report back!

To have an idea of what's actually being sent to SecureCRT, you can turn on "Raw" logging (File -> Raw Log Session), run your script, and then go take a look at the raw log file to see what the remote is sending to SecureCRT (some of which doesn't get displayed to the terminal screen because it's part of an escape sequence).

Ermmm... OK... I'm actually disappointed I didn't think of that myself. But I guess my problem is so obscure I'll give myself some slack!


Thanks for the initial look at this. I'll try your last two suggestions and let you know how it goes!

[umbongo]

OK, it looks like we might be getting somewhere...

First off on further research it isn't down to using SSH; it's a partciular device type when using SSH. So far I know the Nexus 9300 platform is an offender while the 7000 series isn't; I don't yet know how far in to the Cisco Nexus product line this problem goes.

I enabled the IgnoreEscape option, but I was surprised it made no difference.

The reason I say surprised is because when I raw logged it came back with some rather interesting results. There were some escape sequences and 3 occurrences of the router name that didn't show up in the screen:

router#
<ESC>[J<ESC>
router#
router#

Unsurprisingly <ESC>[J = Erase Down

So it's starting to become evident why the IgnoreEscape didn't made a difference; I reckon it is still matching on the lines that didn't show on screen.

However I'm not sure how to script around this problem. Any suggestions?

[umbongo]

Some further information; if I switch off synchronous and program a sleep in before the "sh ver" command it works... but I haven't tested how robust this is, and I don't like switching off synchronous behaviour as it could get unpredictable.


Is this a good idea?
Is there a more elegant solution?

[jdev]

It looks like you're up against a situation where the "start" of a session looks different on some machines vs. others.

Since the "start" of one session has a bunch of data that differs in many ways from each other machine (content, size, time it takes to appear, etc.), you'll probably want to introduce a trick known as "Wait until any new text stops appearing on the screen". Here's an example function:

' -----------------------------------------------------------------------------
Sub WaitForScreenContentsToStopChanging(nMsDataReceiveWindow)
    ' This function relies on new data received being different from the
    ' data that was already received.  It won't work if, as one example, you
    ' have a screenful of 'A's and more 'A's arrive (because one screen
    ' "capture" will look exactly like the previous screen "capture").
    Dim bOrig
    
    ' Store Synch flag for later restoration
    bOrig = crt.Screen.Synchronous
    ' Turn Synch off since speed is of the essence; we'll turn it back on (if
    ' it was already on) at the end of this function
    crt.Screen.Synchronous = False
    
    strLastScreen = crt.Screen.Get(1,1,crt.Screen.Rows,crt.Screen.Columns)
    Do
        crt.Sleep nMsDataReceiveWindow
        
        strNewScreen = crt.Screen.Get(1,1,crt.Screen.Rows, crt.Screen.Columns)
        If strNewScreen = strLastScreen Then Exit Do

        strLastScreen = strNewScreen
    Loop
    
    ' Restore the Synch setting
    crt.Screen.Synchronous = bOrig
End Sub

Note that this function takes as a parameter nMsDataReceiveWindow, which represents the largest number of milliseconds that you ever would expect to legitimately transpire and the contents of the screen would not be updated.

It's usually only useful at the beginning of a script, where you want to know it's "safe" to start sending commands. Here, "safe" is defined as, "SecureCRT hasn't received any new data from the remote system (within the ReceiveWindow defined) that has appeared on the screen... this likely means that the remote is waiting for input now."

Here's an example script that puts this into action:

' WaitForScreenContentsToStopChanging.vbs
' 
' Example of a "WaitForScreenContentsToStopChanging()" method used as one
' way to determine if there's any new data arriving from the remote within
' a specified window of milliseconds.
'
' The method protrayed in this example will work in situations where lines of
' data coming from the remote are each unique, or the data you're waiting to
' receive is the initial data received in the session and the number of lines
' of data you're waiting to recieve has fewer lines than the number of rows
' on the terminal screen.

nMsDataReceiveWindow = 150
strCommandThatGeneratesOutput = "ls -alR /"

Sub Main()
    Do
        Do
            nMsDataReceiveWindow = _
                InputBox("Enter the max number of milliseconds the script " & _
                    "should wait for data to arrive before timing out and " & _
                    "considering the data ""stopped"".", _
                    "Specify Data Receive Interval (ms)", _
                    nMsDataReceiveWindow)
            If nMsDataReceiveWindow = "" Then Exit Sub
            If Not IsNumeric(nMsDataReceiveWindow) Then
                MsgBox "Please specify a valid number"
            Else
                Exit Do ' inner Do..Loop
            End If
        Loop

        Do
            strCommandThatGeneratesOutput = _
                InputBox("Enter the command you wish to run on the remote.", _
                    "Specify Remote Command", _
                    strCommandThatGeneratesOutput)
            If strCommandThatGeneratesOutput = "" Then Exit Sub
            If MsgBox("Are you sure you want to run this command?" & vbcrlf & _
                vbcrlf & _
                vbtab & _
                strCommandThatGeneratesOutput, vbYesNo) = vbYes Then Exit Do
        Loop

        ' Send the command to the remote, and wait for the command to be echoed
        ' back to us, including the CR that indicates the command has been
        ' received AND is being acted upon.
        crt.Screen.Send strCommandThatGeneratesOutput & vbcr
        crt.Screen.WaitForString strCommandThatGeneratesOutput
        crt.Screen.WaitForString vbcr
        
        nBeginTime = Timer
        WaitForScreenContentsToStopChanging(nMsDataReceiveWindow)
        MsgBox "Took: " & Timer - nBeginTime & _
            " seconds with a data receive window/interval of " & _
            nMsDataReceiveWindow & " milliseconds." & vbcrlf & vbcrlf & _
            "If new text is still arriving in the terminal window, it is " & _
            "a strong indication that your interval is too short."
    Loop
End Sub

' -----------------------------------------------------------------------------
Sub WaitForScreenContentsToStopChanging(nMsDataReceiveWindow)
    ' This function relies on new data received being different from the
    ' data that was already received.  It won't work if, as one example, you
    ' have a screenful of 'A's and more 'A's arrive (because one screen
    ' "capture" will look exactly like the previous screen "capture").
    Dim bOrig
    
    ' Store Synch flag for later restoration
    bOrig = crt.Screen.Synchronous
    ' Turn Synch off since speed is of the essence; we'll turn it back on (if
    ' it was already on) at the end of this function
    crt.Screen.Synchronous = False
    
    strLastScreen = crt.Screen.Get(1,1,crt.Screen.Rows,crt.Screen.Columns)
    Do
        crt.Sleep nMsDataReceiveWindow
        
        strNewScreen = crt.Screen.Get(1,1,crt.Screen.Rows, crt.Screen.Columns)
        If strNewScreen = strLastScreen Then Exit Do

        strLastScreen = strNewScreen
    Loop
    
    ' Restore the Synch setting
    crt.Screen.Synchronous = bOrig
End Sub

It's not a solve-all solution, but it can add reliability to your script without you needing to wait any longer than necessary on each device you're connecting to.

If you add this WaitForScreenContentsToStopChanging() function to your script (probably at the very end), and then add an "Else" clause to be a companion to your Telnet's "If" which calls this function, you can be relatively "sure" that the remote is ready to receive your commands...
Oh. You can also get the text on the current line after you've called this
function, and consider with relative accuracy that the text you get on the current line to the left of the cursor actually represents the shell prompt on your secondary host.

So, in the code you provide, this:

	objTab.Screen.Send cnxnString & chr(13)
	If protocol = "Telnet" then
		objTab.Screen.WaitForStrings "sername:","ogin:"
		objTab.Screen.Send UID & chr(13)
		objTab.Screen.WaitForString "assword:"
		objTab.Screen.Send Password & chr(13)
	End If
	objTab.Screen.WaitForStrings vPossiblePrompts

...would become this:

	objTab.Screen.Send cnxnString & chr(13)
	If protocol = "Telnet" then
		objTab.Screen.WaitForStrings "sername:","ogin:"
		objTab.Screen.Send UID & chr(13)
		objTab.Screen.WaitForString "assword:"
		objTab.Screen.Send Password & chr(13)
		WaitForScreenContentsToStopChanging(350)
	Else
		WaitForScreenContentsToStopChanging(350)
	End If
	
	' Get the text to the left of the cursor
	strPrompt = objTab.Screen.Get(_
		objTab.Screen.CurrentRow, _
		1, _
		objTab.Screen.CurrentRow, _
		objTab.Screen.CurrentColumn - 1)

	' Now that I have THE secondary prompt, I can reliably
	' just wait for it after sending each command in sequence.

--Jake