[Python] Weird newlines using ReadString

[vysh]

Hi guys, this code give me something unexpected:

crt.GetTab(tabNum).Screen.IgnoreEscape = True

# Capture upstream interface information
licaShowUSCommand = "show interface upstream\r"

crt.GetTab(tabNum).Screen.Send(licaShowUSCommand)
licaUS = crt.GetTab(tabNum).Screen.ReadString(licaPrompt, 3)

licaUS = re.sub(licaShowUSCommand, '', licaUS)

This is what shows in SecureCRT after running the code:

miniCMTS>show interface upstream
Channel  Status   Frequency  Channel Width  Modulation              D30Mode  Channel  Inner Power
                  (MHz)      (MHz)          Profile                          Type     (dbmv)     
1        enable   32.0       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0          
2        enable   38.4       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0          
3        enable   44.8       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0          
4        enable   51.2       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0          
miniCMTS>

This is what gets captured in "licaUS" variable:

Channel  Status   Frequency  Channel Width  Modulation              D30Mode  Channel  Inner Power

                  (MHz)      (MHz)          Profile                          Type     (dbmv)     

1        enable   30.4       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0          

2        enable   36.8       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0          

3        enable   43.2       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0          

4        enable   49.6       6.4            AtdmaMediumNoiseQpsk    disable  ATDMA    0

Hidden characters enabled (see attachment) -- shows extra CRs. Where do they come from? I am really puzzled...

[bgagnon]

Hi vysh,

The information is likely being sent to SecureCRT that way by the remote.

However, having multiple CRs (carriage returns) would not necessarily cause "new lines". There is only one LF (line feed) being shown in your editor when hidden characters are enabled.

To get a true understanding of the data the remote is sending, you would want to capture a raw log probably.

To create a raw log:

• Before connecting with the session, select Raw Log Session from SecureCRT's File menu.
  
• In the Select Log File dialog, choose a folder and filename in which you would like the log text to be stored and press the Save button. At this point, if you open the File menu, you will notice that the Raw Log Session menu item now has a check-mark next to it; this indicates that raw logging is activated.
  
• Now connect to the remote machine and perform the actions that cause the behavior.
  
• Disconnect the session and select Raw Log Session from the File menu to turn off the raw logging feature.

If you want us to look at it, since a raw log can contain sensitive information, feel free to send it as an attachment via email to support@vandyke.com. Please reference "Attn Brenda - Forum Thread #12442" in the subject line.

[vysh]

Thanks for the reply.

I have found out that the extra CRs get added here:

results_path = os.path.join(os.path.expanduser("~"), "lica_info.txt")
f = open(results_path, 'w')
    f.write(final_results,)
    f.close()

Immediately before that if I crt.Dialog.MessageBox(final_results) everything looks fine, no extra CRs. But in the file it writes "\r\r\n". I am unsure why :-/
For now I just implemented a dirty workaround with

f.write(final_results.replace('\r', ''))

. It strips all '\r's from the string and Windows transforms single \n's to \r\n and all looks good, but it doesn't solve the problem itself.
While I was looking into my newline issues I decided to additionally copy everything to Clipboard. But I immediately ran into another issue.

def main():
	text = "First line.\nSecond line\n\nFourth line."

	crt.Clipboard.Text = text

main()

It will not convert \n to \r\n when run in SecureCRT 8.0.2 for Windows. Is this normal?

[bgagnon]

Hi vysh,

I am not sure about "extra CRs" (I have not seen a raw log, so I don't know what the remote sends), but the line ending conventions on the differing operating systems *are* different. Windows uses CR/LF, Mac/Linux/Unix uses LF only.

It will not convert \n to \r\n when run in SecureCRT 8.0.2 for Windows. Is this normal?

I am not sure what you mean by "run in" ...?

You are putting that literal text into the clipboard, it's not going to convert the line endings.

What problem are you trying to solve at this juncture?

[vysh]

I assumed new lines would be converted automatically into their os-specific versions when putting them into the clipboard just as it is the case with open():

https://docs.python.org/2/library/functions.html#open:

The default is to use text mode, which may convert '\n' characters to a platform-specific representation on writing and back on reading.

However it looks like they do not get converterd, so you have to use code like:

    if os.name == 'nt':
        final_results = final_results.replace('\n', '\r\n')

    crt.Clipboard.Text = final_results

[bgagnon]

Hi vysh,

I assumed new lines would be converted automatically into their os-specific versions when putting them into the clipboard

No, as you found, that is not the case. We do believe there is a Python-specific function that you could use to achieve the desired conversion, but we just don't have that info on-hand. If you find the specific function, or any community members know of it, feel free to post the info.

[Neisany]

is there any update to fixing this issue?

	data=""
	varCommands=[
		"show system details | nomore",
		"show system status| nomore | tab",
		"show system package-info| nomore",""]
		
	#crt.Dialog.MessageBox(str(len(varCommands)))
	for varCommand in varCommands:
		data += CaptureOutputOfCommand(varCommand,"admin@")
#		crt.Dialog.MessageBox(data)
	data= data.strip('\r\n')
	crt.Clipboard.Format = "CF_TEXT"
	crt.Clipboard.Text = data	





def CaptureOutputOfCommand(varCommand, prompt):

	# First, send the command to the remote.
	crt.Screen.Send(varCommand + '\r')

	# Second, wait for the carriage return to be echoed by the remote device.
	# This allows us to capture only the output of the command, not the line
	# on which the command was issued (which would include the prompt + cmd).
	# If you want to capture the command that was issued, simply comment out
	# the following line of code.
	# crt.Screen.WaitForString('\r')

	# Now that the command has been sent, use Screen.ReadString to capture
	# all the data that is received up to the point at which the shell
	# prompt appears (the captured data does not include the shell prompt).
	return crt.Screen.ReadString(prompt)

main()

output =
text from the command
empty line
text form the command
empty line

[bgagnon]

Hi Neisany,

I cannot replicate those results.

What version of SecureCRT are you using?

What is the remote device you run those commands against?

Have you confirmed the line endings the device sends *is only* \r\n (CR/LF)? I have seen some Cisco devices that send \n\r\n (LF/CR/LF).

If you have just a Linux box with OpenSSH running you can connect to and you run the script against it (changing the commands array to something more appropriate), do you get the same results?

[Neisany]

here is my version
Version 8.3.4 (x64 build 1699) - Official Release - July 12, 2018

not sure on how to turn on hidden characters.

[bgagnon]

Hi Neisany,

You would have to have some editor you can view non-printing characters in, or, just change them to something you can view:

strOutput.replace('\n', '[LF]')
strOutput.replace('\r', '[CR]')

Then MessageBox the output and see what you have.

[Neisany]

thanks Bagagon!

I now see the CR and LF
example 1

[ok][2019-11-30 03:07:12][CR][CR]
FADV-ATLANTA-PRI-cli> show coredump| nomore[CR][CR]
total 0[CR][CR]
[ok][2019-11-30 03:07:12][CR][CR]
FADV-ATLANTA-PRI-cli> show commit list 5| nomore[CR][CR]
2019-11-30 03:07:13[CR][CR]
SNo. ID       User       Client      Time Stamp          Label       Comment[CR][CR]
~~~~ ~~       ~~~~       ~~~~~~      ~~~~~~~~~~          ~~~~~       ~~~~~~~[CR][CR]

example 2

[ok][2019-11-30 03:07:12][CR][CR]
FADV-ATLANTA-PRI-cli> show coredump| nomore[CR][CR]
total 0[CR][CR]
[ok][2019-11-30 03:07:12][CR][CR]
FADV-ATLANTA-PRI-cli> show commit list 5| nomore[CR][CR]
2019-11-30 03:07:13[CR][CR]
SNo. ID       User       Client      Time Stamp          Label       Comment[CR][CR]
~~~~ ~~       ~~~~       ~~~~~~      ~~~~~~~~~~          ~~~~~       ~~~~~~~[CR][CR]

But I don't understand why the .rstrip isn't working. I will use the replace command and see if that fixes my issue.