UTF-8: problems with characters and boxes

[Motley Fool]

I'm still new to Unicode and all the various bits that need to be set in order for it to work right, so I may have missed something. I can't figure out what it is though, so any help is appreciated. I'm trying to get UTF-8 working in SecureCRT 5.5.1 which connects to an Ubuntu 6.06 box. Some UTF-8 characters are displayed fine, while others are not. I tested SecureCRT alongside PuTTY - PuTTY seemed to get the display "right" more of the time.

Here's the setup. First, on the Linux side of things:

> echo $TERM
xterm

> locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

and over in SecureCRT 5.5.1 (Windows XP):

Options -> Session Options -> Terminal -> Emulation:
  Terminal = Xterm
  ANSI Color = (checked)
  Use color scheme = (not checked)

Options -> Session Options -> Terminal -> Appearance:
  Normal Font = Lucida Console 9pt
    (after clicking "Font" button)  Script = Western
  Character Encoding = UTF (tried OEM as well - it was even worse)
  Use Unicode line-drawing characters = (checked)

and in PuTTY 0.60 (Windows XP):

Window -> Appearance
  Font = Lucida Console, 10-point
Window -> Translation
  Received data assumed to be... = UTF-8
  Handling of line drawing characters = Use Unicode line drawing code points

Window -> Colours
  Allow terminal to specify ANSI colours = (checked)

I looked at the good ol' UTF-8-demo.txt file in both SecureCRT and PuTTY. I took screenshots in 3 different parts of the file - the Greek anthem, the Russian conference invitation, and the first four box drawing tests at the bottom of the file.

Here's what I hoped to see: screenshot-actual.gif
Here's what I saw in PuTTY: screenshot-putty.gif
Here's what I saw in SecureCRT: screenshot-securecrt.gif

Now, getting a tip from this thread, I tried going into SecureCRT's Options -> Session Options -> Terminal -> Appearance window, clicking the "Font" button, and changing "Script" from "Western" to something else. When I set it to "Greek", the Greek anthem looked fine. When I set it to "Cyrillic", the Russian invitation came out fine. However, when the "Script" was set to "Greek" or "Cyrillic", it shot the box rendering all to pieces. Look here: bad-boxes.gif

So it seems like there are two problems:

1. Incorrect UTF-8 character rendering (which I assume Sinpi accurately described in the thread I mentioned...?)
2. Incorrect box rendering in any Script (whether Western, Greek or Cyrillic) that doesn't appear to be problem in PuTTY, even using the same Lucida Console font. Even in its best mode (i.e. in Western Script), SecureCRT seems to be using the "poor man's" box rendering method of pluses, hyphens and pipes rather than actual Unicode box characters. (I should note that it made no difference whether the "Use Unicode line-drawing characters" SecureCRT setting was checked or not - the boxes are drawn the same way.)

So, is there a problem with my setup, or is there a problem with SecureCRT 5.5.1?

Thanks to anyone who can help!

[Maureen]

Originally Posted by Motley Fool

So, is there a problem with my setup, or is there a problem with SecureCRT 5.5.1?

Thanks for posting such a detailed description of what you are trying to do. Currently, SecureCRT doesn't support multiple international character sets at the same time. This is something we hope to address in a future version. I will add your request to our development database. When we have something for you to try, I will post a note here. If you would also like to be notified by e-mail, please send e-mail to me at Maureen.Jett@vandyke.com.

Maureen

[Motley Fool]

Originally Posted by Maureen

Currently, SecureCRT doesn't support multiple international character sets at the same time.

Maureen, thanks for the reply. What you mentioned answers my questions about issue #1.

How about issue #2?

2. Incorrect box rendering in any Script (whether Western, Greek or Cyrillic) that doesn't appear to be problem in PuTTY, even using the same Lucida Console font. Even in its best mode (i.e. in Western Script), SecureCRT seems to be using the "poor man's" box rendering method of pluses, hyphens and pipes rather than actual Unicode box characters. (I should note that it made no difference whether the "Use Unicode line-drawing characters" SecureCRT setting was checked or not - the boxes are drawn the same way.)

Can that be solved?

I found a similar box-drawing thread over here. The final post by troybtj said that he had resolved it by setting his LANG variable to "en_US.UTF-8". Mine was already set that way, however (see the "locale" section of my Linux box in my first posting), and yet the problem remains.

Thanks again.

[Maureen]

Originally Posted by Motley Fool

How about issue #2?
Can that be solved?

I found a similar box-drawing thread over here. The final post by troybtj said that he had resolved it by setting his LANG variable to "en_US.UTF-8". Mine was already set that way, however (see the "locale" section of my Linux box in my first posting), and yet the problem remains.

Thanks for following up and sorry I only answered one of your questions. The line drawing problem is related to the multiple character set problem. The sample file contains pure Unicode characters and currently, SecureCRT doesn't have a transformer that can handle these characters.

Maureen

[Motley Fool]

Originally Posted by Maureen

The sample file contains pure Unicode characters and currently, SecureCRT doesn't have a transformer that can handle these characters.

Maureen, I may just not be catching on, but let me ask one more question. To get rid of all the multiple character sets, I tried creating a file with just spaces and Unicode box characters. (i.e. using U+2500 through U+255D to make a simple single-line box and double-line box). I set both PuTTY and SecureCRT to use Lucida Console, which contains the characters needed for box drawing. I tried setting SecureCRT's "Use Unicode Line-drawing characters" checkbox to both on and off.

Here's what I got: two-boxes.gif

So, in this situation where the font has the necessary characters and everything is set to UTF-8, what's preventing SecureCRT from drawing the boxes correctly?

Thanks again.

[Maureen]

Originally Posted by Motley Fool

So, in this situation where the font has the necessary characters and everything is set to UTF-8, what's preventing SecureCRT from drawing the boxes correctly?

You're correct that the font has the necessary characters. The problem is that the file is Unicode and currently, SecureCRT can only display line-drawing characters for single-byte character sets.

It seems like the samples you've posted are for testing purposes. I was wondering how close your "real" files are to what you've posted. For example, do you need to be able to display multiple language character sets in the same file? What are your line-drawing needs?

Maureen

[Motley Fool]

Well, for example, here are a couple of tools I use. This is "aptitude", the Ubuntu console-based package management system. aptitude.gif

And here's dpkg-config system for configuring Ubuntu's various packages. configuration.gif

The box-drawing quality is acceptable... just not as good as it could be.

[mjr]

I'd like to chip into this thread. This is the only reason I have not yet purchased SecureCRT. If a simple program like mIRC or PuTTY can handle UTF-8 near flawlessly, why can't a program purchased by international defense contractors, banks, and manufacturing companies? Surely clients such as Microsoft and Hitachi need to work with both Japanese and Western script at the same time, and clients such as Verizon, a Vodafone subsidiary, need to work with other subsidiaries that use Eastern European, Arab, and Japanese scripts? I really love the software, but I often have to work with multiple scripts at the same time (such as editing a file in Japanese while using a Western editor).

[mdella]

...I'll chip in as well ;-)

For the last couple of years I've been working on several international projects that are targeted to Japan, Korea, India, and China. To some degree I've given up on actually using SecureCRT to work mostly right with the UTF-8 version of the US character set and supporting characters. Rather I have been taking files and using SecureFX to transfer them to my desktop, editing them with EditPlus (and SuperEdit) to get the correct character set encodings (Could never trust what I say being typed by SecureCRT was actually the correct encoding, it just looked correctly on my screen rarely elsewhere) then upload the files back to the box once done.

Granted this is time consuming and the like, but I use SecureCRT for a plethera of other things and getting the Unicode stuff was a low priority for me. But since others are having this issue, I thought I'd chime in.

Oh by the way, slowly but surely, other countries (esp Europe) are converting to UTF-8 or -16 and away from their "custom" character sets. The US is the slowest in the world to conform since everyone here uses ASCII (ok, so its dating me by 20+ years) and (cough cough) Color ANSI Color (invented back in the 80s ;-)

Course most of the US stuff I use falls into this category so its hasn't been that big a deal until I took on this last international job...

Marcos

[schickb]

Wow, very old thread, but it appears this is still broken. I sure wish SecureCRT could correctly display Debian package configuration under Ubuntu or Debian.

Any plans to fix this?

[Maureen]

Thanks for following up on this. We have spent some time looking into the Unicode issue and have made some code changes that will make it easier to implement this. We don't have a timeline for implementing full Unicode support, but I'll definitely post here when we have something to try.

Maureen

[Maureen]

SecureCRT 6.5 has improved UTF-8 support in the following ways.

• character sets from multiple languages can be viewed in a session
• improved support for right-to-left reading order languages
• UTF-8 characters can be mapped in the Keymap Editor
• Send/Receive ASCII works with UTF-8 data
• UTF-8 filenames can be used in the SFTP tab
• pass-through printing supports UTF-8 data
• better behavior with scripts that use combining marks to compose characters

This version is currently in pre-beta testing. If you would like to try it, please send e-mail to me at Maureen.Jett@vandyke.com.

Maureen

[solarein]

I'm using SecureCRT 6.1.4 and I'm having the same problems with unicode line drawing characters with UTF-8 not displaying correctly. After some searching I was able to find this ancient thread, and it looks like this is the same problem remaining after all this time. I'm using Linux emulation mode with ANSI Color enabled, and Courier New 10pt font. Like those before, my line drawing characters in aptitude and other programs are displayed as dashes and plusses, i.e. "poor man's line drawing characters" rather than as unicode line drawing characters. I haven't tried the 6.5 beta yet because Maureen's message doesn't list a fix to unicode line drawing. Could someone tell me if this has been fixed with version 6.5 or is planned to be fixed in a later release? Thanks.

[miked]

The SecureCRT 6.5 History describes changes to Unicode support. If you you try 6.5 and it doesn't work better for you, we can try to help. In the worst case scenario you can easily revert to 6.1 again.

[solarein]

I just upgraded to SecureCRT 6.5 and unicode line drawing is now working perfectly for me. I also like the new interface. Thank you very much and keep up the great work!