Re: Unicode-based FreeBSD

Alexander Churanov wrote:
2008/8/24 Tz-Huan Huang <tzhuan@xxxxxxxx>
I'm a Chinese living in Taiwan and I am probably sure that Unicode is
larger
than any other Chinese character sets (including traditional and simplified
Chinese). The UTF-8 support in FreeBSD/Xorg is good enough for me.
I can read/type all Unicode 4.0 characters (including CJKV extension A/B)
in Firefox or any gtk/qt programs if I have the needed font; I can produce
documents with any Unicode characters by LaTeX+CJK package.
It's much better than MS IE and Word because IE and Word only support
Unicode 2.0 (or maybe 3.0, I'm not so sure).

There are two reasons to use any character sets other than UTF-8:
1. compatibility for old programs/services or other OS.
2. the old man wrote the document when Unicode was not so popular and
newbies read the old document.

UTF-8 is more and more popular in Chinese, at least in Taiwan.
Almost everything works well in my daily jobs (of course under the X).
The major missing part is the kiconv UTF-8 support -- currently the kiconv
doesn't support more than two bytes character conversion so there
is no UTF-8 support for Chinese (most Chinese characters are 3-byte or
more). I should mount msdosfs/cd9660 in zh_TW.Big5 and convert the
filename to UTF-8 by lint or screen.

IMHO, If I need Chinese support, I'll go into X. I have no reason to use
Chinese under console even if I can read/type in Chinese. I prefer Firefox
rather than w3m or links. :-)

Regards,
Tz-Huan
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"



Tz-Huan,

Working with Chinese text is the hard part of my solution (described in full
in freebsd-current@xxxxxxxxxxx). In brief it's about moving FreeBSD to UTF-8
completely and making syscons map UTF-8 to selected 8-bit charset for
displaying (a failsafe solution). It seems that this makes syscons somewhat
more usable for some people, but not for from East Asia, am I right?

I was thinking of how to make working with Chinese filenames possible under
syscons, but the help of a native speaker/writer would help much, because I
know only basic facts about that matter.

I see two alternatives of displaying unicode code points that do not fit
into selected 8-bit display charset:

1) Substituting with some character, like '?'. This is very affordable
solutiuon, but makes inconvenient working with files having names that do
not fit into selected charset.

2) Substituting with encoded code point value like "#1234;". This is more
complex solutuon, if correct baskspacing and things like that are required.
I am not ready to implement it.

In any case, it would be nice to have some "magic" implemented: if copying
a text with substitued code points and then pasting it would case the
original UTF-8 sequence to be inserted.

For all folks I'd like to explain again that I'm not discussing correct
rendering of non-latin scripts. It's not possible to render Devanagari in
character mode. And approach that Linux console takes is partial. The cost
of full solution is like X, freetype, freebidi and so on.

Tz-Huan, could you comment on the proposed solution? From your point of
view, are proposed changes in syscons useful?

Again, this does not affect X, Firefox, etc, but would make possible to have
the whole system using UTF-8 out of box.

Alexander Churanov
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"
Hi.

First time poster so be nice, please.

I am not an expert in Unicode but I am Icelandic and need to manage filenames which have some "special characters" in the Latin alphabet. Like á, ð, é, í, ó, ú, þ, æ and ö. Even though these characters are defined in ISO-8859-1 and -15, they cannot be directly typed (by default) in the console in FreeBSD (also applies to Debian Linux). This means I have to resort to other programs - under certain circumstances - if I want to rename, copy or delete files which have these aforementioned alphabetic characters. For me it is an annoying situation and I would gladly support ideas to fix that and especially those who enable one to type and display most Unicode characters in the console.

On to my point... My suggestion is to go as far as possible with the proposed solution. There should be UTF-32, UTF-16 and UTF-8 support and the first-mentioned should be the primary charset with the others as fallback. I think only enabling UTF-8 is not going far enough and therefore I do not support Churanov's ideas to obscure non-displayable characters with other symbols. We should take future needs into account and have UTF-32 and UTF-16 solutions already available instead of having to take on a yet another rewrite when it is needed. I do not know much about other system components but the same should apply to them. I think there should be total Unicode support in FreeBSD as soon as possible. It might take some time for those ISO-8859 charsets to die out but it should not delay the transfer to Unicode. But again, I am not the most knowledgeable about Unicode.

I suggest that the FreeBSD Foundation set a policy which states that every new component must have Unicode support (if applicable) and support projects to bring in Unicode support where there is none or only partial. In FreeBSD 9 or 10, we might start seeing pretty reliable Unicode support and maybe, just maybe, be able to migrate everything from other charsets to Unicode and keep it as the default. This is the chicken and the egg problem. FreeBSD does not have a good enough Unicode support because hardly anybody uses it and hardly anybody uses it because the support is not good enough. To break out of this circle, we need to acknowledge the problem and solve it.

Með kveðju / With regards,
Svavar Kjarrval (svavar@xxxxxxxxxxx)
s. 863-9900

_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: RfD: c-addr/len
    ... I intend to go for 32-bit characters ... Gforth and others support Unicode ... other Forth systems already work with UTF-8 in many situations, ... On SP-Forth, there is no command-line editing, only backspace. ...
    (comp.lang.forth)
  • Re: UTF8 settings (was: Can scp be used to update a directory?)
    ... Anne Wilson wrote: ... bytes to store most common non-ASCII characters, ... What you've just illustrated is an ISO-8859 name viewed in an UTF-8 ... support UTF8 in $LANG properly. ...
    (Fedora)
  • Re: Eiffel and portability, which choice ?
    ... And the support is full of not errors. ... either UTF-8 or UTF-16 are the two formats that ... characters in these scripts, not a single additional bit is needed. ... like the ones described in Unicode or ISO 10646? ...
    (comp.lang.eiffel)
  • Re: console font limits
    ... > Font size is not a sane place to draw the line. ... UTF-8 and large fonts, but no compositing or wide characters. ... because of the rarity and because of the general lack of support ...
    (Linux-Kernel)
  • Re: Reading a text file with spanish accents
    ... Well, it's possible that you are reading the file correctly from UTF-8, but the font for the console doesn't support those characters. ... When using StreamReader with Encoding.UTF8 the ñ ...
    (microsoft.public.dotnet.languages.csharp)