Re: Unicode-based FreeBSD

Alexander Churanov wrote:

Svavar,

I am trying to understand you.

2008/8/26 Svavar Lúthersson <svavar@xxxxxxxxxxx <mailto:svavar@xxxxxxxxxxx>>

The Icelandic alphabet works in editors like pico but I have not
found another editor where it actually displays the characters
correctly. I checked edit, vi and vipw to be sure. It might be a
configuration problem or a lack of it but it's better for the user
experience if it works out-of-the-box. It should be enough to
configure it in one place and it should work "everywhere".

Hmm. A minute ago I've pressed Ctrl-Alt-F3, switched to syscons console, started "emacs /tmp/test", where "test" was written in russian, typed some russian text into, closed the editor and then started "cat /tmp/test". No problems. I still can not understand what's the difference between ISO-8859-1 and KOI8-R from the implementation point of view. It seems that I need to try to configure a system for Icelandic. I'll do that tomorrow on a dedicated box. I promise to help you with configuration in case It's at all possible.


The primary problem of the character support in syscons is
displaying specialised characters on the screen/tty. When I use
the special Icelandic characters in UTF-8, each character is
displayed as "??" which is very confusing to see if there are 2 or
more in a row...

This is exactly what I am trying to solve, examining opinions on this list at the same time.

Of course there are certain problems with changing the filenames
between languages like Russian and Icelandic since the normal
keyboard only has about 100 keys and cannot possible contain all
the characters in the Unicode specification.

There are special Input Methods for the rest of Unicode (more than 200K code points currently assigned).


It however should not stop me from reading the filenames in the
language they were written. As for writing characters in other
languages, the "Windows approach" steps in the right direction by
enabling me to change the input language and therefore type in
characters I would not otherwise be able to with the Icelandic
keyboard. If the characters are translated to Unicode, it should
not matter what keyboard layout is used. As for how it would be
carried out in FreeBSD, I will leave it up to the developers.

For switching I use CapsLock when in plain syscons console and Alt+Shift when in X. By the way, how Windows displays non-ASCII characters in plain text console? I'll wonder if better than suggested by me.

The aforementioned is why I am suggesting that the system should
be moved directly to UTF-32. If it is moved to UTF-8 and there is
a need in the future for UTF-16 or -32, the conversion process has
to start again.

I'm sure that it is not necessary. Again, all UTFs encode THE SAME SET. But UTF-32 is better for single characters. And UTF-8 is better for UNIX-like systems.


Like I mentioned in my former answer, the program writers do not
write Unicode compatible programs because there is almost no
Unicode support and the FreeBSD developers see little reason to
speed up Unicode implementation because there are so few programs
Unicode compatible. Therefore I think that FreeBSD should
implement a Unicode support policy and move straight to UTF-32 and
make it the FreeBSD default. I am not pretending that this project
will be easy, painless and quick but it is better done sooner than
later. Said policy could begin by announcing an active plan for
Unicode support and suggest that every new FreeBSD project should
support Unicode. At the same time it should suggest the same to
other developers which write software for FreeBSD. When the time
is right or after further steps, the FreeBSD Foundation should
announce that after version X, Unicode will be default charset. At
that time, the software which has Unicode support will (I hope)
work flawlessly with Unicode characters. When UTF-32 would be
fully supported in FreeBSD, the developers could wait for the end
of the support cycle for the first version with full UTF-32
support and then make it the default in the versions to come. That
way the backward compatibility would be great and for all
supported versions of FreeBSD.

Probably, this is useful, but I'm sure that this is out of scope of my little project. I do not have enough power to enforce such a policy.

Með kveðju / With regards,
Svavar Kjarrval (svavar@xxxxxxxxxxx <mailto:svavar@xxxxxxxxxxx>)
s. 863-9900

Alexander Churanov

Hi.

Not everything was directed to you so please do not take things personally. :þ (The Icelandic "þ" at work.)

I am not against your idea of adding the Unicode support in general, just that it doesn't go far enough. I did not misunderstand your point that it uses the same set but we would end up in the same situation when it will be time to add display and writing support for UTF-16 and UTF-32 although it would be slightly easier since we do not have to make an actual conversion of existing characters. Still we have to think about the programs that only "upgrade" to UTF-8 and have to go through yet another change to support UTF-16 or UTF-32. It is much easier in the long run to just go all the way to UTF-32 to begin with. Going to UTF-8 might fix some of the character issues but we would be in the same shoes when it comes to characters which are in -16 and -32 but not in -8. I am not a user of X in FreeBSD (but soon, I hope) so my FreeBSD environment is limited to the console.

Windows (XP) displays the Icelandic alphabet with no problems at the command prompt. I also have no problems typing it. Somebody else has to try to see if other charsets, like Russian or Chinese, work there as well.

If the result of this discussion is that UTF-8 support is enough, I suggest the future expansion to UTF-16 and UTF-32 will be taken into account and make is as easy as possible for somebody in the future to carry it out. If my programming was any better, I would have volunteered for the job.

The policy suggestion was directed to the mailing list in general, not as a directive that you should or must enforce.

Með kveðju / With regards,
Svavar Kjarrval (svavar@xxxxxxxxxxx)
s. 863-9900

_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Unicode-based FreeBSD
    ... another editor where it actually displays the characters correctly. ... There are special Input Methods for the rest of Unicode (more than 200K code ... would be carried out in FreeBSD, I will leave it up to the developers. ... FreeBSD should implement a Unicode support policy and move straight to ...
    (freebsd-current)
  • Re: Unicode Support
    ... > compiler/assembler is unicode support. ... > but the fact that most assemblers, ... Most popular *editing* tools support this data format. ... Is there really a need to put unicode characters into identifiers? ...
    (alt.lang.asm)
  • Re: Unicode Support
    ... >> compiler/assembler is unicode support. ... >> but the fact that most assemblers, ... Most popular *editing* tools support this data ... > Is there really a need to put unicode characters into identifiers? ...
    (alt.lang.asm)
  • Re: Unicode-based FreeBSD
    ... You have to type "special characters" that are high-bit characters of ... I am not an expert in Unicode and I am not even suggesting that I know everything about FreeBSD. ... The primary problem of the character support in syscons is displaying specialised characters on the screen/tty. ...
    (freebsd-current)
  • Re: =?ISO-8859-1?Q?Soup=E7on_of_cedilles_and_aper=E7us?=
    ... Note that most of these support only ASCII ... plus a few accented characters, ... Unicode supports, in principle, the characters of every written ... The catch is that not all newsreaders are new enough to ...
    (alt.usage.english)