Problem with German "Umlaute"

[Paul Kramer]

Hello,

it seems that cmus cannot display German Umlaute, i.e. À ö ÃŒ ß Ä Ö Ü

I attached a screenshot, which shows the problem. In every row where
there should be an Umlaut, the line from there on is not updated and
contains the lines of the previous view.

I think this has something to do with the encoding of the id3-tags.

In file-browser view I get Die ￜ~Drzte/ as an artist (it should be "Die
Ärzte").

[Johannes Weißl]

Post by Paul Kramer
it seems that cmus cannot display German Umlaute, i.e. À ö ÃŒ ß Ä Ö Ü
I attached a screenshot, which shows the problem. In every row where
there should be an Umlaut, the line from there on is not updated and
contains the lines of the previous view.
I think this has something to do with the encoding of the id3-tags.
In file-browser view I get Die ￜ~Drzte/ as an artist (it should be "Die
Ärzte").

Wow, that is really a strange bug! cmus normally should handle German
umlauts (as well as other non-ASCII characters) really well...

What version of cmus are you using? Can you mail the output of
~/cmus-debug.txt?


Johannes

[Storm Dragon]

Hi,
I am using Ubuntu 10.04. I have pulled the latest from git, However,
when I do ./configure and make I get:
$ ./configure
checking for program gcc... /usr/bin/gcc
checking for program gcc... /usr/bin/gcc
checking for CFLAGS -std=gnu99 -pipe -Wall -Wshadow -Wcast-align
-Wpointer-arith -Wwrite-strings -Wundef -Wmissing-prototypes
-Wredundant-decls -Wextra -Wno-sign-compare -Wformat-security... yes
checking for CFLAGS -Wdeclaration-after-statement... yes
checking for CFLAGS -Wold-style-definition... yes
checking for CFLAGS -Wno-pointer-sign... yes
checking for CFLAGS -Werror-implicit-function-declaration... yes
checking for CFLAGS -Wno-unused-parameter... yes
checking if CC can generate dependency information... yes
checking byte order... little-endian
checking for DL_LIBS (-ldl -Wl,--export-dynamic)... yes
checking for PTHREAD_LIBS (-lpthread)... yes
checking for realtime scheduling... yes
checking for NCURSES_LIBS (-lncursesw)... yes
checking for working ncurses setup... yes
checking for function resizeterm... yes
checking for function use_default_colors... yes
checking for ICONV_LIBS (-liconv)... no
assuming libc contains iconv
checking for working iconv... yes
checking for header <byteswap.h>... yes
checking for program pkg-config... /usr/bin/pkg-config
checking for FLAC_LIBS (pkg-config)... -lFLAC -lm
checking for FLAC_CFLAGS (pkg-config)... -I/usr/include/FLAC
checking for MAD_LIBS (pkg-config)... -lmad
checking for MAD_CFLAGS (pkg-config)...
checking for MODPLUG_LIBS (pkg-config)... -lmodplug
checking for MODPLUG_CFLAGS (pkg-config)... -I/usr/include/libmodplug
checking for header <mpc/mpcdec.h>... no
checking for MPC_LIBS (-lmpcdec)... yes
checking for VORBIS_LIBS (pkg-config)... -lvorbisfile -lvorbis -lm -logg
checking for VORBIS_CFLAGS (pkg-config)...
checking for WAVPACK_LIBS (pkg-config)... -lwavpack -lm
checking for WAVPACK_CFLAGS (pkg-config)...
checking for header <mp4v2/mp4v2.h>... no
checking for header <neaacdec.h>... yes
checking for MP4_LIBS (-lmp4v2 -lfaad -lm)... no
checking for header <neaacdec.h>... yes
checking for AAC_LIBS (-lfaad -lm)... yes
checking for FFMPEG_LIBS (pkg-config)... -pthread -L/usr/local/lib
-lavformat -lavcodec -ldl -lX11 -lXext -lXfixes -lasound -lxvidcore
-lx264 -lvorbisenc -lvorbis -ltheoraenc -ltheoradec -logg
-lopencore-amrwb -lopencore-amrnb -lmp3lame -lfaac -lm -lz -lavcore
-lavutil
checking for FFMPEG_CFLAGS (pkg-config)... -I/usr/local/include
checking for header <ffmpeg/avcodec.h>... no
checking for PULSE_LIBS (pkg-config)... -lpulse
checking for PULSE_CFLAGS (pkg-config)... -D_REENTRANT
checking for ALSA_LIBS (pkg-config)... -lasound
checking for ALSA_CFLAGS (pkg-config)... -I/usr/include/alsa
checking for AO_LIBS (pkg-config)... -lao -lpthread -ldl
checking for AO_CFLAGS (pkg-config)...
checking for program artsc-config... no
checking for header <sys/soundcard.h>... yes
checking for header <sys/audioio.h>... no
checking for ROAR_LIBS (pkg-config)... no
checking for ROAR_LIBS (-lroar)... no
***@stormdragon-laptop:~/cmus$ make
CC ape.o
In file included from ip.h:23,
from ape.c:7:
comment.h:4:20: error: keyval.h: No such file or directory
In file included from ip.h:23,
from ape.c:7:
comment.h:6: warning: 'struct keyval' declared inside parameter list
comment.h:6: warning: its scope is only this definition or declaration,
which is probably not what you want
comment.h:7: warning: 'struct keyval' declared inside parameter list
comment.h:9: warning: 'struct keyval' declared inside parameter list
comment.h:10: warning: 'struct keyval' declared inside parameter list
comment.h:12: warning: 'struct keyval' declared inside parameter list
comment.h:13: warning: 'struct keyval' declared inside parameter list
comment.h:14: warning: 'struct keyval' declared inside parameter list
comment.h:16: warning: 'struct growing_keyvals' declared inside
parameter list
comment.h:17: warning: 'struct growing_keyvals' declared inside
parameter list
In file included from ape.c:7:
ip.h:24:16: error: sf.h: No such file or directory
In file included from ape.c:7:
ip.h:75: error: expected specifier-qualifier-list before
'sample_format_t'
ip.h:85: warning: 'struct keyval' declared inside parameter list
ape.c:9:18: error: file.h: No such file or directory
ape.c:10:21: error: xmalloc.h: No such file or directory
ape.c: In function 'read_header':
ape.c:80: error: implicit declaration of function 'read_all'
ape.c: In function 'ape_parse_one':
ape.c:150: error: implicit declaration of function 'xstrdup'
ape.c:150: warning: assignment makes pointer from integer without a cast
ape.c:154: error: implicit declaration of function 'xstrndup'
ape.c:154: warning: assignment makes pointer from integer without a cast
ape.c:160: warning: assignment makes pointer from integer without a cast
ape.c: In function 'ape_read_tags':
ape.c:219: error: implicit declaration of function 'xnew'
ape.c:219: error: expected expression before 'char'
make: *** [ape.o] Error 1
Thanks
Storm
--
Vinux Publicity Coordinator
Registered Linux user number 508465: http://counter.li.org/
My blog, Thoughts of a Dragon: http://www.stormdragon.us/
Vinux is Linux done right: http://www.vinuxproject.org/
Fe fi fo fum, I smell the blood...
MMMMM BLOOD!

Post by Johannes Weißl

Post by Paul Kramer
it seems that cmus cannot display German Umlaute, i.e. À ö ÃŒ ß Ä Ö Ü
I attached a screenshot, which shows the problem. In every row where
there should be an Umlaut, the line from there on is not updated and
contains the lines of the previous view.
I think this has something to do with the encoding of the id3-tags.
In file-browser view I get Die ￜ~Drzte/ as an artist (it should be "Die
Ärzte").

Wow, that is really a strange bug! cmus normally should handle German
umlauts (as well as other non-ASCII characters) really well...
What version of cmus are you using? Can you mail the output of
~/cmus-debug.txt?
Johannes
------------------------------------------------------------------------------
Xperia(TM) PLAY
It's a major breakthrough. An authentic gaming
smartphone on the nation's most reliable network.
And it wants your games.
http://p.sf.net/sfu/verizon-sfdev

[Johannes Weißl]

Hi,

Post by Storm Dragon
I am using Ubuntu 10.04. I have pulled the latest from git, However,
$ ./configure
[...]

Configure output looks all good!

Post by Storm Dragon
comment.h:4:20: error: keyval.h: No such file or directory

This is strange! Can you remove the source directory (rm -rf ~/cmus) and
check out the git source again? If the error remains, can you compile
with "make V=2" and mail the output?


Johannes

[Storm Dragon]

Hi,
Redownloading it did the trick.
Thanks
Storm
--
Vinux Publicity Coordinator
Registered Linux user number 508465: http://counter.li.org/
My blog, Thoughts of a Dragon: http://www.stormdragon.us/
My last.fm profile: http://www.last.fm/user/stormdragon2976
"And for every crossing where two roads diverged. I fell one false decision. But still you find me on this earth."
Van Canto

[Jason Woofenden]

Paul,

I'm pretty fuzzy on how the unicode stuff works, but I have umlauts
displaying correctly in my library (view 2) over little e.

It might matter which version of ncurses you're linking with. I try
to make sure I'm using ncursesw (the "w" is important I think.)
Check your config.log (in the cmus build directory)... mine has
this:

checking for NCURSES_LIBS (-lncursesw)... yes

I think if it can't find a "w" version, it complains (warning) in
that file.


The other factor I can think of is your language/encoding settings.
I've got this env var:

LANG=en_US.UTF-8


And of course you need a terminal emulator that supports unicode,
but I'm sure you've got that.

That's the extent of my knowledge. If this doesn't point you to the
solution, I hope someone else on the list can.


Take care, - Jason

[Paul Kramer]

Hello,

I looked into it and saw that I didn't have Umlaute in my console at all.

So I set my locale to en_us.UTF8 (strangely enough it was on "POSIX")
and set my console font to lat9w-16

After rebooting everything works like a charm again.

Thanks!

Regards,
Paul

Post by Jason Woofenden
Paul,
I'm pretty fuzzy on how the unicode stuff works, but I have umlauts
displaying correctly in my library (view 2) over little e.
It might matter which version of ncurses you're linking with. I try
to make sure I'm using ncursesw (the "w" is important I think.)
Check your config.log (in the cmus build directory)... mine has
checking for NCURSES_LIBS (-lncursesw)... yes
I think if it can't find a "w" version, it complains (warning) in
that file.
The other factor I can think of is your language/encoding settings.
LANG=en_US.UTF-8
And of course you need a terminal emulator that supports unicode,
but I'm sure you've got that.
That's the extent of my knowledge. If this doesn't point you to the
solution, I hope someone else on the list can.
Take care, - Jason

[Johannes Weißl]

Hey Paul!

Post by Paul Kramer
I looked into it and saw that I didn't have Umlaute in my console at all.
So I set my locale to en_us.UTF8 (strangely enough it was on "POSIX")
and set my console font to lat9w-16
After rebooting everything works like a charm again.

Great that it works now! Still, cmus shouldn't do such nasty things even
if `locale charmap` = POSIX. I will investigate that! Thanks for the
report!


Johannes

[Johannes Weißl]

If iconv fails, replace every multibyte sequence by '?'.
This way cmus is still 100% usable if e.g. locale is POSIX.

Reported-by: Paul Kramer <***@gmx.de>
---
uchar.c | 10 ++++++++++
uchar.h | 12 ++++++++++++
ui_curses.c | 23 +++++++++++++++--------
3 files changed, 37 insertions(+), 8 deletions(-)

diff --git a/uchar.c b/uchar.c
index 8ea6ba4..65580b7 100644
--- a/uchar.c
+++ b/uchar.c
@@ -493,6 +493,16 @@ int u_copy_chars(char *dst, const char *src, int *width)
return di;
}

+int u_to_ascii(char *dst, const char *src, int len)
+{
+ int i, idx = 0;
+ for (i = 0; i < len && src[idx]; i++) {
+ uchar u = u_get_char(src, &idx);
+ dst[i] = (u < 128) ? u : '?';
+ }
+ return i;
+}
+
int u_skip_chars(const char *str, int *width)
{
int w = *width;
diff --git a/uchar.h b/uchar.h
index f676a64..500b929 100644
--- a/uchar.h
+++ b/uchar.h
@@ -154,6 +154,18 @@ void u_set_char(char *str, int *idx, uchar uch);
int u_copy_chars(char *dst, const char *src, int *width);

/*
+ * @dst destination buffer
+ * @src null-terminated UTF-8 string
+ * @len how many bytes are available in @dst
+ *
+ * Copies at most @len bytes, less if null byte was hit. Replaces every
+ * non-ascii character by '?'. Null byte is _never_ copied.
+ *
+ * Returns number of bytes written to @dst.
+ */
+int u_to_ascii(char *dst, const char *src, int len);
+
+/*
* @str null-terminated UTF-8 string, must be long enough
* @width how much to skip
*
diff --git a/ui_curses.c b/ui_curses.c
index 50a0df5..8ce9a9a 100644
--- a/ui_curses.c
+++ b/ui_curses.c
@@ -291,6 +291,7 @@ int track_format_valid(const char *format)

static void utf8_encode(const char *buffer)
{
+ int n;
#ifdef HAVE_ICONV
static iconv_t cd = (iconv_t)-1;
size_t is, os;
@@ -303,7 +304,7 @@ static void utf8_encode(const char *buffer)
cd = iconv_open("UTF-8", charset);
if (cd == (iconv_t)-1) {
d_print("iconv_open failed: %s\n", strerror(errno));
- return;
+ goto fallback;
}
}
i = buffer;
@@ -314,15 +315,19 @@ static void utf8_encode(const char *buffer)
*o = 0;
if (rc == -1) {
d_print("iconv failed: %s\n", strerror(errno));
- return;
+ goto fallback;
}
-#else
- strcpy(conv_buffer, buffer);
+ return;
#endif
+fallback:
+ n = min(sizeof(conv_buffer) - 1, strlen(buffer));
+ memmove(conv_buffer, buffer, n);
+ conv_buffer[n] = '\0';
}

static void utf8_decode(const char *buffer)
{
+ int n;
#ifdef HAVE_ICONV
static iconv_t cd = (iconv_t)-1;
size_t is, os;
@@ -335,7 +340,7 @@ static void utf8_decode(const char *buffer)
cd = iconv_open(charset, "UTF-8");
if (cd == (iconv_t)-1) {
d_print("iconv_open failed: %s\n", strerror(errno));
- return;
+ goto fallback;
}
}
i = buffer;
@@ -346,11 +351,13 @@ static void utf8_decode(const char *buffer)
*o = 0;
if (rc == -1) {
d_print("iconv failed: %s\n", strerror(errno));
- return;
+ goto fallback;
}
-#else
- strcpy(conv_buffer, buffer);
+ return;
#endif
+fallback:
+ n = u_to_ascii(conv_buffer, buffer, sizeof(conv_buffer) - 1);
+ conv_buffer[n] = '\0';
}

/* screen updates {{{ */

[Johannes Weißl]

---
ui_curses.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/ui_curses.c b/ui_curses.c
index 8ce9a9a..f604a13 100644
--- a/ui_curses.c
+++ b/ui_curses.c
@@ -318,8 +318,8 @@ static void utf8_encode(const char *buffer)
goto fallback;
}
return;
-#endif
fallback:
+#endif
n = min(sizeof(conv_buffer) - 1, strlen(buffer));
memmove(conv_buffer, buffer, n);
conv_buffer[n] = '\0';
@@ -354,8 +354,8 @@ static void utf8_decode(const char *buffer)
goto fallback;
}
return;
-#endif
fallback:
+#endif
n = u_to_ascii(conv_buffer, buffer, sizeof(conv_buffer) - 1);
conv_buffer[n] = '\0';
}

[Gregory Petrosyan]

Thanks Paul and Johannes, I've merged the fix to both master and maint. Hope
it's all good now!

Gregory