unacceptable character '?' (0xFFFD) special characters are not allowed

Hi,

I'm getting the following errors on linux systems (some of them) when I load config yaml files via the built in config classes:

Code:

[WARNING] [DeathTpPlus] [DeathTpPlus] Error in the deathmessages configuration
org.bukkit.configuration.InvalidConfigurationException: unacceptable character '?' (0xFFFD) special characters are not allowed
in "<string>", position 11310

as you can see in these supplied config files http://dev.bukkit.org/media/attachments/25/852/DeathTpPlus.zip

the file in question is UTF-8

strangly the others aren't and don't cause those issues.

Another strange thing is that I save the files as UTF-8

Code:

            OutputStream outputStream = new FileOutputStream(pluginPath + deathMessageFileName);
            stream = new PrintWriter(new OutputStreamWriter(outputStream, "utf-8"));

Any idea on how to fix this? Google is full of this and it looks like issues with the yaml processor

 

Don Redhorse my guess is that the '?' character is not supported in the UTF-8 charset, though that is a bit strange. I am pretty sure ? can be read/written from file...

Maybe it is different on Linux systems?

 

It's flipping over your 'ä' on line 302. Remove that and it would probably work.

I don't know the cause of this, as that character is a valid UTF-8 character.

 

well that is the point... the file is valid but still causing issues... the file I posted above uses special characters, but this is now even happening with my stock default english messages ... only on linux though...

On windows it works, even with special characters like ä but only if I use utf-8.

well the ? is just a placeholder... for a character in the file which can't be parsed by the yaml configuration classes... i guess that the issue would go away if the file is saved without utf-8 on linux. But this is required for windows systems.

 

Save it as Unix ANSI to make it work on both linux and windows AFAIK..

 

will that support foreign characters like öäü (german) or danish, french etc?

 

I have the same Problem but dont know how to fix it.
Im using everywhere Linux, therefore i have no Wordpad or something like that.
But i need to "convert" from UTF-8 to ANSI right?
But how to?
I dont understand where i can do this or HOW i can do this.

I have the Problem with AuthDB, it shows me mysterious letters (an a with a line over it ^^)
And after that theres the default text. But theres no way to translate it from English to german... i have really no idea whats wrong.

 

Whatever 0xFFFD is, it's not a valid character: http://www.unicodemap.org/details/0xFFFD/index.html
But I can't find the character in your yml files, so I assume it's a charset issue. Let me check bukkit sources... Okay, as far as I see bukkit doesn't load with a specific charset. That means Java falls back to the systems charset. Execute the following command on the infected machines to confirm that UTF-8 is not their system charset:
locale
Example output from a machine using UTF-8:

Code:

LANG=de_DE.UTF-8
LANGUAGE=de_DE:en
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

//EDIT: I almost forgot the fix... I think you load the files (from within your jar), then save them (into you data folder). So tell Java to load them as UTF-8, but do not tell it which charset to use while you save them. That way it will convert the file from UTF-8 to whatever charset the system uses and it should be loadable by bukkits yaml API later.

 

if do "local" the output looks wrong ^^

Code:

LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

That should be really wrong But how to tell java to use UTF8?
Or how to install UTF8 on Ubuntu?
Im asking aunt google, perhaps she helps me...

Changed to UTF8 with this "update-locale LANG=de_DE.UTF-8 LC_MESSAGES=POSIX".
Rebooting now and test it again

 

I have the same problem with some plugins i have linux how can i fix this?:

Code:

org.bukkit.configuration.InvalidConfigurationException: unacceptable character '�' (0xFFFD) special characters are not allowed
in "<string>", position 44

 

The iconv utility on Linux will convert files to UTF-8 (or pretty much any encoding).

 

So i need to use icony utility is that what you mean? i really dont have a clue how to fix this.

 

You're most likely seeing that error ("unacceptable character '�' (0xFFFD)") because the file you're trying to load isn't either plain ASCII or UTF-8 encoded (and ASCII is a subset of UTF-8 anyway). You can convert files to UTF-8 encoding with iconv, like this:

Code:

$ iconv -f ISO8859-1 -t UTF-8 file.txt > newfile.txt

Of course, you need to know what encoding your existing file is (ISO8859-1 is a common encoding for western European languages). "file file.txt" might be able to tell you. If you're OK with sharing the file, feel free to upload it to pastebin or equivalent and I'll take a look.

 

btw, for a german language support and the 'Ä', 'ä', 'Ö', 'ö', 'Ü', 'ü' characters you can use '\u00C4', '\u00E4', '\u00D6', '\u00F6', '\u00DC', '\u00FC', that will solve the problem too!
In german:
Du kannst statt 'Ä', 'ä', 'Ö', 'ö', 'Ü', 'ü' auch '\u00C4', '\u00E4', '\u00D6', '\u00F6', '\u00DC', '\u00FC' nutzen, damit wäre der Fehler auch "behoben"!

 

If it's really a "german file" it may be encoded in ISO8859-15.

//EDIT: Or some windows encoding like Windows-1252.

 

If I remember correctly, there was recentlyish(a few months ago) a bug with the way that bukkit uses SnakeYML, I am not sure if has been fixed since, but I was told that bug was the cause the nearly identical problem to yours that I had. Also I am positive it only affects UTF-8 files

 

sorry mabyy stupid question which file do i need to pastebin?

 

Show us the full stacktrace and we may be able to tell. But from that snippet it almost sounds like its hardcoded into some plugin you use.

 

It's probably coming from here: https://github.com/Bukkit/Bukkit/bl...configuration/file/YamlConfiguration.java#L55 in the loadFromString() method, which is also used when you load an external configuration file.

monir it's a YAML file that you need to post - might be your config.yml, or does your plugin load another YAML files?

 

Here i have this file its Yml and its modifyworld for permissionsex When i change the config i get this charachter error but this is orginal config and that work.

http://pastebin.com/9a8EFFDc

 

monir Now the question is how should we identify the charset of the wrongly encoded characters and encode it to the right one if you give us the file without it?

Also I think it would be better to upload to a binary file hoster as pastbin&co may correct the character issues automatically.

 

guys.. could you make sure to add any findings to the issue about this on the bukkit bug tracker... will try to get the issue number but atm it is not possible

 

The map chatsensor is how i edited the config after restart it resets when i get the error.
Here is pastebin:http://pastebin.com/Bxk78pzb
yml file and jar: http://speedy.sh/C7knb/chatsensor.rar

I have this problem with 3 diffrent plugins.

 

OK, your ChatCensor2/config.yml is in ISO8859-1 encoding, as I suspected - in particular, this string: "Det ordet är blockerat!"

(ChatCensor/config.yml is plain ASCII so shouldn't be a problem)

If you convert that config.yml with iconv like this:

Code:

$ cp config.yml config.yml.SAFE
$ iconv -f ISO8859-1 -t UTF-8 config.yml -o config.yml

that should fix your problem.

 

Hi, I just wish to add my two cents in the bug:
I have Ubuntu11.10 laptop and my UTF-8 files were working great.
Then after the setup I sent the files to Ubuntu Server and I got that error.
So the problem is in the environment.
I list my 'locale' outputs here:

Laptop - works great:

Code:

mine@tonigh:~/MCServer$ locale
LANG=bg_BG.UTF-8
LANGUAGE=bg:en
LC_CTYPE=bg_BG.UTF-8
LC_NUMERIC="bg_BG.UTF-8"
LC_TIME="bg_BG.UTF-8"
LC_COLLATE=bg_BG.UTF-8
LC_MONETARY="bg_BG.UTF-8"
LC_MESSAGES=bg_BG.UTF-8
LC_PAPER="bg_BG.UTF-8"
LC_NAME="bg_BG.UTF-8"
LC_ADDRESS="bg_BG.UTF-8"
LC_TELEPHONE="bg_BG.UTF-8"
LC_MEASUREMENT="bg_BG.UTF-8"
LC_IDENTIFICATION="bg_BG.UTF-8"
LC_ALL=

The server - throws error:

Code:

root@Server27:~# locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=bg_BG.UTF-8
LANGUAGE=
LC_CTYPE=bg_BG.UTF-8
LC_NUMERIC="bg_BG.UTF-8"
LC_TIME="bg_BG.UTF-8"
LC_COLLATE=bg_BG.UTF-8
LC_MONETARY="bg_BG.UTF-8"
LC_MESSAGES=bg_BG.UTF-8
LC_PAPER="bg_BG.UTF-8"
LC_NAME="bg_BG.UTF-8"
LC_ADDRESS="bg_BG.UTF-8"
LC_TELEPHONE="bg_BG.UTF-8"
LC_MEASUREMENT="bg_BG.UTF-8"
LC_IDENTIFICATION="bg_BG.UTF-8"
LC_ALL=

Hope that helps