Parsing system locales

[Finchiedev]

Hello all!

I've been experimenting with integrating icu4x into user-facing applications for a while now, and was very excited to see the progress made by GSoC student @ashu26jha in #5081 and #5358 on retrieving the system locale. If there is interest in doing so, I would be very interested in contributing to the env_preferences crate that came out of this work to make it easier to use out-of-the-box.

This is motivated by my computer (x86_64-unknown-linux-gnu) returning Ok({All: "C"}) from the linux implementation of env_preferences::get_locales. Ideally, env_preferences::get_locales would return Locale instead of String, so I can immediately use the unwrapped result with icu_datetime, icu_plurals etc. I assume parsing these strings into locales is the kind of logic that should be handled by icu4x rather than duplicated across downstream users - but please correct me if I'm wrong!

At least on Linux, I would expect to solve this according to TR35's section on legacy variants. If possible, I'd love feedback on this solution, and if there's interest in such a contribution at all. I am very happy to do the work but don't want to interfere with anyone's existing progress or create an unwanted PR :)

[Manishearth]

Yes, we'd welcome contributions!

I think the legacy variant section is correct, though note that ICU4X doesn't ship any data around these by default.

@sffc to confirm that this is the route we should take.

[sffc]

Thanks for your interest in this area!

CC @robertbastian @zbraniecki to advise on design.

[zbraniecki]

Ideally, env_preferences::get_locales would return Locale instead of String, so I can immediately use the unwrapped result with icu_datetime, icu_plurals etc. I assume parsing these strings into locales is the kind of logic that should be handled by icu4x rather than duplicated across downstream users - but please correct me if I'm wrong!

You are correct, the student that started this crate did not get to this phase, but the intention is to normalize API between env_preferences and anything else use Locales everywhere.

IIRC the original idea is that there should be that you can use per-OS specialization to get an unprocessed response from the OS (so, for example windows::get_raw_locales or some similar name returns Strings), and have universal cross-OS method like env_preferences::get_requested_locales_lossy(options) -> Vec<Locale> which returns parsed locales, omitting ones that could not be resolved and env_preferences::get_requested_locales(options) -> Result<Vec<Locale>, _> failing on errors.

I'd be happy to brainstorm that with you and review your design.

[Finchiedev]

Thanks! I've managed to set aside some time to work on this (POSIX-only to start) and have run into a few problems where the solution is unclear, so I've written them down along with how I've solved them so far. I would very much appreciate any feedback on improvements or alternative solutions you may have!

If it would be easier to open a draft pull request and discuss there, please let me know :)

P.S. I've generated a dataset for testing which is available as a gist

1. Default "C"/"POSIX" locale

From reading the POSIX spec on localization environment variables and locales in general, it seems like the default locale "C"/"POSIX" doesn't imply a specific language as the default - so if the user's locale is "C" or "POSIX" env_preferences couldn't assume that to be equivalent to "en-US" for example.

I've solved this for now by returning an Option, where Some is any non-default locale and None is the default "C"/"POSIX" locale, so that the user can choose their own default:

// Rough example of a French-speaking app choosing their own default locale
let locale: Option<Locale> = get_locales_lossy();
let locale = locale.unwrap_or(locale!("fr"));

I'm a bit concerned about this solution deliberately ignoring the defaults that the "C"/"POSIX" locale specifies. For example, if an app chooses a default locale with different collation rules to the "C" locale, it may behave in a way that ignores what could be a user-defined preference. This would also ignore modifiers, so preferences such as "C@euro" would not be respected - even though the user has explicitly requested to use Euros as the currency symbol.

Not saying this is necessarily the wrong thing to do, but would very much appreciate some input on the what the right choice is for this library.

2. Converting POSIX locales to BCP-47 identifiers

From what I understand, the POSIX format is language[_territory][.codeset][@modifier]. For now, I'm parsing that format, and creating an icu_locale::Locale equivalent to {language}-{territory}-u-va-posix. For example, the POSIX locale en_US.utf8 would become en-US-u-va-posix.

This has worked quite well on my test dataset, but it feels like a flimsy solution and just wanted to check if I'm missing anything here. The codesets and modifiers are deliberately not handled yet, and questions 4 & 5 below are on that topic.

3. Data source for legacy variants (e.g. "bokmal", "nynorsk")

I've found some POSIX locales that use legacy variants, for example "bokmal" or "nynorsk", so I'd expect I need to convert them to BCP-47 identifiers as per the legacy variant mappings table.

Should I use CLDR data to perform these conversions, or just hard-code the table via a simple match statement? I'd expect CLDR to be the better solution, but have been quite overwhelmed with the dataset and haven't managed to find anything specific to use as a data source. If someone would be able to point me in the right direction, that would be much appreciated!

4. Data source for known aliases (e.g. "czech", "korean")

It seems like some POSIX locales use a full name, for example "czech" or "thai". I'm sure that CLDR has this information somewhere, but again I've not been able to find what exactly to use as a source of known aliases.

5. POSIX modifiers (e.g. "@euro", "@Latin")

From what I can tell, most modifiers are implementation-defined and aren't formally specified. They seem to mostly be a mix of currency preferences (e.g. "@euro"), collation rules (e.g. "@abegede") and scripts (e.g. "@Latin", "@cyrillic"). I've also seen some other modifiers such as "@iqtelif", "@valencia" and "@saaho" which I'm unsure how to categorize.

I'd expect the CLDR data wouldn't be able to match all of these modifiers, so I'm wondering how much of the modifiers can be handled using CLDR vs a hard-coded list of well-known modifiers and their mappings.

6. Alternative charsets (e.g. "iso88591", "iso88599")

For now, I am parsing the codeset field but ignoring it. Is it worth returning an error on non-utf8 locales, or is there a better way to handle them?

[robertbastian]

• We don't really want to do Option<Locale>, as Locale already has the special Locale::unknown() (und), which mainly behaves like en, but is more neutral where it can be. The unknown locale can also be extended into things like und-u-co-foo.
• Regarding -u-va-posix, this could be expressed as a Variant as well, i.e. de-posix. We actually have data in this format (en-US-posix in collator), and we don't have any data using -u-va-posix, so I think we should parse to a variant.
• Similar with @valencia, I've seen this as both a variant and a subdivision tag (-u-sd-valencia). We don't currently have data keyed by either.
• For @euro, Unicode has -u-cu-eur, but ICU4X will not do anything with that. We don't consider currency a user settings, it is a property of a numeric value, and the only way to convert currencies is to use variable exchange rates.
• I'm not sure if we'd want to error on non-UTF-8 charsets. ICU4X will always return UTF-8, and our Rust and FFI APIs are documented as such (except for a few UTF-16 methods).

[Finchiedev]

Thank you very much! It seems like the best approach for now is to hard-code support for all of the modifier edge cases you mentioned, and fall back to checking the modifier against valid script names if possible.

I’ll make sure to remove Option from the return signature in favour of “und”, thanks for the feedback!

Ignoring the codeset section sounds reasonable, especially in a _lossy function. I assume there’s no way to encode alternative charsets into a BCP-47 identifier anyways.

[Finchiedev]

I've almost finished my draft POSIX implementation, and the only edge case in my dataset that isn't handled is full locale names, for example "french", "deutsch", "eesti". I'd like to use CLDR if possible to match each full name to the language ID, for example "french"->"fr".

Most of the locale names (29/35) have a matching language entry in CLDR's common/main/en.xml, for example "french" corresponds to line 216:

<language type="fr">French</language>

I'd like to use this CLDR data if possible, and potentially handle the remaining 6/35 edge cases manually. If anyone is able to let me know how I can integrate the relevant data into my Rust code, that would be very much appreciated! The icu_provider_source crate seems relevant, but I'm not sure what the exact process is for adding a data source like this.

Thought it might be worth mentioning that the CLDR file langtags.json seems to match all 35 of these edge cases under the "localnames" field, but I'm not sure if that is directly usable.

List of full locale names in en.xml

catalan
croatian
czech
danish
dutch
estonian
finnish
french
galician
german
greek
hebrew
hungarian
icelandic
italian
japanese
korean
lithuanian
norwegian
polish
portuguese
romanian
russian
slovak
slovenian
spanish
swedish
thai
turkish

List of full locale names NOT in en.xml

dansk
deutsch
eesti
galego
hrvatski
slovene

[robertbastian]

I don't think you should use CLDR data for this. The data in en.xml is for user-facing display names, and they are subject to change. For example, TR's English name changed from "Turkey" to "Türkiye" a while ago. You need stability for this, so you should hardcode them for now, and then we can look into adding them to CLDR in a stable locale-agnostic file.

[Finchiedev]

I've found 2 valid Windows locales that Locale::try_from_str returns an error on, and would very much appreciate feedback on how to correctly handle them if anyone is able to help!

1. es-ES_tradnl
2. zh-yue-HK

[robertbastian]

1. According to https://learn.microsoft.com/en-us/dotnet/api/system.globalization.cultureinfo?view=net-8.0, this would be something like es-ES-u-co-trad
2. zh has historically been used for "Chinese", whereas in BCP-47 it means "Mandarin Chinese". Cantonese now has its own language tag, yue.

[Finchiedev]

Perfect, thank you! I will add those in :)