How can I have support in Hebrew and russian subtitles at the same time?

Question

I'm trying to set up Firebase's new crash reporting via their docs and running into an error. When I build the project I get this error from the run script phase:

Pods/FirebaseCrash/upload-sym-util.bash:384: error: symbolFileMappings:upsert: Request contains an invalid argument.

After debugging a bit I found the VERBOSE flag and set that for more info as seen below (keys removed)

/Pods/FirebaseCrash/upload-sym-util.bash:376: note: another thing
== Info:   Trying 216.58.216.47...
== Info: Connected to mobilecrashreporting.googleapis.com (216.58.216.47) port 443 (#0)
== Info: TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
== Info: Server certificate: *.googleapis.com
== Info: Server certificate: Google Internet Authority G2
== Info: Server certificate: GeoTrust Global CA
=> Send header, 413 bytes (0x19d)
0000: POST /v1/apps/1:000000000000:ios:0000000000000000/symbolFileMapp
0040: ings:upsert?key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX HTTP/1.1
0082: Host: mobilecrashreporting.googleapis.com
00ad: User-Agent: curl/7.43.0
00c6: Accept: */*
00d3: Content-Type: application/json
00f3: X-Ios-Bundle-Identifier: com.jakecraige.Inventry
0125: Authorization: Bearer XXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
0165: XXXXXXXXXXXXXXXXXXXXXXXXXXX-XXX
0186: Content-Length: 186
019b: 
=> Send data, 186 bytes (0xba)
0000: {"upload_key":"1:000000000000:ios:0000000000000000-00000000-0000
0040: -0000-0000-000000000000","symbol_file_mapping":{"symbol_type":2,
0080: "app_version":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}}
== Info: upload completely sent off: 186 out of 186 bytes
<= Recv header, 26 bytes (0x1a)
0000: HTTP/1.1 400 Bad Request
<= Recv header, 16 bytes (0x10)
0000: Vary: X-Origin
<= Recv header, 15 bytes (0xf)
0000: Vary: Referer
<= Recv header, 47 bytes (0x2f)
0000: Content-Type: application/json; charset=UTF-8
<= Recv header, 37 bytes (0x25)
0000: Date: Mon, 30 May 2016 21:47:10 GMT
<= Recv header, 13 bytes (0xd)
0000: Server: ESF
<= Recv header, 24 bytes (0x18)
0000: Cache-Control: private
<= Recv header, 33 bytes (0x21)
0000: X-XSS-Protection: 1; mode=block
<= Recv header, 29 bytes (0x1d)
0000: X-Frame-Options: SAMEORIGIN
<= Recv header, 33 bytes (0x21)
0000: X-Content-Type-Options: nosniff
<= Recv header, 30 bytes (0x1e)
0000: Alternate-Protocol: 443:quic
<= Recv header, 69 bytes (0x45)
0000: Alt-Svc: quic=":443"; ma=2592000; v="34,33,32,31,30,29,28,27,26,
0040: 25"
<= Recv header, 21 bytes (0x15)
0000: Accept-Ranges: none
<= Recv header, 30 bytes (0x1e)
0000: Vary: Origin,Accept-Encoding
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 2 bytes (0x2)
0000: 
<= Recv data, 138 bytes (0x8a)
0000: 7f
0004: {.  "error": {.    "code": 400,.    "message": "Request contains
0044:  an invalid argument.",.    "status": "INVALID_ARGUMENT".  }.}.
0085: 0
0088: 
== Info: Connection #0 to host mobilecrashreporting.googleapis.com left intact
/Pods/FirebaseCrash/upload-sym-util.bash:385: note: symbolFileMappings:upsert: The metadata for the symbol file failed to update.

So it looks like the POST to mobilecrashreporting.googleapis.com/v1/apps/$GOOGLE_APP_ID/symbolFileMappings:upsert?key=$FIREBASE_API_KEY is failing.

Looking through all the params, they seem to match up with my config and nothing is empty, so I'm not really sure what to do next.

Has anyone else run into this? Idea on how to fix it?

Thanks!

Solution 1

There are several steps you have to take:

1. You must use a DirectShow filter for subtitles rendering that supports Unicode (like DirectVobSub). Your movie playing software must of course use that filter, it's impossible to say how exactly since you didn't say what movie player you use.

2. You must use a font that has all the characters you need (e.g. Arial, the default, works fine, don't change it to something fancy as Hebrew support in fonts is patchy at best)

3. Most important: your subtitles must be encoded in UTF-8.

   a. If your subtitles are external, just use software like iconv (Windows version) to convert your subtitles into UTF-8, just make sure you use correct source encoding (Windows-1255 for Hebrew and Windows-1251 for Russian) and the target must always be UTF-8.

   b. If your subtitles are internal to your video file, you'll need to extract them first. For MKV files, use mkvtoolnix to extract subtitles, transcode them same as in "a" above and pack them back into MKV. For other video file containers, use their appropriate tools to extract subtitles.

Solution 2

First, use a font that has characters for all languages (actually scripts) you want to use, like Arial. Otherwise, characters that the font doesn't have will appear as squares.

Now, you need to get the encoding right, or you'll get incorrect/garbage characters like you mention. You have 3 options:

1. Video with subtitles encoded in the video itself

That's the easiest, If you can get it...

2. Subtitles in Language-Specific Charset, Like Windows-1255 (Hebrew) or Windows-1251 (Russian)

• Use subtitles that are encoded in the language-specific charset - most are nowadays.
• Tell your player/DirectShow filter which charset the subtitles are encoded in. In [VLC], set it in Tools -> Preferences... -> Subtitles & OSD -> Default encoding. You will have have to set it every time - not fun.
• If your player/DirectShow filter doesn't have that option, you will have to tell it via the system's non-english support language, and reboot. Even less fun.

3. Subtitles in [UTF-8] - the way of the future

• Use a player/DirectShow filter which is able to show UTF-8 subtitles, like example VLC or [DirectVobSub].
• Use subtitles that are encoded in UTF-8.

Unfortunately, most available subtitles nowadays are not encoded in UTF-8. You can convert them:

a. If your subtitles are external, use software like [iconv] ([Windows version]) or [SubtitleEdit]. Make sure you use correct source encoding (Windows-1255 for Hebrew and Windows-1251 for Russian) and target encoding (UTF-8).

b. If your subtitles are internal to your video file, you'll need to extract them first. For MKV files, use [mkvtoolnix] to extract subtitles, convert them like in "a" above and pack them back into MKV. For other video file containers, use their appropriate tools.

Comments

[11alex11]

My player is vlc. What do I need to configure and how?

[haimg]

@11alex11: See "subtitles" tab in preferences. If you didn't touch there anything, the defaults are just fine, but you need to follow #3. Or, alternatively, you can change the "default" language in VLC options each time you change from Russian to Hebrew subtitles or back.

[jakecraige]

Hey Robert, this is what I'm seeing for that line MODULE mac x86_64 45FA3BDF3866336FACCE04D28744BAF30 Inventry

[jakecraige]

I'm actually seeing two different strings like that, MODULE mac x86_64 45FA3BDF3866336FACCE04D28744BAF30 Inventry and MODULE mac x86_64 45FA3BDF3866336FACCE04D28744BAF30 Inventry.FILE, perhaps that second one is the culprit? There's something weird going on with newlines there. A bit more context here: gist.github.com/jakecraige/79ac463e7da0e5605bdf502bdd0edf47

[Rizwan Sattar]

@robert-menke I have the same issue, and verified after running on verbose=3 that the MODULE mac stuff was fine. However, it revealed the arguments being passed up for upsert, and one of them, app_version is that's causing the failure. In my case I was doing a debug build, and my debug build version is 3.7-dev (i.e. not all numbers and dots). Changing this to 3.7 makes the upsert request complete successfully. I'd be interested to see if @jakecraige has a similar issue with the app version.

[jakecraige]

@RizwanSattar That was it! I had a build script that sets the build number to a git commit hash so when I removed that and set to a standard version number it worked.

[Robert Menke]

Sorry about that. The backend team rejects any version that is not in Dewey decimal format (=~ m/^\d+(?:\.\d+)*$/). It's not a valid restriction as the version is only used for labeling, but they felt it was necessary to filter out corrupted uploads.

[Rizwan Sattar]

Thanks for writing back @robert-menke. I figured it must be a restriction for some reason, though it hampers a common workflow for developers, as you know. Could you elaborate on the corrupted uploads? I'm not sure how a version string can help detect that. Regardless, I'd add an enhancement request to provide a more descriptive error message (than "contains invalid argument") when you guys get a chance. I've modified my Script to only upload on Release builds, skipping debug builds.

[Robert Menke]

The general idea at the time was that only formal release builds would be symbolicated, and anything not looking like a formal release would be held in suspicion. In retrospect, it was a poor decision—the version is used only for display purposes; the LC_UUID is the functional identifier. We are working on having that restriction removed so any arbitrary string would be accepted as a version identifier.