UTF-8 and Unicode Filename Hygiene for Cloud Mac CI

The same iOS project may compile successfully in a developer’s terminal yet fail in non-interactive CI on a cloud Mac when it reads Chinese configuration data, resolves resource names containing accented characters, or captures logs. These failures are especially troublesome because they are often mistaken for broken dependencies: a file clearly exists, but a script reports that it cannot be found; two paths look identical in the logs, but Git treats them as different. The root cause is usually not UTF-8 itself. It is a mismatch between the job entry point, language runtimes, and the repository’s interpretation of filename encoding.

First identify the layer where the mismatch occurs

Do not start by changing system settings. First separate the problem into four layers: the job process’s locale, the script runtime’s default encoding, the behavior of individual commands in a pipeline, and the Unicode normalization form used by filenames.

Run the following both in an interactive terminal and at the beginning of the CI job:

printf 'shell=%s\n' "$SHELL"
locale
printf 'LANG=%s\nLC_ALL=%s\n' "${LANG:-unset}" "${LC_ALL:-unset}"
python3 -c 'import locale,sys; print(locale.getpreferredencoding(False), sys.getfilesystemencoding())'
ruby -e 'p [Encoding.default_external, Encoding.default_internal]'

Save the output as a regular build artifact rather than leaving it only in a scrolling log. If the terminal reports en_US.UTF-8 while the variables are empty inside the job, the problem is at the process startup boundary. If the locale matches but Ruby still reads files using another encoding, check whether the script explicitly passes an incorrect option.

“The machine supports UTF-8” does not mean “every job uses UTF-8.” The environment received by the process is the CI system’s real configuration, not the environment shown in a login terminal.

Pin UTF-8 at the job entry point

The most reliable approach is to set the environment in a runner wrapper script or at the pipeline job entry point instead of relying on .zshrc. Non-interactive shells usually do not load these files as developers expect, and processes started by launchd may receive a different environment.

#!/bin/zsh
set -euo pipefail

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

locale
exec "$@"

Save the script as ci/run-utf8.zsh, make it executable, and route every build command through it:

chmod +x ci/run-utf8.zsh
ci/run-utf8.zsh ./ci/build.zsh

Do not globally run export LC_ALL=C to “fix” logs. The C locale is appropriate for an individual command that requires stable bytewise sorting, but it also changes character classification, case conversion, and regular-expression behavior. When it is genuinely needed, limit its scope to the command itself, as in LC_ALL=C sort input.txt.

Also verify how each script reads files

Python should specify encoding="utf-8" explicitly, while Ruby can use File.read(path, encoding: "UTF-8"). When reading text line by line in Shell, use IFS= read -r to prevent backslashes and leading or trailing whitespace from being rewritten. JSON, plist, and YAML files should be handled by their corresponding parsers rather than by using grep and sed to guess their structure.

Audit Unicode filename normalization

The character é can be represented as a single code point or as a letter followed by a combining mark. The two forms look identical but have different byte sequences. When a repository moves between macOS, Linux, and archive files, this difference can lead to duplicate resources, scripts that cannot be found, or commits containing case-only changes that do not apply correctly.

Add a read-only check to CI:

from pathlib import Path
import sys
import unicodedata

bad = []

for path in Path(".").rglob("*"):
    if ".git" in path.parts:
        continue
    raw = path.as_posix()
    normalized = unicodedata.normalize("NFC", raw)
    if raw != normalized:
        bad.append((raw, normalized))

for raw, normalized in bad:
    print(f"non-nfc: {raw!r} -> {normalized!r}")

sys.exit(1 if bad else 0)

Save it as ci/check_unicode_paths.py and run it before installing dependencies. The checker should report problems without renaming anything automatically, because a normalized path may conflict with an existing file. Before fixing a path, confirm its actual name in the Git index:

git -c core.quotepath=false ls-files
git -c core.quotepath=false status --short

Then use git mv and commit the rename separately. For a case-only change, first move the file to a temporary name and then move it to the target name. Afterward, clone and build the repository in a new directory. Do not validate the change in an old workspace already affected by filesystem caching.

Preserve the original evidence in logs

After an encoding failure passes through tee, a log collector, or JSON escaping, the original bytes may be replaced, leaving only question marks or replacement characters. During investigation, preserve both readable logs and a byte-level view of critical files:

file -I Config/环境.json
xxd -g 1 -l 96 Config/环境.json
plutil -lint App/Info.plist

Do not treat the output of file as definitive; it provides clues, not proof. The program that actually reads the file should still specify UTF-8 explicitly and return a nonzero status when decoding fails. Build scripts should also avoid ignoring the preceding command’s exit status after running cmd | tee build.log. In zsh, inspect $pipestatus, or separate critical commands from log collection.

Logs should also record an escaped representation of the failing path. Python’s repr() and Ruby’s String#dump are better than direct output for identifying combining characters, line breaks, and invisible spaces.

Establish a pre-merge gate and repair order

A maintainable gate should check known facts without modifying the repository on its own. Run the checks in this order:

1. Print the locale and the encoding used by each runtime.
2. Check whether repository paths use NFC.
3. Check for duplicate names after normalization.
4. Read critical text configuration files with an explicit encoding.
5. Run a minimal build in a clean workspace.
6. Save the environment probe, path report, and failure logs.

If a job fails only on one runner, compare the environment probes first and then compare the Git indexes. Do not begin by clearing every cache. If the path report identifies a problem, fix the filename and create a separate commit. If the paths are valid but parsing still fails, inspect the file’s encoding declaration and the parameters used to read it. This sequence keeps environment, repository, and tool failures separate.

Jobs on RentMini should keep these checks in the repository and version them with the project instead of relying on manual settings applied to a particular cloud Mac. When the entry-point script, Unicode checker, and log evidence all go through code review, rebuilding a workspace or switching nodes will produce consistent behavior.

Frequently asked questions