Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 155 additions & 0 deletions tools/i18n/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# WLED i18n Architecture

## Overview

Two-repository architecture separating **build toolchain** (core repo) from **translation files** (community repo).

```
┌─────────────────────────────────────────────────────────────┐
│ Core Repo (WLED/tools/i18n/) │
│ ├── extract.py # Extract translatable strings from HTML │
│ ├── build.py # Apply translations at build time │
│ └── locales/ # Locale configuration │
└─────────────────────────────────────────────────────────────┘
↓ calls
┌─────────────────────────────────────────────────────────────┐
│ Translation Repo (WLED-translations/<locale>/) │
│ ├── static.json # Layer 1: Static HTML (429 entries) │
│ ├── js.json # Layer 2: JS strings (45 entries) │

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Count actual JS strings in the English template
if [ -f tools/i18n/locales/en_template.json ]; then
  echo "Counting js: keys in en_template.json:"
  rg -o '"js:[^"]+":' tools/i18n/locales/en_template.json | wc -l
else
  echo "en_template.json not found; run extract.py first"
fi

Repository: wled/WLED

Length of output: 95


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show the referenced lines in both docs
echo "---- tools/i18n/ARCHITECTURE.md (lines 1-220) excerpts around 18, 32, 152 ----"
sed -n '10,40p' tools/i18n/ARCHITECTURE.md
sed -n '140,170p' tools/i18n/ARCHITECTURE.md
echo

echo "---- README.md (excerpts around lines 46 and 132) ----"
sed -n '35,60p' README.md
sed -n '120,150p' README.md
echo

# Find occurrences of the claimed counts
echo "---- Occurrences of '45 entries' and '716 entries' ----"
rg -n "45 entries|716 entries|Layer 2|JS strings" tools/i18n/ARCHITECTURE.md README.md || true
echo

# Compute js-string count by parsing JSON (authoritative)
if [ -f tools/i18n/locales/en_template.json ]; then
  echo "---- Count js:* keys in tools/i18n/locales/en_template.json via JSON parsing ----"
  python3 - <<'PY'
import json
from pathlib import Path

p = Path("tools/i18n/locales/en_template.json")
data = json.loads(p.read_text(encoding="utf-8"))

keys = list(data.keys()) if isinstance(data, dict) else []
js_keys = [k for k in keys if isinstance(k, str) and k.startswith("js:")]
print("Total keys:", len(keys))
print("js:* keys:", len(js_keys))
print("First 10 js keys:", js_keys[:10])
PY
else
  echo "tools/i18n/locales/en_template.json not found"
fi

Repository: wled/WLED

Length of output: 2021


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "---- Repo root files (top-level) ----"
ls -la

echo
echo "---- Find README* files ----"
fd -H --max-depth 4 "README" -t f . || true
# Also fall back to common names if fd misses
ls -la README* 2>/dev/null || true

echo
echo "---- Authoritative count: js:* keys in en_template.json ----"
python3 - <<'PY'
import json
from pathlib import Path

p = Path("tools/i18n/locales/en_template.json")
data = json.loads(p.read_text(encoding="utf-8"))
js_keys = [k for k in data.keys() if isinstance(k, str) and k.startswith("js:")]
print("tools/i18n/locales/en_template.json js:* keys:", len(js_keys))
PY

echo
echo "---- Search docs for '716 entries' and '45 entries' near translation coverage ----"
rg -n "716 entries|45 entries|Layer 2: JS strings|js\\.json" -S . || true

Repository: wled/WLED

Length of output: 3680


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "---- tools/i18n/ARCHITECTURE.md: around claimed '45' ----"
sed -n '1,60p' tools/i18n/ARCHITECTURE.md
echo
sed -n '110,170p' tools/i18n/ARCHITECTURE.md

echo
echo "---- tools/i18n/README.md: around claimed '716' ----"
sed -n '35,70p' tools/i18n/README.md
echo
sed -n '110,160p' tools/i18n/README.md

echo
echo "---- en_template.json structure peek ----"
python3 - <<'PY'
import json
from pathlib import Path
p = Path("tools/i18n/locales/en_template.json")
txt = p.read_text(encoding="utf-8")
print("File bytes:", len(txt))
print("First 300 chars:\n", txt[:300].replace("\n","\\n"),"\n")
try:
    data = json.loads(txt)
    print("Top-level type:", type(data).__name__)
    if isinstance(data, dict):
        print("Top-level dict keys sample:", list(data.keys())[:10])
    elif isinstance(data, list):
        print("Top-level list length:", len(data))
except Exception as e:
    print("json.loads failed:", e)
PY

echo
echo "---- en_template.json: count js:* occurrences (same logic as original comment) ----"
rg -o '"js:[^"]+":' tools/i18n/locales/en_template.json | wc -l

echo
echo "---- en_template.json: count unique js:* keys via regex extraction ----"
python3 - <<'PY'
import re
from pathlib import Path
txt = Path("tools/i18n/locales/en_template.json").read_text(encoding="utf-8")
keys = re.findall(r'"(js:[^"]+)"\s*:', txt)
print("Unique js:* keys:", len(set(keys)))
print("First 10 unique js keys:", sorted(set(keys))[:10])
PY

Repository: wled/WLED

Length of output: 7763


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "---- Search for count/stat logic in tools/i18n/*.py ----"
rg -n "JS strings|Layer 2|716|45|js\.json|en_template\.json|--stats" tools/i18n -S || true

echo
echo "---- Show extract.py relevant sections ----"
ls -ლა tools/i18n
python3 - <<'PY'
from pathlib import Path
p=Path("tools/i18n/extract.py")
print("extract.py exists:", p.exists(), "size:", p.stat().st_size if p.exists() else None)
PY
sed -n '1,220p' tools/i18n/extract.py

echo
echo "---- Show build.py relevant sections (first 260 lines) ----"
sed -n '1,260p' tools/i18n/build.py

echo
echo "---- Search for 'entries' formatting in docs/scripts ----"
rg -n "entries" tools/i18n -S || true

Repository: wled/WLED

Length of output: 2113


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Extract the extract.py stats implementation
sed -n '260,420p' tools/i18n/extract.py

echo
echo "---- extract.py: look for counting logic ----"
rg -n "stats|JS strings|716|45|js\.json|innerHTML|innerText|alert" tools/i18n/extract.py -S

echo
echo "---- extract.py: show nearby lines around JS-string stats marker ----"
rg -n "JS strings" tools/i18n/extract.py -n -S || true

Repository: wled/WLED

Length of output: 4408


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Run the i18n extractor in stats mode to see how it counts "JS strings"
python3 tools/i18n/extract.py --stats | head -n 200

echo
echo "---- Additionally, regenerate _template.json and count js:* keys inside it ----"
python3 tools/i18n/extract.py --output tools/i18n/locales/_template_generated.json
python3 - <<'PY'
import re, pathlib
p = pathlib.Path("tools/i18n/locales/_template_generated.json")
txt = p.read_text(encoding="utf-8")
keys = re.findall(r'"(js:[^"]+)"\s*:', txt)
print("Generated _template_generated.json unique js:* keys:", len(set(keys)))
print("Occurrences of js:* keys in JSON:", len(keys))
PY

Repository: wled/WLED

Length of output: 2443


Fix inconsistent Layer 2 JS-string entry counts in i18n documentation

  • tools/i18n/ARCHITECTURE.md states Layer 2 js.json has 45 entries.
  • tools/i18n/README.md states Layer 2 js.json has 716 entries.
  • The current English template (tools/i18n/locales/en_template.json) contains 43 unique js:* keys.

Update both docs to the same count (or explicitly document the counting method if 716 comes from a different metric/source).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/i18n/ARCHITECTURE.md` at line 18, The Layer 2 js.json entry counts are
inconsistent across docs and the actual template: reconcile and correct the
documentation by (A) counting unique keys prefixed with "js:" in
tools/i18n/locales/en_template.json (current count = 43) and then update the
count cited in tools/i18n/ARCHITECTURE.md and tools/i18n/README.md to that
number, or (B) if the 716 figure comes from a different metric, explicitly
document the counting method and source (e.g., combined keys across locales vs
unique template keys) in both files; ensure you reference the files/tools
mentioned (tools/i18n/locales/en_template.json, tools/i18n/ARCHITECTURE.md,
tools/i18n/README.md, js.json) so the change is applied consistently.

│ ├── effects.json # Layer 3: Effect names (216 entries) │
│ ├── palettes.json # Layer 4: Palette names (72 entries) │
│ └── metadata.json # Version, coverage, maintainer │
└─────────────────────────────────────────────────────────────┘
```

---

## Four-Layer Translation Architecture

| Layer | Content | File | Implementation | Coverage |
|-------|---------|------|----------------|----------|
| **L1** | Static HTML | `static.json` | Regex replacement in HTML text | 429 strings |
| **L2** | JS strings | `js.json` | Replace JS string literals | 45 strings |
| **L3** | Effect names | `effects.json` | C++ PROGMEM `#undef` + redefine | 216/216 (100%) |
| **L4** | Palette names | `palettes.json` | C++ PROGMEM array replacement | 72/72 (100%) |

---

## Build Flow

### PlatformIO Configuration

```ini
# platformio_override.ini
[env:esp32dev_zh_CN]
extends = env:esp32dev
custom_usermods = https://github.com/foxlesbiao/WLED-translations
build_flags = ${env:esp32dev.build_flags} -D WLED_LOCALE=zh_CN
extra_scripts = pre:tools/i18n/build.py
```

PlatformIO automatically clones the translation repo to `.pio/libdeps/`.

### Build Steps

1. PlatformIO clones `WLED-translations` to `.pio/libdeps/<env>/WLED-translations/`
2. `build.py` auto-detects translations in `.pio/libdeps/`
3. Applies L1/L2 translations via regex replacement
4. Generates `i18n_effects.h` / `i18n_palettes.h` for L3/L4 (PROGMEM replacement)
5. Output to `build/i18n/<locale>/`

---

## How Dynamic Content Works

The key insight: **PROGMEM replacement happens at compile time**, so JSON endpoints return translated strings automatically.

```
Browser ESP32 Firmware
│ │
├─ GET /json/palettes ─────────→│ PROGMEM array was replaced at compile time
│ ← {"0":"默认","1":"* 随机循环",...} ↓
│ │ palettes.json → i18n_palettes.h
├─ GET /json/effects ─────────→│ #undef _data_FX_MODE_STATIC
│ ← {"0":"常亮","1":"闪烁",...}│ #define _data_FX_MODE_STATIC "常亮"
```

No firmware code changes needed. The C++ PROGMEM strings are the single source of truth.

---

## Grammar and Word Order

WLED UI uses **short labels**, not full sentences:

| Pattern | Example | i18n Impact |
|---------|---------|-------------|
| Single word | "Brightness", "Speed" | ✅ No issue |
| Label + value | "255 segments" | ✅ Works ("255 个段") |
| Full sentences | Almost none | ✅ N/A |
| Plural forms | Not used | ✅ N/A |
| Date formats | Not used in UI | ✅ N/A |

The architecture **intentionally avoids** complex i18n patterns (ICU MessageFormat, plural rules) because WLED's UI doesn't need them.

---

## What's NOT Translated (By Design)

| Content | Reason |
|---------|--------|
| User-defined preset names | Belongs to user |
| Usermod settings pages | Dynamic HTML from firmware, varies by hardware |
| System info (IP, memory) | Universal data |
| Effect slider tooltips | Generated from mode data arrays |

---

## Repository Structure

### Core repo (WLED)

```
tools/i18n/
├── extract.py # String extraction tool
├── build.py # Build-time translation applicator
├── ARCHITECTURE.md # This file
├── README.md # Usage documentation
└── locales/
└── en_template.json # English template for translators
```

### Translation repo (WLED-translations)

```
<locale>/
├── static.json # Layer 1: Static HTML text
├── js.json # Layer 2: JavaScript strings
├── effects.json # Layer 3: Effect names (PROGMEM)
├── palettes.json # Layer 4: Palette names (PROGMEM)
└── metadata.json # {"version":"1.0","coverage":"100%","maintainer":"..."}
en_template/ # English template for translators
```

---

## Adding a New Language

1. Fork `WLED-translations`
2. Copy `en_template/` to `<locale>/`
3. Translate JSON files
4. Submit PR to translation repo

No changes to WLED core needed.

---

## Coverage Summary (zh_CN)

| Layer | Content | Count | Status |
|-------|---------|-------|--------|
| L1 | Static HTML | 429 | ✅ Complete |
| L2 | JS strings | 45 | ✅ Complete |
| L3 | Effect names | 216/216 | ✅ 100% |
| L4 | Palette names | 72/72 | ✅ 100% |
| **Total** | | **762** | **100%** |
187 changes: 187 additions & 0 deletions tools/i18n/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
# WLED i18n Toolchain

Build-time internationalization for WLED Web UI. Translates HTML/JS strings at compile time — replaces English text, does not add to it.

## How It Works

```
English HTM files (wled00/data/)
extract.py → en_template.json
Translator creates locale repo (WLED-translations)
build.py → Translated HTM files
npm run build → html_*.h / js_*.h (C headers)
pio run → Firmware with translated UI
```

## Quick Start (User)

Add to `platformio_override.ini`:

```ini
[env:esp32dev_zh_CN]
extends = env:esp32dev
custom_usermods = https://github.com/foxlesbiao/WLED-translations
build_flags = ${env:esp32dev.build_flags} -D WLED_LOCALE=zh_CN
extra_scripts = pre:tools/i18n/build.py
```

Then: `pio run -e esp32dev_zh_CN`

PlatformIO automatically clones the translations repo to `.pio/libdeps/`. The build script finds translations there automatically.

## Quick Start (Translator)

Translations live in a separate repo: [WLED-translations](https://github.com/foxlesbiao/WLED-translations)

```
WLED-translations/
├── library.json # PlatformIO dependency manifest
├── zh_CN/
│ ├── static.json # Layer 1: static HTML (429 entries)
│ ├── js.json # Layer 2: JS strings (716 entries)
│ ├── effects.json # Layer 3: effect names (216 entries)
│ ├── palettes.json # Layer 4: palette names (72 entries)
│ └── metadata.json
├── de_DE/
│ └── ...
└── en_template/ # English template (generated by extract.py)
```

### Adding a new language

```bash
# 1. Clone translations repo
git clone https://github.com/foxlesbiao/WLED-translations
cd WLED-translations

# 2. Generate English template (from WLED source)
python3 /path/to/WLED/tools/i18n/extract.py --stats
cp -r /path/to/WLED/tools/i18n/locales/* en_template/

# 3. Create your locale
mkdir de_DE
cp en_template/*.json de_DE/

# 4. Fill in "translation" fields in each JSON file
# 5. Commit and push
```

## Quick Start (Developer)

### Extract strings (generate template)

```bash
python3 tools/i18n/extract.py --stats
# Output: tools/i18n/locales/en_template.json
```

### Build translated firmware

```bash
# Build translated HTM files
python3 tools/i18n/build.py --locale zh_CN \
--translations-dir /path/to/WLED-translations/zh_CN \
--output-dir build/i18n/zh_CN

# Validate translations
python3 tools/i18n/build.py --locale zh_CN --validate

# Build web UI headers
npm ci && npm run build

# Build firmware
pio run -e esp32dev
```

### Version updates (diff)

When WLED releases a new version, compare templates to find changes:

```bash
# Generate old and new templates
python3 tools/i18n/extract.py --stats # on old version
cp tools/i18n/locales/en_template.json en_template_old.json

python3 tools/i18n/extract.py --stats # on new version
cp tools/i18n/locales/en_template.json en_template_new.json

# Compare
python3 tools/i18n/diff.py --old en_template_old.json --new en_template_new.json
```

Output shows added/removed/modified strings. Translators update only changed entries.

## Translation Search Order

`build.py` searches for translations in this order:

1. `--translations-dir` (explicit path)
2. `.pio/libdeps/*/WLED-translations/<locale>/` (PlatformIO out-of-tree)
3. `tools/i18n/locales/<locale>.json` (local fallback)

## Coverage

| Layer | Content | Method | Count |
|-------|---------|--------|-------|
| 1. Static HTML | Labels, buttons, placeholders | DOM text matching | 429 |
| 2. JS strings | `alert()`, `innerHTML`, `innerText` | Script block regex | 716 |
| 3. Effect names | FX names in `FX.cpp` | PROGMEM replacement | 216 |
| 4. Palette names | Palette names in `FX_fcn.cpp` | PROGMEM replacement | 72 |

## Known Limitations

### Cannot translate (technical)

- **Dynamic runtime text** — OTA update errors, Info page content, usermod settings, Pin Info page
- **External tools** — PixelForge add-ons (always English, downloaded on-the-fly)
- **JavaScript template literals** — strings with `${...}` interpolation
- **C++ server-side strings** — ~12 strings in `xml.cpp` need `#ifdef WLED_LOCALE_*`

### Language-specific issues (acknowledged)

- Word order differences (e.g., "X of Y" patterns)
- Number formats (decimal point vs comma)
- Grammar rules (singular/plural, countable/uncountable)
- Date formats

These are known limitations. The tool handles short labels and UI fragments, not full sentences.

## Layer 3/4: Effect & Palette Names

Effect names (216) and palette names (72) are translated via C++ PROGMEM replacement:

```c
// locale_effects.h (auto-generated)
#pragma once
#ifdef WLED_LOCALE
#undef _data_FX_MODE_STATIC
static const char _data_FX_MODE_STATIC[] PROGMEM = "常亮";
// ...
#endif
```

The `.h` files are generated in the translation repo. Users copy them to their local build.

## Architecture

```
WLED (core repo)
└── tools/i18n/
├── extract.py # Extract strings from HTML/JS
├── build.py # Apply translations (pre-build script)
├── diff.py # Compare template versions
└── README.md

WLED-translations (community repo)
├── zh_CN/
│ ├── static.json # Layer 1: HTML text
│ ├── js.json # Layer 2: JS strings
│ ├── effects.json # Layer 3: effect names
│ └── palettes.json # Layer 4: palette names
└── en_template/ # English template
```
Loading
Loading