Skip to content

Commit 676f92e

Browse files
ch4r10t33rch4r10t33r
committed
leanpoint: add convert-validator-config.py, sync-leanpoint-upstreams.sh, update README
1 parent f307de9 commit 676f92e

4 files changed

Lines changed: 232 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ A single command line quickstart to spin up lean node(s)
2424
3. **yq**: YAML processor for automated configuration parsing
2525
- Install on macOS: `brew install yq`
2626
- Install on Linux: See [yq installation guide](https://github.com/mikefarah/yq#install)
27+
4. **Python 3 + PyYAML** (optional, for leanpoint upstreams sync): Required only if you use the automatic leanpoint upstreams sync (tooling server). Install with `pip install pyyaml` or `uv add pyyaml`.
2728

2829
## Quick Start
2930

@@ -82,6 +83,29 @@ Grafana is started with the two pre-provisioned dashboards from [leanMetrics](ht
8283

8384
> **Note:** The `--metrics` flag only affects local deployments. When using Ansible deployment mode, this flag is ignored. Metrics ports are always exposed by clients regardless of this flag.
8485
86+
### Leanpoint upstreams sync (tooling server)
87+
88+
After all validator nodes are spun up (local or Ansible), the quickstart can sync leanpoint upstreams to a tooling server so [leanpoint](https://github.com/leanEthereum/leanpoint) monitors the current set of nodes. This runs automatically at the end of `spin-node.sh` unless disabled.
89+
90+
**What runs:**
91+
1. `convert-validator-config.py` reads `validator-config.yaml` and generates `upstreams.json` (validator URLs for health checks).
92+
2. `sync-leanpoint-upstreams.sh` rsyncs `upstreams.json` to the tooling server and restarts the leanpoint Docker container.
93+
94+
**Defaults:** Tooling server `46.225.10.32`, user `root`, remote path `/opt/leanpoint/upstreams.json`, container name `leanpoint`. Override with env vars (see script header in `sync-leanpoint-upstreams.sh`).
95+
96+
**Disable sync:** Set `LEANPOINT_SYNC_DISABLED=1` before running `spin-node.sh`, or when the convert script or validator config is missing the sync is skipped without failing the run.
97+
98+
**Standalone use of convert script:** You can generate `upstreams.json` for local leanpoint without the tooling server:
99+
100+
```sh
101+
# From lean-quickstart root
102+
python3 convert-validator-config.py local-devnet/genesis/validator-config.yaml upstreams.json
103+
# With --docker for leanpoint in Docker reaching a host devnet:
104+
python3 convert-validator-config.py local-devnet/genesis/validator-config.yaml upstreams-local-docker.json --docker
105+
```
106+
107+
Requires Python 3 and PyYAML (`pip install pyyaml`).
108+
85109
## Args
86110

87111
1. `NETWORK_DIR` is an env to specify the network directory. Should have a `genesis` directory with genesis config. A `data` folder will be created inside this `NETWORK_DIR` if not already there.
@@ -536,6 +560,7 @@ NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --forceKeyG
536560
This quickstart includes automated configuration parsing:
537561

538562
- **Official Genesis Generation**: Uses PK's `eth-beacon-genesis` docker tool from [PR #36](https://github.com/ethpandaops/eth-beacon-genesis/pull/36)
563+
- **Leanpoint upstreams sync**: After nodes are spun up, `convert-validator-config.py` and `sync-leanpoint-upstreams.sh` generate `upstreams.json` from `validator-config.yaml`, rsync it to the tooling server, and restart the leanpoint container (see [Leanpoint upstreams sync](#leanpoint-upstreams-sync-tooling-server))
539564
- **Complete File Set**: Generates `validators.yaml`, `nodes.yaml`, `genesis.json`, `genesis.ssz`, and `.key` files
540565
- **QUIC Port Detection**: Automatically extracts QUIC ports from `validator-config.yaml` using `yq`
541566
- **Node Detection**: Dynamically discovers available nodes from the validator configuration

‎convert-validator-config.py‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
#!/usr/bin/env python3
2+
"""
3+
Convert validator-config.yaml to upstreams.json for leanpoint.
4+
5+
This script reads a validator-config.yaml file (used by lean-quickstart)
6+
and generates an upstreams.json file that leanpoint can use to monitor
7+
multiple lean nodes.
8+
9+
Usage:
10+
python3 convert-validator-config.py [validator-config.yaml] [output.json] [--docker]
11+
12+
Options:
13+
--docker Use host.docker.internal so leanpoint running in Docker can
14+
reach a devnet on the host (e.g. upstreams-local-docker.json).
15+
16+
Examples:
17+
python3 convert-validator-config.py \\
18+
local-devnet/genesis/validator-config.yaml \\
19+
upstreams.json
20+
21+
python3 convert-validator-config.py \\
22+
ansible-devnet/genesis/validator-config.yaml \\
23+
upstreams.json
24+
25+
python3 convert-validator-config.py \\
26+
local-devnet/genesis/validator-config.yaml \\
27+
upstreams-local-docker.json --docker
28+
"""
29+
30+
import sys
31+
import json
32+
import yaml
33+
34+
35+
def convert_validator_config(
36+
yaml_path: str,
37+
output_path: str,
38+
base_port: int = 8081,
39+
docker_host: bool = False,
40+
):
41+
"""
42+
Convert validator-config.yaml to upstreams.json.
43+
44+
Args:
45+
yaml_path: Path to validator-config.yaml
46+
output_path: Path to output upstreams.json
47+
base_port: Base HTTP port for beacon API (default: 8081)
48+
docker_host: If True, use host.docker.internal so leanpoint in Docker
49+
can reach a devnet running on the host (Docker Desktop/Orbstack).
50+
"""
51+
with open(yaml_path, 'r') as f:
52+
config = yaml.safe_load(f)
53+
54+
if 'validators' not in config:
55+
print("Error: No 'validators' key found in config", file=sys.stderr)
56+
sys.exit(1)
57+
58+
upstreams = []
59+
60+
for idx, validator in enumerate(config['validators']):
61+
name = validator.get('name', f'validator_{idx}')
62+
63+
# Try to get IP from enrFields, default to localhost
64+
ip = "127.0.0.1"
65+
if 'enrFields' in validator and 'ip' in validator['enrFields']:
66+
ip = validator['enrFields']['ip']
67+
if docker_host:
68+
ip = "host.docker.internal"
69+
70+
# Use metricsPort from config when present (validator-config uses it for API)
71+
http_port = validator.get('metricsPort', base_port + idx)
72+
73+
upstream = {
74+
"name": name,
75+
"url": f"http://{ip}:{http_port}",
76+
"path": "/v0/health" # Health check endpoint
77+
}
78+
79+
upstreams.append(upstream)
80+
81+
output = {"upstreams": upstreams}
82+
83+
with open(output_path, 'w') as f:
84+
json.dump(output, f, indent=2)
85+
86+
print(f"✅ Converted {len(upstreams)} validators to {output_path}")
87+
print(f"\nGenerated upstreams:")
88+
for u in upstreams:
89+
print(f" - {u['name']}: {u['url']}{u['path']}")
90+
91+
print(f"\n💡 To use: leanpoint --upstreams-config {output_path}")
92+
93+
94+
def main():
95+
args = [a for a in sys.argv[1:] if a != "--docker"]
96+
docker_host = "--docker" in sys.argv
97+
98+
if len(args) < 2:
99+
if len(args) == 0:
100+
print(__doc__)
101+
print("\nUsing default paths...")
102+
yaml_path = "local-devnet/genesis/validator-config.yaml"
103+
output_path = "upstreams.json"
104+
else:
105+
yaml_path = args[0]
106+
output_path = "upstreams-local-docker.json" if docker_host else "upstreams.json"
107+
else:
108+
yaml_path = args[0]
109+
output_path = args[1]
110+
111+
try:
112+
convert_validator_config(yaml_path, output_path, docker_host=docker_host)
113+
except FileNotFoundError as e:
114+
print(f"Error: File not found: {e}", file=sys.stderr)
115+
sys.exit(1)
116+
except yaml.YAMLError as e:
117+
print(f"Error: Invalid YAML: {e}", file=sys.stderr)
118+
sys.exit(1)
119+
except Exception as e:
120+
print(f"Error: {e}", file=sys.stderr)
121+
sys.exit(1)
122+
123+
124+
if __name__ == "__main__":
125+
main()

‎spin-node.sh‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,10 @@ if [ "$deployment_mode" == "ansible" ]; then
214214
echo "❌ Ansible deployment failed. Exiting."
215215
exit 1
216216
fi
217-
217+
218+
# Sync leanpoint upstreams to tooling server and restart leanpoint container
219+
"$scriptDir/sync-leanpoint-upstreams.sh" "$validator_config_file" "$scriptDir" "$sshKeyFile" "$useRoot" || true
220+
218221
# Ansible deployment succeeded, exit normally
219222
exit 0
220223
fi
@@ -423,6 +426,9 @@ if [ -n "$enableMetrics" ] && [ "$enableMetrics" == "true" ]; then
423426
echo ""
424427
fi
425428

429+
# Sync leanpoint upstreams to tooling server and restart leanpoint container
430+
"$scriptDir/sync-leanpoint-upstreams.sh" "$validator_config_file" "$scriptDir" "$sshKeyFile" "$useRoot" || true
431+
426432
container_names="${spin_nodes[*]}"
427433
process_ids="${spinned_pids[*]}"
428434

‎sync-leanpoint-upstreams.sh‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/bin/bash
2+
# sync-leanpoint-upstreams.sh: Regenerate upstreams.json from validator-config.yaml,
3+
# rsync it to the tooling server, and restart the leanpoint container.
4+
#
5+
# Used after validator nodes are spun up so leanpoint monitors the current set
6+
# of nodes. Called at the end of spin-node.sh (both Ansible and local deployment).
7+
#
8+
# Usage:
9+
# sync-leanpoint-upstreams.sh <validator_config_file> <script_dir> [ssh_key_file] [use_root]
10+
#
11+
# Env (optional):
12+
# TOOLING_SERVER Tooling server host (default: 46.225.10.32)
13+
# TOOLING_SERVER_USER SSH user on tooling server (default: root)
14+
# LEANPOINT_DIR Path containing convert-validator-config.py (default: script_dir)
15+
# REMOTE_UPSTREAMS_PATH Remote path for upstreams.json (default: /opt/leanpoint/upstreams.json)
16+
# LEANPOINT_CONTAINER Docker container name on tooling server (default: leanpoint)
17+
# LEANPOINT_SYNC_DISABLED Set to 1 to skip sync (e.g. when tooling server is not used)
18+
19+
set -e
20+
21+
validator_config_file="${1:?Usage: sync-leanpoint-upstreams.sh <validator_config_file> <script_dir> [ssh_key_file] [use_root]}"
22+
scriptDir="${2:?Usage: sync-leanpoint-upstreams.sh <validator_config_file> <script_dir> [ssh_key_file] [use_root]}"
23+
sshKeyFile="${3:-}"
24+
useRoot="${4:-false}"
25+
26+
TOOLING_SERVER="${TOOLING_SERVER:-46.225.10.32}"
27+
TOOLING_SERVER_USER="${TOOLING_SERVER_USER:-root}"
28+
LEANPOINT_DIR="${LEANPOINT_DIR:-$scriptDir}"
29+
REMOTE_UPSTREAMS_PATH="${REMOTE_UPSTREAMS_PATH:-/opt/leanpoint/upstreams.json}"
30+
LEANPOINT_CONTAINER="${LEANPOINT_CONTAINER:-leanpoint}"
31+
32+
if [ "${LEANPOINT_SYNC_DISABLED:-0}" = "1" ]; then
33+
echo "Leanpoint sync disabled (LEANPOINT_SYNC_DISABLED=1), skipping."
34+
exit 0
35+
fi
36+
37+
convert_script="$LEANPOINT_DIR/convert-validator-config.py"
38+
if [ ! -f "$convert_script" ]; then
39+
echo "Warning: convert-validator-config.py not found at $convert_script, skipping leanpoint sync."
40+
exit 0
41+
fi
42+
43+
if [ ! -f "$validator_config_file" ]; then
44+
echo "Warning: validator config not found at $validator_config_file, skipping leanpoint sync."
45+
exit 0
46+
fi
47+
48+
# Build SSH/rsync target and optional key args
49+
remote_target="${TOOLING_SERVER_USER}@${TOOLING_SERVER}"
50+
ssh_cmd="ssh -o StrictHostKeyChecking=no"
51+
if [ -n "$sshKeyFile" ]; then
52+
key_path="$sshKeyFile"
53+
[[ "$key_path" == ~* ]] && key_path="${key_path/#\~/$HOME}"
54+
if [ -f "$key_path" ]; then
55+
ssh_cmd="ssh -i $key_path -o StrictHostKeyChecking=no"
56+
fi
57+
fi
58+
59+
# Generate upstreams.json (no --docker: use real validator IPs for remote leanpoint)
60+
out_file=$(mktemp)
61+
trap "rm -f $out_file" EXIT
62+
python3 "$convert_script" "$validator_config_file" "$out_file" || {
63+
echo "Warning: convert-validator-config.py failed, skipping leanpoint sync."
64+
exit 0
65+
}
66+
67+
# Ensure remote directory exists, then rsync
68+
remote_dir=$(dirname "$REMOTE_UPSTREAMS_PATH")
69+
$ssh_cmd "$remote_target" "mkdir -p $remote_dir"
70+
rsync -e "$ssh_cmd" "$out_file" "${remote_target}:${REMOTE_UPSTREAMS_PATH}"
71+
72+
# Restart leanpoint container on tooling server
73+
$ssh_cmd "$remote_target" "docker restart $LEANPOINT_CONTAINER"
74+
75+
echo "Leanpoint upstreams synced to $TOOLING_SERVER and container '$LEANPOINT_CONTAINER' restarted."

0 commit comments

Comments
 (0)