Skip to content

redhat-performance/chronicler

BranchesTags

Repository files navigation

Chronicler

Export Zathras benchmark results to OpenSearch for centralized analysis, dashboards, and performance tracking. Horreum export is available as a stub (not implemented) for future use.

Quick start: For a local Python workflow, follow Installation & Setup first (venv, install, verify, optional tests, then Configuration). Then use How to run for CLI examples. Prefer Running with a container? Skip pip there; you still supply config and volume mounts. Deeper topics (benchmark matrix, CI, OpenSearch queries) come later in the document.

Installation & Setup

Complete this section on your machine before How to run (unless you use only the container image).

Prerequisites

  • Python 3.9+
  • Zathras benchmark results
  • OpenSearch access (optional if you only use --output-json)

Install the package

Use a project virtual environment; do not install into the system Python.

cd /path/to/chronicler
python3 -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate

# OpenSearch export (most common):
pip install ".[opensearch]"

# JSON-only output (no OpenSearch Python client):
pip install .

# Development (editable install, tests, and OpenSearch client):
pip install -e ".[opensearch,dev]"

Verify the install:

python3 -c "import chronicler; print(chronicler.__file__)"

Dependencies and optional extras are defined in pyproject.toml. Prefer the commands above over requirements.txt (kept as a legacy flat list; see comments in that file).

Running tests (with venv activated):

cd /path/to/chronicler
pip install -e ".[opensearch,dev]"   # if not already installed for development
pytest tests/ -v         # Run full suite (209 tests)

# Run by category
pytest -m unit           # Fast unit tests only (178 tests)
pytest -m integration    # Integration tests with file I/O (20 tests)

The test suite covers:

  • Schema: dataclass serialization, validation, content hashing
  • Utilities: timestamp validation, parsing, run helpers
  • Archive handling: zip/tar extraction
  • Processors: timestamp validation, run parsing, metric extraction

Dependencies (see pyproject.toml):

  • pyyaml — configuration and result parsing (always installed)
  • opensearch-py — OpenSearch export (install with .[opensearch])
  • requests — optional Horreum client path (install with .[horreum]; Horreum export is still a stub)

Configuration

Create your configuration file next to the Chronicler Python package (so it is found without passing --config):

PKG_CONFIG=$(python3 -c "from pathlib import Path; import chronicler; print(Path(chronicler.__file__).parent/'config')")
cp "$PKG_CONFIG/export_config_example.yml" "$PKG_CONFIG/export_config.yml"
vim "$PKG_CONFIG/export_config.yml"

Discovery order (when --config is omitted):

  1. CHRONICLER_CONFIG — absolute or relative path to a YAML file (recommended for CI/orchestrators).
  2. Package path — <chronicler>/config/export_config.yml (same directory as export_config_example.yml in the install).
  3. Current working directory, first match: ./export_config.yml → ./config/export_config.yml → ./chronicler/config/export_config.yml.

If no file is found, the CLI logs a warning and runs with an empty config (OpenSearch code defaults only). Passing --config PATH always uses that path and overrides discovery. Each run logs which file is used (or that the explicit path is missing).

Orchestrators (e.g. Zathras) can rely on CHRONICLER_CONFIG or on a pre-installed export_config.yml beside the package, avoiding a hard-coded path to the Chronicler source tree.

Example config:

opensearch:
  url: "https://opensearch.example.com"
  summary_index: "zathras-results"        # Summary documents
  timeseries_index: "zathras-timeseries"  # Individual time series points
  username: "example-user"
  password: "your-password"
  verify_ssl: false  # Set to true for production

# Horreum export is not implemented (stub only)
horreum:
  url: "http://localhost:8080"
  username: "your-horreum-username"
  password: "your-horreum-password"
  test_name: "Zathras Benchmarks"

processing:
  batch_size: 500
  continue_on_error: true
  verbose: false

How to Run

After completing Installation & Setup above (including Install the package and Configuration), run the post-processing CLI on your result directories (same activated venv you used for pip install):

# Process and export (config is discovered automatically; see Configuration section)
python3 -m chronicler.run_postprocessing \
    --input /path/to/results \
    --opensearch

# Or just generate JSON files
python3 -m chronicler.run_postprocessing \
    --input /path/to/results \
    --output-json /tmp/json_output

What it does:

  • Recursively discovers all result directories
  • Auto-detects benchmark types (coremark, streams, pyperf, etc.)
  • Batch processes all results
  • Exports to OpenSearch or saves JSON (Horreum is a stub)
  • Provides detailed summary report

Example output:

2025-11-06 21:00:31 - Searching for results in: production_data
2025-11-06 21:00:31 - Found 34 result directory(ies)
2025-11-06 21:00:34 - Processing coremark: results_coremark.zip
  [SUCCESS] Parsed coremark: coremark_Standard_D128ds_v6_2_20251107_020034
  [EXPORT] Exported to OpenSearch (summary): coremark_Standard_D128ds_v6_2_20251107_020034

======================================================================
PROCESSING SUMMARY
======================================================================
Total: 109 files
Successful: 78 benchmarks
Skipped: 31 (unknown types)
Duration: 109.38 seconds

Tests Processed:
  - coremark: 12
  - streams: 11
  - pyperf: 6 (34,080 time series points)
  - specjbb: 11
  - And more...

Running with a Container

Use this path when you prefer not to install Chronicler with pip locally. The image includes the CLI entrypoint; you still provide a config file and mount your result directories.

A pre-built image is available at quay.io/zathras/chronicler.

Prepare your config

The container expects the config file at /config/config.yaml (set via CHRONICLER_CONFIG). Copy and edit the example:

cp config/export_config_example.yml /path/to/your/config.yaml
vim /path/to/your/config.yaml   # Add your credentials

Run the container

Mount your config file and your benchmark results directory into the container, then pass the same CLI flags as the local invocation:

# Podman — export to OpenSearch
podman run --rm \
    -v /path/to/your/config.yaml:/config/config.yaml:z \
    -v /path/to/results:/results:z \
    quay.io/zathras/chronicler \
    --input /results \
    --opensearch

# Docker — export to OpenSearch
docker run --rm \
    -v /path/to/your/config.yaml:/config/config.yaml \
    -v /path/to/results:/results \
    quay.io/zathras/chronicler \
    --input /results \
    --opensearch

# Write JSON output instead of exporting
podman run --rm \
    -v /path/to/results:/results:z \
    -v /tmp/json_output:/output:z \
    quay.io/zathras/chronicler \
    --input /results \
    --output-json /output

Note: The :z flag on volume mounts is required by SELinux on Fedora/RHEL hosts. Omit it on other platforms.

Override the config path at runtime

If you prefer to place the config file elsewhere, override the environment variable:

podman run --rm \
    -e CHRONICLER_CONFIG=/data/my_config.yaml \
    -v /path/to/your/config.yaml:/data/my_config.yaml:z \
    -v /path/to/results:/results:z \
    quay.io/zathras/chronicler \
    --input /results \
    --opensearch

Detailed Usage

Benchmark Support Status

Benchmark Post-Processing Support Processor Notes
CoreMark Supported coremark_processor.py Single-thread CPU performance
CoreMark Pro Supported coremark_pro_processor.py 9 workload types
FIO Supported fio_processor.py Flexible I/O tester
HPL (autohpl) Supported autohpl_processor.py High Performance Computing Linpack
Passmark Supported passmark_processor.py CPU & Memory marks
Phoronix Test Suite Supported phoronix_processor.py 51 sub-tests (BOPs)
Pig Supported pig_processor.py Processor scheduler efficiency
PyPerf Supported pyperf_processor.py 90 Python benchmarks, 5,680 time series points
SPEC CPU 2017 Supported speccpu2017_processor.py Compute-intensive performance suite
SpecJBB Supported specjbb_processor.py Java business benchmark (Critical/Max-jOPS)
STREAM Supported streams_processor.py Memory bandwidth (Copy, Scale, Add, Triad)
Uperf Supported uperf_processor.py Network performance (IOPS, latency, throughput)
HammerDB Not Supported - Database benchmarking (MariaDB, PostgreSQL)
IOzone Not Supported - File system benchmarking
Linpack Not Supported - Licensed Linpack benchmark
NUMA STREAM Not Supported - NUMA memory bandwidth extension

12 of 16 benchmarks supported | 35,000+ time series points per production run

CI/CD Integration with Burden

Automatic Export After Every Benchmark Run

Burden has built-in Chronicler integration. No script modifications needed.

Setup:

  1. Install Chronicler with OpenSearch support:

    pip install 'chronicler[opensearch]'

  2. Configure export credentials (choose one method):

    # Option 1: Environment variable (recommended for CI/orchestrators)
export CHRONICLER_CONFIG=/path/to/export_config.yml

# Option 2: Package config directory (standard location)
PKG_CONFIG=$(python3 -c "from pathlib import Path; import chronicler; print(Path(chronicler.__file__).parent/'config')")
cp "$PKG_CONFIG/export_config_example.yml" "$PKG_CONFIG/export_config.yml"
vim "$PKG_CONFIG/export_config.yml"   # Add your credentials

# Option 3: Zathras config directory
# Place export_config.yml in the Zathras top-level config/ directory
cd /path/to/zathras
cp /path/to/chronicler/config/export_config_example.yml config/export_config.yml
vim config/export_config.yml   # Add your credentials

    Config is discovered automatically in this order: CHRONICLER_CONFIG environment variable, package export_config.yml, Zathras config/export_config.yml, or legacy ../chronicler/config/export_config.yml.

Usage:

# Standard mode (continues on Chronicler errors):
burden --run_chronicler [other burden options]

# Strict mode (aborts burden if Chronicler fails):
burden --run_chronicler_strict [other burden options]

# Alternatively, use environment variable with --run_chronicler:
CHRONICLER_STRICT=1 burden --run_chronicler [other burden options]

# Verbose Chronicler output:
CHRONICLER_VERBOSE=1 burden --run_chronicler [other burden options]
# OR
burden --run_chronicler --ansible_noise_level dense [other burden options]

Benefits:

  • Zero manual script modification
  • Results exported automatically after test completion
  • Graceful failure handling with --run_chronicler or strict abort with --run_chronicler_strict
  • Automatic config discovery from multiple locations
  • Works with both archived and non-archived results
  • Verbose logging available for troubleshooting

View Your Results

OpenSearch Dashboards

Access your exported data:

Quick Queries

# Get latest results
GET /zathras-results/_search
{
  "size": 10,
  "sort": [{ "metadata.test_timestamp": "desc" }]
}

# Find CoreMark results
GET /zathras-results/_search
{
  "query": { "term": { "test.name": "coremark" }}
}

# Performance over time (summary data)
GET /zathras-results/_search
{
  "query": { "term": { "test.name": "coremark" }},
  "sort": [{ "metadata.test_timestamp": "asc" }],
  "_source": ["metadata.test_timestamp", "results.runs.run_0.mean"]
}

# Get time series data points
GET /zathras-timeseries/_search
{
  "query": { 
    "bool": {
      "must": [
        { "term": { "test.name": "pyperf" }},
        { "term": { "results.run.benchmark_name": "2to3" }}
      ]
    }
  },
  "sort": [{ "metadata.sequence": "asc" }]
}

# Compare by CPU architecture
GET /zathras-results/_search
{
  "size": 0,
  "aggs": {
    "by_arch": {
      "terms": { "field": "system_under_test.hardware.cpu.architecture.keyword" },
      "aggs": {
        "avg_performance": { "avg": { "field": "results.runs.run_0.mean" }}
      }
    }
  }
}

What Gets Extracted

The post-processing pipeline automatically extracts and structures data from Zathras result files:

Benchmark Results (results_*.zip)

  • Performance metrics: Mean, median, min, max, standard deviation for each run
  • Time series data: Individual data points with sequence ordering (for benchmarks like PyPerf)
  • Configuration details: Test-specific settings and parameters
  • Execution metadata: Start time, duration, status (PASS/FAIL)
  • Validation data: Checksums, compiler information

System Configuration (sysconfig_info.tar)

Hardware:

  • CPU: Vendor, model, architecture, cores, threads, cache sizes, CPU flags (as boolean objects)
  • Memory: Total capacity, speed, NUMA topology (node-based objects)
  • Storage: Devices, capacity, types, mount points
  • Network: Interfaces, speeds, addresses

Operating System:

  • Distribution, version, kernel version
  • Tuned profile settings
  • Sysctl parameters
  • SELinux configuration

Test Configuration (ansible_vars.yml)

  • Test parameters and iteration counts
  • System tuning settings applied during test
  • Zathras scenario information

Directory Metadata (from path structure)

Automatically parsed from result directory paths:

  • OS Vendor: e.g., rhel, ubuntu, fedora
  • Cloud Provider: e.g., azure, aws, gcp, local
  • Instance Type: e.g., Standard_D8ds_v6, m5.xlarge
  • Iteration Number: e.g., 0, 1, 2
  • Scenario Name: e.g., az_rhel_10_ga

Example directory structure: production_data/az_rhel_10_ga/rhel/azure/Standard_D8ds_v6_1/

Export to Different Targets

OpenSearch (Two-Index Architecture)

Zathras uses two OpenSearch indices to handle high-volume time series data:

  1. zathras-results - Summary documents with aggregated statistics (mean, median, stdev, etc.)
  2. zathras-timeseries - Individual time series data points (one document per point)

Why two indices? Benchmarks like PyPerf generate 5,680+ time series points per test, exceeding OpenSearch's 5,000 field limit for a single document. The two-index approach keeps summaries queryable while preserving full time series data.

python3 -m chronicler.run_postprocessing \
    --input /path/to/results \
    --opensearch
# Optional: --config /path/to/export_config.yml overrides discovery

Horreum (stub)

Horreum export is not implemented. The HorreumExporter class is a stub: it accepts the same constructor and method calls for compatibility, but export_zathras_document (and other methods) raise NotImplementedError. Implement chronicler/exporters/horreum_exporter.py to enable Horreum.

from chronicler.exporters.horreum_exporter import HorreumExporter

# Stub accepts url, username, password (and **kwargs for future use)
exporter = HorreumExporter(
    url="https://horreum.example.com",
    username="user",
    password="pass"
)
# exporter.export_zathras_document(document)  # Raises NotImplementedError

Duplicate Prevention

Zathras uses content-based checksums to prevent duplicate documents when reprocessing the same test results.

How It Works

Each document is assigned a deterministic ID based on a SHA256 hash of its content:

Document ID = {test_name}_{content_hash[:16]}
Example: coremark_fdcfbbf0e6a525ea

The hash includes:

  • Test name, version, and configuration
  • System details (CPU, memory, OS)
  • All benchmark results and metrics
  • Original test timestamp

The hash excludes processing_timestamp, ensuring identical test results always generate the same ID.

Benefits

Prevents Duplicates:

# Process results
python3 -m chronicler.run_postprocessing --input results/ --opensearch

# Reprocess same results (e.g., after fixing a bug)
python3 -m chronicler.run_postprocessing --input results/ --opensearch

# Result: Same document updated in OpenSearch, no duplicate created

Safe Reprocessing:

  • Fix processor bugs without creating duplicates
  • Add new fields to existing documents
  • Update metadata or extractors
  • Track last processing time via processing_timestamp

OpenSearch Behavior:

# First upload
POST /zathras-results/_doc/coremark_fdcfbbf0e6a525ea
→ Creates new document

# Second upload (same test results, different processing time)
POST /zathras-results/_doc/coremark_fdcfbbf0e6a525ea
→ Updates existing document (no duplicate)

When Duplicates WILL Occur

Duplicates only happen when actual test data differs:

  • Different test runs (different timestamps/results)
  • Different systems (different CPU/memory)
  • Different configurations (different parameters)

This is expected behavior representing genuinely different tests.

Testing

Deduplication is built-in and automatic. To verify it's working, process the same results twice and query OpenSearch - you'll see only one document exists with an updated processing_timestamp.

Schema & Data Structure

Two-Index Architecture

Zathras uses two separate OpenSearch indices to efficiently handle benchmarks with large time series datasets:

1. zathras-results - Summary Documents

Contains aggregated statistics for each benchmark execution:

{
  "metadata": {
    "document_id": "coremark_Standard_D8ds_v6_1_20251106",
    "test_timestamp": "2025-11-06T12:00:00Z",
    "processing_timestamp": "2025-11-06T12:05:00Z",
    "os_vendor": "rhel",
    "cloud_provider": "azure",
    "instance_type": "Standard_D8ds_v6",
    "iteration": 1,
    "scenario_name": "az_rhel_10_ga"
  },
  "test": {
    "name": "coremark",
    "version": "1.0"
  },
  "system_under_test": {
    "hardware": {
      "cpu": {
        "vendor": "Intel",
        "model": "Xeon Platinum 8370C",
        "architecture": "x86_64",
        "cores": 8,
        "threads": 16,
        "flags": { "avx2": true, "avx512": true, "sse4_2": true }
      },
      "memory": {
        "total_gb": 32,
        "speed_mhz": 3200
      }
    },
    "os": {
      "vendor": "rhel",
      "version": "10.0",
      "kernel": "6.11.0-0.rc5.20240828git6a0e38f.45.el10.x86_64"
    }
  },
  "results": {
    "runs": {
      "run_0": {
        "status": "PASS",
        "mean": 193245.2,
        "median": 193500.0,
        "stdev": 1234.5,
        "min": 191000.0,
        "max": 195000.0
      }
    }
  }
}

2. zathras-timeseries - Individual Time Series Points

Stores each time series data point as a separate, fully denormalized document:

{
  "metadata": {
    "document_id": "pyperf_Standard_D8ds_v6_1_20251106",
    "timeseries_id": "pyperf_Standard_D8ds_v6_1_20251106_run0_2to3_seq0",
    "timestamp": "2025-11-06T12:00:00Z",
    "sequence": 0,
    "test_timestamp": "2025-11-06T12:00:00Z",
    "processing_timestamp": "2025-11-06T12:05:00Z",
    "os_vendor": "rhel",
    "cloud_provider": "azure",
    "instance_type": "Standard_D8ds_v6",
    "iteration": 1
  },
  "test": {
    "name": "pyperf",
    "version": "1.0"
  },
  "system_under_test": {
    /* Full SUT details included */
  },
  "results": {
    "run": {
      "run_key": "run_0",
      "run_number": 0,
      "status": "PASS",
      "benchmark_name": "2to3"
    },
    "value": 1.23,
    "unit": "seconds"
  }
}

Design Principles

Fully Denormalized

  • Each document contains complete context (test, SUT, configuration)
  • No joins required for querying
  • Optimized for document-oriented datastores

Object-Based Structure

  • Dynamic keys like run_0, node_0, device_0 instead of arrays
  • Better OpenSearch performance for aggregations
  • Avoids nested object limitations

Hierarchical Metadata

  • Structured extraction from directory paths
  • Consistent field naming across indices
  • Enables filtering by cloud provider, instance type, iteration

Dual Timestamps

  • test_timestamp: When the benchmark was executed
  • processing_timestamp: When the JSON document was created
  • Enables tracking of both test execution and data pipeline timing

Boolean CPU Flags

  • CPU features represented as objects: {"avx2": true, "avx512": false}
  • Efficient term queries for hardware capability filtering

Advanced Queries

Performance Regression Detection

GET /zathras-results/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "test.name": "coremark" }},
        { "range": { "metadata.test_timestamp": { "gte": "now-7d" }}}
      ]
    }
  },
  "sort": [{ "metadata.test_timestamp": "asc" }],
  "_source": ["metadata.test_timestamp", "results.runs.run_0.mean", "metadata.instance_type"]
}

Hardware Comparison

GET /zathras-results/_search
{
  "query": { "term": { "test.name": "coremark" }},
  "aggs": {
    "by_cpu_model": {
      "terms": { "field": "system_under_test.hardware.cpu.model.keyword", "size": 10 },
      "aggs": {
        "avg_performance": { "avg": { "field": "results.runs.run_0.mean" }},
        "max_performance": { "max": { "field": "results.runs.run_0.max" }}
      }
    }
  }
}

Find Systems with Specific Features

# Systems with AVX-512 and more than 64 cores
GET /zathras-results/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "system_under_test.hardware.cpu.flags.avx512": true }},
        { "range": { "system_under_test.hardware.cpu.cores": { "gt": 64 }}}
      ]
    }
  },
  "_source": ["system_under_test.hardware.cpu", "metadata.instance_type", "results.runs.run_0.mean"]
}

Time Series Analysis

# Get all time series points for a specific benchmark run
GET /zathras-timeseries/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "metadata.document_id": "pyperf_Standard_D8ds_v6_1_20251107" }},
        { "term": { "results.run.benchmark_name": "2to3" }}
      ]
    }
  },
  "sort": [{ "metadata.sequence": "asc" }],
  "size": 100
}

# Aggregate time series data
GET /zathras-timeseries/_search
{
  "query": { "term": { "test.name": "pyperf" }},
  "aggs": {
    "by_benchmark": {
      "terms": { "field": "results.run.benchmark_name.keyword" },
      "aggs": {
        "avg_value": { "avg": { "field": "results.value" }},
        "min_value": { "min": { "field": "results.value" }},
        "max_value": { "max": { "field": "results.value" }}
      }
    }
  }
}

Additional Resources

Documentation

External Resources

About

Benchmark results processing tool to extract, transform, and export performance data to OpenSearch.

Resources

Readme

License

GPL-3.0 license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages