Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ext/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ go_library(
"//common/types/traits:go_default_library",
"//interpreter:go_default_library",
"//parser:go_default_library",
"@in_yaml_go_yaml_v3//:go_default_library",
"@org_golang_google_protobuf//encoding/protojson:go_default_library",
"@org_golang_google_protobuf//proto:go_default_library",
"@org_golang_google_protobuf//reflect/protoreflect:go_default_library",
Expand Down Expand Up @@ -82,6 +83,7 @@ go_test(
"//test:go_default_library",
"//test/proto2pb:go_default_library",
"//test/proto3pb:go_default_library",
"@in_yaml_go_yaml_v3//:go_default_library",
"@org_golang_google_protobuf//encoding/protojson:go_default_library",
"@org_golang_google_protobuf//proto:go_default_library",
"@org_golang_google_protobuf//types/known/wrapperspb:go_default_library",
Expand Down
45 changes: 45 additions & 0 deletions ext/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ Local bindings are not guaranteed to be evaluated before use.

Encoding utilities for marshalling data into standardized representations.

Note: Version 2 of this library depends on the CEL optional type. Please ensure that
cel.OptionalTypes() is enabled when using encoder extensions at version 2 or greater.

### Base64.Decode

**Introduced in version 0 (cost support in version 1)**
Expand Down Expand Up @@ -104,6 +107,48 @@ Examples:
json.encode([1, 'two', true]) // return '[1,"two",true]'
json.encode({'items': [1, 'two', false]}) // return '{"items":[1,"two",false]}'

### JSON.Parse

Introduced at version: 2

Parses a JSON string to a CEL value or a specific type.

json.parse(<string>) -> <optional_type(dyn)>
json.parse(<string>, <type(T)>) -> <optional_type(T)>

Examples:

json.parse('{"hello":"world"}') // return optional.of({'hello': 'world'})
json.parse('123', int) // return optional.of(123)

### YAML.Encode

Introduced at version: 2

Encodes a CEL value to a YAML string.

yaml.encode(<dyn>) -> <string>

Examples:

yaml.encode('hello') // return "hello\n"
yaml.encode([1, 'two', true]) // return "- 1\n- two\n- true\n"
yaml.encode({'items': [1, 'two', false]}) // return "items:\n - 1\n - two\n - false\n"

### YAML.Parse

Introduced at version: 2

Parses a YAML string to a CEL value or a specific type.

yaml.parse(<string>) -> <optional_type(dyn)>
yaml.parse(<string>, <type(T)>) -> <optional_type(T)>

Examples:

yaml.parse('hello: world') // return optional.of({'hello': 'world'})
yaml.parse('123', int) // return optional.of(123)

## Math

Math helper macros and functions.
Expand Down
39 changes: 31 additions & 8 deletions ext/design/encoders.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@

## 1. Overview

The `encoders` extension library provides standard encoding, decoding, and parsing functions for Common Expression Language (CEL). It allows expressions to safely interact with serialized data representations like Base64 and JSON without compromising execution safety, determinism, or resource boundaries.
The `encoders` extension library provides standard encoding, decoding, and parsing functions for Common Expression Language (CEL). It allows expressions to safely interact with serialized data representations like Base64, JSON, and YAML without compromising execution safety, determinism, or resource boundaries.

The library defines functions under two primary namespaces:
The library defines functions under three primary namespaces:
- `base64`: For standard Base64 binary-to-text encoding and decoding.
- `json`: For serializing CEL values to JSON and parsing JSON payloads into dynamic or strongly-typed CEL representations.
- `yaml`: For serializing CEL values to YAML and parsing YAML payloads into dynamic or strongly-typed CEL representations.

---

Expand Down Expand Up @@ -73,27 +74,50 @@ json.parse(string, type(T)) -> optional_type(T)
- `json:",omitempty"`: omits zero-value fields during serialization.
- Returns `optional.none()` if the JSON value does not match the expected target schema or type.

### 3.3. YAML Functions

```text
yaml.encode(dyn) -> string
yaml.parse(string) -> optional_type(dyn)
yaml.parse(string, type(T)) -> optional_type(T)
```

#### `yaml.encode(val)`
- Converts any CEL value into a YAML string, building directly on top of JSON serialization:
- First converts CEL `val` to JSON via `jsonEncodeValue(val)` (handling primitives, lists, maps, protobuf messages, native Go structs with `json` struct tags).
- Unmarshals the normalized intermediate representation and marshals it into formatted YAML via `go.yaml.in/yaml/v3`.
- Produces valid YAML ending with a newline.

#### `yaml.parse(string)` & `yaml.parse(string, type(T))`
- Decodes YAML into a sanitized intermediate JSON structure, then leverages `json.parse` / `jsonParseWithType` engine:
- Validates a single root YAML document (multiple documents return `optional.none()`).
- Supports all dynamic and strongly-typed schema targets (primitives, lists, maps, timestamps, durations, protobuf messages, Go structs).
- Returns `optional.none()` on malformed YAML syntax or schema mismatches.

---

## 4. Security & Resource Bounds

### 4.1. Payload Size Limit
To prevent memory exhaustion attacks (e.g., billion-laughs-style JSON bombs, deeply nested objects, or multi-gigabyte payloads), `json.parse` enforces a strict 10MB maximum input size:
To prevent memory exhaustion attacks (e.g., billion-laughs-style bombs, deeply nested structures, or multi-gigabyte payloads), `json.parse` and `yaml.parse` enforce a strict 10MB maximum input size:
```go
const maxJSONSize = 10 * 1024 * 1024 // 10MB
const (
maxJSONSize = 10 * 1024 * 1024 // 10MB
maxYAMLSize = 10 * 1024 * 1024 // 10MB
)
```
Inputs exceeding this limit immediately return a CEL runtime error.

### 4.2. Cost Modeling
JSON parsing and serialization complexity cannot be statically bounded without inspecting runtime payloads and target schema depths. Therefore:
JSON and YAML parsing and serialization complexity cannot be statically bounded without inspecting runtime payloads and target schema depths. Therefore:
- **Cost Estimation**: Overload cost estimators return `checker.UnknownCostEstimate()` (`CostEstimate{Min: 0, Max: math.MaxUint64}`).
- **Cost Tracking**: Runtime cost trackers return `math.MaxUint64`.

---

### 5.1. Instantiation via `Provider.NewValue`

`json.parse` utilizes `types.Provider.NewValue(typeName, map[string]ref.Val{})` to instantiate target types:
`json.parse` and `yaml.parse` utilize `types.Provider.NewValue(typeName, map[string]ref.Val{})` to instantiate target types:
1. **Protobuf Messages**: `NewValue` returns a proto value whose underlying `proto.Message` is cloned and populated using `protojson.Unmarshal`.
2. **Native Go Objects**: `NewValue` returns a native struct value whose `reflect.Type` is used to instantiate a pointer (`reflect.New(rt)`) and populated via standard `json.Unmarshal`.
3. **No Private Registry Exposure**: Avoids leaking internal registry structures or requiring custom reflection getters on `Registry`.
Expand All @@ -102,8 +126,7 @@ JSON parsing and serialization complexity cannot be statically bounded without i

To maintain a clean separation of concerns:
- **`common/types/native.go` & `ext/native.go`**: Contain all Go struct reflection, struct tag inspection (`json:"..."`, `cel:"..."`, `omitempty`, `json:"-"`), anonymous embedded struct promotion, and zero-value omission rules. `nativeObj.ConvertToNative(types.JSONValueType)` / `nativeObj.ConvertToNative(types.JSONStructType)` transforms native objects into standard structured representations.
- **`ext/encoders.go`**: Acts as a format-level encoder/decoder. It performs payload size verification, invokes `val.ConvertToNative(types.JSONValueType)`, and calls the format-specific marshal/unmarshal engine (`protojson`).
- **Future Format Support (YAML, XML)**: Adding support for formats such as YAML or XML will follow the exact same architecture: format encoders in `ext/` remain lightweight wrappers over `ConvertToNative` / `NewValue`, while native type reflection and tag mappings reside centrally in `common/types/native.go` and `ext/native.go`.
- **`ext/encoders.go`**: Acts as a format-level encoder/decoder. It performs payload size verification, invokes `val.ConvertToNative(types.JSONValueType)`, and delegates to format engines (`protojson`, `yaml/v3`). YAML support builds seamlessly on top of JSON transformation.

---

Expand Down
Loading
Loading