diff --git a/content/en/docs/operating/setup/installation.md b/content/en/docs/operating/setup/installation.md index c677253..f542521 100644 --- a/content/en/docs/operating/setup/installation.md +++ b/content/en/docs/operating/setup/installation.md @@ -98,6 +98,19 @@ rules: ... ``` +Alternatively you can directly grant more permissions via Helm values: + +```yaml +manager: + rbac: + strict: true + clusterRole: + extraResources: + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch", "update", "patch"] +``` + If you are missing permissions you will see an error status for the respective tenants reflecting ```bash diff --git a/content/en/docs/reference.md b/content/en/docs/reference.md index 6687396..8da1bc1 100644 --- a/content/en/docs/reference.md +++ b/content/en/docs/reference.md @@ -2053,10 +2053,25 @@ Enforcement for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **[registries](#rulestatusspecindexenforceregistriesindex)** | []object | Define registries which are allowed to be used within this tenant
The rules are aggregated, since you can use Regular Expressions the match registry endpoints | false | +| **action** | enum | Declare the action being performed on the enforcement rule:
deny: On match, deny admission request
allow: On match, allowed admission request
audit: On match, audit (post event) of admission request
*Enum*: allow, deny, audit
*Default*: deny
| false | +| **[workloads](#rulestatusspecindexenforceworkloads)** | object | Enforcement for Workloads (Pods) | false | -### RuleStatus.spec[index].enforce.registries[index] +### RuleStatus.spec[index].enforce.workloads + + + +Enforcement for Workloads (Pods) + + +| **Name** | **Type** | **Description** | **Required** | +| :---- | :---- | :----------- | :-------- | +| **qosClasses** | []string | Define Pod QoS classes matched by this enforcement rule.
Supported values are Guaranteed, Burstable and BestEffort. | false | +| **[registries](#rulestatusspecindexenforceworkloadsregistriesindex)** | []object | Define registries which are allowed to be used within this tenant
The rules are aggregated, since you can use Regular Expressions the match registry endpoints | false | +| **targets** | []enum | Define the enforcement targets this rule applies to.
If empty, each webhook applies its own backwards-compatible default.
*Enum*: pod/initcontainers, pod/ephemeralcontainers, pod/containers, pod/volumes
| false | + + +### RuleStatus.spec[index].enforce.workloads.registries[index] @@ -2065,9 +2080,9 @@ Enforcement for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **url** | string | OCI Registry endpoint, is treated as regular expression. | true | +| **exp** | string | Expression used to evaluate regex | false | +| **negate** | boolean | Negate regular Expression
*Default*: false
| false | | **policy** | []string | Allowed PullPolicy for the given registry. Supplying no value allows all policies. | false | -| **validation** | []enum | Requesting Resources
*Enum*: pod/images, pod/volumes
*Default*: [pod/images pod/volumes]
| false | ### RuleStatus.status @@ -2081,7 +2096,8 @@ RuleStatus contains the accumulated rules applying to namespace it's deployed in | :---- | :---- | :----------- | :-------- | | **[conditions](#rulestatusstatusconditionsindex)** | []object | Conditions | true | | **observedGeneration** | integer | ObservedGeneration is the most recent generation the controller has observed.
*Format*: int64
| false | -| **[rule](#rulestatusstatusrule)** | object | Managed Enforcement properties per Namespace (aggregated from rules) | false | +| **[rule](#rulestatusstatusrule)** | object | Deprecated: use Rules.
Rule contains a legacy flattened view and cannot fully represent action-aware rules.
| false | +| **[rules](#rulestatusstatusrulesindex)** | []object | Rules contains the effective namespace rules after tenant rule selection.
Order is preserved from the originating Tenant rules. | false | ### RuleStatus.status.conditions[index] @@ -2105,7 +2121,8 @@ Condition contains details for one aspect of the current state of this API Resou -Managed Enforcement properties per Namespace (aggregated from rules) +Deprecated: use Rules. +Rule contains a legacy flattened view and cannot fully represent action-aware rules. | **Name** | **Type** | **Description** | **Required** | @@ -2122,10 +2139,25 @@ Enforcement for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **[registries](#rulestatusstatusruleenforceregistriesindex)** | []object | Define registries which are allowed to be used within this tenant
The rules are aggregated, since you can use Regular Expressions the match registry endpoints | false | +| **action** | enum | Declare the action being performed on the enforcement rule:
deny: On match, deny admission request
allow: On match, allowed admission request
audit: On match, audit (post event) of admission request
*Enum*: allow, deny, audit
*Default*: deny
| false | +| **[workloads](#rulestatusstatusruleenforceworkloads)** | object | Enforcement for Workloads (Pods) | false | + + +### RuleStatus.status.rule.enforce.workloads + + + +Enforcement for Workloads (Pods) + + +| **Name** | **Type** | **Description** | **Required** | +| :---- | :---- | :----------- | :-------- | +| **qosClasses** | []string | Define Pod QoS classes matched by this enforcement rule.
Supported values are Guaranteed, Burstable and BestEffort. | false | +| **[registries](#rulestatusstatusruleenforceworkloadsregistriesindex)** | []object | Define registries which are allowed to be used within this tenant
The rules are aggregated, since you can use Regular Expressions the match registry endpoints | false | +| **targets** | []enum | Define the enforcement targets this rule applies to.
If empty, each webhook applies its own backwards-compatible default.
*Enum*: pod/initcontainers, pod/ephemeralcontainers, pod/containers, pod/volumes
| false | -### RuleStatus.status.rule.enforce.registries[index] +### RuleStatus.status.rule.enforce.workloads.registries[index] @@ -2134,9 +2166,62 @@ Enforcement for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **url** | string | OCI Registry endpoint, is treated as regular expression. | true | +| **exp** | string | Expression used to evaluate regex | false | +| **negate** | boolean | Negate regular Expression
*Default*: false
| false | +| **policy** | []string | Allowed PullPolicy for the given registry. Supplying no value allows all policies. | false | + + +### RuleStatus.status.rules[index] + + + +For future implementation where users might manage RuleStatus CRs themselves + + +| **Name** | **Type** | **Description** | **Required** | +| :---- | :---- | :----------- | :-------- | +| **[enforce](#rulestatusstatusrulesindexenforce)** | object | Enforcement for given rule | false | + + +### RuleStatus.status.rules[index].enforce + + + +Enforcement for given rule + + +| **Name** | **Type** | **Description** | **Required** | +| :---- | :---- | :----------- | :-------- | +| **action** | enum | Declare the action being performed on the enforcement rule:
deny: On match, deny admission request
allow: On match, allowed admission request
audit: On match, audit (post event) of admission request
*Enum*: allow, deny, audit
*Default*: deny
| false | +| **[workloads](#rulestatusstatusrulesindexenforceworkloads)** | object | Enforcement for Workloads (Pods) | false | + + +### RuleStatus.status.rules[index].enforce.workloads + + + +Enforcement for Workloads (Pods) + + +| **Name** | **Type** | **Description** | **Required** | +| :---- | :---- | :----------- | :-------- | +| **qosClasses** | []string | Define Pod QoS classes matched by this enforcement rule.
Supported values are Guaranteed, Burstable and BestEffort. | false | +| **[registries](#rulestatusstatusrulesindexenforceworkloadsregistriesindex)** | []object | Define registries which are allowed to be used within this tenant
The rules are aggregated, since you can use Regular Expressions the match registry endpoints | false | +| **targets** | []enum | Define the enforcement targets this rule applies to.
If empty, each webhook applies its own backwards-compatible default.
*Enum*: pod/initcontainers, pod/ephemeralcontainers, pod/containers, pod/volumes
| false | + + +### RuleStatus.status.rules[index].enforce.workloads.registries[index] + + + + + + +| **Name** | **Type** | **Description** | **Required** | +| :---- | :---- | :----------- | :-------- | +| **exp** | string | Expression used to evaluate regex | false | +| **negate** | boolean | Negate regular Expression
*Default*: false
| false | | **policy** | []string | Allowed PullPolicy for the given registry. Supplying no value allows all policies. | false | -| **validation** | []enum | Requesting Resources
*Enum*: pod/images, pod/volumes
*Default*: [pod/images pod/volumes]
| false | ## TenantOwner @@ -3449,10 +3534,25 @@ Enforcement for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **[registries](#tenantspecrulesindexenforceregistriesindex)** | []object | Define registries which are allowed to be used within this tenant
The rules are aggregated, since you can use Regular Expressions the match registry endpoints | false | +| **action** | enum | Declare the action being performed on the enforcement rule:
deny: On match, deny admission request
allow: On match, allowed admission request
audit: On match, audit (post event) of admission request
*Enum*: allow, deny, audit
*Default*: deny
| false | +| **[workloads](#tenantspecrulesindexenforceworkloads)** | object | Enforcement for Workloads (Pods) | false | + + +### Tenant.spec.rules[index].enforce.workloads + + + +Enforcement for Workloads (Pods) + + +| **Name** | **Type** | **Description** | **Required** | +| :---- | :---- | :----------- | :-------- | +| **qosClasses** | []string | Define Pod QoS classes matched by this enforcement rule.
Supported values are Guaranteed, Burstable and BestEffort. | false | +| **[registries](#tenantspecrulesindexenforceworkloadsregistriesindex)** | []object | Define registries which are allowed to be used within this tenant
The rules are aggregated, since you can use Regular Expressions the match registry endpoints | false | +| **targets** | []enum | Define the enforcement targets this rule applies to.
If empty, each webhook applies its own backwards-compatible default.
*Enum*: pod/initcontainers, pod/ephemeralcontainers, pod/containers, pod/volumes
| false | -### Tenant.spec.rules[index].enforce.registries[index] +### Tenant.spec.rules[index].enforce.workloads.registries[index] @@ -3461,9 +3561,9 @@ Enforcement for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **url** | string | OCI Registry endpoint, is treated as regular expression. | true | +| **exp** | string | Expression used to evaluate regex | false | +| **negate** | boolean | Negate regular Expression
*Default*: false
| false | | **policy** | []string | Allowed PullPolicy for the given registry. Supplying no value allows all policies. | false | -| **validation** | []enum | Requesting Resources
*Enum*: pod/images, pod/volumes
*Default*: [pod/images pod/volumes]
| false | ### Tenant.spec.rules[index].namespaceSelector @@ -3503,10 +3603,10 @@ Permissions for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **[rules](#tenantspecrulesindexpermissionsrulesindex)** | []object | Define Promotion Rules which distributed additional ClusterRoles across the Tenant
for promoted ServiceAccounts. | false | +| **[promotions](#tenantspecrulesindexpermissionspromotionsindex)** | []object | Define Promotion Rules which distributed additional ClusterRoles across the Tenant
for promoted ServiceAccounts. | false | -### Tenant.spec.rules[index].permissions.rules[index] +### Tenant.spec.rules[index].permissions.promotions[index] @@ -3516,10 +3616,10 @@ Permissions for given rule | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | | **clusterRoles** | []string | ClusterRoles granted to the promoted ServiceAccounts across the Tenant
kubebuilder:validation:Minimum=1 | false | -| **[selector](#tenantspecrulesindexpermissionsrulesindexselector)** | object | Match ServiceAccounts which are promoted which are granted these additional ClusterRoles
across the Tenant | false | +| **[selector](#tenantspecrulesindexpermissionspromotionsindexselector)** | object | Match ServiceAccounts which are promoted which are granted these additional ClusterRoles
across the Tenant | false | -### Tenant.spec.rules[index].permissions.rules[index].selector +### Tenant.spec.rules[index].permissions.promotions[index].selector @@ -3529,11 +3629,11 @@ across the Tenant | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **[matchExpressions](#tenantspecrulesindexpermissionsrulesindexselectormatchexpressionsindex)** | []object | matchExpressions is a list of label selector requirements. The requirements are ANDed. | false | +| **[matchExpressions](#tenantspecrulesindexpermissionspromotionsindexselectormatchexpressionsindex)** | []object | matchExpressions is a list of label selector requirements. The requirements are ANDed. | false | | **matchLabels** | map[string]string | matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
map is equivalent to an element of matchExpressions, whose key field is "key", the
operator is "In", and the values array contains only "value". The requirements are ANDed. | false | -### Tenant.spec.rules[index].permissions.rules[index].selector.matchExpressions[index] +### Tenant.spec.rules[index].permissions.promotions[index].selector.matchExpressions[index] @@ -3832,9 +3932,9 @@ Managed Metadata | **Name** | **Type** | **Description** | **Required** | | :---- | :---- | :----------- | :-------- | -| **url** | string | OCI Registry endpoint, is treated as regular expression. | true | +| **exp** | string | Expression used to evaluate regex | false | +| **negate** | boolean | Negate regular Expression
*Default*: false
| false | | **policy** | []string | Allowed PullPolicy for the given registry. Supplying no value allows all policies. | false | -| **validation** | []enum | Requesting Resources
*Enum*: pod/images, pod/volumes
*Default*: [pod/images pod/volumes]
| false | ### Tenant.status.spaces[index].metadata diff --git a/content/en/docs/resource-management/customquotas/_index.md b/content/en/docs/resource-management/customquotas/_index.md index 8efec3f..0ce2663 100644 --- a/content/en/docs/resource-management/customquotas/_index.md +++ b/content/en/docs/resource-management/customquotas/_index.md @@ -129,6 +129,104 @@ The following constraints apply to the JSONPath: * Values can resolve to array results, which are then summed up. (For example, `.spec.containers[*].resources.limits.cpu` would sum the CPU limits of all containers in a Pod.) * Missing fields are resulting in an error, as it's assumed that if a path requires calculation it should force the targeted sources to define these paths. Meaning if you eg define this JP `.spec.initContainers[*].resources.limits.cpu` on a Pod that has no initContainers, it will error. If you want to only calculate the path if it exists, you can use a [fielselector](#fieldselectors) to only match objects where the path exists, for example with `.spec.initContainers` as fieldSelector. +#### Matching Strategies + +This section describes how JSONPath expressions are evaluated and how their results are interpreted for conditional matching. + +#### Truthy + +When a `fieldSelectors` entry does not contain a top-level `=` or `==`, Capsule treats it as a JSONPath expression. + +The selector matches when the JSONPath result is truthy. + +Truthy evaluation rules: + +* empty result: false +* `false`, case-insensitive: false +* `0`: false +* any other non-empty result: true + +Example: + +```yaml +spec: + sources: + - apiVersion: v1 + kind: PersistentVolumeClaim + op: add + path: .spec.resources.requests.storage + selectors: + - fieldSelectors: + - '.spec.accessModes[?(@=="ReadWriteOnce")]' + - '.status.phase' +``` + +This selector matches only if: + +* `.spec.accessModes[?(@=="ReadWriteOnce")]` returns a non-empty result +* `.status.phase returns a non-empty result` + +For example, this matches a PVC with: + +```yaml +spec: + accessModes: + - ReadWriteOnce +status: + phase: Bound +``` + +#### Equality + +When an entry contains a top-level `=` or `==` (not nested JP expressions), Capsule treats it as an equality comparison. The left side is evaluated as a JSONPath expression. The right side is compared as a **string**. + +```yaml +spec: + sources: + - apiVersion: v1 + kind: Service + op: count + selectors: + - fieldSelectors: + - '.spec.type=ClusterIP' +``` + +The following forms are equivalent: + +```yaml +fieldSelectors: + - '.spec.type=ClusterIP' + - '.spec.type==ClusterIP' + - '.spec.type=="ClusterIP"' + - ".spec.type=='ClusterIP'" +``` + +A `==` inside a JSONPath filter is still treated as part of the JSONPath expression, not as Capsule equals matching. + +For example: + +```yaml +fieldSelectors: + - '.spec.accessModes[?(@=="ReadWriteOnce")]' +``` + +This is interpreted as a truthy JSONPath selector, not as an equals selector. + + +**Use JSONPath filters for arrays:** + +```yaml +fieldSelectors: + - '.spec.accessModes[?(@=="ReadWriteOnce")]' +``` + +**Use equals matching for scalar fields:** + +```yaml +fieldSelectors: + - '.spec.type=ClusterIP' +``` + ### Quota Matches As it's the case with native [ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/#how-resource-quota-works), when a request is made, Capsule evaluates all existing CustomQuotas and GlobalCustomQuotas to determine which ones match the request. Always the smallest quantity of quotas is enforced, meaning that if multiple quotas match a request, the one with the least available capacity will be the one that determines whether the request is allowed or denied. @@ -363,12 +461,7 @@ spec: fieldSelectors are additional per-source filters. Each entry is a JSONPath expression evaluated against the candidate object. -A selector entry matches when its JSONPath result is truthy: - - * empty result, `false` or `0`: false - * any other non-empty result: true - -Given: +[View the available matching semantics](#matching-strategies) for fieldSelectors. ```yaml spec: diff --git a/content/en/docs/tenants/rules.md b/content/en/docs/tenants/rules.md index 6e23863..0dace3b 100644 --- a/content/en/docs/tenants/rules.md +++ b/content/en/docs/tenants/rules.md @@ -22,9 +22,10 @@ spec: rules: # Matches all Namespaces and enforces the rule for all of them - enforce: - registries: - - url: "harbor/v2/customer-registry/.*" - policy: [ "ifNotPresent" ] + workload: + registries: + - exp: "harbor/v2/customer-registry/.*" + policy: [ "ifNotPresent" ] # Select a subset of namespaces (enviornment=prod) to allow further registries - namespaceSelector: @@ -33,9 +34,10 @@ spec: operator: In values: ["prod"] enforce: - registries: - - url: "harbor/v2/prod-registry/.*" - policy: [ "ifNotPresent" ] + workloads: + registries: + - exp: "harbor/v2/prod-registry/.*" + policy: [ "ifNotPresent" ] ``` Note that rules are combined together. In the above example, all namespaces within the `solar` tenant will be enforced to use images from `harbor/v2/customer-registry/*`, while namespaces labeled with `env=prod` will also be allowed to pull images from `harbor/v2/prod-registry/*`. @@ -43,6 +45,7 @@ Note that rules are combined together. In the above example, all namespaces with ## Permissions Declare permission distribution rules for the selected namespaces. + ### Promotions As an administrator, you can define promotion rules . A promotion rule selects ServiceAccounts within a Tenant based on specified conditions and assigns them predefined ClusterRoles. @@ -170,15 +173,67 @@ To revoke the promotion, Alice can just remove the label: kubectl label sa gitops-reconcile -n solar-test projectcapsule.dev/promote- --as alice --as-group projectcapsule.dev ``` +## Enforcement +Namespace rules can enforce admission behavior for selected resources in Tenant namespaces. Each rule block can define an `action` and one or more matchers. For registry enforcement, matchers are evaluated against the full OCI reference string, including registry, path, tag, or digest. +Rules are evaluated in declaration order. If multiple `allow` or `deny` rules match the same reference, the later matching `allow` or `deny` rule has higher precedence. `audit` rules do not deny the request; they emit an event and add a warning to the admission response. +### Action -## Enforcement +Each `enforce` block supports an `action` field: + +| Action | Behavior | +|---|---| +| `allow` | Allows the matching reference. If `policy` is configured, the pull policy must also be one of the configured values. | +| `deny` | Denies the matching reference. A later matching `allow` rule can override it. | +| `audit` | Allows the request, emits a Kubernetes event, and returns an admission warning. | + +If `action` is omitted, it defaults to `deny`. + +This precedence model allows both broad defaults and specific exceptions. For example, you can allow all Harbor images but deny a customer path afterwards: + +```yaml +rules: + - enforce: + action: allow + workloads: + registries: + - exp: "harbor/.*" -### Registries + - enforce: + action: deny + workloads: + registries: + - exp: "harbor/customer/.*" +``` + +In this example, `harbor/nginx:1.14.2` is allowed, while `harbor/customer/app:1.0.0` is denied because the later, more specific deny rule also matches. -Define allowed image registries for `Pods` with rules. Each registry can have specific policies and validation targets. We use Regexp pattern matching for the registry URL, so you can specify patterns like `harbor/v2/customer-registry/*` to match all images from that registry. The matching is done against the full image name, including the path and tag. The ordering is based on the order of declaration, so the first matching rule will be applied. The later the match is found, the higher the precedence. This allows you to build constructs like these: +You can also deny broadly and allow a more specific exception afterwards: + +```yaml +rules: + - enforce: + action: deny + workloads: + registries: + - exp: "harbor/customer/.*" + + - enforce: + action: allow + workloads: + registries: + - exp: "harbor/customer/prod-image/.*" +``` + +In this example, `harbor/customer/test-image/app:1.0.0` is denied, while `harbor/customer/prod-image/app:1.0.0` is allowed. + +#### Audit + +Use `action: audit` to observe workload usage without blocking the request. Audit rules allow the admission request, emit a Kubernetes event, and add a warning to the admission response. + +For registry enforcement: ```yaml --- @@ -190,62 +245,129 @@ spec: ... rules: - enforce: + action: audit + workloads: + targets: + - pod/containers + registries: + - exp: "docker.io/.*" +``` + +Applying a Pod with `docker.io/library/nginx:latest` succeeds, but the API server response contains an admission warning and Capsule emits a related event for the Pod. + +For QoS enforcement: + +```yaml +rules: + - enforce: + action: audit + workloads: + qosClasses: + - Burstable +``` + +Applying a `Burstable` Pod succeeds, but Capsule emits an event and returns an admission warning. + + + +### Workloads + +Enforcement for workloads mainly targets `Pods` and their associated resources. + +Workload enforcement is configured under `spec.rules[].enforce.workloads`. Each rule can define an `action`, optional workload `targets`, and one or more workload matchers such as registry expressions or QoS classes. + +Rules are evaluated in declaration order. If multiple `allow` or `deny` rules match the same request, the **last matching allow or deny rule wins**. `audit` rules do not block the request; they emit a Kubernetes event and add an admission warning. + +Supported actions are: + +| Action | Behavior | +|---|---| +| `allow` | Allows the matching request. If the matcher defines additional constraints, such as image pull policy, those constraints must also be satisfied. | +| `deny` | Denies the matching request. | +| `audit` | Allows the request, emits a Kubernetes event, and returns an admission warning. | + +If `action` is omitted, Capsule treats the rule as `deny`. + +--- + +#### Targets + +The `targets` field defines which parts of a workload a rule applies to. + +Targets are configured under `enforce.workloads.targets` and are authoritative for target-aware workload enforcement. Registry entries no longer define their own validation targets. + +```yaml +rules: + - enforce: + action: deny + workloads: + targets: + - pod/containers registries: + - exp: "harbor/customer/.*" +``` - # Enforce PullPolicy "always" for all registries (For Container Images and Volume Images) - - url: ".*" - policy: [ "Always" ] +If `targets` is omitted or empty, the rule applies to all workload targets supported by the matching hook. - # If we are pulling from a harbor registry we want to verify the images for Containers only - - url: "harbor/.*" +Supported workload targets are: - - namespaceSelector: - matchExpressions: - - key: env - operator: In - values: ["prod"] - enforce: +| Target | Description | +|---|---| +| `pod/initcontainers` | Applies to images used by `spec.initContainers`. | +| `pod/containers` | Applies to images used by `spec.containers`. | +| `pod/ephemeralcontainers` | Applies to images used by `spec.ephemeralContainers`. | +| `pod/volumes` | Applies to image volumes under `spec.volumes[].image`. | + +Targets are currently used only by a subset of workload hooks. For example, the registry enforcement hook uses targets to decide which Pod image references are validated. The QoS enforcement hook also respects workload targets when evaluating QoS rules. Other hooks may ignore `targets` until they explicitly support target-aware enforcement. + +Examples: + +```yaml +rules: + - enforce: + action: deny + workloads: + targets: + - pod/initcontainers registries: - - url: "harbor/v2/customer-registry/prod-image/.*" - policy: [ "Always" ] + - exp: "harbor/init-only/.*" ``` -Let's try to apply the following pod in the namespace `solar-test` (Does not match the `env=prod` selector): +This rule denies matching images only when they are used by `initContainers`. The same image reference is not denied when used by regular containers, ephemeral containers, or image volumes unless another rule matches those targets. ```yaml -apiVersion: v1 -kind: Pod -metadata: - name: image-volume -spec: - containers: +rules: + - enforce: + action: deny + workloads: + targets: + - pod/containers + - pod/ephemeralcontainers + registries: + - exp: "debug/.*" +``` - - name: shell - command: ["sleep", "infinity"] - imagePullPolicy: Never - image: harbor/v2/prod-registry/debian - volumeMounts: - - name: volume - mountPath: /volume +This rule applies to regular containers and ephemeral containers, but not to init containers or image volumes. - volumes: - - name: volume - image: - reference: quay.io/crio/artifact:v2 - pullPolicy: IfNotPresent +--- +#### QoS Classes -``` +QoS class enforcement allows tenants to allow, deny, or audit Pods based on their [computed Kubernetes QoS class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/). -**What do you expect to happen?** +QoS rules are configured under `enforce.workloads.qosClasses`. -```bash -kubectl apply -f pod.yaml -n solar-test +Supported QoS classes are: -Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: containers[0] reference "harbor/v2/prod-registry/debian" uses pullPolicy=Never which is not allowed (allowed: Always) -``` +| QoS class | Description | +|---|---| +| `Guaranteed` | The Pod has CPU and memory requests and limits set so that requests equal limits. | +| `Burstable` | The Pod has at least one CPU or memory request or limit, but does not qualify as `Guaranteed`. | +| `BestEffort` | The Pod has no CPU or memory requests or limits. | -Because our first rule enforces all registries to use the `always` image pull policy, the pod creation is denied because it uses the `Never` policy. We can either allow the `Never` policy for this specific registry: +Capsule evaluates the QoS class of the incoming Pod during create and update admission. Pod-level resources are considered when present. If Kubernetes has already populated `status.qosClass`, Capsule can use that value; otherwise it computes the QoS class from the Pod specification. + +Deny `BestEffort` Pods: ```yaml --- @@ -257,28 +379,91 @@ spec: ... rules: - enforce: - registries: + action: deny + workloads: + qosClasses: + - BestEffort +``` - # Enforce PullPolicy "always" for all registries (For Container Images and Volume Images) - - url: ".*" - policy: [ "Always" ] +With this rule, a Pod without CPU or memory requests and limits is denied: - # If we are pulling from a harbor registry we want to verify the images for Containers only - - url: "harbor/.*" - policy: [ "Never" ] +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: best-effort +spec: + containers: + - name: shell + image: harbor/platform/debian:latest + command: ["sleep", "infinity"] +``` - - namespaceSelector: - matchExpressions: - - key: env - operator: In - values: ["prod"] - enforce: - registries: - - url: "harbor/v2/customer-registry/prod-image/.*" - policy: [ "always" ] +Example rejection: + +```bash +Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: pod "best-effort" uses QoS class "BestEffort" which is denied by namespace rule +``` + +Audit `Burstable` Pods: + +```yaml +rules: + - enforce: + action: audit + workloads: + qosClasses: + - Burstable +``` + +A matching Pod is admitted, but Capsule emits an event and the API server response contains an admission warning. + +Allow `BestEffort` only for selected namespaces: + +```yaml +rules: + - enforce: + action: deny + workloads: + qosClasses: + - BestEffort + + - namespaceSelector: + matchLabels: + allow-best-effort: "true" + enforce: + action: allow + workloads: + qosClasses: + - BestEffort ``` -But let's for now also remove the generic rule for all registries and only keep the harbor one: +Because later matching allow or deny rules take precedence, namespaces labeled `allow-best-effort=true` can run `BestEffort` Pods, while other namespaces cannot. + +You can also combine QoS rules with targets: + +```yaml +rules: + - enforce: + action: deny + workloads: + targets: + - pod/containers + qosClasses: + - BestEffort +``` + +If `targets` is omitted, the QoS rule applies to all workload targets supported by the QoS hook. + +--- + +#### Registries + +Define image registry rules for `Pods` with regular expressions. The `exp` field is matched against the full OCI reference string. This includes the registry, repository path, image name, tag, and digest if present. + +Registry rules are configured under `enforce.workloads.registries`. The workload-level `targets` field under `enforce.workloads.targets` controls which Pod image references are validated. + +The following example allows Harbor images by default, denies a more specific customer path for regular containers and image volumes, audits regular container images from an audit registry, and allows a production image path only for namespaces matching `env=prod`: ```yaml --- @@ -290,11 +475,27 @@ spec: ... rules: - enforce: - registries: + action: allow + workloads: + registries: + - exp: "harbor/.*" + + - enforce: + action: deny + workloads: + targets: + - pod/containers + - pod/volumes + registries: + - exp: "harbor/customer/.*" - # If we are pulling from a harbor registry we want to verify the images for Containers only - - url: "harbor/.*" - policy: [ "Never" ] + - enforce: + action: audit + workloads: + targets: + - pod/containers + registries: + - exp: "audit/.*" - namespaceSelector: matchExpressions: @@ -302,56 +503,93 @@ spec: operator: In values: ["prod"] enforce: - registries: - - url: "harbor/v2/customer-registry/prod-image/.*" - policy: [ "Always" ] + action: allow + workloads: + targets: + - pod/containers + - pod/volumes + registries: + - exp: "harbor/customer/prod-image/.*" + policy: ["Always"] ``` -If we try to apply the pod again, it we will still get an error. The problem is that we are mounting an image volume that is not coming from the allowed harbor registry: +Let's try to apply the following Pod in namespace `solar-test`, which does not match the `env=prod` selector: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: image-volume +spec: + containers: + - name: shell + command: ["sleep", "infinity"] + imagePullPolicy: IfNotPresent + image: harbor/customer/test-image/debian:latest + volumeMounts: + - name: volume + mountPath: /volume + + volumes: + - name: volume + image: + reference: quay.io/crio/artifact:v2 + pullPolicy: IfNotPresent +``` + +**What do you expect to happen?** ```bash -Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: volumes[0](volume) reference "quay.io/crio/artifact:v2" is not allowed +kubectl apply -f pod.yaml -n solar-test + +Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: containers[0] reference "harbor/customer/test-image/debian:latest" is denied by registry rule "harbor/customer/.*" ``` -However we would like to only validate the images used in the Pod spec (Containers, InitContainers, EphemeralContainers) and not the ones used in the Volumes. We can achieve this by specifying the [`validation`](#validation) field for the harbor registry: +The Pod is denied because the regular container image matches both `harbor/.*` and `harbor/customer/.*`. Since the deny rule is declared later, it has higher precedence. + +The image volume reference is not denied by the shown deny rule because it does not match `harbor/customer/.*`. If the image volume used a matching reference, for example `harbor/customer/volume-artifact:v1`, the same deny rule would apply because it targets both `pod/containers` and `pod/volumes`. + +In a namespace matching `env=prod`, the more specific production allow rule is also considered: ```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant +apiVersion: v1 +kind: Pod metadata: - name: solar + name: prod-image spec: - ... - rules: - - enforce: - registries: - # If we are pulling from a harbor registry we want to verify the images for Containers only - - url: "harbor/.*" - policy: [ "Never" ] - validation: - - "pod/images" + containers: + - name: shell + command: ["sleep", "infinity"] + imagePullPolicy: Always + image: harbor/customer/prod-image/debian:latest +``` - - namespaceSelector: - matchExpressions: - - key: env - operator: In - values: ["prod"] - enforce: +The request is allowed because the namespace-specific rule matches later and allows `harbor/customer/prod-image/.*` with `imagePullPolicy: Always`. + +Target-specific registry rules allow different behavior for different parts of the same Pod. For example, this rule denies the registry only for init containers: + +```yaml +rules: + - enforce: + action: deny + workloads: + targets: + - pod/initcontainers registries: - - url: "harbor/v2/customer-registry/prod-image/.*" - policy: [ "Always" ] + - exp: "harbor/init-only/.*" ``` -#### Policy +A matching reference under `spec.initContainers` is denied. The same reference under `spec.containers` is ignored by this rule. -Define the allowed image pull policies for the specified registry URL. Supported policies are: +##### Policy - * `Always`: The image is always pulled. - * `IfNotPresent`: The image is pulled only if it is not already present on the node. - * `Never`: The image is never pulled. If the image is not present on the node, the Pod will fail to start. +Define the allowed image pull policies for a matching registry expression. Supported policies are: -This configuration is optional. If no policy is specified, all image pull policies are allowed for the given registry. +* `Always`: The image is always pulled. +* `IfNotPresent`: The image is pulled only if it is not already present on the node. +* `Never`: The image is never pulled. If the image is not present on the node, the Pod fails to start. + +The `policy` field is optional. If no policy is specified, all image pull policies are accepted for the matching registry expression. ```yaml --- @@ -362,22 +600,43 @@ metadata: spec: ... rules: - # Matches all Namespaces and enforces the rule for all of them - enforce: + action: allow + workloads: + targets: + - pod/containers + registries: + - exp: "harbor/v2/customer-registry/.*" + policy: ["IfNotPresent", "Always"] +``` + +If a matching `allow` rule defines `policy`, the Pod must use one of the configured pull policies. For example, this rule allows the registry but only with `Always`: + +```yaml +rules: + - enforce: + action: allow + workloads: + targets: + - pod/containers registries: - - url: "harbor/v2/customer-registry/.*" - policy: [ "IfNotPresent", "Always" ] + - exp: "harbor/v2/customer-registry/.*" + policy: ["Always"] ``` +A Pod using `imagePullPolicy: Never` for that registry is rejected: -#### Validation +```bash +Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: containers[0] reference "harbor/v2/customer-registry/debian:latest" uses pullPolicy=Never which is not allowed (allowed: Always) +``` -Define on which parts of the Pod the registry policy must be validated. Currently supported validation targets are: +Policy is checked only after the final registry decision is `allow`. A final `deny` decision always denies the request, regardless of the configured pull policy. - * `pod/images`: Validate the images used in the Pod spec (For `Containers`, `InitContainers` and `EphemeralContainers`). - * `pod/volumes`: Validate the images used in the Pod `Volumes` ([Read More](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/)) +##### Negated regular expressions -**By default, both targets are validated**. You can override this behavior by specifying the `validation` field for each registry: +A registry expression can be negated with `negate: true`. This means the rule matches references that do **not** match the expression. + +For example, the following rule denies every regular container image that is not from the trusted registry path: ```yaml --- @@ -388,10 +647,44 @@ metadata: spec: ... rules: - # Matches all Namespaces and enforces the rule for all of them - enforce: + action: deny + workloads: + targets: + - pod/containers + registries: + - exp: "trusted/.*" + negate: true +``` + +With this rule: + +* `trusted/backend/api:1.0.0` is allowed because it does not match the negated rule. +* `docker.io/library/nginx:latest` is denied because it does not match `trusted/.*`, so the negated expression evaluates to true. + +You can combine negation with namespace selectors and action precedence. For example, deny all untrusted container images by default, but allow a controlled exception in production namespaces: + +```yaml +rules: + - enforce: + action: deny + workloads: + targets: + - pod/containers registries: - - url: "harbor/v2/customer-registry/.*" - validation: - - "pod/images" + - exp: "trusted/.*" + negate: true + + - namespaceSelector: + matchLabels: + env: prod + enforce: + action: allow + workloads: + targets: + - pod/containers + registries: + - exp: "partner-registry/prod-approved/.*" ``` + +In a namespace labeled `env=prod`, `partner-registry/prod-approved/app:1.0.0` is allowed because the later matching allow rule overrides the earlier deny rule.