Design Records
Design records document the engineering decisions behind PGSTY SILO: root causes, S3 and IAM protocol invariants, trade-offs against rejected alternatives, operational limits, and regression coverage.
November 9: Console Languages
- Adding Multilingual Support to SILO: Ten UI languages across Console 2.6.0 and Console 2.7.0 (including Portuguese from #83 and Turkish requested in silo#277), maintenance tiers, English fallback and Arabic RTL, with measured gzip and cache tradeoffs behind per-language assets and eager loading.
November 6: Found in Release Acceptance
- A Partial View Must Not Report an Existing Object as Missing: Why a node that reaches only some drives can answer 404 for an object that exists, and why “not found” must require an answer from every drive.
- A Failed Multipart Completion Can Strand the Upload: Why a completion that misses write quorum can strand the upload until stale-upload cleanup, and why the repair waits for fencing of storage operations.
November 4: Release-Blocking Lifecycle Fixes
- Multi-Pool Lifecycle Expiry: Decide on the Whole Object (#278, #279): Why an expiry decided on the versions of one pool hides, deletes, or resurrects versions in another pool, and why a recheck on the merged version list under the object lock is the minimum complete repair.
- Restored Copies: Expire the Copy, Not the Object (#281, #282): Why the expiry of a restored current version hid the object or stripped an older null version, and which storage and native-restore safeguards ship with the exact-version fix.
- Native Restore in Two Phases (#284): Why a native restore held the object lock for its whole tier download, why the first #281 safeguards also stranded failed ordinary uploads, and how staging outside the lock with a short, checked commit replaces the private upload.
- Audit Webhooks After a Peer Reload (#259): Why a runtime audit_webhook change leaves every server but the serving one with a stopped target, which logger subsystems share it, and why building logger targets on the server context plus opening the queue store first is the minimum complete repair.
- Failed Writes and Phantom Empty Prefixes (#285, #286): Why a write whose staged data vanished before the remote rename leaves an empty object directory that later lists as an empty prefix, and why removing it on both rename failure paths is sufficient.
- Heal Dry Run Must Not Delete Data (from #280): Why a dry-run heal could delete the last metadata copies of a dangling object and run a real bucket heal on remote nodes, and why an early return plus forwarding the dry-run flag to peers closes both paths.
- Lifecycle Delete-All Must Stay on One Key (found during review of #278): Why ExpiredObjectAllVersions, DelMarkerExpiration, and the default batch-expiration purge delete every other key under the expired key’s name, and why purging the key’s own versions by ID after a recheck is the minimum safe repair.
- Bucket Quota in the Console: List Display and Numeric Validation (silo-console#72, silo-console#73): Why a Console-set quota showed as “-” in the bucket list, why bucket creation could turn a huge quota into one byte, where both came from, and what the merged Console fix covers.
- Lock Domains of Large Single-Pool Clusters: Why DELETE and CopyObject fail to exclude PUT on the same key in single-pool clusters of 18 or more hosts, which layouts that covers, and the routing change that closes it.
November 1: Pool Migration Integrity Review
- Pool Migration Integrity Review: #261, #263, and #265: A maintainer review of three reports and their fixes, including the contributor merge of #266 and maintainer follow-up #276.
- Mixed EC Layouts and Migration Listing (#261, #262): Why a readable object can disappear from decommission and rebalance listings after an interrupted degraded overwrite, and why the narrow recount in #262 is sufficient.
- Tier Free Versions Are Internal Cleanup Records (#263, #264): Why a free version must stay visible to healing but must not become a user-visible delete marker or a migration version.
- ILM Expiry and Pool Migration Ordering (#265, #266, #276): Why a versioned Expiration marker must not cause the current version to be skipped, and how the maintainer follow-up preserves tier cleanup and completion semantics.
October 24: FULL_OBJECT Checksum Replication
- FULL_OBJECT Checksum Replication: Preserve the Type at Upload Initiation (
#254,#255): Why dropping the checksum type makes valid multipart CRC objects fail replication, how one shared header fix also restores the existing CopyObject path, and the source-upgrade and acceptance boundaries.
October 24: Multi-Pool Deletion and Placement
-
Multi-Pool Deletion: Preserve the Latest Copy on Failure (
#256,#257): Why an online-drive preflight cannot protect against later I/O failures, how staged cleanup preserves each item’s authoritative copy, and the retry, tier-reference, performance, and compatibility boundaries of the merged fix. -
Overwrite Pool Affinity: Why #245 Was Not Adopted (
#245): The original placement problem, request-aware quorum and health improvements, unversioned-only scope, two rounds of m3 performance measurements, and the decision to stop after small-object overwrite regressions remained. The experimental changes were not merged.
October 23: Alias Regions
- Alias regions: Persistent per-alias regions, client-cache isolation, and upgrade, downgrade and credential-rotation boundaries, reconciled with mcli 20261109.
October 23 IAM & KMS Design Notes
- LDAP Long DNs: Restoring Login Without Hiding Revocation Errors (
#250): How\xxhex escaping inNormalizeDNtriggers the 255-byte single-component filename limit on the object IAM backend, and whyloadIAMConfigBytesWithMetadataclassifies only impossible pure-ASCII path components (> 255bytes) aserrConfigNotFound. - IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets (
#251): Moving policy reload and KMS decryption outside the authentication-cache lock while retainingwriteMu, threading request cancellation end-to-end toKES.Decrypt, and keepingetcd30-second RPC deadlines separate from total decryption budgets.
Server 20261018 Key Design Notes
For the Server 20261018 release, see these core architecture notes:
- Object Lock Governance & Overwrite Protection (
SN-2026-015,SN-2026-016) - IAM Admin Credential Authorization (
SN-2026-017) - Durable IAM & Scoped STS Revocations (
SN-2026-013) - LIST Recovery Lock Cost
- ListMultipartUploads Rollback & Capacity Boundaries
- Multi-Pool Consistency & Rebalance Protection
- KES & OpenBao Integration
-
IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets
Commit 1b5477f3d (pgsty/silo#251) resolves two coupled failure modes under slow KMS/KES responses: Authentication-cache lock contention: ListPolicies and ListPolicyDocs now load and decrypt policy replacements outside the in-memory …
Commit 1b5477f3d (pgsty/silo#251) resolves two coupled failure modes under slow KMS/KES responses: Authentication-cache lock contention: ListPolicies and ListPolicyDocs now load and decrypt policy replacements outside the in-memory …
-
Alias regions: a small configuration change with compatibility boundaries
Decision, 2026-10-23: mcli PR #44 is merged on main at 91006a587. We accepted the per-alias region design, kept its small production implementation and supplemented the regression coverage. This records the original design decision. The capability is …
Decision, 2026-10-23: mcli PR #44 is merged on main at 91006a587. We accepted the per-alias region design, kept its small production implementation and supplemented the regression coverage. This records the original design decision. The capability is …
-
SILO, KES and OpenBao: Key Ownership and Recovery
SILO integrates with external key-management systems through PGSTY KES (MINIO_KMS_KES_ENDPOINT). The current release is KES 20261026 (built with Go 1.27.3 and updated cloud dependencies while retaining the API, configuration, and key formats …
SILO integrates with external key-management systems through PGSTY KES (MINIO_KMS_KES_ENDPOINT). The current release is KES 20261026 (built with Go 1.27.3 and updated cloud dependencies while retaining the API, configuration, and key formats …
-
LIST Recovery: Confirm Only Uncertain Names and Bound the Waiting Cost
Shipped in SILO Server 20261018, PR #239 (merged as fbcb51197, closing #218 and retaining mr javad seydi’s original signed commits from #221) and the October 18 directory/common-prefix follow-ups (3dd53eb81 and 2080b3d03) resolve silent object …
Shipped in SILO Server 20261018, PR #239 (merged as fbcb51197, closing #218 and retaining mr javad seydi’s original signed commits from #221) and the October 18 directory/common-prefix follow-ups (3dd53eb81 and 2080b3d03) resolve silent object …
-
IAM Admin API Authorization Boundaries
Server 20261018 ships Server PR #242 (commit 2616d43fb on baseline 00a375c4e, merged as 4e55fc7db) and the bulk self-listing follow-up 995439260 to close SN-2026-017. This design note records the problem scope, handler-level authorization contract, …
Server 20261018 ships Server PR #242 (commit 2616d43fb on baseline 00a375c4e, merged as 4e55fc7db) and the bulk self-listing follow-up 995439260 to close SN-2026-017. This design note records the problem scope, handler-level authorization contract, …
-
Clearing Stale Fields in Legacy Metadata and Scanner Responses
This article records the design and verification of PR #233 (tested head 86c5a92eef3d, merged as d94b3b3c9251), which shipped in Server 20261018. It fixes stale fields carried over when legacy xl.meta version headers and remote namespace-scanner …
This article records the design and verification of PR #233 (tested head 86c5a92eef3d, merged as d94b3b3c9251), which shipped in Server 20261018. It fixes stale fields carried over when legacy xl.meta version headers and remote namespace-scanner …
-
Object Lock Retention Bypass and Version-Overwrite Repair Design (SN-2026-015 / SN-2026-016)
This document records the verification, Git provenance, repair design, critical-path evaluation, and candidate trade-offs for two Object Lock (WORM) security vulnerabilities identified through external reports and an internal write-path audit: …
This document records the verification, Git provenance, repair design, critical-path evaluation, and candidate trade-offs for two Object Lock (WORM) security vulnerabilities identified through external reports and an internal write-path audit: …
-
Pinned TLS Parameters and Handshake Compatibility
After the Go TLS default repair 48e184652 shipped in Server 20260916, the reporter of issue #154 retested on that release and still saw connection reset by peer on OIDC discovery. Meanwhile, upstream Go issues reported the exact same symptom and …
After the Go TLS default repair 48e184652 shipped in Server 20260916, the reporter of issue #154 retested on that release and still saw connection reset by peer on OIDC discovery. Meanwhile, upstream Go issues reported the exact same symptom and …
-
A Listing Must Not Drop a Null Version That Still Has Quorum
This record covers the listing omission investigated on 2026-09-16: an unversioned bucket under concurrent overwrites returned HTTP 200 listings that lacked keys the same endpoint could still GET. It describes the confirmed mechanism, the targeted …
This record covers the listing omission investigated on 2026-09-16: an unversioned bucket under concurrent overwrites returned HTTP 200 listings that lacked keys the same endpoint could still GET. It describes the confirmed mechanism, the targeted …
-
Go 1.27 TLS Defaults and OIDC Discovery Failure Modes
After SILO’s toolchain moved to Go 1.27, the Server TLS repair 48e184652 (“fix(tls): honor Go key exchange defaults across transports”, shipped in Server 20260916) removed explicit curve overrides across eight Server TLS configuration points. This …
After SILO’s toolchain moved to Go 1.27, the Server TLS repair 48e184652 (“fix(tls): honor Go key exchange defaults across transports”, shipped in Server 20260916) removed explicit curve overrides across eight Server TLS configuration points. This …
-
Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts
This article records the repair of Server’s HTTP read deadlines, merged in PR #196 (fix 055030ea5) and shipped in Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a). The investigation was prompted by a slow-HTTP DoS scanner report (#183) and …
This article records the repair of Server’s HTTP read deadlines, merged in PR #196 (fix 055030ea5) and shipped in Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a). The investigation was prompted by a slow-HTTP DoS scanner report (#183) and …
-
Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject
Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #194 (commit 4fcdf37ce, merged as 9f3037e941, retaining Mikhail Khadarenka’s original implementation from PR #187) fixes a regression where a trusted replication receiver …
Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #194 (commit 4fcdf37ce, merged as 9f3037e941, retaining Mikhail Khadarenka’s original implementation from PR #187) fixes a regression where a trusted replication receiver …
-
Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection
Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #193 (commit 03027727d) and PR #196 (commit 680eac66e) fix two defects in how object tags preserve ordering across bucket replication: SSE-KMS destination copies silently …
Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #193 (commit 03027727d) and PR #196 (commit 680eac66e) fix two defects in how object tags preserve ordering across bucket replication: SSE-KMS destination copies silently …
-
SSE-C Replica Integrity
This article records the SSE-C replica integrity contract formed by #122, #123, #124, #126, and #134, which shipped in Server 20260916, building on the earlier zero-byte/read-attribute authentication and destination-key checksum repairs from Server …
This article records the SSE-C replica integrity contract formed by #122, #123, #124, #126, and #134, which shipped in Server 20260916, building on the earlier zero-byte/read-attribute authentication and destination-key checksum repairs from Server …
-
Object Lock Replication Ordering
Initial Object Lock replication timestamp ordering (f4c1286c9) shipped in SILO Server 20260903 (recorded in the security advisory ledger). Subsequent fixes for timestamp-only removals, SSE-C replica retransmission, and multi-pool Object Lock …
Initial Object Lock replication timestamp ordering (f4c1286c9) shipped in SILO Server 20260903 (recorded in the security advisory ledger). Subsequent fixes for timestamp-only removals, SSE-C replica retransmission, and multi-pool Object Lock …
-
Multi-Pool Object Consistency
Shipped in SILO Server 20260916, PR #178 repairs multi-pool mutation ordering, PR #207 extends current-object selection to conditional PUT (closing #133 and #144), and commit fced86303 preserves object tags across pool rebalance and decommission. …
Shipped in SILO Server 20260916, PR #178 repairs multi-pool mutation ordering, PR #207 extends current-object selection to conditional PUT (closing #133 and #144), and commit fced86303 preserves object tags across pool rebalance and decommission. …
-
Federated CopyObject: Preserve the Destination Contract
Server 20260916 repairs the legacy etcd bucket-federation CopyObject path across PR #157, PR #159, PR #163, PR #177, and PR #179. When CopyObject forwards a copy across federated deployments as a remote PutObject, the proxy streams logical source …
Server 20260916 repairs the legacy etcd bucket-federation CopyObject path across PR #157, PR #159, PR #163, PR #177, and PR #179. When CopyObject forwards a copy across federated deployments as a remote PutObject, the proxy streams logical source …
-
Durable IAM Revocations
SILO retains the revision of every deleted IAM record so an offline site cannot resurrect a deleted identity or policy grant when it reconnects. This mechanism covers built-in users, service accounts, groups, policy documents, and policy mappings …
SILO retains the revision of every deleted IAM record so an offline site cannot resurrect a deleted identity or policy grant when it reconnects. This mechanism covers built-in users, service accounts, groups, policy documents, and policy mappings …
-
Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence
#77 is a site-replication correctness defect repaired in Server 20260916 (PR #180, commit 48ec10312). Before this repair, a receiving site overwrote the peer’s source timestamp with local arrival time (UTCNow) and could then reject a genuinely newer …
#77 is a site-replication correctness defect repaired in Server 20260916 (PR #180, commit 48ec10312). Before this repair, a receiving site overwrote the peer’s source timestamp with local arrival time (UTCNow) and could then reject a genuinely newer …
-
Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation
This article records the root-cause analysis, design tradeoffs, and validation across three rounds of replication reliability repairs shipped in Server 20260916: Round 1 (PR #162, merged as d1105bbb): addresses #153, #152, and #137, covering …
This article records the root-cause analysis, design tradeoffs, and validation across three rounds of replication reliability repairs shipped in Server 20260916: Round 1 (PR #162, merged as d1105bbb): addresses #153, #152, and #137, covering …