Run 1B iterations of the "scenarios" test without errors.

[coryan]

I am writing a new stress test for the client library. This test performs different scenarios for the new bidi streaming reads, including:

• Call open_object() and discard the result without making any reads.
• Call open_object() then call read_range() on the result and read all the data.
• Call open_object() then call read_range() read some data and then discard the reader and the descriptor.
• Call open_object() then call read_range() then drop the object descriptor and then read all the data.

The idea is to shake out problems (deadlocks, leaks, race conditions) by trying all these things concurrently, multiple times. We should be able to complete about 1B operations without errors.

[coryan]

I am getting several different errors:

bm-1765912602.txt:447,83741,14125855910,open_read_after_drop,334319711,,1wMOsOatnBm8Q60Oyt3Ong9bwaQ0LkTf,ERROR=the service reports an error with code UNAVAILABLE described as: 502:Bad Gateway;;Caused by:;    status: 'The service is currently unavailable'; self: "502:Bad Gateway"; metadata: {"content-type": "application/grpc"; "date": "Tue; 16 Dec 2025 23:18:14 GMT"; "content-length": "0"}

bm-1765912602.txt:261,87342,14126379567,open_read,360002144,,eMVtWLHCatx9b6ssdSKmLYmPwCBVUUFe,ERROR=the service reports an error with code UNKNOWN described as: 408:Request Timeout;;Caused by:;    status: 'Unknown error'; self: "408:Request Timeout"; metadata: {"content-type": "application/grpc"; "date": "Tue; 16 Dec 2025 23:18:41 GMT"; "content-length": "0"}

bm-1765912602.txt:139,363552,64542187410,open_read_discard,357274547,,tQ9QebJak3bf4MWHGx2t7ut7BofMYC8z,ERROR=the service reports an error with code UNAUTHENTICATED described as: Request had invalid authentication credentials. Expected OAuth 2 access token; login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.;;Caused by:;    status: 'The request does not have valid authentication credentials'; self: "Request had invalid authentication credentials. Expected OAuth 2 access token; login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."; metadata: {"pcx-high-bwd-bin": "KgUYBJgBBA"; "google.rpc.errorinfo-bin": "ChRBQ0NFU1NfVE9LRU5fRVhQSVJFRBIOZ29vZ2xlYXBpcy5jb20aMgoGbWV0aG9kEihnb29nbGUuc3RvcmFnZS52Mi5TdG9yYWdlLkJpZGlSZWFkT2JqZWN0GiEKB3NlcnZpY2USFnN0b3JhZ2UuZ29vZ2xlYXBpcy5jb20"; "endpoint-load-metrics-bin": "MVOuS884KydAOYtBk5wSr+8/SWZmZmZm5h1A"; "grpc-server-stats-bin": "AABXKLguUwAAAA"; "pc-high-bwd-bin": "S2dVWUJKZ0JCQQ"}

[coryan]

The errors go away with 60s RPC timeouts configured in the client:

$ grep -c ,ERROR= bm-1766412318.*
bm-1766412318.log:0
bm-1766412318.txt:0

$ wc -l bm-1766412318.*
        6717 bm-1766412318.log
  1000000513 bm-1766412318.txt
  1000007230 total

This suggests a component in the service is creating timeouts (the origin of the "408:Request Timeout" messages), and something else in the service converts that to a Unknown gRPC status code.

[coryan]

Second run with 60s timeouts, no errors:

$ TS=$(date +%s); RUSTFLAGS="--cfg google_cloud_unstable_storage_bidi -C target-cpu=native"   cargo run --release --package storage-scenarios --   --bucket-name coryan-test-us-central1-bm001 --task-count=512   --grpc-subchannel-count=512 --rampup-period=1s   --iterations 1000000000   >bm-${TS}.txt 2>bm-${TS}.log </dev/null &

$ grep -c ,ERROR= bm-${TS}.*
bm-1766701067.log:0
bm-1766701067.txt:0

$ wc -l bm-${TS}.*
      120783 bm-1766701067.log
  1000000513 bm-1766701067.txt
  1000121296 total

[coryan]

Another successful run with a zonal bucket. Still using the development branch, testing with the main branch next.

[coryan]

This is done.

$ TS=$(date +%s); RUSTFLAGS="-C target-cpu=native"   cargo run --release --package storage-scenarios --   --bucket-name coryan-test-us-central1-bm001 --task-count=1024   --grpc-subchannel-count=1024 --rampup-period=1s   --iterations 1000000000   >bm-${TS}.txt 2>bm-${TS}.log </dev/null &

$ wc -l bm-${TS}.*
      256306 bm-1767912096.log
  1000001025 bm-1767912096.txt
  1000257331 total

$ grep -c ,ERROR= bm-${TS}.*
bm-1767912096.log:0
bm-1767912096.txt:0