Last month, a single container in my GKE cluster (Sao Paulo region) entered an error loop, outputting to stdout at ~2k logs/second.
I discovered the hard way that GKE's default behavior is to ingest 100% of this into Cloud Logging with no rate limiting.
My bill jumped nearly 1000% before alerts caught it.
I fixed the loop and paused the `_Default` sink immediately.
I opened a billing ticket requesting a "one-time courtesy adjustment" for a runaway resource—standard practice for first-time anomalies on AWS/Azure.
I have been rejected twice.
The latest response: "The team has declined the adjustment request due to our internal policies."
If you run GKE, the `_Default` sink in Log Router captures all container stdout/stderr.
There is NO DEFAULT CAP on ingestion volume which is an absurd!
A simple while(true); do echo "error"; done can bankrupt a small project.
Go to Logging -> Log Router.
Edit _Default sink.
Add an exclusion filter: resource.type="k8s_container" severity=INFO (or exclude specific namespaces).
Has anyone successfully escalated a billing dispute past Tier 1 support recently?
It seems their policy is now to enforce full payment even on obvious runaway/accidental usage.
Last month, a single container in my GKE cluster (Sao Paulo region) entered an error loop, outputting to stdout at ~2k logs/second.
I discovered the hard way that GKE's default behavior is to ingest 100% of this into Cloud Logging with no rate limiting. My bill jumped nearly 1000% before alerts caught it.
Infrastructure (Compute): ~$140 (R$821 BRL) Cloud Logging: ~$1,300 (R$7,554 BRL)
Ratio: Logging cost 9.2x the actual servers.
https://imgur.com/jGrxnkh
I fixed the loop and paused the `_Default` sink immediately.
I opened a billing ticket requesting a "one-time courtesy adjustment" for a runaway resource—standard practice for first-time anomalies on AWS/Azure.
I have been rejected twice.
The latest response: "The team has declined the adjustment request due to our internal policies."
If you run GKE, the `_Default` sink in Log Router captures all container stdout/stderr.
There is NO DEFAULT CAP on ingestion volume which is an absurd!
A simple while(true); do echo "error"; done can bankrupt a small project.
Go to Logging -> Log Router. Edit _Default sink.
Add an exclusion filter: resource.type="k8s_container" severity=INFO (or exclude specific namespaces).
Has anyone successfully escalated a billing dispute past Tier 1 support recently?
It seems their policy is now to enforce full payment even on obvious runaway/accidental usage.