Enabling retention is incompatible with setting up a proxy through env vars in SingleBinary and SimpleScalable modes #17100
Comments
|
Hi, So digging a little bit deeper, it seems that ":9095" requests are still being forwarded to the proxy even with ruler disabled. Only thing is that the backend pod is not complaining about it unless some rules are loaded in the ruler... Looking in the code base related to the compactor, I found this : func (t *Loki) compactorAddress() (string, bool, error) {
legacyReadMode := t.Cfg.LegacyReadTarget && t.Cfg.isTarget(Read)
if t.Cfg.isTarget(All) || legacyReadMode || t.Cfg.isTarget(Backend) {
// In single binary or read modes, this module depends on Server
return net.JoinHostPort(t.Cfg.Server.GRPCListenAddress, strconv.Itoa(t.Cfg.Server.GRPCListenPort)), true, nil
}
if t.Cfg.Common.CompactorAddress == "" && t.Cfg.Common.CompactorGRPCAddress == "" {
return "", false, errors.New("query filtering for deletes requires 'compactor_grpc_address' or 'compactor_address' to be configured")
}
if t.Cfg.Common.CompactorGRPCAddress != "" {
return t.Cfg.Common.CompactorGRPCAddress, true, nil
}
return t.Cfg.Common.CompactorAddress, false, nil
}In my case, the backend pod enters the first condition, as the target is "Backend" : if t.Cfg.isTarget(All) || legacyReadMode || t.Cfg.isTarget(Backend) {
// In single binary or read modes, this module depends on Server
return net.JoinHostPort(t.Cfg.Server.GRPCListenAddress, strconv.Itoa(t.Cfg.Server.GRPCListenPort)), true, nil
}The address returned is ":9095", without a hostname, because the grpc_listen_address is set to "" in the configuration. But in SingleBinary and SimpleScalableSetup the GRPC client try to connect to ":9095", falling into the first condition. Not sure if it's expected. |
|
Renaming the Issue according to the latest findings. |
rurod
changed the title
Uh oh!
There was an error while loading. Please reload this page.
Describe the bug
Loki forwards ":9095" requests to HTTP Proxy when both ruler and retention are enabled.
To Reproduce
Steps to reproduce the behavior:
Issue does not occur if no alerting rule is configured.
Issue disappears if retention is disabled
Error message :
Expected behavior
Environment:
Screenshots, Promtail config, or terminal output
Backend pod logs :
Squid proxy access logs :
SimpleScalableValues.yaml
Ruler rules configmap:
The text was updated successfully, but these errors were encountered: