Skip to content

Grpc DNS resolver timing out for simple localhost lookup #8356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
peterchenadded opened this issue May 24, 2025 · 3 comments
Closed

Grpc DNS resolver timing out for simple localhost lookup #8356

peterchenadded opened this issue May 24, 2025 · 3 comments

Comments

@peterchenadded
Copy link

peterchenadded commented May 24, 2025

What version of gRPC are you using?

1.71.1

What version of Go are you using (go version)?

1.24.2 Darwin/arm64

What operating system (Linux, Windows, …) and version?

Mac m1 Sequoia 15.5

What did you do?

This is a follow up of #7429 (comment)

Essentially tried running:

https://github.com/grpc/grpc-go/blob/master/examples/helloworld/greeter_client/main.go

But got:

rpc error: code = DeadlineExceeded desc= context deadline exceeded

This was with

localhost:4317

It worked fine with

127.0.01:4317

What did you expect to see?

I expected both localhost and 127.0.01 to work.

What did you see instead?

Only 127.0.0.1 worked

I had to remove the two DNS nameservers entries on my machine to get localhost to work. But this is not an appropriate fix because I needed root to do this and breaks other functionality that does need it.

I could have also
used passthrough://localhost:4317, but for some libraries such as OTEL exporters that use grpc it is currently not possible to pass through the scheme.

I tried net.DefaultResolver.LookupIP(ctx, 'ip', 'localhost') and it resolved with no issues quickly.

The issue seems to be that grpc DNS resolver is trying to contact the name servers even for simple /etc/hosts entries that net.DefaultResolver has no issues working with.

@peterchenadded
Copy link
Author

peterchenadded commented May 24, 2025

Ok after a deep dive into the code I can see

  1. The code that does the resolution is at

    https://github.com/grpc/grpc-go/blob/master/internal/resolver/dns/dns_resolver.go#L336

  2. The lookupHost is working since it is using net.DefaultResolver

  3. but after that it tries to do

    lookupTXT on _grpc_config.localhost

    That is timing out after 5 seconds with

    I/o timeout

  4. So question is how to disable that Txt lookup

    grpc.WithDisableServiceConfig()

Is it possible to have an environment variable to turn this off, to work with problematic nameservers?

@arjan-bal
Copy link
Contributor

arjan-bal commented May 26, 2025

Hi @peterchenadded, there isn't a way to selectively disable service config lookups for hosts. You will need to use the WithDisableServiceConfig() dial option or provide an IP address as the target (e.g: "127.0.0.1" instead of "localhost") to bypass service config lookups. To allow users to configure gRPC channels, libraries usually accept their own options structs which they convert to gRPC DialOptions internally. GCP client libraries are an example of this pattern.

@peterchenadded
Copy link
Author

Thanks @arjan-bal problem is I don't have direct access to change the client options for the library am using e.g. otel grpc.

If there no plans to add a environment variable to disable it, we can close this issue off.

I will have to live with 127.0.0.1 for my local Mac setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants