Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fleet-server fips140=only test failures #4618

Open
michel-laterman opened this issue Mar 21, 2025 · 3 comments
Open

fleet-server fips140=only test failures #4618

michel-laterman opened this issue Mar 21, 2025 · 3 comments
Labels
Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Comments

@michel-laterman
Copy link
Contributor

When testing on a Linux VM with microsoft/go and a FIPS provider, running CGO_ENABLED=1 FIPS=true make test-unit will result in the following tests failing.
These are all tests around our handling of certs, so this is unlikely to be an issue with the binary and probably is an issue with our test code:

=== RUN   Test_server_ClientCert
=== RUN   Test_server_ClientCert/no_client_certs
    server.go:98: {"level":"info","message":"Listening on localhost:41397"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:34782: EVP_KDF_derive\nopenssl error(s):\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 644 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc000049b00?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000307140, {0xc001904100, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc001904100, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc00024f688, 0x320e7a0, 0x2, {0xc001904100, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc00024f508, {0x2511018, 0xc00018ed20})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc00024f508, {0x2510fe0, 0xc001f65740})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc001847dd0, {0x2510fe0, 0xc00205fd10})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 634\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server_test.go:169:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/api/server_test.go:169
        	Error:      	Received unexpected error:
        	            	Get "https://localhost:41397/api/status": EOF
        	Test:       	Test_server_ClientCert/no_client_certs
=== RUN   Test_server_ClientCert/valid_client_certs
    server.go:98: {"level":"info","message":"Listening on localhost:43115"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:37202: EVP_KDF_derive\nopenssl error(s):\nerror:0308010C:digital envelope routines::unsupported\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../crypto/evp/evp_fetch.c:349\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 638 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc0004191f0?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000025140, {0xc0003e6980, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc0003e6980, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc0000cc508, 0x320e7a0, 0x2, {0xc0003e6980, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000025610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000025610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc0000cc388, {0x2511018, 0xc001755810})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc0000cc388, {0x2510fe0, 0xc002128690})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc0001cd5f0, {0x2510fe0, 0xc0020140f0})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 650\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server_test.go:241:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/api/server_test.go:241
        	Error:      	Received unexpected error:
        	            	Get "https://localhost:43115/api/status": EOF
        	Test:       	Test_server_ClientCert/valid_client_certs
=== RUN   Test_server_ClientCert/invalid_client_certs
    server.go:98: {"level":"info","message":"Listening on localhost:42115"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:58804: EVP_KDF_derive\nopenssl error(s):\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 678 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc0002ca010?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000307140, {0xc0003e76c0, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc0003e76c0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc000148508, 0x320e7a0, 0x2, {0xc0003e76c0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc000148388, {0x2511018, 0xc000134550})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc000148388, {0x2510fe0, 0xc000d471a0})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc000155950, {0x2510fe0, 0xc00042cc60})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 657\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server.go:74: {"level":"warn","error":"close tcp 127.0.0.1:42115: use of closed network connection","message":"server.Run: error while closing listener."}
=== RUN   Test_server_ClientCert/valid_client_certs_no_certs_requested
    server.go:98: {"level":"info","message":"Listening on localhost:39959"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:42778: EVP_KDF_derive\nopenssl error(s):\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 666 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc000261940?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000307140, {0xc001904ee0, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc001904ee0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc0000ccc08, 0x320e7a0, 0x2, {0xc001904ee0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc0000cca88, {0x2511018, 0xc0003ee6e0})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc0000cca88, {0x2510fe0, 0xc000df1380})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc0010fb440, {0x2510fe0, 0xc000d47a10})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 662\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server_test.go:396:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/api/server_test.go:396
        	Error:      	Received unexpected error:
        	            	Get "https://localhost:39959/api/status": EOF
        	Test:       	Test_server_ClientCert/valid_client_certs_no_certs_requested
--- FAIL: Test_server_ClientCert (4.89s)
    --- FAIL: Test_server_ClientCert/no_client_certs (0.63s)
    wserver.go:74: {"level":"warn","error":"close tcp 127.0.0.1:41397: use of closed network connection","message":"server.Run: error while closing listener."}
    --- FAIL: Test_server_ClientCert/valid_client_certs (0.76s)
    wserver.go:74: {"level":"warn","error":"close tcp 127.0.0.1:43115: use of closed network connection","message":"server.Run: error while closing listener."}
    --- PASS: Test_server_ClientCert/invalid_client_certs (1.03s)
    --- FAIL: Test_server_ClientCert/valid_client_certs_no_certs_requested (1.28s)

=== RUN   TestAPMHTTPTransportOptions/custom_cert
    instrumentation_test.go:94: start test server to verify TLSClientConfig...
2025/03/21 19:20:33 http: panic serving 127.0.0.1:43508: EVP_KDF_derive
openssl error(s):
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 177 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0xfb4720?, 0xc0003e6100?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc0000ed140, {0xc0003be1e0, 0x20, 0x20}, {0x10a2d39, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x170b040, {0xc0003be1e0, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0000aa508, 0x170b040, 0x2, {0xc0003be1e0, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc0000ed610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc0000ed610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0000aa388, {0x11d7d88, 0xc000194050})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0000aa388, {0x11d7d50, 0xc0003e42a0})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc000188240, {0x11d7d50, 0xc0003e4120})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 178
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
    instrumentation_test.go:112:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/config/instrumentation_test.go:112
        	Error:      	Received unexpected error:
        	            	Get "https://127.0.0.1:43423": EOF
        	Test:       	TestAPMHTTPTransportOptions/custom_cert

=== RUN   TestClientCerts/no_certs
2025/03/21 19:20:39 http: panic serving 127.0.0.1:43928: EVP_KDF_derive
openssl error(s):
error:0308010C:digital envelope routines::unsupported
	../crypto/evp/evp_fetch.c:349
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 97 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0x18c4820?, 0xc000037850?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc00003b140, {0xc00002b380, 0x20, 0x20}, {0x1a29d2f, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x25aacc0, {0xc00002b380, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0001a3a08, 0x25aacc0, 0x2, {0xc00002b380, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc00003b610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc00003b610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0001a3888, {0x1bed290, 0xc000362230})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0001a3888, {0x1bed258, 0xc0003721e0})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc0001b4870, {0x1bed258, 0xc0003720f0})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 8
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
    client_test.go:63:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/es/client_test.go:63
        	Error:      	Received unexpected error:
        	            	EOF
        	Test:       	TestClientCerts/no_certs
=== RUN   TestClientCerts/uses_certs
2025/03/21 19:20:39 http: panic serving 127.0.0.1:38270: EVP_KDF_derive
openssl error(s):
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 13 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0x18c4820?, 0xc000037b10?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc0003df140, {0xc00002b580, 0x20, 0x20}, {0x1a29d2f, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x25aacc0, {0xc00002b580, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0000af688, 0x25aacc0, 0x2, {0xc00002b580, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc0003df610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc0003df610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0000af508, {0x1bed290, 0xc0000fa870})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0000af508, {0x1bed258, 0xc00048ec60})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc0000ee3f0, {0x1bed258, 0xc00048eb70})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 99
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
    client_test.go:112:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/es/client_test.go:112
        	Error:      	Received unexpected error:
        	            	EOF
        	Test:       	TestClientCerts/uses_certs
=== RUN   TestClientCerts/client_cert_does_not_match
2025/03/21 19:20:40 http: panic serving 127.0.0.1:58120: EVP_KDF_derive
openssl error(s):
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 104 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0x18c4820?, 0xc000036da0?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc00015f140, {0xc00002b040, 0x20, 0x20}, {0x1a29d2f, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x25aacc0, {0xc00002b040, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0001a2508, 0x25aacc0, 0x2, {0xc00002b040, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc00015f610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc00015f610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0001a2388, {0x1bed290, 0xc0003622d0})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0001a2388, {0x1bed258, 0xc00018cb70})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc0001b4360, {0x1bed258, 0xc00048e690})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 114
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
--- FAIL: TestClientCerts (2.14s)
    --- FAIL: TestClientCerts/no_certs (0.36s)
    --- FAIL: TestClientCerts/uses_certs (0.81s)
@michel-laterman michel-laterman added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Mar 21, 2025
@michel-laterman
Copy link
Contributor Author

michel-laterman commented Mar 21, 2025 •
edited
Loading

Note this seems to only occur when using the microsoft/go toolchain.

Digging into cert generation for a test; the test is configured with:

fleet-server/internal/pkg/es/client_test.go

Lines 69 to 87 in 2e51560

ca := certs.GenCA(t)
server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Elastic-Product", "Elasticsearch")
fmt.Fprintln(w, "You know, For Search.")
}))
certPool := x509.NewCertPool()
certPool.AddCert(ca.Leaf)
// test server will verify a client cert if present
server.TLS = &tls.Config{
Certificates: []tls.Certificate{ca},
ClientAuth: tls.VerifyClientCertIfGiven,
ClientCAs: certPool,
MinVersion: tls.VersionTLS12,
}
server.StartTLS()
defer server.Close()
cert := certs.GenCert(t, ca)

The error messages for keylength indicate the issue is with key length, we use 2048 (fips 140 2/3 requirement):

fleet-server/internal/pkg/testing/certs/certs.go

Line 93 in 2e51560

caKey, err := rsa.GenerateKey(rand.Reader, 2048) // less secure key for quicker testing.

I'm not sure what is causing these failures

@michel-laterman
Copy link
Contributor Author

michel-laterman commented Mar 21, 2025 •
edited
Loading

Looks like something similar has been reported and fixed: golang-fips/openssl#253

Manually patching in the changes from the PR associated with the fix: golang-fips/openssl#260 fixed all above test failures
could not recreate the original failure within the multipass vm, this does not appear to solve the failures on the test ami used to discover them

@michel-laterman
Copy link
Contributor Author

It may be something that was patched within openssl.
The test AMI where this is occuring is running

OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

However the multipass instance I tried to recreate within ran

OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@michel-laterman