ScanStruct with a Nullable tuple value results in pointer value as text

[Viktor-Nordell]

Observed

When using ScanStruct to scan a tuple containing a nullable string the driver will insert the pointer address as a string into the struct.

Or with other words:

1. Row containing tuple with a Nullable string.
2. ScanStruct the row into a struct with a string field.
3. A pointer is stored into the field as text, e.g. '0xc00052f790'

(The behavior is correct when not using nullable values)

Expected behaviour

Either the ScanStruct operation should fail, or null values are mapped to "".

Code example

The code does the following:

1. Creates a new table with a tuple.
2. Inserts test data.
3. Reads the row into Row using ScanStruct.
4. The tuple values in the struct (myRow.MyTuple.TupleOne) now contains a pointer as string, instead of the actual value.

package code

type Tuple struct {
	TupleOne string `ch:"tuple_one"`
	TupleTwo string `ch:"tuple_two"`
}

type Row struct {
	MyId    uint64 `ch:"my_id"`
	MyTuple Tuple  `ch:"my_tuple"`
}

func TestBug(t *testing.T) {
	ctx := context.Background()

        db := // Db setup code

	createdDb := `
create table if not exists my_table(
    my_id    UInt64,
    my_tuple Tuple(tuple_one Nullable(String), tuple_two Nullable(String))
) engine = MergeTree primary key (my_id) order by (my_id)
`

	if err := db.Exec(ctx, createdDb); err != nil {
		t.Fatal(err)
	}

	if err := db.Exec(ctx, "insert into my_table(my_id, my_tuple) values (1, tuple('one', 'two'))"); err != nil {
		t.Fatal(err)
	}

	myRow := Row{}

	if err := db.QueryRow(ctx, "select * from my_table limit 1").ScanStruct(&myRow); err != nil {
		t.Fatal(err)
	}

	if myRow.MyTuple.TupleOne != "one" {
		t.Errorf("Expected 'one' but got '%s'", myRow.MyTuple.TupleOne)
	}

	if myRow.MyTuple.TupleTwo != "two" {
		t.Errorf("Expected 'two' but got '%s'", myRow.MyTuple.TupleTwo)
	}
}

The result will be:

    bug_test.go:48: Expected 'one' but got '0xc00052f790'
    bug_test.go:52: Expected 'two' but got '0xc00052f7c0'

Details

Environment

• clickhouse-go version: v2.30.0
• Interface: ClickHouse API / database/sql compatible driver
• Go version: 1.23.4
• Operating system: linux/amd64
• ClickHouse version: 24.5.4
• Is it a ClickHouse Cloud? No
• ClickHouse Server non-default settings, if any: None
• CREATE TABLE statements for tables involved: See code example
• Sample data for all these tables, use [clickhouse-obfuscator]: (https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary

[jkaflik]

@SpencerTorres could you take a look?

[SpencerTorres]

Hello! Thanks for submitting a code sample. I was able to re-create this issue immediately.

Problem

The issue is happening here:

// lib/column/tuple.go:204

if field.Kind() == reflect.String {
    if v := reflect.ValueOf(fmt.Sprint(value.Interface())); v.Type().AssignableTo(field.Type()) {
        field.Set(v)
        return nil
    }
}

The intent of this code is to allow your row value to be assigned to your struct's string. Whether it's a string, a number, a bool, etc. it will try to convert it to a string via fmt.Sprint(value.Interface()). It also checks whether this resulting string is assignable to your string field... which in this case, it will ALWAYS be assignable since fmt.Sprint will always return a string.

The reason it's returning a pointer is because the Nullable column makes it a *string, but the error message failed to tell you this. fmt.Sprint is making this the pointer address.

What you should've seen is this:

clickhouse [ScanRow]: (my_tuple) converting *string to string is unsupported

Solution

Simply change string to *string.

I believe the code I linked should be deleted. It's trying to force a conversion by using fmt.Sprint. I could fix this so that it supports dereferencing the Nullable's pointer, but if the type is more complex then this, people will still be getting silent errors for these failed conversions. If the user doesn't know the type then they can simply use any. If this code is removed then it will still try converting the primitives as well as trying the sql.Scanner interface. If it can't be converted, then you'll receive a clearer error.

I'll submit a PR to apply the suggested fix. Let me know if you have any further questions. Thanks!

[Viktor-Nordell]

Thanks @SpencerTorres !

What you should've seen is this:

clickhouse [ScanRow]: (my_tuple) converting *string to string is unsupported

Odd didn't see it, or at least didn't notice it.

I could fix this so that it supports dereferencing the Nullable's pointer, but if the type is more complex then this, people will still be getting silent errors for these failed conversions

Wouldn't it be better if the call to ScanStruct failed with an actual error, instead of giving an error in the log?

[SpencerTorres]

Wouldn't it be better if the call to ScanStruct failed with an actual error, instead of giving an error in the log?

Yes! The change in #1465 will return an error all the way to ScanStruct. If the conversion is not possible you will see the correct error returned

[Viktor-Nordell]

Yes! The change in #1465 will return an error all the way to ScanStruct. If the conversion is not possible you will see the correct error returned

Awesome, then I think you can close issue, thanks for the help!