Progressive generics support

[fbiville]

Go 1.18 was released on March 2022 and introduced support for the long-awaited generics.
It comes to no-one's surprise that such support was requested soon after for this driver, as you can see here.

Quick Feature overview

Since the initial request, a few generic functions have been added.

The first generic top-level functions appeared in 5.4.0:

• ExecuteRead[T any] and ExecuteWrite[T any], generic variants of the session-level API ExecuteRead and ExecuteWrite
• GetRecordValue[T RecordValue] to extract a record value in a type-safe way
• GetProperty[T PropertyValue] to extract a node or relationship property in a type-safe way

In the 5.5.0 release, the experimental ExecuteQuery[T] also appeared.

While these additions work great and allow to reduce some of usual boilerplate, they also expose some limitations of the current generics support.

These generic APIs are defined at the surface. The core of the driver is still pretty much unsafe and often defaults to the any (Go 1.18 equivalent to interface{}) where a proper type could work better.

Fundamental limitation

The main reason (beyond my own bandwidth) is that Go generics have one very impactful limitation, best illustrated as follows:

type Foo[T any] interface {
    DoSomething() T // ✅ valid!
}

func DoSomething[T any]() T // ✅ valid!

type Bar interface {
    DoSomething[T any]() T // ❌ invalid!
}

type Baz[U any] interface {
    DoSomething[T any]() T // ❌ still invalid!
}

Since most the driver has been built with interfaces and struct receivers, you can imagine how this restriction (as technically sound as it is) is very limiting.

Hypothetical Generic Interfaces

Let's say we could retrofit generic type information into the existing interfaces as follows (this is obviously a major breaking change and would not occur before 6.0).

type DriverWithContext[T any] interface {
	DefaultExecuteQueryBookmarkManager() BookmarkManager
	Target() url.URL
	NewSession(ctx context.Context, config SessionConfig) SessionWithContext[T]
	VerifyConnectivity(ctx context.Context) error
	Close(ctx context.Context) error
	IsEncrypted() bool
	GetServerInfo(ctx context.Context) (ServerInfo, error)
}

type SessionWithContext[T any] interface {
	LastBookmarks() Bookmarks
	BeginTransaction(ctx context.Context, configurers ...func(*TransactionConfig)) (ExplicitTransaction[T], error)
	ExecuteRead(ctx context.Context, work ManagedTransactionWork[T], configurers ...func(*TransactionConfig)) (any, error)
	ExecuteWrite(ctx context.Context, work ManagedTransactionWork[T], configurers ...func(*TransactionConfig)) (any, error)
	Run(ctx context.Context, cypher string, params map[string]any, configurers ...func(*TransactionConfig)) (ResultWithContext[T], error)
	Close(ctx context.Context) error
}

type ExplicitTransaction[T any] interface {
	Run(ctx context.Context, cypher string, params map[string]any) (ResultWithContext[T], error)
	Commit(ctx context.Context) error
	Rollback(ctx context.Context) error
	Close(ctx context.Context) error
}

type ManagedTransactionWork[T any] func(tx ManagedTransaction[T]) (T, error)

type ManagedTransaction[T any] interface {
	Run(ctx context.Context, cypher string, params map[string]any) (ResultWithContext[T], error)
}

type ResultWithContext[T any] interface {
	Keys() ([]string, error)
	NextRecord(ctx context.Context, record *T, mapper RecordMapper[T]) bool
	Next(ctx context.Context) bool
	PeekRecord(ctx context.Context, record *T, mapper RecordMapper[T]) bool
	Peek(ctx context.Context) bool
	Err() error
	Record(mapper RecordMapper[T]) T
	Collect(ctx context.Context, mapper RecordMapper[T]) ([]T, error)
	Single(ctx context.Context, mapper RecordMapper[T]) (T, error)
	Consume(ctx context.Context) (ResultSummary, error)
	IsOpen() bool
}

type RecordMapper[T any] func(*Record) (T, error)

Since I have spent roughly 1 minute on this, maybe there are better solutions than this.
However, there is 1 fundamental limitation that one cannot avoid, in my opinion.

There must be as many DriverWithContext[T] (and SessionWithContext[T]...) as required T, and this is a big no-no.
Imagine you need an int64 for one query and a *User for the other.
The only way you would be able to do it with these generic interfaces would be create one driver instance for each: DriverWithContext[int64], DriverWithContext[*User], ...
This would mean create as many costly connection pools as required result types.

Alternatively, you could define a single type constraint, "union-ing" all the required result types, but that becomes quite cumbersome and inflexible.

I'm happy to be proven wrong here, but I think the only possible solution for the time being is to define new top-level functions.

Needless to say that does not help at all with the current neo4j package cruft 😅

GetPropertyValue limitation

Let's rewind and look at one of the existing top-level generic function and how it's limited by the current lack of generics support internally.

Let's look at GetProperty[T PropertyValue], and zoom in on the PropertyValue definition:

type PropertyValue interface {
	bool | int64 | float64 | string |
		Point2D | Point3D |
		Date | LocalTime | LocalDateTime | Time | Duration | /* OffsetTime == Time == dbtype.Time */
		[]byte | []any
}

PropertyValue is defined as a type that can be either of bool, int64, ..., []any.

Since this is used in the context of GetProperty, this type is supposed to reflect all the possible types of a node's or relationship's property.

Where it falls short is that []any is actually way too broad of a type.
A node or relationship can define array properties, but these arrays must be:

• homegeneous (you cannot mix integers and booleans e.g. within the same array)
• made of simple types (you cannot have an array of array (inc. byte arrays), or array of maps)

A better definition would be:

type PropertyValue interface {
	bool | int64 | float64 | string |
		Point2D | Point3D |
		Date | LocalTime | LocalDateTime | Time | Duration |
		[]byte |
		[]bool | []int64 | []float64 | []string |
		[]Point2D | []Point3D |
		[]Date | []LocalTime | []LocalDateTime | []Time | []Duration
}

(Note: this is the least ugly definition I could find that makes the compiler happy)

It's not too costly to perform a type assertion of a single any to one of the simple types, and that's what GetProperty does:

	value, ok := rawValue.(T)
	if !ok {
		zeroValue := *new(T)
		return zeroValue, fmt.Errorf("expected value to have type %T but found type %T", zeroValue, rawValue)
	}

It's not the same story when one gets an []any and needs to get to []T.
The easiest implementation would be to iterate over every element and type-assert them to the desired T and append them to the final []T slice, but it can get pretty costly very quickly.

Supporting better slice types will likely require a rewrite of the internal core message read/write pieces to generic variants and it's not a small undertaking.

Long story short, the current property support is limited to []any.

Note: it gets even more complex with record values, since users can return really any kind of type, so proper type safety is really limited:

type RecordValue interface {
	bool | int64 | float64 | string |
		Point2D | Point3D |
		Date | LocalTime | LocalDateTime | Time | Duration | /* OffsetTime == Time == dbtype.Time */
		[]byte | []any | map[string]any |
		Node | Relationship | Path
}

We cannot really get beyond []any and map[string]any.

Feedback

Is there any other low-hanging fruit(s) you see when it comes to generic support in the Go driver?
Is the limitation of GetProperty a big blocker for you?
Did I miss better solutions?

Please add your voice to the discussion!