gopgext/am

import "gopgext/am"

Package am writes index access methods in Go: new kinds of indexes, like pgvector’s IVFFlat and HNSW, created with CREATE ACCESS METHOD. The extension registers a Method under the access method’s name (in main):

am.Register("ivfflat", am.Method{
	Options: []am.Option{{Name: "lists", Default: 100, Min: 1, Max: 32768}},
	OrderBy: true,  // ORDER BY col <-> query uses the index
	Build:   build,  // CREATE INDEX: reads the table's rows, writes records
	Insert:  insert, // a new row
	Vacuum:  vacuum, // removes the records of deleted rows
	Scan:    scan,   // the rows matching scan keys, or nearest first
})

and its sql/ files create the access method, with one handler function per access method (all with this package’s C symbol), and its operator classes:

CREATE FUNCTION ivfflat_handler(internal) RETURNS index_am_handler
	AS 'MODULE_PATHNAME', 'gopgext_am_handler' LANGUAGE C;
CREATE ACCESS METHOD ivfflat TYPE INDEX HANDLER ivfflat_handler;
CREATE OPERATOR CLASS vector_l2_ops FOR TYPE vector USING ivfflat AS
	OPERATOR 1 <-> (vector, vector) FOR ORDER BY float_ops;

Storage: an Index is a meta blob (Meta, SetMeta) and chains of records (NewChain, Append, Records, Get, Update, Delete): gopgext keeps them in the index’s pages, WAL-logged (so they survive crashes and replicate). Each of those operations is atomic. Modifications only happen in Build, Insert and Vacuum, which never run concurrently for one index (they are serialized by a lock), while scans run concurrently with them and see each operation either entirely or not at all. A record holds at most about 8,100 bytes, the meta blob about 8,000.

Go errors (and panics) become PostgreSQL errors, as in SQL functions.

Index#

Functions#

func Register#

func Register(name string, m Method)

Register registers the access method named name (call it in main).

Types#

type ChainID#

type ChainID int64

ChainID identifies a chain of records.

type Index#

type Index struct {
	// contains filtered or unexported fields
}

Index is an index of the access method.

func (*Index) Append#

func (idx *Index) Append(chain ChainID, b []byte) RecordID

Append adds a record at the end of a chain.

func (*Index) Columns#

func (idx *Index) Columns() int

Columns returns the number of key columns.

func (*Index) Delete#

func (idx *Index) Delete(id RecordID)

Delete deletes a record.

func (*Index) Get#

func (idx *Index) Get(id RecordID) []byte

Get returns a copy of a record.

func (*Index) ID#

func (idx *Index) ID() int64

ID identifies the index’s storage: it changes when the index is rebuilt (REINDEX, TRUNCATE), so it can key caches of its contents.

func (*Index) Meta#

func (idx *Index) Meta() []byte

Meta returns the meta blob (empty in a new index).

func (*Index) Name#

func (idx *Index) Name() string

Name returns the access method’s name.

func (*Index) NewChain#

func (idx *Index) NewChain() ChainID

NewChain creates an empty chain of records.

func (*Index) OpFamily#

func (idx *Index) OpFamily(col int) string

OpFamily returns the name of the operator family of column col (0-based): its operator class’s name, unless created with FAMILY.

func (*Index) Option#

func (idx *Index) Option(name string) int

Option returns the value of an option of the index.

func (*Index) Records#

func (idx *Index) Records(chain ChainID) iter.Seq2[RecordID, []byte]

Records returns the records of a chain, in order (copies). Records appended while ranging may or may not be seen.

func (*Index) SetMeta#

func (idx *Index) SetMeta(b []byte)

SetMeta replaces the meta blob.

func (*Index) Update#

func (idx *Index) Update(id RecordID, b []byte)

Update replaces a record with one of the same size.

type Method#

type Method struct {
	Options []Option
	// SupportFunctions is the number of support functions operator
	// classes can declare (FUNCTION n).
	SupportFunctions int
	// OrderBy: scans can return rows ordered by an operator (ORDER BY col
	// <-> value): operator classes declare it FOR ORDER BY.
	OrderBy bool
	// MultiColumn: indexes can have several columns.
	MultiColumn bool

	// Build fills a new, empty index with the rows of the table, and
	// returns the number of index entries.
	Build func(idx *Index, rows Rows) (int, error)
	// Insert indexes a new row.
	Insert func(idx *Index, tid TID, values []Value) error
	// Vacuum removes the entries of the rows for which dead is true, and
	// returns how many it removed and how many remain.
	Vacuum func(idx *Index, dead func(TID) bool) (removed, remaining int, err error)
	// Cost returns the fraction of the index a scan reads (for the
	// planner); nil: all of it.
	Cost func(idx *Index) float64
	// Scan starts a scan: keys are conditions, orderBys orderings.
	Scan func(idx *Index, keys, orderBys []ScanKey) (Scanner, error)
}

Method is an index access method.

type Option#

type Option struct {
	Name     string
	Desc     string
	Default  int
	Min, Max int
}

Option is an integer option of the indexes (CREATE INDEX … WITH (lists = 100)).

type RecordID#

type RecordID int64

RecordID identifies a record. IDs of deleted records may be reused.

type Result#

type Result struct {
	TID TID
	// Distances are the values of the ORDER BY operators for the row (one
	// per ordering key), in ascending order across results.
	Distances []float64
	// Recheck asks PostgreSQL to check the conditions on the row.
	Recheck bool
	// RecheckOrder means Distances are lower bounds: PostgreSQL computes
	// the exact ones (with the operators) and reorders.
	RecheckOrder bool
	// NullDistance: the distances are NULL (a NULL value, or a NULL
	// query): such rows come last.
	NullDistance bool
}

Result is a row a scan returns.

type Rows#

type Rows = iter.Seq2[TID, []Value]

Rows are the rows of the table being indexed (Build): their TID and the values of the index’s columns (NULL ones included). Ranging over it scans the table; it can be scanned more than once.

type ScanKey#

type ScanKey struct {
	Column   int // 0-based
	Strategy int
	Arg      Value
}

ScanKey is a condition (WHERE col op Arg) or an ordering (ORDER BY col op Arg) of a scan: Strategy is the operator’s number in the operator class.

type Scanner#

type Scanner interface {
	Next() (r Result, ok bool, err error)
}

Scanner returns the results of a scan, one at a time, until ok is false.

func Results#

func Results(rs []Result) Scanner

Results returns a Scanner over rs.

type TID#

type TID struct {
	Block  uint32
	Offset uint16
}

TID identifies a table row (a heap tuple).

func TIDFromBytes#

func TIDFromBytes(b []byte) TID

TIDFromBytes is the inverse of TID.Bytes.

func (TID) Bytes#

func (t TID) Bytes() []byte

Bytes returns t as 6 bytes, to store it in a record.

type Value#

type Value struct {
	// contains filtered or unexported fields
}

Value is an indexed value or the argument of a scan key.

func (Value) Bytes#

func (v Value) Bytes() []byte

Bytes returns a copy of the contents of a variable-length value.

func (Value) Datum#

func (v Value) Datum() uint64

Datum returns the raw Datum.

func (Value) Float64#

func (v Value) Float64() float64

Float64 returns a double precision value.

func (Value) Int64#

func (v Value) Int64() int64

Int64 returns a bigint (or a smaller integer type) value.

func (Value) IsNull#

func (v Value) IsNull() bool

IsNull reports whether the value is NULL.

func (Value) View#

func (v Value) View() []byte

View returns the contents of a variable-length value (bytea, or a type stored like it), not copied: valid during the call (scan keys: until the scan ends).