gopgext/tsearch

Full text search dictionaries and parsers.

API reference: gopgext/tsearch.

Full text search dictionary templates and parsers in Go, and text search services for Go code. A dictionary template is registered under the name of its init function; the Dictionary it returns from the options of CREATE TEXT SEARCH DICTIONARY is the dictionary’s state (PostgreSQL caches it per backend; it’s dropped when PostgreSQL drops it, e.g. after ALTER):

type intDict struct{ maxLen int }

func (d *intDict) Lexize(word string) ([]tsearch.Lexeme, error) {
	if len(word) > d.maxLen {
		return tsearch.StopWord, nil // drop it; nil: unknown word, for the next dictionary
	}
	return []tsearch.Lexeme{{Text: word}}, nil
}

func main() {
	tsearch.RegisterDictionary("dintdict_init", func(opts []tsearch.Option) (tsearch.Dictionary, error) {
		d := &intDict{maxLen: 6}
		for _, o := range opts { ... } // o.Name, o.Value, o.Text(), o.Bool() (defGetString/defGetBoolean)
		return d, nil
	})
}
CREATE FUNCTION dintdict_init(internal) RETURNS internal
	AS 'MODULE_PATHNAME', 'gopgext_tsearch_init' LANGUAGE C STRICT;
CREATE FUNCTION dintdict_lexize(internal, internal, internal, internal) RETURNS internal
	AS 'MODULE_PATHNAME', 'gopgext_tsearch_lexize' LANGUAGE C STRICT;
CREATE TEXT SEARCH TEMPLATE intdict_template (LEXIZE = dintdict_lexize, INIT = dintdict_init);
APIPurpose
RegisterDictionary(init, f)a dictionary template (symbols gopgext_tsearch_init, gopgext_tsearch_lexize)
Lexeme{Text, Variant, Flags}, Prefix, Filter, AddPos, StopWordLexize results: variants (nvariant), TSL_* flags; nil is an unknown word, an empty slice a stop word
RegisterParser(start, lextype, p)a parser: TokenTypes() and Parse(text) returning a Tokenizer (Next() (type, token), 0 at the end); symbols gopgext_tsearch_start, _gettoken, _end, _lextype (headlines: prsd_headline)
ConfigFile(name, ext), ReadConfigFile(path)data files in $SHAREDIR/tsearch_data (get_tsearch_config_filename, tsearch_readline: lines in the server encoding, a per-line error for characters it lacks, the “line N of configuration file” context while reading)
Lexize(dict, word), FindDictionary(name)call any dictionary (like ts_lexize); the dictionary name in the schema of the SQL function being called
Lower(s), CharLen(s)lower-casing as text search does (str_tolower, default collation); character length in the server encoding

A project’s tsearch_data/ files (rules, synonyms, stop words) are installed into $SHAREDIR/tsearch_data by gopgext install. Dictionaries don’t get the thesaurus’ multi-word state (DictSubState). examples/textsearch has a parser and a compound-word dictionary; contrib/dict_int, contrib/dict_xsyn and contrib/unaccent are ports of the contrib dictionaries.