Zen: GiST Algo

GiST README

Introduction

GiST stands for Generalized Search Tree. It was introduced in seminal paper "Generalized Search Trees for Database Systems", 1995,Joseph M. Hellerstein,Jeffrey F. Naughton,Avi Pfeffer (PS, 390Kb) and implemented by J. Hellerstein and P.Aoki in an early version of PostgreSQL ( more details is available from The GiST Indexing Project at Berkeley). As an "university" project it had a limited number of features and was in rare use. Since 7.1 version of PostgreSQL the GiST was taken up by Oleg Bartunov and Teodor Sigaev, see GiST development for information about current development.

Current implementation of GiST supports:

Variable length keys
Composite keys (multi-key)
provides NULL-safe interface to GiST core
Concurrency (as of 8.1 version)
Recovery support via WAL logging (as of PostgreSQL 8.1)

GiST is used in several public projects, like PosGIS project, which adds support for geographic objects to the PostgreSQL, OpenFTS - Open Source Full Text Search engine, which provides online indexing of data and relevance ranking for database searching. Check list of popular GiST based extensions.

Algorithms

This document describes algorithms implemented in GiST PostgreSQL v.8.1X

Source files are located under src/backend/access/gist and header files - under src/include/access in PostgreSQL source tree.

Algoritms implemented in PostgreSQL were developed following paper "Access Methods for Next-Generation Database Systems" by Marcel Kornaker (PDF.gz).

The original algorithms were modified by following reasons:

They should be adapted to PostgreSQL conventions. For example, SEARCH algorithm was considerably changed, because in PostgreSQL function search should return one tuple (next), not all tuples at once. Also, it should release page locks between calls.
since we added support of variable length keys, it's not possible to guarantee enough free space for all keys on pages after splitting. User defined function picksplit doesn't have information about size of tuples (each tuple may contain several keys as in multicolumn index while picksplit could work with only one key ) and pages.
We modified original INSERT algorithm for perfomance reason. In particularly, it's single-pass algorithm.
Since the paper were theoretical, some details were omited and we have to find out ourself how to solve some specific problems.

Because of the above reasons, we have to revised interaction of GiST CORE and PostgreSQL WAL system. Moreover, we encountered (and solved) a problem of uncompleted insertions when recovering after crash, which was not touched in the paper.

SEARCH algorithm

Function gettuple finds a tuple, which satisfies the search predicate. It store their state and returns next tuple under subsequent calls. Stack contains page, its LSN and LSN of parent page and currentposition is saved between calls.

gettuple(search-pred)
	if ( firsttime )
		push(stack, [root, 0, 0]) // page, LSN, parentLSN
		currentposition=0
	end
	top = top of stack
	while(true)
		latch( top->page, S-mode )
		if ( top->page->lsn != top->lsn ) 
			top->lsn = top->page->lsn
			currentposition=0
			if ( top->parentlsn < top->page->nsn )
				add to stack rightlink
		else
			currentposition++
		end

		while(true)
			currentposition = find_first_match( currentposition )
			if ( currentposition is invalid )
				unlatch( top->page )
				pop stack
				top = top of stack
				break loop
			else if ( top->page is leaf )
				unlatch( top->page )
				return tuple
			else 
				add to stack child page
			end
			currentposition++
		end
	end

INSERT ALGORITHM

INSERT guarantees that the GiST tree remains balanced. User defined key method Penalty is used for choosing a subtree to insert; method PickSplit is used for the node splitting algorithm; method Union is used for propagating changes upward to maintain the tree properties.

NOTICE: We modified original INSERT algorithm for perfomance reason. In particularly, it's now single-pass algorithm.

Function findLeaf is used to identify subtree for insertion. Page, in which insertion is proceeded, is locked as well as its parent page. Functions findParent and findPath are used to find parent pages, which could be changed because of concurrent access. Function pageSplit is reccurrent and could split page by more than 2 pages, which could be necessary if keys have different lengths or more than one key are inserted (in such situation, user defined function pickSplit cannot guarantee free space on page).

findLeaf(new-key)
	push(stack, [root, 0]) //page, LSN
	while(true)
		top = top of stack
		latch( top->page, S-mode )
		top->lsn = top->page->lsn
		if (  top->parent->lsn < top->page->nsn )
			unlatch( top->page )
			pop stack
		else if ( top->page is not leaf )
			push( stack, [get_best_child(top->page, new-key), 0] )
			unlatch( top->page )
		else
			unlatch( top->page )
			latch( top->page, X-mode )
			if ( top->page is not leaf )
				unlatch( top->page )
				pop stack
			else if ( top->parent->lsn < top->page->nsn )
				unlatch( top->page )
				pop stack
			else
				break loop
			end
		end
	end

findPath( stack item )
	push stack, [root, 0, 0] // page, LSN, parent 
	while( stack )
		top = top of stack
		latch( top->page, S-mode )
		if ( top->parent->page->lsn < top->page->nsn )
			push stack, [ top->page->rightlink, 0, top->parent ]
		end
		for( each tuple on page )
			if ( tuple->pagepointer == item->page )
				break loop
			else
				add to stack at the end [tuple->pagepointer,0, top]
			end
		end
		unlatch( top->page )
		pop stack
	end
	
findParent( stack item )
	parent = item->parent
	latch( parent->page, X-mode )
	if ( parent->page->lsn != parent->lsn )
		while(true) 
			search parent tuple on parent->page, if found the return
			rightlink = parent->page->rightlink
			unlatch( parent->page )
			if ( rightlink is incorrect )
				break loop
			end
			parent->page = rightlink
			latch( parent->page, X-mode )
		end
		findPath( item->parent )
		replace part of stack to new one
		findParent( item )
	end

pageSplit(page, allkeys)
	(lkeys, rkeys) = pickSplit( allkeys )
	if ( page is root )
		lpage = new page
	else
		lpage = page
	rpage = new page
	if ( no space left on rpage )
		newkeys = pageSplit( rpage, rkeys )
	else
		push newkeys, union(rkeys)
	end
	if ( no space left on lpage )
		push newkeys, pageSplit( lpage, lkeys )
	else
		push newkeys, union(lkeys)
	end
	return newkeys


placetopage(page, keysarray)
	if ( no space left on page )
		keysarray = pageSplit(page, [ extract_keys(page), keysarray])
		last page in chain gets old NSN,
		original and others - new NSN from current LSN
		if ( page is root )
			make new root with keysarray
		end
	else
		put keysarray on page
		if ( length of keysarray > 1 )
			keysarray = [ union(keysarray) ]
		end
	end
	
insert(new-key)
	findLeaf(new-key)
	keysarray = [new-key]
	top = top of stack
	while(true)
		findParent( top )
		keysarray = placetopage(top->page, keysarray)
		unlatch( top->page )
		pop stack;
		top = top of stack
		if (length of keysarray == 1)
			newboundingkey = union(oldboundingkey, keysarray)
			if (newboundingkey == oldboundingkey)
				unlatch top->page
				break loop
			end
		end
	end

GiST Algo

Projects

Links

GiST README

Introduction

Algorithms

SEARCH algorithm

INSERT ALGORITHM

References