Want a simple, persistent, key-value store in Go? Something handy to have in your toolbox that’s easy to use and performant? Let’s write one!

Here’s what we’d like it to do:

// everything is persisted to disk
store, err := skv.Open("/path/to/store.db")

// store a complex object without making a fuss
var info session.Info
store.Put("sess-341356", info)

// get it back later, identifying the object with a string key
store.Get("sess-341356", &info)

// delete it when we no longer need it
store.Delete("sess-341356")

// bye
store.Close()

Note that we want to directly store and retrieve Go values, much like the fmt.Print and fmt.Scan functions. That should keep things simple for the caller, by not requiring (de)serialization code for the types.

We also want the Put, Get and Delete methods to be goroutine-safe, so that the calling code has one less thing to worry about.

Of course, we also expect the store to be fast and scalable too!

Step 1: Encoders

Encoders convert in-memory objects to byte streams.

Go has a bunch of built-in encoders under the encoding package – like json and xml. There are also other popular formats like msgpack, protocol buffers and flatbuffers.

They all have their strengths, but for our needs we’ll choose gob. Gob is a built-in package (encoding/gob) that is performant and easy-to-use. You can read more about gob here, and see some of the other contenders here.

Using the gob encoding, you can encode a Go value into a byte slice like so:

// the result goes into this buffer
var buf bytes.Buffer
// make an encoder that will write into the buffer
encoder := gob.NewEncoder(&buf)
// actually encode
var obj SomeComplexType
encoder.Encode(obj)
// get the result out as a []byte
result := buf.Bytes()

Decoding is also equally easy:

// make a reader for the input (which is a []byte)
reader := bytes.NewReader(input)
// make a decoder
decoder := gob.NewDecoder(reader)
// decode it int
var obj SomeComplexType
decoder.Decode(&obj)

Step 2: Persistent Storage

Probably the most widely used embedded storage library is SQLite. It does have database/sql bindings for Go, but being a C library it needs cgo and the compile times are frustratingly high.

LevelDB and RocksDB are two other candidates, but perhaps a bit of an overkill for our use case.

We’ll settle on a pure-Go, dependency-free, memory-mapped B+tree implementation called BoltDB. It can store key-value pairs (with both keys and values being []byte), segregated into “buckets”.

Here’s how you can open the database and create a bucket:

// open the db
db, err := bolt.Open("/path/to/store", 0640, nil)

// create a bucket
db.Update(func(tx *bolt.Tx) error {
	_, err := tx.CreateBucketIfNotExists([]byte("this.is.a.bucket.name"))
	return err
})

And within a bucket, you can store key-value pairs like this:

db.Update(func(tx *bolt.Tx) error {
	err := tx.Bucket(bucketName).Put(key, value)
	return err
})

To fetch a particular record, you can seek using a cursor:

db.View(func(tx *bolt.Tx) error {
	c := tx.Bucket(bucketName).Cursor()
	if k, v := c.Seek(key); k == nil {
		// db is empty, key not found
	} else if bytes.Equal(k, key) {
		// found, use 'v'
	} else {
		// key not found
	}
	return nil
})

As you can see, BoltDB is fairly simple to use. You can read more about it here.

Step 3: Putting the pieces together

Putting together BoltDB to manage the persistent store, and using the Gob encoder to (de)serialize Go values, we get skv - the Simple Key-Value store! It works exactly like we described above:

import "github.com/rapidloop/svk"

// open the store
store, err := svk.Open("/var/lib/mywebapp/sessions.db")

// put: encodes value with gob and updates the boltdb
err := svk.Put(sessionId, info)

// get: fetches from boltdb and does gob decode
err := svk.Get(sessionId, &info)

// delete: seeks in boltdb and deletes the record
err := svk.Delete(sessionId)

// close the store
store.Close()

To import it into your project, use:

go get -u github.com/rapidloop/skv

which will import and build both skv and boltdb.

The whole code is only one file, has tests, lives in GitHub, and is MIT licensed. It even has documentation! Feel free to poke around, fork the code, and use it as you will. Happy coding!