3 Diskv (disk-vee) is a simple, persistent key-value store written in the Go
4 language. It starts with an incredibly simple API for storing arbitrary data on
5 a filesystem by key, and builds several layers of performance-enhancing
6 abstraction on top. The end result is a conceptually simple, but highly
7 performant, disk-backed storage system.
9 [![Build Status][1]][2]
11 [1]: https://drone.io/github.com/peterbourgon/diskv/status.png
12 [2]: https://drone.io/github.com/peterbourgon/diskv/latest
17 Install [Go 1][3], either [from source][4] or [with a prepackaged binary][5].
21 $ go get github.com/peterbourgon/diskv
24 [3]: http://golang.org
25 [4]: http://golang.org/doc/install/source
26 [5]: http://golang.org/doc/install
36 "github.com/peterbourgon/diskv"
40 // Simplest transform function: put all the data files into the base dir.
41 flatTransform := func(s string) []string { return []string{} }
43 // Initialize a new diskv store, rooted at "my-data-dir", with a 1MB cache.
44 d := diskv.New(diskv.Options{
45 BasePath: "my-data-dir",
46 Transform: flatTransform,
47 CacheSizeMax: 1024 * 1024,
50 // Write three bytes to the key "alpha".
52 d.Write(key, []byte{'1', '2', '3'})
54 // Read the value back out of the store.
55 value, _ := d.Read(key)
56 fmt.Printf("%v\n", value)
58 // Erase the key+value from the store (and the disk).
63 More complex examples can be found in the "examples" subdirectory.
70 At its core, diskv is a map of a key (`string`) to arbitrary data (`[]byte`).
71 The data is written to a single file on disk, with the same name as the key.
72 The key determines where that file will be stored, via a user-provided
73 `TransformFunc`, which takes a key and returns a slice (`[]string`)
74 corresponding to a path list where the key file will be stored. The simplest
78 func SimpleTransform (key string) []string {
83 will place all keys in the same, base directory. The design is inspired by
84 [Redis diskstore][6]; a TransformFunc which emulates the default diskstore
85 behavior is available in the content-addressable-storage example.
87 [6]: http://groups.google.com/group/redis-db/browse_thread/thread/d444bc786689bde9?pli=1
89 **Note** that your TransformFunc should ensure that one valid key doesn't
90 transform to a subset of another valid key. That is, it shouldn't be possible
91 to construct valid keys that resolve to directory names. As a concrete example,
92 if your TransformFunc splits on every 3 characters, then
95 d.Write("abcabc", val) // OK: written to <base>/abc/abc/abcabc
96 d.Write("abc", val) // Error: attempted write to <base>/abc/abc, but it's a directory
99 This will be addressed in an upcoming version of diskv.
101 Probably the most important design principle behind diskv is that your data is
102 always flatly available on the disk. diskv will never do anything that would
103 prevent you from accessing, copying, backing up, or otherwise interacting with
104 your data via common UNIX commandline tools.
108 An in-memory caching layer is provided by combining the BasicStore
109 functionality with a simple map structure, and keeping it up-to-date as
110 appropriate. Since the map structure in Go is not threadsafe, it's combined
111 with a RWMutex to provide safe concurrent access.
115 diskv is a key-value store and therefore inherently unordered. An ordering
116 system can be injected into the store by passing something which satisfies the
117 diskv.Index interface. (A default implementation, using Google's
118 [btree][7] package, is provided.) Basically, diskv keeps an ordered (by a
119 user-provided Less function) index of the keys, which can be queried.
121 [7]: https://github.com/google/btree
123 ## Adding compression
125 Something which implements the diskv.Compression interface may be passed
126 during store creation, so that all Writes and Reads are filtered through
127 a compression/decompression pipeline. Several default implementations,
128 using stdlib compression algorithms, are provided. Note that data is cached
129 compressed; the cost of decompression is borne with each Read.
133 diskv also now provides ReadStream and WriteStream methods, to allow very large
134 data to be handled efficiently.
139 * Needs plenty of robust testing: huge datasets, etc...
140 * More thorough benchmarking
141 * Your suggestions for use-cases I haven't thought of