implement SST for streaming store to reduce memory usage
Sorted String Table in combination with mmap() can greatly reduce the memory requirements of the Streaming Store, as no data needs to be stored for every entry.
Should start with an array of indexes of all entries in the file.
Links:
- explanation about leveled compaction (in our case the one with the most tombstones should be selected first; if equal then the largest part)
Changes needed to using store:
- domotica should store in a directory with the date in it (otherwise the whole DB is visited and memory is trashed)
ToDo:
-
merge of L0 only with overlapping parts (instead of the whole L1); -
peer sync: easy way to select last n entries in a directory (needed for syncing between hosts), possibly use a secondary index (in sstable) for this; -
initial peer sync and subscribe: easy way to resume iteration (one way is to record id of every item send to check when resuming); -
implement custom filters over objects allowing for selecting the entries that are currently valid (assuming every entry has a date in the metadata). Preferably as a generic operation (which should work over multiple items); -
incorporate timestamped entries by setting the timestamp as an extra key, for example: name
\0
timestamp
(in big-endian). Does this conflict with normal operations? Is this useful for more scenarios than timestamped mode? -
check put speed
Edited by Bernard van Gastel