dets - A Disk Based Term Storage

The module dets provides a term storage on file. The stored terms, in this module called objects, are tuples such that one element is defined to be the key. A Dets table is a collection of objects with the key at the same position stored on a file.

Dets is used by the Mnesia application, and is provided as is for users who are interested in an efficient storage of Erlang terms on disk only. Many applications just need to store some terms in a file. Mnesia adds transactions, queries, and distribution. The size of Dets files cannot exceed 2 GB. If larger tables are needed, Mnesia's table fragmentation can be used.

There are three types of Dets tables: set, bag and duplicate_bag. A table of type set has at most one object with a given key. If an object with a key already present in the table is inserted, the existing object is overwritten by the new object. A table of type bag has zero or more different objects with a given key. A table of type duplicate_bag has zero or more possibly equal objects with a given key.

Dets tables must be opened before they can be updated or read, and when finished they must be properly closed. If a table has not been properly closed, Dets will automatically repair the table. This can take a substantial time if the table is large. A Dets table is closed when the process which opened the table terminates. If several Erlang processes (users) open the same Dets table, they will share the table. The table is properly closed when all users have either terminated or closed the table. Dets tables are not properly closed if the Erlang runtime system is terminated abnormally.

Note:

access() = read | read_write
auto_save() = infinity | int()
bindings_cont() = tuple()
bool() = true | false
file() = string()
int() = integer() >= 0
keypos() = integer() >= 1
name() = atom() | ref()
no_slots() = integer() >= 0 | default
object() = tuple()
object_cont() = tuple()
select_cont() = tuple()
type() = bag | duplicate_bag | set
version() = 8 | 9 | default

all() -> [Name]

Types

Name = name()

Returns a list of the names of all open tables on this node.

bchunk(Name, Continuation) -> {Continuation2, Data} | '$end_of_table' | {error, Reason}

Types

Name = name()

Continuation = start | cont()

Continuation2 = cont()

Data = binary() | tuple()

Returns a list of objects stored in a table. The exact representation of the returned objects is not public. The lists of data can be used for initializing a table by giving the value bchunk to the format option of the init_table/3 function. The Mnesia application uses this function for copying open tables.

Unless the table is protected using safe_fixtable/2, calls to bchunk/2 may not work as expected if concurrent updates are made to the table.

The first time bchunk/2 is called, an initial continuation, the atom start, must be provided.

The bchunk/2 function returns a tuple {Continuation2, Data}, where Data is a list of objects. Continuation2 is another continuation which is to be passed on to a subsequent call to bchunk/2. With a series of calls to bchunk/2 it is possible to extract all objects of the table.

bchunk/2 returns '$end_of_table' when all objects have been returned, or {error, Reason} if an error occurs.

close(N) -> ok | {error, Reason}

Types

Name = name()

Closes a table. Only processes that have opened a table are allowed to close it.

All open tables must be closed before the system is stopped. If an attempt is made to open a table which has not been properly closed, Dets automatically tries to repair the table.

delete(Name, Key) -> ok | {error, Reason}

Types

Name = name()

Deletes all objects with the key Key from the table Name.

delete_all_objects(N) -> ok | {error, Reason}

Types

Name = name()

Deletes all objects from a table in almost constant time. However, if the table if fixed, delete_all_objects(T) is equivalent to match_delete(T, '_').

delete_object(Name, Object) -> ok | {error, Reason}

Types

Name = name()

Object = object()

Deletes all instances of a given object from a table. If a table is of type bag or duplicate_bag, the delete/2 function cannot be used to delete only some of the objects with a given key. This function makes this possible.

first(N) -> Key | '$end_of_table'

Types

Key = term()

Name = name()

Returns the first key stored in the table Name according to the table's internal order, or '$end_of_table' if the table is empty.

Unless the table is protected using safe_fixtable/2, subsequent calls to next/2 may not work as expected if concurrent updates are made to the table.

Should an error occur, the process is exited with an error tuple {error, Reason}. The reason for not returning the error tuple is that it cannot be distinguished from a key.

There are two reasons why first/1 and next/2 should not be used: they are not very efficient, and they prevent the use of the key '$end_of_table' since this atom is used to indicate the end of the table. If possible, the match, match_object, and select functions should be used for traversing tables.

foldl(Function, Acc0, Name) -> Acc1 | {error, Reason}

Types

Function = fun(Object, AccIn) -> AccOut

Acc0 = Acc1 = AccIn = AccOut = term()

Name = name()

Object = object()

Calls Function on successive elements of the table Name together with an extra argument AccIn. The order in which the elements of the table are traversed is unspecified. Function must return a new accumulator which is passed to the next call. Acc0 is returned if the table is empty.

foldr(Function, Acc0, Name) -> Acc1 | {error, Reason}

Types

Function = fun(Object, AccIn) -> AccOut

Acc0 = Acc1 = AccIn = AccOut = term()

Name = name()

Object = object()

Calls Function on successive elements of the table Name together with an extra argument AccIn. The order in which the elements of the table are traversed is unspecified. Function must return a new accumulator which is passed to the next call. Acc0 is returned if the table is empty.

from_ets(Name, EtsTab) -> ok | {error, Reason}

Types

Name = name()

EtsTab = - see ets(3) -

Deletes all objects of the table Name and then inserts all the objects of the Ets table EtsTab. The order in which the objects are inserted is not specified. Since ets:safe_fixtable/2 is called the Ets table must be public or owned by the calling process.

info(N) -> InfoList | undefined

Types

Name = name()

InfoList = [{Item, Value}]

Returns information about the table Name as a list of {Item, Value} tuples:

*

{file_size, int()}, the size of the file in bytes.

*

{filename, file()}, the name of the file where objects are stored.

*

{keypos, keypos()}, the position of the key.

*

{size, int()}, the number of objects stored in the table.

*

{type, type()}, the type of the table.

info(Name, Item) -> Value | undefined

Types

Name = name()

Returns the information associated with Item for the table Name. In addition to the {Item, Value} pairs defined for info/1, the following items are allowed:

*

{access, access()}, the access mode.

*

{auto_save, auto_save()}, the auto save interval.

*

{bchunk_format, binary()}, an opaque binary describing the format of the objects returned by bchunk/2. The binary can be used as argument to is_compatible_chunk_format/2. Only available for version 9 tables.

*

{hash, Hash}. Describes which BIF is used to calculate the hash values of the objects stored in the Dets table. Possible values of Hash are hash, which implies that the erlang:hash/2 BIF is used, phash, which implies that the erlang:phash/2 BIF is used, and phash2, which implies that the erlang:phash2/1 BIF is used.

*

{memory, int()}, the size of the file in bytes. The same value is associated with the item file_size.

*

{no_keys, int()}, the number of different keys stored in the table. Only available for version 9 tables.

*

{no_objects, int()}, the number of objects stored in the table.

*

{no_slots, {Min, Used, Max}}, the number of slots of the table. Min is the minimum number of slots, Used is the number of currently used slots, and Max is the maximum number of slots. Only available for version 9 tables.

*

{owner, pid()}, the pid of the process that handles requests to the Dets table.

*

{ram_file, bool()}, whether the table is kept in RAM.

*

{safe_fixed, SafeFixed}. If the table is fixed, SafeFixed is a tuple {FixedAtTime, [{Pid, RefCount}]}. FixedAtTime is the time when the table was first fixed, and Pid is the pid of the process that fixes the table RefCount times. There may be any number of processes in the list. If the table is not fixed, SafeFixed is the atom false.

*

{version, int()}, the version of the format of the table.

init_table(Name, InitFun [, Options]) -> ok | {error, Reason}

Types

Name = atom()

InitFun = fun(Arg) -> Res

Arg = read | close

Res = end_of_input | {[object()], InitFun} | {Data, InitFun} | term()

Data = binary() | tuple()

Replaces the existing objects of the table Name with objects created by calling the input function InitFun, see below. The reason for using this function rather than calling insert/2 is that of efficiency. It should be noted that the input functions are called by the process that handles requests to the Dets table, not by the calling process.

When called with the argument read the function InitFun is assumed to return end_of_input when there is no more input, or {Objects, Fun}, where Objects is a list of objects and Fun is a new input function. Any other value Value is returned as an error {error, {init_fun, Value}}. Each input function will be called exactly once, and should an error occur, the last function is called with the argument close, the reply of which is ignored.

If the type of the table is set and there is more than one object with a given key, one of the objects is chosen. This is not necessarily the last object with the given key in the sequence of objects returned by the input functions. Extra objects should be avoided, or the file will be unnecessarily fragmented. This holds also for duplicated objects stored in tables of type duplicate_bag.

It is important that the table has a sufficient number of slots for the objects. If not, the hash list will start to grow when init_table/2 returns which will significantly slow down access to the table for a period of time. The minimum number of slots is set by the open_file/2 option min_no_slots and returned by the info/2 item no_slots. See also the min_no_slots option below.

The Options argument is a list of {Key, Val} tuples where the following values are allowed:

*

{min_no_slots, no_slots()}. Specifies the estimated number of different keys that will be stored in the table. The open_file option with the same name is ignored unless the table is created, and in that case performance can be enhanced by supplying an estimate when initializing the table.

*

{format, Format}. Specifies the format of the objects returned by the function InitFun. If Format is term (the default), InitFun is assumed to return a list of tuples. If Format is bchunk, InitFun is assumed to return Data as returned by bchunk/2. This option overrides the min_no_slots option.

insert(Name, Objects) -> ok | {error, Reason}

Types

Name = name()

Objects = object() | [object()]

Inserts one or more objects into the table Name. If there already exists an object with the same key as some of the given objects and the table type is set, the old object will be replaced.

insert_new(Name, Objects) -> Bool

Types

Name = name()

Objects = object() | [object()]

Bool = bool()

Inserts one or more objects into the table Name. If there already exists an object with the same key as some of the given objects the table is not updated and false is returned, otherwise the objects are inserted and true returned.

is_compatible_bchunk_format(Name, BchunkFormat) -> Bool

Types

Name = name()

BchunkFormat = binary()

Bool = bool()

Returns true if it would be possible to initialize the table Name, using init_table/3 with the option {format,  bchunk}, with objects read with bchunk/2 from some table T such that calling info(T,  bchunk_format) returns BchunkFormat.

is_dets_file(FileName) -> Bool | {error, Reason}

Types

FileName = file()

Bool = bool()

Returns true if the file FileName is a Dets table, false otherwise.

lookup(Name, Key) -> [Object] | {error, Reason}

Types

Key = term()

Name = name()

Object = object()

Returns a list of all objects with the key Key stored in the table Name. For example:

2>dets:open_file(abc, [{type, bag}]).

{ok,abc} 3>dets:insert(abc, {1,2,3}).

ok 4>dets:insert(abc, {1,3,4}).

ok 5>dets:lookup(abc, 1).

[{1,2,3},{1,3,4}]

If the table is of type set, the function returns either the empty list or a list with one object, as there cannot be more than one object with a given key. If the table is of type bag or duplicate_bag, the function returns a list of arbitrary length.

Note that the order of objects returned is unspecified. In particular, the order in which objects were inserted is not reflected.

match(Continuation) -> {[Match], Continuation2} | '$end_of_table' | {error, Reason}

Types

Continuation = Continuation2 = bindings_cont()

Match = [term()]

Matches some objects stored in a table and returns a list of the bindings that match a given pattern in some unspecified order. The table, the pattern, and the number of objects that are matched are all defined by Continuation, which has been returned by a prior call to match/1 or match/3.

When all objects of the table have been matched, '$end_of_table' is returned.

match(Name, Pattern) -> [Match] | {error, Reason}

Types

Name = name()

Pattern = tuple()

Match = [term()]

Returns for each object of the table Name that matches Pattern a list of bindings in some unspecified order. See ets(3) for a description of patterns. If the keypos'th element of Pattern is unbound, all objects of the table are matched. If the keypos'th element is bound, only the objects with the right key are matched.

match(Name, Pattern, N) -> {[Match], Continuation} | '$end_of_table' | {error, Reason}

Types

Name = name()

Pattern = tuple()

N = default | int()

Match = [term()]

Continuation = bindings_cont()

Matches some or all objects of the table Name and returns a list of the bindings that match Pattern in some unspecified order. See ets(3) for a description of patterns.

A tuple of the bindings and a continuation is returned, unless the table is empty, in which case '$end_of_table' is returned. The continuation is to be used when matching further objects by calling match/1.

If the keypos'th element of Pattern is bound, all objects of the table are matched. If the keypos'th element is unbound, all objects of the table are matched, N objects at a time. The default, indicated by giving N the value default, is to let the number of objects vary depending on the sizes of the objects. If Name is a version 9 table, all objects with the same key are always matched at the same time which implies that more than N objects may sometimes be matched.

The table should always be protected using safe_fixtable/2 before calling match/3, or errors may occur when calling match/1.

match_delete(Name, Pattern) -> N | {error, Reason}

Types

Name = name()

N = int()

Pattern = tuple()

Deletes all objects that match Pattern from the table Name, and returns the number of deleted objects. See ets(3) for a description of patterns.

If the keypos'th element of Pattern is bound, only the objects with the right key are matched.

match_object(Continuation) -> {[Object], Continuation2} | '$end_of_table' | {error, Reason}

Types

Continuation = Continuation2 = object_cont()

Object = object()

Returns a list of some objects stored in a table that match a given pattern in some unspecified order. The table, the pattern, and the number of objects that are matched are all defined by Continuation, which has been returned by a prior call to match_object/1 or match_object/3.

When all objects of the table have been matched, '$end_of_table' is returned.

match_object(Name, Pattern) -> [Object] | {error, Reason}

Types

Name = name()

Pattern = tuple()

Object = object()

Returns a list of all objects of the table Name that match Pattern in some unspecified order. See ets(3) for a description of patterns.

If the keypos'th element of Pattern is unbound, all objects of the table are matched. If the keypos'th element of Pattern is bound, only the objects with the right key are matched.

Using the match_object functions for traversing all objects of a table is more efficient than calling first/1 and next/2 or slot/2.

match_object(Name, Pattern, N) -> {[Object], Continuation} | '$end_of_table' | {error, Reason}

Types

Name = name()

Pattern = tuple()

N = default | int()

Object = object()

Continuation = object_cont()

Matches some or all objects stored in the table Name and returns a list of the objects that match Pattern in some unspecified order. See ets(3) for a description of patterns.

A list of objects and a continuation is returned, unless the table is empty, in which case '$end_of_table' is returned. The continuation is to be used when matching further objects by calling match_object/1.

If the keypos'th element of Pattern is bound, all objects of the table are matched. If the keypos'th element is unbound, all objects of the table are matched, N objects at a time. The default, indicated by giving N the value default, is to let the number of objects vary depending on the sizes of the objects. If Name is a version 9 table, all matching objects with the same key are always returned in the same reply which implies that more than N objects may sometimes be returned.

The table should always be protected using safe_fixtable/2 before calling match_object/3, or errors may occur when calling match_object/1.

member(Name, Key) -> Bool | {error, Reason}

Types

Name = name()

Key = term()

Bool = bool()

Works like lookup/2, but does not return the objects. The function returns true if one or more elements of the table has the key Key, false otherwise.

next(Name, Key1) -> Key2 | '$end_of_table'

Types

Name = name()

Key1 = Key2 = term()

Returns the key following Key1 in the table Name according to the table's internal order, or '$end_of_table' if there is no next key.

Should an error occur, the process is exited with an error tuple {error, Reason}.

Use first/1 to find the first key in the table.

open_file(Filename) -> {ok, Reference} | {error, Reason}

Types

FileName = file()

Reference = ref()

Opens an existing table. If the table has not been properly closed, the error {error, need_repair} is returned. The returned reference is to be used as the name of the table. This function is most useful for debugging purposes.

open_file(Name, Args) -> {ok, Name} | {error, Reason}

Types

Name = atom()

Opens a table. An empty Dets table is created if no file exists.

The atom Name is the name of the table. The table name must be provided in all subsequent operations on the table. The name can be used by other processes as well, and several process can share one table.

If two processes open the same table by giving the same name and arguments, then the table will have two users. If one user closes the table, it still remains open until the second user closes the table.

The Args argument is a list of {Key, Val} tuples where the following values are allowed:

*

{access, access()}. It is possible to open existing tables in read-only mode. A table which is opened in read-only mode is not subjected to the automatic file reparation algorithm if it is later opened after a crash. The default value is read_write.

*

{auto_save, auto_save()}, the auto save interval. If the interval is an integer Time, the table is flushed to disk whenever it is not accessed for Time milliseconds. A table that has been flushed will require no reparation when reopened after an uncontrolled emulator halt. If the interval is the atom infinity, auto save is disabled. The default value is 180000 (3 minutes).

*

{estimated_no_objects, int()}. Equivalent to the min_no_slots option.

*

{file, file()}, the name of the file to be opened. The default value is the name of the table.

*

{max_no_slots, no_slots()}, the maximum number of slots that will be used. The default value is 2 M, and the maximal value is 32 M. Note that a higher value may increase the fragmentation of the table, and conversely, that a smaller value may decrease the fragmentation, at the expense of execution time. Only available for version 9 tables.

*

{min_no_slots, no_slots()}. Application performance can be enhanced with this flag by specifying, when the table is created, the estimated number of different keys that will be stored in the table. The default value as well as the minimum value is 256.

*

{keypos, keypos()}, the position of the element of each object to be used as key. The default value is 1. The ability to explicitly state the key position is most convenient when we want to store Erlang records in which the first position of the record is the name of the record type.

*

{ram_file, bool()}, whether the table is to be kept in RAM. Keeping the table in RAM may sound like an anomaly, but can enhance the performance of applications which open a table, insert a set of objects, and then close the table. When the table is closed, its contents are written to the disk file. The default value is false.

*

{repair, Value}. Value can be either a bool() or the atom force. The flag specifies whether the Dets server should invoke the automatic file reparation algorithm. The default is true. If false is specified, there is no attempt to repair the file and {error, need_repair} is returned if the table needs to be repaired.

The value force means that a reparation will take place even if the table has been properly closed. This is how to convert tables created by older versions of STDLIB. An example is tables hashed with the deprecated erlang:hash/2 BIF. Tables created with Dets from a STDLIB version of 1.8.2 and later use the erlang:phash/2 function or the erlang:phash2/1 function, which is preferred.

The repair option is ignored if the table is already open.

*

{type, type()}, the type of the table. The default value is set.

*

{version, version()}, the version of the format used for the table. The default value is 9. Tables on the format used before OTP R8 can be created by giving the value 8. A version 8 table can be converted to a version 9 table by giving the options {version, 9} and {repair, force}.

pid2name(Pid) -> {ok, Name} | undefined

Types

Name = name()

Pid = pid()

Returns the name of the table given the pid of a process that handles requests to a table, or undefined if there is no such table.

This function is meant to be used for debugging only.

repair_continuation(Continuation, MatchSpec) -> Continuation2

Types

Continuation = Continuation2 = select_cont()

MatchSpec = match_spec()

This function can be used to restore an opaque continuation returned by select/3 or select/1 if the continuation has passed through external term format (been sent between nodes or stored on disk).

The reason for this function is that continuation terms contain compiled match specifications and therefore will be invalidated if converted to external term format. Given that the original match specification is kept intact, the continuation can be restored, meaning it can once again be used in subsequent select/1 calls even though it has been stored on disk or on another node.

See also ets(3) for further explanations and examples.

Note:

1>dets:open_file(t, []),
dets:insert(t, [{1,a},{2,b},{3,c},{4,d}]),
MS = ets:fun2ms(fun({X,Y}) when (X > 1) or (X < 5) -> {Y} end),
QH1 = dets:table(t, [{traverse, {select, MS}}]).

2>QH2 = qlc:q([{Y} || {X,Y} <- dets:table(t), (X > 1) or (X < 5)]).

3>qlc:info(QH1) =:= qlc:info(QH2).

fun(X) -> io:format("~p~n", [X]), continue end.

fun(X) -> {continue, X} end.

ets(3), mnesia(3), qlc(3)

Claes Wikström - support@erlang.ericsson.se
Hans Bolinder - support@erlang.ericsson.se