The DistKV command¶
DistKV exports one command line tool, appropriately named distkv
. It
provides various sub- and sub-sub-commands to start and control your server.
Each command and sub-command accepts a distinct set of options which must be used directly after the (sub)command affected by them.
distkv¶
The main entry point for all commands.
Subcommands:
- distkv client
- distkv server
-
-v
,
--verbose
¶
Increase debugging verbosity. Broadly speaking, the default is
ERROR
; this option selectsWARNING
,INFO
orDEBUG
depending on how often you use it.
-
-q
,
--quiet
¶
Decrease debugging verbosity: the opposite of
distkv -v
, reducing the verbosity toFATAL
.
-
-l
,
--log
<source=LEVEL>
¶ You can selectively adjust debugging verbosity if you need to print, or ignore, some types of messages. Example:
-vvv --log asyncserf.actor=ERROR
would suppress most chattiness of AsyncSerf’s actor.
-
-D
,
--debug
¶
Some DistKV methods, in particular cryptographic operations, are not noted for their speed. This option reduces the bit count of these options in order to speed them up significantly.
It also reduces their security unacceptably. Thus, this option should only used while debugging.
-
-c
,
--cfg
<FILE>
¶ Specify a YAML configuration file.
Data in this file override the corresponding entries in the
distkv.defaults.CFG
directory.
-
-C
,
--conf
<location=value>
¶ Set a specific configuration value. This option takes precedence over
distkv -c
.
Run the DistKV server.
A DistKV server holds all data and syncs with all other DistKV servers. You can’t run distkv client unless you have at least one running server.
There is no separate option to set the address for clients to connect to;
use server.bind_default.port=57589
(or your own port number) to change
it from the default of 27589
, or use a configuration file.
-
-h
,
--host
<address>
¶ The Serf server’s IP address. The default is
localhost
.This option is available in the configuration file as
server.serf.host
.
-
-p
,
--port
<port>
¶ The TCP port to connect to. The Serf default is 7373.
This option is available in the configuration file as
server.serf.port
.
-
-l
,
--load
<file>
¶ Pre-load the saved data from this file into the server before starting it.
Do not use this option with an out-of-date savefile.
-
-s
,
--save
<file>
¶ Log all changes to this file. This includes the initial data.
You can later adapt this with
distkv client control save
A network of servers needs to contain some data before it becomes operational. When starting the first server, you can use an initial
-
-i
,
--init
<value>
¶ Initialize the server by storing this value in the root entry.
-
-e
,
--eval
¶
Evaluate the initial value, as a standard Python expression.
You can also use distkv client data set to update this value later.
-
name
¶
Each DistKV server requires a unique name. If you recycle a name, the old server using it will die (unless your Serf network is disjoint – in that case, one or both will terminate some random time after the networks are reconnecting, and you’ll get inconsistent data). So don’t do that.
This subcommand collects all sub-subcommand which talk to a DistKV server.
-
-h
,
--host
<address>
¶ The address to connect to. Defaults to
localhost
.This setting is also available as the
connect.host
configuration setting.
-
-p
,
--port
<port>
¶ The port to connect to. Defaults to 27586.
This setting is also available as the
connect.port
configuration setting.
-
-a
,
--auth
<params>
¶ Parameters for authorizing this client. Use
=file
to load the data from a file, ormethod data=value…
to provide them inline.The default is
_anon
, i.e. no authorization.
-
-m
,
--metadata
¶
The results of many commands will include the metadata associated with the entry or entries in question. This allows you to safely modify a value.
Basic data access.
This subcommand does not have options.
Read a DistKV value.
If you read a sub-tree recursively, be aware that the whole subtree will
be read before anything is printed. Use the watch --state
subcommand
for incremental output.
-
-r
,
--recursive
¶
Print all entries below this entry.
-
-d
,
--as-dict
<text>
¶ When you use this option, the data is printed as a dictionary. The argument of this option controls which key is used for the actual value; this string should not occur as a path element.
The customary value to use is a single underscore.
Using this option in conjunction with
--recursive
requires keeping the whole data set in memory before starting to print anything. This may take a long time or eat a lot of memory.
-
-m
,
--mindepth
<integer>
¶ When printing recursively, start at this depth off the given path.
The default is zero, i.e. include the entry itself.
-
-M
,
--maxdepth
<integer>
¶ When printing recursively, stop at this depth (inclusive).
The default is to print the whole tree. Use
1
to print the entry itself (assuming that it has a value and you didn’t use--mindepth=1
) and its immediate children.
-
path…
¶
Access the entry at this location. The default is the root node, which usually isn’t what you want.
List DistKV values.
This command is basically like distkv client data get
, except that
--recursive
and empty
are always set. mindepth
and maxdepth
default to 1.
-
-r
,
--recursive
¶
Print all entries below this entry.
-
-d
,
--as-dict
<text>
¶ When you use this option, the data is printed as a dictionary. The argument of this option controls which key is used for the actual value; this string should not occur as a path element.
The customary value to use is a single underscore.
Using this option in conjunction with
--recursive
requires keeping the whole data set in memory before starting to print anything. This may take a long time or eat a lot of memory.
-
-m
,
--mindepth
<integer>
¶ When printing recursively, start at this depth off the given path.
The default is zero, i.e. include the entry itself.
-
-M
,
--maxdepth
<integer>
¶ When printing recursively, stop at this depth (inclusive).
The default is to print the whole tree. Use
1
to print the entry itself (assuming that it has a value and you didn’t use--mindepth=1
) and its immediate children.
-
path…
¶
Access the entry at this location. The default is the root node, which usually isn’t what you want.
Store a value at some DistKV position.
If you update a value, you should use --last
(preferred) or
--prev
(if you must), to ensure that no other change collides
with yours.
When adding a new entry, use --new
to ensure that you don’t
accidentally overwrite something.
-
-v
,
--value
<value>
¶ The value to store. This option is mandatory.
-
-e
,
--eval
¶
Treat the
value
as a Python expression, to store anything that’s not a string.
-
-l
,
--last
<node> <count>
¶ The chain link which last modified this entry.
-
-n
,
--new
¶
Use this option instead of
--last
orprev
if the entry is new, or has been deleted.
-
-p
,
--prev
<value>
¶ The value which this entry needs to have in order to be affected.
Try not to use this option;
--last
is much better.This value is also affected by
--eval
.
-
path…
¶
Write to the entry at this location. The default is the root node, which usually isn’t what you want.
Delete the value at some DistKV position.
If you delete a value, you should use --last
(preferred) or
--prev
(if you must), to ensure that no other change collides
with your deletion.
Recursive changes only check the entry you mention on the command line.
-
-l
,
--last
<node> <count>
¶ The chain link which last modified this entry.
-
-e
,
--eval
¶
Treat the
value
as a Python expression, to store anything that’s not a string.
-
-p
,
--prev
<value>
¶ The value which this entry needs to have in order to be affected.
Try not to use this option;
--last
is much better.This value is also affected by
--eval
.
-
path…
¶
Write to the entry at this location. The default is the root node, which usually isn’t what you want.
Monitor changes to the state of an entry, or rather its subtree.
-
-s
,
--state
¶
Before emitting changes, emit the current state of this subtree.
A flag entry will be printed when this step is completed.
Note
The current state may already include updates, due to DistKV’s asynchonous nature. You should simply replace existing values.
-
-m
,
--msgpack
¶
Write the output as
MsgPack
data. XXX TODOThe default is to use YAML.
-
path…
¶
Monitor the subtree at this location. The default is the root node.
Stream a list of changes from standard input to DistKV.
-
-m
,
--msgpack
¶
Interpret the input as
MsgPack
data. XXX TODOThe default is to use YAML. XXX TODO
-
path…
¶
Interpret the streamed data relative to this subtree.
Control your server. XXX TODO
Control logging of changes on the server.
Set the file to log to. The old file is closed as soon as the new file is ready (i.e. the current state is saved).
-
-i
,
--incremental
¶
The save file will only contain changes, but not the current state.
-
path
¶
The file to write to. Note that this file is on the server.
Save the current state of the server to this file.
-
path
¶
The file to write to. Note that this file is on the server.
Stop logging.
Set up and change client authorization.
If you have never setup authorization, this sub-subcommands’ --help
options may not work. Use -m root
as a workaround. XXX TODO
-
-m
,
--method
<name>
¶ Affect the named method.
DistKV supports multiple authorization methods. The default is the one that has been changed to with
distkv client auth init
.If you want to do anything with authorization, you’ll need to use this flag to set up the initial users.
See `Auth `
Set up this method.
-
-s
,
--switch
¶
Actually swtich to using this method. This is the default for initial set-up.
List configured auth methods.
XXX TODO
Manage DistKV users.
Each authorization method has its own schema for validating users.
Add a new user.
Example: distkv client -a root auth -m password user add name=foo password=barbaz
The identifier which you’d use to subsequently refer to that user is printed when this command completes.
-
-a
,
--add
<key>=<value>
¶ Set an additional parameter, i.e. one that controls DistKV’s handling of that user. Used e.g. for setting the user’s ACL.
-
<key>
=<value>
¶ Set an auth-specific parameter, i.e. one that’s controlled by the auth mode.
Modify a user.
-
-a
,
--add
<key>=<value>
¶ Set or change an additional parameter, i.e.
-
<userid>
¶
The ID of the user to modify. Printed when adding or listing the user.
Check that authorizing a user works.
XXX TODO seems broken
-
options…
¶
Whichever auth options you’d normally use in
distkv client -a TYPE …
.
Dump data of a user.
-
-c
,
--chain
<int>
¶
The chain length to return, for subsequent modification.
-
ident
¶
The user identifier, as reported by
add
.
List users.
XXX TODO add verbosity
Set up DistKV’s type control: verify the data that clients write.
See Verifying and Translating Entries for details.
Retrieve a type entry.
-
-y
,
--yaml
¶
Emit the schema as YAML data. Default: JSON.
-
-s
,
--script
<filename>
¶ Save the script to this file. Default: include in the output.
-
-S
,
--schema
<filename>
¶ Save the schema to this file. Default: include in the output.
-
name…
¶
The type data to retrieve.
Add or modify a type entry.
For setting up a type, you need at least two good and one bad test value. (If there’s only one possible good value, you don’t need the entry; if there’s no bad value you don’t need the type check.)
Type checks accumulate: Both ‘good’ and ‘bad’ test values for “int percent” must be accepted by “int”.
Tests can use Python code, a JSON schema, or both. In the latter case the schema is tested first.
To modify a record, use distkv client type get <path>… > <tempfile>
, edit
the tempfile, then restore with distkv client type set -d <tempfile> <path>…
.
-
-y
,
--yaml
¶
Read the schema as YAML. Default: JSON.
-
-s
,
--script
<filename>
¶ Load the script from this file. Default: no script.
-
-S
,
--schema
<filename>
¶ Load the schema from this file. Default: no schema.
-
-g
<value>
¶ A known-good value to test the codec. It will be Python-evaluated.
-
-b
<value>
¶ A known-bad value to test the codec. It will be Python-evaluated.
-
-a
,
--all
¶
Load the complete record from the
script
file.
-
name…
¶
The type data to set.
Read, set or delete type matches, i.e. which part of your DistKV tree is constricted by which type.
-
-t
,
type
<name>
¶ The type name to use. Use multiple –type` options to access subtypes. Skip this option to display which type corresponds to the given path.
-
-d
,
--delete
¶
Delete the match record instead of printing it.
-
path…
¶
The DistKV entry to affect. Path elements ‘+’ and ‘#’ match exactly-one and one-or-more subpaths. The most specific path wins.
Set up codecs: manipulate the data that clients see, sort of like a database view.
Codecs consist of code that encodes, i.e. converts to the user’s view, and decodes, i.e. converts to the server’s storage.
Codecs cannot translate path names, or access other entries. The decoder may modify an existing entry (or rather, use the currently-stored version when assembling an entry’s new value).
Unlike types, the codec hierarchy is strictly for convenience.
Retrieve information about a codec, including its scripts.
-
-e
,
--encode
<file>
¶ The file which the encoder’s Python code is written to.
If this option is not used, the code is part of the script’s output.
-
-d
,
--decode
<file>
¶ The file which the decoder’s Python code is written to.
If this option is not used, the code is part of the script’s output.
-
-s
,
--script
<file>
¶ The YAML file to which to write any other data.
This file will also contain the scripts, if not stored otherwise.
-
<name>
¶
The name of the codec group from which this codec should be retrieved.
-
<path>
¶
The DistKV entry that would be affected. Path elements ‘+’ and ‘#’ match exactly-one and one-or-more subpaths. The most specific path wins.
Add or modify a codec.
To modify a codec, use distkv client codec get <path>… > <tempfile>
, edit the
tempfile, then restore with distkv client codec set -d <tempfile> <path>…
.
-
-e
,
--encode
<file>
¶ The file which contains the encoder’s Python code.
-
-d
,
--decode
<file>
¶ The file which contains the decoder’s Python code.
-
-i
,
--in
<source> <dest>
¶ A pair of test values for the decoding branch of the codec. Both are
eval
-uated.
-
-o
,
--out
<source> <dest>
¶ A pair of test values for the encoding branch of the codec. Both are
eval
-uated.
-
-d
,
--data
<file>
¶ The YAML file which contains any other data.
Required: two arrays “in” and “out” containing tuples with before/after values for the decoder and encoder, respectively.
You may store the scripts in this file, using “encode” and “decode” keys.
-
<name>
¶
The name of the codec group to which this codec should be saved or modified.
-
<path>
¶
The DistKV entry to affect. Path elements ‘+’ and ‘#’ match exactly-one and one-or-more subpaths. The most specific path wins.
Read, set or delete codec matches, i.e. which part of your DistKV tree is managed by which codec. To this effect, matches are tagged by a group name.
Which codec group to apply to a given user is stored in that user’s
auxiliary data as conv=NAME
. If no such entry is present, that user’s
data are not converted.
-
-c
,
--codec
¶
The codec to use on the given path. Use this option multiple times if the codec has a multi-level name.
-
<name>
¶
The name of the codec group to which this codec should be saved or modified.
-
<path>
¶
The DistKV entry to affect. Path elements ‘+’ and ‘#’ match exactly-one and one-or-more subpaths. The most specific path wins.
Manipulate access control lists stored in DistKV.
Generate a list of known ACLs.
Retrieve the flags at a specific ACL path.
If the path does not contain any flags, print -
.
-
<acl>
¶
The ACL to modify. Mandatory.
-
<path>
¶
The ACL path from which to retrieve the flags.
Set the flags at a specific ACL path.
-
-a
,
--acl
<MODES>
¶ The flag values to set. Start with
+
to add,-
to remove modes. Use an empty argument (''
) to remove all rights. A lone-
removes the entry.
-
<acl>
¶
The ACL to modify. Mandatory.
-
<path>
¶
The ACL path to add or modify.
Check whether an ACL allows access.
-
-m
,
--mode
<mode>
¶ Check this mode (single letter). The default is “x”.
-
-a
,
--acl
<acl>
¶ In addition to the user’s current ACL, also check the flag on the named ACL.
There is no indication which of the two failed. This is intentional.
-
<path>
¶
The path to check.
Dump an ACL’s content.
-
-d
,
--as-dict
TEXT
¶ Print as dictionary.
TEXT
is the key used for the ACL data.Default: Emit a list.
Using this flag requires storing the whole ACL in memory, which is usually not a problem (unlike for data).
-
<name>
¶
The name of the ACL to dump. Mandatory.
-
<path>
¶
The path to start dumping at. Default: the root.
Manipulate code stored in DistKV.
Retrieve Python code stored in the server.
-
-s
,
--script
<filename>
¶ Save the code to <filename> instead of including it in the output.
-
<path>
…
¶ Path to the code in question.
Store or replace Python code stored in the server.
This code will not run in the server; the purpose of these calls is to upload code for use by client runners.
To modify some code, use distkv client code get <path>… > <tempfile>
, edit
the tempfile, then restore with distkv client code set -d <tempfile> <path>…
.
-
-d
,
--data
<filename>
¶ Load the metadata from this file.
-
-s
,
--script
<filename>
¶ Load the code from this file.
-
-a
,
--async
¶
The code will run asynchronously, i.e. it may use
async
andawait
statements.You should only use the
anyio
module for sleeping, locking etc..
-
-t
,
--thread
¶
The code will run in a worker thread.
This option is incompatible with
--async
.
-
name…
¶
The path to the code to write.
TODO: Old versions of the code continue to run; DistKV does not yet restart users.
Manipulate modules stored in DistKV.
Modules are replaced immediately, but code using them is not auto-restarted.
This code is experimental and frankly just plain wrong: Module loading is not deferred until “import” time. This code needs sever refactoring. For now, please store modules in the file system.
Retrieve Python module stored in the server.
-
-s
,
--script
<filename>
¶ Save the code to <filename> instead of including it in the output
-
<path>
…
¶ Path to the code in question.
Store or replace Python code stored in the server.
This code will not run in the server; the purpose of these calls is to upload code for use by client-side runners.
To modify a module, use distkv client code module get <path>… > <tempfile>
, edit
the tempfile, then restore with distkv client code module set -d <tempfile> <path>…
.
-
-d
,
--data
<filename>
¶ Load the metadata from this file.
-
-s
,
--script
<filename>
¶ Load the module’s code from this file.
-
name…
¶
The path to the code to set, below
.distkv code proc
or whatever else is configured undercodes
.
TODO: Old versions of the code continue to run; DistKV does not yet restart users.
Subcommand for controlling and executing code stored in DistKV.
-
-n
,
--node
<node>
¶ The node where the code in question might run.
Not using this option means that the code in question may run on any node.
This is the actual runner, i.e. the program that runs stored tasks.
This program does not terminate.
List available run entries.
-
-s
,
--state
¶
Add the current state.
-
-S
,
--state-only
¶
Only print the current state.
-
<prefix>
…
¶ Limit listing to this prefix.
Read a runner entry.
Create or change a runner entry.
-
-c
,
--code
<code>
¶ Path to the code that this entry should execute. This value is either split by spaces or, if
--eval
is used, interpreted as a Python expression.
-
-t
,
--time
<when>
¶ Time at which the runner should fire next. Seconds in the future.
-
-r
,
--repeat
<seconds>
¶ Time after a successful execution when the runner should fire again.
-
-d
,
--delay
<seconds>
¶ Time after an unsuccessful execution when the runner should fire again.
-
-b
,
--backoff
¶
Back-off exponent. The effective delay is
delay * backoff ^ n_failures
.To retry a failure immediately, simply use
--time now
.
Subcommand for viewing and modifying the internal state of a DistKV server.
This command emits DistKV’s internal state.
The output is comparable to distkv client data dump -rd_
, but for internal
data.
-
<path>
…
¶ Path prefix for DistKV’s internal data structure.
This command queries the internal state of a DistKV server.
All lists of tick
values are sorted and consist of either single
entries, or [begin,end)
tuples, i.e. the starting value is part of the
range but the end is not.
-
-y
,
--yaml
¶
Print the result of this operation as YAML data.
-
-n
,
--nodes
¶
Add a list of known nodes and their current
tick
value.
-
-d
,
--deleted
¶
Add a list of per-node deleted
tick
values, i.e. those whose entries have been purged from the system.
-
-k
,
--known
¶
Add a list of per-node known
tick
values, i.e. those which have been superseded by subsequent changes.
-
-m
,
--missing
¶
Add a list of per-node missing
tick
values, i.e. those neither in theknown
list nor seen in any entries’ chains.
-
-r
,
--remote-missing
¶
Add a list of per-node missing
tick
values that have been requested from other servers.
See Server protocol <server_protocol> for details.
Mark ticks as known or deleted. This is used to clean up the missing
range(s) when there’s a consistency problem.
-
-d
,
--deleted
¶
Add the nodes to the
deleted
list instead of theknown
list. The effect is that if they subsequently re-surcace they’ll be ignored.
-
-b
,
--broadcast
¶
Send the changes to the whole network, not just the node you’re a client of. (The local node is still targeted first, to ensure that if your message should crash the server at least it’ll only crash one.)
-
<node>
¶
The node whose ticks shall be used.
-
<item>
…
¶ The tick values you want to clear. Taken from the current
missing
list if not specified here; in this case, an emptynode
means to take the whole list, not just the ones fornode
.
Manage the list of nodes that collectively manage cleaning deleted entries from the DistKV tree.
All of these nodes must be online for clean-up to work.
-
-d
,
--delete
¶
Remove the mentioned nodes. Default is to add them.
-
<node>
…
¶ Nodes to add or delete. If none are given, list the current state, or (with
--delete
) clear the list, disabling node deletion.If you want to shut deletion down temporarily, you can also add a nonexistent node to the list.
This subcommand imports the debugger and then continues to process arguments.
This can be used to drop into the debugger when an exception occurs, set breakpoints, or whatever.
Note
Stepping over async function calls may or may not work. If not, your best bet is to set a breakpoint on the next line.