Code in DistKV¶
DistKV can store Python code and modules, either for direct use by your client or for a runner daemon.
TODO: There is no dependency resolution. Thus, while you can store Python modules in DistKV, there’s no guarantee yet that they’ll actually be present when your code loads.
Code¶
Python code stored in DistKV is wrapped with a procedure context, mainly to make returning a result more straightforward. This is done by indenting the code before compiling it: don’t depend on multi-line strings to be flush left.
Storage¶
The location for executable scripts is configurable and defaults to “.distkv code proc”. Scripts are stored as a dict with these attributes:
code
: the actual textis_async
: a flag whether the procedure is synchronous (None
), sync but should run in a worker thread (False
), or async (True
).vars
: Input parameters of your procedure. Parameters not mentioned here are available in akw
dict.requires
: modules which this code needs. XXX TODO
There’s no way for code written for a specific async library to run under
another, with the possible exception of “asyncio top of Trio” (via
trio-asyncio
. DistKV itself uses anyio
in order to avoid this
poroblem. The author strongly recommends to follow this practice if at all
possible.
The required modules must be stored in DistKV. Accessing modules from your Python installation or the virtualenv you’ve set up for DistKV is of course possible, but DistKV does not try to keep them up-to-date for you.
Call cr = await CodeRoot.as_handler(client)
. Then, call some code by
simply naming it: cr("forty.two")
will run the code stored at .distkv
code proc forty two
.
Modules¶
Python modules are stored to DistKV as plain code.
Recursive dependencies are not allowed.
Storage¶
The location for Python modules is configurable and defaults to “.distkv code module”. Modules are stored as a dict with these attributes:
code
: the actual program textrequires
: other modules which this module needs to be loaded.
Usage¶
Call await ModuleRoot.as_handler(client)
. All modules in your DistKV
store are loaded into the Python interpreter; use normal import statements
to access them.
TODO: Modules are not yet loaded incrementally.
Runners¶
The distributed nature of DistKV lends itself to running arbitrary code on any node that can accomodate it.
DistKV has two built-in types of code runners.
Single-node runner¶
This runner executes code on a specific node. This is useful e.g. if you need to access actual hardware.
In order to be able to implement “emergency operation when disconnected” or similar fallback strategies, single-node runners can be configured to provide an Actor group that represents the “central” nodes. Your code can access a queue that reports whether the central group is partially or completely (in)visible.
Any-node runner¶
This runner executes code on some node, largely determined by chance, startup order, or phase of the moon.
TODO: Load balancing is not yet implemented.
All nodes in a runner form an Actor group; the node that holds the Tag checks whether jobs need to start.
Runner configuration¶
Runner entries don’t hold code, they merely point to it.
See distkv.runner.RunnerEntry
for details.
The actcual runtime information is stored in a separate “state” node, mainly to avoid race conditions.
See distkv.runner.StateEntry
for details.