Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -124,3 +124,4 @@ misc/data/
# Claude Code local working files. Handoffs summarize in-flight work and may
# name local paths / private packages; they are deliberately never committed.
.claude/handoffs/
docsrc/
34 changes: 27 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,26 @@ To install: ```pip install dol```

[Documentation here](https://i2mint.github.io/dol/)

<!-- epythet:agentic-readme:start -->
## For AI agents

`dol` ships tooling for coding agents. If you are one, start here.

**Skills** ([Agent Skills](https://agentskills.io) format), for any agent host.

| Skill | Use it to |
|---|---|
| `dol-dev-portability` | keep dol working on Windows as well as Linux/macOS |
| `dol-dev-wrap-kvs` | understand and safely modify dol's core wrapping machinery — wrap_kvs, store_decorator, Store.wrap, and how transforms are applied |
| `dol-store-building` | build a dol store: wrap any storage backend |

**Instruction files**: `CLAUDE.md` (Claude Code).

**The documentation, machine-readable**: [`llms.txt`](https://i2mint.github.io/dol/llms.txt) indexes every page; [`dol.md`](https://i2mint.github.io/dol/dol.md) is the whole documentation in one file; every page has a `.md` twin; [`objects.inv`](https://i2mint.github.io/dol/objects.inv) maps symbols to URLs. The full list, with install lines, is on the site's [For AI agents](https://i2mint.github.io/dol/ai-agents.html) page.

If you like writing your own code, the rest of this README is written for you, starting at [Example use](#example-use).
<!-- epythet:agentic-readme:end -->

## Example use

Say you have a source backend that has pickles of some lists-of-lists-of-strings,
Expand Down Expand Up @@ -217,13 +237,13 @@ how the content is stored should be specified, but StoreInterface offers a dict-
__delitem__ calls: _id_of_key
__iter__ calls: _key_of_id

```pydocstring
```python
>>> from dol import Store
```

A Store can be instantiated with no arguments. By default it will make a dict and wrap that.

```pydocstring
```python
>>> # Default store: no key or value conversion ################################################
>>> s = Store()
>>> s['foo'] = 33
Expand All @@ -236,7 +256,7 @@ Now let's make stores that have a key and value conversion layer
input keys will be upper cased, and output keys lower cased
input values (assumed int) will be converted to ascii string, and visa versa

```pydocstring
```python
>>>
>>> def test_store(s):
... s['foo'] = 33 # write 33 to 'foo'
Expand All @@ -262,7 +282,7 @@ We can introduce this conversion layer in several ways.
Here are few...

## by subclassing
```pydocstring
```python
>>> # by subclassing ###############################################################################
>>> class MyStore(Store):
... def _id_of_key(self, k):
Expand All @@ -280,7 +300,7 @@ Here are few...

## by assigning functions to converters

```pydocstring
```python
>>> # by assigning functions to converters ##########################################################
>>> class MyStore(Store):
... def __init__(self, store, _id_of_key, _key_of_id, _data_of_obj, _obj_of_data):
Expand All @@ -301,7 +321,7 @@ Here are few...

## using a Mixin class

```pydocstring
```python
>>> # using a Mixin class #############################################################################
>>> class Mixin:
... def _id_of_key(self, k):
Expand All @@ -322,7 +342,7 @@ Here are few...

## adding wrapper methods to an already made Store instance

```pydocstring
```python
>>> # adding wrapper methods to an already made Store instance #########################################
>>> s = Store(dict())
>>> s._id_of_key=lambda k: k.upper()
Expand Down
16 changes: 15 additions & 1 deletion dol/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
"""Core tools to build simple interfaces to complex data sources and bend the interface to your will (and need)"""
"""Core tools to build simple interfaces to complex data sources and bend the interface to your will (and need).

``dol`` wraps any storage backend (files, S3, databases, dicts) behind a dict-like
interface, and transforms that interface with composable layers. Start with
``wrap_kvs`` (key/value transforms), the file stores (``Files``, ``TextFiles``,
``JsonFiles``, ``PickleFiles``), the ready-made codecs (``ValueCodecs``, ``KeyCodecs``),
``filt_iter`` (key filtering) and ``cache_this`` (caching).

>>> from dol import wrap_kvs
>>> import json
>>> s = wrap_kvs({}, obj_of_data=json.loads, data_of_obj=json.dumps)
>>> s['a'] = {'x': 1}
>>> s['a'], s.store
({'x': 1}, {'a': '{"x": 1}'})
"""

import os

Expand Down
4 changes: 2 additions & 2 deletions dol/_interface_wrap.py
Original file line number Diff line number Diff line change
Expand Up @@ -490,8 +490,8 @@ def _validate_stack_seams(stack):
def _compile_method_plan(name, sites, leaf_method, sig, encoders, decoders):
"""Compile one boundary method: encode role args, call leaf, decode result.

``sites``: {param_name_or_'return': ((role, path, ann), ...)}.
Returns a callable(*args, **kwargs) with the leaf method baked in.
``sites``: ``{param_name_or_'return': ((role, path, ann), ...)}``.
Returns a ``callable(*args, **kwargs)`` with the leaf method baked in.
"""
# Build per-parameter transformers (outer -> inner). Integer site keys
# mean positional index (dict-form specs), resolved against the outer
Expand Down
71 changes: 41 additions & 30 deletions dol/appendable.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
"""
Tools to add append-functionality to key-val stores. The main function is
`appendable_store_cls = add_append_functionality_to_store_cls(store_cls, item2kv, ...)`
You give it the `store_cls` you want to sub class, and a item -> (key, val) function, and you get a store (subclass) that
has a `store.append(item)` method. Also includes an extend method (that just called appends in a loop.

See add_append_functionality_to_store_cls docs for examples.
"""Tools to add append-functionality to key-val stores.

The main function is ``appendable(store_cls, item2kv=...)``: you give it the store
class you want to subclass and an item -> (key, val) function, and you get a store
(subclass) that has a ``store.append(item)`` method (and an ``extend``, which appends in
a loop). ``mk_item2kv_for`` holds ready-made item2kv factories (timestamps, uuids,
fields of the item, ...).

>>> from dol.appendable import appendable
>>> S = appendable(dict, item2kv=lambda item: (item['id'], item))
>>> s = S()
>>> s.append({'id': 1})
>>> s
{1: {'id': 1}}
"""

import time
Expand All @@ -24,7 +31,8 @@ def define_extend_as_seq_of_appends(obj):
Args:
obj: Class (type) or instance of an object that has an "append" method.

Returns: The obj, but with that extend method.
Returns:
The obj, but with that extend method.

>>> class A:
... def __init__(self):
Expand All @@ -48,7 +56,6 @@ def define_extend_as_seq_of_appends(obj):
>>> a.extend([10, 20])
>>> a.t
[1, 2, 3, 10, 20]

"""
assert hasattr(obj, "append"), (
f"Your object needs to have an append method! Object was: {obj}"
Expand Down Expand Up @@ -113,15 +120,16 @@ def attr(attr_name):

Args:
attr_name: The attribute name to use as the key
Returns: an item -> (key, val) function

Returns:
an item -> (key, val) function

>>> ref_getter =mk_item2kv_for.attr("ref")
>>> from collections import namedtuple
>>> A = namedtuple('A', ['ref'])
>>> a = A(ref='some_ref')
>>> ref_getter(a)
('some_ref', A(ref='some_ref'))

"""

def item2kv(item):
Expand All @@ -147,7 +155,8 @@ def item_to_key(item2key):
Args:
item2key: an item -> key function

Returns: an item -> (key, val) function
Returns:
an item -> (key, val) function

>>> item2key = lambda item: item['G'] # use value of 'L' as the key
>>> item2key({'L': 'let', 'I': 'it', 'G': 'go'})
Expand All @@ -166,8 +175,9 @@ def item2kv(item):
def field(field, keep_field_in_value=True, dflt_if_missing=NotSpecified):
"""item2kv that uses a specific key of a (mapping) item as the key

Note: If keep_field_in_value=False, the field will be popped OUT of the item.
If that's not the desired effect, one should feed copies of the items (e.g. map(dict.copy, items))
Note:
If keep_field_in_value=False, the field will be popped OUT of the item.
If that's not the desired effect, one should feed copies of the items (e.g. map(dict.copy, items))

:param field: The field (value) to use as the returned key
:param keep_field_in_value: Whether to leave the field in the item. If False, will pop it out
Expand All @@ -183,7 +193,6 @@ def field(field, keep_field_in_value=True, dflt_if_missing=NotSpecified):
>>> item2kv = mk_item2kv_for.field('G', dflt_if_missing=None)
>>> item2kv({'L': 'let', 'I': 'it', 'DIE': 'go'})
(None, {'L': 'let', 'I': 'it', 'DIE': 'go'})

"""
if dflt_if_missing is NotSpecified:
if keep_field_in_value:
Expand Down Expand Up @@ -216,9 +225,11 @@ def utc_key(offset_s=0, factor=1, *, time_postproc: Callable | None = None):
or to get a more accurate timestamp of an event.

Use case for offset_s:

* Align to another system's clock
* Get more accurate timestamping of an event. For example, in situations where the item is a chunk of live
streaming data and we want the key (timestamp) to represent the timestamp of the beginning of the chunk.
streaming data and we want the key (timestamp) to represent the timestamp of the beginning of the chunk.

Without an offset_s, the timestamp would be the timestamp after the last byte of the chunk was produced,
plus the time it took to reach the present function. If we know the data production rate (e.g. sample rate)
and the average lag to get to the present function, we can get a more accurate timestamp for the beginning
Expand All @@ -227,14 +238,14 @@ def utc_key(offset_s=0, factor=1, *, time_postproc: Callable | None = None):
Args:
offset_s: An offset (in seconds, possibly negative) to add to the current time.

Returns: an item -> (current_utc_s, item) function
Returns:
an item -> (current_utc_s, item) function

>>> import time
>>> item2key = mk_item2kv_for.utc_key()
>>> k, v = item2key('some data')
>>> assert abs(time.time() - k) < 0.01 # which asserts that k is indeed a (current) utc timestamp
>>> assert v == 'some data' # just the item itself

"""
if time_postproc is None:

Expand All @@ -260,7 +271,8 @@ def uuid_key(hex=True):
One advantage though, is that the uuid is time-based, so it can be used to sort
the keys in the order they were IDed.

Returns: an item -> (uuid, item) function
Returns:
an item -> (uuid, item) function

>>> import uuid
>>> item2key = mk_item2kv_for.uuid_key()
Expand All @@ -275,7 +287,6 @@ def uuid_key(hex=True):
>>> k, v = item2key('some data')
>>> isinstance(k, uuid.UUID)
True

"""
import uuid

Expand All @@ -298,12 +309,11 @@ def item_to_key_params_and_val(item_to_key_params_and_val, key_str_format):

Args:
item_to_key_params_and_val: an item -> (key_params, val) function
key_str_format: A string format such that
key_str_format.format(*key_params) or
key_str_format.format(**key_params)
will produce the desired key string
key_str_format: A string format such that ``key_str_format.format(*key_params)``
or ``key_str_format.format(**key_params)`` will produce the desired key string

Returns: an item -> (key, val) function
Returns:
an item -> (key, val) function

>>> # Using tuple key params with unnamed string format fields
>>> item_to_kv = mk_item2kv_for.item_to_key_params_and_val(lambda x: ((x['L'], x['I']), x['G']), '{}/{}')
Expand All @@ -330,14 +340,16 @@ def item2kv(item):
def fields(fields, keep_field_in_value=False, key_as_tuple=False):
"""Make item2kv from specific fields of a Mapping (i.e. dict-like object) item.

Note: item2kv will not mutate item (even if keep_field_in_value=False).
Note:
item2kv will not mutate item (even if keep_field_in_value=False).

Args:
fields: The sequence (list, tuple, etc.) of item fields that should be used to create the key.
keep_field_in_value: Set to True to return the item as is, as the value
key_as_tuple: Set to True if you want keys to be tuples (note that the fields order is important here!)

Returns: an item -> (item[fields], item[not in fields]) function
Returns:
an item -> (item[fields], item[not in fields]) function

>>> item_to_kv = mk_item2kv_for.fields('L')
>>> item_to_kv({'L': 'let', 'I': 'it', 'G': 'go'})
Expand All @@ -351,7 +363,6 @@ def fields(fields, keep_field_in_value=False, key_as_tuple=False):
>>> item_to_kv = mk_item2kv_for.fields(('G', 'L'), key_as_tuple=True) # but ('G', 'L') order is respected here
>>> item_to_kv({'L': 'let', 'I': 'it', 'G': 'go'})
(('go', 'let'), {'I': 'it'})

"""
if isinstance(fields, str):
fields_set = {fields}
Expand Down Expand Up @@ -392,7 +403,8 @@ def appendable(store_cls=None, *, item2kv, return_keys=False):
item2kv: The function that produces a (key, val) pair from an item
new_store_name: The name to give the new class (default will be 'Appendable' + store_cls.__name__)

Returns: A subclass of store_cls with two additional methods: append, and extend.
Returns:
A subclass of store_cls with two additional methods: append, and extend.


>>> item_to_kv = lambda item: (item['L'], item) # use value of 'L' as the key, and value is the item itself
Expand Down Expand Up @@ -516,7 +528,6 @@ class Extender:
>>> b_extender += ' split'
>>> store
{'a': 'pplesauce', 'b': 'anana split'}

"""

def __init__(
Expand Down
Loading
Loading