diff --git a/claude-skill.mdx b/claude-skill.mdx index 3b2cc30..a41c7fe 100644 --- a/claude-skill.mdx +++ b/claude-skill.mdx @@ -41,11 +41,12 @@ The skill is structured so Claude only loads the reference it needs: | Module | Covers | | --- | --- | -| `startup` | Installation, starting the server, connecting, loading a graph | -| `querying` | `MATCH`, `WHERE`, joins, built-in functions | -| `writing` | `CREATE`, `SET`, the change and commit workflow | -| `algorithms` | Shortest path, vector search | -| `introspection` | Schema exploration, Python SDK reference | +| `startup` | Installation, starting the server, connecting (HTTP, authentication, embedded), loading a graph | +| `querying` | `MATCH`, `OPTIONAL MATCH`, paths, `WITH`, aggregation, `UNWIND`, subqueries, built-in functions, `LOAD CSV` | +| `writing` | `CREATE`, `MERGE`, `SET`, `DELETE`, the change and commit workflow | +| `importing` | CSV, Parquet, JSONL, GML and Neo4j imports | +| `algorithms` | Shortest path, vector search, embeddings, GNN sampling | +| `introspection` | Schema exploration, versioning and time travel, Python SDK reference | ## Links diff --git a/commands.mdx b/commands.mdx index 5080d6f..1a0cec1 100644 --- a/commands.mdx +++ b/commands.mdx @@ -25,6 +25,7 @@ turingdb start [options] | `-reset-default` | Delete the default graph before starting | off | | `-ui` | Launch the built-in visualizer proxy | off | | `-ui-port ` | Visualizer proxy port | `8080` | +| `-auth-on` | Require a bearer token on every request. The token is read from the `TURINGDB_AUTH_TOKEN` environment variable, and must be set when you use this flag | off | | `-start-timeout ` | Time to wait for daemon readiness | `500` | ### Examples @@ -53,6 +54,34 @@ Load a graph at startup: turingdb -load mygraph ``` +Require authentication: + +```bash +TURINGDB_AUTH_TOKEN=s3cret turingdb start -demon -auth-on +``` + +Serve the binary protocol instead of JSON (see [Environment variables](#environment-variables)): + +```bash +USE_TURING_PROTO=1 turingdb start -demon +``` + + + Server logs are written to `/logs/turingdb.log`. Additional details about an import, such as a `LOAD JSONL`, are written there. + + +### Interactive shell + +Without `-demon`, `turingdb start` serves HTTP **and** opens an interactive Cypher shell. Besides Cypher queries and [engine commands](/query/procedures#commands), the shell has a few local commands: + +| Command | Description | +|---|---| +| `cd ` | Switch the active graph | +| `checkout change-` | Work on a change. `checkout ` opens a past commit, and `checkout` returns to the latest commit of main | +| `read ` | Run the queries in a file | +| `help` | List the available commands | +| `quit` / `exit` | Leave the shell | + --- ## UI @@ -116,3 +145,29 @@ Stop an instance running in a custom directory: ```bash turingdb stop -turing-dir /data/myturing ``` + +--- + +## Environment variables + +| Variable | Read by | Effect | +|---|---|---| +| `USE_TURING_PROTO` | server | `1` makes the server speak TuringDB's **binary protocol** instead of JSON over HTTP. The server then serves the binary protocol only: connect with `TuringDB(type="native")`, and use a server in the default mode for `curl`, the JSON client and the visualizer. See [client backends](/pythonsdk/reference#client-backends) | +| `TURINGDB_AUTH_TOKEN` | server and Python SDK | With `-auth-on`, the token the server requires. The SDK sends it as a bearer token when set | +| `TURINGDB_TYPE` | Python SDK | Default client backend (`json`, `native` or `embedded`) when `type` is not passed | + +--- + +## HTTP API + +The Python SDK is a thin client over a single endpoint, so you can query a server from any language. With the default (JSON) server: + +```bash +curl -s -X POST 'http://localhost:6666/query?graph=mygraph' -d "MATCH (n) RETURN count(n)" +``` + +- The request body is the query text. The optional query-string parameters `change=` and `commit=` target a change or a past commit. +- With `-auth-on`, send `Authorization: Bearer `. A missing or incorrect token returns HTTP `401` with an empty body. +- A successful response is `{"header": {"column_names": [...], "column_types": [...]}, "data": [chunk, ...], "time": }`, where each chunk is a list of columns. +- An error response is `{"error": "PARSE_ERROR", "error_details": "...", "time": }`. The code is one of `PARSE_ERROR`, `ANALYZE_ERROR`, `PLAN_ERROR`, `EXEC_ERROR`, `CHANGE_NOT_FOUND` or `COMMIT_NOT_FOUND`. +- Queries on a non-HEAD commit need `LOAD COMMIT ''` first when you talk to the API directly. The CLI and the SDK do this for you. diff --git a/docs.json b/docs.json index 7c0ceb1..9b64f3b 100644 --- a/docs.json +++ b/docs.json @@ -64,6 +64,13 @@ "group": "Query Language", "pages": [ "query/cypher_subset", + "query/matching", + "query/paths", + "query/aggregation", + "query/subqueries", + "query/functions", + "query/writing", + "query/procedures", "query/cheatsheet" ] }, diff --git a/graph_dev/create_change.mdx b/graph_dev/create_change.mdx index 37e3e71..a18e682 100644 --- a/graph_dev/create_change.mdx +++ b/graph_dev/create_change.mdx @@ -18,10 +18,11 @@ description: "Creating a change on a TuringDB graph is similar to creating a new ```python - # Create a new change on the graph + # Create a new change on the graph. + # The client is switched onto the new change automatically. change = client.new_change() - # Checkout into the change + # Optional: checkout into a change that already exists client.checkout(change=change) ``` diff --git a/graph_dev/create_nodes_edges.mdx b/graph_dev/create_nodes_edges.mdx index dc0fa80..8ffc3a9 100644 --- a/graph_dev/create_nodes_edges.mdx +++ b/graph_dev/create_nodes_edges.mdx @@ -69,4 +69,13 @@ description: "Once you have a valid change, you can start adding new nodes and e In the case you create the nodes then the edges, you need to commit your changes after node creation. - \ No newline at end of file + + +## Merge, delete and bulk loading + +- Use `MERGE` instead of `CREATE` to avoid duplicates: `MERGE (n:Person {name: 'Jane'})`. Run `COMMIT` first when `MERGE` should find nodes created earlier in the same change. +- Remove data with `DELETE` (edges, or nodes without edges) and `DETACH DELETE` (a node and all its edges). +- Every node needs at least one label, and every edge exactly one edge type. +- For more than a few thousand nodes or edges, build the graph with [`LOAD PARQUET`](/import_data/parquet) instead of Cypher writes. + +See [Writing data](/query/writing) for the full reference. diff --git a/graph_dev/submit_change.mdx b/graph_dev/submit_change.mdx index fc785dc..a410fa2 100644 --- a/graph_dev/submit_change.mdx +++ b/graph_dev/submit_change.mdx @@ -23,4 +23,10 @@ description: "Submit changes done to the database (just like a commit being subm client.checkout() ``` - \ No newline at end of file + + + + `CHANGE SUBMIT` commits any pending writes itself, so a `COMMIT` right before it is optional. In the Python SDK, call `client.checkout()` afterwards to return to main: the client points at the submitted change until you do. + + +If two changes edit the same node, the second submit reports a conflict, for example `This change attempted to update Node 0 (...) which has been modified on main.` The failed change stays open: fix it and submit again, or discard it with `CHANGE DELETE`. See [Writing data](/query/writing#failed-queries-and-conflicts). diff --git a/graph_dev/time_travelling.mdx b/graph_dev/time_travelling.mdx index a457b3d..11a8e2b 100644 --- a/graph_dev/time_travelling.mdx +++ b/graph_dev/time_travelling.mdx @@ -39,6 +39,14 @@ description: "You can time travel in you graph databases and see the commit hist +## About `db.history()` + +- Commits are listed newest first, and the latest one is marked with a literal `(HEAD)` suffix on its hash: strip it before passing the hash to `checkout`. +- Every `COMMIT` and every submitted change adds an entry. +- `nodeCount` and `edgeCount` count what **that commit** added, not the graph totals. +- Use hashes exactly as printed (they are not zero-padded). An unknown hash returns `COMMIT_NOT_FOUND` from `client.checkout(commit=...)`, and zeros from `CALL db.describeCommit('')`. +- A past commit is read-only, so make writes on main or in a change. Call `client.checkout()` to return to main. + ## Lazy Commit Loading TuringDB lazily loads commits from disk. When a graph is loaded, only the **HEAD commit** (the latest commit on the current branch) is fully loaded into memory. All other commits are registered but their data remains on disk until explicitly requested. diff --git a/graph_dev/update_properties.mdx b/graph_dev/update_properties.mdx index e8fa194..038081b 100644 --- a/graph_dev/update_properties.mdx +++ b/graph_dev/update_properties.mdx @@ -84,12 +84,28 @@ The `SET` clause is used to update property values on existing nodes and edges. SET n.isMinor = true ``` - - `CREATE ... SET` is not supported. When creating nodes or edges, use brace initialisation to set properties directly in the `CREATE` clause: + + When creating nodes or edges, you can set properties directly in the `CREATE` clause with braces: ```jsx CREATE (:Person {name: 'Alice', age: 30}) ``` + `CREATE ... SET` also works: `CREATE (n:Item {sku: 'A1'}) SET n.price = 9.99`. + + + ## Clearing and removing properties + + Set a property to `null` to clear its value, or use `REMOVE` to remove properties: + + ```jsx + MATCH (n:Person {name: 'Alice'}) SET n.age = null + MATCH (n:Person {name: 'Alice'}) REMOVE n.position, n.surname + ``` + + + Each property name has a single type across the whole graph, so `SET n.age = 'old'` is reported as a type incompatibility when `age` is an Integer. See [Writing data](/query/writing#set-and-remove) for the full rules. + + Labels cannot be changed after creation. ```python @@ -172,11 +188,27 @@ The `SET` clause is used to update property values on existing nodes and edges. """) ``` - - `CREATE ... SET` is not supported. When creating nodes or edges, use brace initialisation to set properties directly in the `CREATE` clause: + + When creating nodes or edges, you can set properties directly in the `CREATE` clause with braces: ```python client.query("CREATE (:Person {name: 'Alice', age: 30})") ``` + `CREATE ... SET` also works: `client.query("CREATE (n:Item {sku: 'A1'}) SET n.price = 9.99")`. + + + ## Clearing and removing properties + + Set a property to `null` to clear its value, or use `REMOVE` to remove properties: + + ```python + client.query("MATCH (n:Person {name: 'Alice'}) SET n.age = null") + client.query("MATCH (n:Person {name: 'Alice'}) REMOVE n.position, n.surname") + ``` + + + Each property name has a single type across the whole graph, so `SET n.age = 'old'` is reported as a type incompatibility when `age` is an Integer. See [Writing data](/query/writing#set-and-remove) for the full rules. + + Labels cannot be changed after creation. diff --git a/import_data/csv.mdx b/import_data/csv.mdx index a38413d..61df179 100644 --- a/import_data/csv.mdx +++ b/import_data/csv.mdx @@ -6,7 +6,7 @@ description: "Stream rows from a CSV file in a query with LOAD CSV" `LOAD CSV` streams the rows of a CSV file into a query. Unlike `LOAD JSONL` / `LOAD GML` (which build a whole graph from a file), `LOAD CSV` is a **reading statement** that yields one row at a time, bound to a variable you can reference in `RETURN`, `MATCH`, or `CREATE`. -The file must live in the `data` subdirectory of the TuringDB working directory (set by `-turing-dir`, default `$HOME/.turing`). +The file must live in the `data` subdirectory of the TuringDB working directory (set by `-turing-dir`, default `$HOME/.turing`). Absolute paths are re-rooted inside that directory. ## Syntax @@ -21,7 +21,7 @@ LOAD CSV "" [WITH HEADERS] [ON ERROR SKIP | ON ERROR FAIL] AS | `ON ERROR FAIL` | Abort on the first malformed row (the default) | | `AS ` | Variable bound to each row | -All values come back as **strings** — wrap them in `toInteger(...)` / `toFloat(...)` if you need numbers. +All values come back as **strings**. Wrap them in `toInteger(...)` / `toFloat(...)` / `datetime(...)` if you need other types. ## With headers @@ -38,19 +38,36 @@ RETURN row[0] AS name, row[1] AS age LIMIT 5 ``` +Read a row's fields as `row.col` or `row[]`. Without `WITH HEADERS`, the header line comes back as an ordinary data row. + +## Combining with the rest of a query + +`LOAD CSV` composes with `MATCH`, `WITH` / `WHERE`, aggregates, `ORDER BY` and `UNWIND`. For example, join the rows with nodes that are already in the graph: + +```jsx +LOAD CSV "people.csv" WITH HEADERS AS row +MATCH (p:Person {name: row.name}) +RETURN p.name, row.city +``` + + + To use `row.x` inside an `OPTIONAL MATCH` pattern that directly follows `LOAD CSV`, project the field first: `... AS row WITH row.name AS nm OPTIONAL MATCH (p:Person {name: nm}) ...`. + + ## Building a graph from CSV Because `LOAD CSV` yields rows, you can drive `CREATE` from it inside a change: ```python -change = client.new_change() -client.checkout(change=change) +client.new_change() client.query('LOAD CSV "people.csv" WITH HEADERS AS row CREATE (:Person {name: row.name, age: toInteger(row.age)})') -client.query("COMMIT") +client.query("COMMIT") # make the new nodes visible to the next query client.query("CHANGE SUBMIT") client.checkout() ``` - - `LOAD CSV` followed by `MATCH` in the same read statement is not yet supported. - + + This approach suits small files and additions to an existing graph. Creating edges with `LOAD CSV` matches both endpoints for every row, so it is best kept to a few thousand edges. For bulk loads, convert the CSV to Parquet and use [`LOAD PARQUET`](/import_data/parquet), which loads 100,000 nodes and 200,000 edges in about 0.2 seconds. + + +To skip malformed rows, add `ON ERROR SKIP` (the default, `ON ERROR FAIL`, stops at the first one). See [Writing data](/query/writing#bulk-writes) for a single-pass `MERGE` pattern that needs no `COMMIT`. diff --git a/import_data/gml.mdx b/import_data/gml.mdx index fc225f6..ea2842e 100644 --- a/import_data/gml.mdx +++ b/import_data/gml.mdx @@ -32,12 +32,20 @@ graph [ | `source` / `target` | Required. Must reference the `id` of a node defined in the same file. | | other fields | Optional. Stored as edge properties. | -**Supported property value types:** String, Integer, Double, Boolean. All property values are stored as strings in the TuringDB graph regardless of their original type in the GML file. +**Property values:** all property values are stored as **strings** in the TuringDB graph, regardless of their original type in the GML file. Convert them with `toInteger` / `toFloat` in your queries. All imported nodes are assigned the label `GMLNode` and all edges the type `GMLEdge`, regardless of any fields present in the file. Any field other than `id`, `source`, and `target` is treated as a plain property. To query your data after import, use `MATCH (n:GMLNode)` and `MATCH ()-[e:GMLEdge]->()`. + + Property names also get a **type suffix**, for example `label (String)` and `weight (String)`. Quote them with backticks in queries, and run `CALL db.propertyTypes()` to see the exact names: + + ```jsx + MATCH (n:GMLNode) RETURN n.`label (String)`, toInteger(n.`age (String)`) + ``` + + ## Import steps diff --git a/import_data/jsonl.mdx b/import_data/jsonl.mdx index 118b44e..5a7c721 100644 --- a/import_data/jsonl.mdx +++ b/import_data/jsonl.mdx @@ -27,7 +27,7 @@ Each line is a self-contained JSON object representing either a node or a relati | Field | Requirement | | ------------ | ------------------------------------------------------------------------------------------------------ | | `type` | Must be the string `"node"` | -| `id` | Required. Integer or string parseable as integer. Must start at `0` and increment by `1` with no gaps. | +| `id` | Required. Integer or string parseable as integer. Ids do not need to be contiguous. | | `labels` | Required. Array with at least one label. | | `properties` | Optional. Dictionary with property keys and values. | @@ -36,20 +36,48 @@ Each line is a self-contained JSON object representing either a node or a relati | Field | Requirement | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Must be the string `"relationship"` | -| `id` | Required. Integer or string parseable as integer. Must start at `0` and increment by `1` with no gaps. | +| `id` | Required. Integer or string parseable as integer. | | `label` | Required. Exactly one label. | | `start` / `end` | Required. Either:
- a bare integer (or string parseable as integer) referencing a node `id`
- a dictionary with an `id` key holding an integer (or string parseable as integer) referencing a node `id`. | | `properties` | Optional. Dictionary with property keys and values. | -## Loading embedding properties +## Property values -By default, an array-valued property in the JSONL is stored as a string. To load specific array properties as **embeddings** (type `Embedding`, usable with `cosine_similarity` / `euclidean_distance`), append a `WITH EMBEDDINGS` clause listing each embedding property and its dimension: +- Scalars (strings, integers, doubles, booleans) become properties of the matching type, and `null` values are skipped. +- Arrays load as **List** properties. +- Nested objects are stored as **JSON strings**. +- ISO date strings stay strings, and integer durations stay integers, unless you declare them (see below). + +Each property name has a single type across the whole graph, so use the same type for a property on every node. + +## Loading date, duration and embedding properties + +Typed options can be chained after the graph name: + +```jsx +LOAD JSONL 'mydata.jsonl' AS mygraph + WITH EMBEDDINGS [{"emb", 384}, {"summary_vec", 768}] + WITH DATETIMES [born, updatedAt] + WITH DURATIONS [tenure] +``` + +- `WITH DATETIMES [prop, ...]` parses ISO 8601 strings into `DateTime` values. +- `WITH DURATIONS [prop, ...]` reads integers as microseconds and stores them as `Duration` values. +- `WITH EMBEDDINGS [{"prop", dim}, ...]` stores arrays as `Embedding` values, see below. + +### Loading embedding properties + +An array-valued property is stored as a List by default. To load specific array properties as **embeddings** (type `Embedding`, usable with `cosine_similarity` / `euclidean_distance`), append a `WITH EMBEDDINGS` clause listing each embedding property and its dimension: ```jsx LOAD JSONL 'mydata.jsonl' AS mygraph WITH EMBEDDINGS [{"embedding", 384}, {"summary_vec", 768}] ``` -Each spec is `{"", }`. The dimension must be **greater than 1**, and every array under that property must have exactly that many numeric elements (otherwise the load fails). This works for both node and relationship properties. Array properties not listed in `WITH EMBEDDINGS` are still stored as strings. +Each spec is `{"", }`. The dimension must be **greater than 1**, and every array under that property must have exactly that many numeric elements. This works for both node and relationship properties. Array properties not listed in `WITH EMBEDDINGS` are stored as Lists. + + + When an import fails, additional details are written to `/logs/turingdb.log`. + This sets embeddings as node/edge **properties**. To bulk-load embeddings onto an *existing* graph from a Parquet file, use [`LOAD EMBEDDING FROM`](/vector-search#embeddings-as-node-properties) instead. diff --git a/import_data/parquet.mdx b/import_data/parquet.mdx index eda7de9..0974bc3 100644 --- a/import_data/parquet.mdx +++ b/import_data/parquet.mdx @@ -1,42 +1,91 @@ --- title: "Import Parquet files" sidebarTitle: "Parquet" -description: "Build a TuringDB graph from node and edge Parquet files with the turing-parquet CLI" +description: "Build a TuringDB graph from node and edge Parquet files with LOAD PARQUET or the turing-parquet CLI" --- -To load Parquet files containing columns for node IDs, node properties, edge source and target IDs, and edge properties, see: `LOAD PARQUET`. +To load Parquet files containing columns for node IDs, node properties, edge source and target IDs, and edge properties, see [`LOAD PARQUET`](#load-parquet). - -To load Parquet files containing a JSON string column, which is then converted into a graph structure, see: `turing-parquet`. +To load Parquet files containing a JSON string column, which is then converted into a graph structure, see [`turing-parquet`](#turing-parquet). # `LOAD PARQUET` -`LOAD PARQUET` queries may be used to load a graph into TuringDB. The command is of the following format: +`LOAD PARQUET` is the fastest way to get data into TuringDB, and the recommended path for anything beyond a few thousand entities. + ``` -LOAD PARQUET " {AS }" +LOAD PARQUET '' [AS ] [WITH DURATIONS [, ...]] ``` -where `{AS }` may be omitted, in which case the name of the graph will be that of the directory. -The directory specified need contain `nodes.parquet` and `edges.parquet`. +The graph name defaults to the name of the directory. The directory is resolved inside the `data` subdirectory of the TuringDB working directory (`-turing-dir`) and must contain `nodes.parquet` and `edges.parquet`. + + + `LOAD PARQUET` creates a **new graph**, so choose a graph name that does not exist yet. To add a large volume of data to an existing graph, write the existing data plus the new data to Parquet and load it under a new graph name. Use Cypher writes only for small, incremental changes. + + +## Required columns + +`nodes.parquet` has one row per node: + +- `__id`: `INT64`, unique. Your own node key. Edges refer to it. +- `__labels`: a list of strings (`list`), non-empty. Each element is a label of the node. The path of each element must be `__labels.list.element`. + +`edges.parquet` has one row per edge: + +- `__source`: `INT64`, the `__id` of the source node. +- `__target`: `INT64`, the `__id` of the target node. +- `__type`: a string, the edge type. Every edge has exactly one. -Each row in `nodes.parquet` defines a node, and the file has a required format of: -- a column of name `__id` of type `INT64` - defines the relative node ID for the row; -- a column of name `__labels` of type `BYTE_ARRAY` with exactly 1 level of nesting, forming a byte array of byte arrays - each inner byte array is treated as a string (the name of that label), the path of each element *must* be `__labels.list.element`. +## Property columns -Each row in `edges.parquet` defines an edge, and the file has a required format of: -- a column of name `__source` of type `INT64` - defines the node ID (present in `nodes.parquet`) of the source node of this edge; -- a column of name `__target` of type `INT64` - defines the node ID (present in `nodes.parquet`) of the target node of this edge; -- a column of name `__type` of type `BYTE_ARRAY` with exactly 0 levels of nesting - interpreted as a string (the name of this edge type). +Every other column becomes a property of the node or edge on that row. The column name is the property name. Null values are allowed. -As well as the above required columns, `nodes.parquet` and `edges.parquet` may contain any number of extra columns, all of which are interpreted as a property value for the node/edge of that row. These columns may have any name (other than the names of the columns explicitly required above). +| Parquet type | TuringDB property type | +| ------------ | ---------------------- | +| `INT64` | Integer | +| `DOUBLE` | Double | +| `BOOLEAN` | Boolean | +| `BYTE_ARRAY` (string) | String | +| timestamp | DateTime | +| list of the types above | List | -The supported types for property columns are: -- `BOOLEAN` -- `INT64` -- `DOUBLE` -- `BYTE_ARRAY` with exactly 0 levels of nesting +Use 64-bit columns: cast `INT32` and `FLOAT` columns to `int64` / `float64` first. To load an `INT64` column of microseconds as a `Duration`, name it in `WITH DURATIONS []`. + +## From pandas + +To go from CSV, JSON, SQL extracts or API data, load it into pandas, map your natural keys to `INT64` ids, and write the two files: + +```python +import pandas as pd, pyarrow as pa, pyarrow.parquet as pq +from pathlib import Path + +people = pd.read_csv("people.csv") # columns: email, name, age +knows = pd.read_csv("knows.csv") # columns: src, dst, since (src and dst are emails) + +ids = {key: i for i, key in enumerate(people["email"])} # natural key -> int64 __id +nodes = pd.DataFrame({ + "__id": pd.Series(range(len(people)), dtype="int64"), + "__labels": [["Person"]] * len(people), + "email": people["email"], + "name": people["name"], + "age": people["age"].astype("Int64"), # int64 +}) +edges = pd.DataFrame({ + "__source": knows["src"].map(ids).astype("int64"), + "__target": knows["dst"].map(ids).astype("int64"), + "__type": "KNOWS", + "since": knows["since"].astype("int64"), +}) + +out = Path.home() / ".turing" / "data" / "people_graph" # /data/ +out.mkdir(parents=True, exist_ok=True) +pq.write_table(pa.Table.from_pandas(nodes, preserve_index=False), out / "nodes.parquet") +pq.write_table(pa.Table.from_pandas(edges, preserve_index=False), out / "edges.parquet") + +client.query("LOAD PARQUET 'people_graph' AS people") +client.set_graph("people") +``` -Note: all property columns of `BYTE_ARRAY` are treated as strings +For several node types, concatenate them into a single `nodes.parquet` with non-overlapping `__id` ranges, each row carrying its own `__labels`. Keep the natural key (`email` above) as a property so you can match on it in later queries. # `turing-parquet` diff --git a/index.mdx b/index.mdx index d18f7be..40ca18e 100644 --- a/index.mdx +++ b/index.mdx @@ -20,7 +20,7 @@ The platform has three main components: the [TuringDB engine](/concepts/overview Learn about the core concepts and architecture of TuringDB - TuringDB query language based on OpenCypher + Cypher in TuringDB v3: most of openCypher, plus versioning and vector search See how TuringDB performs against other graph databases diff --git a/pythonsdk/get_started.mdx b/pythonsdk/get_started.mdx index 5f32949..2268be1 100644 --- a/pythonsdk/get_started.mdx +++ b/pythonsdk/get_started.mdx @@ -130,7 +130,7 @@ from turingdb import TuringDB client = TuringDB(host="http://localhost:6666") # Or run the engine in-process, with no server: -# client = TuringDB(type="embedded", data_dir="~/.turing") +# client = TuringDB(type="embedded", data_dir="/home/me/.turing") # use an absolute path # Create and set working graph client.create_graph("people") diff --git a/pythonsdk/reference.mdx b/pythonsdk/reference.mdx index 4336c47..32ba583 100644 --- a/pythonsdk/reference.mdx +++ b/pythonsdk/reference.mdx @@ -22,6 +22,8 @@ or using `pip`: pip install turingdb ``` +The SDK targets TuringDB v3: pin `turingdb>=3.0` in your project (older projects may pin `1.x`, which uses the previous engine and a much smaller Cypher subset). The wheel supports Python 3.10 to 3.14 on Linux (x86_64, aarch64) and macOS 15 or later (arm64). It also contains the `turingdb` command-line tool, the `turing-parquet` import tool, and the in-process engine. + ## Getting Started ```python @@ -42,25 +44,81 @@ print(client.list_available_graphs()) | `type` | Encoding / transport | Constructor arguments | | ------ | -------------------- | --------------------- | -| `"json"` *(default)* | JSON payloads over HTTP against a running daemon | `host="http://localhost:6666"` | -| `"native"` | Binary (Turing proto) payloads over HTTP — faster; requires the compiled extension | `host="localhost"`, `port=6666` | -| `"embedded"` | In-process engine, no daemon or socket | `data_dir="/path/to/.turing"` | - -Both `json` and `native` talk to a daemon over HTTP; they differ in the wire encoding (JSON vs. binary). `embedded` runs the engine in-process with no network at all. +| `"json"` *(default)* | JSON over HTTP against a running server | `host="http://localhost:6666"`, optional `token` | +| `"native"` | TuringDB's binary protocol against a running server. Faster for large results | `host="localhost"`, `port=6666`, optional `token` | +| `"embedded"` | In-process engine, no server or socket | `data_dir="/path/to/.turing"` | ```python -# Default HTTP client +# Default HTTP/JSON client client = TuringDB(host="http://localhost:6666") # Native binary protocol client = TuringDB(type="native", host="localhost", port=6666) -# Embedded, in-process — no server needed -client = TuringDB(type="embedded", data_dir="~/.turing") +# Embedded, in-process: no server needed +client = TuringDB(type="embedded", data_dir="/home/me/.turing") +``` + +Use the default `json` backend unless you have a reason not to. The other two are optimisations for specific situations: + +- **`native`** when you control the server and need more throughput on large results. +- **`embedded`** when you want a self-contained engine with no server to start, stop or connect to. + +### The native binary protocol + +The binary protocol is a **server-side mode**, selected once when the server starts with the environment variable `USE_TURING_PROTO=1`: + +```bash +USE_TURING_PROTO=1 turingdb start -demon -p 6666 +``` + +```python +client = TuringDB(type="native", host="localhost", port=6666) +``` + +A server speaks either JSON or the binary protocol on a given port: + +| Server started | `type="json"` client | `type="native"` client | +| -------------- | -------------------- | ---------------------- | +| normally (JSON, the default) | works | returns `Proto header dataLen does not match chunk payload size` | +| with `USE_TURING_PROTO=1` | returns `Invalid response from the server` | works | + +A server in binary mode serves the binary protocol only, so `curl`, the JSON client and the browser visualizer (`-ui`) connect to a server running in the default mode. To use both, run two servers on different ports. + +- `host` accepts `"localhost"` together with `port=`, or a full URL such as `"http://localhost:6666"`. Use one of these two forms. +- The query API is identical, and results come back as the same DataFrames. Embeddings are returned as numpy `float32` arrays instead of Python lists. +- To list the graphs on disk, use the JSON client. On the native backend, `list_loaded_graphs()` lists the loaded graphs. +- `token`, `reconnect()` and the S3 helpers work as on the JSON backend. + +### Authentication + +A server started with `-auth-on` requires a bearer token. Pass it with `token=` or set the `TURINGDB_AUTH_TOKEN` environment variable, which the SDK reads automatically. This works on both the `json` and `native` backends: + +```bash +TURINGDB_AUTH_TOKEN=s3cret turingdb start -demon -auth-on +``` + +```python +client = TuringDB(host="http://localhost:6666", token="s3cret") +``` + +A missing or incorrect token returns HTTP `401`, which the JSON client raises as an `httpx.HTTPStatusError`. The embedded backend runs in-process and needs no authentication, so it does not take a `token`. + +### The embedded backend + +```python +import os + +client = TuringDB(type="embedded", data_dir=os.path.expanduser("~/.turing")) ``` +- `data_dir` is the TuringDB root directory (the one that holds `graphs/`, `data/` and `vector/`). Omit it to use `~/.turing`. +- **Use an absolute path**, for example with `os.path.expanduser("~/...")`, since a leading `~` is not expanded. +- Writes still persist to disk when you submit a change, so your work survives the process ending. +- The embedded backend is designed for a single process. For the browser visualizer, `list_available_graphs()`, concurrent access from several processes, the S3 helpers and `token=`, use a server. + - `list_available_graphs()` and the S3 helpers (`s3_connect`, `transfer`) are only supported on the default JSON backend (`type="json"`). + `list_available_graphs()` is only supported on the default JSON backend. The S3 helpers (`s3_connect`, `transfer`) work on the JSON and native backends. ## Core Methods @@ -125,15 +183,16 @@ Returns whether the current graph is loaded in memory. ### `new_change() → int` -Creates a new isolated change (equivalent to `CHANGE NEW`), makes it the active change, and returns its integer ID. +Creates a new isolated change (equivalent to `CHANGE NEW`), makes it the active change, and returns its integer ID. The client is already on the new change, so a separate `checkout(change=...)` is not needed. ```python change = client.new_change() -client.checkout(change=change) ``` +`new_change()` raises if the client is already on a change or on a past commit. Call `client.checkout()` first. + - There is no `submit()` method — submit a change by running `client.query("CHANGE SUBMIT")`. + Submit a change by running `client.query("CHANGE SUBMIT")`, then call `client.checkout()` to return to main. The client points at the submitted change until you do, so call `checkout()` before the next query. ## Version Control Helpers @@ -184,7 +243,7 @@ You can also read the current context through the `current_graph`, `current_comm ### `reconnect()` -Resets the connection. A no-op for the HTTP and embedded backends; reopens the socket for the native backend. +Resets the connection. A no-op for the HTTP and embedded backends. Reopens the socket for the native backend. ### `try_reach(timeout: int = 5)` / `warmup(timeout: int = 5)` @@ -196,7 +255,7 @@ Return the server-side execution time and the full round-trip time (in milliseco ## S3 / Data Transfer -These helpers move graph data between local paths, S3, and the TuringDB server. They are only available on the JSON backend (`type="json"`). +These helpers move graph data between local paths, S3, and the TuringDB server. They are available on the JSON and native backends, but not on the embedded backend. ### `s3_connect(bucket_name, access_key=None, secret_key=None, region=None, use_scratch=True)` @@ -219,7 +278,7 @@ The equivalent engine commands (issued directly via `query`) are `S3 CONNECT " + Over the JSON backend, Doubles are serialised with 6 decimal places. Use the native or embedded backend if you need full precision. + ## Error Handling -All SDK errors raise a custom `TuringDBException`. +Query errors raise `TuringDBException`. The message starts with the error code (`PARSE_ERROR`, `ANALYZE_ERROR`, `PLAN_ERROR`, `EXEC_ERROR`, `CHANGE_NOT_FOUND`, `COMMIT_NOT_FOUND`, ...) and points at the offending part of the query. ```python +from turingdb import TuringDB, TuringDBException + try: - client.query("MATCH (x) RETURN x") + client.query("MATCH (n RETURN n") except TuringDBException as e: print("Query failed:", e) ``` +On the JSON backend, transport failures (the server is down, or HTTP 401 for a bad or missing token) surface as `httpx` exceptions instead. + ## Example Workflow ```python diff --git a/query/aggregation.mdx b/query/aggregation.mdx new file mode 100644 index 0000000..0eb4121 --- /dev/null +++ b/query/aggregation.mdx @@ -0,0 +1,239 @@ +--- +title: "Results, Aggregation and Lists" +sidebarTitle: "Results and Aggregation" +description: "Shape results with WITH, DISTINCT, ORDER BY, SKIP, LIMIT and UNION, then aggregate, unwind and transform lists." +--- + +These examples use the [sample graph](/query/matching) from the matching page. + +## RETURN, DISTINCT and expressions + +```cypher +MATCH (n:Person) +RETURN n.name, n.age + 5 AS adjusted + +MATCH (n:Person) +RETURN DISTINCT n.city + +MATCH (n:Person) +RETURN n.name AS nm +ORDER BY size(nm) DESC, nm + +MATCH (n:Person) +RETURN * +``` + +`RETURN *` returns every variable in scope (nodes as IDs). A column without an alias is named after its expression, for example `n.name`. + +## ORDER BY, SKIP and LIMIT + +```cypher +MATCH (n:Person) +RETURN n.name, n.age +ORDER BY n.age + +MATCH (n:Person) +RETURN n.name, n.age +ORDER BY n.age DESC + +MATCH (n:Person) +RETURN n.name, n.city, n.age +ORDER BY n.city, n.age DESC + +MATCH (n:Person) +RETURN n.name, n.age +ORDER BY n.age DESC SKIP 1 LIMIT 2 +``` + +- You can sort by several keys, by aliases and by expressions. Ascending is the default. +- `SKIP` and `LIMIT` combine for pagination: `SKIP 10 LIMIT 10` returns the 11th to 20th rows. +- `SKIP` and `LIMIT` take constant expressions. +- `RETURN DISTINCT ... ORDER BY` can only sort by returned columns. +- Nulls sort last in ascending order and first in descending order. + +## WITH + +`WITH` pipes the result of one part of a query into the next. It is how you filter on aggregates and chain several stages: + +```cypher +MATCH (n:Person) +WITH n +WHERE n.age > 28 +RETURN n.name + +MATCH (n:Person) +WITH n.name AS name, n.age AS age +ORDER BY age DESC LIMIT 2 +RETURN name, age + +MATCH (n:Person) +WITH n +WHERE n.age > 26 +WITH n +WHERE n.age < 36 +RETURN n.name + +MATCH (n:Person) +WITH n.city AS city, count(*) AS c +WHERE c > 1 +RETURN city, c +``` + +## UNION + +```cypher +MATCH (p:Person) +RETURN p.name AS name +UNION +MATCH (c:Company) +RETURN c.name AS name + +MATCH (p:Person) +RETURN p.name AS name +UNION ALL +MATCH (c:Company) +RETURN c.name AS name +``` + +`UNION` removes duplicates and `UNION ALL` keeps them. All branches must return the same column names, and each column must keep one value type across the branches. + +## Aggregation + +Grouping is implicit, as in Neo4j: the non-aggregated items in the `RETURN` are the grouping keys. + +```cypher +MATCH (a:Person)-[:WORKS_AT]->(c:Company) +RETURN c.name, count(a) AS employees, collect(a.name) AS who, avg(a.age) AS avgAge +ORDER BY employees DESC +``` + +```cypher +MATCH (n:Person) +RETURN count(*), count(n.age), sum(n.age), avg(n.age), min(n.age), max(n.age) + +MATCH (a)-[w:WORKS_AT]->(c) +RETURN w.role, count(DISTINCT c) AS companies + +MATCH (n:Person) +RETURN n.city, max(n.age) - min(n.age) AS spread + +MATCH (n:Person) +RETURN sum(CASE WHEN n.age > 30 THEN 1 ELSE 0 END) AS seniors + +MATCH (n:Person) +RETURN collect(n.name)[0..3] AS firstThree + +MATCH ()-[r]->() +RETURN type(r) AS t, count(*) AS c +ORDER BY c DESC + +MATCH (n) +RETURN labels(n) AS l, count(*) AS c +ORDER BY c DESC +``` + +| Aggregate | Notes | +| --------- | ----- | +| `count(x)`, `count(*)`, `count(DISTINCT x)` | `count(x)` ignores nulls | +| `collect(x)`, `collect(DISTINCT x)` | Nulls are dropped | +| `sum(x)` | | +| `avg(x)` | Always a Double | +| `min(x)`, `max(x)` | Numbers, strings, booleans, `DateTime` and `Duration` | + +Statistical aggregates such as `stdev` and `percentileCont` are not yet available. Compute them in your application, for example in pandas. + +**Rules:** + +- `WHERE` runs before aggregation, so to filter on an aggregate, compute it in a `WITH` first and filter on its alias (see the example below). +- To aggregate an aggregate, compute the first one in a `WITH` and aggregate the result in the next stage. +- Aggregates can be combined with each other and with constants in one expression (`max(n.age) - min(n.age)`, `count(*) * 10`). To use a property in the same expression, return it as its own item so it is a grouping key: `RETURN n.age, n.age + count(*)` works. +- With no grouping keys, an empty input still returns one row: `count` is `0`, `sum` is `0`, `avg` is `null` and `collect` is `[]`. With grouping keys, an empty input returns no rows. + +### Filtering on an aggregate + +```cypher +MATCH (n:Person) +WITH n.city AS city, count(*) AS people +WHERE people > 1 +RETURN city, people +``` + +## UNWIND + +`UNWIND` expands a list into one row per element. It accepts any list expression, including lists you computed earlier in the query: + +```cypher +UNWIND [1, 2, 3] AS x +RETURN x * 10 + +UNWIND range(1, 10, 3) AS x +RETURN x + +MATCH (n:Person) +WITH collect(n.name) AS names +UNWIND names AS nm +RETURN nm + +MATCH (n:Person) +WITH collect(n) AS ns +UNWIND ns AS p +RETURN p.name + +MATCH (n:Person) +UNWIND n.tags AS t +RETURN t, count(*) +``` + +`range(start, end[, step])` includes both bounds. Unwound nodes stay nodes, so `p.name` works after `UNWIND ns AS p`. You can also `UNWIND` a list of maps and read their keys: `UNWIND [{a: 1}, {a: 2}] AS m RETURN m.a`. + + + Write the list into the query text, for example `f"UNWIND {ids} AS x ..."` in Python. Query parameters such as `UNWIND $list` are not yet supported. + + +## Lists + +```cypher +RETURN [1, 2, 3, 4, 5][0], [1, 2, 3, 4, 5][-1], [1, 2, 3, 4, 5][1..3], [1, 2, 3][10] + +RETURN size([1, 2]), head([1, 2]), last([1, 2]), tail([1, 2]), reverse([1, 2, 3]) + +MATCH (n:Person {name: 'Alice'}) +RETURN n.tags[0], size(n.tags) + +RETURN [1, 2] + [3], 'a' || 'b' +``` + +- Indexing and slicing work on literals, list properties and function results (`labels(n)[0]`). Negative indexes count from the end, and an out-of-range index returns `null`. +- Lists and maps compare by value with `=`. +- `size(null)` and `head([])` return `null`. + +### List comprehensions + +```cypher +RETURN [x IN range(1, 6) WHERE x % 2 = 0 | x * x] + +MATCH (a:Person) +RETURN a.name, [(a)-[:KNOWS]->(b) | b.name] AS friends + +MATCH (n:Person) +WHERE any(t IN n.tags WHERE t = 'vip') +RETURN n.name + +RETURN reduce(total = 0, x IN [1, 2, 3] | total + x) +``` + +- `[x IN list WHERE condition | expression]` filters and maps a list. +- A **pattern comprehension** `[(a)-[:KNOWS]->(b) | b.name]` collects a value for every match of a pattern. +- `all`, `any`, `none` and `single` test a condition over a list: `any(t IN n.tags WHERE t = 'vip')`. +- `reduce(acc = init, x IN list | expression)` folds a list into a single value. + +## CASE + +```cypher +MATCH (n:Person) +RETURN n.name, + CASE WHEN n.age >= 35 THEN 'senior' WHEN n.age IS NULL THEN 'unknown' ELSE 'junior' END AS band, + CASE n.city WHEN 'London' THEN 'UK' WHEN 'Paris' THEN 'FR' ELSE 'other' END AS country +``` + +The simple form (`CASE n.city WHEN 'London' THEN ...`) and the searched form (`CASE WHEN condition THEN ...`) both work. TuringDB also accepts comparison tests in the simple form, such as `CASE n.age WHEN > 30 THEN 'old' WHEN IS NULL THEN '?' END`. diff --git a/query/cheatsheet.mdx b/query/cheatsheet.mdx index 8340edb..134d321 100644 --- a/query/cheatsheet.mdx +++ b/query/cheatsheet.mdx @@ -1,477 +1,406 @@ --- title: "TuringDB Query Language Cheatsheet" sidebarTitle: "CheatSheet" -description: "A compact reference for querying, creating, and exploring graphs in TuringDB using Cypher-like syntax + TuringDB-specific extensions." +description: "A compact reference for querying, writing and exploring graphs in TuringDB v3: Cypher plus TuringDB-specific extensions." --- -## MATCH: Pattern Matching +Each section links to the page that explains it in full. -```jsx -MATCH (n) RETURN n -MATCH (n:Person {name: 'Alice'}) RETURN n.age -MATCH (a:Person)-->(b:Company) RETURN a.name, b.name -``` +## MATCH: pattern matching -- `()` → Node -- `[]` → Edge -- `-` → Undirected edge -- Node labels: `:Label` - e.g. `(n:Person)` -- Edge types: `:EDGE_TYPE` - e.g. `-[:KNOWS]->` (one edge type per edge) -- Properties: `{key: 'value', age: 30}` +[Matching patterns](/query/matching) -## CREATE: Add Nodes & Edges +```cypher +MATCH (n) +RETURN n -```jsx -CREATE (:Person {name: 'Alice', age: 30}) -CREATE (:Person {name: 'Mick'})-[:FRIEND_OF]->(:Person {name: 'John'}) -CREATE (a)-[:EDGE]->(b), (b)-[:EDGE]->(c), (c)-[:EDGE]->(a) - -CREATE (gabby:Person {name: 'Gabby', age: 27}), -(susan:Person {name: 'Susan', age: 35}), -(lynette:Person {name: 'Lynette', age: 41}), -(bree:Person {name: 'Bree', age: 42}), -(gabby)-[FRIEND_OF]->(susan), -(gabby)-[FRIEND_OF]->(lynette), -... -``` +MATCH (n:Person {name: 'Alice'}) +RETURN n.age -- Labels **required** for node creation -- No `RETURN` clause -- Use commas to create multiple items at once +MATCH (a:Person)-->(b:Company) +RETURN a.name, b.name -## MATCH + CREATE and MATCH + CREATE + RETURN +MATCH (a:Person)-[e:KNOWS {since: 2010}]->(b:Person) +RETURN a.name, e.since, b.name -```jsx -// Create two nodes Person -CREATE (:Person {name: 'Alice', age: 24}), - (:Person {name: 'John', age: 27}) +MATCH (a)-[r:KNOWS|WORKS_AT]->(b) +RETURN a.name, type(r), b.name -// Match two nodes to create the edge between them -MATCH (n:Person {name: 'Alice'}), (m:Person {name: 'John'}) -CREATE (n)-[:FRIENDS_WITH]->(m) +MATCH (a:Person)-[e]-(b) +RETURN a.name, b.name -// RETURN clause can also be added -MATCH (n:Person {name: 'Alice'}), (m:Person {name: 'John'}) -CREATE (n)-[:FRIENDS_WITH]->(m) -RETURN n.name, n.age, m.name, m.age +MATCH (a:Person WHERE a.age > 30)-[r:KNOWS WHERE r.since > 2012]->(b) +RETURN b.name + +MATCH (n:Person) +ORDER BY n.age LIMIT 2 +RETURN n.name ``` -## SET: Update Properties +- `()` is a node, `[]` is an edge, `-` ignores direction. +- Node labels use `:Label` and can be stacked: `(n:Person:Employee)`. +- Edge types use `:EDGE_TYPE`: `-[:KNOWS]->`. An edge has exactly one type, and `[:A|B]` matches either. +- `RETURN n` gives the internal node ID. Return properties explicitly: `RETURN n.name`. +- Unknown labels and properties are forgiving: they match nothing or read as `null`. -```jsx -// Update a single property -MATCH (n:Person {name: 'Alice'}) -SET n.age = 30 +## OPTIONAL MATCH -// Update multiple properties at once -MATCH (n:Person {name: 'Alice'}) -SET n.position = 'Developer', n.surname = 'Taylor' +```cypher +MATCH (a:Person) +OPTIONAL MATCH (a)-[:WORKS_AT]->(c:Company) +RETURN a.name, c.name -// Update from an expression -MATCH (p:Product) -SET p.discountPrice = p.price * (1 - 0.15) +MATCH (a:Person) +OPTIONAL MATCH (a)-[:KNOWS]->(b) +RETURN a.name, count(b) +``` -// Update edge property -MATCH (n:Person {name: 'Alice'})-[e:KNOWS]->(m:Person {name: 'Bob'}) -SET e.since = 2020 +## WHERE: filter -// Set an embedding vector -MATCH (n:Person {name: 'Alice'}) -SET n.emb = (1.2, 2.0, 0.0, 12.0) +```cypher +MATCH (n) +WHERE n:Person +RETURN n.name + +MATCH (n:Person) +WHERE n.age >= 18 AND n.city = 'London' +RETURN n.name + +MATCH (n:Person) +WHERE n.name STARTS +WITH 'A' OR n.name CONTAINS 'ro' +RETURN n.name + +MATCH (n:Person) +WHERE n.name IN ['Alice', 'Bob'] +RETURN n.name -// Set an embedding vector by node ID -MATCH (n) WHERE n = 1234 -SET n.emb = (1.2, 2.0, 0.0, 12.0) +MATCH (n:Person) +WHERE 'admin' IN n.tags +RETURN n.name -// Conditional update with WHERE MATCH (n:Person) -WHERE n.age < 18 -SET n.isMinor = true +WHERE n.tags IS NOT NULL +RETURN n.name + +MATCH (n)-[e]->(m) +WHERE e:KNOWS AND e.since > 2012 +RETURN n.name, m.name + +MATCH (p:Person) +WHERE NOT (p)-[:KNOWS]->() +RETURN p.name ``` -- Always used with `MATCH`, not `CREATE` -- Creates the property if it does not already exist -- Use `COMMIT` to persist changes +- Comparison: `= <> != < <= > >=`, null tests `IS NULL` / `IS NOT NULL`, boolean `AND OR XOR NOT`. +- Strings and lists: `IN`, `STARTS WITH`, `ENDS WITH`, `CONTAINS`, concatenation with `+` or `||`. +- Use `IS NOT NULL` to test for a property (`=~` and `exists(n.prop)` are not yet supported). +- For Doubles, use ranges (`n.score > 1.99 AND n.score < 2.01`) or convert with `toInteger`. -## WHERE: Filter nodes and edges by properties +## Lookup by native node ID -To filter on node label: +[Details](/query/matching#look-up-nodes-by-native-node-id) -```jsx +```cypher MATCH (n) -WHERE n:Person -RETURN n, n.age -``` +WHERE n = 0 +RETURN n.name -To filter on node property: +MATCH (n)-[:KNOWS]->(m) +WHERE n = 0 +RETURN m.name -```jsx +UNWIND [0, 1, 2] AS x MATCH (n) -WHERE n.name = 'Alice' -RETURN n, n.age +WHERE n = x +RETURN id(n), n.name + +EXPLAIN MATCH (n) +WHERE n = 0 +RETURN n.name ``` -To filter on edge label: +- Compare the **variable** (`n = 0`), not `id(n) = 0`. The first is a direct seek and the second scans every node. +- For batches, `UNWIND` a literal list of distinct IDs and return `id(n)` rather than `x`. -```jsx -MATCH (n)-[e]->(m) -WHERE e:PLAY_POKER_TOGETHER -RETURN n.name, m.age -``` +## Joins and Cartesian products + +```cypher +MATCH (p:Person), (c:Company) +RETURN p.name, c.name -To filter on edge property: +MATCH (a:Person)-[:KNOWS]->(b), (b)-[:WORKS_AT]->(c) +RETURN a.name, b.name, c.name -```jsx -MATCH (n)-[e]->(m:Person) -WHERE n.name = 'Gabby' -AND e.play_poker_together = true -RETURN n.name, m.age +MATCH (a:Person)-->(i:Interest)<--(b:Person) +WHERE a <> b +RETURN a.name, b.name, i.name + +MATCH (a:Person), (b:Person) +WHERE a.city = b.city AND a <> b +RETURN a.name, b.name ``` -## Edge type filtering +- A comma separates independent patterns. Shared variables create joins, unrelated patterns create Cartesian products (N × M rows). -```jsx -// Match only edges of a given type -MATCH (n:Person)-[:KNOWS]->(m:Person) RETURN n.name, m.name +## Variable-length and quantified paths -// Name the edge to also return its properties -MATCH (n:Person)-[e:KNOWS]->(m:Person) RETURN n.name, e.since, m.name +[Paths](/query/paths) -// One edge type per hop -MATCH (a:Person)-[:KNOWS]->(b:Person)-[:WORKS_AT]->(c:Company) RETURN a.name, b.name, c.name +```cypher +MATCH (a:Person {name: 'Alice'})-[:KNOWS]->+(b) +RETURN DISTINCT b.name -// List the edge types in the graph -CALL db.edgeTypes() -``` +MATCH (a:Person {name: 'Alice'})-[:KNOWS]->*(b) +RETURN DISTINCT b.name -- An edge has exactly one edge type, so a pattern takes a single `[:EDGE_TYPE]` constraint +MATCH (a:Person {name: 'Alice'})-[:KNOWS]->{1,2}(b) +RETURN DISTINCT b.name -## SKIP and LIMIT +MATCH (a:Person {name: 'Alice'})-[:KNOWS*1..3]->(b) +RETURN DISTINCT b.name -```jsx -// Return first 10 results -MATCH (n) RETURN n LIMIT 10 +MATCH (a:Person {name: 'Alice'}) ((x)-[:KNOWS]->(y) WHERE y.age > x.age)+ (b) +RETURN b.name -// Pagination: skip 10, return next 10 -MATCH (n) RETURN n SKIP 10 LIMIT 10 +MATCH p = (a:Person {name: 'Alice'})-[:KNOWS]->+(b) +RETURN [n IN nodes(p) | n.name], length(p) ``` -## ORDER BY +- Postfix `->*` is **0** or more hops. Bracket `-[:KNOWS*]->` is **1** or more. +- Each edge is used at most once per path. Bound the length on large graphs. -```jsx -// Sort ascending (default) -MATCH (n:Person) RETURN n.name, n.age ORDER BY n.age +## WITH, ORDER BY, SKIP, LIMIT, UNION -// Sort descending -MATCH (n:Person) RETURN n.name, n.age ORDER BY n.age DESC +[Results and aggregation](/query/aggregation) -// Sort by multiple properties -MATCH (n:Person) RETURN n.name, n.age, n.city ORDER BY n.city, n.age DESC - -// Combined with pagination -MATCH (n:Person) RETURN n.name, n.age ORDER BY n.age DESC SKIP 10 LIMIT 10 -``` +```cypher +MATCH (n:Person) +WITH n.city AS city, count(*) AS c +WHERE c > 1 +RETURN city, c -## Joins and Cartesian Products +MATCH (n:Person) +RETURN DISTINCT n.city -```jsx -// Cartesian product: all combinations of Person × Interest -MATCH (a:Person), (b:Interest) RETURN a.name, b.name +MATCH (n:Person) +RETURN n.name, n.age +ORDER BY n.age DESC SKIP 1 LIMIT 2 + +MATCH (p:Person) +RETURN p.name AS name +UNION +MATCH (c:Company) +RETURN c.name AS name +``` -// Three-way cartesian product -MATCH (p:Person), (i:Interest), (c:Category) RETURN p.name, i.name, c.name +- Expressions in `WITH` must be aliased. +- Nulls sort last ascending and first descending. -// Join on shared variable: persons sharing an interest -MATCH (a:Person)-->(b:Interest)<--(c:Person) -WHERE a.name <> c.name -RETURN a.name, b.name, c.name +## Aggregation -// With explicit edge types -MATCH (a:Person)-[:INTERESTED_IN]->(b:Interest)<-[:INTERESTED_IN]-(c:Person) -WHERE a.name <> c.name -RETURN a.name, b.name, c.name +```cypher +MATCH (n:Person) +RETURN count(*), count(n.age), sum(n.age), avg(n.age), min(n.age), max(n.age) -// Multi-hop join -MATCH (a:Person)-->(i:Interest)-->(c:Category) -RETURN a.name, i.name, c.name +MATCH (a:Person)-[:WORKS_AT]->(c:Company) +RETURN c.name, count(a) AS employees, collect(a.name) AS who -// Mix path + cartesian product -MATCH (a:Person)-->(i:Interest), (c:Category) -WHERE c.name = 'Cat1' -RETURN a.name, i.name, c.name +MATCH (n:Person) +RETURN count(DISTINCT n.city) -// Two independent paths (cartesian of each path's results) -MATCH (a:Person)-->(i1:Interest), (b:Person)-->(i2:Interest) -WHERE a.name <> b.name -RETURN a.name, i1.name, b.name, i2.name +MATCH ()-[r]->() +RETURN type(r) AS t, count(*) AS c +ORDER BY c DESC ``` -- Comma `,` separates independent patterns -- Shared variables between patterns create joins -- Unrelated patterns create cartesian products (N × M rows) +- Grouping is implicit. To filter on an aggregate, compute it in a `WITH` and filter afterwards. +- `stdev` and `percentile*` aggregates are not yet available. -## Property Match Operators +## UNWIND, lists, CASE -```jsx -MATCH (n {name: 'Alice'}) RETURN n -// exact equivalent of -MATCH (n) WHERE n.name = 'Alice' RETURN n -``` +```cypher +UNWIND [1, 2, 3] AS x +RETURN x * 10 -- `:` → exact match using property filter in node directly -- `=` → exact match using property filter in WHERE clause - -## Boolean operators - -- `OR` - - ```jsx - MATCH (n:Person) - WHERE n.medication = "Aspirin" - OR n.medication = "Ibuprofen" - RETURN n.name - ``` -- `AND` - - ```jsx - MATCH (n) - WHERE n.name = 'Matt' - AND n.age = 20 - AND n.hasPhD = false - RETURN n - ``` - -## Comparison operators - -- Equal: `=` - - ```jsx - # Find antibodies targeting proteins in Human - MATCH (ab:Antibody)-->(prot:Protein) - WHERE prot.host = 'Human' - RETURN ab.name, prot.name - ``` -- Inequal: `<>` - - ```jsx - # Find antibodies (associated to a protein) used together - # in same publication (2-hop) - MATCH (ab1:Antibody)-->(prot:Protein), (ab2:Antibody)-->(prot:Protein) - WHERE ab1.name <> ab2.name - RETURN ab1.name, ab2.name, prot.name, prot.gene_name - ``` -- Less than: `<` -- Less than or equal to: `<=` -- Greater than: `>` -- Greater than or equal to: `>=` - - ```jsx - # Publications published on 2020 or after - MATCH (pub:Publication) - WHERE pub.published_year >= 2020 - RETURN pub.displayName, pub.pubmedid, pub.published_year, pub.country - ``` -- is null:`IS NULL` -- is not null:`IS NOT NULL` - - ```jsx - # Publications from the United States - MATCH (pub:Publication) - WHERE pub.country = 'United States' - AND pub.institution IS NOT NULL - RETURN pub.displayName, pub.institution, pub.published_year - ``` - -## Data Types - -| Type | Example | -| --------- | ------------------------ | -| String | `'hello'` or `"hello"` | -| Integer | `age=30` | -| Boolean | `flag=true` | -| Double | `score=3.14` | -| Embedding | `emb=(1.2, 2.0, 0.0)` | - -## RETURN Clause - -```jsx -MATCH (n)-->(m) RETURN n.name, n.age, m.name, m.age -``` +MATCH (n:Person) +WITH collect(n.name) AS names +UNWIND names AS nm +RETURN nm -- Select fields with `RETURN n.prop, m.prop` +MATCH (n:Person) +UNWIND n.tags AS t +RETURN t, count(*) + +RETURN [x IN range(1, 6) WHERE x % 2 = 0 | x * x] -## Expression evaluation +MATCH (a:Person) +RETURN a.name, [(a)-[:KNOWS]->(b) | b.name] AS friends -```jsx -// Multiply property by constant -MATCH (n) RETURN n.price * 1.1 +MATCH (n:Person) +WHERE any(t IN n.tags WHERE t = 'vip') +RETURN n.name -// Divide two properties -MATCH ()-[r]->() RETURN r.a / r.b +RETURN reduce(total = 0, x IN [1, 2, 3] | total + x) -// Add two properties -MATCH (n) RETURN n.val + n.tax +MATCH (n:Person) +RETURN n.name, CASE WHEN n.age >= 35 THEN 'senior' ELSE 'junior' END AS band ``` -## LOAD graphs +## Subqueries + +[Subqueries](/query/subqueries) -### Load TuringDB graph +```cypher +MATCH (p:Person) +WHERE EXISTS { (p)-[:WORKS_AT]->(:Company {name: 'Acme'}) } +RETURN p.name -Files have to be available in `graphs` subdirectory of `turing-dir`. +MATCH (p:Person) +RETURN p.name, COUNT { (p)-[:KNOWS]->() } AS outDegree -```jsx -LOAD GRAPH graph_name +MATCH (p:Person) +CALL (p) { + MATCH (p)-[:KNOWS]->(f) + RETURN f.name AS friend LIMIT 1 +} +RETURN p.name, friend + +MATCH (p:Person) +OPTIONAL CALL (p) { + MATCH (p)-[:WORKS_AT]->(c) + RETURN c.name AS co +} +RETURN p.name, co ``` -### Load external data into TuringDB graph +## Functions + +[Functions and data types](/query/functions) -Files have to be available in `data` subdirectory of `turing-dir`. +| Category | Functions | +| -------- | --------- | +| Entity | `id`, `labels`, `type`, `startNode`, `endNode` | +| Path and list | `nodes`, `relationships`, `length`, `size`, `head`, `last`, `tail`, `reverse`, `range`, `reduce` | +| Conversion | `toInteger`, `toFloat`, `toString`, `toBoolean`, `coalesce` | +| String | `toUpper`, `toLower`, `trim`, `ltrim`, `rtrim`, `reverse` | +| Math | `abs`, `sign`, `ceil`, `floor`, `round`, `sqrt`, `exp`, `log`, `log10`, `pi`, `e`, `sin`, `cos`, `tan` | +| Temporal | `datetime`, `duration` | +| Embedding | `cosine_similarity`, `euclidean_distance` | +| Aggregate | `count`, `collect`, `sum`, `avg`, `min`, `max` | -```jsx -LOAD GML 'mygml.gml' AS my_graph -``` +```cypher +MATCH (n)-[e]->(m) +RETURN labels(n), type(e), labels(m) + +RETURN datetime('2024-03-15T10:30:00Z') + duration({days: 1}) -```jsx -LOAD JSONL 'myjsonl.jsonl' AS my_graph +MATCH (n:Document) +RETURN n.title, cosine_similarity(n.emb, (0.4, 0.3, 0.8, 0.1)) ``` -## Procedures (`CALL`) +- Function names are case-sensitive, except `count` and `collect`. +- Use `type(e)` for an edge's type (it replaces `edgeType()`). -| Query | Description | -| ------------------------- | ----------------------------------- | -| `CALL db.labels()` | All node labels | -| `CALL db.propertyTypes()` | All node/edge property keys & types | -| `CALL db.edgeTypes()` | All edge types | -| `CALL db.history()` | Commit history | -| `CALL db.describeCommit('')` | Counts for a single commit | -| `CALL db.procedures()` | List available procedures | -| `CALL db.showIndexes()` | List property indexes | +## Data types -Procedure outputs can be `YIELD`ed and chained: `CALL db.labels() YIELD label MATCH (n) WHERE n.name = label RETURN n`. +| Type | Example | +| ---- | ------- | +| String | `'hello'` or `"hello"` | +| Integer | `30`, `0x1F` | +| Double | `3.14` | +| Boolean | `true` | +| List | `[1, 'a', [2]]` | +| Map | `{a: 1}` | +| DateTime | `datetime('2024-03-15T10:30:00Z')` | +| Duration | `duration({days: 2})` | +| Embedding | `(1.2, 2.0, 0.0)` (parentheses, not brackets) | -## DELETE +## Writing (inside a change) -```jsx -// Delete a node -MATCH (n:Person {name: 'Alice'}) DELETE n +[Writing data](/query/writing) -// Delete an edge -MATCH (n)-[e:KNOWS]->(m) DELETE e -``` +```cypher +CHANGE NEW -## Variable-length paths +CREATE (:Person {name: 'Alice', age: 30}) -```jsx -// Postfix quantifiers (not Neo4j's -[*1..3]-) -MATCH (a:Person)-[e]->+(b:Person) RETURN a, b // one or more hops -MATCH (a:Person)-[e]->*(b:Person) RETURN a, b // zero or more hops -MATCH (a:Person)-[e]->{2,4}(b:Interest) RETURN a, b // 2 to 4 hops -``` +CREATE (:Person {name: 'Mick'})-[:FRIEND_OF {since: 2020}]->(:Person {name: 'John'}) -## Graph algorithms +COMMIT -| Function | Description | -| ------------------------------------------ | -------------------------------------------------------------- | -| `shortestPath(n, m, distanceProperty, distOutputVar, pathOutputVar)` | Shortest path between nodes using the Dijkstra processor | +MATCH (a:Person {name: 'Alice'}), (b:Person {name: 'Mick'}) +CREATE (a)-[:KNOWS]->(b) -## Functions +MERGE (n:Person {name: 'Alice'}) + ON CREATE SET n.created = true + ON MATCH SET n.seen = true -| Function | Description | -| ----------------- | ------------------------------------------------------------------ | -| `labels(n)` | Node label as a string | -| `edgeType(e)` | Edge type as a string | -| `toInteger(expr)` | Parses a string as an integer | -| `toFloat(expr)` | Parses a string as a float | -| `toBoolean(expr)` | Parses a string as a boolean | -| `cosine_similarity(a, b)` | Cosine similarity of two embeddings | -| `euclidean_distance(a, b)` | Euclidean distance of two embeddings | - -## Aggregate Functions - -| Function | Description | -| ----------------- | ------------------------------------------------------------------ | -| `count(expr)` | Returns the number of non-null rows in the column corresponding to `expr` | -| `avg(expr)` | Returns the average value of non-null rows in the numeric-valued column corresponding to `expr` | - -```jsx -// Introspection -MATCH (n)-[e]->(m) RETURN labels(n), edgeType(e) - -// Type conversion -MATCH (n) WHERE n.year > toInteger("2020") RETURN n.name -MATCH (n) RETURN n.price * toFloat("1.07") - -// Aggregation -MATCH (n:Person) RETURN count(*) -``` +MATCH (n:Person {name: 'Alice'}) +SET n.age = 31, n.position = 'Developer' -## Engine Commands - -| Command | Description | -| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| `CREATE GRAPH ` | Create a new named graph | -| `LOAD GRAPH ` | Load an existing graph | -| `LOAD GML 'mygraph.gml' AS my_graph` | Load the specified GML as TuringDB graph. Requires the GML to be accessible in TuringDB directory (`-turing-dir`, `data` subdir) | -| `LOAD JSONL 'mygraph.jsonl' AS my_graph` | Load the specified JSONL as TuringDB graph. Requires the JSONL to be accessible in TuringDB directory (`-turing-dir`, `data` subdir) | -| `COMMIT` | Persist intermediate state within a change (needed between separate node/edge create steps) | -| `LOAD COMMIT ''` | Load a past commit into memory (needed for REST API queries on non-HEAD commits; CLI and SDK do this automatically) | -| `LIST GRAPH` | List all available graphs | -| `HISTORY` | Show commit history (versioning) | -| `CHANGE NEW` | Create a new isolated change | -| `CHANGE SUBMIT` | Merge current change into main | -| `CHANGE DELETE` | Delete current change | -| `CHANGE LIST` | List active uncommitted changes | -| `CREATE VECTOR INDEX WITH DIMENSION METRIC ` | Create a vector index | -| `VECTOR SEARCH IN FOR () YIELD ids` | k-nearest-neighbor vector search | -| `SHOW VECTOR INDEXES` / `DELETE VECTOR INDEX ` | List / delete vector indexes | -| `CREATE INDEX FOR (n) ON n.prop` / `FOR [e] ON e.prop` | Create a property index (write; commit to take effect) | -| `DROP INDEX ` | Drop a property or vector index | -| `MERGE_DATAPARTS` | Compact the current graph's DataParts into one | -| `INSTALL ` / `SHOW EXTENSIONS` / `SHOW PROCEDURES` | Load a procedure extension / list extensions / list procedures | -| `LOAD CSV "" [WITH HEADERS] [ON ERROR SKIP\|FAIL] AS row` | Stream CSV rows (see [CSV import](/import_data/csv)) | - -## UNWIND - -```jsx -// Expand a literal list into rows (literal lists only) -UNWIND [1, 2, 3] AS x RETURN x - -// Return a list literal directly -RETURN [1, 2, 3] AS nums -``` +MATCH (n:Person {name: 'Alice'}) +REMOVE n.position -## Property indexes +MATCH (n:Person {name: 'Alice'}) +DETACH DELETE n -```jsx -// Indexes are writes — create inside a change, then commit -CHANGE NEW -CREATE INDEX age_index FOR (n) ON n.age -CHANGE SUBMIT -MATCH (n) WHERE n.age = 30 RETURN n // optimizer uses the index +MATCH (n:Person)-[e:KNOWS]->(m) +DELETE e -DROP INDEX age_index // remove +CHANGE SUBMIT ``` -## Useful Patterns +- Every node needs at least one label and every edge exactly one type. Labels are set at creation. +- Run `COMMIT` after creating entities that a later query must match. `CHANGE SUBMIT` commits for you. +- In the Python SDK, call `client.checkout()` after `CHANGE SUBMIT`. +- For more than a few thousand entities, use [`LOAD PARQUET`](/import_data/parquet). + +## Procedures -```jsx -// Two-hop connection -MATCH (a)-[r1]->(b)-[r2]->(c) RETURN c +[Procedures and commands](/query/procedures) -// All people named John -MATCH (n:Person {name: 'John'}) RETURN n -/// exact equivalent of -MATCH (n:Person) WHERE n.name = 'John' RETURN n +| Query | Description | +| ----- | ----------- | +| `CALL db.labels()` | All node labels | +| `CALL db.edgeTypes()` | All edge types | +| `CALL db.propertyTypes()` | All property keys and their types | +| `CALL db.history()` | Commit history | +| `CALL db.describeCommit('')` | Counts for a single commit | +| `CALL db.getNodes([0, 1])` | Labels and all properties of specific nodes | +| `CALL db.procedures()` | All procedures and signatures | -// Find co-authors: two persons linked to the same publication -MATCH (a:Person)-->(p:Publication)<--(b:Person) -WHERE a.name <> b.name -RETURN a.name, b.name, p.title +Outputs can be chained: `CALL db.labels() YIELD label MATCH (n) WHERE label IN labels(n) RETURN label, count(n)`. -// Match existing nodes then create edge between them -MATCH (n:Person {name: 'Alice'}), (m:Person {name: 'Bob'}) -CREATE (n)-[:KNOWS]->(m) +## Graph algorithms and search -// Cartesian: pair every person with every city -MATCH (p:Person), (c:City) RETURN p.name, c.name +```cypher +MATCH (a:Station {name: 'Ashchurch'}), (b:Station {name: 'Pershore'}) +shortestPath(a, b, distance, dist, path) +RETURN dist, path + +VECTOR SEARCH IN my_index FOR 5 (1.2, 0.5, 3.0, 0.1) YIELD ids, score +RETURN ids, score ``` + +See [Shortest path](/shortest-path) and [Vector search](/vector-search). + +## Engine commands + +| Command | Description | +| ------- | ----------- | +| `CREATE GRAPH ` | Create a new named graph | +| `LOAD GRAPH ` | Load an existing graph | +| `LOAD GML 'f.gml' AS g` | Load a [GML](/import_data/gml) file as a graph (file in `/data`) | +| `LOAD JSONL 'f.jsonl' AS g` | Load a [JSONL](/import_data/jsonl) file as a graph, optionally `WITH EMBEDDINGS`, `WITH DATETIMES`, `WITH DURATIONS` | +| `LOAD PARQUET '' AS g` | Load [Parquet](/import_data/parquet) files as a graph | +| `LOAD CSV 'f.csv' [WITH HEADERS] [ON ERROR SKIP\|FAIL] AS row` | Stream rows from a [CSV](/import_data/csv) file | +| `LIST GRAPH` / `LIST AVAILABLE GRAPHS` | Loaded graphs / graphs on disk | +| `CHANGE NEW` / `SUBMIT` / `DELETE` / `LIST` | Create, merge, discard and list changes | +| `COMMIT` | Seal pending writes inside a change | +| `LOAD COMMIT ''` | Load a past commit (REST API only; the CLI and SDK do it for you) | +| `MERGE_DATAPARTS` | Compact the graph's DataParts into one (renumbers nodes) | +| `CREATE VECTOR INDEX WITH DIMENSION METRIC ` | Create a vector index | +| `SHOW VECTOR INDEXES` / `DELETE VECTOR INDEX ` | List / delete vector indexes | +| `INSTALL ` / `SHOW EXTENSIONS` / `SHOW PROCEDURES` | Load a procedure extension / list extensions / list procedures | +| `EXPLAIN ` | Show the query plan without running it | diff --git a/query/cypher_subset.mdx b/query/cypher_subset.mdx index 3a0d2fa..7f57456 100644 --- a/query/cypher_subset.mdx +++ b/query/cypher_subset.mdx @@ -1,621 +1,112 @@ --- title: "Query Language in TuringDB" -sidebarTitle: "Query Language" -description: "TuringDB uses Cypher as a query language and has some unique types of queries" +sidebarTitle: "Overview" +description: "TuringDB v3 runs Cypher on a new engine and supports most of openCypher: OPTIONAL MATCH, WITH, UNWIND, UNION, MERGE, variable-length paths, subqueries and more." --- -TuringDB supports most of the **Cypher query language**, extended with versioning, metadata search, and flexible property matching. +TuringDB v3 (`turingdb` 3.0 on PyPI) runs every query on a new MLIR-based engine and supports **most of openCypher**, extended with versioning, vector search and fast lookup by native node ID. -This guide covers some examples of queries types, including `MATCH`, `CREATE`, property filters, procedures, and available data types. +If you know Cypher from Neo4j, you can write the queries you are used to. This page covers the basics and the handful of places where TuringDB differs. The rest of the language is documented in the pages that follow. -# Basics +## What is supported -Queries are built by referencing nodes and edges. TuringDB supports the standard CYPHER syntax, in that nodes are denoted using parentheses `()`, whilst edges are denoted using square brackets `[]`. +| Area | Supported | +| ---- | --------- | +| Reading | `MATCH`, `OPTIONAL MATCH`, `WHERE`, `WITH`, `RETURN [DISTINCT]`, `ORDER BY`, `SKIP`, `LIMIT`, `UNWIND`, `UNION [ALL]`, `LOAD CSV` | +| Patterns | Multiple labels, edge type alternatives (`[:A\|B]`), inline `WHERE`, variable-length paths (`*1..3`, `->+`, `{2,4}`), quantified path patterns, named paths, pattern predicates | +| Expressions | Arithmetic, string and list concatenation, `CASE`, list and pattern comprehensions, `reduce`, `all` / `any` / `none` / `single`, list slicing, maps | +| Aggregation | `count`, `sum`, `avg`, `min`, `max`, `collect`, `DISTINCT`, implicit grouping, aggregates inside expressions | +| Subqueries | `EXISTS { }`, `COUNT { }`, `CALL { }` and `OPTIONAL CALL { }` | +| Writing | `CREATE`, `MERGE` with `ON CREATE` / `ON MATCH`, `SET`, `REMOVE`, `DELETE`, `DETACH DELETE` | +| Types | String, Integer, Double, Boolean, List, Map, `DateTime`, `Duration`, Embedding | +| TuringDB extras | Changes, commits and time travel, `shortestPath`, `VECTOR SEARCH`, `gnn.graphSAGE`, lookup by native node ID | -For example, `(n)` would denote a node named `n`, whilst `[e]` would denote an edge named `e`. +## Basics -Nodes and edges can have both **label** and **property** constraints. As in standard CYPHER, property constraints are specified using curly brackets `{}`, whilst labels are specified using colon `:` syntax. +Queries are built from **nodes** and **edges**: -An example of a node with a label constraint would be `(n:Person)`. The edge equivalent is the **edge type**, written the same way: `[e:FRIENDS_WITH]`. An edge has exactly one edge type. - -An example of an edge with a property constraint would be `[e {duration: 10}]`. - -Property and label constraints may be combined, for instance, `(n:Person {name: 'John'})` specifies a node which both has the label `Person`, and a `name` property with the value `John`. - -Nodes and edges can have label and property multiple constraints, which are specified in a comma-separated list: `(n:Person:Man {name: 'John', age: 20})`. These lists can be arbitrarily long. - -By convention: - -- we prefer using single quotes around strings (even if double quotes alsowork ) -- node label are written in Pascal case (no spaces): e.g. `Person`, `BankAccount`, `BloodType` -- edge label are written in upper case (spaces replaced by underscores): e.g. `TRANSACTION`, `FRIENDS_WITH`, `IS_CLIENT_OF` - -### Summary: - -Queries are built around **nodes** and **edges**: - -- **Nodes** are written in parentheses `()` e.g. `(n)` - a node with alias `n` -- **Edges** are written in square brackets `[]` e.g. `[e]` - an edge with alias `e` - -You can add: - -- **Labels** with a colon `:` - e.g. `(n:Person)` -- **Edge types** with a colon `:` - e.g. `[e:FRIENDS_WITH]` or `[:FRIENDS_WITH]` -- **Property constraints** with curly braces `{}` - e.g. `[e {duration: 10}]` -- **Both** at once - e.g. `(n:Person {name: 'John'})` - -Multiple labels and properties can be specified: - -```jsx -(n:Person:Man {name: 'John', age: 20}) -``` - -# Queries - -Queries are built up of combinations of nodes and edges, as specified above. - -## `MATCH` queries - -`MATCH` queries are used to retrieve nodes, edges, and their property values from the database via specification of relationships and properties. - -It is easiest to demonstrate the syntax of a `MATCH` query with a concrete example: - -```jsx -MATCH (n)-[e]->(m) RETURN n, e, m -MATCH (m)<-[e]-(n) RETURN n, e, m // also works -``` - -The above query will look for all the directed edges in the graph. The query will return the internal ID of the node `n`. - - - When using `RETURN n`, other implementations of CYPHER may return all properties of `n`, whilst TuringDB only returns the internal ID of `n`. - - -`MATCH` queries are flexible: they can contain a single node and no edges, or any number of node, edge pairs. For example: - -```jsx -MATCH (n) RETURN n -``` - -will match all nodes in the database. An example of a multi-hop `MATCH` query would be: - -```jsx -MATCH (n)-[e]->(m)-[f]->(p) RETURN n, m, p -``` - - - Queries have _variables_, which in the above examples are those such as `n`, `m`, `e`, etc. Variables are a way to give a name to a node or an edge, so that those nodes or edges can be specified in the `RETURN` clause. However, if you do not want to return an edge or its properties, the edge need not have a name. For example: - - ```jsx - MATCH (n)-->(m) RETURN m - ``` - - Both node and edges can omit a variable name if they specify at least a label constraint: - - ```jsx - MATCH (:Person)-[:FRIENDS_WITH]->(m) RETURN m - ``` - - -You can also return multiple properties using a comma separated list: - -```jsx -MATCH (n:Person)-->(m:Person) RETURN n.name, m.age -``` - -Combining the syntax of `MATCH` queries, and the ability to specify constraints, here are a few examples of some syntactically correct TuringDB `MATCH` queries: - -- `MATCH (n:Person) RETURN n.name` -- `MATCH (:Person)-->(n:Person) RETURN n.name` -- `MATCH (n:Person:Woman:SoftwareEngineer) RETURN n.name` -- `MATCH (n:Person:Woman:SoftwareEngineer)-->(m)-[e]->(p:Man)-[f]->(q) RETURN e, f` - - - Note that whilst all the above the queries are all _syntactically_ valid, if the graph does not have a node property which is used in a query, it will fail to execute. - - -## `CREATE` queries - -`CREATE` queries follow exactly the same syntax as `MATCH` queries when it comes to specifying nodes, edges, and property/label constraints. `CREATE` queries may have `RETURN` clauses, but they do not need them. There is also the additional requirement that all nodes and edges _must have at least one label_. This means a query such as - -```jsx -CREATE (n) -``` - -is not valid, whilst - -```jsx -CREATE (n:Person) -``` - -is a valid query. - -There is no requirement to declare any names for any variables. This means we can have queries such as - -```jsx -CREATE (:Person)-[:FRIENDS_WITH]->(:Person) -``` - -However, naming variables can be useful if you want to create multiple edges to or from a given node. For instance, if you would like to create a triangle pattern, this can be achieved using the following approach - -```jsx -CREATE (a:Corner)-[:EDGE]->(b:Corner), (b)-[:EDGE]->(c:Corner), (c)-[:EDGE]->(a) -``` - -## `MATCH ... CREATE ...` and `MATCH ... CREATE ... RETURN ...` queries - -`MATCH`, `CREATE` and `RETURN` statements can be used together in queries. - -For example, to create a edge between two existing graph, the following queries can be used: - -```jsx -// Create two nodes Person -CREATE (:Person {name: 'Alice', age: 24}) -CREATE (:Person {name: 'John', age: 27}) - -// Match two nodes to create the edge between them -MATCH (n:Person {name: 'Alice'}), (m:Person {name: 'John'}) -CREATE (n)-[:FRIENDS_WITH]->(m) - -// RETURN clause can also be added -MATCH (n:Person {name: 'Alice'}), (m:Person {name: 'John'}) -CREATE (n)-[:FRIENDS_WITH]->(m) -RETURN n.name, n.age, m.name, m.age -``` - -## `WHERE` queries - -The `WHERE` clause allows to filter the results on node and/or edge labels and/or properties. - -To filter on node label: - -```jsx -MATCH (n) -WHERE n:Person -RETURN n, n.age -``` - -To filter on node property: - -```jsx -MATCH (n) -WHERE n.name = 'Alice' -RETURN n, n.age -``` - -To filter on edge label: - -```jsx -MATCH (n)-[e]->(m) -WHERE e:PLAYPOKER -RETURN n.name, m.age -``` - -To filter on edge property: - -```jsx -MATCH (n)-[e]->(m:Person) -WHERE n.name = 'Gabby' -AND e.play_poker = true -RETURN n.name, m.age -``` - -## Filtering on edge types - -`MATCH` queries can be restricted to a given edge type with the `[:EDGE_TYPE]` syntax, the edge counterpart of a node label: - -```cypher -// Only KNOWS edges -MATCH (n:Person)-[:KNOWS]->(m:Person) -RETURN n.name, m.name - -// Name the edge to also return its properties -MATCH (n:Person)-[e:KNOWS]->(m:Person) -RETURN n.name, e.since, m.name - -// Each hop can have its own edge type -MATCH (a:Person)-[:KNOWS]->(b:Person)-[:WORKS_AT]->(c:Company) -RETURN a.name, b.name, c.name -``` - -An edge carries exactly one edge type, so each edge pattern takes a single `[:EDGE_TYPE]` constraint. Use `CALL db.edgeTypes()` to list the edge types available in the graph. - -## Multi-pattern queries (Joins and Cartesian Products) - -TuringDB supports matching multiple patterns in a single query by separating them with commas. Depending on whether the patterns share variables, this results in either a **join** or a **cartesian product**. - -### Cartesian Product - -When patterns in a `MATCH` clause are separated by commas and do **not** share any variables, TuringDB computes a cartesian product of the results. Every row from the first pattern is combined with every row from the second. - -```cypher -MATCH (a:Person), (b:Interest) -RETURN a.name, b.name -``` - -This returns all combinations of Person nodes with Interest nodes. If there are 6 persons and 5 interests, the result contains 30 rows. - -You can use `WHERE` to filter the cartesian product: - -```cypher -MATCH (p:Person), (i:Interest) -WHERE p.isFrench = true AND i.name = 'MegaHub' -RETURN p.name, i.name -``` - -Three or more patterns can be combined: - -```cypher -MATCH (p:Person), (i:Interest), (c:Category) -WHERE p.name = 'A' AND i.name = 'Shared' AND c.name = 'Cat1' -RETURN p.name, i.name, c.name -``` - - - Cartesian products can produce very large result sets. A product of N nodes by M nodes produces N × M rows. Use `WHERE` filters or `LIMIT` to keep result sizes manageable. - - -### Pattern-based Joins - -When two paths converge on a shared node variable, TuringDB performs a **hash join** on that variable. This is useful for finding entities that share a common connection. - -```cypher -MATCH (a:Person)-->(b:Interest)<--(c:Person) -WHERE a.name <> c.name -RETURN a.name, b.name, c.name -``` - -This finds pairs of different persons who share a common interest. The variable `b` acts as the join point. - -Multi-hop joins are also supported: +- Nodes are written in parentheses: `(n)` is a node named `n`. +- Edges are written in square brackets: `[e]` is an edge named `e`. +- **Labels** are added with a colon: `(n:Person)`. A node can have several: `(n:Person:Employee)`. +- **Edge types** use the same syntax: `[e:FRIENDS_WITH]`. An edge has exactly one edge type. +- **Properties** are constrained with curly braces: `(n:Person {name: 'John', age: 20})`. ```cypher -MATCH (a:Person)-->(i:Interest)-->(c:Category) -RETURN a.name, i.name, c.name +MATCH (a:Person {name: 'Alice'})-[e:KNOWS]->(b:Person) +RETURN a.name, e.since, b.name ``` -You can specify edge types explicitly: +By convention, node labels are written in PascalCase (`Person`, `BankAccount`), edge types in upper case with underscores (`WORKS_AT`, `FRIENDS_WITH`), and strings use single quotes. -```cypher -MATCH (a:Person)-[:INTERESTED_IN]->(b:Interest)<-[:INTERESTED_IN]-(c:Person) -WHERE a.name <> c.name -RETURN a.name, b.name, c.name -``` - -### Mixing Paths and Cartesian Products - -Connected paths and independent patterns can be combined in the same query. Shared variables create joins, while unrelated patterns create cartesian products. - -```cypher -MATCH (a:Person)-->(i:Interest), (c:Category) -WHERE c.name = 'Cat1' -RETURN a.name, i.name, c.name -``` - -This returns every Person→Interest connection combined with the Cat1 category node. - -Two independent paths can also be combined: - -```cypher -MATCH (a:Person)-->(i1:Interest), (b:Person)-->(i2:Interest) -WHERE a.name = 'Alice' AND b.name = 'Bob' AND i1.name <> i2.name -RETURN a.name, i1.name, b.name, i2.name -``` - -This finds all combinations of Alice's interests with Bob's interests, excluding pairs where both interests are the same. - -## `LIMIT` keyword - -**LIMIT** restricts the number of returned results. The following query returns only the first 10 results: - -```cypher -MATCH (n) -RETURN n -LIMIT 10 -``` - -## `SKIP` keyword - -**SKIP** skips the first N results before returning the rest. The following query skips the first 10 results: - -```cypher -MATCH (n) -RETURN n -SKIP 10 -``` - -`SKIP` and `LIMIT` can be combined for pagination. The following query will return the 11th to 20th results: - -```cypher -MATCH (n) -RETURN n -SKIP 10 -LIMIT 10 -``` - -## Sorting with `ORDER BY` - -**ORDER BY** sorts the results by one or more properties. By default, results are sorted in ascending order. - -```cypher -MATCH (n:Person) -RETURN n.name, n.age -ORDER BY n.age -``` - -Use `DESC` to sort in descending order: - -```cypher -MATCH (n:Person) -RETURN n.name, n.age -ORDER BY n.age DESC -``` - -You can sort by multiple properties. Results are sorted by the first property, then ties are broken by subsequent properties: +Keywords are case-insensitive (`MATCH` and `match` are the same). Labels, edge types and property names are case-sensitive. Only `//` starts a comment: ```cypher -MATCH (n:Person) -RETURN n.name, n.age, n.city -ORDER BY n.city, n.age DESC -``` - -`ORDER BY` can be combined with `SKIP` and `LIMIT` for sorted pagination: - -```cypher -MATCH (n:Person) -RETURN n.name, n.age -ORDER BY n.age DESC -SKIP 10 -LIMIT 10 -``` - -# Data Types - -TuringDB offers the following data types for node and edge properties: - -- String -- Boolean -- Integer (signed) -- Double (decimal) -- Embedding (vector of floats, e.g. `(1.2, 2.0, 0.0)`) - -String properties can be enclosed using double quotes (`"`), single quotes (`'`), or backticks (\`\`\`). - -# Operators - -## Boolean operators - -The `OR` and `AND` operartors are used to filter on multiple conditions: - -```jsx -MATCH (n:Person) -WHERE n.medication = "Aspirin" -OR n.medication = "Ibuprofen" -RETURN n.name -``` - -```jsx -MATCH (n) -WHERE n.name = 'Matt' -AND n.age = 20 -AND n.hasPhD = false -RETURN n +// a line comment +MATCH (n:Person) RETURN n.name // also fine at the end of a line ``` -## Comparison operators + + Use `//` for comments. SQL-style `--` and `/* */` comments are not yet supported. + -TuringDB allows you to query against node and edge properties using the `:` operator for exact matching of the property value. +### Running a query -```jsx -MATCH (n {name: 'Matt', age: 20, hasPhD: false}) RETURN n -``` +Queries run through the Python SDK, the interactive shell, or the HTTP endpoint. Read queries run directly against the current graph; [write queries](/query/writing) must run inside a change. -You can also pass through the WHERE clause to do the exact equivalent query: +```python +from turingdb import TuringDB -```text -MATCH (n) -WHERE n.name = 'Matt' -AND n.age = 20 -AND n.hasPhD = false -RETURN n +client = TuringDB(host="http://localhost:6666") +df = client.query("MATCH (n:Person) RETURN n.name, n.age") # a pandas DataFrame ``` -Implemented comparison operators: - -- Equal: `=` - - ```text - # Find antibodies targeting proteins in Human - MATCH (ab:Antibody)-->(prot:Protein) - WHERE prot.host = 'Human' - RETURN ab.name, prot.name - ``` -- Inequal: `<>` - - ```jsx - # Find antibodies (associated to a protein) used together - # in same publication (2-hop) - MATCH (ab1:Antibody)-->(prot:Protein), (ab2:Antibody)-->(prot:Protein) - WHERE ab1.name <> ab2.name - RETURN ab1.name, ab2.name, prot.name, prot.gene_name - ``` -- Less than: `<` -- Less than or equal to: `<=` -- Greater than: `>` -- Greater than or equal to: `>=` - - ```jsx - # Publications published on 2020 or after - MATCH (pub:Publication) - WHERE pub.published_year >= 2020 - RETURN pub.displayName, pub.pubmedid, pub.published_year, pub.country - ``` -- is null:`IS NULL` -- is not null:`IS NOT NULL` - - ```jsx - # Publications from the United States - MATCH (pub:Publication) - WHERE pub.country = 'United States' - AND pub.institution IS NOT NULL - RETURN pub.displayName, pub.institution, pub.published_year - ``` - -# Expression Evaluation in RETURN - -Since v1.22.0, TuringDB supports arithmetic expressions directly in `RETURN` projections. - -**Supported operators:** `+`, `-`, `*`, `/` - -```cypher --- Single property with constant -MATCH (n) -WHERE n.type = 'Patient' -RETURN n.age * 12 - --- Two properties combined -MATCH ()-[r]->() -WHERE r.value_satoshi > 0 -AND r.value_usd > 0 -RETURN r.value_usd / r.value_satoshi * 100000000.0 -``` - -# Built-in Functions - -TuringDB supports built-in functions for type conversion, introspection, and embedding comparison. - -## Introspection: `labels()`, `edgeType()` - -```jsx -MATCH (n)-[e]->(m) RETURN labels(n), edgeType(e), labels(m) -``` - -## Type conversion: `toInteger()`, `toFloat()`, `toBoolean()` - -```jsx -MATCH (n) WHERE n.year > toInteger("2020") RETURN n.name, n.year -MATCH (n) RETURN n.name, n.price * toFloat("1.07") -MATCH (n) RETURN toBoolean("true") -``` - -## Embedding comparison: `cosine_similarity()`, `euclidean_distance()` - -Compare two embeddings (embedding literals use parentheses — see [Vector Search](/vector-search)): - -```jsx -MATCH (n:Document) RETURN n.title, cosine_similarity(n.emb, (0.4, 0.3, 0.8, 0.1)) -MATCH (n:Document) RETURN n.title, euclidean_distance(n.emb, (0.4, 0.3, 0.8, 0.1)) -``` - -## Aggregation: `count()`, `avg()` - -TuringDB supports the `count()` and `avg()` aggregating functions. Aggregates may only be used when the query has a single return item: - -```jsx -MATCH (n:Person) RETURN count(*) -MATCH (n:Person) RETURN count(n.age) -MATCH (n:Person) RETURN avg(n.age) -``` - -Aggregates may be combined within that single return item: - -```jsx -MATCH (n:Person) RETURN count(n) + avg(n.age) -``` - -# Procedures - -On top of supporting queries which return or alter information _in the graph_, TuringDB supports a number of _procedures_ which return information, or metadata, _about the graph_. - -These queries follow the `CALL` syntax, and the following variants are supported: - -1. `CALL db.propertyTypes()` - all node and edge property keys and their types -2. `CALL db.labels()` - all node labels -3. `CALL db.edgeTypes()` - all edge types (edge equivalent of node labels) -4. `CALL db.history()` - the commit history (commit, nodeCount, edgeCount, partCount) -5. `CALL db.describeCommit('')` - node/edge/part counts for a single commit -6. `CALL db.procedures()` - list the available procedures and their signatures -7. `CALL db.showIndexes()` - list property indexes and their sizes - -These procedures are useful for exploring the data which is available in the graph, and using this to plan `MATCH` queries. - -A procedure's output columns can be `YIELD`ed and fed into a following `MATCH`/`WHERE`: - -```jsx -CALL db.propertyTypes() YIELD propertyType, valueType RETURN * -CALL db.labels() YIELD label, id MATCH (n) WHERE n.name = label RETURN n, label -``` - -# Commands - -Outside of CYPHER, there are a number of commands which you can use to interact with the TuringDB engine. - -| Command | Explanation | -| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `CREATE GRAPH ` | Create a graph with the specified name | -| `LOAD GRAPH ` | Load the specified TuringDB graph. Requires the graph files to be accessible in TuringDB directory (`-turing-dir`, `graphs` subdir) | -| `LOAD GML 'mygraph.gml' AS my_graph` | Load the specified GML as TuringDB graph. Requires the GML to be accessible in TuringDB directory (`-turing-dir`, `data` subdir) | -| `LOAD JSONL 'mygraph.jsonl' AS my_graph` | Load the specified JSONL as TuringDB graph. Requires the JSONL to be accessible in TuringDB directory (`-turing-dir`, `data` subdir) | -| `LOAD JSONL 'mygraph.jsonl' AS my_graph WITH EMBEDDINGS [{"emb", 384}]` | Load JSONL and mark array properties as embeddings (see [JSONL import](/import_data/jsonl#loading-embedding-properties)) | -| `COMMIT` | Persist intermediate state within a change (needed between separate node and edge create steps) | -| `CHANGE NEW` | Creates a new change, returning a column with the ID of the created change | -| `CHANGE SUBMIT` | When checked out on a specific change, submits all changes made to the “master branch” | -| `CHANGE DELETE` | Deletes the currently checked out change | -| `CHANGE LIST` | Lists the currently active (uncommitted) changes | -| `LOAD COMMIT ''` | Load a past commit into memory for querying. Required when sending queries directly to the REST API for non-HEAD commits. The CLI `checkout` and Python SDK handle this automatically. | -| `LIST GRAPH` | Lists the available graphs | -| `CREATE VECTOR INDEX WITH DIMENSION METRIC ` | Create a vector index (see [Vector Search](/vector-search)) | -| `VECTOR SEARCH IN FOR () YIELD ids` | k-nearest-neighbor search over a vector index | -| `CREATE INDEX FOR (n) ON n.prop` | Create a property index on a node property (write — see Indexes below) | -| `CREATE INDEX FOR [e] ON e.prop` | Create a property index on an edge property | -| `DROP INDEX ` | Drop a property or vector index by name | -| `MERGE_DATAPARTS` | Compact all DataParts of the current graph into one (see [DataParts](/concepts/dataparts)) | -| `INSTALL ` | Load a procedure extension (see Extensions below) | -| `SHOW EXTENSIONS` | List installed extensions | -| `SHOW PROCEDURES` | List available procedures and signatures (same data as `CALL db.procedures()`) | - -# UNWIND - -`UNWIND` expands a list into one row per element. It is a reading statement and composes with `RETURN` / `MATCH`. + + `EXPLAIN ` returns the compiled query plan without running the query. It is the easiest way to check that a lookup uses a constant node scan. + -```jsx -UNWIND [1, 2, 3] AS x RETURN x -``` +## Coming from Neo4j + +TuringDB follows openCypher closely, so the queries you know carry over. These are the TuringDB specifics worth knowing: + +- **`RETURN n` returns the node's internal ID**, an integer. Return the properties you need with `RETURN n.name, n.age`, or use [`db.getNodes`](/query/procedures#inspecting-nodes-and-edges) to see every property of a few nodes. +- **Values go in the query text.** Queries are plain strings, so put literal values directly in the text and escape quotes in your application. Query parameters such as `$name` are not yet supported. +- **One statement per request.** Send each statement as its own request (a single trailing `;` is fine). +- **Labels are set at creation.** Every node has at least one label and every edge exactly one edge type, chosen when it is created. +- **Property types are global.** A property name has a single value type across the whole graph: if `age` is an Integer, it is an Integer on every node. +- **Typed numeric equality.** Compare Integers with Integers, and match Doubles with a range (`n.score > 1.99 AND n.score < 2.01`) or convert with `toInteger`. `<`, `>` and arithmetic work across numeric types. +- **Forgiving names.** An unknown label matches nothing and a misspelled property reads as `null`. Check names with [`CALL db.labels()` and `CALL db.propertyTypes()`](/query/procedures). +- **Label alternatives.** Write `WHERE n:A OR n:B` or `WHERE NOT n:A`. Edge type alternatives (`[:A|B]`) work directly in patterns. +- **String matching.** Use `STARTS WITH`, `ENDS WITH` and `CONTAINS`. Regular expressions (`=~`) are not yet supported. +- **Loops and uniqueness.** Use `UNWIND` for loops and `MERGE` to avoid duplicates. +- **Shortest paths** use a TuringDB statement, see [Shortest path](/shortest-path). +- **Function names are case-sensitive** (`toInteger`, not `TOINTEGER`), except `count` and `collect`. + + + Coming from TuringDB 1.x? The v3 engine replaces the previous one, and many earlier limits are gone: `OPTIONAL MATCH`, `WITH`, `DISTINCT`, `UNION`, `MERGE`, `DELETE`, aggregates in any position, `UNWIND` over any list, `LOAD CSV` followed by `MATCH`, and `CREATE ... SET` all work now. One function changed name: `edgeType(e)` is now `type(e)`. + + +## Data types + +TuringDB offers these types for node and edge properties and for values in queries: + +| Type | Literal | Notes | +| ---- | ------- | ----- | +| String | `'Alice'` or `"Alice"` | Backslash escapes (`\'`). Backticks quote identifiers, not strings | +| Integer | `30`, `0x1F` | Signed 64-bit. Integer division truncates | +| Double | `3.14`, `1e3` | `^` always returns a Double | +| Boolean | `true`, `false` | | +| List | `[1, 'a', [2]]` | May mix types and nest | +| Map | `{a: 1, b: [1, 2]}` | See [Maps](/query/functions#maps) | +| DateTime | `datetime('2024-03-15T10:30:00Z')` | Microsecond precision, always UTC | +| Duration | `duration({days: 2})` | Microsecond precision, no month component | +| Embedding | `(1.2, 2.0, 0.0)` | Parentheses, at least 2 numbers. See [Vector search](/vector-search) | - `UNWIND` currently accepts **literal lists only** — `UNWIND $param`, `UNWIND someVariable`, and `UNWIND collect(...)` are not yet supported. List literals (`[...]`) may also be returned directly, e.g. `RETURN [1, 2, 3] AS nums`, but their elements must be literals. + Embedding literals use **parentheses**. `[1.2, 2.0, 0.0]` is a List, which is a different type. -# Property Indexes - -A property index speeds up equality lookups on a node or edge property. The query optimizer automatically rewrites `WHERE n.prop = ` into an index lookup when a matching index exists. - -Creating an index is a **write**, so it must run inside a change and only takes effect after the change is committed/submitted: - -```jsx -CHANGE NEW -CREATE INDEX age_index FOR (n) ON n.age -- node property index -CREATE INDEX since_index FOR [e] ON e.since -- edge property index -CHANGE SUBMIT - --- now this uses the index instead of a full scan: -MATCH (n) WHERE n.age = 30 RETURN n -``` - -Inspect existing indexes with `CALL db.showIndexes()` (yields `name`, `size`); remove one with `DROP INDEX age_index`. - -# Extensions - -Extensions are native shared libraries that register extra procedure namespaces callable via `CALL`. Install one by name, then call its procedures: - -```jsx -INSTALL greeter -SHOW EXTENSIONS -CALL greeter.hello() YIELD message RETURN message -``` - -`SHOW PROCEDURES` lists every available procedure and its signature (the same information as `CALL db.procedures()`). +More detail on types, conversion and results is in [Functions and data types](/query/functions). -# Roadmap +## Roadmap -Whilst TuringDB currently supports most of CYPHER, 100% of CYPHER can be parsed but we are working on supporting query execution for some rare CYPHER queries types. TuringDB also has a CALL function which over time will contain more and more algorithms. +We keep extending Cypher coverage and growing the library of procedures available through `CALL`. If there is a query form you would like to see, please [tell us](/troubles/troubles#contact-us). diff --git a/query/functions.mdx b/query/functions.mdx new file mode 100644 index 0000000..cec5adf --- /dev/null +++ b/query/functions.mdx @@ -0,0 +1,166 @@ +--- +title: "Functions and Data Types" +sidebarTitle: "Functions and Types" +description: "Built-in functions, how values come back to the Python SDK, dates and durations, maps, and embedding comparison in TuringDB v3." +--- + +## Built-in functions + +Function names are **case-sensitive** (`toInteger`, not `TOINTEGER`), except `count` and `collect`. + +| Category | Functions | Notes | +| -------- | --------- | ----- | +| Entity | `id(x)`, `labels(n)`, `type(r)`, `startNode(r)`, `endNode(r)` | `labels` returns a **list**. `type` returns a string. `startNode(r).name` works | +| Path | `nodes(p)`, `relationships(p)`, `length(p)` | Lists of IDs. Read properties through a [comprehension](/query/aggregation#list-comprehensions) | +| List | `size`, `length`, `head`, `last`, `tail`, `reverse`, `range(start, end[, step])`, `reduce` | `size`, `length` and `reverse` also work on strings. `range` bounds are inclusive | +| Predicates | `all`, `any`, `none`, `single` `(x IN list WHERE ...)` | | +| Conversion | `toInteger`, `toFloat`, `toString`, `toBoolean`, `coalesce(a, b, ...)` | `toInteger` and `toFloat` accept strings, integers and doubles, and return `null` for a bad string. `toBoolean` accepts strings. `coalesce` arguments must share one type | +| String | `toUpper`, `toLower`, `trim`, `ltrim`, `rtrim`, `reverse` | Plus the operators `STARTS WITH`, `ENDS WITH`, `CONTAINS` and `\|\|` | +| Math | `abs`, `sign`, `ceil`, `floor`, `round`, `sqrt`, `exp`, `log`, `log10`, `pi()`, `e()` | Trigonometry: `sin`, `cos`, `tan`, `cot`, `asin`, `acos`, `atan`, `degrees`, `radians`, `haversin` | +| Temporal | `datetime(isoString)`, `datetime(epochSeconds)`, `datetime()`, `duration({days: 2, hours: 3})`, `duration(micros)` | See [Dates and durations](#dates-and-durations) | +| Embedding | `cosine_similarity(a, b)`, `euclidean_distance(a, b)` | Arguments must be Embeddings, not lists. See [Vector search](/vector-search) | +| Aggregate | `count`, `collect`, `sum`, `avg`, `min`, `max` | See [Aggregation](/query/aggregation#aggregation) | + +```cypher +MATCH (n)-[e]->(m) +RETURN labels(n), type(e), labels(m) + +MATCH (n:Person) +RETURN toUpper(n.name), toString(n.age) + +MATCH (n:Person) +WHERE n.age > toInteger('29') +RETURN n.name + +MATCH (n:Person) +RETURN n.name, n.age * toFloat('1.07') + +RETURN toBoolean('true') + +RETURN abs(-3), round(2.5), sqrt(16.0), pi() + +MATCH (n:Person) +RETURN n.name, coalesce(n.nickname, n.name) +``` + + + In v3, `type(e)` returns an edge's type as a string. It replaces `edgeType()` from earlier versions. + + +The function library keeps growing. Some functions, such as `substring`, `split`, `rand()`, `keys` and `properties`, are not yet available. For now, filter strings with `STARTS WITH`, `ENDS WITH` and `CONTAINS`, and do any other post-processing in your application. + +## Arithmetic in `RETURN` and `SET` + +Arithmetic expressions work anywhere an expression is allowed: + +```cypher +MATCH (n:Person) +RETURN n.age * 12 + +MATCH (n:Person) +RETURN n.name, n.age + 5 AS adjusted, n.age / 2 + +MATCH ()-[r]->() +WHERE r.since > 0 +RETURN r.since / 10 * 100 +``` + +`+ - * / % ^` are supported. Integer `/` truncates, and `^` returns a Double. `+` and `||` concatenate strings and lists, and `'x' + 1` gives `'x1'`. + +## Dates and durations + +`DateTime` and `Duration` are first-class types: + +```cypher +RETURN datetime('2024-03-15T10:30:00Z') AS d + +RETURN datetime('2024-03-15') AS midnight + +RETURN datetime(1700000000) AS fromEpoch + +RETURN datetime('2024-03-15T10:30:00Z') + duration({days: 1, hours: 2}) AS later + +RETURN datetime('2024-03-16T00:00:00Z') - datetime('2024-03-15T00:00:00Z') AS elapsed + +RETURN datetime('2024-03-15T10:30:00Z').year, datetime('2024-03-15T10:30:00Z').month + +RETURN duration({days: 2, hours: 3}).hours +``` + +Durations can be added, subtracted, scaled and compared with each other and with datetimes: + +```cypher +RETURN duration({hours: 36}) + duration({hours: 12}) AS total + +RETURN duration({days: 2, hours: 12}) - duration({days: 1}) AS remaining + +RETURN duration({days: 2}) * 3 AS threeTimes + +RETURN duration({hours: 36}) / 2 AS half + +RETURN duration({hours: 1}) > duration({minutes: 30}) AS longer +``` + +- `datetime` values are stored as microseconds since the Unix epoch, **normalised to UTC**. A date-only string means midnight, and an offset (`+02:00`) is converted to the equivalent UTC instant. +- `duration` is a signed count of microseconds. It has no month component, because a month has no fixed length. `duration()` reads the number as microseconds. +- Components are available as properties: `d.year`, `d.month`, `d.day`, `d.hour`, and so on. On a duration, `dur.days`, `dur.hours`, ... give the **total** in that unit, so `duration({days: 2, hours: 3}).hours` is `51`. +- `<`, `>`, `min` and `max` work on temporal values. +- When you [import JSONL](/import_data/jsonl) you can declare which properties are `DateTime` or `Duration` with `WITH DATETIMES [...]` and `WITH DURATIONS [...]`. + +## Maps + +Map literals can be built, returned, compared with `=`, and stored as node or edge properties: + +```cypher +RETURN {name: 'Alice', tags: ['a', 'b']} AS m + +WITH {a: 1, b: 'x'} AS m +RETURN m.a, m.b + +UNWIND [{a: 1}, {a: 2}] AS m +RETURN m.a +``` + +You can read a key from a map **value** that is in scope, for example one bound by `WITH` or `UNWIND`, with `m.key`. + + + To read individual values from a map stored as a property, return the property as a whole or store the values as separate properties. Dereferencing a stored map inside a query (`n.meta.level`, `m['key']`) and map projection (`n {.name, .age}`) are not yet supported. + + +## How results come back + +`client.query()` returns a pandas DataFrame. The column types follow the query's value types: + +| Type | HTTP `column_types` | pandas dtype | +| ---- | ------------------- | ------------ | +| String | `String` | `string` | +| Integer | `Int64` | `Int64` | +| Unsigned integer (IDs, `count`, `length(p)`) | `UInt64` | `UInt64` | +| Double | `Double` | `float64` | +| Boolean | `Bool` | `boolean` | +| List | `List` | object (Python list) | +| Map | `Map` | object (dict) | +| Embedding | `Embedding` | object (list over HTTP, numpy `float32` array over the native and embedded backends) | +| DateTime | `DateTime` | `datetime64[us, UTC]` | +| Duration | `Duration` | `timedelta64[us]` | +| Node or edge (`RETURN n`) | `UInt64` | `UInt64` (internal ID) | +| Named path (`RETURN p`) | `EntityList` | object (list of `{type, id}` dicts) | + + + Over the default HTTP/JSON backend, Doubles are serialised with **6 decimal places** (`3.14159265` becomes `3.141593`). If you need full precision, scale the value in the query, or use the [native or embedded backend](/pythonsdk/reference#client-backends). + + +## Embeddings + +Embeddings are stored as ordinary properties (type `Embedding`) and compared with `cosine_similarity` (true cosine) and `euclidean_distance`. Both work in `RETURN`, `WHERE` and `ORDER BY`: + +```cypher +MATCH (n:Document) +RETURN n.title, cosine_similarity(n.emb, (0.4, 0.3, 0.8, 0.1)) AS sim +ORDER BY sim DESC LIMIT 10 + +MATCH (n:Document) +RETURN n.title, euclidean_distance(n.emb, (0.4, 0.3, 0.8, 0.1)) +``` + +This is a brute-force scan over the matched nodes. For large collections, build a [vector index](/vector-search). diff --git a/query/matching.mdx b/query/matching.mdx new file mode 100644 index 0000000..6caaf37 --- /dev/null +++ b/query/matching.mdx @@ -0,0 +1,285 @@ +--- +title: "Matching Patterns" +sidebarTitle: "Matching" +description: "Read data with MATCH, OPTIONAL MATCH and WHERE, join patterns, and look up nodes quickly by native node ID." +--- + +Read queries run directly against the current graph, with no change to open first. + +The examples on this page use a small graph of people, companies and interests. Create it once in a change (see [Writing data](/query/writing)) if you want to follow along: + +```cypher +CREATE + (alice:Person {name: 'Alice', age: 30, city: 'London', active: true, tags: ['admin', 'vip']}), + (bob:Person {name: 'Bob', age: 35, city: 'London', active: true, tags: ['dev']}), + (carol:Person {name: 'Carol', age: 28, city: 'Paris', active: false, tags: ['dev', 'vip']}), + (dave:Person {name: 'Dave', age: 41, city: 'Paris', active: true}), + (acme:Company {name: 'Acme', founded: 1999}), + (globex:Company {name: 'Globex', founded: 2010}), + (music:Interest {name: 'Music'}), + (alice)-[:KNOWS {since: 2010}]->(bob), + (bob)-[:KNOWS {since: 2015}]->(carol), + (carol)-[:KNOWS {since: 2018}]->(dave), + (alice)-[:WORKS_AT {role: 'CTO'}]->(acme), + (bob)-[:WORKS_AT {role: 'Engineer'}]->(acme), + (carol)-[:WORKS_AT {role: 'Engineer'}]->(globex), + (alice)-[:LIKES]->(music), + (carol)-[:LIKES]->(music) +``` + +## MATCH + +```cypher +MATCH (n) +RETURN count(n) + +MATCH (n:Person) +RETURN n.name, n.age + +MATCH (n:Person {name: 'Alice', age: 30}) +RETURN n.name + +MATCH (n:Person:Employee) +RETURN n.name + +MATCH (a:Person)-[:KNOWS]->(b:Person) +RETURN a.name, b.name + +MATCH (a)-[e:KNOWS {since: 2010}]->(b) +RETURN a.name, e.since, b.name + +MATCH (a)<-[e]-(b) +RETURN a.name, b.name + +MATCH (a)-[e]-(b) +RETURN a.name, b.name + +MATCH (a)-[r:KNOWS|WORKS_AT]->(b) +RETURN a.name, type(r), b.name + +MATCH (a {name: 'Alice'}) +MATCH (a)-[:KNOWS]->(b) +RETURN b.name +``` + +- `(n:Person:Employee)` matches nodes that have **both** labels. +- `(a)-[e]-(b)` (or `(a)--(b)`) ignores direction. +- `[:KNOWS|WORKS_AT]` matches either edge type. An edge has exactly one type, so each edge pattern takes a single type or a list of alternatives. +- `type(r)` returns the edge type as a string. +- Several `MATCH` clauses in a row are allowed, and a later one can reuse variables from an earlier one. +- Variables are only needed for entities you want to return or reuse. `MATCH (:Person)-[:KNOWS]->(m) RETURN m.name` is valid, and a node or edge may omit its variable when it has a label or type. + + + `RETURN n` returns the node's internal ID, not its properties. Return the properties you need (`RETURN n.name, n.age`). This is different from Neo4j, where `RETURN n` returns the whole node. + + +### Filtering inside the pattern + +A `WHERE` can sit inside a node or edge pattern. The variable must be named: + +```cypher +MATCH (a:Person WHERE a.age > 30)-[r:KNOWS WHERE r.since > 2012]->(b) +RETURN b.name +``` + +### Ordering and limiting in `MATCH` + +As a TuringDB extension, `ORDER BY`, `SKIP` and `LIMIT` can also appear right after the pattern, before `RETURN`: + +```cypher +MATCH (n:Person) +ORDER BY n.age LIMIT 2 +RETURN n.name +``` + + + Unknown labels and property names are forgiving: `MATCH (n:Nope) RETURN n` returns no rows and `RETURN n.misspelled` returns `null`. Use [`CALL db.labels()`, `CALL db.edgeTypes()` and `CALL db.propertyTypes()`](/query/procedures) to check what exists in a graph. + + +## OPTIONAL MATCH + +`OPTIONAL MATCH` keeps the row when the pattern has no match and returns `null` for its variables: + +```cypher +MATCH (a:Person) +OPTIONAL MATCH (a)-[:WORKS_AT]->(c:Company) +RETURN a.name, c.name + +MATCH (a:Person) +OPTIONAL MATCH (a)-[:KNOWS]->(b) +RETURN a.name, count(b) +``` + +The second query returns `0` for people who know nobody. A `WHERE` attached to an `OPTIONAL MATCH` belongs to the optional part, so rows are kept, as in Neo4j. + +Return a property (`c.name`) or `id(c)` from an optional node, rather than the bare variable. + + +## WHERE and operators + +```cypher +MATCH (n) +WHERE n:Person +RETURN n.name + +MATCH (n:Person) +WHERE n.age >= 18 AND n.city = 'London' +RETURN n.name + +MATCH (n:Person) +WHERE n.name STARTS +WITH 'A' OR n.name ENDS +WITH 'e' OR n.name CONTAINS 'ro' +RETURN n.name + +MATCH (n:Person) +WHERE n.name IN ['Alice', 'Bob'] +RETURN n.name + +MATCH (n:Person) +WHERE 'admin' IN n.tags +RETURN n.name + +MATCH (n:Person) +WHERE n.tags IS NOT NULL +RETURN n.name + +MATCH (n:Person) +WHERE n.active +RETURN n.name + +MATCH (n)-[e]->(m) +WHERE e:KNOWS AND e.since > 2012 +RETURN n.name, m.name + +MATCH (p:Person) +WHERE (p)-[:LIKES]->() +RETURN p.name + +MATCH (p:Person) +WHERE NOT (p)-[:KNOWS]->() +RETURN p.name +``` + +- **Label and type tests:** `n:Person` tests a node label, `e:KNOWS` tests an edge type. Combine them with `AND`, `OR` and `NOT`. +- **Pattern predicates:** `WHERE (p)-[:LIKES]->()` keeps the nodes for which the pattern exists. For richer conditions see [`EXISTS` subqueries](/query/subqueries). + +| Operators | | +| --------- | - | +| Comparison | `=`, `<>`, `!=`, `<`, `<=`, `>`, `>=`, and chained comparisons such as `1 < x < 3` | +| Null tests | `IS NULL`, `IS NOT NULL` | +| Boolean | `AND`, `OR`, `XOR`, `NOT`, with three-valued logic for `null` as in Neo4j | +| Membership and strings | `IN`, `STARTS WITH`, `ENDS WITH`, `CONTAINS` | +| Arithmetic | `+`, `-`, `*`, `/`, `%`, `^`. Integer `/` truncates and `^` returns a Double | +| Concatenation | `+` and `\|\|` join strings and lists | + +Property constraints in a pattern, `MATCH (n {name: 'Matt', age: 20})`, are exactly equivalent to the same equality tests in `WHERE`. + + + Use `n.prop IS NOT NULL` to test for a property, and `STARTS WITH`, `ENDS WITH` and `CONTAINS` to match strings. Regular expressions (`=~`) and `exists(n.prop)` are not yet supported. + + +### Comparing numbers + +Equality compares numbers of the same kind. An Integer compared with a Double using `=`, or two Doubles compared with `=`, is reported when the query is analysed: + +```text +MATCH (n:Person) WHERE n.age = 30.0 RETURN n.name +ANALYZE_ERROR: Operands are not valid or compatible types +``` + +Use range comparisons for Doubles, or convert the value: + +```cypher +MATCH (n:Person) +WHERE n.age > 29.5 AND n.age < 30.5 +RETURN n.name + +MATCH (n:Person) +WHERE n.age = toInteger(30.0) +RETURN n.name +``` + +Mixed-type `<`, `>` and arithmetic work as expected. Division by zero returns an execution error, and Int64 arithmetic wraps around on overflow. + +## Joins and Cartesian products + +Comma-separated patterns in a `MATCH` produce either a **join** or a **Cartesian product**, depending on whether they share a variable. + +```cypher +// Cartesian product: every Person with every Company (N x M rows) +MATCH (p:Person), (c:Company) +RETURN p.name, c.name + +// Join: a shared variable links the patterns +MATCH (a:Person)-[:KNOWS]->(b), (b)-[:WORKS_AT]->(c) +RETURN a.name, b.name, c.name + +// Join on a shared node: people who like the same thing +MATCH (a:Person)-->(i:Interest)<--(b:Person) +WHERE a <> b +RETURN a.name, b.name, i.name + +// Join on a value through WHERE (hash join) +MATCH (a:Person), (b:Person) +WHERE a.city = b.city AND a <> b +RETURN a.name, b.name +``` + + + A Cartesian product of N nodes by M nodes produces N × M rows. Always constrain it with `WHERE` or `LIMIT` before running it on a large graph. + + +Connected paths and independent patterns can be mixed in one query. Shared variables create joins and unrelated patterns create Cartesian products: + +```cypher +MATCH (a:Person)-->(i:Interest), (c:Company) +WHERE c.name = 'Acme' +RETURN a.name, i.name, c.name +``` + +## Look up nodes by native node ID + +Every node has a **native node ID**, the integer you get from `RETURN n` or `id(n)`. When you already know which nodes you want, **compare the node variable itself with the ID**, written as a literal. The planner turns this into a constant scan of exactly those nodes: + +```cypher +MATCH (n) +WHERE n = 0 +RETURN n.name + +MATCH (n)-[:KNOWS]->(m) +WHERE n = 0 +RETURN m.name + +UNWIND [0, 1, 2] AS x +MATCH (n) +WHERE n = x +RETURN id(n), n.name +``` + +`EXPLAIN` shows whether it worked. The plan should contain `const_scan_nodes` rather than `scan_nodes`: + +```cypher +EXPLAIN MATCH (n) +WHERE n = 0 +RETURN n.name +``` + +On large graphs this is orders of magnitude faster than looking nodes up again through a key property such as `n.uuid`. The rules that keep it fast: + +- **Compare the variable, not the function.** `WHERE n = 123` is a direct seek. `WHERE id(n) = 123` scans every node. Use `id(n)` only to *return* the ID. +- **Use `UNWIND` for batches, not `IN`.** `WHERE n IN [...]` and `WHERE id(n) IN [...]` scan every node. The `UNWIND [] AS x MATCH (n) WHERE n = x` form is rewritten at plan time into one constant scan, which costs roughly a microsecond per node. +- **The batch form needs an exact shape:** + - the list is a **literal written in the query text** (build it in Python: `f"UNWIND {ids} AS x ..."`), not a variable, `collect(...)`, `range(...)` or a list property; + - its elements are **distinct** integers, so deduplicate first; + - `x` is used **only** in `n = x`. Return `id(n)` rather than `x`. +- **Fetch IDs once and reuse them.** Look nodes up with whatever filter you need (`MATCH (n:Person {name: 'Alice'}) RETURN id(n)`), keep the IDs in your application, and address the nodes by ID from then on. +- **Keep IDs constant.** Use literal IDs rather than IDs computed per row, and match one node per `MATCH`. Forms such as `WHERE a = p[0] AND b = p[1]` are not rewritten into a constant scan, so they run as a Cartesian product of two full node scans, which is expensive on large graphs. To connect two known nodes, put both IDs in as literals, one pair per query: `MATCH (a) WHERE a = 5 MATCH (b) WHERE b = 7`. For bulk edges, see [Parquet import](/import_data/parquet). +- **Vector search results are already nodes.** `VECTOR SEARCH ... YIELD ids` binds `ids` as nodes, so use `ids.title` or `MATCH (ids)-[...]->(...)` directly. See [Vector search](/vector-search). + +**Stability.** The native IDs of existing nodes are stable across queries, change submits and server restarts, so it is safe to cache them. Two exceptions: + +- Nodes created in an open change get their final IDs at `COMMIT` or `CHANGE SUBMIT`. Read them after that. +- [`MERGE_DATAPARTS`](/concepts/dataparts) renumbers all nodes. Re-fetch any cached IDs afterwards. + +Keep a business key as a property as well, for durable identity outside TuringDB. Use `id(n)` for node IDs. For edges, `MATCH ()-[r]->() WHERE id(r) = 0 RETURN startNode(r), endNode(r)` works. diff --git a/query/paths.mdx b/query/paths.mdx new file mode 100644 index 0000000..6505a3d --- /dev/null +++ b/query/paths.mdx @@ -0,0 +1,116 @@ +--- +title: "Paths" +sidebarTitle: "Paths" +description: "Variable-length paths, quantified path patterns and named paths in TuringDB." +--- + +These examples use the [sample graph](/query/matching) from the matching page. + +## Variable-length paths + +TuringDB supports both its own postfix quantifiers and the Neo4j bracket form: + +| Syntax | Hops | +| ------ | ---- | +| `-[:KNOWS]->+` | 1 or more | +| `-[:KNOWS]->*` | **0** or more (includes the start node itself) | +| `-[:KNOWS]->{2,4}`, `{2}`, `{,3}`, `{2,}` | between, exactly, up to, at least | +| `-[:KNOWS*1..3]->`, `-[:KNOWS*2]->`, `-[*..3]-`, `-[*0..1]->` | Neo4j style | +| `-[:KNOWS*]->` | **1** or more (unlike the postfix `*`) | + +```cypher +MATCH (a:Person {name: 'Alice'})-[:KNOWS]->+(b) +RETURN DISTINCT b.name + +MATCH (a:Person {name: 'Alice'})-[:KNOWS*1..3]->(b) +RETURN DISTINCT b.name + +MATCH (a)-[:KNOWS|WORKS_AT]->{1,2}(b) +RETURN a.name, b.name + +MATCH (a)-[e:KNOWS WHERE e.since > 2012]->+(b) +RETURN a.name, b.name + +MATCH (a)-[:KNOWS*1..3 {since: 2010}]->(b) +RETURN b.name + +MATCH (a {name: 'Alice'})-[e:KNOWS]->+(b) +RETURN b.name, size(e) +``` + +- **Edge type alternatives** (`[:KNOWS|WORKS_AT]`) work on variable-length hops. +- **A predicate or property map applies to every hop.** `-[e:KNOWS WHERE e.since > 2012]->+` only follows edges that satisfy it, and `-[:KNOWS*1..3 {since: 2010}]->` requires every edge on the path to have that property value. +- **Trail semantics.** Each edge is used at most once within a path, while nodes may repeat. Each distinct path is its own row, so use `DISTINCT` or an aggregate to collapse them. +- **One quantifier per edge.** Choose either `[*1..3]` or `{1,3}` for an edge. +- **The edge variable of a variable-length hop is a list of edge IDs.** `size(e)`, `e[0]`, `e[-1]` and `UNWIND e` work on it. To read edge properties along a path, name the path and use `relationships(p)` (below). + + + Variable-length matching **enumerates** paths. On dense graphs the number of paths can grow very quickly, so always bound the length (`*1..6` or `{1,6}`). For weighted shortest paths on large graphs, use the [`shortestPath` statement](/shortest-path). + + +## Quantified path patterns + +A quantified path pattern repeats a one-hop sub-pattern, with its own `WHERE`, a number of times: + +```cypher +MATCH (a:Person {name: 'Alice'}) ((x)-[r:KNOWS]->(y) WHERE r.since > 2009){1,3} (b) +RETURN b.name + +MATCH (a:Person {name: 'Alice'}) ((x)-[:KNOWS]->(y) WHERE y.age > x.age)+ (b) +RETURN b.name +``` + +The second query follows `KNOWS` edges only while ages strictly increase. Inside the parentheses only a **single hop** of the form `(node)-[edge]->(node) [WHERE ...]` is allowed, and the inner variables (`x`, `r`) become lists. + +## Named paths + +Give a path a name to inspect its nodes and edges: + +```cypher +MATCH p = (a:Person {name: 'Alice'})-[:KNOWS]->+(b) +RETURN [n IN nodes(p) | n.name] AS names, + [r IN relationships(p) | r.since] AS sinces, + length(p) AS hops +ORDER BY hops +``` + +- `nodes(p)` and `relationships(p)` return **lists of IDs**. Read properties through a [list comprehension](/query/aggregation#list-comprehensions). +- `length(p)` is the number of hops. +- A bare `RETURN p` comes back as a list of `{"type": "node" | "edge", "id": ...}` dictionaries. +- `count(p)` counts paths. +- `OPTIONAL MATCH p = ... RETURN p IS NULL` works. + +## Finding routes + +For hop-count questions, undirected search, or when you need the properties along a route, combine variable-length paths with comprehensions: + +```cypher +// Fewest hops, returning the names along the route +MATCH p = (a:Person {name: 'Alice'})-[:KNOWS*1..6]->(b:Person {name: 'Dave'}) +RETURN [n IN nodes(p) | n.name] AS route, length(p) AS hops +ORDER BY hops LIMIT 1 + +// k-hop neighbourhood and reachability +MATCH (a:Person {name: 'Alice'})-[:KNOWS]-{1,3}(b) +RETURN DISTINCT b.name + +MATCH (a:Person {name: 'Alice'})-[:KNOWS]->+(b) +RETURN count(DISTINCT b) +``` + +To sum a property along a route, build a list with a comprehension and aggregate over its elements: + +```cypher +MATCH p = (a:Person {name: 'Alice'})-[:KNOWS*1..6]->(b:Person {name: 'Dave'}) +WITH [n IN nodes(p) | n.name] AS route, [r IN relationships(p) | r.since] AS years +UNWIND years AS y +WITH route, sum(y) AS total +RETURN route, total +ORDER BY total LIMIT 1 +``` + + + To sum a property along a route, compute the list with a comprehension first, as above. + + +For a true weighted shortest path, see [Shortest path](/shortest-path). diff --git a/query/procedures.mdx b/query/procedures.mdx new file mode 100644 index 0000000..095b5ee --- /dev/null +++ b/query/procedures.mdx @@ -0,0 +1,156 @@ +--- +title: "Procedures and Commands" +sidebarTitle: "Procedures and Commands" +description: "Explore a graph with CALL db.* procedures, and manage graphs, changes, vector indexes and extensions with TuringDB engine commands." +--- + +Beyond queries over the data in a graph, TuringDB has **procedures**, which return information about the graph, and **commands**, which drive the engine itself. + +## Procedures + +Procedures use the `CALL` syntax. Use them to explore a graph before writing queries against it: + +```cypher +CALL db.labels() +CALL db.edgeTypes() +CALL db.propertyTypes() +CALL db.history() +CALL db.procedures() +``` + +| Procedure | Yields | +| --------- | ------ | +| `db.labels()` | `id`, `label` | +| `db.edgeTypes()` | `id`, `edgeType` | +| `db.propertyTypes()` | `id`, `propertyType`, `valueType` | +| `db.history()` | `commit`, `nodeCount`, `edgeCount`, `partCount` | +| `db.describeCommit(commit)` | `nodeCount`, `edgeCount`, `partCount` | +| `db.procedures()` | `name`, `signature` (same as `SHOW PROCEDURES`) | +| `db.hierarchicalLabelCounts(labels)` | `label`, `nodeCount` | +| `db.listNodes(labels, properties, skip, limit)` | `id`, `labels`, `properties` | +| `db.getNodes(nodeIDs)` | `id`, `labels`, `inEdgeCount`, `outEdgeCount`, `properties` | +| `db.getEdges(edgeIDs)` | `id`, `src`, `tgt`, `edgeTypeID`, `properties` | +| `db.getNodeEdges(...)` | `id`, `outgoingEdges`, `incomingEdges`, `outEdgeCounts`, `inEdgeCounts` | +| `gnn.neighbourhoodSample(node, sampleSize, seed)` | `src`, `edge`, `edgeType`, `tgt` | +| `gnn.graphSAGE(seeds, fanouts, seed)` | `dst_nodes0..2`, `src_nodes0..2`, `tgt_nodes0..2` | + +`db.history()` lists commits newest first. Every `COMMIT` and every submitted change adds an entry, and `nodeCount` / `edgeCount` count what **that commit** added, not the graph totals. + +### Chaining procedures with YIELD + +A procedure's output columns can be renamed, filtered and used by the rest of the query with `YIELD`: + +```cypher +CALL db.propertyTypes() YIELD * + +CALL db.labels() YIELD label AS l +WHERE l STARTS +WITH 'P' +RETURN l + +CALL db.labels() YIELD label +MATCH (n) +WHERE label IN labels(n) +RETURN label, count(n) + +CALL db.history() YIELD `commit`, nodeCount +RETURN `commit`, nodeCount +``` + +- **Constant arguments.** Procedure arguments are constants, so pass literal values (for example `db.getNodes([0, 1, 2])`) rather than values read from a row. +- **`commit` is a keyword.** Yield it with backticks: ``YIELD `commit` ``. +- To filter on `valueType`, filter the resulting DataFrame in pandas. +- `OPTIONAL CALL` is for subqueries. It is not yet available for procedures. + +### Inspecting nodes and edges + +`RETURN n` gives a node's internal ID. To see every property of a few nodes, use `db.getNodes`, which returns the properties as a JSON string: + +```cypher +CALL db.getNodes([0, 1, 2]) YIELD id, labels, properties +RETURN id, labels, properties + +CALL db.listNodes(['Person'], {name: 'ali'}, 0, 10) YIELD id, labels, properties +``` + +`db.listNodes(labels, properties, skip, limit)` does a case-insensitive substring search on property values. + +```python +import json + +df = client.query("CALL db.getNodes([0, 1, 2]) YIELD id, labels, properties") +df["properties"] = df["properties"].map(json.loads) +``` + +### Profiling a graph with plain Cypher + +```cypher +MATCH (n) +RETURN labels(n) AS labels, count(*) AS n +ORDER BY n DESC + +MATCH (a)-[r]->(b) +RETURN labels(a) AS src, type(r) AS rel, labels(b) AS dst, count(*) AS n +ORDER BY n DESC + +MATCH (n:Person) +RETURN count(n.age) AS withAge, count(*) AS total +``` + +## Commands + +Outside of Cypher, a set of commands controls the engine. Send them through `client.query()` or from the interactive shell. + +### Graphs + +| Command | Description | +| ------- | ----------- | +| `CREATE GRAPH ` | Create an empty graph | +| `LOAD GRAPH ` | Load an on-disk graph. Its files must be in the TuringDB directory (`-turing-dir`, `graphs` subdirectory) | +| `LIST GRAPH` | List the graphs currently **loaded** in the server | +| `LIST AVAILABLE GRAPHS` | List the graphs on disk, with `isLoaded` and `isLoading` | +| `LOAD JSONL 'f.jsonl' AS ` | Build a graph from a [JSONL](/import_data/jsonl) file in the `data` subdirectory | +| `LOAD GML 'f.gml' AS ` | Build a graph from a [GML](/import_data/gml) file in the `data` subdirectory | +| `LOAD PARQUET '' AS ` | Build a graph from Parquet files, see [Parquet import](/import_data/parquet) | +| `LOAD CSV '' ... AS row` | Stream rows from a CSV file, see [CSV import](/import_data/csv) | +| `EXPLAIN ` | Show the compiled plan without running the query | + +Dropping and unloading graphs from a running server is not yet available. + +### Changes and versions + +| Command | Description | +| ------- | ----------- | +| `CHANGE NEW` | Create a new change and return its ID | +| `CHANGE SUBMIT` | Commit pending writes and merge the current change into main | +| `CHANGE DELETE` | Discard the current change | +| `CHANGE LIST` | List the open changes on the current graph | +| `COMMIT` | Seal the pending writes inside a change. Needed before a later query can `MATCH` or `MERGE` newly created entities | +| `LOAD COMMIT ''` | Load a past commit for querying. Only needed when sending queries directly to the REST API. The CLI `checkout` and the Python SDK do it for you | +| `MERGE_DATAPARTS` | Compact all [DataParts](/concepts/dataparts) of the graph into one. Run it on main with no change open on any graph | + +`MERGE_DATAPARTS` **renumbers** all nodes. Re-fetch any cached node IDs afterwards, and rebuild vector indexes keyed by node IDs. + +### Vector indexes + +| Command | Description | +| ------- | ----------- | +| `CREATE VECTOR INDEX WITH DIMENSION METRIC ` | Create a vector index, see [Vector search](/vector-search) | +| `VECTOR SEARCH IN FOR () YIELD ids` | k-nearest-neighbour search | +| `SHOW VECTOR INDEXES` | List vector indexes | +| `DELETE VECTOR INDEX ` | Delete a vector index | + +## Extensions + +Extensions are native libraries that register extra procedure namespaces. Install one by name and then call its procedures: + +```cypher +INSTALL greeter +SHOW EXTENSIONS +SHOW PROCEDURES + +CALL greeter.hello() YIELD message +RETURN message +``` + +`SHOW PROCEDURES` lists every available procedure and its signature, including the ones from extensions. The `greeter` example ships with the Python wheel. Extensions are looked up in `/lib/turingdb/extensions/.so`, then in `/extensions/.so`. diff --git a/query/subqueries.mdx b/query/subqueries.mdx new file mode 100644 index 0000000..998daf0 --- /dev/null +++ b/query/subqueries.mdx @@ -0,0 +1,95 @@ +--- +title: "Subqueries" +sidebarTitle: "Subqueries" +description: "Use EXISTS, COUNT and CALL subqueries to filter, count and compute per-row results in TuringDB." +--- + +These examples use the [sample graph](/query/matching) from the matching page. + +## EXISTS + +`EXISTS { ... }` tests whether a pattern, or a full `MATCH` with its own `WHERE`, has at least one match for the current row. It works in `WHERE` and as a returned value: + +```cypher +MATCH (p:Person) +WHERE EXISTS { (p)-[:WORKS_AT]->(:Company {name: 'Acme'}) } +RETURN p.name + +MATCH (p:Person) +WHERE EXISTS { + MATCH (p)-[:KNOWS]->(f) + WHERE f.age > p.age +} +RETURN p.name + +MATCH (p:Person) +WHERE NOT EXISTS { (p)-[:KNOWS]->() } +RETURN p.name + +MATCH (p:Person) +RETURN p.name, EXISTS { (p)-[:LIKES]->() } AS likesSomething +``` + +## COUNT + +`COUNT { ... }` returns the number of matches for each row: + +```cypher +MATCH (p:Person) +RETURN p.name, COUNT { (p)-[:KNOWS]->() } AS outDegree +ORDER BY outDegree DESC + +MATCH (p:Person) +WHERE COUNT { + MATCH (p)-[:KNOWS]-(x) + RETURN x +} >= 2 +RETURN p.name +``` + +## CALL subqueries + +A `CALL { ... }` subquery runs once per incoming row. It is the way to compute a per-row top-k or a per-row aggregate. Import the outer variables either in the `CALL (p)` header or with a `WITH p` as the first clause of the body: + +```cypher +MATCH (p:Person) +CALL (p) { + MATCH (p)-[:KNOWS]->(f) + RETURN f.name AS friend + ORDER BY f.name LIMIT 1 +} +RETURN p.name, friend + +MATCH (p:Person) +CALL { + WITH p + MATCH (p)-[:KNOWS]->(f) + RETURN count(f) AS deg +} +RETURN p.name, deg + +MATCH (p:Person) +OPTIONAL CALL (p) { + MATCH (p)-[:WORKS_AT]->(c) + RETURN c.name AS co +} +RETURN p.name, co + +CALL { + MATCH (p:Person) + RETURN p.name AS nm + UNION + MATCH (c:Company) + RETURN c.name AS nm +} +RETURN nm +ORDER BY nm +``` + +- **Dropped rows.** A non-optional `CALL` drops the outer rows for which its body returns nothing. Use `OPTIONAL CALL` to keep them with `null` values. +- **Imported variables.** To return an imported variable from a `CALL` body, give it an alias: `RETURN p AS me`. +- A `CALL` body can contain `UNION`, aggregation, `ORDER BY` and `LIMIT`. Write-only subqueries also work inside a [change](/query/writing). + + + `COLLECT { ... }`, `CALL (*) { }`, `CALL { } IN TRANSACTIONS` and `OPTIONAL CALL` on a procedure are not yet available. To collect values, use `collect()` in a `CALL` body or a [pattern comprehension](/query/aggregation#list-comprehensions). + diff --git a/query/writing.mdx b/query/writing.mdx new file mode 100644 index 0000000..842eef9 --- /dev/null +++ b/query/writing.mdx @@ -0,0 +1,287 @@ +--- +title: "Writing Data" +sidebarTitle: "Writing Data" +description: "CREATE, MERGE, SET, REMOVE and DELETE in TuringDB, and the change workflow that every write goes through." +--- + +All writes (`CREATE`, `MERGE`, `SET`, `REMOVE`, `DELETE`) must run **inside a change**. A change is TuringDB's equivalent of a git branch: you write in an isolated copy and then submit it to the main graph. A write sent outside a change returns a clear message such as `Cannot perform CREATE outside of a write transaction.` + +## The change workflow + +```python +client.new_change() # CHANGE NEW: the client is now on this change + +client.query("CREATE (:Person {name: 'Alice'})-[:KNOWS]->(:Person {name: 'Bob'})") +# ... more writes ... + +client.query("CHANGE SUBMIT") # merge the change into main +client.checkout() # return the client to main +``` + +- `new_change()` returns the change ID and puts the client on that change. +- After `CHANGE SUBMIT` the client still points at the submitted change. Call `client.checkout()` to return to main before running the next query. +- To abandon a change, run `CHANGE DELETE` and then `client.checkout()`. +- `new_change()` raises if the client is already on a change or on a past commit. Call `checkout()` first. + +See [Create a change](/graph_dev/create_change), [Submit a change](/graph_dev/submit_change) and [Time travelling](/graph_dev/time_travelling) for more on the versioning model. + +### When you need `COMMIT` inside a change + +`COMMIT` seals the pending writes of a change. It adds a commit to the history but does not publish anything to main. Whether a later query can see an earlier write depends on what the earlier write did: + +| An earlier query in the same change did... | Visible to later queries before `COMMIT`? | +| ------------------------------------------ | ----------------------------------------- | +| `CREATE` nodes or edges | **After `COMMIT`.** Run `COMMIT` so that a later `MATCH`, `MATCH ... CREATE` or `MERGE` sees the new entities | +| `SET`, `REMOVE` or `DELETE` on entities that already existed | Yes, immediately | +| anything, followed by `COMMIT` | Yes | +| writes earlier in the **same query** | Yes | + +**Rule of thumb: run `COMMIT` after any query that creates entities that a later query must `MATCH` or `MERGE`.** You can use several `COMMIT`s in one change. `CHANGE SUBMIT` commits whatever is still pending, so no `COMMIT` is needed right before it. + +```python +client.new_change() +client.query("CREATE (:Person {name: 'Alice'}), (:Person {name: 'Bob'})") +client.query("COMMIT") # make the new nodes visible to the next query +client.query("MATCH (a:Person {name: 'Alice'}), (b:Person {name: 'Bob'}) CREATE (a)-[:KNOWS]->(b)") +client.query("CHANGE SUBMIT") +client.checkout() +``` + +### Failed queries and conflicts + +- **Recovering from a failed query.** If a query stops with an execution error in the middle of a change, the writes it already made stay in the change. The safest recovery is `CHANGE DELETE`, then redoing the work. +- **Changes are isolated.** Main does not see a change's writes, even committed ones, until the change is submitted. Each change is rebased onto main at submit time. +- **Disjoint edits submit cleanly.** When two changes edit the same entity, the second `CHANGE SUBMIT` reports a conflict, for example `This change attempted to update Node 0 (...) which has been modified on main.` Conflict detection is coarse: creating an edge to a node that was modified on main also conflicts. +- **A failed submit rejects the whole change but leaves it open.** You can fix it and submit again, or run `CHANGE DELETE`. +- **IDs of newly created nodes and edges are provisional until `COMMIT` or `CHANGE SUBMIT`.** Read them after committing. IDs of existing nodes are stable across submits and restarts (see [node ID lookup](/query/matching#look-up-nodes-by-native-node-id)). + +## CREATE + +```cypher +CREATE (:Person {name: 'Alice', age: 30}) + +CREATE (a:Person:Employee {name: 'Bob'}) +RETURN a.name, id(a) + +CREATE (:Person {name: 'Mick'})-[:FRIEND_OF {since: 2020}]->(:Person {name: 'John'}) + +CREATE (a:T {x: 1}), (b:T {x: 2}), (a)-[:R]->(b) + +CREATE p = (:City {name: 'Paris'})-[:ROAD {km: 450}]->(:City {name: 'Lyon'}) +RETURN p + +CREATE (c:City {name: 'Li' + 'lle', pop: 2 * 1000, tags: ['a', 'b'], info: {zone: 1}, emb: (0.1, 0.2, 0.3)}) + +CREATE (n:Item {sku: 'A1'}) +SET n.price = 9.99 +RETURN n.sku + +CREATE (a:W {k: 'a'}) +WITH a +CREATE (b:W {k: 'b'})-[:NEXT]->(a) +``` + +To connect nodes that already exist, match them first. They must be committed, or created earlier **in the same query**: + +```cypher +MATCH (a:Person {name: 'Alice'}), (b:Person {name: 'Bob'}) +CREATE (a)-[:KNOWS {since: 2021}]->(b) +RETURN a.name, b.name +``` + +If you know their [native node IDs](/query/matching#look-up-nodes-by-native-node-id), address them directly. That is the fastest option, but the IDs must be **constants** written in the query, one pair per query: + +```cypher +MATCH (a) +WHERE a = 0 +MATCH (b) +WHERE b = 1 +CREATE (a)-[:KNOWS {since: 2021}]->(b) +``` + +**Rules:** + +- **Labels and edge types.** Create nodes with a label, `CREATE (n:Label)`, and give each edge a single type, `[:TYPE]`. +- **Labels are set at creation** and cannot be changed afterwards. +- **Property types are global.** Each property name has **one value type across the whole graph** (see `CALL db.propertyTypes()`). If `age` is an Integer, `CREATE (:X {age: 'old'})` and `SET n.age = 2.5` are both reported as type incompatibilities, and values are not widened from Integer to Double automatically. +- **Nulls.** Leave a key out of `CREATE` when it has no value, rather than writing `{name: null}`. +- **Write patterns use single hops.** Variable-length edges are for reading, and a matched edge variable is not reused in `CREATE`. +- **Parallel edges and self-loops are allowed.** An undirected `CREATE (a)-[:R]-(b)` is accepted and creates `a->b`. + + + Match the two endpoints of an edge with separate `MATCH` clauses and constant IDs. Taking both from one `UNWIND` row (`a = p[0] AND b = p[1]`) plans as a Cartesian product of two full node scans, which is expensive on large graphs. For thousands of edges, use [`LOAD PARQUET`](/import_data/parquet). + + +## MERGE + +`MERGE` matches the whole pattern or creates it. It supports `ON CREATE SET` and `ON MATCH SET`: + +```cypher +MERGE (n:Person {name: 'Alice'}) + ON CREATE SET n.created = true + ON MATCH SET n.seen = true +RETURN n.name +``` + +```cypher +MATCH (a:Person {name: 'Alice'}), (b:Person {name: 'Carol'}) +MERGE (a)-[e:KNOWS]->(b) + ON CREATE SET e.since = 2024 + +MERGE (a:Tag {name: 'x'})-[:LINKS]->(b:Tag {name: 'y'}) + +MATCH (p:Person {name: 'Bob'}) +MERGE (p)-[:OWNS]->(c:Car {model: 'Z'}) + +UNWIND ['a', 'b', 'a'] AS k +MERGE (t:Key {k: k}) +RETURN t.k +``` + +`MERGE` deduplicates within one query, so the last example creates two `Key` nodes, not three. + + + Run `COMMIT` before a `MERGE` that should find nodes created by an earlier query in the same change, otherwise `MERGE` creates them again. `MERGE` is the way to avoid duplicates. + + +## SET and REMOVE + +```cypher +MATCH (n:Person {name: 'Alice'}) +SET n.age = 31 + +MATCH (n:Person {name: 'Alice'}) +SET n.score = n.age * 1.1, n.updated = true + +MATCH (p:Product) +SET p.discountPrice = p.price * (1 - 0.15) + +MATCH (a:Person) +SET a.band = CASE WHEN a.age > 30 THEN 'senior' ELSE 'junior' END + +MATCH (a:Person) +SET a.place = coalesce(a.city, 'unknown') + +MATCH (n:Person {name: 'Alice'}), (m:Person {name: 'Bob'}) +SET n.age = m.age + +MATCH (n:Person {name: 'Alice'})-[e:KNOWS]->(m) +SET e.weight = 0.5 + +MATCH (n:Person) +WHERE n.age < 18 +SET n.isMinor = true + +MATCH (n:Person {name: 'Alice'}) +SET n.tags = ['a', 'b'], n.meta = {level: 3} + +MATCH (n:Person {name: 'Alice'}) +SET n.emb = (1.2, 2.0, 0.0, 12.0) + +MATCH (n:Person {name: 'Alice'}) +SET n.joined = datetime('2024-03-15T10:30:00Z') + +MATCH (a:Person) +WITH count(a) AS c +MATCH (s:Stats {name: 'people'}) +SET s.total = c + +MATCH (n) +WHERE n = 0 +SET n.flag = true + +UNWIND [0, 1, 2] AS x +MATCH (n) +WHERE n = x +SET n.flag = true + +MATCH (n:Person {name: 'Alice'}) +SET n.age = null + +MATCH (n:Person {name: 'Alice'}) +REMOVE n.score, n.flag +``` + +- `SET` creates the property if it does not exist yet. Setting a property to `null` clears it, and `REMOVE n.prop` removes it. +- The value can be any expression, including a `CASE`, a function call or a value from another matched entity. +- Address nodes by [native node ID](/query/matching#look-up-nodes-by-native-node-id) for the fastest updates. Batch updates use the same literal `UNWIND` form as reads. +- For a one-off embedding, `SET n.emb = (...)` is fine. To write embeddings onto many existing nodes, use [`LOAD EMBEDDING`](/vector-search#embeddings-as-node-properties) instead of a `SET` loop. + + + To update several properties at once, list the assignments: `SET n.a = 1, n.b = 2`. Map forms (`SET n += {...}`, `SET n = {...}`), label changes (`SET n:Label`) and aggregates directly inside `SET` (`SET n.c = count(*)`) are not yet supported. Compute aggregates in a `WITH` first, as in the `Stats` example above. + + +## DELETE and DETACH DELETE + +```cypher +MATCH (n:Person {name: 'Alice'}) +DETACH DELETE n + +MATCH (n:Person {name: 'Alice'})-[e:KNOWS]->(m) +DELETE e + +MATCH (n:Temp) +WHERE n.v > 3 +DELETE n + +MATCH (n:Person {name: 'Remy'})-[e]->(m:Person) +DELETE e, n +``` + +- `DETACH DELETE` removes a node together with all of its edges. +- Use `DETACH DELETE` for a node that still has edges. A plain `DELETE` reports `Cannot delete a node with relationships; use DETACH DELETE`. +- To delete a path, delete its nodes or edges. + +## Bulk writes + + + **Beyond a few thousand entities, use [`LOAD PARQUET`](/import_data/parquet) rather than Cypher writes.** Above roughly 5,000 nodes or 1,000 edges, write your data to Parquet and load it into a new graph. It loads 100,000 nodes and 200,000 edges in about 0.2 seconds, while Cypher-driven loads that `MATCH` both endpoints of every edge take minutes at a few thousand edges. Use Cypher writes for small, incremental changes. + + +For small volumes, inline the data with `UNWIND`: + +```cypher +UNWIND range(1, 1000) AS i +CREATE (:Num {v: i}) + +UNWIND ['Brest', 'Rennes', 'Nantes'] AS nm +MERGE (:City {name: nm}) + +UNWIND [['Brest', 'FR-29'], ['Rennes', 'FR-35']] AS row +CREATE (:City {name: row[0], code: row[1]}) +``` + +For rows with mixed types, such as `['Brest', 139000]`, use parallel lists: + +```cypher +WITH ['Brest', 'Rennes'] AS names, [139000, 220000] AS pops +UNWIND range(0, size(names) - 1) AS i +CREATE (:City {name: names[i], pop: pops[i]}) +``` + +`LOAD CSV` can drive writes too. Create the nodes, `COMMIT`, then create the edges (see [CSV import](/import_data/csv)). Alternatively `MERGE` both endpoints in a single pass, which deduplicates across the rows of one query so no `COMMIT` is needed: + +```cypher +LOAD CSV 'knows.csv' WITH HEADERS AS row + MERGE (a:Person {pid: toInteger(row.src)}) + MERGE (b:Person {pid: toInteger(row.dst)}) + CREATE (a)-[:KNOWS {since: toInteger(row.since)}]->(b) +``` + +Use `UNWIND` to loop over a list. + +**Building the query text in Python.** Queries are plain text, so escape quotes in your application: + +```python +def lit(v): + if isinstance(v, str): + return "'" + v.replace("\\", "\\\\").replace("'", "\\'") + "'" + if isinstance(v, bool): + return "true" if v else "false" + return repr(v) + +names = ["Alice", "O'Brien"] +client.query(f"UNWIND [{', '.join(lit(n) for n in names)}] AS nm MERGE (:Person {{name: nm}})") +``` + +Keep each query to a few thousand rows at most. diff --git a/shortest-path.mdx b/shortest-path.mdx index 5f80d43..c47037f 100644 --- a/shortest-path.mdx +++ b/shortest-path.mdx @@ -1,10 +1,14 @@ --- title: "Shortest Path" sidebarTitle: "Shortest Path" -description: "Find the shortest path between nodes in a TuringDB graph using the Dijkstra processor" +description: "Find the weighted shortest path between nodes in a TuringDB graph with the shortestPath statement (Dijkstra)" --- -TuringDB's `shortestPath` processor runs a Dijkstra search from a source set of nodes to a target set of nodes, returning the shortest distance and the path taken based on a numeric edge weight property. +TuringDB's `shortestPath` statement runs a Dijkstra search from a source set of nodes to a target set of nodes, returning the shortest distance and the path taken based on a numeric edge weight property. + + + `shortestPath` is a TuringDB statement for weighted shortest paths. For hop-count questions, or when you need the properties along the route, use [variable-length paths](/query/paths). + ## Sample dataset @@ -44,9 +48,9 @@ shortestPath(sourceSet, targetSet, edgePropName, distanceReturnVar, pathReturnVa | ------------------- | ------------------------------------------------------------------- | | `sourceSet` | Variable bound to the source node(s) via a preceding `MATCH` clause | | `targetSet` | Variable bound to the target node(s) via a preceding `MATCH` clause | -| `edgePropName` | Name of the numeric edge property to use as weight | -| `distanceReturnVar` | Output variable name for the total shortest distance | -| `pathReturnVar` | Output variable name for the path string | +| `edgePropName` | Name of the numeric edge property to use as weight, written as a bare identifier (`distance`), **not** a quoted string. Edges where it is null are skipped | +| `distanceReturnVar` | Output variable name for the total shortest distance. A Double, or an Integer if the weight is an integer | +| `pathReturnVar` | Output variable name for the path: a flat list of alternating IDs `[node, edge, node, edge, ..., node]` | ## Basic usage @@ -71,13 +75,12 @@ shortestPath(sourceSet, targetSet, edgePropName, distanceReturnVar, pathReturnVa ``` -+-------+--------------+ -| dist | path | -+-------+--------------+ -| 11.29 | (0)-[1]->(6) | -+-------+--------------+ + dist path +0 11.29 [0, 1, 6] ``` +`path` alternates node and edge IDs: node `0`, edge `1`, node `6`. + ### Returning only the distance or only the path Both output variables are always computed — you can choose to return one or both: @@ -103,11 +106,8 @@ Both output variables are always computed — you can choose to return one or bo ``` -+-------+ -| dist | -+-------+ -| 11.29 | -+-------+ + dist +0 11.29 ``` ## Multiple sources or targets @@ -137,17 +137,16 @@ You can pass multiple nodes to the source or target set — `shortestPath` will ``` -+------+--------------+ -| dist | path | -+------+--------------+ -| 12.6 | (1)-[5]->(6) | -+------+--------------+ + dist path +0 12.6 [1, 5, 6] ``` -## Limitations +If a set matches several source and target nodes, only the single best pair is returned. + +## Usage notes - Any variable on a pattern going into a `shortestPath` processor cannot be returned in the `RETURN` clause — they are consumed by the processor. The following query will produce an error: + The variables in the pattern going into `shortestPath` are consumed by the statement, so `RETURN` takes `dist` and `path`. Asking for a pattern variable such as `n` is reported as an error: ```jsx MATCH (n{name:"Ashchurch"}), (m{name:"Worcestershire Parkway"}) @@ -156,24 +155,30 @@ You can pass multiple nodes to the source or target set — `shortestPath` will ``` ``` - PLAN_ERROR: Unexpected exception: projection variable n not found in output column + ANALYZE_ERROR: Cannot return n after SHORTESTPATH. ``` - - You can still `MATCH` additional nodes outside of the source and target sets and include them in `RETURN`. The result will be a cartesian product with the shortest path output: +- **Directed.** Outgoing edges are followed. When the target cannot be reached from the source, the query returns zero rows. +- **`RETURN` follows directly.** Put `ORDER BY` and `LIMIT` on that `RETURN`. +- **Reading `path`.** `path` is a flat list of IDs. To get names and properties, map the IDs back with a second query (see below). +- **Where the other nodes come from.** A join can feed an endpoint: `MATCH (x {name: "Bromsgrove"})-[:LINK]->(a), (b {name: "Worcester Shrub Hill"}) shortestPath(a, b, distance, dist, path) RETURN dist, path`. - ```jsx - MATCH (n{name:"Ashchurch"}), (m{name:"Worcestershire Parkway"}), (p{name:"Cheltenham Spa"}) - shortestPath(n, m, distance, dist, path) - RETURN dist, path, p.name - ``` +## Naming the stations along the route - ``` - +-------+--------------+----------------+ - | dist | path | p.name | - +-------+--------------+----------------+ - | 11.29 | (0)-[1]->(6) | Cheltenham Spa | - +-------+--------------+----------------+ - ``` - \ No newline at end of file +To turn the IDs in `path` back into names, run a second query. The even positions of `path` are nodes: + +```python +row = client.query(""" +MATCH (a:Station {name: 'Bromsgrove'}), (b:Station {name: 'Worcester Shrub Hill'}) +shortestPath(a, b, distance, dist, path) +RETURN dist, path +""").iloc[0] + +node_ids = sorted({int(i) for i in row["path"][0::2]}) +names = client.query(f"UNWIND {node_ids} AS x MATCH (n) WHERE n = x RETURN id(n) AS id, n.name AS name") +``` + +The `UNWIND` list is a literal of distinct IDs and `x` is not returned, so this is a constant scan (see [lookup by node ID](/query/matching#look-up-nodes-by-native-node-id)). + +For large graphs prefer `shortestPath` over variable-length paths: variable-length matching enumerates every path. diff --git a/troubles/troubles.mdx b/troubles/troubles.mdx index 90c82f5..3a2cf45 100644 --- a/troubles/troubles.mdx +++ b/troubles/troubles.mdx @@ -27,6 +27,49 @@ turingdb -reset-default -turing-dir /path/to/your/turing/dir --- +## `Invalid response from the server` or `Proto header dataLen does not match chunk payload size` + +These two errors mean the client and the server disagree on the protocol. A TuringDB server speaks **either** JSON over HTTP **or** the binary protocol, chosen when it starts: + +| Error | Cause | Fix | +|---|---|---| +| `Invalid response from the server: Expecting value...` from `TuringDB(host=...)` | The server was started with `USE_TURING_PROTO=1` (binary mode) but you used the default JSON client | Connect with `TuringDB(type="native", host="localhost", port=6666)`, or restart the server without `USE_TURING_PROTO` | +| `Proto header dataLen does not match chunk payload size` from `TuringDB(type="native", ...)` | The server is a normal JSON server | Use the default `TuringDB(host="http://localhost:6666")`, or restart the server with `USE_TURING_PROTO=1` | + +See [client backends](/pythonsdk/reference#client-backends) for details. + +--- + +## `401 Unauthorized` + +The server was started with `-auth-on` and your client did not send a valid token. Pass `token="..."` to `TuringDB(...)`, or set the `TURINGDB_AUTH_TOKEN` environment variable. The JSON client raises this as an `httpx.HTTPStatusError`. + +--- + +## `CHANGE_NOT_FOUND` after `CHANGE SUBMIT` + +After `CHANGE SUBMIT` the Python client still points at the submitted change. Call `client.checkout()` to return to main before running the next query. + +--- + +## `Cannot perform CREATE outside of a write transaction` + +All writes (`CREATE`, `MERGE`, `SET`, `REMOVE`, `DELETE`) must run inside a change. Call `client.new_change()` first, and submit with `CHANGE SUBMIT`. See [Writing data](/query/writing). + +--- + +## A `MATCH` right after a `CREATE` returns no rows + +Entities created earlier in a change are not visible to later queries until you run `COMMIT`. Run `COMMIT` after creating nodes and before matching them to create edges or to `MERGE`. See [Writing data](/query/writing#when-you-need-commit-inside-a-change). + +--- + +## `Function 'X' does not exist` + +Function names are case-sensitive (`toInteger`, not `TOINTEGER`), and some functions are not yet available in TuringDB. In v3, `type()` replaces `edgeType()`. See [Functions and data types](/query/functions). + +--- + ## Contact us If you have any feedback, bugs to report, or features requests please send us an email: [team@turingdb.ai](team@turingdb.ai) diff --git a/tutorials/create_graph.mdx b/tutorials/create_graph.mdx index e36246f..de9e6d2 100644 --- a/tutorials/create_graph.mdx +++ b/tutorials/create_graph.mdx @@ -149,7 +149,7 @@ graph [ ## Key Points -1. **Single CREATE Query**: Due to TuringDB's current Cypher limitations, the entire graph must be created in a single CREATE statement. +1. **Single CREATE Query**: This tutorial builds the whole graph in one `CREATE` statement, which lets the edges refer to the nodes by variable. You can also create the nodes first, `COMMIT`, and then `MATCH` them to create the edges, see [Writing data](/query/writing). To import data in bulk, use [`LOAD PARQUET`](/import_data/parquet) or [`LOAD JSONL`](/import_data/jsonl). 2. **Change Management**: Always use the change system when modifying data: - Create a new change with `CHANGE NEW` - Checkout into the change @@ -160,13 +160,13 @@ graph [ ## Troubleshooting -- **Large Graphs**: For very large GML files, you might hit query size limits. Consider splitting the import or contact support for guidance. +- **Large Graphs**: For very large GML files, you might hit query size limits. Convert the graph to Parquet and use [`LOAD PARQUET`](/import_data/parquet), which loads hundreds of thousands of nodes and edges in well under a second, or use [`LOAD JSONL`](/import_data/jsonl). - **Invalid Characters**: The sanitization function handles common invalid characters, but you may need to extend it for specific use cases. - **Memory Usage**: Large graphs are loaded entirely into memory before conversion, so ensure you have sufficient RAM. ## Next Steps -After importing your GML file, you can query the data using TuringDB's supported Cypher subset: +After importing your GML file, you can query the data with [Cypher](/query/cypher_subset): ```python # Find all Person nodes diff --git a/vector-search.mdx b/vector-search.mdx index 304eb0b..b276125 100644 --- a/vector-search.mdx +++ b/vector-search.mdx @@ -6,43 +6,54 @@ description: "Search through high-dimensional embeddings and connect the results TuringDB has a built-in vector index that lets you run k-nearest-neighbor searches over embedding vectors. You bring your own embeddings, from any model or provider, and TuringDB handles the indexing, storage, and fast retrieval. -Each vector is associated with a numerical ID. That ID can be a node property, an edge property, or a foreign key referencing data in an external system. This keeps the index lightweight and flexible: the vector store doesn't need to know what your data looks like. +Each vector is stored with a numerical ID. When that ID is the **native node ID** of the node the vector describes (`id(n)`), search results are the nodes themselves, so you can read their properties and expand from them with no extra lookup. - Vector indexes live at the TuringDB root level, independent of graphs and versioning. A single vector index can serve searches across multiple graphs and commits. + Vector indexes live at the TuringDB root level (`/vector/`), independent of graphs and versioning. A single vector index can serve searches across multiple graphs and commits, and it persists across restarts. + + Embedding literals use **parentheses**: `(1.2, 2.0, 0.0)`. Square brackets `[1.2, 2.0, 0.0]` make a list, which is a different type. An embedding literal needs at least 2 numbers. + + ## Create a vector index -A vector index is defined by a name, a dimension, and a distance metric. +A vector index is defined by a name, a dimension, a distance metric and, optionally, an index type. **Syntax:** ``` -CREATE VECTOR INDEX WITH DIMENSION METRIC +CREATE VECTOR INDEX WITH DIMENSION METRIC [TYPE ] ``` | Parameter | Description | | ---------- | --------------------------------------------------------------------------- | | `` | Identifier for the vector index | -| `` | Dimension of the embedding vectors (positive integer) | -| `` | Distance metric: `EUCLID` (Euclidean distance) or `COSINE` (cosine similarity) | +| `` | Dimension of the embedding vectors (positive integer, at most 8192) | +| `` | `COSINE` or `EUCLID` (uppercase) | +| `` | Optional. `FLAT` (exact, the default) or `HNSW` (approximate, for large collections) | ```jsx CREATE VECTOR INDEX doc_embeddings WITH DIMENSION 768 METRIC COSINE + CREATE VECTOR INDEX doc_hnsw WITH DIMENSION 768 METRIC EUCLID TYPE HNSW ``` ```python client.query("CREATE VECTOR INDEX doc_embeddings WITH DIMENSION 768 METRIC COSINE") + client.query("CREATE VECTOR INDEX doc_hnsw WITH DIMENSION 768 METRIC EUCLID TYPE HNSW") ``` + + The metrics return raw scores. `COSINE` scores are the **inner product** of the stored and query vectors, so L2-normalise both to get true cosine similarity. `EUCLID` scores are the **squared** L2 distance. + + ## Load embeddings -Once the index exists, load your pre-computed embeddings from a file. Each row in the file maps a numerical ID to a vector. +Once the index exists, load your pre-computed embeddings from a header-less CSV file. Each line is `id,v1,v2,...,vd`, where `id` is normally the native TuringDB node ID (`id(n)`). The file path is relative to your TuringDB data directory (`~/.turing/data` by default). @@ -73,36 +84,48 @@ LOAD VECTOR FROM "" IN +To build that file from your graph, export `id(n)` next to the text you want to embed, compute the embeddings in Python, and write `id,v1,...,vd` lines: + +```python +docs = client.query("MATCH (d:Document) RETURN id(d) AS id, d.text AS text") +vectors = model.encode(docs["text"].tolist()) # any embedding model + +with open("/home/me/.turing/data/document_vectors.csv", "w") as f: + for node_id, vec in zip(docs["id"], vectors): + f.write(f"{node_id}," + ",".join(f"{x:.6f}" for x in vec) + "\n") +``` + ## Search -`VECTOR SEARCH` finds the k nearest neighbors of a query vector and yields their IDs. It is a read statement, so you can chain it with `MATCH` to pull back the actual graph data. +`VECTOR SEARCH` finds the k nearest neighbors of a query vector and yields their IDs and scores. It is a read statement, so you can chain it with `MATCH`, `WHERE` and `RETURN` to pull back the actual graph data. **Syntax:** ``` -VECTOR SEARCH IN FOR () YIELD +VECTOR SEARCH IN FOR () YIELD ids [AS ] [, score [AS ]] ``` | Parameter | Description | | -------------- | ---------------------------------------------------------- | | `` | Name of the vector index to search | | `` | Number of nearest neighbors to return (positive integer) | -| `` | Query vector as a list literal of float values | -| `` | Variable name to hold the result IDs | +| `` | Query vector as an embedding literal, in parentheses. Its dimension must match the index | +| `ids` | The matched IDs, best first. Bound as nodes when the index is keyed by native node IDs | +| `score` | The score of each match | ### Standalone search ```jsx - VECTOR SEARCH IN doc_embeddings FOR 5 (0.12, 0.45, 0.78, 0.33) YIELD ids - RETURN ids + VECTOR SEARCH IN doc_embeddings FOR 5 (0.12, 0.45, 0.78, 0.33) YIELD ids, score + RETURN ids, score ``` ```python df = client.query(""" - VECTOR SEARCH IN doc_embeddings FOR 5 (0.12, 0.45, 0.78, 0.33) YIELD ids - RETURN ids + VECTOR SEARCH IN doc_embeddings FOR 5 (0.12, 0.45, 0.78, 0.33) YIELD ids, score + RETURN ids, score """) print(df) ``` @@ -111,29 +134,37 @@ VECTOR SEARCH IN FOR () YIELD ### Combining with MATCH -This is where it gets interesting. Chain `VECTOR SEARCH` with a `MATCH` clause to join the nearest-neighbor IDs back to your graph: +This is where it gets interesting. When the vector file is keyed by native node IDs, `ids` **is the node**. Read its properties directly, filter on `score`, or expand from it with a pattern: ```jsx - VECTOR SEARCH IN doc_embeddings FOR 10 (0.12, 0.45, 0.78, 0.33) YIELD ids - MATCH (d:Document) WHERE d.id = ids - RETURN d.title, d.summary + VECTOR SEARCH IN doc_embeddings FOR 10 (0.12, 0.45, 0.78, 0.33) YIELD ids AS doc, score + WHERE score > 0.8 + MATCH (doc)-[:ABOUT]->(t:Topic) + RETURN doc.title, t.name, score ``` ```python df = client.query(""" - VECTOR SEARCH IN doc_embeddings FOR 10 (0.12, 0.45, 0.78, 0.33) YIELD ids - MATCH (d:Document) WHERE d.id = ids - RETURN d.title, d.summary + VECTOR SEARCH IN doc_embeddings FOR 10 (0.12, 0.45, 0.78, 0.33) YIELD ids AS doc, score + WHERE score > 0.8 + MATCH (doc)-[:ABOUT]->(t:Topic) + RETURN doc.title, t.name, score """) print(df) ``` -The `ids` variable works exactly like a variable introduced by `CALL ... YIELD`, so any subsequent `MATCH` clause can reference it. + + **Use `ids` directly.** `ids` is already the node, so read properties from it (`ids.title`) or expand from it (`MATCH (ids)-[...]->(...)`). Re-matching it with `MATCH (d:Document) WHERE d.id = ids` scans the whole label and cross-joins it with the results, so key the vector file by native node IDs, as above. If you key the file by your own IDs, joining them back needs a label scan. + + + + Node IDs are renumbered by [`MERGE_DATAPARTS`](/concepts/dataparts). After compacting a graph, rebuild any vector index keyed by node IDs. + ## Manage indexes @@ -143,6 +174,8 @@ The `ids` variable works exactly like a variable introduced by `CALL ... YIELD`, SHOW VECTOR INDEXES ``` +This returns the name and dimension of each index. + ### Delete a vector index ```jsx @@ -159,13 +192,12 @@ This removes the index and frees the associated resources. // 1. Create a vector index for product embeddings CREATE VECTOR INDEX product_vectors WITH DIMENSION 384 METRIC COSINE - // 2. Load embeddings generated by your model + // 2. Load embeddings generated by your model (keyed by native node ID) LOAD VECTOR FROM "product_embeddings.csv" IN product_vectors // 3. Find the 10 most similar products to a query embedding - VECTOR SEARCH IN product_vectors FOR 10 (0.15, 0.82, 0.44, 0.91) YIELD ids - MATCH (p:Product) WHERE p.id = ids - RETURN p.name, p.price, p.category + VECTOR SEARCH IN product_vectors FOR 10 (0.15, 0.82, 0.44, 0.91) YIELD ids AS p, score + RETURN p.name, p.price, p.category, score // 4. Inspect existing indexes SHOW VECTOR INDEXES @@ -183,14 +215,13 @@ This removes the index and frees the associated resources. # 1. Create a vector index for product embeddings client.query("CREATE VECTOR INDEX product_vectors WITH DIMENSION 384 METRIC COSINE") - # 2. Load embeddings generated by your model + # 2. Load embeddings generated by your model (keyed by native node ID) client.query('LOAD VECTOR FROM "product_embeddings.csv" IN product_vectors') # 3. Find the 10 most similar products to a query embedding df = client.query(""" - VECTOR SEARCH IN product_vectors FOR 10 (0.15, 0.82, 0.44, 0.91) YIELD ids - MATCH (p:Product) WHERE p.id = ids - RETURN p.name, p.price, p.category + VECTOR SEARCH IN product_vectors FOR 10 (0.15, 0.82, 0.44, 0.91) YIELD ids AS p, score + RETURN p.name, p.price, p.category, score """) print(df) @@ -205,9 +236,7 @@ This removes the index and frees the associated resources. ## Embeddings as node properties -The vector index above is a standalone, root-level structure keyed by numeric IDs. Separately, you can store an embedding directly as a **node or edge property** (type `Embedding`) and compare embeddings inline with the `cosine_similarity` and `euclidean_distance` functions — no index required. - -Embedding literals use **parentheses** `(...)`, not square brackets (which are list literals): +The vector index above is a standalone, root-level structure keyed by numeric IDs. Separately, you can store an embedding directly as a **node or edge property** (type `Embedding`) and compare embeddings inline with the `cosine_similarity` and `euclidean_distance` functions, with no index required. ```jsx // Set an embedding property (inside a change) @@ -218,9 +247,11 @@ MATCH (n:Document) RETURN n.name, cosine_similarity(n.emb, (0.12, 0.45, 0.78, 0. MATCH (n:Document) RETURN n.name, euclidean_distance(n.emb, (0.12, 0.45, 0.78, 0.33)) ``` +`cosine_similarity` is true cosine similarity and `euclidean_distance` is true (not squared) L2 distance. Both accept Embeddings only, not lists, and work in `RETURN`, `WHERE` and `ORDER BY`. This is a brute-force scan, so for large collections use a vector index. + ### Bulk-loading embeddings from Parquet — `LOAD EMBEDDING FROM` -To attach embeddings to **existing** nodes in bulk, load them from a Parquet file. This is a write, so run it inside a change. +To attach embeddings to **existing** nodes in bulk, load them from a Parquet file. This is a write, so run it inside a change. `LOAD EMBEDDING` is the fastest way to write many embeddings: about 0.1 s for 20,000 embeddings of 384 dimensions, against about 2.5 ms per node for a loop of `SET n.emb = (...)`. **Syntax:** ``` @@ -231,21 +262,36 @@ The Parquet file (relative to the TuringDB `data` directory) must have exactly t | Column | Type | Description | | ----------- | -------------------------------------- | -------------------------------------------------------- | -| `node_id` | `INT64` | Internal TuringDB node ID to attach the embedding to | +| `node_id` | `INT64` | Native TuringDB node ID (`id(n)`) to attach the embedding to | | `embedding` | `FIXED_LEN_BYTE_ARRAY` (`dim × 4` bytes) | Little-endian float32 vector; dimension inferred from width | ```python -change = client.new_change() -client.checkout(change=change) -client.query('LOAD EMBEDDING FROM "embeddings.parquet" AS emb') -client.query("COMMIT") +import numpy as np, pyarrow as pa, pyarrow.parquet as pq +from pathlib import Path + +docs = client.query("MATCH (d:Document) RETURN id(d) AS node_id, d.text AS text") +vecs = model.encode(docs["text"].tolist()).astype(" - When importing a graph from JSONL, you can mark embedding properties at load time instead — see [`LOAD JSONL ... WITH EMBEDDINGS`](/import_data/jsonl#loading-embedding-properties). + When importing a graph from JSONL, you can mark embedding properties at load time instead. See [`LOAD JSONL ... WITH EMBEDDINGS`](/import_data/jsonl#loading-embedding-properties). -