Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions local-preview-playbook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,13 +191,6 @@ asciidoc:
nodejs-driver-url: 'https://github.com/apache/cassandra-nodejs-driver'
python-driver-url: 'https://github.com/apache/cassandra-python-driver'
scala-driver-url: 'https://github.com/apache/cassandra-spark-connector'
cass-driver-cpp-shield: 'image:https://img.shields.io/github/v/tag/datastax/cpp-driver?label=latest[alt="Latest cpp-driver release on GitHub",link="{cpp-driver-url}/tags"]'
cass-driver-csharp-shield: 'image:https://img.shields.io/nuget/v/CassandraCSharpDriver?label=latest[alt="Latest CassandraCSharpDriver release on NuGet",link="https://www.nuget.org/packages/CassandraCSharpDriver"]'
cass-driver-go-shield: 'image:https://img.shields.io/github/v/tag/apache/cassandra-gocql-driver?label=latest%20gocql[alt="Latest gocql release on GitHub",link="{go-driver-url}/tags"]'
cass-driver-java-shield: 'image:https://img.shields.io/github/v/tag/apache/cassandra-java-driver?label=latest[alt="Latest cassandra-java-driver release on GitHub",link="{java-driver-url}/tags"]'
cass-driver-nodejs-shield: 'image:https://img.shields.io/github/v/tag/apache/cassandra-nodejs-driver?label=latest[alt="Latest cassandra-driver release on GitHub",link="{nodejs-driver-url}/tags"]'
cass-driver-python-shield: 'image:https://img.shields.io/github/v/tag/apache/cassandra-python-driver?label=latest[alt="Latest cassandra-driver release on GitHub",link="{python-driver-url}/tags"]'
cass-driver-scala-shield: 'image:https://img.shields.io/github/v/tag/apache/cassandra-spark-connector?label=latest[alt="Latest cassandra-spark-connector release on GitHub",link="{scala-driver-url}/releases"]'
data-api: 'Data API'
csharp-client-api-ref-url: 'xref:astra-api-docs:ROOT:attachment$csharp-client'
py-client-api-ref-url-2x: 'xref:astra-api-docs:ROOT:attachment$python-client/astrapy'
Expand All @@ -209,10 +202,6 @@ asciidoc:
java-client-repo-url: 'https://github.com/datastax/astra-db-java'
csharp-client-repo-url: 'https://github.com/datastax/astra-db-csharp'
python-client-python-version: '3.9 to 3.14'
dataapi-java-client-shield: 'image:https://img.shields.io/maven-central/v/com.datastax.astra/astra-db-java.svg?label=latest[alt="Latest astra-db-java release on Maven Central",link="https://search.maven.org/artifact/com.datastax.astra/astra-db-java"]'
dataapi-python-client-shield: 'image:https://img.shields.io/github/v/tag/datastax/astrapy?label=latest[alt="Latest astrapy release on GitHub",link="{python-client-repo-url}/releases"]'
dataapi-typescript-client-shield: 'image:https://img.shields.io/github/v/tag/datastax/astra-db-ts?label=latest[alt="Latest astra-db-ts release on GitHub",link="{typescript-client-repo-url}/releases"]'
dataapi-csharp-client-shield: 'image:https://img.shields.io/github/v/tag/datastax/astra-db-csharp?label=latest[alt="Latest astra-db-csharp release on GitHub",link="{csharp-client-repo-url}/releases"]'
agent: 'DataStax Agent'
repair-service: 'Repair Service'
backup-service: 'Backup Service'
Expand Down
3 changes: 2 additions & 1 deletion modules/ROOT/partials/sinks/get-started.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ You can pass the configuration directly or with a configuration file.
--sink-config-file configs.json
----
+
.Example configuration data structure
Example configuration data structure:
+
[source,json]
----
include::common:streaming:example$connectors/sinks/{connectorType}/sample-data.json[]
Expand Down
3 changes: 2 additions & 1 deletion modules/ROOT/partials/sources/get-started.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ You can pass the configuration directly or with a configuration file.
--source-config-file configs.json
----
+
.Example configuration data structure
Example configuration data structure:
+
[source,json]
----
include::common:streaming:example$connectors/sources/{connectorType}/sample-data.json[]
Expand Down
5 changes: 4 additions & 1 deletion modules/components/pages/heartbeat-vm.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,13 @@ Run the binary file `pulsar-heartbeat-<version>-<os>-<arch>`.

Here is an example of how to set up and run the application:

[source,bash,subs="+attributes"]
[source,bash]
----
export PULSAR_OPS_MONITOR_CFG=/Users/username/go/src/github.com/datastax/pulsar-heartbeat/config/runtime.yml
----

[source,console,subs="+attributes"]
----
./pulsar-heartbeat-{heartbeat-version}-linux-amd64
2022/01/11 15:16:57 info config file /Users/username/go/src/github.com/datastax/pulsar-heartbeat/config/runtime.yml
----
Expand Down
8 changes: 3 additions & 5 deletions modules/components/pages/pulsar-sql.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,6 @@ trino> show catalogs;
+
Notice the similarities between your {pulsar-short} tenant/namespaces and Trino's output:
+
.Result
[source,console]
----
Catalog
Expand All @@ -152,7 +151,6 @@ trino> show tables in pulsar."public/default";
+
Notice the topic that you created from the data-generator source connector <<create-topic,earlier in this guide>>:
+
.Result
[source,console]
----
Table
Expand Down Expand Up @@ -254,15 +252,15 @@ javac PrestoExample.java

. Download the https://repo1.maven.org/maven2/io/trino/trino-jdbc/443/trino-jdbc-443.jar[version 443 Trino JDBC driver] to the same folder as the Java class.

. Run the client and observe the output.
`-cp` tells Java to use the downloaded trino-jdbc jar in your classpath to connect.
. Run the client and observe the output:
+
[source,shell]
----
java -cp .:trino-jdbc-443.jar PrestoExample
----
+
.Result
`-cp` tells Java to use the downloaded trino-jdbc jar in your classpath to connect.
+
[source,console]
----
Connection established......
Expand Down
74 changes: 51 additions & 23 deletions modules/install-upgrade/pages/cluster-sizing-reference.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,80 +14,94 @@ The number of nodes in a highly available cluster depends on the number of zones
{company} recommends aligning the replication factor with the AZ count.
For example, if there are 3 zones, then set a replication factor of 3.

[cols=4*,options=header]
[cols=4*]
|===
|Environment
|Server Component
|Node Count
|Note
|Environment |Server Component |Node Count |Note

|SANDBOX (STANDALONE)
|{zookeeper-short}, broker, and bookie (combined)
|1
|

.3+|DEVELOPMENT (1 region, 1 zone)
|DEVELOPMENT (1 region, 1 zone)
|{zookeeper-short}
|1
|

|DEVELOPMENT (1 region, 1 zone)
|Broker
|1
|

|DEVELOPMENT (1 region, 1 zone)
|Bookie
|1
|

.5+|SINGLE REGION TESTING ENVIRONMENT (1 region, 1 zone)
|SINGLE REGION TESTING ENVIRONMENT (1 region, 1 zone)
|{zookeeper-short}
|3
|

|SINGLE REGION TESTING ENVIRONMENT (1 region, 1 zone)
|Broker
|3
|

|SINGLE REGION TESTING ENVIRONMENT (1 region, 1 zone)
|Bookie
|3
|

|SINGLE REGION TESTING ENVIRONMENT (1 region, 1 zone)
|{pulsar-short} proxy
|3
|

|SINGLE REGION TESTING ENVIRONMENT (1 region, 1 zone)
|Function worker (dedicated)
|3
|

.6+|HIGH-AVAILABILITY PRODUCTION ENVIRONMENT (1 region, 3 zones)
|HIGH-AVAILABILITY PRODUCTION ENVIRONMENT (1 region, 3 zones)
|{zookeeper-short}
|5
|2/2/1 nodes^*^
|2/2/1 nodes (assuming a replication factor of 3)

|HIGH-AVAILABILITY PRODUCTION ENVIRONMENT (1 region, 3 zones)
|Broker
|3
|1 nodes per AZ^*^
|1 nodes per AZ (assuming a replication factor of 3)

|HIGH-AVAILABILITY PRODUCTION ENVIRONMENT (1 region, 3 zones)
|Function worker (dedicated)
|3
|1 node per AZ^*^
|1 node per AZ (assuming a replication factor of 3)
The number of function workers depends on the cluster's functions workload.

|HIGH-AVAILABILITY PRODUCTION ENVIRONMENT (1 region, 3 zones)
|Bookie
|6
|2 nodes per AZ^*^
|2 nodes per AZ (assuming a replication factor of 3)

|HIGH-AVAILABILITY PRODUCTION ENVIRONMENT (1 region, 3 zones)
|{pulsar-short} proxy
|3
|1 node per AZ^*^
|1 node per AZ (assuming a replication factor of 3)

|HIGH-AVAILABILITY PRODUCTION ENVIRONMENT (1 region, 3 zones)
|AutoRecovery
|3
|1 per AZ^*^
|1 per AZ (assuming a replication factor of 3)
|===

^*^ Assuming a replication factor of 3.

== Hardware sizing

The following table lists the minimum hardware requirements for a {pulsar-short} cluster.

[cols=4*,options=header]
[cols=4*]
|===
|Environment
|Server Component
|Hardware Spec
|Note
|Environment |Server Component |Hardware Spec |Note

|DEVELOPMENT
|{zookeeper-short}, broker, and {bookkeeper-short}
Expand All @@ -96,48 +110,62 @@ a|* CPU: 8 vCPU
* Data Disk: 64 GB SSD
a|Data disks are shared by {zookeeper-short} data, {bookkeeper-short} journal, and {bookkeeper-short} ledger

.4+|TEST
|TEST
|{zookeeper-short}
a|* CPU: 2 vCPU
* Memory: 4 GB
* Data Disk: 32 GB SSD
|

|TEST
|Broker and function worker
a|* CPU: 8 vCPU
* Memory: 16 GB
|No data disk

|TEST
|Bookie
a|* CPU: 4 vCPU
* Memory: 16 GB
* Data Disk Journal: 32 GB SSD
* Data Disk Ledger: 256 GB SSD
|

|TEST
|{pulsar-short} proxy, function worker
a|* CPU: 4 vCPU
* Memory: 8 GB
|

.6+|PRODUCTION
|PRODUCTION
|{zookeeper-short}
a|* CPU: 4 vCPU
* Memory: 4 GB
* Data Disk: 64 GB SSD
|

|PRODUCTION
|Broker
a|* CPU: 8 vCPU
* Memory: 16 GB
* No data disk
|

|PRODUCTION
|Bookie
a|* CPU: 8 vCPU
* Memory: 32 GB
* Data Disk Journal: 256 GB SSD
* Data Disk Ledger: 1024 GB SSD
|Ledger disk capacity can be beyond 1TB.

|PRODUCTION
|{pulsar-short} proxy, AutoRecovery
a|* CPU: 4 vCPU
* Memory: 16 GB
|

|PRODUCTION
|Function worker (dedicated)
a|* CPU: 4 vCPU
* Memory: 8 GB
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Until all subscribers have acknowledged receipt of a message, the broker will no

Unfortunately, this isn't realistic. Things happen. Processes lock up. Networks go down. If we tell {pulsar-short} to indefinitely attempt message delivery to all subscribers on all topics, the backlog would grow out of control, with bookie disks continuously filling and never draining. So, guaranteed message delivery must be managed with some rules.

{pulsar-short} has different ways of managing the broker's backlog (ie: guaranteed message delivery). Combining these different settings make up the rules of message retention. The rules of message retention directly impact how full a bookie's disk can be. We can't cover every possability within {pulsar-short}'s message retention system, so we will focus on 3 key areas and let those drive our sizing calculation. For more on message retention and expiration, see https://pulsar.apache.org/docs/concepts-messaging/#message-retention-and-expiry[{pulsar-short}'s message retention and expiry documentation].
{pulsar-short} has different ways of managing the broker's backlog (ie: guaranteed message delivery). Combining these different settings make up the rules of message retention. The rules of message retention directly impact how full a bookie's disk can be. We can't cover every possibility within {pulsar-short}'s message retention system, so we will focus on 3 key areas and let those drive our sizing calculation. For more on message retention and expiration, see https://pulsar.apache.org/docs/concepts-messaging/#message-retention-and-expiry[{pulsar-short}'s message retention and expiry documentation].

=== Retention policy

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ include::ROOT:partial$luna-name.adoc[]

The resulting {product} deployment includes:

* *Tiered Storage:* Offload historical messages to more cost effective object storages such as AWS S3, Azure Blob, Google Cloud Storage, and HDFS.
* *Tiered Storage:* Offload historical messages to more cost effective object storage such as AWS S3, Azure Blob, Google Cloud Storage, and HDFS.
* *Built-in Schema Registry:* Guarantee messaging type safety on a per-topic basis without relying on any external facility.
* *{pulsar-short} I/O connectors:* Enables {pulsar-short} to exchange data with external systems, either as sources or sinks.
* *{pulsar-short} Function:* Lightweight compute extensions of {pulsar-short} brokers which enable real-time simple event processing within {pulsar-short}.
Expand Down
1 change: 0 additions & 1 deletion modules/operations/pages/auth.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,6 @@ pulsar@pulsar-bastion-85c9b777f6-gt9ct:/pulsar$ curl -d "client_id=test-client"
"http://test-keycloak/auth/realms/pulsar/protocol/openid-connect/token"
----
+
.Results
[source,json]
----
{
Expand Down
1 change: 0 additions & 1 deletion modules/operations/pages/functions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ There are multiple ways to add functions using the {pulsar-short} Admin CLI.
+
For example purposes, you can use the following sample function that consumes a message from one topic, adds an exclamation point, and then publishes the result to another topic.
+
.function.py
[source,python]
----
from pulsar import Function
Expand Down
Loading