Introducing a new metric to understand if pipelines are fully synced

Lag is the quantity of events which have been consumed by Streamkap but not yet delivered to the destination. 

While this should be 0, there are occasions where it can build up, such as:

• Sudden large influx of events and this could include snapshots
• The destination connector throughput capacity needs increasing
• The destination vendor service drops or is deterioated

Pipeline Summaries

Destination Connectors

Topic Level

Excited to announce that we now support additional services per account

What is a Service?

A service is the underlying architecture behind Streamkap. For example, this includes Apache Kafka, Apache Flink, Debezium, Connectors and so on, configured in such a way to be highly performant and scalable. 

At present when you login to Streamkap you are presented one service to work with.

What does this mean?

The problem with one service is that your development or staging pipelines are mixed in with your production pipelines and this can result in excess noise from alerts or the possibility of introducing mistakes. 

Equally, the development pipelines are sitting on top of a production grade service which is likely un-necessary. 

Going forward we now support additional services, allowing you to split out production from development/staging. 

Today, we support additional services in the same region. Soon, we will be supporting additional services of varying specifications and in other regions for increased flexibility

Alerts have now been moved to a dedicated page accessible on the sidebar. 

The gear icon is still used to configure the types of alerts and channels to receive on. 

We will be adding soon additional filters and a better UI to set your preferences

Snapshots (Also known as backfilling) previously happened when creating the source connector or adding new tables to the source connector. 

Going forward, this will now happen at the pipeline stage

• Created or edited source connectors will not snapshot upon creation
• When creating a new pipeline, a toggle will be default ticked to snapshot all the tables
• When editing a pipeline and adding new tables, a toggle will be default ticked to snapshot the new tables

We're excited to announce that Streamkap is now an official partner with ClickHouse along with the release of an updated connector.

We have collaborated with ClickHouse over the last few months to ensure this is the best integration available for streaming CDC (Change Data Capture) and other event data into ClickHouse. The new connector has brings support for more data types, upserts and high throughput with extremely low latency. 🚀

Learn more on the ClickHouse blog where we wrote about the technical challenges, features and performance benchmarks in building this connector.

Streamkap now supports an additional 2 deployment options for Customers who need an on-prem solution

• Proxy Agent - For use when you wish to connect on-prem databases and send through to Streamkap Cloud
• BYOC - Bring Your Own Cloud - This allows for deployment of Streamkap within your cloud environment. Available only on an Enterprise plan

Please get in touch if you would like to learn more

Elasticsearch is a popular search and analytics engine. While it's great for search, it's painful to analyse data within the platform. This connector allows you to send the data elsewhere for analysis. 

Elasticsearch Setup Docs

Under each connector, there is now real-time time series metrics for your usage analysis

Event Metrics Tracked

The metrics we support for analysis are:

• Total Written
• Created/Updated/Deleted
• Snapshots
• Volume (GB)
• Latency

With Streamkap, you can now stream real-time data into CockroachDB.

CockroachDB is a popular high performant distributed database.

CoackroachDB Setup Docs