running kafka streams inside kafka connect

[RK Sing]

We have a requirement to do Single row transformations, basic joins,
deduping and routing from the source to the destination Kafka topics.

We want to use Kafka-connect as the platform which is running Kafka stream
inside. Has anyone used Kafkastreams inside Kafka connect ??

Is this pattern ok for Kafka-connect?

regards
dhurandar

[Ryanne Dolan]

Dhurandar, definitely! Connect and Streams are both agnostic to how their
workers are run. They aren't really platforms per se. You just need to spin
up one or more workers and they do their thing. So a Streams app doesn't
run "inside" Connect, but you can certainly have Connect and Streams
workers talk to each other through Kafka topics.

Ryanne

Post by RK Sing
We have a requirement to do Single row transformations, basic joins,
deduping and routing from the source to the destination Kafka topics.
We want to use Kafka-connect as the platform which is running Kafka stream
inside. Has anyone used Kafkastreams inside Kafka connect ??
Is this pattern ok for Kafka-connect?
regards
dhurandar

[RK Sing]

Thank you Ryanne for the answer, My question is can I run Streams app
inside Connect, what I mean is since Streams is a library, can I build a
custom Source connector and use Streams in there . or using Streams in
transforms??
We want to containerize Kafka connect as in wrap it in a docker, and since
all the Kafka-connect configs are stored in Kafka itself, so the container
can be stateless, we can spin up containers as and when needed, we want
this approach to fetch data from microservices and golden gate oracle. etc.

Post by Ryanne Dolan
Dhurandar, definitely! Connect and Streams are both agnostic to how their
workers are run. They aren't really platforms per se. You just need to spin
up one or more workers and they do their thing. So a Streams app doesn't
run "inside" Connect, but you can certainly have Connect and Streams
workers talk to each other through Kafka topics.
Ryanne

Post by RK Sing
We have a requirement to do Single row transformations, basic joins,
deduping and routing from the source to the destination Kafka topics.
We want to use Kafka-connect as the platform which is running Kafka

stream

Post by RK Sing
inside. Has anyone used Kafkastreams inside Kafka connect ??
Is this pattern ok for Kafka-connect?
regards
dhurandar

[Matthias J. Sax]

Streams is not designed to be run inside Connect, and this won't work.

What you can do is, to import the data via connect into a "staging
topic" and then read this "staging topic" with a Kafka Streams
application and apply the transformations etc to write the data into the
actual target topics.


-Matthias

Post by RK Sing
Thank you Ryanne for the answer, My question is can I run Streams app
inside Connect, what I mean is since Streams is a library, can I build a
custom Source connector and use Streams in there . or using Streams in
transforms??
We want to containerize Kafka connect as in wrap it in a docker, and since
all the Kafka-connect configs are stored in Kafka itself, so the container
can be stateless, we can spin up containers as and when needed, we want
this approach to fetch data from microservices and golden gate oracle. etc.

Post by Ryanne Dolan
Dhurandar, definitely! Connect and Streams are both agnostic to how their
workers are run. They aren't really platforms per se. You just need to spin
up one or more workers and they do their thing. So a Streams app doesn't
run "inside" Connect, but you can certainly have Connect and Streams
workers talk to each other through Kafka topics.
Ryanne

Post by RK Sing
We have a requirement to do Single row transformations, basic joins,
deduping and routing from the source to the destination Kafka topics.
We want to use Kafka-connect as the platform which is running Kafka

stream

Post by RK Sing
inside. Has anyone used Kafkastreams inside Kafka connect ??
Is this pattern ok for Kafka-connect?
regards
dhurandar

[RK Sing]

Thank you that's helpful.

So that means resource management of streams needs to a separate concern
than that of Kafka connect

Post by Matthias J. Sax
Streams is not designed to be run inside Connect, and this won't work.
What you can do is, to import the data via connect into a "staging
topic" and then read this "staging topic" with a Kafka Streams
application and apply the transformations etc to write the data into the
actual target topics.
-Matthias

Post by RK Sing
Thank you Ryanne for the answer, My question is can I run Streams app
inside Connect, what I mean is since Streams is a library, can I build a
custom Source connector and use Streams in there . or using Streams in
transforms??
We want to containerize Kafka connect as in wrap it in a docker, and

since

Post by RK Sing
all the Kafka-connect configs are stored in Kafka itself, so the

container

Post by RK Sing
can be stateless, we can spin up containers as and when needed, we want
this approach to fetch data from microservices and golden gate oracle.

etc.

Post by RK Sing

Post by Ryanne Dolan
Dhurandar, definitely! Connect and Streams are both agnostic to how

their

Post by RK Sing

Post by Ryanne Dolan
workers are run. They aren't really platforms per se. You just need to

spin

Post by RK Sing

Post by Ryanne Dolan
up one or more workers and they do their thing. So a Streams app doesn't
run "inside" Connect, but you can certainly have Connect and Streams
workers talk to each other through Kafka topics.
Ryanne

Post by RK Sing
We have a requirement to do Single row transformations, basic joins,
deduping and routing from the source to the destination Kafka topics.
We want to use Kafka-connect as the platform which is running Kafka

stream

Post by RK Sing
inside. Has anyone used Kafkastreams inside Kafka connect ??
Is this pattern ok for Kafka-connect?
regards
dhurandar

[Guozhang Wang]

Hi RK,

You may find this blog post helpful for incorporating Streams with Connect:

https://yokota.blog/2018/04/23/embedding-kafka-connect-in-kafka-streams-ksql/

I've also cc'ed Robert, the author of this blog post if you are interested
in his code and want to learn more.


Guozhang

Post by RK Sing
Thank you that's helpful.
So that means resource management of streams needs to a separate concern
than that of Kafka connect

Post by Matthias J. Sax
Streams is not designed to be run inside Connect, and this won't work.
What you can do is, to import the data via connect into a "staging
topic" and then read this "staging topic" with a Kafka Streams
application and apply the transformations etc to write the data into the
actual target topics.
-Matthias

Post by RK Sing
Thank you Ryanne for the answer, My question is can I run Streams app
inside Connect, what I mean is since Streams is a library, can I build

a

Post by Matthias J. Sax

Post by RK Sing
custom Source connector and use Streams in there . or using Streams in
transforms??
We want to containerize Kafka connect as in wrap it in a docker, and

since

Post by RK Sing
all the Kafka-connect configs are stored in Kafka itself, so the

container

Post by RK Sing
can be stateless, we can spin up containers as and when needed, we want
this approach to fetch data from microservices and golden gate oracle.

etc.

Post by RK Sing

Post by Ryanne Dolan
Dhurandar, definitely! Connect and Streams are both agnostic to how

their

Post by RK Sing

Post by Ryanne Dolan
workers are run. They aren't really platforms per se. You just need to

spin

Post by RK Sing

Post by Ryanne Dolan
up one or more workers and they do their thing. So a Streams app

doesn't

Post by Matthias J. Sax

Post by RK Sing

Post by Ryanne Dolan
run "inside" Connect, but you can certainly have Connect and Streams
workers talk to each other through Kafka topics.
Ryanne

Post by RK Sing
We have a requirement to do Single row transformations, basic joins,
deduping and routing from the source to the destination Kafka topics.
We want to use Kafka-connect as the platform which is running Kafka

stream

Post by RK Sing
inside. Has anyone used Kafkastreams inside Kafka connect ??
Is this pattern ok for Kafka-connect?
regards
dhurandar