Stream Metrics - Memory Analysis

[Patrik Kleindl]

Hello

During the analysis of JVM memory two possible issues were shown which I
would like to bring to your attention:
1) Duplicate strings
Top findings:
string_content="stream-processor-node-metrics" count="534,277"
string_content="processor-node-id" count="148,437"
string_content="stream-rocksdb-state-metrics" count="41,832"
string_content="punctuate-latency-avg" count="29,681"

"stream-processor-node-metrics" seems to be used in Sensors.java as a
literal and not interned.

2) The HashMap parentSensors
from org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl
was reported multiple times as suspicious for potentially keeping alive a
lot of objects. In our case the reported size was 40-50MB each.
I haven't looked too deep in the code but noticed that the class
Sensor.java which is used as a key in the HashMap does not implement equals
or hashCode method. Not sure this is a problem though.

Maybe someone can shed some light on this

best regards

Patrik

[Guozhang Wang]

Hello Patrik,

Could you file a JIRA for your findings? Also what Kafka versions are you
using (could you add that to the ticket as well)?

Could you provide some more elaborations on what you did the JVM analysis,
so that I can try to re-produce the observations.


Guozhang

Post by Patrik Kleindl
Hello
During the analysis of JVM memory two possible issues were shown which I
1) Duplicate strings
string_content="stream-processor-node-metrics" count="534,277"
string_content="processor-node-id" count="148,437"
string_content="stream-rocksdb-state-metrics" count="41,832"
string_content="punctuate-latency-avg" count="29,681"
"stream-processor-node-metrics" seems to be used in Sensors.java as a
literal and not interned.
2) The HashMap parentSensors
from
org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl
was reported multiple times as suspicious for potentially keeping alive a
lot of objects. In our case the reported size was 40-50MB each.
I haven't looked too deep in the code but noticed that the class
Sensor.java which is used as a key in the HashMap does not implement equals
or hashCode method. Not sure this is a problem though.
Maybe someone can shed some light on this
best regards
Patrik

[Patrik Kleindl]

Done.
https://issues.apache.org/jira/browse/KAFKA-7660
br, Patrik

Post by Guozhang Wang
Hello Patrik,
Could you file a JIRA for your findings? Also what Kafka versions are you
using (could you add that to the ticket as well)?
Could you provide some more elaborations on what you did the JVM analysis,
so that I can try to re-produce the observations.
Guozhang

Post by Patrik Kleindl
Hello
During the analysis of JVM memory two possible issues were shown which I
1) Duplicate strings
string_content="stream-processor-node-metrics" count="534,277"
string_content="processor-node-id" count="148,437"
string_content="stream-rocksdb-state-metrics" count="41,832"
string_content="punctuate-latency-avg" count="29,681"
"stream-processor-node-metrics" seems to be used in Sensors.java as a
literal and not interned.
2) The HashMap parentSensors
from

org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl

Post by Patrik Kleindl
was reported multiple times as suspicious for potentially keeping alive a
lot of objects. In our case the reported size was 40-50MB each.
I haven't looked too deep in the code but noticed that the class
Sensor.java which is used as a key in the HashMap does not implement

equals

Post by Patrik Kleindl
or hashCode method. Not sure this is a problem though.
Maybe someone can shed some light on this
best regards
Patrik

--
-- Guozhang

[Guozhang Wang]

Thank you!

Post by Patrik Kleindl
Done.
https://issues.apache.org/jira/browse/KAFKA-7660
br, Patrik

Post by Guozhang Wang
Hello Patrik,
Could you file a JIRA for your findings? Also what Kafka versions are you
using (could you add that to the ticket as well)?
Could you provide some more elaborations on what you did the JVM

analysis,

Post by Guozhang Wang
so that I can try to re-produce the observations.
Guozhang

Post by Patrik Kleindl
Hello
During the analysis of JVM memory two possible issues were shown which

I

Post by Guozhang Wang

Post by Patrik Kleindl
1) Duplicate strings
string_content="stream-processor-node-metrics" count="534,277"
string_content="processor-node-id" count="148,437"
string_content="stream-rocksdb-state-metrics" count="41,832"
string_content="punctuate-latency-avg" count="29,681"
"stream-processor-node-metrics" seems to be used in Sensors.java as a
literal and not interned.
2) The HashMap parentSensors
from

org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl

Post by Guozhang Wang

Post by Patrik Kleindl
was reported multiple times as suspicious for potentially keeping

alive a

Post by Guozhang Wang

Post by Patrik Kleindl
lot of objects. In our case the reported size was 40-50MB each.
I haven't looked too deep in the code but noticed that the class
Sensor.java which is used as a key in the HashMap does not implement

equals

Post by Patrik Kleindl
or hashCode method. Not sure this is a problem though.
Maybe someone can shed some light on this
best regards
Patrik

--
-- Guozhang