jackdaw

https://github.com/FundingCircle/jackdaw
2019-11-05T09:31:43.012300Z

@dcj where would the metadata come from?

2019-11-05T09:31:59.012900Z

you would write it explicitly in Kafka?

2019-11-05T16:03:00.013900Z

I might have a related question, are the ConsumerWrappers accessible from jackdaw? I want to log committed offsets

2019-11-05T16:08:38.017100Z

Interesting question @jgerman. Do you mean that you have some app implemented using the kafka streams DSL, and that you basically want to track the progress of the underlying consumers by logging the committed offsets as they are committed? Not sure how I'd do that. I think again it might be something for which you'd need to bust out the processor api.

2019-11-05T16:10:28.018400Z

yeah, I've only been looking at Streams and Jackdaw for a day or so, but in the monitoring section of the Manning book on streams there's an example of implementing interceptors (in java) to get that sort of information

2019-11-05T16:10:34.018700Z

public class StockTransactionConsumerInterceptor implements
 ConsumerInterceptor<Object, Object> {

    // some details left out for clarity
    private static final Logger LOG =
 LoggerFactory.getLogger(StockTransactionConsumerInterceptor.class);

    public StockTransactionConsumerInterceptor() {
        <http://LOG.info|LOG.info>("Built StockTransactionConsumerInterceptor");
    }

    @Override
    public ConsumerRecords&lt;Object, Object&gt;
 (ConsumerRecords&lt;Object, Object&gt; consumerRecords) {
        <http://LOG.info|LOG.info>("Intercepted ConsumerRecords {}",
                 buildMessage(consumerRecords.iterator()));              1
        return consumerRecords;
    }

    @Override
    public void onCommit(Map&lt;TopicPartition, OffsetAndMetadata&gt; map) {
         <http://LOG.info|LOG.info>("Commit information {}",  map);                        2
    }

2019-11-05T16:11:05.019400Z

if I'm going to jump into the ProcessorAPI with jackdaw I'm off in custom interop land correct?

2019-11-05T16:12:32.020Z

my intention here is to be able to report consumer lag to datadog, we've had trouble getting jmx metrics out of kafka

2019-11-05T16:54:42.022Z

Ah yeah interceptors is probably a good tool to use for that purpose. You can also do them in clojure. You just need to make sure they are AOT compiled so that kafka. Hopefully jackdaw neither helps nor hinders in this area. It should just get out of your way

2019-11-05T19:17:27.022400Z

cool, I'll poke around and see if I can figure out how to handle it, thanks!