public void increment(String label, String topic) { + Stats.incr(label); + }
import io.prometheus.client.Counter; public static final Counter requests = Counter.build() .name("requests_total").help("Total requests.").register();
public void increment(String label, String topic) { requests.inc(); // Your code here. } }
The component is pluggable, you can configure ingestion of metrics to any monitoring system by implementing MetricCollector interface. By default metrics are sent to Ostrich, statistic library bundled into secor.
Just inject prometheus simple client into this module and you could get prometheus working for secor.
Designing a High Level Consumer
By far the most important thing you need to know to make SECOR operate with Kafkaf
In merced, we used the low-level simple consumer and wrote our own work dispatcher to get precise control.
difference between merced and secor
Kafka consumer offset management protocol to keep track of what’s been uploaded to S3
consumers keep track of what's written and where it left off by looking at kafka consumer offsets rather than checking S3 since S3 is an eventually consistent system.
Data lost or corrupted at this stage isn’t recoverable so the greatest design objective for Secor is data integrity.
data loss in S3 is being mitigated.