24 Matching Annotations
  1. Nov 2023
    1. Economies that are heavily reliant on oil and gas revenues face some stark choices and pressures in energy transitions.
      • for: stats - oil and gas - steep drop in revenues of fossil fuel producer economies

      • stats: oil and gas - steep drop in revenues of fossil fuel reliant economies

        • per capita net income from oil and natural gas among producer economies will be 60% lower in 2030 in a 1.5 °C scenario.relative to revenues between 2010 and 2022.
      • question

        • many producer economies are not diversifying into clean energy fast enough to compensate for these steep revenue drops
  2. Oct 2023
  3. Aug 2023
    1. Kafka Broker

      producer key properties => linger time -> linger.ms batch size -> batch.size

      these determine the throughput and latency of kafkaproducers

  4. Oct 2021
    1. Jurgis Pranas Didžiulis Valencia (born 5 November 1979 in Bogotá , Colombia ) is a performer, song and music author, producer, public figure and activist.
  5. Aug 2021
  6. Apr 2021
  7. Mar 2021
  8. Oct 2020
    1. Richard Curtis

      Richard Curtis is a screenwriter, producer and film director. He produced Love Actually, Bridget Jone's Diary, Notting Hill and many more rom-coms

  9. May 2020
  10. Jun 2017
    1. You measure the throughout that you can achieve on a single partition for production (call it p) and consumption (call it c). Let’s say your target throughput is t.

      t = throughput (QPS) p = single partition for production c = consumption

    1. By electing a new leader as soon as possible messages may be dropped but we will minimized downtime as any new machine can be leader.

      two scenarios to get the leader back: 1.) Wait to bring the master back online. 2.) Or elect the first node that comes back up. But in this scenario if that replica partition was a bit behind the master then the time from when this replica went down to when the master went down. All that data is Lost.

      SO there is a trade off between availability and consistency. (Durability)

    2. keep in mind that these guarantees hold as long as you are producing to one partition and consuming from one partition.

      This is very important a 1-to-1 mapping between writer and reader with partition. If you have more producers per partition or more consumers per partition your consistency is going to go haywire

    1. incidents are an unavoidable reality of working with distributed systems, no matter how reliable. A prompt alerting solution should be an integral part of the design,

      see how it can hook into the current logging mechanism

  11. May 2017
    1. With Flume & FlumeNG, and a File channel, if you loose a broker node you will lose access to those events until you recover that disk.

      In flume you loose events if the disk is down. This is very bad for our usecase.

    1. The Kafka cluster retains all published records—whether or not they have been consumed—using a configurable retention period. For example, if the retention policy is set to two days, then for the two days after a record is published, it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so storing data for a long time is not a problem.

      irrespective of the fact that the consumer has consumed the message that message is kept in kafka for the entire retention policy duration.

      You can have two or more consumer groups: 1 -> real time 2 -> back up consumer group

    2. Kafka for Stream Processing

      Could be something we can consider for directing data from a raw log to a tenant based topic.

    3. replication factor N, we will tolerate up to N-1 server failures without losing any records

      Replication Factor means number of nodes/brokers which could go down before we start losing data.

      So if you have a replication factor of 6 for a 11 node cluster, then you will be fault tolerant till 5 nodes go down. After that point you are going to loose data for a particular partition.

    4. Messages sent by a producer to a particular topic partition will be appended in the order they are sent. That is, if a record M1 is sent by the same producer as a record M2, and M1 is sent first, then M1 will have a lower offset than M2 and appear earlier in the log.

      ordering is guaranteed.

    1. The offset the ordering of messages as an immutable sequence. Kafka maintains this message ordering for you.

      Kafka maintains the ordering for you...

    1. The producer is responsible for choosing which record to assign to which partition within the topic.

      Producer can publish to a specific topics

    2. individual partition must fit on the servers that host it

      Each Partition is bounded by the server that hosts that partition.

    3. the only metadata retained on a per-consumer basis is the offset or position of that consumer in the log. This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads records, but, in fact, since the position is controlled by the consumer it can consume records in any order it likes.

      partition offset maintained by kafka. Offset number is maintained so that if the consumer goes down nothing breaks.