Best way to persist and start from a Offset

[marcofranssen]

I am currently struggling to find the best way to persist the current offset and how I can now if I should keep an offset for various partitions.

Furthermore I have the same question when starting the consumption of messages. How do I know which Partition I should start.

My publisher uses no key, so that means it is doing RoundRobin if I am not mistaking.

When I create a consumer I can provide the OffsetPosition, but I'm not sure how I can know what is the correct Offset and partition to use.

This is basically my consumer code atm.

        public Task Run(CancellationToken token)
        {
            return Task.Run(() =>
            {
                var offset = ReadPersistedOffset();
                using (var consumer = CreateConsumer(_topic, offset))
                {
                    while (!token.IsCancellationRequested)
                    {
                        foreach (var message in consumer.Consume())
                        {
                            var eventJson = GetJson(message.Value);
                            var @event = GetPayload(eventJson);
                            DispatchToSubscriber(@event);
                            PersistOffset(message.Meta);
                        }
                    }
                }
            });
        }

So is it enough to just perist the message meta propery? Offset and PartitionId, should I keep a dictionary of partitions and offsets?

How do I create the consumer based on the offset? Should I start consuming form multiple partitions? Can this actually happen?

[ashishbhatia22]

Hi,
Did you find a solution to this problem? I see that offset and partition are managed by zookeeper. If a consumer wants to replay all messages for a given date, is there a recommended solution to handle that?