thoughts on how to distribute a kafka_topic module?

[josephholsten]

I've got a module for managing kafka topics, seems like it would pair nicely with your role. It still need to update partition & replication count, plus support removing topics entirely. Here's the rough version:

#!/usr/bin/python
# -*- coding: utf-8 -*-

class KafkaTopic(object):
    def __init__(self, **kwargs):

        # Initial var for storing current defaults value
        self.current_state = None

        # Just set all given parameters
        for key, val in kwargs.iteritems():
            setattr(self, key, val)

        self.executable = self.kafka_dir + '/bin/kafka-topics.sh'

    def read(self):
        args = [
            self.executable,
            "--describe",
            "--zookeeper",
            self.zk_nodes,
            "--topic",
            self.name
        ]
        rc, out, err = self.module.run_command(args=args)
        if rc != 0:
            raise Exception("An error occurred while describing topic. args: " + " ".join(args) + " out: " + out)
        if len(out) == 0:
            self.current_state = "absent"
        else:
            self.current_state = "present"


    def create(self):
        args = [
            self.executable,
            "--create",
            "--zookeeper",
            self.zk_nodes,
            "--replication-factor",
            str(self.replication_factor),
            "--partitions",
            str(self.partitions),
            "--topic",
            self.name,
        ]
        rc, out, err = self.module.run_command(args=args)
        if rc != 0:
            raise Exception(
                "An error occurred while describing topic. args: " + str(args) + " out: " + out + " err: " + err)

    # returns whether state was changed
    def run(self):
        self.read()

        if self.current_state == self.state:
            return False

        return True


def main():
    module = AnsibleModule(
        argument_spec=dict(
            name=dict(
                required=True,
            ),
            zk_nodes=dict(
                required=True,
            ),
            replication_factor=dict(
                default=2,
            ),
            partitions=dict(
                default=6,
            ),
            state=dict(
                default="present",
            ),
            kafka_dir=dict(
                default="/opt/kafka",
            ),
        ),
        supports_check_mode=True,
    )

    name = module.params['name']
    replication_factor = module.params['replication_factor']
    partitions = module.params['partitions']
    zk_nodes = module.params['zk_nodes']
    kafka_dir = module.params['kafka_dir']
    state = module.params['state']

    try:
        topic = KafkaTopic(
            module=module,
            name=name,
            replication_factor=replication_factor,
            partitions=partitions,
            zk_nodes=zk_nodes,
            kafka_dir=kafka_dir,
            state=state,
        )
        changed = topic.run()
        if not module.check_mode and changed:
            topic.create()
        module.exit_json(changed=changed)
    except Exception as e:
        module.fail_json(msg=e.message)


from ansible.module_utils.basic import *

main()

[tkuhlman]

Cool, I make topics right now in another role(https://github.com/hpcloud-mon/ansible-monasca-schema/blob/master/tasks/kafka_topics.yaml) and am always annoyed on how it is a hack with ugly output. I keep meaning to get around to a module, I'm glad to see your work! I think it makes sense to be paired with this kafka role, when you think it is ready just make a pull request and drop it into the library dir and update the readme, etc. I'll then convert my topic creation over to it and do some testing.

[EddyP23]

No progress on this since?

I am looking for an ansible Kafka Topic Module, I assume this was some sort of draft for it, has this developed into something I could reuse?

[teebee]

@EddyP23 Did you find any suitable Kafka topic module in the meantime? Can you recommend any?