Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added 'Kafka tiny client' #551

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

added 'Kafka tiny client' #551

wants to merge 1 commit into from

Conversation

sqlly
Copy link

@sqlly sqlly commented Apr 14, 2020

I've created a most lightweight implementation of Kafka binary protocol for Espruino. Kafka is now almost de-facto standard for stream messaging and Espruino definitely lack such implementation.

@gfwilliams
Copy link
Member

This is great, thanks! Does the crc32 code you have there actually work?

On new Espruino builds (cutting edge, and 2v05 when released) there's actually a E.CRC32 function which would probably help out: http://www.espruino.com/Reference#l_E_CRC32

If you need it I also have an implementation that doesn't need the table...

@sqlly
Copy link
Author

sqlly commented Apr 16, 2020

Probably my Espruino build was not so fresh (actually I've created that code 3 months ago) and at that time I've found that build-in CRC32 value does not equal to what Kafka expects. After a deep dive to that I've found working solution.
As for implementation that does not need a table - I've tried one with dynamic CRC table generation - it was extremly slow. If your version is really fast, I will be glad to remove CRC table, it is very resource expensive.

@gfwilliams
Copy link
Member

This should do it hopefully:

function CRC32(data) {
  var crc = 0xFFFFFFFF;
  data.forEach(function(d) {
    crc^=d;
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
    crc=(crc>>>1)^(0xEDB88320&-(crc&1));
  });
  return ~crc;
}
print(CRC32([2,4,65,7,5,4,3,5,6]))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants