Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transition away from Java serialization for storing state on disk or in Zookeeper #625

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Transition away from Java serialization for storing state on disk or in Zookeeper #625

wants to merge 13 commits into from

Commits on Jul 21, 2013

  1. Change StormTopology state serialization from Java to Thrift 

    config and nimbus currently use the core Java serialization to store
StormTopology instances. This commit will change this to use Thrift
serialization instead. StormTopology is a Thrift struct so this
basically involves finding all the places we call `Utils/serialize` and
`Utils/deserialize` and replace them with a call to a method that
serializes with Thrift instead.
    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    59d4187 View commit details
    Browse the repository at this point in the history

  2. Move serialization of Storm conf from Java default to Clojure default 

    config.clj and nimbus serialize Storm configuration using the default
Java implementation. This commit will phase out Java default
serialization in favor of Clojure default serialization. There are
obviously many reasons to phase out Java serialization, but the main
rationale is that Java serialization will complain about version
mismatch even if the change is semantically backward compatible.
    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    46ca749 View commit details
    Browse the repository at this point in the history

  3. Fix serialization concurrency bug 

    TSerializer is not threadsafe. In `Utils` we instantiate a static final
Tserializer, but this can (and will) cause odd bugs if we start calling
`serialize()` in different threads. Thus, every time we call
`Utils/serializeTopology`, we create a new TSerializer.

Another way to do this would be to lock it, which performance may or may
not merit.
    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    44cfa94 View commit details
    Browse the repository at this point in the history

  4. Make serialization funcitons more generic 

    Serialization of configuration is handled in config, but it is not
different from a generic method for Clojure form serialization. This
commit will move this method to utils so that we can use it for other
things, like serialization in cluster.
    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    50070c7 View commit details
    Browse the repository at this point in the history

  5. Transition cluster state serialization to Clojure form serialization 

    cluster.clj uses the stock Java serialization implementation. There are
obviously many reason to not use standard Java serialization, but our
main motivation is that Java will complain about serialized state
when the versions don't match even if they're semantically backwards
compatible.
    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    9492da3 View commit details
    Browse the repository at this point in the history

  6. Source LocalState serialization logic to state serialization interface 

    Supervisor currently just uses stock jvm serialization to communicate
LocalState. This is undesirable for many reasons, so this commit
will introduce a serialization interface which makes code cleaner,
letting us specify which type of serializer to use without populating
Supervisor with unnecessary boxing/unboxing behavior, or LocalState
with too much knowledge about what's happening in the Supervisor.

Also this commit will introduce a basic implementation sketch for a
serializer for LocalState (though it will just use the jvm serialization
at this point).
    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    00f0274 View commit details
    Browse the repository at this point in the history

  7. Source LocalState constants to Constants.java 

    To build the serialization interface in Java, we need to put these
constants in a Java file. This commit will put them in Constants.java
    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    9d673a8 View commit details
    Browse the repository at this point in the history

  8. Add sketch of LocalStateSerializer

    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    55682d8 View commit details
    Browse the repository at this point in the history

  9. Add clojure-based serialization implementation for LocalState

    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    1aba69c View commit details
    Browse the repository at this point in the history

  10. Update Supervisor to use clojure serialization

    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    8a83664 View commit details
    Browse the repository at this point in the history

  11. Require LocalState to take a serializer in constructor

    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    a70c170 View commit details
    Browse the repository at this point in the history

  12. Remove Java implementation of LocalStateSerializer

    @hausdorff
    hausdorff authored and Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    6366e75 View commit details
    Browse the repository at this point in the history

  13. Fixup for rebase of hausdorff/ser_dev onto nathanmarz/master

    Philip (flip) Kromer committed Jul 21, 2013
    Configuration menu
    Copy the full SHA
    b99a90b View commit details
    Browse the repository at this point in the history