Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logstash to Security Lake pipeline #147

Closed
wants to merge 82 commits into from
Closed

Logstash to Security Lake pipeline #147

wants to merge 82 commits into from

Commits on Feb 5, 2024

  1. Adding Python script that receives a continuous json stream over stdi… 

    …n and outputs parquet to Security Lake
    @f-galland
    f-galland committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    e6784f3 View commit details
    Browse the repository at this point in the history

  2. Adding logstash pipeline for python script

    @f-galland
    f-galland committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    116b22b View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. encode_parquet() function fixed to handle lists of dictionaries

    @f-galland
    f-galland committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    288c40a View commit details
    Browse the repository at this point in the history

  2. Correct error in encode_parquet()

    @f-galland
    f-galland committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    6ac3c99 View commit details
    Browse the repository at this point in the history

  3. Avoid storing the block ending in the output buffer

    @f-galland
    f-galland committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    4ad01c2 View commit details
    Browse the repository at this point in the history

  4. Add comments on handling files and streams with pyarrow for future re… 

    …ference
    @f-galland
    f-galland committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    1638b17 View commit details
    Browse the repository at this point in the history

  5. Add s3 handling reference links

    @f-galland
    f-galland committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    17e5dfb View commit details
    Browse the repository at this point in the history

  6. Write parquet directly to bucket

    @f-galland
    f-galland committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    0b5adc9 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Added basics of map_to_ocsf() function

    @f-galland
    f-galland committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    10824ed View commit details
    Browse the repository at this point in the history

  2. Minor fixes

    @f-galland
    f-galland committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    c81239b View commit details
    Browse the repository at this point in the history

  3. Map alerts to OCSF as they are read

    @f-galland
    f-galland committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    210541d View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. Add script to convert Wazuh events to OCSF 

    Also adds a simple test script
    @AlexRuiz7
    AlexRuiz7 committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    5e3c0fa View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2024

  1. Add OCSF converter + Parquet encoder + test scripts

    @AlexRuiz7
    AlexRuiz7 committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    0995134 View commit details
    Browse the repository at this point in the history

  2. Update .gitignore

    @AlexRuiz7
    AlexRuiz7 committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    d82ed21 View commit details
    Browse the repository at this point in the history

  3. Include the contents of the alert under unmapped

    @f-galland
    f-galland committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    17dac0c View commit details
    Browse the repository at this point in the history

  4. Add support for different OCSF schema versions

    @f-galland
    f-galland committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    a4f74db View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. Use custom ocsf module to map alerts

    @f-galland
    f-galland committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    34f295b View commit details
    Browse the repository at this point in the history

  2. Modify script to use converter class

    @f-galland
    f-galland committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    fd63e9e View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. Code polish and fix errors

    @f-galland
    f-galland committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    d32e06d View commit details
    Browse the repository at this point in the history

  2. Remove unnecessary type declaration from debug flag

    @f-galland
    f-galland committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    ab56e89 View commit details
    Browse the repository at this point in the history

  3. Improved parquet encoding

    @f-galland
    f-galland committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    7fc49e7 View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2024

  1. Initial commit for test env's docker-compose.yml

    @f-galland
    f-galland committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    67b785f View commit details
    Browse the repository at this point in the history

  2. Merge branch '4.9.0' into logstash-pipe-output 

    Merging to incorporate event generator script into logstash-pipe-output branch
    @f-galland
    f-galland committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    1d8efe3 View commit details
    Browse the repository at this point in the history

  3. Remove sudo references from docker-compose.yml

    @f-galland
    f-galland committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    0bf697d View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. Adding Python script that receives a continuous json stream over stdi… 

    …n and outputs parquet to Security Lake
    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    159adcb View commit details
    Browse the repository at this point in the history

  2. Adding logstash pipeline for python script

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    6e17aae View commit details
    Browse the repository at this point in the history

  3. encode_parquet() function fixed to handle lists of dictionaries

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    a05c23c View commit details
    Browse the repository at this point in the history

  4. Correct error in encode_parquet()

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    e04f0d5 View commit details
    Browse the repository at this point in the history

  5. Avoid storing the block ending in the output buffer

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    93935fc View commit details
    Browse the repository at this point in the history

  6. Add comments on handling files and streams with pyarrow for future re… 

    …ference
    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    1db384c View commit details
    Browse the repository at this point in the history

  7. Add s3 handling reference links

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    c60045f View commit details
    Browse the repository at this point in the history

  8. Write parquet directly to bucket

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    8949097 View commit details
    Browse the repository at this point in the history

  9. Added basics of map_to_ocsf() function

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    eb7ace3 View commit details
    Browse the repository at this point in the history

  10. Minor fixes

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    3d7b8ff View commit details
    Browse the repository at this point in the history

  11. Map alerts to OCSF as they are read

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    545f855 View commit details
    Browse the repository at this point in the history

  12. Add script to convert Wazuh events to OCSF 

    Also adds a simple test script
    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    f753b12 View commit details
    Browse the repository at this point in the history

  13. Add OCSF converter + Parquet encoder + test scripts

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    dcc119e View commit details
    Browse the repository at this point in the history

  14. Update .gitignore

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    5c5ff24 View commit details
    Browse the repository at this point in the history

  15. Include the contents of the alert under unmapped

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    a39ef90 View commit details
    Browse the repository at this point in the history

  16. Add support for different OCSF schema versions

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    97725bc View commit details
    Browse the repository at this point in the history

  17. Use custom ocsf module to map alerts

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    e313572 View commit details
    Browse the repository at this point in the history

  18. Modify script to use converter class

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    4896d15 View commit details
    Browse the repository at this point in the history

  19. Code polish and fix errors

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    7fd25d1 View commit details
    Browse the repository at this point in the history

  20. Remove unnecessary type declaration from debug flag

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    e06203c View commit details
    Browse the repository at this point in the history

  21. Improved parquet encoding

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    6826e12 View commit details
    Browse the repository at this point in the history

  22. Initial commit for test env's docker-compose.yml

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    9cfc247 View commit details
    Browse the repository at this point in the history

  23. Remove sudo references from docker-compose.yml

    @f-galland @AlexRuiz7
    f-galland authored and AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    324d1f5 View commit details
    Browse the repository at this point in the history

  24. Add operational Python module to transform events to OCSF

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    cb5ac73 View commit details
    Browse the repository at this point in the history

  25. Create minimal Docker environment to test and develop the integration.

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    05ae2d1 View commit details
    Browse the repository at this point in the history

  26. Fix events-generator's Inventory starvation

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    17f47ca View commit details
    Browse the repository at this point in the history

  27. Remove files present in #147

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    204948f View commit details
    Browse the repository at this point in the history

  28. Cleanup

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    5fcc9a3 View commit details
    Browse the repository at this point in the history

  29. Add FQDN hostnames to services for certificates creation

    @AlexRuiz7
    AlexRuiz7 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    a246410 View commit details
    Browse the repository at this point in the history

  30. Add certificate generator service

    @f-galland
    f-galland committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    bf3f1ff View commit details
    Browse the repository at this point in the history

  31. Add certificate config to docker compose file

    @f-galland
    f-galland committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    4279b6e View commit details
    Browse the repository at this point in the history

  32. Use secrets for certificates

    @f-galland
    f-galland committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    65b3b10 View commit details
    Browse the repository at this point in the history

  33. Disable permission handling inside cert's generator entrypoint.sh

    @f-galland
    f-galland committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    c0d6d2c View commit details
    Browse the repository at this point in the history

  34. Back to using a bind mount for certs

    @f-galland
    f-galland committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    017a908 View commit details
    Browse the repository at this point in the history

  35. Have entrypoint.sh generate certs with 1000:1000 ownership

    @f-galland
    f-galland committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    2a60c41 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. Correct certificate permissions and bind mounting

    @f-galland
    f-galland committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    91da2c2 View commit details
    Browse the repository at this point in the history

  2. Add security initialization variable to compose file

    @f-galland
    f-galland committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    55f0726 View commit details
    Browse the repository at this point in the history

  3. Add S3 Ninja (Mock)

    @AlexRuiz7
    AlexRuiz7 committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    454d6fd View commit details
    Browse the repository at this point in the history

  4. Fix permissions on certs generator entrypoint

    @f-galland
    f-galland committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    a9f9572 View commit details
    Browse the repository at this point in the history

  5. Add cert generator config file

    @f-galland
    f-galland committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    c854dc5 View commit details
    Browse the repository at this point in the history

  6. Remove old cert generator dir

    @f-galland
    f-galland committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    422cf1c View commit details
    Browse the repository at this point in the history

  7. Set indexer hostname right in pipeline file

    @f-galland
    f-galland committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    21f89f8 View commit details
    Browse the repository at this point in the history

  8. Merging changes from 160-security-lake-logstash-certificates

    @f-galland
    f-galland committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    826f06b View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Change timestamp field in pipeline

    @f-galland
    f-galland committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    6938a54 View commit details
    Browse the repository at this point in the history

  2. Clean up unneeded files

    @f-galland
    f-galland committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    118b477 View commit details
    Browse the repository at this point in the history

  3. Made script available as volume and changed permissions for quick tes… 

    …ting
    @f-galland
    f-galland committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    17b71f0 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Temporary fix on ocsf class for testing purposes

    @f-galland
    f-galland committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    b9b21a8 View commit details
    Browse the repository at this point in the history

  2. Add function to handle local s3 buckets

    @f-galland
    f-galland committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    81ecb7c View commit details
    Browse the repository at this point in the history

  3. Allow to set the bucket name from the cli

    @f-galland
    f-galland committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    f4a7336 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Renamed script to match convention

    @f-galland
    f-galland committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    811f940 View commit details
    Browse the repository at this point in the history

  2. Move argparse into its own function

    @f-galland
    f-galland committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    c421235 View commit details
    Browse the repository at this point in the history

  3. Add volumes to mount script and dependencies in compose file

    @f-galland
    f-galland committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    3713fb1 View commit details
    Browse the repository at this point in the history

  4. Delete integrations/ocsf-mapping.json 

    Signed-off-by: Federico Gustavo Galland <[email protected]>
    @f-galland
    f-galland authored Feb 28, 2024
    Configuration menu
    Copy the full SHA
    f6329f4 View commit details
    Browse the repository at this point in the history

  5. Delete integrations/amazon-security-lake/logstash/pipe-output.conf 

    Signed-off-by: Federico Gustavo Galland <[email protected]>
    @f-galland
    f-galland authored Feb 28, 2024
    Configuration menu
    Copy the full SHA
    5cb2c38 View commit details
    Browse the repository at this point in the history

  6. merging s3 mock changes into pipeline development branch

    @f-galland
    f-galland committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    259622b View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Merge branch '4.9.0' of github.com:wazuh/wazuh-indexer into logstash-… 

    …pipe-output
    @AlexRuiz7
    AlexRuiz7 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    25ba5a0 View commit details
    Browse the repository at this point in the history

  2. Remove old files

    @AlexRuiz7
    AlexRuiz7 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    9950369 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. Merge branch '4.9.0-2.11.1' into logstash-pipe-output

    @AlexRuiz7
    AlexRuiz7 authored Mar 12, 2024
    Configuration menu
    Copy the full SHA
    b75134e View commit details
    Browse the repository at this point in the history