Logstash to Security Lake pipeline #147
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n and outputs parquet to Security Lake
Also adds a simple test script
Merging to incorporate event generator script into logstash-pipe-output branch
…n and outputs parquet to Security Lake
Signed-off-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Mar 4, 2024
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Mar 4, 2024
* Add events generator tool for `wazuh-alerts` (#152) * Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> * Add `wazuh.manager.name` to VD mappings (#158) * Create compatibility_request.md (#163) Signed-off-by: Álex Ruiz <[email protected]> * Add Python module to accomplish OCSF compliant events (#159) * Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Mar 8, 2024
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
8 tasks
|
Superseded by #186 |
AlexRuiz7
added a commit
that referenced
this pull request
Apr 9, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line
AlexRuiz7
added a commit
that referenced
this pull request
Apr 24, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Apr 24, 2024
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Apr 24, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line
AlexRuiz7
added a commit
that referenced
this pull request
Apr 24, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Jun 28, 2024
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Jun 28, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line
AlexRuiz7
added a commit
that referenced
this pull request
Jun 28, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Aug 20, 2024
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Aug 20, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line
AlexRuiz7
added a commit
that referenced
this pull request
Aug 20, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Sep 9, 2024
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Sep 9, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line
AlexRuiz7
added a commit
that referenced
this pull request
Sep 9, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Sep 9, 2024
* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7
added a commit
that referenced
this pull request
Sep 9, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line
AlexRuiz7
added a commit
that referenced
this pull request
Sep 9, 2024
* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a logstash pipeline configuration file that takes data from Wazuh Indices and pushes them to a Python script through
stdinby means of the pipe output plugin.Issues Resolved
Resolves #144
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.