Skip to content

Commit

Permalink
Add s3 handling reference links
Browse files Browse the repository at this point in the history
  • Loading branch information
f-galland committed Feb 6, 2024
1 parent 1638b17 commit 17e5dfb
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion integrations/stdin_to_securitylake.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,19 @@
import time
import json
from datetime import datetime
from pyarrow import parquet, Table
from pyarrow import parquet, Table, fs

block_ending = { "block_ending": True }

s3 = fs.S3FileSystem(region='eu-west-3')

def map_to_ocsf():
## Code that translates fields to OCSF

def encode_parquet(list):
### We can write directly to S3 from pyarrow:
### https://arrow.apache.org/docs/python/filesystems.html#s3
### https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html#pyarrow.fs.S3FileSystem.open_output_stream
###
### Credentials can be stored in /root/.aws/credentials
### https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/credentials.html
Expand Down

0 comments on commit 17e5dfb

Please sign in to comment.