Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Elasticsearch v8 support #93

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Elasticsearch v8 support #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f86b127
Fix code to elasticsearch 8.x version
thoscilo Apr 6, 2023
406ddd6
Remove Python 2.7 support
thoscilo Apr 6, 2023
81fefba
version 1.1.0
thoscilo Apr 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 9 additions & 16 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,6 @@ Install using pip::

pip install CMRESHandler

Requirements Python 2
=====================
This library requires the following dependencies
- elasticsearch
- requests
- enum


Requirements Python 3
=====================
This library requires the following dependencies
Expand All @@ -42,20 +34,20 @@ This library requires the following dependencies

Additional requirements for Kerberos support
============================================
Additionally, the package support optionally kerberos authentication by adding the following dependecy
Additionally, the package support optionally kerberos authentication by adding the following dependency
- requests-kerberos

Additional requirements for AWS IAM user authentication (request signing)
=========================================================================
Additionally, the package support optionally AWS IAM user authentication by adding the following dependecy
Additionally, the package support optionally AWS IAM user authentication by adding the following dependency
- requests-aws4auth

Using the handler in your program
==================================
To initialise and create the handler, just add the handler to your logger as follow ::

from cmreslogging.handlers import CMRESHandler
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200}],
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200, 'scheme': 'http'}],
auth_type=CMRESHandler.AuthType.NO_AUTH,
es_index_name="my_python_index")
log = logging.getLogger("PythonTest")
Expand All @@ -65,7 +57,7 @@ To initialise and create the handler, just add the handler to your logger as fol
You can add fields upon initialisation, providing more data of the execution context ::

from cmreslogging.handlers import CMRESHandler
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200}],
handler = CMRESHandler(hosts=[{'host': 'localhost', 'port': 9200, 'scheme': 'http'}],
auth_type=CMRESHandler.AuthType.NO_AUTH,
es_index_name="my_python_index",
es_additional_fields={'App': 'MyAppName', 'Environment': 'Dev'})
Expand Down Expand Up @@ -95,17 +87,18 @@ Kibana on top of elasticsearch
Initialisation parameters
=========================
The constructors takes the following parameters:
- hosts: The list of hosts that elasticsearch clients will connect, multiple hosts are allowed, for example ::
- hosts: The list of hosts that elasticsearch clients will connect, multiple hosts are allowed.
Use ```'scheme'``` to determinate if use SSL (`use_ssl` is deprecated). To use SSL set ```'scheme': 'https'```, or if you don't need SSL Sset ```'scheme': 'http'```.
for example::

[{'host':'host1','port':9200}, {'host':'host2','port':9200}]
[{'host':'host1','port':9200, 'scheme': 'https'}, {'host':'host2','port':9200, 'scheme': 'http'}]


- auth_type: The authentication currently support CMRESHandler.AuthType = NO_AUTH, BASIC_AUTH, KERBEROS_AUTH
- auth_details: When CMRESHandler.AuthType.BASIC_AUTH is used this argument must contain a tuple of string with the user and password that will be used to authenticate against the Elasticsearch servers, for example ('User','Password')
- aws_access_key: When ``CMRESHandler.AuthType.AWS_SIGNED_AUTH`` is used this argument must contain the AWS key id of the the AWS IAM user
- aws_secret_key: When ``CMRESHandler.AuthType.AWS_SIGNED_AUTH`` is used this argument must contain the AWS secret key of the the AWS IAM user
- aws_region: When ``CMRESHandler.AuthType.AWS_SIGNED_AUTH`` is used this argument must contain the AWS region of the the AWS Elasticsearch servers, for example ``'us-east'``
- use_ssl: A boolean that defines if the communications should use SSL encrypted communication
- verify_ssl: A boolean that defines if the SSL certificates are validated or not
- buffer_size: An int, Once this size is reached on the internal buffer results are flushed into ES
- flush_frequency_in_sec: A float representing how often and when the buffer will be flushed
Expand Down Expand Up @@ -139,7 +132,7 @@ they can be plotted on Kibana, or the SQL statements that Django executed. ::
'elasticsearch': {
'level': 'DEBUG',
'class': 'cmreslogging.handlers.CMRESHandler',
'hosts': [{'host': 'localhost', 'port': 9200}],
'hosts': [{'host': 'localhost', 'port': 9200, 'scheme': 'http'}],
'es_index_name': 'my_python_app',
'es_additional_fields': {'App': 'Test', 'Environment': 'Dev'},
'auth_type': CMRESHandler.AuthType.NO_AUTH,
Expand Down
31 changes: 11 additions & 20 deletions cmreslogging/handlers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from threading import Timer, Lock
from enum import Enum
from elasticsearch import helpers as eshelpers
from elasticsearch import Elasticsearch, RequestsHttpConnection
from elasticsearch import Elasticsearch

try:
from requests_kerberos import HTTPKerberosAuth, DISABLED
Expand Down Expand Up @@ -58,13 +58,12 @@ class IndexNameFrequency(Enum):
YEARLY = 3

# Defaults for the class
__DEFAULT_ELASTICSEARCH_HOST = [{'host': 'localhost', 'port': 9200}]
__DEFAULT_ELASTICSEARCH_HOST = [{'host': 'localhost', 'port': 9200, 'scheme': 'http'}]
__DEFAULT_AUTH_USER = ''
__DEFAULT_AUTH_PASSWD = ''
__DEFAULT_AWS_ACCESS_KEY = ''
__DEFAULT_AWS_SECRET_KEY = ''
__DEFAULT_AWS_REGION = ''
__DEFAULT_USE_SSL = False
__DEFAULT_VERIFY_SSL = True
__DEFAULT_AUTH_TYPE = AuthType.NO_AUTH
__DEFAULT_INDEX_FREQUENCY = IndexNameFrequency.DAILY
Expand Down Expand Up @@ -129,7 +128,6 @@ def __init__(self,
aws_secret_key=__DEFAULT_AWS_SECRET_KEY,
aws_region=__DEFAULT_AWS_REGION,
auth_type=__DEFAULT_AUTH_TYPE,
use_ssl=__DEFAULT_USE_SSL,
verify_ssl=__DEFAULT_VERIFY_SSL,
buffer_size=__DEFAULT_BUFFER_SIZE,
flush_frequency_in_sec=__DEFAULT_FLUSH_FREQ_INSEC,
Expand All @@ -142,11 +140,13 @@ def __init__(self,
""" Handler constructor

:param hosts: The list of hosts that elasticsearch clients will connect. The list can be provided
in the format ```[{'host':'host1','port':9200}, {'host':'host2','port':9200}]``` to
make sure the client supports failover of one of the instertion nodes
:param auth_details: When ```CMRESHandler.AuthType.BASIC_AUTH``` is used this argument must contain
a tuple of string with the user and password that will be used to authenticate against
the Elasticsearch servers, for example```('User','Password')
in the format ```[{'host':'host1','port':9200, 'scheme': 'http'},
{'host':'host2','port':9200, 'scheme': 'https'}]``` to make sure the client supports
failover of one of the insertion nodes
:param auth_details: When ```CMRESHandler.AuthType.BASIC_AUTH``` or ```CMRESHandler.AuthType.NTLM_AUTH```
is used this argument must contain a tuple of string with the user and password
that will be used to authenticate against the Elasticsearch servers,
for example```('User','Password')
:param aws_access_key: When ```CMRESHandler.AuthType.AWS_SIGNED_AUTH``` is used this argument must contain
the AWS key id of the the AWS IAM user
:param aws_secret_key: When ```CMRESHandler.AuthType.AWS_SIGNED_AUTH``` is used this argument must contain
Expand All @@ -155,7 +155,6 @@ def __init__(self,
the AWS region of the the AWS Elasticsearch servers, for example```'us-east'
:param auth_type: The authentication type to be used in the connection ```CMRESHandler.AuthType```
Currently, NO_AUTH, BASIC_AUTH, KERBEROS_AUTH are supported
:param use_ssl: A boolean that defines if the communications should use SSL encrypted communication
:param verify_ssl: A boolean that defines if the SSL certificates are validated or not
:param buffer_size: An int, Once this size is reached on the internal buffer results are flushed into ES
:param flush_frequency_in_sec: A float representing how often and when the buffer will be flushed, even
Expand All @@ -182,7 +181,6 @@ def __init__(self,
self.aws_secret_key = aws_secret_key
self.aws_region = aws_region
self.auth_type = auth_type
self.use_ssl = use_ssl
self.verify_certs = verify_ssl
self.buffer_size = buffer_size
self.flush_frequency_in_sec = flush_frequency_in_sec
Expand Down Expand Up @@ -212,19 +210,15 @@ def __get_es_client(self):
if self.auth_type == CMRESHandler.AuthType.NO_AUTH:
if self._client is None:
self._client = Elasticsearch(hosts=self.hosts,
use_ssl=self.use_ssl,
verify_certs=self.verify_certs,
connection_class=RequestsHttpConnection,
serializer=self.serializer)
return self._client

if self.auth_type == CMRESHandler.AuthType.BASIC_AUTH:
if self._client is None:
return Elasticsearch(hosts=self.hosts,
http_auth=self.auth_details,
use_ssl=self.use_ssl,
verify_certs=self.verify_certs,
connection_class=RequestsHttpConnection,
serializer=self.serializer)
return self._client

Expand All @@ -233,11 +227,10 @@ def __get_es_client(self):
raise EnvironmentError("Kerberos module not available. Please install \"requests-kerberos\"")
# For kerberos we return a new client each time to make sure the tokens are up to date
return Elasticsearch(hosts=self.hosts,
use_ssl=self.use_ssl,
verify_certs=self.verify_certs,
connection_class=RequestsHttpConnection,
http_auth=HTTPKerberosAuth(mutual_authentication=DISABLED),
serializer=self.serializer)
serializer=self.serializer,
node_class='requests')

if self.auth_type == CMRESHandler.AuthType.AWS_SIGNED_AUTH:
if not AWS4AUTH_SUPPORTED:
Expand All @@ -247,9 +240,7 @@ def __get_es_client(self):
self._client = Elasticsearch(
hosts=self.hosts,
http_auth=awsauth,
use_ssl=self.use_ssl,
verify_certs=True,
connection_class=RequestsHttpConnection,
serializer=self.serializer
)
return self._client
Expand Down
8 changes: 1 addition & 7 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
# To use a consistent encoding
from codecs import open
from os import path
import sys

here = path.abspath(path.dirname(__file__))

Expand All @@ -22,10 +21,6 @@
'requests'
]

# If python version is above 3.4 (built in enums supported enums)
if sys.version_info <= (3,4):
dependencies.append('enum')

print("List of dependencies : {0}".format(str(dependencies)))

setup(
Expand All @@ -34,7 +29,7 @@
# Versions should comply with PEP440. For a discussion on single-sourcing
# the version across setup.py and the project code, see
# https://packaging.python.org/en/latest/single_source_version.html
version='1.0.0',
version='1.1.0',

description='Elasticsearch Log handler for the logging library',
long_description=long_description,
Expand Down Expand Up @@ -67,7 +62,6 @@

# Specify the Python versions you support here. In particular, ensure
# that you indicate whether you support Python 2, Python 3 or both.
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.6',
],

Expand Down
2 changes: 1 addition & 1 deletion sonar-project.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
sonar.projectKey=cmr.python:python-elasticsearch-logger
sonar.projectName=Python Elasticsearch Logger
sonar.projectVersion=1.0.0b4
sonar.projectVersion=1.1.0
sonar.verbose=DEBUG
sonar.language=py
sonar.sources=cmreslogging
Expand Down
Loading