Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Year out of range #95

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open

Year out of range #95

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
1f61ddf
Update setup.py
aroder Jun 17, 2020
46a5773
added support for SSH tunneling and for using an array for the filter…
aroder Jun 18, 2020
6a19c27
cast string 'true' in config to boolean
aroder Jun 18, 2020
f919ac7
handle SSL status if there are multiple connections for the same user
aroder Jun 18, 2020
9fc707e
logging
aroder Jun 18, 2020
e6d030d
support catalog as well as properties
aroder Jun 18, 2020
ef59742
support default_replication_key
aroder Jun 18, 2020
0546163
timing
aroder Jun 19, 2020
724dbc0
merged from singer-io/tap-postgres
aroder Jul 13, 2020
479ac87
updated readme
aroder Jul 13, 2020
4623e21
updated unit tests
aroder Jul 14, 2020
a1ee787
merged
aroder Jul 14, 2020
f62e590
handle invalid timestamp 0001-01-01 00:00:00
aroder Jul 14, 2020
4baf45d
import fix
aroder Jul 14, 2020
5c93241
different import
aroder Jul 14, 2020
43ea174
different import
aroder Jul 14, 2020
561fa71
ahh modules
aroder Jul 14, 2020
6aee3ee
modules
aroder Jul 14, 2020
68f902c
modules
aroder Jul 14, 2020
a34310f
imports
aroder Jul 14, 2020
8c480fc
imports
aroder Jul 14, 2020
3f070bf
packaging
aroder Jul 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,4 +106,5 @@ ENV/
env-vars.txt
tap_oracle/__pycache__/
*~
config.json
*config.json
.vscode/
37 changes: 37 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,43 @@ using the wal2json decoder plugin.
SELECT * FROM pg_create_logical_replication_slot('stitch', 'wal2json');
```

## Configuration
Check out `config.json.sample` for an example configuration file.

| Field | Required? | Default | Details |
|----------------------------|-----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| | | | |
| dbname | Yes | | The name of the database to connect to |
| default_replication_method | No | | Allows setting a default replication method. If the replication method for a stream is not set in the catalog.json, then this replication method is used. Can be one of ` |
| default_replication_key | No | | Allows setting a default replication key. Should be a string value containing the field name, e.g. `updated_at` |
| host | Yes | | The host to be used to connect |
| password | Yes | | The password to be used to connect |
| port | Yes | | The port to be used to connect |
| user | Yes | | The user to be used to connect |
| use_ssh_tunnel | No | False | Set to true to open an SSH tunnel and connect to the database through the tunnel |
| ssh_jump_server | No | | Only used if `use_ssh_tunnel` is set to true. This is the URL or IP address of the jump server that the connection should tunnel through |
| ssh_jump_server_port | No | 22 | Only used if `use_ssh_tunnel` is set to true. This is the port of the jump server the SSH tunnel will attempt to connect to |
| ssh_private_key_path | No | | Only used if `use_ssh_tunnel` is set to true. This is the path on the local machine to the private SSH key |
| ssh_username | No | | Only used if `use_ssh_tunnel` is set to true. This is the username to be used to connect to the jump server |
| ssl | No | | If true, the sslmode value is set to "require" otherwise sslmode value is not set |

## Development
### Running tests
Install requirements for development: `pip install -e .[test]`

Install postgres to run tests:
1. sudo apt update
1. sudo apt install postgresql postgresql-contrib
1. sudo service postgresql start
1. sudo passwd postgres (set to postgres)
1. sudo -u postgres psql
1. alter user postgres password 'postgres'
1. \q

Run unit tests: `nosetests --where tests/unittests`



---

Copyright © 2018 Stitch
15 changes: 15 additions & 0 deletions config.json.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"dbname": "dbname",
"default_replication_method": "INCREMENTAL",
"default_replication_key": "updated_at",
"host": "host",
"password": "password",
"port": "5432",
"user": "user",
"use_ssh_tunnel": true,
"ssh_jump_server": "jumpserver.mydomain.com",
"ssh_jump_server_port": 22,
"ssh_private_key_path": "~/.ssh/my-key.pem",
"ssh_username": "username",
"ssl": true
}
15 changes: 11 additions & 4 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,20 @@
classifiers=['Programming Language :: Python :: 3 :: Only'],
install_requires=[
'singer-python==5.3.1',
'psycopg2==2.7.4',
'psycopg2==2.8.4',
'strict-rfc3339==0.7',
'nose==1.3.7'
'sshtunnel==0.1.5'
],
extras_require={
'dev': [
'autopep8>=1.5.3',
'python-dotenv>=0.14.0',
'nose>=1.3.7',
]
},
entry_points='''
[console_scripts]
tap-postgres=tap_postgres:main
''',
packages=['tap_postgres', 'tap_postgres.sync_strategies']
)
packages=['tap_postgres', 'tap_postgres.sync_strategies', 'tap_postgres.typecasters']
)
Loading