-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support PostgreSQL Databases #437
Merged
Changes from 19 commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
9e3d274
add postgres driver support
carsonab 5538cba
add postgres migrations
carsonab e310469
add postgres tls support
carsonab f3734f0
add script to generate tls certs
carsonab 58f1a6d
gitleaks ignore postgres test certs
carsonab cf3dba8
add postgres connection check
carsonab 94ad19b
wrapping new database errors
carsonab aa2b10a
generate test certs on setup
carsonab b9037a3
add postgres test for non tls
carsonab 2d691e0
add postgres uniqueviolation
carsonab 527bc58
use alloydbomni docker image instead of postgres
carsonab dc7edc4
add alloydb connector support
carsonab cc8581a
cleanup postgres configs
carsonab cdf8853
update gencerts.sh
carsonab de0dc47
add private service connect options
carsonab e6ba4d7
alloydb-go-connector tests
carsonab 33fbfac
test alloydb migrations
carsonab b21fc44
Added in setting the connections for postgres connections
InfernoJJ 11d6cdf
Adding in the logging for errors
InfernoJJ e76d2ae
test with postgres image
carsonab 59fbc9d
build: add health check to postgres container
adamdecaf 2563704
test with different host port
carsonab a4e8a48
Revert "test with different host port"
adamdecaf b91310f
database: use 127.0.0.1 instead of localhost
adamdecaf 112ae62
add ip to cert SANs
carsonab 97ae1c2
build: show docker-compose logs on failure
adamdecaf e0b1055
build: forgot github removed docker-compose
adamdecaf 1798985
chore: chmod postgres key/certs for Github actions
adamdecaf ae459f4
build: capture certs on startup
adamdecaf bcc444c
build: try owning as root
adamdecaf c544239
test: use directory outside of /var/lib/postgresql
adamdecaf c21a935
build: skip TLS postgres setup
adamdecaf File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -50,4 +50,6 @@ coverage.txt | |
|
||
*.pyc | ||
|
||
.idea/* | ||
.idea/* | ||
|
||
testcerts/* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
testcerts/client.key:private-key:1 | ||
testcerts/root.key:private-key:1 | ||
testcerts/server.key:private-key:1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,145 @@ | ||
package database | ||
|
||
import ( | ||
"context" | ||
"database/sql" | ||
"errors" | ||
"fmt" | ||
"net" | ||
|
||
"cloud.google.com/go/alloydbconn" | ||
"github.com/jackc/pgx/v5" | ||
"github.com/jackc/pgx/v5/pgconn" | ||
"github.com/jackc/pgx/v5/stdlib" | ||
_ "github.com/jackc/pgx/v5/stdlib" | ||
"github.com/moov-io/base/log" | ||
) | ||
|
||
const ( | ||
// PostgreSQL Error Codes | ||
// https://www.postgresql.org/docs/current/errcodes-appendix.html | ||
postgresErrUniqueViolation = "23505" | ||
) | ||
|
||
func postgresConnection(ctx context.Context, logger log.Logger, config PostgresConfig, databaseName string) (*sql.DB, error) { | ||
var connStr string | ||
if config.Alloy != nil { | ||
c, err := getAlloyDBConnectorConnStr(ctx, config, databaseName) | ||
if err != nil { | ||
return nil, logger.LogErrorf("creating alloydb connection: %w", err).Err() | ||
} | ||
connStr = c | ||
} else { | ||
c, err := getPostgresConnStr(config, databaseName) | ||
if err != nil { | ||
return nil, logger.LogErrorf("creating postgres connection: %w", err).Err() | ||
} | ||
connStr = c | ||
} | ||
|
||
db, err := sql.Open("pgx", connStr) | ||
if err != nil { | ||
return nil, logger.LogErrorf("opening database: %w", err).Err() | ||
} | ||
|
||
err = db.Ping() | ||
if err != nil { | ||
_ = db.Close() | ||
return nil, logger.LogErrorf("connecting to database: %w", err).Err() | ||
} | ||
|
||
return db, nil | ||
} | ||
|
||
func getPostgresConnStr(config PostgresConfig, databaseName string) (string, error) { | ||
url := fmt.Sprintf("postgres://%s:%s@%s/%s", config.User, config.Password, config.Address, databaseName) | ||
|
||
params := "" | ||
|
||
if config.TLS != nil { | ||
params += "sslmode=verify-full" | ||
|
||
if config.TLS.CACertFile == "" { | ||
return "", fmt.Errorf("missing TLS CA file") | ||
} | ||
params += "&sslrootcert=" + config.TLS.CACertFile | ||
|
||
if config.TLS.ClientCertFile != "" { | ||
params += "&sslcert=" + config.TLS.ClientCertFile | ||
} | ||
|
||
if config.TLS.ClientKeyFile != "" { | ||
params += "&sslkey=" + config.TLS.ClientKeyFile | ||
} | ||
} | ||
|
||
connStr := fmt.Sprintf("%s?%s", url, params) | ||
return connStr, nil | ||
} | ||
|
||
func getAlloyDBConnectorConnStr(ctx context.Context, config PostgresConfig, databaseName string) (string, error) { | ||
if config.Alloy == nil { | ||
return "", fmt.Errorf("missing alloy config") | ||
} | ||
|
||
var dialer *alloydbconn.Dialer | ||
var dsn string | ||
|
||
if config.Alloy.UseIAM { | ||
d, err := alloydbconn.NewDialer(ctx, alloydbconn.WithIAMAuthN()) | ||
if err != nil { | ||
return "", fmt.Errorf("creating alloydb dialer: %v", err) | ||
} | ||
dialer = d | ||
dsn = fmt.Sprintf( | ||
// sslmode is disabled because the alloy db connection dialer will handle it | ||
// no password is used with IAM | ||
"user=%s dbname=%s sslmode=disable", | ||
config.User, databaseName, | ||
) | ||
} else { | ||
d, err := alloydbconn.NewDialer(ctx) | ||
if err != nil { | ||
return "", fmt.Errorf("creating alloydb dialer: %v", err) | ||
} | ||
dialer = d | ||
dsn = fmt.Sprintf( | ||
// sslmode is disabled because the alloy db connection dialer will handle it | ||
"user=%s password=%s dbname=%s sslmode=disable", | ||
config.User, config.Password, databaseName, | ||
) | ||
} | ||
|
||
// TODO | ||
//cleanup := func() error { return d.Close() } | ||
|
||
connConfig, err := pgx.ParseConfig(dsn) | ||
if err != nil { | ||
return "", fmt.Errorf("failed to parse pgx config: %v", err) | ||
} | ||
|
||
var connOptions []alloydbconn.DialOption | ||
if config.Alloy.UsePSC { | ||
connOptions = append(connOptions, alloydbconn.WithPSC()) | ||
} | ||
|
||
connConfig.DialFunc = func(ctx context.Context, _ string, _ string) (net.Conn, error) { | ||
return dialer.Dial(ctx, config.Alloy.InstanceURI, connOptions...) | ||
} | ||
|
||
connStr := stdlib.RegisterConnConfig(connConfig) | ||
return connStr, nil | ||
} | ||
|
||
func PostgresUniqueViolation(err error) bool { | ||
if err == nil { | ||
return false | ||
} | ||
var pgError *pgconn.PgError | ||
if errors.As(err, &pgError) { | ||
if pgError.Code == postgresErrUniqueViolation { | ||
return true | ||
} | ||
} | ||
return false | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason we format a DSN here but for vanilla postgres use their URI syntax?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Either should work. I just preferred the
key=value
syntax when not giving a server (since thats handled by thealloydbconn
library when connecting to a hosted AlloyDB instance)