Skip to content

Commit

Permalink
SQL: introduce SQLite storage (#2595)
Browse files Browse the repository at this point in the history
  • Loading branch information
@reinkrul
reinkrul authored Nov 12, 2023
1 parent 06ea85a commit 478a8ee
Show file tree
Hide file tree
Showing 17 changed files with 225 additions and 19 deletions.
5 changes: 4 additions & 1 deletion Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ ARG GIT_VERSION=undefined
LABEL maintainer="[email protected]"

RUN apk update \
&& apk add --no-cache \
gcc \
musl-dev \
&& update-ca-certificates

ENV GO111MODULE on
Expand All @@ -22,7 +25,7 @@ COPY go.sum .
RUN go mod download && go mod verify

COPY . .
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -ldflags="-w -s -X 'github.com/nuts-foundation/nuts-node/core.GitCommit=${GIT_COMMIT}' -X 'github.com/nuts-foundation/nuts-node/core.GitBranch=${GIT_BRANCH}' -X 'github.com/nuts-foundation/nuts-node/core.GitVersion=${GIT_VERSION}'" -o /opt/nuts/nuts
RUN CGO_ENABLED=1 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -ldflags="-w -s -X 'github.com/nuts-foundation/nuts-node/core.GitCommit=${GIT_COMMIT}' -X 'github.com/nuts-foundation/nuts-node/core.GitBranch=${GIT_BRANCH}' -X 'github.com/nuts-foundation/nuts-node/core.GitVersion=${GIT_VERSION}'" -o /opt/nuts/nuts

# alpine
FROM alpine:3.18.4
Expand Down
1 change: 1 addition & 0 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ The following options can be configured on the server:
storage.redis.sentinel.password Password for authenticating to Redis Sentinels.
storage.redis.sentinel.username Username for authenticating to Redis Sentinels.
storage.redis.tls.truststorefile PEM file containing the trusted CA certificate(s) for authenticating remote Redis servers. Can only be used when connecting over TLS (use 'rediss://' as scheme in address).
storage.sql.connection Connection string for the SQL database. If not set, it defaults to a SQLite database stored inside the configured data directory
**VCR**
vcr.openid4vci.definitionsdir Directory with the additional credential definitions the node could issue (experimental, may change without notice).
vcr.openid4vci.enabled true Enable issuing and receiving credentials over OpenID4VCI.
Expand Down
3 changes: 2 additions & 1 deletion docs/pages/deployment/cli-reference.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The following options apply to the server commands below:
--http.default.log string What to log about HTTP requests. Options are 'nothing', 'metadata' (log request method, URI, IP and response code), and 'metadata-and-body' (log the request and response body, in addition to the metadata). (default "metadata")
--http.default.tls string Whether to enable TLS for the default interface, options are 'disabled', 'server', 'server-client'. Leaving it empty is synonymous to 'disabled',
--internalratelimiter When set, expensive internal calls are rate-limited to protect the network. Always enabled in strict mode. (default true)
--jsonld.contexts.localmapping stringToString This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. (default [https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson])
--jsonld.contexts.localmapping stringToString This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. (default [https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson])
--jsonld.contexts.remoteallowlist strings In strict mode, fetching external JSON-LD contexts is not allowed except for context-URLs listed here. (default [https://schema.org,https://www.w3.org/2018/credentials/v1,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json])
--loggerformat string Log format (text, json) (default "text")
--network.bootstrapnodes strings List of bootstrap nodes ('<host>:<port>') which the node initially connect to.
Expand All @@ -70,6 +70,7 @@ The following options apply to the server commands below:
--storage.redis.sentinel.username string Username for authenticating to Redis Sentinels.
--storage.redis.tls.truststorefile string PEM file containing the trusted CA certificate(s) for authenticating remote Redis servers. Can only be used when connecting over TLS (use 'rediss://' as scheme in address).
--storage.redis.username string Redis database username. If set, it overrides the username in the connection URL.
--storage.sql.connection string Connection string for the SQL database. If not set, it defaults to a SQLite database stored inside the configured data directory
--strictmode When set, insecure settings are forbidden. (default true)
--tls.certfile string PEM file containing the certificate for the server (also used as client certificate).
--tls.certheader string Name of the HTTP header that will contain the client certificate when TLS is offloaded.
Expand Down
1 change: 1 addition & 0 deletions docs/pages/deployment/server_options.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@
storage.redis.sentinel.password Password for authenticating to Redis Sentinels.
storage.redis.sentinel.username Username for authenticating to Redis Sentinels.
storage.redis.tls.truststorefile PEM file containing the trusted CA certificate(s) for authenticating remote Redis servers. Can only be used when connecting over TLS (use 'rediss://' as scheme in address).
storage.sql.connection Connection string for the SQL database. If not set, it defaults to a SQLite database stored inside the configured data directory
**VCR**
vcr.openid4vci.definitionsdir Directory with the additional credential definitions the node could issue (experimental, may change without notice).
vcr.openid4vci.enabled true Enable issuing and receiving credentials over OpenID4VCI.
Expand Down
16 changes: 16 additions & 0 deletions docs/pages/deployment/storage-configuration.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,22 @@ The server's certificate will be verified against the OS' CA bundle.

Make sure to `configure persistence for your Redis server <https://redis.io/docs/manual/persistence/>`_.

SQL
===

.. note::

SQL storage is still in development, for now you'll still need the other storage options described by this document.

As we're transitioning to protocols with less shared state, we foresee Nuts' data models to become more relational.
To simplify things, we intent to move towards SQL based storage in the future.
The first database to be supported in SQLite, to aid development and demo/workshop setups. Other, supported SQL databases might be:
- MySQL family (MariaDB, Percona)
- PostgreSQL

By default, storage SQLite will be used in a file called ``sqlite.db`` in the configured data directory.
This can be overridden by configuring a connection string in ``storage.sql.connection`` (only SQLite for now).

Redis Sentinel
^^^^^^^^^^^^^^

Expand Down
11 changes: 8 additions & 3 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ require (
github.com/chromedp/chromedp v0.9.3
github.com/dlclark/regexp2 v1.10.0
github.com/golang-jwt/jwt v3.2.2+incompatible
github.com/golang-migrate/migrate/v4 v4.16.2
github.com/goodsign/monday v1.0.1
github.com/google/uuid v1.4.0
github.com/hashicorp/vault/api v1.10.0
github.com/jinzhu/now v1.1.5 // indirect
github.com/knadh/koanf v1.5.0
github.com/labstack/echo/v4 v4.11.3
github.com/lestrrat-go/jwx/v2 v2.0.16
Expand All @@ -33,6 +35,7 @@ require (
github.com/prometheus/client_golang v1.17.0
github.com/prometheus/client_model v0.5.0
github.com/redis/go-redis/v9 v9.3.0
github.com/santhosh-tekuri/jsonschema v1.2.4
github.com/sirupsen/logrus v1.9.3
github.com/spf13/cobra v1.8.0
github.com/spf13/pflag v1.0.5
Expand All @@ -49,9 +52,8 @@ require (
gopkg.in/Regis24GmbH/go-phonetics.v2 v2.0.3
gopkg.in/yaml.v3 v3.0.1
schneider.vip/problem v1.8.1
)

require github.com/santhosh-tekuri/jsonschema v1.2.4
)

require (
github.com/PaesslerAG/gval v1.2.2 // indirect
Expand Down Expand Up @@ -90,7 +92,7 @@ require (
github.com/gobwas/pool v0.2.1 // indirect
github.com/gobwas/ws v1.3.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/golang-jwt/jwt/v4 v4.4.1 // indirect
github.com/golang-jwt/jwt/v4 v4.4.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
Expand All @@ -117,6 +119,7 @@ require (
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-sqlite3 v2.0.1+incompatible // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 // indirect
Expand Down Expand Up @@ -165,5 +168,7 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d // indirect
gopkg.in/Regis24GmbH/go-diacritics.v2 v2.0.3 // indirect
gorm.io/driver/sqlite v1.5.4
gorm.io/gorm v1.25.5
rsc.io/qr v0.2.0 // indirect
)
21 changes: 14 additions & 7 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,10 @@ github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt v3.2.2+incompatible h1:IfV12K8xAKAnZqdXVzCZ+TOjboZ2keLg81eXfW3O+oY=
github.com/golang-jwt/jwt v3.2.2+incompatible/go.mod h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I=
github.com/golang-jwt/jwt/v4 v4.4.1 h1:pC5DB52sCeK48Wlb9oPcdhnjkz1TKt1D/P7WKJ0kUcQ=
github.com/golang-jwt/jwt/v4 v4.4.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v4 v4.4.2 h1:rcc4lwaZgFMCZ5jxF9ABolDcIHdBytAFgqFPbSJQAYs=
github.com/golang-jwt/jwt/v4 v4.4.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-migrate/migrate/v4 v4.16.2 h1:8coYbMKUyInrFk1lfGfRovTLAW7PhWp8qQDT2iKfuoA=
github.com/golang-migrate/migrate/v4 v4.16.2/go.mod h1:pfcJX4nPHaVdc5nmdCikFBWtm+UBpiZjRNNsyBbp0/o=
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe h1:lXe2qZdvpiX5WZkZR4hgp4KJVfY3nMkvmwbVkpv1rVY=
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
Expand Down Expand Up @@ -218,8 +220,8 @@ github.com/gomodule/redigo v1.8.2/go.mod h1:P9dn9mFrCBvWhGE1wpxx6fgq7BAeLBk+UUUz
github.com/goodsign/monday v1.0.1 h1:yJogH0uQNn4blHjoC3ESbdV0P1OhDtGYdd6x0w7QZBo=
github.com/goodsign/monday v1.0.1/go.mod h1:r4T4breXpoFwspQNM+u2sLxJb2zyTaxVGqUfTBjWOu8=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/flatbuffers v1.12.1 h1:MVlul7pQNoDzWRLTw5imwYsl+usrS1TXG2H4jg6ImGw=
github.com/google/flatbuffers v1.12.1/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v2.0.8+incompatible h1:ivUb1cGomAB101ZM1T0nOiWz9pSrTMoa9+EiY7igmkM=
github.com/google/flatbuffers v2.0.8+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
Expand Down Expand Up @@ -300,8 +302,9 @@ github.com/jinzhu/gorm v1.9.16 h1:+IyIjPEABKRpsu/F8OvDPy9fyQlgsg2luMV2ZIH5i5o=
github.com/jinzhu/gorm v1.9.16/go.mod h1:G3LB3wezTOWM2ITLzPxEXgSkOXAntiLHS7UdBefADcs=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
github.com/jinzhu/now v1.0.1 h1:HjfetcXq097iXP0uoPCdnM4Efp5/9MsM0/M+XOTeR3M=
github.com/jinzhu/now v1.0.1/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/joho/godotenv v1.3.0 h1:Zjp+RcGpHhGlrMbJzXTrZZPrWj+1vfm90La1wgB6Bhc=
Expand Down Expand Up @@ -612,8 +615,8 @@ go.etcd.io/bbolt v1.3.8/go.mod h1:N9Mkw9X8x5fupy0IKsmuqVtoGDyxsaDlbk4Rd05IAQw=
go.etcd.io/etcd/api/v3 v3.5.4/go.mod h1:5GB2vv4A4AOn3yk7MftYGHkUfGtDHnEraIjym4dYz5A=
go.etcd.io/etcd/client/pkg/v3 v3.5.4/go.mod h1:IJHfcCEKxYu1Os13ZdwCwIUTUVGYTSAM3YSwc9/Ac1g=
go.etcd.io/etcd/client/v3 v3.5.4/go.mod h1:ZaRkVgBZC+L+dLCjTcF1hRXpgZXQPOvnA/Ak/gq3kiY=
go.opencensus.io v0.22.5 h1:dntmOdLpSpHlVqbW5Eay97DelsZHe+55D+xC6i0dDS0=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
Expand Down Expand Up @@ -854,6 +857,10 @@ gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gorm.io/driver/sqlite v1.5.4 h1:IqXwXi8M/ZlPzH/947tn5uik3aYQslP9BVveoax0nV0=
gorm.io/driver/sqlite v1.5.4/go.mod h1:qxAuCol+2r6PannQDpOP1FP6ag3mKi4esLnB/jHed+4=
gorm.io/gorm v1.25.5 h1:zR9lOiiYf09VNh5Q1gphfyia1JpiClIWG9hQaxB/mls=
gorm.io/gorm v1.25.5/go.mod h1:hbnx/Oo0ChWMn1BIhpy1oYozzpM15i4YPuHDmfYtwg8=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
rsc.io/qr v0.2.0 h1:6vBLea5/NRMVTz8V66gipeLycZMl/+UlFmk8DvqQ6WY=
Expand Down
1 change: 1 addition & 0 deletions storage/cmd/cmd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,6 @@ func FlagSet() *pflag.FlagSet {
flagSet.StringSlice("storage.redis.sentinel.nodes", defs.Redis.Sentinel.Nodes, "Addresses of the Redis Sentinels to connect to initially. Setting this property enables Redis Sentinel.")
flagSet.String("storage.redis.sentinel.username", defs.Redis.Sentinel.Username, "Username for authenticating to Redis Sentinels.")
flagSet.String("storage.redis.sentinel.password", defs.Redis.Sentinel.Password, "Password for authenticating to Redis Sentinels.")
flagSet.String("storage.sql.connection", defs.SQL.ConnectionString, "Connection string for the SQL database. If not set, it defaults to a SQLite database stored inside the configured data directory")
return flagSet
}
Loading

0 comments on commit 478a8ee

Please sign in to comment.