diff --git a/docs/changelog.md b/docs/changelog.md index 3e6a6ebd2..83388e56a 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -23,6 +23,7 @@ - gh-435 Fix CLI to read log dir path from NLog config file. - New Virtual Application Entity support for DICOMWeb STOW-RS APIs to enable dynamic endpoints. +- New data [plug-ins](./plug-ins/overview.md) feature to manipulate incoming outgoing data. ## 0.3.21 diff --git a/docs/plug-ins/overview.md b/docs/plug-ins/overview.md new file mode 100644 index 000000000..4488b1089 --- /dev/null +++ b/docs/plug-ins/overview.md @@ -0,0 +1,136 @@ + + +# Data Plug-ins + +Data plug-ins enable manipulation of incoming data before they are saved to the storage service or outgoing data right before they are exported. + +## Using Data Plug-ins + +The Informatics Gateway allows you to configure data plug-ins in the following services: + +- (DIMSE) MONAI Deploy DICOM Listener: configure each listening AE Title with zero or more data plug-ins via the [CLI](../setup/cli.md) or via the [Configuration API](../api/rest/config.md). +- (DIMSE) DICOM Export: configure the `PluginAssemblies` with one or more data plug-ins in the [ExportRequestEvent](https://github.com/Project-MONAI/monai-deploy-messaging/blob/main/src/Messaging/Events/ExportRequestEvent.cs#L85). +- (DICOMWeb) STOW-RS: + - The Virtual AE endpoints (`/dicomweb/vae/...`) can be configured similarly to the DICOM listener by using the [DICOMWeb STOW API](../api/rest/dicomweb-stow.md##post-dicomwebvaeaetworkflow-idstudiesstudy-instance-uid). + - For the default `/dicomweb/...` endpoints, set zero or more plug-ins under `InformaticsGateway>dicomWeb>plug-ins` in the `appsettings.json` [configuration](../setup/schema.md) file. +- (DICOMWeb) Export: configure the `PluginAssemblies` with one or more data plug-ins in the [ExportRequestEvent](https://github.com/Project-MONAI/monai-deploy-messaging/blob/main/src/Messaging/Events/ExportRequestEvent.cs#L85). + +> [!Note] +> When one or more plug-ins are defined, the plug-ins are executed in the order as they are listed. + +## Available Plug-ins + +The following plug-ins are available: + +| Name | Description | Fully Qualified Assembly Name | +| ------------------------------------ | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| [DicomDeidentifier](./remote-app.md) | A plug-in that de-identifies a set of configurable DICOM tags with random data before DICOM data is exported. | `Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution.DicomDeidentifier, Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution` | +| [DicomReidentifier](./remote-app.md) | A plug-in to be used together with the `DicomDeidentifier` plug-in to restore the original DICOM metadata. | `Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution.DicomReidentifier, Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution` | + + +## Writing Your Plug-ins + +To write an input data plug-in, implement the [IInputDataPlugin](xref:Monai.Deploy.InformaticsGateway.Api.PlugIns.IInputDataPlugIn) interface and +put the [dynamic link library](https://learn.microsoft.com/en-us/troubleshoot/windows-client/deployment/dynamic-link-library) (DLL) in +the `plug-ins/` directories. Similarly, for output data plug-ins, implement the [IOutputDataPlugin](xref:Monai.Deploy.InformaticsGateway.Api.PlugIns.IOutputDataPlugIn) interface. + +Refer to the [Configuration API](../api/rest/config.md) page to retrieve available [input](../api/rest/config.md#get-configaeplug-ins) and [output](../api/rest/config.md#get-configdestinationplug-ins) data plug-ins. + + +### Database Extensions + +If a plug-in requires to persist data to the database, extend the [DatabaseRegistrationBase](xref:Monai.Deploy.InformaticsGateway.Database.Api.DatabaseRegistrationBase) class to register your database context and repositories. + +Refer to the _Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution_ plug-in as a reference. + +> [!Important] +> The Informatics Gateway requires all plug-ins to extend both Entity Framework (sqlite) and MongoDB databases. + +#### Entity Framework + +Implement the [IDatabaseMigrationManagerForPlugIns](xref:Monai.Deploy.InformaticsGateway.Database.Api.IDatabaseMigrationManagerForPlugIns) interface to register your Entity Framework (EF) database context. A `connectionString` is provided to the `Configure(...)` +function when you extend the [DatabaseRegistrationBase](xref:Monai.Deploy.InformaticsGateway.Database.Api.DatabaseRegistrationBase) class and allows you to create your [code-first](https://learn.microsoft.com/en-us/ef/ef6/modeling/code-first/workflows/new-database) +EF database context and generate your migration code using [dotnet ef](https://learn.microsoft.com/en-us/ef/core/cli/dotnet) CLI tool. + +In the following example, we extend [DatabaseRegistrationBase](xref:Monai.Deploy.InformaticsGateway.Database.Api.DatabaseRegistrationBase) class to register a new EF database context named `RemoteAppExecutionDbContext`. + +In the method, we first register the database context, then register our Migration Manager by implementing the [IDatabaseMigrationManagerForPlugIns](xref:Monai.Deploy.InformaticsGateway.Database.Api.IDatabaseMigrationManagerForPlugIns) interface. +Lastly, we register our repository for the `RemoteAppExecutions` table. + +```csharp +public class DatabaseRegistrar : DatabaseRegistrationBase +{ + public override IServiceCollection Configure(IServiceCollection services, DatabaseType databaseType, string? connectionString) + { + Guard.Against.Null(services, nameof(services)); + + switch (databaseType) + { + case DatabaseType.EntityFramework: + Guard.Against.Null(connectionString, nameof(connectionString)); + services.AddDbContext(options => options.UseSqlite(connectionString), ServiceLifetime.Transient); + services.AddScoped(); + + services.AddScoped(typeof(IRemoteAppExecutionRepository), typeof(EntityFramework.RemoteAppExecutionRepository)); + break; + ... + } + + return services; + } +} +``` + +#### MongoDB + +Similar to the [Entity Framework](#entity-framework) section above, For MongoDB, we first register our Migration Manager by implementing the [IDatabaseMigrationManagerForPlugIns](xref:Monai.Deploy.InformaticsGateway.Database.Api.IDatabaseMigrationManagerForPlugIns) interface, +and then we register our repository for the `RemoteAppExecutions` collection. + +```csharp +public class DatabaseRegistrar : DatabaseRegistrationBase +{ + public override IServiceCollection Configure(IServiceCollection services, DatabaseType databaseType, string? connectionString) + { + Guard.Against.Null(services, nameof(services)); + + switch (databaseType) + { + case DatabaseType.MongoDb: + services.AddScoped(); + + services.AddScoped(typeof(IRemoteAppExecutionRepository), typeof(MongoDb.RemoteAppExecutionRepository)); + break; + ... + } + + return services; + } +} +``` + +In the `MigrationManager`, we configure the `RemoteAppExecutions` collection. + +```csharp +public class MigrationManager : IDatabaseMigrationManagerForPlugIns +{ + public IHost Migrate(IHost host) + { + RemoteAppExecutionConfiguration.Configure(); + return host; + } +} +``` diff --git a/docs/plug-ins/remote-app.md b/docs/plug-ins/remote-app.md new file mode 100644 index 000000000..0f870d2cc --- /dev/null +++ b/docs/plug-ins/remote-app.md @@ -0,0 +1,61 @@ + + +# Remote App Execution Plug-ins + +The **Remote App Execution Plug-ins** allow the users to configure a set of DICOM metadata to be replaced with dummy data before +being exported using the `DicomDeidentifier` plug-in. The original data is stored in the database so when the data returns +via DICOM DIMSE or DICOMWeb, the data can be restored using the `DicomReidentifier` plug-in. + +## Supported Data Types + +- DICOM + +## Configuration + +One or more DICOM tags may be configured in the `appsettings.json` file. For example, the following snippet will replace +`AccessionNumber`, `StudyDescription`, and `SeriesDescription`. + +```json +{ + "InformaticsGateway": { + "plugins": { + "remoteApp": { + "ReplaceTags": "AccessionNumber, StudyDescription, SeriesDescription" + } + } + } +} +``` + +Refer to [NEMA](https://dicom.nema.org/medical/dicom/current/output/chtml/part06/chapter_6.html) for a complete list of DICOM tags +and use the value from the **Keyword** column. + +> [!Note] +> `StudyInstanceUID`, `SeriesInstanceUID` and `SOPInstanceUID` are always replaced and tracked to ensure the same +> studies and series get the same UIDs. + +> [!Important] +> Only top-level DICOM metadata can be replaced at this time. + +## Fully Qualified Assembly Names + +The following plug-ins are available: + +| Name | Fully Qualified Assembly Name | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `DicomDeidentifier` | `Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution.DicomDeidentifier, Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution` | +| `DicomReidentifier` | `Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution.DicomReidentifier, Monai.Deploy.InformaticsGateway.PlugIns.RemoteAppExecution` | diff --git a/docs/plug-ins/toc.yml b/docs/plug-ins/toc.yml new file mode 100644 index 000000000..72462cfec --- /dev/null +++ b/docs/plug-ins/toc.yml @@ -0,0 +1,18 @@ +# Copyright 2021-2022 MONAI Consortium +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- name: Data Plug-ins + href: overview.md +- name: Remote App Execution Plug-ins + href: remote-app.md diff --git a/docs/setup/schema.md b/docs/setup/schema.md index 1d152d0b6..b104328fb 100644 --- a/docs/setup/schema.md +++ b/docs/setup/schema.md @@ -20,19 +20,18 @@ The configuration file (`appsettings.json`) controls the behaviors and parameters of the internal services. The file is stored next to the main application binary and provides a subset of the default configuration options by default. Please refer to the [Monai.Deploy.InformaticsGateway.Configuration](xref:Monai.Deploy.InformaticsGateway.Configuration.InformaticsGatewayConfiguration) namespace for complete reference. - ### Configuration Sections `appsettings.json` contains the following top-level sections: ```json { - "ConnectionStrings": "connection string to the database", - "InformaticsGateway": "configuration options for the Informatics Gateway & its internal services", - "Logging": "logging configuration options", - "Kestrel": "web server configuration options. See https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-5.0", - "AllowedHosts": "host filtering option. See https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/host-filtering?view=aspnetcore-5.0", - "Cli": "configurations used by the CLI" + "ConnectionStrings": "connection string to the database", + "InformaticsGateway": "configuration options for the Informatics Gateway & its internal services", + "Logging": "logging configuration options", + "Kestrel": "web server configuration options. See https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-5.0", + "AllowedHosts": "host filtering option. See https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/host-filtering?view=aspnetcore-5.0", + "Cli": "configurations used by the CLI" } ``` @@ -49,6 +48,7 @@ The `InformaticsGateway` configuration section contains the following sub-sectio | hl7 | HL7 listener configuration options | [Hl7Configuration](xref:Monai.Deploy.InformaticsGateway.Configuration.Hl7Configuration) | | storage | Storage configuration options, including storage service and disk usage monitoring | [StorageConfiguration](xref:Monai.Deploy.InformaticsGateway.Configuration.StorageConfiguration) | | messaging | Message broker configuration options | [MessageBrokerConfiguration](xref:Monai.Deploy.InformaticsGateway.Configuration.MessageBrokerConfiguration) | +| plug-ins | Configuration options for plug-ins | [PlugInConfiguration](xref:Monai.Deploy.InformaticsGateway.Configuration.PlugInConfiguration) | | Cli | The configuration used by the CLI | - | --- @@ -57,89 +57,91 @@ The `InformaticsGateway` configuration section contains the following sub-sectio ```json { - "ConnectionStrings": { - "InformaticsGatewayDatabase": "Data Source=/database/mig.db" - }, - "InformaticsGateway": { - "dicom": { - "scp": { - "port": 104, - "logDimseDatasets": false, - "rejectUnknownSources": true - }, - "scu": { - "aeTitle": "MONAISCU", - "logDimseDatasets": false, - "logDataPDUs": false - } + "ConnectionStrings": { + "InformaticsGatewayDatabase": "Data Source=/database/mig.db" }, - "messaging": { - "publisherServiceAssemblyName": "Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessagePublisherService, Monai.Deploy.Messaging.RabbitMQ", - "publisherSettings": { - "endpoint": "localhost", - "username": "username", - "password": "password", - "virtualHost": "monaideploy", - "exchange": "monaideploy" + "InformaticsGateway": { + "dicom": { + "scp": { + "port": 104, + "logDimseDatasets": false, + "rejectUnknownSources": true + }, + "scu": { + "aeTitle": "MONAISCU", + "logDimseDatasets": false, + "logDataPDUs": false + } + }, + "messaging": { + "publisherServiceAssemblyName": "Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessagePublisherService, Monai.Deploy.Messaging.RabbitMQ", + "publisherSettings": { + "endpoint": "localhost", + "username": "username", + "password": "password", + "virtualHost": "monaideploy", + "exchange": "monaideploy" + }, + "subscriberServiceAssemblyName": "Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessageSubscriberService, Monai.Deploy.Messaging.RabbitMQ", + "subscriberSettings": { + "endpoint": "localhost", + "username": "username", + "password": "password", + "virtualHost": "monaideploy", + "exchange": "monaideploy", + "exportRequestQueue": "export_tasks", + "deadLetterExchange": "monaideploy-dead-letter", + "deliveryLimit": 3, + "requeueDelay": 30 + } + }, + "storage": { + "localTemporaryStoragePath": "/payloads", + "remoteTemporaryStoragePath": "/incoming", + "bucketName": "monaideploy", + "storageRootPath": "/payloads", + "temporaryBucketName": "monaideploy", + "serviceAssemblyName": "Monai.Deploy.Storage.MinIO.MinIoStorageService, Monai.Deploy.Storage.MinIO", + "watermarkPercent": 75, + "reserveSpaceGB": 5, + "settings": { + "endpoint": "localhost:9000", + "accessKey": "admin", + "accessToken": "password", + "securedConnection": false, + "region": "local", + "executableLocation": "/bin/mc", + "serviceName": "MinIO" + } + }, + "hl7": { + "port": 2575, + "maximumNumberOfConnections": 10, + "clientTimeout": 60000, + "sendAck": true + }, + "dicomWeb": { + "plugins": [] }, - "subscriberServiceAssemblyName": "Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessageSubscriberService, Monai.Deploy.Messaging.RabbitMQ", - "subscriberSettings": { - "endpoint": "localhost", - "username": "username", - "password": "password", - "virtualHost": "monaideploy", - "exchange": "monaideploy", - "exportRequestQueue": "export_tasks", - "deadLetterExchange": "monaideploy-dead-letter", - "deliveryLimit": 3, - "requeueDelay": 30 - } }, - "storage": { - "localTemporaryStoragePath": "/payloads", - "remoteTemporaryStoragePath": "/incoming", - "bucketName": "monaideploy", - "storageRootPath": "/payloads", - "temporaryBucketName": "monaideploy", - "serviceAssemblyName": "Monai.Deploy.Storage.MinIO.MinIoStorageService, Monai.Deploy.Storage.MinIO", - "watermarkPercent": 75, - "reserveSpaceGB": 5, - "settings": { - "endpoint": "localhost:9000", - "accessKey": "admin", - "accessToken": "password", - "securedConnection": false, - "region": "local", - "executableLocation": "/bin/mc", - "serviceName": "MinIO" - } + "Kestrel": { + "EndPoints": { + "Http": { + "Url": "http://+:5000" + } + } }, - "hl7": { - "port": 2575, - "maximumNumberOfConnections": 10, - "clientTimeout": 60000, - "sendAck": true + "AllowedHosts": "*", + "Cli": { + "Runner": "Docker", + "HostDataStorageMount": "~/.mig/data", + "HostPlugInsStorageMount": "~/.mig/plug-ins", + "HostDatabaseStorageMount": "~/.mig/database", + "HostLogsStorageMount": "~/.mig/logs", + "InformaticsGatewayServerEndpoint": "http://localhost:5000", + "DockerImagePrefix": "ghcr.io/project-monai/monai-deploy-informatics-gateway" } - }, - "Kestrel": { - "EndPoints": { - "Http": { - "Url": "http://+:5000" - } - } - }, - "AllowedHosts": "*", - "Cli": { - "Runner": "Docker", - "HostDataStorageMount": "~/.mig/data", - "HostPlugInsStorageMount": "~/.mig/plug-ins", - "HostDatabaseStorageMount": "~/.mig/database", - "HostLogsStorageMount": "~/.mig/logs", - "InformaticsGatewayServerEndpoint": "http://localhost:5000", - "DockerImagePrefix": "ghcr.io/project-monai/monai-deploy-informatics-gateway" - } } - ``` ### Configuration Validation diff --git a/docs/setup/setup.md b/docs/setup/setup.md index e7d5de64c..7f3092af9 100644 --- a/docs/setup/setup.md +++ b/docs/setup/setup.md @@ -354,12 +354,12 @@ will group instances by the Series Instance UID (0020,000E) with a timeout value Each listening AE Title may be configured with zero or more plug-ins to manipulate incoming DICOM files before saving to the storage service and dispatching a workflow request. To include input data plug-ins, first create your plug-ins by implementing the -[IInputDataPlugin](xref:Monai.Deploy.InformaticsGateway.Api.IInputDataPlugin) interface and then use `-p` argument with the fully -qualified type name with the `mig-cli aet add` command. For example, the following command adds `MyNamespace.AnonymizePlugin` +[IInputDataPlugIn](xref:Monai.Deploy.InformaticsGateway.Api.PlugIns.IInputDataPlugIn) interface and then use `-p` argument with the fully +qualified type name with the `mig-cli aet add` command. For example, the following command adds `MyNamespace.AnonymizePlugIn` and `MyNamespace.FixSeriesData` plug-ins from the `MyNamespace.Plugins` assembly file. ```bash -mig-cli aet add -a BrainAET -grouping 0020,000E, -t 30 -p "MyNamespace.AnonymizePlugin, MyNamespace.Plugins" "MyNamespace.FixSeriesData, MyNamespace.Plugins" +mig-cli aet add -a BrainAET -grouping 0020,000E, -t 30 -p "MyNamespace.AnonymizePlugIn, MyNamespace.PlugIns" "MyNamespace.FixSeriesData, MyNamespace.PlugIns" ``` > [!Note] @@ -383,6 +383,8 @@ The above command tells the Informatics Gateway to accept instances from AE Titl > The Informatics Gateway validates both the source IP address and AE Title when `rejectUnknownSources` is set to `true`. > When the Informatics Gateway is running in a container and data is coming from the localhost, the IP address may not be the same as the host IP address. In this case, open the log file and locate the association that failed; the log should indicate the correct IP address under `Remote host`. +See [Data Plug-ins](../plug-ins/overview.md) to configure data plug-ins or create your own data plug-ins. + ## Export Processed Results If exporting via DIMSE is required, add a DICOM destination: @@ -396,12 +398,3 @@ The command adds a DICOM export destination with AE Title `WORKSTATION1` at IP ` ## Logging See [schema](./schema.md#logging) page for additional information on logging. - -## Data Plug-ins - -You may write your own data plug-ins to manipulate incoming data before they are saved to the storage service or outgoing data right before they are exported. - -To write an input data plug-in, implement the [IInputDataPlugin](xref:Monai.Deploy.InformaticsGateway.Api.IInputDataPlugin) interface and put the [dynamic link library](https://learn.microsoft.com/en-us/troubleshoot/windows-client/deployment/dynamic-link-library) (DLL) in the -plug-ins directories. Similarly for output data plug-ins, implement the [IOutputDataPlugin](xref:Monai.Deploy.InformaticsGateway.Api.IOutputDataPlugin) interface. - -Refer to [Configuration API](../api/rest/config.md) page to retrieve available [input](../api/rest/config.md#get-configaeplug-ins) and [output](../api/rest/config.md#get-configdestinationplug-ins) data plug-ins. diff --git a/docs/toc.yml b/docs/toc.yml index 1c2c30e13..7d052bf97 100644 --- a/docs/toc.yml +++ b/docs/toc.yml @@ -18,6 +18,8 @@ href: setup/ - name: API Documentation href: api/ +- name: Plug-ins + href: plug-ins/ - name: Compliance href: compliance/ - name: Changelog diff --git a/src/InformaticsGateway/appsettings.json b/src/InformaticsGateway/appsettings.json index 2a4f6e86b..3eb753c58 100755 --- a/src/InformaticsGateway/appsettings.json +++ b/src/InformaticsGateway/appsettings.json @@ -99,7 +99,7 @@ }, "plugins": { "remoteApp": { - "ReplaceTags": "StudyInstanceUID, AccessionNumber, SeriesInstanceUID, SOPInstanceUID" + "ReplaceTags": "AccessionNumber" } } },