DIRAC 8.0

Python3

DIRAC 8.0 drops the python2 support, both client and server. Before moving to DIRAC 8.0, all clients, pilots and servers need to be moved to Python3. Instructions can be found in https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-7.3-(v7r3).

The default python version for running DIRAC services is provided by DIRACOS2 (3.9).

The python2 support is kept in certain sections of the code specifically for starting Pilots on nodes that don't provide Python 3.

ElasticSearch support

This version drops the support for ElasticSearch 6. If you are using ES6, you need to move to ES7 (or OpenDistro, or OpenSearch) before installing version 8.0 of DIRAC.

Core

Change of DIRAC Time Module into TimeUtilities

Following https://github.com/DIRACGrid/DIRAC/pull/6091 the Time module was renamed TimeUtilities and was significantly simplified. One of these updates was to implement UTC usage in the module (as it was previously dependent on local timezone). This can result in a time gap especially in the Monitoring data for the hours in which the update comes into effect due to the change of timezone.

The following changes were done:

• Removed the Time.time(), Time.date(), Time.dateTime(), Time.to2K() and Time.from2K().
• Added TimeUtilities.toEpochMilliSeconds()
• Replaced Time.dateTime() -> datetime.datetime.utcnow()
• Replaced Time.toString() -> str(datetime.datetime.utcnow())

Changes in MySQL module

https://github.com/DIRACGrid/DIRAC/pull/6347 introduced the possibility to profile the queries. One change that this brought is that the named parameters MUST be named for the _query, _update, executeStoredProcedureWithCursor, and executeStoredProcedure

DataManagement System

SRM + HTTPs

Following https://github.com/DIRACGrid/DIRAC/pull/6061, it is now possible to submit transfers TPC between SRM and https via FTS. Note that the TURL parameter must be set by the FTS team.

StorageElement changes

https://github.com/DIRACGrid/DIRAC/pull/6283 introduces some breaking API changes as well as a different behavior in corner cases.

The main API changes are:

• se.storages was a list, it is now a dict. The keys are the protocol section name
• se.localPlugins becomes se.localProtocolSections
• se.remotePlugins becomes se.remoteProtocolSections
• se.getPlugins() becomes se.getProtocolSections
• se.getRemotePlugins() becomes se.getRemoteProtocolSections()
• se.getLocalPlugins() becomes se.getLocalProtocolSections()
• se.getStorageParameters: plugin parameter is replaced with protocolSection

In terms of behavior changes:

• StorageElements/DefaultProtocols is not used anymore, and only the outputProtocols config of the SE SRM config is used
• The local plugin is favored in case a local and a remote section provide the same protocol (before, the order was random)

Except for the API changes, we do not think that there will be any disruption caused by the behavior changes, as they are limited to edge cases.

FTS Token support

Following https://github.com/DIRACGrid/DIRAC/pull/7440 the Files table should be updated

alter table Files CHANGE `status` `status` enum('New','Submitted','Ready','Active','Finished','Canceled','Staging','Failed','Defunct','Started','Not_used','Archiving', 'Token_prep') DEFAULT 'New'

LHCb DFC manager update

Following https://github.com/DIRACGrid/DIRAC/pull/6402, the stored procedure should be updated in the FileCatalogDB

DROP PROCEDURE IF EXISTS ps_get_direct_children;
DELIMITER //
CREATE PROCEDURE ps_get_direct_children
(IN dir_id INT )
BEGIN
   SELECT SQL_NO_CACHE ChildID FROM FC_DirectoryClosure WHERE ParentID = dir_id and Depth = 1;
END //
DELIMITER ;

Following https://github.com/DIRACGrid/DIRAC/pull/6682, a new procedure needs to be created

DROP PROCEDURE IF EXISTS ps_get_directory_dump;
DELIMITER //
CREATE PROCEDURE ps_get_directory_dump
(IN dir_id INT)
BEGIN


  DECLARE exit handler for sqlexception
    BEGIN
    ROLLBACK;
    RESIGNAL;
  END;




  (SELECT d.Name ,NULL, d.CreationDate
    FROM FC_DirectoryList d
    JOIN FC_DirectoryClosure c
      ON d.DirID = c.ChildID
    WHERE c.ParentID = dir_id
      AND Depth != 0
  )
  UNION ALL
  (SELECT CONCAT(d.Name, '/', f.FileName), Size, f.CreationDate
    FROM FC_Files f
    JOIN FC_DirectoryList d
      ON f.DirID = d.DirID
    JOIN FC_DirectoryClosure c
      ON c.ChildID = f.DirID
    WHERE ParentID = dir_id
  );
END //
DELIMITER ;

Framework System

Removal of gMonitor and Framework/Monitoring

• The following services must to be uninstalled (the code supporting them is gone):
  • Framework/Monitoring
  • Framework/Plotting

On the machine hosting the Framework/Monitoring service, the directory /opt/dirac/data/monitoring can be removed. The MySQL database "ComponentMonitoringDB" can be removed.

Following this, the ActivityMonitor on the WebApp has also been removed.

The code of any DIRAC extension using gMonitor or contacting the Framework/Monitoring service should be removed. If you happen to have this case, an alternative is provided through an ElasticSearch backend.

Flag for disabling the SecurityLogging service

Please see https://github.com/DIRACGrid/DIRAC/pull/5760 for details on how to use ElasticSearch for reaching the same result. The SecurityLogging service will be removed from later releases.

New agent

The new agent Framework/ProxyRenewalAgent has to be installed. The agent Framework/MyProxyRenewalAgent is instead discontinued and should be removed (its functionalities are part of ProxyRenewalAgent). The new agent Framework/ProxyRenewalAgent has to be installed even if you were not previously using Framework/MyProxyRenewalAgent.

UserProfile

DB change

Increase the size of one field in UserProfileDB:

USE UserProfileDB;
ALTER Table `up_Users` MODIFY COLUMN UserName VARCHAR(64);

If any dashboards containing plots are stored in UserProfileDB then it will need to be updated to reflect the changes in the webapp.

1. It's recommended to backup the exisiting data into a new column.

ALTER TABLE up_ProfilesData ADD DataBak mediumblob default null AFTER Data;
UPDATE up_ProfilesData SET DataBak = Data;

2. Run the following Python code to see what will be changed:

from copy import deepcopy
import json
import zlib
import base64

import DIRAC
DIRAC.initialize()
from DIRAC.FrameworkSystem.DB.UserProfileDB import UserProfileDB
from DIRAC.Core.Utilities import DEncode
from DIRAC.Core.Utilities.ReturnValues import returnValueOrRaise


def fix_plot(params):
    if all(k.startswith("_") for k in params):
        return {k[1:]: v for k, v in params.items()}
    else:
        return params


def fix_plots(data):
    for plot in data["plots"]:
        plot["params"] = fix_plot(plot["params"])


def do_fixes(dry_run):
    updb = UserProfileDB()

    escape = lambda x: returnValueOrRaise(updb._escapeString(x))

    for a, b in returnValueOrRaise(UserProfileDB()._query("select Data, DataBak from up_ProfilesData;")):
        if a != b:
            raise NotImplementedError("Mismatch between Data and DataBak")

    for Profile, VarName, UserId, GroupId, raw in returnValueOrRaise(updb._query("select Profile, VarName, UserId, GroupId, Data from up_ProfilesData;")):
        raw, _ = DEncode.decode(raw)
        if isinstance(raw, (dict, list, bool)):
            continue
        try:
            base64.b64decode(raw)
        except Exception:
            continue
        x = json.loads(DEncode.decode(zlib.decompress(base64.b64decode(raw)))[0])
        if isinstance(x, bool):
            continue
        y = deepcopy(x)

        if "plots" in y:
            fix_plots(y)
        elif "data" in y:
            for a in y["data"]:
                if "module" not in a:
                    continue
                if a["module"] not in ["LHCbDIRAC.Accounting.classes.Accounting", "DIRAC.Accounting.classes.Accounting"]:
                    continue
                fix_plots(a["data"])
        elif "plotParams" in y:
            y["plotParams"] = fix_plot(y["plotParams"])
        else:
            assert any(k in y for k in {"columns", "link", "expandedNodes", "linkToLoad", "leftMenu", "text"}) or y == {}
        if x == y:
            print("Skipping", Profile, VarName, UserId, GroupId)
        else:
            print("Fixing", Profile, VarName, UserId, GroupId)
            print(x)
            print(y)
            new_value = base64.b64encode(DEncode.encode(base64.b64encode(zlib.compress(DEncode.encode(json.dumps(y))))))
            if not dry_run:
                returnValueOrRaise(updb._query(
                    f"update up_ProfilesData set data = from_base64({escape(new_value)}) "
                    f"where Profile = {escape(Profile)} and VarName = {escape(VarName)} and UserId = {int(UserId)} and GroupId = {int(GroupId)} "
                ))


do_fixes(dry_run=True)

3. Once you're happy with the changed modify dry_run so that changes are actually committed to the DB.

4. Once you're happy the migration was sucessful you should remove the DataBak column.

WorkloadManagement System

Lookup in DIRAC/VOPolicy// removed

The PR https://github.com/DIRACGrid/DIRAC/pull/6160 fully removes the lookup in CS for location DIRAC/VOPolicy// that was discontinued in DIRAC v7r2: https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r2#move-vopolicyinputdatamodule-to-operations

JobDB: move from BLOB to TEXT

While introduced with https://github.com/DIRACGrid/DIRAC/pull/5840, this can of course be done at any time (also while running 7.3)

use JobDB;
ALTER TABLE `JobJDLs` MODIFY COLUMN `JDL` MEDIUMTEXT, MODIFY COLUMN `JobRequirements` TEXT, MODIFY COLUMN `OriginalJDL` MEDIUMTEXT;
ALTER TABLE `JobParameters` MODIFY COLUMN `Value` TEXT;
ALTER TABLE `OptimizerParameters` MODIFY COLUMN `Value` MEDIUMTEXT;
ALTER TABLE `AtticJobParameters` MODIFY COLUMN `Value` TEXT;
ALTER TABLE `SiteMask` MODIFY COLUMN `Comment` TEXT;
ALTER TABLE `SiteMaskLogging` MODIFY COLUMN `Comment` TEXT;
ALTER TABLE `HeartBeatLoggingInfo` MODIFY COLUMN `Value` TEXT;
use PilotAgentsDB;
ALTER TABLE `PilotAgents` MODIFY COLUMN `GridRequirements` TEXT;
ALTER TABLE `PilotOutput` MODIFY COLUMN `StdOutput` MEDIUMTEXT, MODIFY COLUMN `StdError` MEDIUMTEXT;

Production System

ProductionDB: move from BLOB to TEXT

While introduced with https://github.com/DIRACGrid/DIRAC/pull/5931, this can of course be done at any time (also while running 7.3)

USE ProductionDB;
ALTER Table `Productions` MODIFY COLUMN Description LONGTEXT;
ALTER Table `ProductionSteps` MODIFY COLUMN LongDescription TEXT;
ALTER Table `ProductionSteps` MODIFY COLUMN Body LONGTEXT;
ALTER Table `ProductionSteps` MODIFY COLUMN InputQuery LONGTEXT;
ALTER Table `ProductionSteps` MODIFY COLUMN OutputQuery LONGTEXT;

Request Management System

ReqDB: move from BLOB to TEXT

While introduced with https://github.com/DIRACGrid/DIRAC/pull/6114, this can of course be done at any time (also while running 7.3)

USE ReqDB;
ALTER Table `Operation` MODIFY COLUMN Arguments TEXT;
ALTER Table `Request` MODIFY COLUMN SourceComponent VARCHAR(255);

Transformation System

TransformationDB: move from BLOB to TEXT

While introduced with https://github.com/DIRACGrid/DIRAC/pull/5828, this can of course be done at any time

ALTER TABLE `TransformationMetaQueries` MODIFY COLUMN `MetaDataValue` TEXT;
ALTER TABLE `AdditionalParameters` MODIFY COLUMN `ParameterValue` LONGTEXT;
ALTER TABLE `Transformations` MODIFY COLUMN `Body` LONGTEXT, MODIFY COLUMN `LongDescription` TEXT;

Monitoring System

Record committing to ES

Following https://github.com/DIRACGrid/DIRAC/pull/6076 the way records are committed to Monitoring has been changed. Given the default milliseconds unit for storing records in ES, the timestamps of each record now need to be in milliseconds when being added, instead of when they are being committed as it was previously done. This can be easily done e.g. with timestamp = int(toEpochMilliseconds()).

New Flag System

In order to simplify the system of flags that are used to enable the monitoring in DIRAC, there is now a new section in the CS under Operations called MonitoringBackends, where there is a flag Default which can be set as Accounting(always set as default) and Monitoring and will decide the monitoring backend for all monitoring types.

There is also an option to override this default flag to set a specific backend for a monitoring type, for which you would need to create a new specific flag. More information on https://dirac.readthedocs.io/en/integration/AdministratorGuide/Systems/MonitoringSystem/index.html#enable-the-monitoring-system.

Note: Please do remove the old flags that are specified for each type in the following sections.

New Monitoring Types

New PilotSubmissionMonitoring

PR https://github.com/DIRACGrid/DIRAC/pull/5788 introduces a new PilotSubmissionMonitoring. The functionalities are the same of PilotSubmissionAccounting, but using the DIRAC Monitoring system (ElasticSearch) as backend.

Old flags to be removed: SendPilotSubmissionAccounting, located in WorkloadManagement/SiteDirector in the CS. Enabled by setting Monitoring value in MonitoringBackends flag.

New DataOperationMonitoring

Data Operation can now also be monitored by the DIRAC Monitoring System. Whether the data is sent to Accounting and/or Monitoring depends on the MonitoringBackends flag.

Replacing ComponentMonitoring with two new types

Following this Pull Request, ComponentMonitoring is replaced by two new types that monitor agents and services respectively instead of all in one: AgentMonitoring and ServiceMonitoring. These won't be present on the DIRAC WebApp but will be available on Kibana/Grafana dashboards.

Also these are enabled by setting Monitoring value in MonitoringBackends flag.

Old flag to be removed: EnableActivityMonitoring in Operations/Defaults.

New PilotsHistoryMonitoring

New monitoring type that sends a snapshot of PilotAgentsDB to Elasticsearch every 15m, as it is similarly done with WMSHistory. Won't be implemented for Accounting.

Enabled by setting Monitoring value in MonitoringBackends flag.

Nginx

If requests to DIRAC server are processed by nginx, then you need to make the following updates:

• pass the X-SSL-CERT header to the escaped user certificate pem by adding the following to the nginx configuration:

location ~ /DIRAC/ {
    ...
    proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;

WebApp portal

A portal compatible with this version no longer contains path "/" to "/DIRAC/" redirects, administrator must add this to the nginx configuration as needed:

location = / {
    rewrite ^ https://$server_name/DIRAC/ permanent;
}

WebApp

A new style of webapp handler has been adopted. Any WebAppDIRAC extensions need to be adapted using the step described in How to migrate to new WebHandler class.

Technology Previews

OAuth2 authorization

OAuth2 authorization - a feature that you can try, for this you will need more actions:

Nginx

• add new upstream that describe REST endpoints

upstream tornadoserver_8443 {
  server 127.0.0.1:8443;
}

• add location to describe REST endpoints access

location ~ ^/(?!(DIRAC|pilot)) {
  proxy_pass_header Server;
  proxy_set_header Host $http_host;
  proxy_redirect off;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Scheme $scheme;
  proxy_pass https://tornadoserver_8443;
  proxy_read_timeout 3600;
  proxy_send_timeout 3600;

  proxy_set_header X-Ssl_client_verify $ssl_client_verify;
  proxy_set_header X-Ssl_client_s_dn $ssl_client_s_dn;
  proxy_set_header X-Ssl_client_i_dn $ssl_client_i_dn;
  proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;

  gzip on;
  gzip_proxied any;
  gzip_comp_level 9;
  gzip_types text/plain text/css application/javascript application/xml application/json;

  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  break;
}

DIRAC configuration

• Describe WebApp client credentials in /DIRAC/Security/Authorization

Authorization
{
  Clients
  {
    DIRACWeb
    {
      client_id = <any string>       # Should be in the local dirac.cfg as secret information
      client_secret = <any string>   # Should be in the local dirac.cfg as secret information
      redirect_uri = https://<your domain>/DIRAC/loginComplete
    } 
  }
}

• register an OAuth 2 client on the your Identity Provider and write received client credentials in a /Resources/IdProviders

IdProviders
{
  CheckIn                                                         # Identity Provider name
  {
    ProviderType = CheckIn                                        # Can be also IAM or just OAuth2
    issuer = https://aai.egi.eu/auth/realms/egi                   # Authorization Server endpoint url
    scope = openid+profile+offline_access+eduperson_entitlement   # Default scope
    logoURL = https://csirt.egi.eu/files/2016/10/egi-logo.png     # Logo to show users in the authentication dialog
    client_id = <EGI client ID>                                   # Should be in the local dirac.cfg as secret information
    client_secret = <EGI client secret>                           # Should be in the local dirac.cfg as secret information
  }
}

• Describe TokenManager service, OAuth 2 REST API and databases in a /Systems/Framework/<instance name> section

Services
{
  TokenManager
  {
    Protocol = https
  }
}
URLs
{
  TokenManager = https://<domain name>:8443/Framework/TokenManager  # Service that will manage tokens 
  AuthAPI = https://<domain name>/auth                              # OAuth 2 REST API
}
Databases
{
  AuthDB                                                            
  {
    DBName = AuthDB                                                 # Registers long sessions
  }
  TokenDB
  {
    DBName = TokenDB                                                # Store user refresh tokens
  }
}

To forbid receiving a proxy as a result of authorization, set /Systems/Framework/<instance name>/APIs/Auth/downloadablePersonalProxy configuration option to False.

Deprecated

DIRACScript

Please, use:

from DIRAC.Core.Base.Script import Script

@Script
def main(self):
   Script.parseCommandLine()
   ...

OR to load configuration without parsing arguments:

from DIRAC import initialize

initialize()

instead of:

from DIRAC.Core.Utilities.DIRACScript import DIRACScript
...