GOALS for 2024

[belforte]

based on discussion in CRAB meeting on Sept 14, we have a first set of goals for 2024 (things moved there !)
Similarly to #7635

goals we expect we will achieve:

• Move all VM's to alma9 port all VM's to alma9 #8257
• stop using deprecated HTC API's modernize HTCondor calls #8337
• adapt to new HTCondor python bindings v2
• Install via piypi Migrate CRAB to PYPI ecosystem #7560
• automatic CI pipeline Add new CI pipelines (wishlist) #7958
• Improved recovery task make it easy to deal with tails #7595
  • only a small bit remains FileBased splitting: add FileSplitting to use cases covered by crab recovery CRABClient#5330
• define and implement inferface of CRABClient and Rucio python client.  CRABClient#5221
• Canary deployment for TW, Canary deployments - REST, TW, Publisher #8650
• Task scheduler (something before TaskWorker) Limit task submission #7485
• reboot each schedd every 6 months or so (automatically). We can do via a crontab (in puppet) which stops submissions to Vanilla pool, holds al jobs, wa and reboots. Ned to syncronize with TW putting in drain the day before. Proposal: have hiera variable with list of reboot dates placed in TW config and parsed by TW code and placed in scheduler disk and parsed by crontab
• fix HC dealyed submission is disrupted when condor restarts on the scheduler deferred job start (for HC) gets delayed affer schedd restart #7410
• complete "improved monitoring" based on Summer Students output Develop a "What CRAB is used for" historical dashboard #7169 and more
• make preparelocal use S3 for tarball and unify with --dryrun #6544 change how we handle user sandbox #7461

Things which go to 2025

• use of tokens for talking to CMSWEB Use tokens for crab for backend service-to-service communication #7845 (predicated on CMS TokenTeam progresses)
• refactor/re-evalluate crab resubmit and revisit JobRetry.py policies refactor crab resubmit #6270
• adopt Rahuls tool to insert user data into rucio Adopting cmsrucio_import to CRABClient  #7755
• Utilitiies for users to deal with data management via Rucio (list/delete rules e.g.)
• a job status table for each task (avoid accessing status cache on schedd)
• new copycat functionality for our testing and use in automatic recovery deploy updated copycat functionality #7877
• [ ]
• cleanup code source (remove "transients" from past migrations)

things that we may realistically do, but it is not clear if/when we will want to do

• crab recall implement crab recall #7594
• cleanup an update documentation
• ...