Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[14_1] HLT-compatible LHCInfoPer* PopCon (duringFill mode) #46572

Open
wants to merge 1 commit into
base: CMSSW_14_1_X
Choose a base branch
from
Open

[14_1] HLT-compatible LHCInfoPer* PopCon (duringFill mode) #46572

wants to merge 1 commit into from

Conversation

JanChyczynski
Copy link
Contributor

PR description:

Update of the duringFill mode of LHCInfoPer* O2Os to be able to write to HLT and express tags complying to the synchronization rules. It writes only one (or none) payload at execution and writes to an IOV corresponding to a lumisection not yet processed by HLT. It's achieved by using OnlineDBOutputService::writeIOVForNextLumisection instead of PoolDBOutputService::writeMany. The O2Os in duringFill now uses OnlinePopCon and OnlinePopConAnalyzer (implemented in #44927) instead of the original ones.

Depends on the backport of #44927 which is #46571 (has to be tested on top of it and merged after it)

There is a plan to add better unit tests (not necessarily in this PR).

PR validation:

Tested by running the workflow in various states of LHC (stable beams, no stable beams etc.) for a few fills. Each time the logs procured and the uploaded payloads were examined.
The unit tests for the duringFill mode have been updated (although its extension is planned).
Example commands for running the workflow:

cd CondTools/RunInfo/test/
cmsRun ../python/LHCInfoPerLSPopConAnalyzer_cfg.py mode=duringFill destinationConnection=sqlite:test.db \
lastLumiFile='last_lumi.txt'  tag=perls_test_tag startTime='2022-06-26 15:43:20.000'

cmsRun ../python/LHCInfoPerFillPopConAnalyzer_cfg.py mode=duringFill destinationConnection=sqlite:fill_during/test.db \
lastLumiFile='last_lumi.txt' tag=perfill_test_tag  startTime='2022-06-26 15:43:20.000'

Backport

Backport of #45395

@cmsbuild cmsbuild added this to the CMSSW_14_1_X milestone Oct 31, 2024
@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 31, 2024 •
edited
Loading

A new Pull Request was created by @JanChyczynski for CMSSW_14_1_X.

It involves the following packages:

  • CondTools/RunInfo (db)

@atpathak, @cmsbuild, @consuegs, @francescobrivio, @perrotta can you please review it and eventually sign? Thanks.
@JanChyczynski, @PonIlya, @mmusich, @rsreds, @yuanchao this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

cmsbuild commented Oct 31, 2024 •
edited
Loading

cms-bot internal usage

@JanChyczynski
Copy link
Contributor Author

@perotta Can you please trigger the tests? Remember about testing it on top of #46571

@JanChyczynski JanChyczynski mentioned this pull request Oct 31, 2024
Merged
@perrotta
Copy link
Contributor

perrotta commented Nov 1, 2024

backport of #45395

@perrotta
Copy link
Contributor

perrotta commented Nov 4, 2024

test parameters:

@perrotta
Copy link
Contributor

perrotta commented Nov 4, 2024

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 4, 2024

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cef566/42577/summary.html
COMMIT: 7ca1ca0
CMSSW: CMSSW_14_1_X_2024-11-04-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46572/42577/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 70 lines to the logs
  • Reco comparison results: 10 differences found in the comparisons
  • DQMHistoTests: Total files compared: 46
  • DQMHistoTests: Total histograms compared: 3565669
  • DQMHistoTests: Total failures: 6
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3565643
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
  • Checked 199 log files, 169 edm output root files, 46 DQM output files
  • TriggerResults: no differences found

@JanChyczynski
HLT-compatible LHCInfoPer* PopCon (duringFill mode)
38b493d 
adjust LHCInfoPer*PopConSourceHandler duringFill mode for HLT-compatibility

fix LHCInfoPerLS PopCon duringFill startSampleTime error

fix filtering in  duringFill mode (prevPayload error)

Remove adding empty payload to empty tag

Extract SourceHandler from LHCInfoPer*PopConAnalyzer

Make perLS and perFill duringFill use OnlinePopCon

Attempt to configure OnlineDBOutputService in LHCInfoPer*PopConAnalyzer_cfg.py

finish LHCInfoPer*PopConAnalyzer_cfg files for updated duringFill

For perLS duringfill implement adding defaultPayload and skipping invalid payloads

implement adding defaultPayload in duringFill mode

duringFill perLS: skip upload if invalid values

Adjust duringFill unit tests

remove duplicated kLumisectionsQueryLimit

fix unit tests, print logs of failed lhcInfoNewPopCon test

add connection strings as cfg parameters, fix frontier key parameter for perFill
@JanChyczynski JanChyczynski force-pushed the 14_1-pr-HLT-compatible-LHCInfoPer-PopCon branch from 7ca1ca0 to 38b493d Compare November 5, 2024 14:51
@JanChyczynski JanChyczynski changed the title HLT-compatible LHCInfoPer* PopCon (duringFill mode) [14_1] HLT-compatible LHCInfoPer* PopCon (duringFill mode) Nov 5, 2024
@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 5, 2024

Pull request #46572 was updated. @atpathak, @cmsbuild, @consuegs, @francescobrivio, @perrotta can you please check and sign again.

@perrotta
Copy link
Contributor

perrotta commented Nov 5, 2024

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 5, 2024

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cef566/42601/summary.html
COMMIT: 38b493d
CMSSW: CMSSW_14_1_X_2024-11-05-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46572/42601/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 6 differences found in the comparisons
  • DQMHistoTests: Total files compared: 46
  • DQMHistoTests: Total histograms compared: 3565669
  • DQMHistoTests: Total failures: 6
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3565643
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
  • Checked 199 log files, 169 edm output root files, 46 DQM output files
  • TriggerResults: no differences found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport-ok db-pending orp-pending pending-signatures requires-external tests-approved
Projects
None yet
Milestone
CMSSW_14_1_X
Development

Successfully merging this pull request may close these issues.
3 participants
@JanChyczynski @cmsbuild @perrotta