Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drop require lock when not needed during loading to allow parallel precompile loading #56291

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

vtjnash
Copy link
Member

@vtjnash vtjnash commented Oct 22, 2024

Fixes _require_search_from_serialized to first acquire all start_loading locks (using a deadlock-free batch-locking algorithm) before doing stalechecks and the rest, so that all the global computations happen behind the require_lock, then the rest can happen behind module-specific locks, then (as before) extensions can be loaded in parallel eventually after require returns.

@vtjnash vtjnash added packages Package management and loading multithreading Base.Threads and related functionality labels Oct 22, 2024
Fixes _require_search_from_serialized to acquire all start_loading locks
(using a deadlock-free batch-locking algorithm) before doing stalechecks
and the rest, so that all the global computations happen behind the
require_lock then the rest can happen behind module-specific locks.
@vtjnash
Copy link
Member Author

vtjnash commented Oct 23, 2024

This looks like it is now more easily running into an existing race-condition introduced by #54788 (@Keno) in the import_module function

package loading: Error During Test at /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-c05e8f6358/share/julia/test/threads_exec.jl:1124
2024-10-22 19:55:19 UTC	  Got exception outside of a @test
2024-10-22 19:55:19 UTC	  TaskFailedException
2024-10-22 19:55:19 UTC	
2024-10-22 19:55:19 UTC	      nested task error: importing TestPkg into Main conflicts with an existing global
2024-10-22 19:55:19 UTC	      Stacktrace:
2024-10-22 19:55:19 UTC	       [1] eval
2024-10-22 19:55:19 UTC	         @ ./boot.jl:459 [inlined]
2024-10-22 19:55:19 UTC	       [2] (::var"#229#230"{Event, Channel{Bool}})()
2024-10-22 19:55:19 UTC	         @ Main /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-c05e8f6358/share/julia/test/threads_exec.jl:1137
2024-10-22 19:55:19 UTC	  Stacktrace:
2024-10-22 19:55:19 UTC	    [1] sync_end(c::Channel{Any})
2024-10-22 19:55:19 UTC	      @ Base ./task.jl:601
2024-10-22 19:55:19 UTC	    [2] macro expansion
2024-10-22 19:55:19 UTC	      @ ./task.jl:634 [inlined]
2024-10-22 19:55:19 UTC	    [3] macro expansion
2024-10-22 19:55:19 UTC	      @ /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-c05e8f6358/share/julia/test/threads_exec.jl:1132 [inlined]
2024-10-22 19:55:19 UTC	    [4] macro expansion
2024-10-22 19:55:19 UTC	      @ /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-c05e8f6358/share/julia/stdlib/v1.12/Test/src/Test.jl:1707 [inlined]
2024-10-22 19:55:19 UTC	    [5] macro expansion
2024-10-22 19:55:19 UTC	      @ /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-c05e8f6358/share/julia/test/threads_exec.jl:1125 [inlined]
2024-10-22 19:55:19 UTC	    [6] macro expansion
2024-10-22 19:55:19 UTC	      @ /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-c05e8f6358/share/julia/stdlib/v1.12/Test/src/Test.jl:1707 [inlined]
2024-10-22 19:55:19 UTC	    [7] top-level scope
2024-10-22 19:55:19 UTC	      @ /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-c05e8f6358/share/julia/test/threads_exec.jl:32
2024-10-22 19:55:19 UTC	    [8] include
2024-10-22 19:55:19 UTC	      @ ./Base.jl:582 [inlined]
2024-10-22 19:55:19 UTC	    [9] exec_options(opts::Base.JLOptions)
2024-10-22 19:55:19 UTC	      @ Base ./client.jl:329
2024-10-22 19:55:19 UTC	   [10] _start()
2024-10-22 19:55:19 UTC	      @ Base ./client.jl:558

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
multithreading Base.Threads and related functionality packages Package management and loading
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant