how to yield in block ULT

[lishujun5]

ULTs can make blocking calls, but users must be careful. If a ULT is blocked without yielding, it will block the entire ES. An efficient user library built on Argobots should design carefully to reduce the blocking time as much as possible.

I am confused about this, how can I yield in a block function call in a ULT. could you give me some example which show me show to use a block function in a ULT but not block the entire ES?

[shintaro-iwasaki]

Thanks for asking a question and sorry for the late reply. Basically, the statement you quoted explains that "Argobots' user-level threading system adopts cooperative multitasking, so the user needs to be careful".

Clarification: "blocking call" in this context

Here "blocking call" assumes that, ultimately, the following form of the code that waits for something.

// Basic form
while (is_complete());

// Example 1: completion of an MPI operation
while (true) {
	MPI_Test(&req, &flag, MPI_STATUS_IGNORE);
	if (flag)
		break;
}

// Example 2: wait for spinlock
while (pthread_mutex_trylock(&mutex) == 0);

// futex would be another blocking primitive, but let me omit some corner cases.

Problem of blocking call in a ULT function

It is allowed to make such blocking calls in a ULT function (i.e., the program doesn't suddenly crash because of this), but users must be careful. If a ULT enters a blocking function (i.e., entering such a loop) without calling a ULT-yield function, such as ABT_self_yield(), it will block the entire execution stream, which is mapped to one OS-level thread (a typical thread created by pthread_create or std::thread).

For example, if you create only one execution stream using ABT_xstream_create(), your program runs all the ULTs on a single execution stream. If one ULT enters a blocking call without a proper ULT-yielding function, the other ULTs are not scheduled. In the following case, ULT2 will never be scheduled while ULT1 is in the while loop. This is counterintuitive since ULT2 will be scheduled if these ULT1() and ULT2() functions are running on typical "threads" created by pthread_create or std::thread.

void ULT1(void *) {
	[...];
	while (true) {
		MPI_Test(&req, &flag, MPI_STATUS_IGNORE);
		if (flag)
			break;
	}
}
void ULT2(void *) {
	[...]; // Never scheduled until an MPI operation above finishes and flag is set to 1
}

How to yield in a block function call

To let ULT2 schedule even if you use only one execution stream and ULT1 is in the while loop, you need to explicitly call a ULT-yield function in ULT1 periodically.

void ULT1(void *) {
	while (true) {
		MPI_Test(&req, &flag, MPI_STATUS_IGNORE);
		if (flag)
			break;
		ABT_self_yield(); // yield: go back to an execution stream that is running ULT1.
		                  // perhaps ULT2 will be scheduled on top of that execution stream.
	}
}

If you cannot insert a ULT-yield function into a blocking function

A blocking function is not always exposed to the user. For example, pthread_spin_lock() internally has a while loop, but its implementation is not exposed to the user. I believe most implementations provide non-blocking versions (like pthread_spin_trylock() or MPI_Test()), so please use these versions with ABT_self_yield().

If the implementation only provides a blocking version, there is no good solution; perhaps you might want to either create more execution streams or ignore blocking calls if that blocking call doesn't cause a deadlock. We conducted research on efficient non-cooperative multitasking on top of Argobots (https://dl.acm.org/doi/10.1145/3437801.3441610), but the current Argobots does not implement this option.

[carns]

Another possibility is to delegate blocking function calls to other execution streams. That's heavy handed, but it might be the only option in some cases. There is a fairly simple example at https://github.com/mochi-hpc/mochi-abt-io if you want to see that approach. It delegates blocking I/O system calls to a separate pool that is set aside for that specific purpose. The original caller suspends on an eventual (which allows other ULTs to execute) until the delegated call is complete.