- Users can install Pydantic V2 without causing pip dependency errors.
- Added support for Pydantic v2.
- Added metadata dict to Result message
- Added details dict to RedisTask
- Quashed warning that would be logged when deserializing
EPStatusReport
messages using theep_status_report
alias forglobal_state
- Renamed funcx-common to globus-compute-common package
-
Renamed the
ep_status_report
field of theEPStatusReport
message toglobal_state
global_state
has an alias ofep_status_report
for backward compatibility
- Container information can now be conveyed directly with Task messages, not just
container_id
- Add an exception and error code for when a serialized function exceeds our size limits.
- Add new exception,
ContainerBuildForbidden
- Added error codes and classes for when submissions to the web service's
batch_run
API are too large
- A
TaskTransition
message type which is used in the Result, ManagerStatusReport, and EPStatusReport to record status events. - Execution-start and execution-end TaskState constants.
- ActorName constants to represent the various entities in the system.
- Add
queue_name
field toRedisTask
. This specifies the name of the AMQP queue where this task's result will be put or found. If not set, this task's result will not be placed into an AMQP queue.
- Added a
.status_log
property to the RedisTask object. The state log requires an atomic append, so this cannot be implemented as another field in the RedisTask hash. Instead, it is implemented as a top-level array.
Task
messages no longer requirecontainer_id
, in support of running tasks that don't require containers.
-
An
sdk_version_sharing
module for centralizing the behavior where the SDK sends its version string to the Web Service for logging. -
A new error class has been added,
messagepack.WrongMessageTypeError
-
All
messagepack.Message
objects now support a new method,assert_one_of_types
, which takesMessage
subclasses as arguments and raises aWrongMessageTypeError
ifisinstance(..., message_types)
does not pass
-
funcx_common.messagepack
now provides a defaultMessagePacker
instance, created at import time. This also allows forpack
andunpack
methods to be provided as functions from the package. The following names are now available for import and use:funcx_common.messagepack.DEFAULT_MESSAGE_PACKER
funcx_common.messagepack.pack
funcx_common.messagepack.unpack
-
funcx_common.messagepack.message_types.Result
now supports two new optional attributes:exec_start_ms
andexec_end_ms
, for execution timing info in milliseconds since epoch. -
funcx_common.messagepack.message_types.Result
has a computed propertyexec_duration_ms
which takesexec_end_ms - exec_start_ms
-
InvalidAuthToken
andInsufficientAuthScope
error classes were added for auth errors that occur in the web service -
Implementation of v1 of the
funcx.messagepack
protocol. See the readme for more info -
The following message types have been added to
messagepack
:Result
,TaskCancel
-
Result
defines an additional model for de/serializing errors:funcx_common.messagepack.message_types.ResultErrorDetails
. TheResultErrorDetails
object is used to wrap a string code and a user-facing message -
TaskCancel
is defined only to have thetask_id
field
-
-
Remove the v0 implementation of
funcx.messagepack
-
The following message types have been removed from
messagepack
:Heartbeat
,HeartbeatReq
,ResultsAck
-
pydantic>=1,<2
is now required byfuncx-common
-
Messages in the
funcx.messagepack
subpackage are now pydantic models, and their members have changed. Themessage_type
is now astr
, not an enum
-
A new class,
funcx_common.redis_task.RedisTask
has been added, which implements theTaskProtocol
backed withRedisField
attributes. This follows the pattern of existing implementations. -
RedisTask.load
is added as a classmethod for loading tasks from Redis with the requirement that the task must exist in Redis -
A new enum has been added,
funcx_common.tasks.InternalTaskState
, for taskinternal_status
values -
funcx_common.task_storage.TaskStorage
and its child classes (ImplicitRedisStorage
andRedisS3Storage
) now supportstore_payload
andget_payload
methods.
-
A new subpackage,
funcx_common.messagepack
provides an implementation of serialization and deserialization of message objects meant to be compatible withfuncx_endpoint.executors.high_throughput.messages
in its on-the-wire representation of messages. Changes between the two implementations are noted in a README formessagepack
.-
messagepack
defines protocol versions, starting with the current and unversioned v0, for handling changes to the protocol over time -
only v0 is implemented, but a suggested plan for v1 of the protocol can be seen in the
messagepack
readme
-
-
Add the
funcx_common.task_storage.get_default_task_storage()
method, which reads theFUNCX_REDIS_STORAGE_THRESHOLD
andFUNCX_S3_BUCKET_NAME
environment variables and constructs the appropriate TaskStorage object-
Detection of S3/Redis vs Redis-only storage is done by the presence/absence of the
FUNCX_S3_BUCKET_NAME
variable. It can be forced to redis-only by settingFUNCX_REDIS_STORAGE_THRESHOLD="-1"
-
The storage threshold has a default value of 20,000 if not set
-
-
Add
funcx_common.task_storage.ImplicitRedisStorage
as a storage class which only reads and writes the task object, on the assumption that it is a RedisTask
-
RedisS3Storage now contains an ImplicitRedisStorage object which is used to implement read/write to redis
-
RedisS3Storage now requires
bucket_name
andredis_threshold
as keyword-only arguments, if constructed directly
-
A new TaskStorage class that is designed to abstract the storage systems for result or payload blobs
-
A RedisS3Storage implementation which is ready for use for storing results.
- This will store small results in Redis and large results in S3 using a configurable threshold.
- The write to redis is implicit, assuming that task objects define
result
as aRedisField
-
TaskProtocol now defines a new dict attribute:
result_reference
, which is written and read by the TaskStorage component. Tasks must add an implementation of this field to comply with the protocol.
- The
HasRedisConnection
class has been removed
- Redis connections created by
default_redis_connection_factory
now default to loading a redis URL from an environment variable,FUNCX_COMMON_REDIS_URL
, with a default ofredis://localhost:6379
- The redis logging decorator has been moved. It is no longer attached to a
class, but is now available as
funcx_common.redis.redis_connection_error_logging
- The signatures for creating
FuncxRedisPubSub
andFuncxEndpointTaskQueue
have changed. They can now be passed aredis.Redis
object, and default to callingdefault_redis_connection_factory
-
funcx_common.redis.HasRedisConnection
no longer provides thelog_connection_errors
decorator -
funcx_common.redis.FuncxRedisPubSub
andfuncx_common.redis.FuncxEndpointTaskQueue
no longer automatically log redis connection errors to thefuncx_common
logger. Users of these classes should either handle these errors themselves or make use of the new error logging decorator
-
A
funcx_common.redis.HasRedisConnection
object now accepts aredis_connection_factory
callable which is used to instantiate theredis.Redis
object, to allow overrides to the connection construction process. The callable must have the signature(str, int) -> redis.Redis
, and the default is a function visible under the namefuncx_common.redis.default_redis_connection_factory
. -
A new context-manager is available as a method of the
HasRedisConnection
class,HasRedisConnection.connection_error_logging
. This can be used to capture and log (at exception-level) redis connection errors to thefuncx_common
logger -
The default connection construction for
HasRedisConnection
now setshealth_check_interval=30
. To override this setting, useredis_connection_factory
- Begin using
scriv
to manage the changelog - The
FuncxRedisConnection
class has been renamed toHasRedisConnection
. The name change better indicates that this class is not a connection itself -- it's just an inheritable way of constructing a connection.
- Add
FuncxRedisPubSub
, which provides a pubsub wrapper over theredis
library'sPubSub
client
- Fix mypy type inference on RedisField descriptors
-
First version of generic task utilities. Constants starting with task states and a
TaskProtocol
class which defines (some) required properties ofTask
objects. Import fromfuncx_common.tasks
-
Add first version of
FuncxRedisConnection
andFuncxEndpointTaskQueue
utilities for wrapping redis-py. Import fromfuncx_common.redis
, as infrom funcx_common.redis import FuncxEndpointTaskQueue
-
A new extra,
redis
defines a requirement for the redis-py lib. Install withfuncx-common[redis]
to pull in the redis requirement. Installation withoutreids
will result inFuncxRedisConnection
andFuncxEndpointTaskQueue
failing to initialize -
funcx_common.redis
now providesRedisField
, a descriptor which is backed byhset
andhget
against the owning object'sredis_client
attribute for presistence. Initialization ofRedisField
requires theHasRedisFieldsMeta
metaclass.HasRedisFields
may be used to apply this metaclass via inheritance. -
De/serialization of
RedisField
data can be defined with "serde" objects.funcx_common.redis
provides:DEFAULT_SERDE
(strings),INT_SERDE
,JSON_SERDE
, andFuncxRedisEnumSerde
(takes an enum class as an input)
-
Bugfix:
FuncxResponseError.unpack()
correctly handles values not in the known response codes enum -
Add
py.typed
to package data, to publish type annotations -
Bugfix: annotate
FuncxResponseError.http_status_code
as a class var
- Fix links to repo/homepage (including in package metadata)
- Initial Release