Flyte v1.13.3 milestone release

Flyte 1.13.3 Release Notes

This is an in-between release in preparation for the big 1.14 release slated for early December.

What's Changed

• Add finalizer to avoid premature CRD Deletion by @RRap0so in #5788
• Handle CORS in secure connections by @eapolinario in #5855
• Upstream contributions from Union.ai by @andrewwdye in #5769
  • Fix cluster pool assignment validation (@iaroslav-ciupin )
  • Don't send inputURI for start-node (@iaroslav-ciupin )
  • Unexpectedly deleted pod metrics (@iaroslav-ciupin )
  • CreateDownloadLink: Head before signing (@iaroslav-ciupin )
  • Fix metrics scale division in timer (@iaroslav-ciupin )
  • Add histogram stopwatch to stow storage (@andrewwdye )
  • Override ArrayNode log links with map plugin (@hamersaw )
  • Fix k3d local setup prefix (@andrewwdye )
  • adjust Dask LogName to (Dask Runner Logs) (@fiedlerNr9 )
  • Dask dashboard should have a separate log config (@EngHabu )
  • Log and monitor failures to validate access tokens (@katrogan )
  • Fix type assertion when an event is missed while connection to apiser… (@EngHabu )
  • Add read replica host config and connection (@squiishyy )
  • added lock to memstore make threadsafe (@hamersaw )
  • Move storage cache settings to correct location (@mbarrien )
  • Add config for grpc MaxMessageSizeBytes (@andrewwdye )
  • Add org to CreateUploadLocation (@katrogan )
  • Histogram Bucket Options (@squiishyy )
  • Add client-go metrics (@andrewwdye )
  • Enqueue owner on launchplan terminal state (@hamersaw )
  • Add configuration for launchplan cache resync duration (@hamersaw )
  • Overlap fetching input and output data (@andrewwdye )
  • Fix async notifications tests (@andrewwdye )
  • Overlap FutureFileReader blob store writes/reads (@andrewwdye )
  • Overlap create execution blob store reads/writes (@andrewwdye )

Full Changelog: v1.13.2...v1.13.3

Flyte v1.13.2 milestone release

Flyte 1.13.2 Release Notes

What's Changed

• Enable echo plugin by default by @pingsutw in #5679
• Do not emit execution id label by default in single binary by @eapolinario in #5704
• Using new offloaded metadata literal message for literal offloading by @pmahindrakar-oss in #5705
• Improve error message for MismatchingTypes by @pingsutw in #5639
• [Docs] Echo Task by @Future-Outlier in #5707
• Improve execution name readability by @wayner0628 in #5637
• Configure imagePullPolicy to be Always pull on flyte sandbox environment by @pmahindrakar-oss in #5709
• Set IsDefault to False for echo plugin by @pingsutw in #5713
• Move default execution name generation to flyteadmin by @wayner0628 in #5714
• Update helm/docs per changes in supported task discovery by @Sovietaced in #5694
• [flyteagent] Add Logging for Agent Supported Task Types by @Future-Outlier in #5718
• extend pod customization to include init containers by @samhita-alla in #5685
• Update "Try Serverless" language in Quickstart guide by @neverett in #5698
• Refactor flyteadmin to pass proto structs as pointers by @Sovietaced in #5717
• fix: Use deterministic execution names in scheduler by @pingsutw in #5724
• [flyteagent] Enable agent-service in every configuration by @Future-Outlier in #5730
• [Docs][flyteagent] Remove agent-service config in flyte agent documentation by @Future-Outlier in #5731
• Fix flytectl returning the oldest workflow when using --latest flag by @RRap0so in #5716
• Remove explicit go toolchain versions by @ddl-ebrown in #5723
• Add listing api to stow storage by @bgedik in #5741
• Use latest upload/download-artifact action version by @Sovietaced in #5743
• Introduced SMTP notification by @robert-ulbrich-mercedes-benz in #5535
• Added literal offloading for array node map tasks by @pmahindrakar-oss in #5697
• pytorch object.inv moved by @wild-endeavor in #5755
• [RFC] Binary IDL With MessagePack Bytes by @Future-Outlier in #5742
• Rename literal offloading flags and enable offloading in local config by @eapolinario in #5754
• Add semver check for dev and beta version by @pmahindrakar-oss in #5757
• Add comments for semver regex by @pmahindrakar-oss in #5758
• Enable caching in flyte-core helmchart by @cpaulik in #5739
• [Flyte][1][IDL] Binary IDL With MessagePack by @Future-Outlier in #5751
• [Flyte][2][Literal Type For Scalar] Binary IDL With MessagePack by @Future-Outlier in #5761
• Backoff on etcd errors by @EngHabu in #5710
• [Docs][flyteagent] Update Databricks Agent Setup to V2 by @Future-Outlier in #5766
• [Docs][flyteagent] Improve Agent Secret Setup by @Future-Outlier in #5765
• [RFC] Offloaded Raw Literals by @wild-endeavor in #5103
• Use pluggable clock for auto refresh cache and make unit tests fast by @Sovietaced in #5767
• [Docs] Flyte Deck example by @Future-Outlier in #5777
• Update ImageSpec documentation by @pingsutw in #5748
• Revert "Improve execution name readability" by @pingsutw in #5740
• [Docs] Flyte Deck example V2 by @Future-Outlier in #5781
• [flytepropeller][flyteadmin] Compiler unknown literal type error handling by @Future-Outlier in #5651
• [Temporary fix] Pin flytekit version for single binary builds by @pmahindrakar-oss in #5798
• Fix propeller crash when inferring literal type for an offloaded literal by @pmahindrakar-oss in #5771

New Contributors

• @wayner0628 made their first contribution in #5637
• @robert-ulbrich-mercedes-benz made their first contribution in #5535
• @cpaulik made their first contribution in #5739

Full Changelog: v1.13.1...v1.13.2

flytectl/v0.9.2

Changelog

• ca04314 Auth/prevent lookup per call (#5686)
• 89efcc6 Fix flytectl returning the oldest workflow when using --latest flag (#5716)
• 96c799c Move default execution name generation to flyteadmin (#5714)
• 5f69589 Refactor flyteadmin to pass proto structs as pointers (#5717)
• 43c9d94 Turn flyteidl and flytectl releases into manual gh workflows (#5635)
• 7d59f10 [flytectl] DataConfig missing from TaskSpec (#5692)

Flyte v1.13.1 milestone release

Flyte 1.13.1 Release Notes

What's Changed

• chore: update runllm widget configuration by @agiron123 in #5530
• Use jsonpb AllowUnknownFields everywhere by @andrewwdye in #5521
• [flyteadmin] Use WithContext in all DB calls by @Sovietaced in #5538
• add new project state: SYSTEM_ARCHIVED by @troychiu in #5544
• Pryce/doc 434 clarify how code is pushed into a given image during pyflyte by @pryce-turner in #5548
• Increase more memory limits in flyteagent by @Future-Outlier in #5550
• Updated map task information to indicate array node is now the default and optional return type by @pryce-turner in #5561
• reverted mockery-v2 on ExecutionContext by @hamersaw in #5562
• Fix issues with helm chart release process by @Sovietaced in #5560
• Add FlyteDirectory to file_types.rst template by @ppiegaze in #5564
• Fully populate Abort task event fields by @va6996 in #5551
• [fix] Add blob typechecker by @ddl-rliu in #5519
• Refactor echo plugin by @pingsutw in #5565
• [Bug] fix ArrayNode state's TaskPhase reset by @pvditt in #5451
• Remove confusing prometheus configuration options in helm charts by @Sovietaced in #5549
• [Housekeeping] Bump Go version to 1.22 by @lowc1012 in #5032
• Fix typos by @omahs in #5571
• Respect original task definition retry strategy for single task executions by @katrogan in #5577
• Clarify the support for the Java/Scala SDK in the docs by @eapolinario in #5582
• Fix spelling issues by @nnsW3 in #5580
• Give flyte binary cluster role permission to create service accounts by @shreyas44 in #5579
• Simplify single task retry strategy check by @eapolinario in #5584
• Fix failures in generate_helm CI check by @eapolinario in #5587
• Update GPU docs by @davidmirror-ops in #5515
• Update azblob 1.1.0 -> 1.4.0 / azcore 1.7.2 -> 1.13.0 by @ddl-ebrown in #5590
• Bump github.com/go-jose/go-jose/v3 from 3.0.0 to 3.0.3 in /flyteadmin by @dependabot in #5591
• add execution mode to ArrayNode proto by @pvditt in #5512
• Fix incorrect YAML for unpartitoned GPU by @davidmirror-ops in #5595
• Another YAML fix by @davidmirror-ops in #5596
• DOC-431 Document pyflyte option --overwrite-cache by @ppiegaze in #5567
• Upgrade docker dependency to address vulnerability by @katrogan in #5614
• Support offloading workflow CRD inputs by @katrogan in #5609
• [flyteadmin] Refactor panic recovery into middleware by @Sovietaced in #5546
• Snowflake agent Doc by @Future-Outlier in #5620
• [flytepropeller][compiler] Error Handling when Type is not found by @Future-Outlier in #5612
• Fix nil pointer when task plugin load returns error by @Sovietaced in #5622
• Log stack trace when refresh cache sync recovers from panic by @Sovietaced in #5623
• [Doc] Fix snowflake agent secret documentation error by @Future-Outlier in #5626
• [Doc] Explain how Agent Secret Works by @Future-Outlier in #5625
• Fix typo in execution manager by @ddl-rliu in #5619
• Amend Admin to use grpc message size by @wild-endeavor in #5628
• [Docs] document the process of setting ttl for a ray cluster by @pingsutw in #5636
• Add CustomHeaderMatcher to pass additional headers by @andrewwdye in #5563
• Turn flyteidl and flytectl releases into manual gh workflows by @eapolinario in #5635
• docs: fix typo by @cratiu222 in #5643
• Use enable_deck=True in docs by @thomasjpfan in #5645
• Fix flyteidl release checkout all tags by @eapolinario in #5646
• Install pyarrow in sandbox functional tests by @eapolinario in #5647
• docs: add documentation for configuring notifications in GCP by @desihsu in #5545
• Correct "sucessfile" to "successfile" by @shengyu7697 in #5652
• Fix ordering for custom template values in cluster resource controller by @katrogan in #5648
• Don't error when attempting to trigger schedules for inactive projects by @katrogan in #5649
• Update Flyte components - v1.13.1-rc0 by @flyte-bot in #5656
• Add offloaded path to literal by @katrogan in #5660
• Improve error messaging for invalid arguments by @pingsutw in #5658
• DOC-462 Update "Try Flyte in the browser" text by @neverett in #5654
• DOC-533 Remove outdated duplicate notification config content by @neverett in #5672
• Validate labels before creating flyte CRD by @pingsutw in #5671
• Add FLYTE_INTERNAL_POD_NAME environment variable that holds the pod name by @bgedik in #5616
• Upstream Using InMemory token cache for admin clientset in propeller by @pvditt in #5621
• [Bug] Update resource failures w/ Finalizers set (#423) by @pvditt in #5673
• [BUG] array node eventing bump version by @pvditt in #5680
• Add custominfo to agents by @ddl-rliu in #5604
• [BUG] use deep copy of bit arrays when getting array node state by @pvditt in #5681
• More concise definition of launchplan by @eapolinario in #5682
• Auth/prevent lookup per call by @wild-endeavor in #5686
• Update Flyte components - v1.13.1-rc1 by @flyte-bot in #5691
• [flytectl] DataConfig missing from TaskSpec by @wild-endeavor in #5692

New Contributors

• @omahs made their first contribution in #5571
• @nnsW3 made their first contribution in #5580
• @shreyas44 made their first contribution in #5579
• @cratiu222 made their first contribution in #5643
• @desihsu made their first contribution in #5545
• @shengyu7697 made their first contribution in #5652
• @bgedik made their first contribution in #5616
  Full Changelog: v1.13.0...v1.13.1

Flyte v1.13.1-rc1 milestone release

Flyte v1.13.0-rc1 Release Notes

What's Changed

• Add CustomHeaderMatcher to pass additional headers by @andrewwdye in #5563
• Turn flyteidl and flytectl releases into manual gh workflows by @eapolinario in #5635
• docs: fix typo by @cratiu222 in #5643
• Use enable_deck=True in docs by @thomasjpfan in #5645
• Fix flyteidl release checkout all tags by @eapolinario in #5646
• Install pyarrow in sandbox functional tests by @eapolinario in #5647
• docs: add documentation for configuring notifications in GCP by @desihsu in #5545
• Correct "sucessfile" to "successfile" by @shengyu7697 in #5652
• Fix ordering for custom template values in cluster resource controller by @katrogan in #5648
• Don't error when attempting to trigger schedules for inactive projects by @katrogan in #5649
• Update Flyte components - v1.13.1-rc0 by @flyte-bot in #5656
• Add offloaded path to literal by @katrogan in #5660
• Improve error messaging for invalid arguments by @pingsutw in #5658
• DOC-462 Update "Try Flyte in the browser" text by @neverett in #5654
• DOC-533 Remove outdated duplicate notification config content by @neverett in #5672
• Validate labels before creating flyte CRD by @pingsutw in #5671
• Add FLYTE_INTERNAL_POD_NAME environment variable that holds the pod name by @bgedik in #5616
• Upstream Using InMemory token cache for admin clientset in propeller by @pvditt in #5621
• [Bug] Update resource failures w/ Finalizers set (#423) by @pvditt in #5673
• [BUG] array node eventing bump version by @pvditt in #5680
• Add custominfo to agents by @ddl-rliu in #5604
• [BUG] use deep copy of bit arrays when getting array node state by @pvditt in #5681
• More concise definition of launchplan by @eapolinario in #5682
• Auth/prevent lookup per call by @wild-endeavor in #5686

New Contributors

• @cratiu222 made their first contribution in #5643
• @desihsu made their first contribution in #5545
• @shengyu7697 made their first contribution in #5652
• @bgedik made their first contribution in #5616

Full Changelog: flytectl/v0.9.1...v1.13.1-rc1

Flyte v1.13.1-rc0 milestone release

Flyte v1.13.1-rc0 Release Notes

Changelog

• chore: update runllm widget configuration by @agiron123 in #5530
• Use jsonpb AllowUnknownFields everywhere by @andrewwdye in #5521
• [flyteadmin] Use WithContext in all DB calls by @Sovietaced in #5538
• add new project state: SYSTEM_ARCHIVED by @troychiu in #5544
• Pryce/doc 434 clarify how code is pushed into a given image during pyflyte by @pryce-turner in #5548
• Increase more memory limits in flyteagent by @Future-Outlier in #5550
• Updated map task information to indicate array node is now the default and optional return type by @pryce-turner in #5561
• reverted mockery-v2 on ExecutionContext by @hamersaw in #5562
• Fix issues with helm chart release process by @Sovietaced in #5560
• Add FlyteDirectory to file_types.rst template by @ppiegaze in #5564
• Fully populate Abort task event fields by @va6996 in #5551
• [fix] Add blob typechecker by @ddl-rliu in #5519
• Refactor echo plugin by @pingsutw in #5565
• [Bug] fix ArrayNode state's TaskPhase reset by @pvditt in #5451
• Remove confusing prometheus configuration options in helm charts by @Sovietaced in #5549
• [Housekeeping] Bump Go version to 1.22 by @lowc1012 in #5032
• Fix typos by @omahs in #5571
• Respect original task definition retry strategy for single task executions by @katrogan in #5577
• Clarify the support for the Java/Scala SDK in the docs by @eapolinario in #5582
• Fix spelling issues by @nnsW3 in #5580
• Give flyte binary cluster role permission to create service accounts by @shreyas44 in #5579
• Simplify single task retry strategy check by @eapolinario in #5584
• Fix failures in generate_helm CI check by @eapolinario in #5587
• Update GPU docs by @davidmirror-ops in #5515
• Update azblob 1.1.0 -> 1.4.0 / azcore 1.7.2 -> 1.13.0 by @ddl-ebrown in #5590
• Bump github.com/go-jose/go-jose/v3 from 3.0.0 to 3.0.3 in /flyteadmin by @dependabot in #5591
• add execution mode to ArrayNode proto by @pvditt in #5512
• Fix incorrect YAML for unpartitoned GPU by @davidmirror-ops in #5595
• Another YAML fix by @davidmirror-ops in #5596
• DOC-431 Document pyflyte option --overwrite-cache by @ppiegaze in #5567
• Upgrade docker dependency to address vulnerability by @katrogan in #5614
• Support offloading workflow CRD inputs by @katrogan in #5609
• [flyteadmin] Refactor panic recovery into middleware by @Sovietaced in #5546
• Snowflake agent Doc by @Future-Outlier in #5620
• [flytepropeller][compiler] Error Handling when Type is not found by @Future-Outlier in #5612
• Fix nil pointer when task plugin load returns error by @Sovietaced in #5622
• Log stack trace when refresh cache sync recovers from panic by @Sovietaced in #5623
• [Doc] Fix snowflake agent secret documentation error by @Future-Outlier in #5626
• [Doc] Explain how Agent Secret Works by @Future-Outlier in #5625
• Fix typo in execution manager by @ddl-rliu in #5619
• Amend Admin to use grpc message size by @wild-endeavor in #5628
• [Docs] document the process of setting ttl for a ray cluster by @pingsutw in #5636
• Add CustomHeaderMatcher to pass additional headers by @andrewwdye in #5563
• Turn flyteidl and flytectl releases into manual gh workflows by @eapolinario in #5635
• docs: fix typo by @cratiu222 in #5643
• Use enable_deck=True in docs by @thomasjpfan in #5645
• Fix flyteidl release checkout all tags by @eapolinario in #5646
• Install pyarrow in sandbox functional tests by @eapolinario in #5647
• docs: add documentation for configuring notifications in GCP by @desihsu in #5545
• Correct "sucessfile" to "successfile" by @shengyu7697 in #5652
• Fix ordering for custom template values in cluster resource controller by @katrogan in #5648
• Don't error when attempting to trigger schedules for inactive projects by @katrogan in #5649

New Contributors

• @omahs made their first contribution in #5571
• @nnsW3 made their first contribution in #5580
• @shreyas44 made their first contribution in #5579
• @cratiu222 made their first contribution in #5643
• @desihsu made their first contribution in #5545
• @shengyu7697 made their first contribution in #5652

Full Changelog: v1.13.0...v1.13.1-rc0

flytectl/v0.9.1

Changelog

• 2b0e8ef Document how to release flytectl (#5537)
• 71779b3 Fix typos (#5571)
• 6292dec Update azblob 1.1.0 -> 1.4.0 / azcore 1.7.2 -> 1.13.0 (#5590)
• 01ecd0a Upgrade docker dependency to address vulnerability (#5614)
• e13bfe3 Use jsonpb AllowUnknownFields everywhere (#5521)
• 02a0d2e [Housekeeping] Bump Go version to 1.22 (#5032)

Flyte v1.13.0 milestone release

Flyte 1.13.0 Release Notes

Flyte 1.13.0 brings a host of new features, optimizations, and fixes, enhancing the platform's functionality and user experience. This release also welcomes several new contributors to the Flyte community. Below are the highlights of this release.

🚀 New Features & Improvements

1. Features

   • Watch agent metadata service
   • Allow controlling in which task phases log links are shown
   • Add executionClusterLabel

2. Helm Charts & Manifests

   • Grafana dashboard updates for better monitoring and observability.
   • Core helm chart updates for propeller configuration of agent service.
   • Helm chart updates related to Prometheus, Webhook HPA, and Flyteconsole probes.

3. Documentation Improvements

   • Added examples and guides, including a guide for setting up the OpenAI batch agent backend and updated Optimization Performance docs.
   • Fix doc link to testing agent on local cluster and replace Azure AD OIDC URL.
   • Clarify networking configurations between data plane propeller and control plane data catalog in multi-cluster deployments.

4. Performance & Bug Fixes

   • Fix cache and token management issues, including addressing auto-refresh cache race conditions and improving token refresh mechanisms.
   • Fix SQL insert operations to handle NULL values appropriately.
   • Keep EnvFrom from pod template
   • Miscellaneous fixes for pod templates, Kubeflow webhook errors, and logging configurations.

5. Flytectl

   • Add prefetch functionality for paginator
   • Fix upgrade and version commands

🔧 Housekeeping & Deprecations

1. Miscellaneous
   • Add OTLP and sampling to otelutils
   • Refactor distributed job using common ReplicaSpec
   • Fix typos using codespell CI job.
   • Remove obsolete Flyte config files.
   • Upgraded various dependencies, including Golang, Docker, and others.
   • Move to upstream mockery, which simplifies the migration to mockery-v2
   • Key-value execution tags

🆕 New Contributors

• @zychen5186 for adding prefetch functionality for paginator.
• @EraYaN for replacing Azure AD OIDC URL.
• @Dlougach for making BaseURL insensitive to trailing slashes for metadata endpoint redirect.
• @flixr for honoring redoc.enabled=false in charts.
• @trevormcguire for including group in apiVersion in plugin_collector.
• @va6996 for inheriting execution cluster label from source execution.
• @mhotan for Helm chart updates related to Prometheus, Webhook HPA, and Flyteconsole probes.
• @eltociear for updating token_source.go.

Full Changelog

flytectl/v0.9.0

Changelog

• f87a049 Add API to get domain (#5443)
• e1d9c5c Add OTLP and sampling to otelutils (#5504)
• ba88e82 Add lint-fix make target, add gci to flytectl, remove goimports make target (#5484)
• 791471c Fix: Make 'flytectl compile' consider launchplans used within workflows (#5463)
• f54b74e Fix: replace with in OSS docs (#5501)
• ed40a94 Remove doc building from flytectl (#5494)
• c97a0e9 Remove github.ref from group (#5518)
• c512571 reimplement functions in go-mouff-update, use ghreposervice (#5470)

Flyte v1.13.0-rc0 milestone release

Flyte v1.13.0-rc0 Release Notes

Major Features and Improvements

• Key-value execution tags: Enhanced execution tagging capabilities. (#5453)
• Watch agent metadata service: Improved monitoring of agent metadata. (#5017)
• Distributed job refactoring: Refactored distributed job using common ReplicaSpec. (#5355)
• Execution cluster label inheritance: Executions now inherit cluster labels from source executions. (#5431)
• Domain API: Added new API to retrieve domain information. (#5443)
• Execution environment versioning: Added version to ExecutionEnv proto message. (#5506)
• OTLP and sampling in otelutils: Enhanced observability with OpenTelemetry Protocol (OTLP) support. (#5504)

Notable Changes

• Removed mmcloud plugin. (#5468)
• Updated k3s version to 1.29.0. (#5475)
• Improved auth flow to support custom base URLs in deployments. (#5192)
• Added flyteconsole URL to FlyteWorkflow CRD. (#5449)
• Enhanced 'flytectl compile' to consider launch plans within workflows. (#5463)
• Introduced control over task phases for log link display. (#4726)
• Improved Helm chart configurations for Prometheus, Webhook HPA, and Flyteconsole probes. (#5508)

Bug Fixes and Optimizations

• Fixed flaky auto_refresh_test. (#5438)
• Corrected NULL to empty string in SQL insert for migrations. (#5482)
• Resolved issues with broken mermaid diagrams in documentation. (#5498)
• Fixed Ray plugin to use default service account if not set in task metadata. (#5499)

Documentation and Usability

• Updated community page. (#5496)
• Improved documentation on logging link lifetime configuration. (#5503)
• Replaced 'uctl' with 'flytectl' in OSS docs for consistency. (#5501)

New Contributors

• @Dlougach, @flixr, @trevormcguire, @va6996, @mhotan, and @eltociear made their first contributions to the project.

For a complete list of changes, please refer to the full changelog.