[chore] add UP rule to ruff and upgrade to Python 3.9 syntax

[danielgafni]

Summary & Motivation

Hello, I really like running ruff on Dagster.

This PR adds the pyupgrade rule which automatically updates Python syntax & usage to the minimal supported Python version (3.9 in our case).

The only exception is dagster-buildkite since it's still running on 3.8 (@gibsondan how hard would it be to bump Python in BuildKite?).

See pyproject.toml diff:

diff --git a/pyproject.toml b/pyproject.toml
index 7fcaa156fe..0ff8cb69a4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -98,7 +98,7 @@ filterwarnings = [
 
 [tool.ruff]
 
-target-version = "py38"
+target-version = "py39"
 
 # *.py, *.ipy are included by default
 extend-include = ["*.ipynb"]
@@ -116,7 +116,7 @@ extend-exclude = [
 line-length = 100
 
 # Fail if Ruff is not running this version.
-required-version = "0.5.5"
+required-version = "0.7.2"
 
 [tool.ruff.lint]
 
@@ -181,6 +181,13 @@ ignore = [
   # (assorted perf rules) We have a lot of violations, enable when autofix is available
   "PERF401", # (manual-list-comprehension)
   "PERF402", # (manual-list-copy)
+
+  # (assorted pyupgrade rules) We have a lot of violations, enable when autofix is available
+  "UP035", # (deprecated-import)
+
+  # (assorted import rules) We have a lot of violations, enable when autofix is available
+  "F401", # (unused-import)
+
 ]
 
 # By default, ruff only uses all "E" (pycodestyle) and "F" (pyflakes) rules.
@@ -248,6 +255,9 @@ select = [
 
   # (invalid escape sequence) flag errant backslashes
   "W605",
+
+  # (pyupgrade) automatically upgrade Python syntax to newer versions
+  "UP",
 ]
 
 [tool.ruff.lint.flake8-builtins]

All other changes are automated.

Initial UP violations:

❯ ruff check --statistics
3561	UP006	[*] non-pep585-annotation
2163	UP035	[ ] deprecated-import
 457	UP008	[*] super-call-with-parameters
 162	UP015	[*] redundant-open-modes
  92	UP031	[*] printf-string-formatting
  39	UP027	[*] unpacked-list-comprehension
  39	UP034	[*] extraneous-parentheses
  22	UP037	[*] quoted-annotation
  15	UP028	[*] yield-in-for-loop
   7	UP024	[*] os-error-alias
   7	UP033	[*] lru-cache-with-maxsize-none
   7	UP036	[*] outdated-version-block
   4	UP009	[*] utf8-encoding-declaration
   4	UP013	[*] convert-typed-dict-functional-to-class
   2	UP010	[*] unnecessary-future-import
   1	UP012	[*] unnecessary-encode-utf8
   1	UP018	[*] native-literals
[*] fixable with `ruff check --fix`

Violations after ruff check --fix (before --unsafe-fixes):

❯ ruff check --statistics
3562	UP006 	[*] non-pep585-annotation
1352	UP035 	[ ] deprecated-import
 457	UP008 	[*] super-call-with-parameters
  92	UP031 	[*] printf-string-formatting
  15	UP028 	[*] yield-in-for-loop
   7	UP036 	[*] outdated-version-block
   4	TCH003	[*] typing-only-standard-library-import
[*] fixable with `ruff check --fix`

And before ignoring the unfixable rules:

❯ ruff check --statistics
237	UP035	deprecated-import
 36	F401 	unused-import

[danielgafni]

• [chore] add UP rule to ruff and upgrade to Python 3.9 syntax #25703 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @danielgafni and the rest of your teammates on Graphite

[github-actions]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-fvce03v3g-elementl.vercel.app
https://11-02--chore-add-up-rule-to-ruff.dagster.dagster-docs.io

Direct link to changed pages:

[github-actions]

Deploy preview for dagster-docs-beta ready!

Preview available at https://dagster-docs-beta-owcusk788-elementl.vercel.app

Direct link to changed pages:

[cmpadden]

I think this is great, and am in favor of inclusion of pyupgrade rules.

Will wait for approval from at least one other though to ensure consensus.

[danielgafni]

Anyone wants to give another review?

[smackesey]

Left a comment on the pyproject.toml.

Also this needs an internal companion PR to keep these OSS/internal in sync.

Otherwise nice!

[danielgafni]

After switching to py39:

ruff check --statistics
3531	UP006 	[*] non-pep585-annotation
1344	UP035 	[ ] deprecated-import
   4	TCH003	[*] typing-only-standard-library-import

After running ruff with --unsafe-fixes:

❯ ruff check --statistics
237	UP035	deprecated-import
 36	F401 	unused-import

Deprecated import errors are similar to:

python_modules/libraries/dagstermill/dagstermill/manager.py:5:1: UP035 `typing.AbstractSet` is deprecated, use `collections.abc.Set` instead
  |
3 | import uuid
4 | from collections.abc import Mapping
5 | from typing import TYPE_CHECKING, AbstractSet, Any, Optional, cast
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UP035
6 |
7 | from dagster import (
  |

Unused import errors look like:

python_modules/libraries/dagster-pandera/dagster_pandera/__init__.py:4:45: F401 `typing.Type` imported but unused
|
2 | import re
3 | from collections.abc import Sequence
4 | from typing import TYPE_CHECKING, Callable, Type, Union
|                                             ^^^^ F401
5 |
6 | import dagster._check as check
|
= help: Remove unused import: `typing.Type`

I've added both rules to exclude.

[schrockn]

• Can you document (or point to a document) the describes all the autofixes variants being applied here? This diff to is too large to review?

With the above, we could also then announce this to end-headsup so people know what the have to "untrain" themselves from writing.

[danielgafni]

Hey @schrockn, I added violation stats to the PR description.

For sure it's impossible to review, but the number of --unsafe-fixes is pretty low (only 2 rules - check my comment above) and since CI was happy it's probably safe to apply them.

[danielgafni]

Actually, there are some failures in BuildKite because some parts of it are still running on Python 3.8 3.7.
I hope there is a single image/entrypoint which we can bump? @gibsondan @smackesey