Implementing Zod in Python (Is it possible)

[s-cork]

Context: zod.dev

Zod is TypeScript library that does two things well:

• parsing data against schemas
• type inference of schemas

I've implemented the parsing of data part in python
But I can't decide whether the typing inference is possible or if it's just a dead end?

---

Example from typescript

import { z } from "zod";

const s = z.string();
const foo = s.parse("foo"); // "foo"
const notString = s.parse(1); // Error thrown
type S = infer<typeof s> // string

The benefit of the inference is that we can use our schemas as type information

function transformString(x: S) {
    ...
}

---

I feel like this should be possible in Python but beyond simple types like str, int, optional I get lost in the lack of features in python typing

Here's a quick implementation for string in python

T = TypeVar("T")

class ZodType(Generic[T]):
    _out: T
 

class ZodString(ZodType[str]):
    ...
    @classmethod
    def create(cls):
        return cls()

string = ZodString.create

def infer(x: ZodType[T]) -> T:
    return x._out

x = string()
z = infer(x) # typechecker understands z is a str

Optional is similar

class ZodOptional(ZodType):
    ... # implementation details
    @classmethod
    def create(cls, schema: ZodType[T]) -> ZodType[Optional[T]):
        return cls(schema)

optional = ZodOptional.create

x = optional(string())
z = infer(x) # typechecker understands z is str | None

We can even do things like Record (or Mapping I guess)

K = TypeVar("K")
V = TypeVar("V")

class ZodRecord(ZodType):
    ... # implementation details
    @classmethod
    def create(cls, keys: ZodType[K], vals: ZodType[V]) -> ZodType[dict[K,V]]):
        return cls(keys, vals)

record = ZodRecord.create

x = record(keys=string(), vals=integer())
z = infer(x) # typechecker understands it's a dict[str, int]

But then we get to things like ZodUnion and I'm lost 🤷

# pseudo code
Ts = TypeVarTuple("Ts")

class ZodUnion(ZodType):
    @classmethod
    def create(*types: *Ts) -> ZodType[Union[Ts]]:
        # But I can't unpack Ts here - and even if I could how would I get the type of the _out attribute from the inferred vars
        ...

# I want to be able to do
x = union(string(), integer(), boolean())
z = infer(x) # str | int | bool

And then there is ZodLiteral - there's no way to infer a literal value from a generic variable since there's no such thing as LiteralVar.

# psuedo code

L = LiteralVar("L")

class ZodLiteral(ZodType):
    @classmethod
    def create(value: L) -> ZodType[Literal[L]]:
        ...

x = literal("foo")
z = infer(x) # typechecker should understand this is "foo"

Am I just hitting the edge of what python typing is capable of?

The ultimate goal would be to be able to do things like

import zod as z

human_age_schema = z.ingeter().min(5).max(127) # each call just returns a ZodInteger with constraints
human_age = z.infer(human_age_schema)

def do_something_with_age(x: human_age): # typechecker infers x is an int
    x = human_age_schema.parse(x)
    ...

As well as

import zod as z

schema = z.union(z.string(), z.integer())

def foo(x: z.infer(schema)):
    x = schema.parse(x)
    ...

foo(3.14) # type checker complains

[decorator-factory]

Re ZodLiteral

this actually works with pyright using LiteralString ( playground link )

L = TypeVar("L", bound=LiteralString)

class ZodLiteral(Generic[L]):
    def __init__(self, value: L) -> None:
        self._value = value

x = ZodLiteral("foo")
reveal_type(x)  # ZodLiteral[Literal["foo"]]

infer

I don't think infer is possible. Here: pyright playground pyright is complaining about:

class ZodSchema(Generic[T]):
    def result(self) -> type[T]:
        return type  # type: ignore

class ZodLiteral(ZodSchema[L]):
    def __init__(self, value: L) -> None:
        self._value = value

x = ZodLiteral("foo")

foo_result: TypeAlias = x.result()   # Invalid expression form for type alias definition

def f(foo: foo_result) -> None:
    reveal_type(foo)

although, strangely, it does reveal foo to be Literal["foo"]

Unions

No need to use TypeVarTuple, but you will need to make T covariant, so that ZodSchema[int] is a subtype of ZodSchema[int | str] and such (playground)

class ZodSchema(Generic[T]): ...
    
class ZodInt(ZodSchema[int]): ...
    
class ZodStr(ZodSchema[str]): ...
    
class ZodList(ZodSchema[tuple[T, ...]], Generic[T]):
    def __init__(self, element: ZodSchema[T]): ...

class ZodAnyOf(ZodSchema[T]):
    def __init__(self, *options: ZodSchema[T]): ...

hmm = ZodAnyOf(ZodInt(), ZodInt(), ZodList(ZodStr()), ZodStr())
reveal_type(hmm)  # ZodAnyOf[int | tuple[str, ...] | str]

Nested records

A bigger problem you're going to run into is parsing 'records'

point = z.record({
  "x": z.integer(),
  "y": z.integer(),
})

Although I suppose you could do some descriptor magic:

class Point(z.Record):
    x = z.field(z.integer())
    y = z.field(z.integer())

P.S. playground doesn't really work on mobile, sorry

[adriangb]

I’m curious as to what the advantage of a zod like API is versus Pydantic? Pydantic V2 will be able to do non-record types, something like:

from typing import Annotated
from pydantic import Gt, validate, parse

myint = Annotated[int, Gt(0)]

assert validate(myint, 1) == 1
assert parse(myint, “1”) == 1
validate(myint, -1) # error
parse(myint, “foo”) #error

(the parse and validate functions are hypothetical but plauisible)

[decorator-factory]

IMHO: one reason is that parsing type annotations (at runtime) and analyzing them is a big pain. Every library that wants to do this also needs to do it form scratch, including e.g. figuring out what to do with generic "models".

Whereas if you derive types from values:

• you will have less type-unsafe shenanigans on the library side, it will generally be easier to understand

• the client can implement their own validators easily, because the library is be based on composition of values

• PEP 563 / PEP 649 are largely irrelevant for this approach

• it decouples the definition of a model from a way of encoding/decoding it. For example, there are many ways of encoding a "tagged union":

  • with an internal tag, like {"kind": "circle", "x": 1, "y": 2, "radius": 5} vs {"kind": "rect", "x1": 1, "y1": 2, "x2": 5, "y2": 6}
  • with an external tag, like {"shape_type": "circle", "shape_data": [circle1, circle2, ...]} vs {"shape_type": "rect", "shape_data": [rect1, rect2, ...]} or {"status": "ok", "data": T} vs {"status": "error", "data": {"msg": "not found", "intcode": 404}}
  • just trying out different possibilities until one matches

  Correct me if I'm wrong but IIRC Pydantic maps a union like Rect | Circle to the third way, so you don't have any choice here.
  Whereas a value-based approach would let you define different decoding strategies, like internally_tagged(field="kind", variants={"circle": decode_cicle, "rect": decode_rect}) and try_out(decode_circle, decode_rect). And you can make your own if the library doesn't cover your use case.

[adriangb]

Good points! I think that makes sense for scalars and homogeneous collections. I’m not so sure about record types (what does parsing return, an instance of Point?).

pydantic-core is going to be much more along the composition of values approach. The pydantic library is going to wrap it and encapsulate the runtime type introspection aspect. So I wonder if it’d be possible to build a Zod like Python API on top of that?

[decorator-factory]

I’m not so sure about record types

Yeah, that's where Python's type system is lacking (for this use case). You can't construct a Parser[{"foo": int, "bar": str}] from a {"foo": Parser[int], "bar": Parser[str]}

(maybe I'm misunderstanding what you mean)

[decorator-factory]

For an alternative strategy, the Elm language takes an even simpler approach with values (official tutorial):

import Json.Decode as Jd

type alias Person =
  { name : String
  , age : Int
  }

personDecoder : Jd.Decoder Person
personDecoder =
  Jd.map2 Person
    (Jd.field "name" Jd.string)
    (Jd.field "age" Jd.int)

(Person can be used as a two argument function, taking a string and and integer)

In other words, it's all built from ordinary functions and simple combinators. (there's also a third-party library that makes it more reasonable to work with larger records)

I also made a similar thing in Python: https://github.com/decorator-factory/py-humbleparser (though it's definitely not "production ready")

[decorator-factory]

Though runtime introspection does have some benefits. For one, it can help you enforce conventions, so that you don't map class into klass in one place, into cls in another place and class_ in a third place -- instead you define a single name-mapping strategy.

[birgersp]

[adriangb]

Does Pydantic not serve that role for you?

[birgersp]

Actually yes it does. Thanks!