'workspace' endgame

[majidaldo]

what's the vision for creating 'local dependencies'? would it be possible to create project deps from workspace dep graphs ?

here's where i'm coming from https://github.com/pnnl/hydraconda/tree/master/project. i've used conda and the ability to compose ugly jinja templated yaml files to create simultaneous execution environments mainly for data science scenarios where i don't care about packaging too much. but it would be nice to converge on some of the ideas expressed in rye.

i've contemplated switching to nix for my programming-language agnostic needs but it's difficult to use!

[bluss]

I would like to see local dependencies work with relative paths primarily. Both of these:

A. depencency = "./subdirectory"
B. dependency = "../sibling_directory"

Because, sibling directories is one way to handle monorepo style projects. Especially those where lots of disparate projects all live in the same repo.

Current workspaces are a way to have local dependencies without declaring them, so they work like an alternative to (A) - all packages in a workspace are just installed together, so they can be used together without even declaring the dependency. Ideally it should be declared, shouldn't it? Workspaces can't support the scenario (B) - not the most general case of it, so that makes it seem like relative path dependencies are more important than workspaces(?)

The ideal I see is something I learned from Rust (too), you declare a dependency like foo = { version = "1", path = "./foo" } and it's a path dependency in-tree, but once turned into a wheel it's just a named dependency (dependency on "foo"), which will work fine if both packages are built and installed together.

[cnpryer]

I would prefer local information instead of central and global, so that each subpart of the repository is understandable mostly as an independent unit

When you say central and global, do you mean global to the workspace?

[bluss]

To just expand on it, hope it clarifies the point:

• local information: subpart A declares - in its pyproject - it depends on subpart B.
• global information (variant 1): A and B can use each other because they are in the same workspace. They don't declare any dependency on each other.
• global information (variant 2): A depends on B but it's written in a makefile in the repository root

local is good because it makes it much easier to lift one subtree of the whole project from one repository to another. And refactorability should be good.

It's not strictly necessary to use file paths, you can just declare A depends on B by other means, too.

[cnpryer]

Ignoring implementation details, why wouldn't local information also work in a workspace that supports a global information variant 2 style option as well? Just like in rust with cargo you can declare local information and defer to workspace information if needed.

I think I'm just struggling to understand why global vs local is important for project composability (implementation details?). My thought process is to first think about what experience we want, and then think about the implementation required for that experience (if any).

[cnpryer]

I get that the support for this in the ecosystem isn't there, but assuming we want to explore new options is there anything else that keeps this off the table?

[bluss]

cargo always uses local information (a crate must define its own dependencies, without that it doesn't compile). This point doesn't need to be exaggerated, I think we're just trying to clarify what we actually mean.

I want to avoid dependency relations that are implicit or never defined. If I want to copy one library from a repo and its dep information is not separately available, that's friction and the kind of problem with composability I mean. If I want to add a new library and have to manually merge together two different build system files, that limits composability. 🙂

[majidaldo]

it should resemble a build system. i don't think using the filesystem to encode dependency is flexible enough.

I would prefer local information instead of central and global, so that each subpart of the repository is understandable mostly as an independent unit, so anything that fits with this. Do you have something in mind with resembling a build system? How do you specify dependencies in that case? The basic unit is a pyproject.toml here, is it? (The unit that can have dependencies and be depended on.)

i very much appreciate this. but the filesystem encoded dependency is not flexible. such a dependency graph should be generated. (what if something else in the 'root' depends on something you have as a deep 'child'?)

see my project above for how i encoded deps via yamls files... makefile style.

[bluss]

Maybe a flatter structure is better, in that case? I'm not sure what's wrong with just giving the path to where the dependency is, even if it's nested. For the yaml files, I don't know what to look at, I'm sorry, they are not too easy to read without introduction to it 🙂 I see a list of dependencies but that's on pip packages, not one part of the repo depending on a lib in another part of the repo, which I'm looking for.

[majidaldo]

Maybe a flatter structure is better, in that case? I'm not sure what's wrong with just giving the path to where the dependency is, even if it's nested. For the yaml files, I don't know what to look at, I'm sorry, they are not too easy to read without introduction to it 🙂 I see a list of dependencies but that's on pip packages, not one part of the repo depending on a lib in another part of the repo, which I'm looking for.

sorry about that. the 'workspace' deps are the 'includes' keys.

[majidaldo]

Maybe a flatter structure is better, in that case? I'm not sure what's wrong with just giving the path to where the dependency is, even if it's nested. For the yaml files, I don't know what to look at, I'm sorry, they are not too easy to read without introduction to it 🙂 I see a list of dependencies but that's on pip packages, not one part of the repo depending on a lib in another part of the repo, which I'm looking for.

sorry about that. the 'workspace' deps are the 'includes' keys.
..oh and since those under 'dependencies' are conda pkgs, they are not necessarily (python) pip.