How to use crate_universe with non-registry crates?

[cfrantz]

I'd like to use crate_universe with some non-registry crates and I'm not quite clear on how to go about it. I'm currently using cargo-raze in my repository and I'm able to accomplish this by also using cargo-raze in my non-registry dependencies and having my main repo load the raze-generated deps of my non-registry dependencies.

I'm migrating my project (opentitan) to crate_universe and I'm not sure what is the best approach.

1. I could put my non-registry deps into a Cargo.toml with git references. This works, but is not a viable solution for me because I need to support airgapped builds and git references in the Cargo.toml file get converted to new_git_repository imports. AFAIK, bazel doesn't support git repositories for airgapped mode.
2. I could use crates_vendor rules and duplicate the structure I currently have with cargo-raze.
3. I could define crates_repository rules in each of my non-registry deps and export starlark functions that load and configure the deps of my non-registry repos and then load those from my root workspace.
4. Something else?

I'd really like to do (1), but I need to download my deps as http_archives. It would be convenient if either cargo or a crate.annotation provided a url override so I could specify an archive to download.

I think (2) is probably the next-best approach. I had hoped to avoid committing the generated BUILD files into my repo. I'm already committing the BUILD files with cargo-raze and I feel that it would be an improvement to not have to do it.

I think (3) is too complex.

Are there other possibilities that would allow crate_universe to unify dependency management between registry-deps and non-registry deps (and perhaps cargo-free deps)?

[cfrantz]

I've run into what I think is a rather nasty footgun using approach (2) above. I've written a minimal demonstration:
https://github.com/cfrantz/bugs/tree/main/minspec

I'm using the nightly feature min_specialization. In the error scenario, I have a library crate which depends on serde and has uses min_specialization to create a default implementation for all T where T: serde::Serialize. Thus, my library has serde as a dependency.

In my main binary crate, I have a type Foo for which I implement the trait from my library crate and derive serde::Serialize. Thus, my main binary crate has serde as a dependency.

When I build the program, the library's instance of serde is distinct from the binary's instance of serde and the program fails to build. If I change the binary's bazel dependency on the library to use repo_mapping to supply a single source of deps (from the binary crate), then the program builds and links.

[ericmcbride]

Thanks for the swift response! I will test this out!

[ericmcbride]

Worked perfectly, thanks again!

[ericmcbride]

Sorry to keep bothering, but we are exploring different angles. Can you show me a short example of how you are doing this with crate_vendor? How are you defining that in your workspace?

[ericmcbride]

The important thing is to make sure the parent project enumerates all of the direct dependencies of the non-registry subproject so that crate_index will satisfy the subproject.

How did you ensure this I am using the local vendor mode, because I am not using remote, so i dont have a crates_repository function to load in my namespace :(

[cfrantz]

My root project is OpenTitan and the rust crates_vendor configuration is defined here and named crate_index.

My non-registry crate is serde-annotate (a crate of my own that I'm not yet ready to publish on crates.io) and its crate_index is defined here.

Unfortunately, the way I'm ensuring the parent project enumerates all of the non-registry dependencies is manually. It would be nice if rules_rust in the root project could map in other Cargo.toml files that aren't part of that project's workspace, but since rules_rust re-uses a lot of the normal cargo libraries and machinery, it can't really do things that Cargo can't do.