Feedback using Nugetizer for the first time

[jzabroski]

I come in ☮️ , but have some constructive criticism that may feel like ❤️ 🗡️ . It's more that I am laying out my initial experiences with Nugetizer, and hoping that this might educate on where the documentation is lacking for someone with my background.

@kzu has mentioned I should give Nugetizer a try several times in an NuGet issue NuGet/Home#3891 that expresses some of my pain points with using SDK Pack. He wrote a blog post, NuGetizer as an alternative to .NET SDK Pack, that is thus far probably the best explanation of what NuGetizer can do. I tried to cobble together information from the official nugetizer readme.md and @kzu's blog post.

Here are some observations.

1. It is really hard to Google search to learn about nugetizer. I have tried several different searches and I get almost nothing interesting.

2. I also don't understand why some Google search results for "nugetizer", like the second search result references a different GitHub project than this one (NuGet/NuGet.Build.Packaging), and the fifth search result, is a blog post that references that GitHub project as well. It seems like the other project is archived, but not actually archived, which is confusing as hell.

3. The official nugetizer readme.md says, under Why, that the motivation was to provide an alternative clean and clear design but the linked explanation contains self-defining vocabulary words that I don't know the meaning of, like "bait-and-switch packages". I can loosely guess that "bait-and-switch packages" refer to reference assemblies, but I find this confusing, because... why not just call them reference assemblies? Examples like this elongate the learning curve to "mastering nugetizer".

4. @kzu claims it is easy to debug Nugetizer projects since it's just MSBuild, but that seems to presume the user knows how to debug MSBuild - this feels like a 0.01%'er type thing. I vaguely recall Visual Studio 2015 adding support for stepping through MSBuild XML logic, but I have no idea how to actually do that. It would add a lot of value to the statement that it's easy to debug, by maybe recording some YouTube videos or Twitch sessions demonstrating common debugging workflows with Nugetizer.

5. It seems like rather than doing nugetize --debug, it would be a lot more efficient to just add verbose output to the nugetizer command line tool instead. For example, if PackagePath is wrong, it would be nice if the there was a clue as to which target raised the error validation, rather than just get "D:\Users\John.Zabroski\.nuget\packages\nugetizer\0.7.0\build\NuGetizer.Shared.targets(277,5): error : Target path 'D:\Users\John.Zabroski\.nuget\packages\myvendor.odata\1.0.0.2385' contains invalid characters. [D:\Users\John.Zabroski\source\infrastructure\master\Prototype\my.csproj]"

   1. In particular, the nugetizer readme.md claims, under Inner Develop, there is a concrete set of steps nugetizer does - why isn't there a verbose mode that outlines these steps happening, so we get a sense for where nugetizer is failing?

6. I don't really understand the relationship between Nugetizer and dotnet-nugetize. I don't particularly understand why I'm adding a PackageReference to Nugetizer and then calling nugetize my.csproj, and I don't really understand the output of nugetize (although I am making a tiny bit of progress there, the initial output was really confusing and didn't make a lot of sense to me).

7. I am struggling to understand some of the examples in @kzu's blog post, for example:

   <!-- Add package dependencies directly too -->
   <PackageFile Include="Newtonsoft.Json" Version="13.0.1" PackFolder="Dependency" />

   Initially, I thought this would automatically download the Newtonsoft.Json command based on the description, but all it is really doing is is adding anything with PackFolder="Dependency" to your nuspec's <dependencies>. I don't understand what's special about this, since it seems like a verbose way to accomplish what SDK Path does. I think the more accurate description of this is that you can "Add ambient package dependencies directly to your nuspec, even if your csproj doesn't include them as a PackageReference." In other words, an ambient package is one that isn't directly referenced by your code but should be included in the final output (e.g., a plug-in or its inverse, the shell that hosts the plug-in). Agree? I don't know, just coming here as a rookie trying to figure things out.

   After studying this more closely, I also think the whole point of PackageFile PackFolder="Dependency" is to increase the speed of building a nuspec file. That seems kind of strange to me, since I don't ordinarily spend a lot of time thinking about my packing process, and it means that I no longer get the benefits of PackageReference restoring the package locally? It seems like it would be a lot cleaner long-term if we stuck with PackageReference and supported an attribute on PackageReference Restore="false" to opt it out of the nuget restore process? Am I thinking about this the wrong way?

8. Task-wise, I am struggling with how I can "transclude" one Nuget package with a csproj in my solution. I want the NuGet package to get dumped into the bin folder and for that NuGet package to be treated as the true host for my csproj - a sort of "Remora" design pattern where the csproj comes along for the ride with the referenced NuGet package. In my case, I view the csproj as the plug-in, and the NuGet package as the host - not to get too philosophical.

9. Coding-wise, I am struggling with how to know if I am doing MSBuild best practices, as this veers pretty far from the mainstream of how I use MSBuild on a daily basis. I feel like the prototype I have created is a giant mess. For example, when I run dotnet build, the bin directory is different from when I run dotnet nugetize. The contents really shouldn't change. But I don't really have a deep understanding of where things are running amiss.

10. @kzu's blog post mentions a couple of tricks, like using the Microsoft.Build.NoTargets SDK or the Microsoft.Build.Traversal, but I don't quite understand what Microsoft.Build.NoTargets does - it seems like dotnet build still works if I use it instead of Microsoft.Project.SDK - and I don't quite understand what the utility is of learning it (I don't see a C# sub-reddit or twitter hashtag dedicated to these esoteric scenarios, so I imagine the ramp up time to learn them is non-trivial).

11. Running dotnet nugetize on a solution produces an error if at least one project doesn't PackageReference Nugetizer - this seems odd, in that I don't see why my csproj's dedicated to nuget packaging need to have other csproj's know about nugetizer. Seems to violate the open-closed principle.

12. When using PackageFile and looking at Dependencies under the csproj in VisualStudio Solution Explorer, it reports a warning icon on the csproj and a red x document icon on the PackageFile dependency. I'm not really sure what these mean, but it's definitely distracting and shows that "drop-in replacement" of Nugetizer for SDK Pack is not really quite there yet.

After all this feedback and write-up, I kind of see why people complain about wanting SDK Pack to just work. This took me a full day just to wrap my head around, and I don't think it was a productive use of my time :( - at least not yet - but hopefully it is helpful in making this project more accessible to others in the future.

[kzu]

Thanks for taking the time to write this! I'll try to address the issues.

1. I only somewhat recently forked and took ownership of the future of nugetizer. Not much I can do to teach google more about it, but I'm open to suggestions!

2. Sent a PR: Add note on project not being maintained anymore here. NuGet/NuGet.Build.Packaging#174. I'm no longer at Microsoft, so I don't have any say on what happens to that repo (archiving, merging PRs or otherwise). This is one main reason why I forked and took ownership of my own copy. I was the original developer on the NuGet repo side too, FWIW.

3. The link is provided for historical purposes. And bait-and-switch packages are also explained elsewhere in quite detail (i.e. https://blog.anaisbetts.org/the-bait-and-switch-pcl-trick/ and others). Knowing what it is (or even reading the original wiki) is not a requirement for understanding it. You don't typically go to the c# language design repo/docs/wiki to learn about new features either, but it's still worth it to access it if you want to understand more deeply some motivation or design choice that seems weird.

4. I don't recall ever saying you need to debug MSBuild itself. Since it's all just plain MSBuild, a simple -bl switch will generate a super helpful binlog where you can see all that's been collected, from where, how, by which targets, with which metadata and so on. In that sense, nugetizer is fully transparent, while SDK Pack is (mostly) opaque since most stuff happens inside a single task that receives a gazillion parameters and targets/items/props haven't been really thought out from the beginning for direct consumption (hence weird and ultra-long names, for example).

5. nugetize --debug isn't documented and isn't supposed to be something you need to use. Why would you need to debug the tasks? That's something I do to debug the tasks while developing nugetizer itself. It's not for regular consumption. I created an issue to track removing that. When packing fails, you can just get a binlog and inspect everything. This is normal MSBuild-based troubleshooting I'd say.

6. I don't know really how to explain in more simple terms that dotnet-nugetize is just a tool for quickly looking at what would be packed: https://github.com/devlooped/nugetizer#dotnet-nugetize. Feel free to send a PR with suggestions for more clear wording. As for the output, it's mostly a dump of what is in the .nuspec. Also open for more details as to what is confusing in particular.

7. That example is under a section on "First-class Custom Packaging". If you're not doing fully custom packaging with a packaging project (as opposed to a normal csproj), then you don't need to even bother with that. The point of the blog post is that even if you had a dependency on a package that for whatever reason doesn't restore properly or get packed properly in SDK Pack, you can add it directly and avoid all nuget behaviors altogether. You don't have to use that if just adding a PackageReference works and packs the way you need. It's not about speed, it's about flexibility. If you had a project targeting net45 and needed to add a dependency on a (say) some package that brings some tools that target net5.0, for example, NuGet would block that because they are incompatible. But if you're bringing it in just because you have some build/tasks dependency on it, you don't really care. That is almost impossible to do with SDK Pack. WRT adding a "Restore=false" on PackageReference, good luck getting nuget itself to recognize that and act accordingly. Just like the linked issue on nuget org, that will likely never happen, or take a decade.

8. No easy solution for that. Nugetizer does not solve that and likely never will, just like the SDK pack offers little (no?) help either. It's a corner use case, IMHO. (I've never needed that, and I've authored hundreds of packages).

9. There's a reason a nuget package isn't just a zip of the bin folder. Neither in SDK Pack, nor nugetizer. When you do dotnet pack, you'd get something different too, depending on the project, contents, dependencies, copylocal and what-not. So, things to pack != things built/copied to output directory. Pretty much by design.

10. Using other SDKs is useful for creating a packaging project. Not a code project. Hence no C# involved, no compiler, no nothing. You can read more about those SDKs and why they are useful in their own repository.

11. See https://github.com/devlooped/nugetizer#projectreference. That explicit protocol is what drives the dotnet nugetize tool. It's the way P2P references can expose what they contribute to a package to a referencing project. That's basically the original Virtual Packages design. Without it, you're just guessing what should be packaged from a referenced project, and that guessing is precisely what causes so much pain with SDK Pack, because there's no way to extend/modify/replace the logic.

12. That's the MS project system support (or lack thereof) for package references in the other SDKs. Again, something entirely unrelated to nugetizer itself. Also, the alternative is a .nuspec that has absolutely ZERO support in the IDE. So having little warning icons you can ignore is a vast improvement if you ask me ;)

Having used nugetizer to pack extremely non-trivial projects I can tell you there is no way the built-in SDK pack will ever accomodate advanced packing scenarios without requiring serious investment in understanding deeply what you want to pack, how to do it and with significant flexibility in the underlying targets/tasks. So, the "complain about wanting SDK Pack to just work" is wishful thinking. I've stopped using it for any serious packaging needs. Now, for the happy-path and really simple library, it's great and I do use it some times (i.e. for dotnet tools, it's great!).

As I mentioned on the nuget issue: it was productive if you learned more about MSBuild 👍 .

[jzabroski]

Got it. Just an observation, but I got the sense that you were encouraging the .NET Nuget team to replace SDK Pack with Nugetizer. Hence why I brought up these incompatibilities. I'm not sure if all these loose threads of discussion are making sense to you, but I have frequently observed that the .NET teams make the same mistake over and over again: Thinking one step is completely independent of others.

As an example of sequential coupling, before framework references were introduced, it was thought to be sufficient to just put that info into the project.assets.json and be done with it. After all, project.assets.json is simply what MSBuild thinks your build dependencies are. However, they then realized in order to properly restore packages, that information needed to be embedded in the nupkg nuspec file, and so the framework-related stuff was born.

The various MSBuild properties I've raised in this discussion are all part of that tightly coupled system. And it's hard for people to see/feel the tight coupling because everything in MSBuild is global and allows for invasive software composition through weaving BeforeTargets/AfterTargets/InitialTargets and creating safe macros through the <MSBuild /> Task and Tasks for creating new identifiers. There's no clear system boundaries. However, I started looking through why a small Nuget team is struggling to close out 900 issues, because I have been on a project with a 1,000 open issues that I got down to 50 open issues mostly cosmetic in nature before. And what I saw was a lot of unclear system boundaries that make it hard to reason about what a design should be and what direction it should go in.

Just my 2 cents.

[jzabroski]

I think I've said this before, but it bears reinforcing: the reason SDK pack is so limiting is precisely because it does not offer a clean contract (a target, with specific output items + metadata that lay out the package contents) that you can extend, and that works across project references. Nugetizer is free from that limitation, and the trade off is that you must reference nugetizer to opt-in to this better, more deterministic, extensible and flexible world of packaging.

@kzu You probably have said this before, but I didn't have a deep, visceral feel for what you meant exactly. I believe I just re-stated what you said in paragraph 3 in my previous reply. I think we're on the same page here?

I think where my disconnect was I thought you wanted Nugetizer to become SDK Pack, which would mean people would no longer have to <PackageReference Include="Nugetizer" /> any more, because it would be packed into the <Project Sdk="Microsoft.NET.Sdk" > targets for Pack.

Just thinking out loud. An analogy (I tend to hate analogies but go with me here) is that you used to need to include a ProjectCapability element to denote that a csproj was a test project so that the Visual Studio Project System could flag it as containing info relevant to the Visual Studio Test Explorer sub-system. But now the Microsoft.NET.Sdk contains some magic somewhere (I believe) that automatically infers somehow that the project is a test project (probably based on whether it has a PackageReference to xunit/nunit/mstest).

[kzu]

Just like xunit will likely never be built-in the .NET Sdk, so is nugetizer, IMHO.

Also, there is no magic nor specific knowledge of xunit by the SDK. It's the xunit package the one that brings in the capability:https://github.com/xunit/xunit/blob/main/src/xunit.v3.core/Package/build/xunit.v3.core.props#L5

So, without the xunit package, there is no test project. Likewise, without nugetizer package reference, there can be no nugetizer "capabilities", including support for nugetize cli command.

I hope this makes sense

[jzabroski]

It seems like it might be cleaner to put the <PackageReference Include="Nugetizer" Version="*" />

BTW, I found this idiom to be incredibly useful when combined with dotnet nuget locals --clear http-cache. However, if you update packages via dotnet CLI, it will overwrite the * with the current version number, which seems wrong to me and inconsistent with how the NuGet tooling is supposed to work.

[jzabroski]

Just like xunit will likely never be built-in the .NET Sdk, so is nugetizer, IMHO.

I can actually see xunit v3 possibly becoming built-in. Personally, I always wished Fixie took off, but xunit is still way better than nunit.

Also, there is no magic nor specific knowledge of xunit by the SDK. It's the xunit package the one that brings in the capability:https://github.com/xunit/xunit/blob/main/src/xunit.v3.core/Package/build/xunit.v3.core.props#L5

That's the v3 code. It used to be that you had to specify a special Guid for the Visual Studio Project System to know it was a test container project. Thankfully, they improved that.

Do you happen to know if ProjectCapability names are reserved by Microsoft, or can we add our own, e.g. especially if we create Visual Studio Code/Visual Studio Pro plug-ins? Just wondering.

So, without the xunit package, there is no test project. Likewise, without nugetizer package reference, there can be no nugetizer "capabilities", including support for nugetize cli command.

I hope this makes sense

Initially, it did. But see my previous question about defining new project capabilities. Ideally, Nugetizer is a capability rather than a dependency. Importing it as a package includes a similar props file that adds the Nugetize capability. That would then allow you to hide nugetize rather than have it as an explicit contract. I believe the transitive closure of all projects can then still be computed without requiring all csproj's to have Nugetizer packagereference. So if people want to pack everything, they just need to pack the entry point application to do so. To me, that is a lot more intuitive.

[mhsimkin]

Hi. I am new to Nugetize. I found it today and have spent the last several hours trying to get my head around how I am supposed to use it.

I feel you really need to have a "Getting Started" guide. Your explanation on what is happening under the covers is interesting, but completely useless to get someone started.

For example, You discuss adding the dotnet-nugetize tool and what that provides. , however no where in that whole discussion was it even mentioned that you need to add nugetize as a package to your projects in order to get it to work.

If it wasn't for this discussion and a couple of issues where there is a discussion on GetPackageContents, I would never have been able to figure this out.

-marc

[kzu]

Agreed. I think the current What section should actually be How, and the What should be the quickstart.

[davidhunter22]

mhsimkin, you are doing better than I am. I have spent quite a while now trying to get Nugetize to produce a single nuget package and have not succeeded. I have tried countless experiments based on what I have read but I either get errors or nothing. I wish there was a HelloWorld example somewhere obvious so that I could at least get on the first rung.

[kzu]

Some real world projects that use it: devlooped/ThisAssembly, devlooped/TableStorage, devlooped/avatar, moq/moq, dotnetconfig/dotnet-config.

Also, nugetizer itself is packed by nugetizer ;)

Agreed that more samples/demos would be nice. Have you looked at the scenarios at https://github.com/devlooped/nugetizer/tree/main/src/NuGetizer.Tests/Scenarios?

[kzu]

Hello world provided: https://github.com/devlooped/nugetizer/tree/main/samples/LibAnalyzerTasks