Is there a way to use types from external proto generation?

[TomTardigradeSEL]

My team is incrementally adopting to Goa for our service design and definitions. We have existing services that use grpc and protobuf. Is there a way to directly use the existing types as payloads when writing the design.go files with the DSL?

[raphael]

I'm assuming that "type" means the actual struct generated by protobuf - not the abstract shape of the data structure. Also I'm assuming "use" means the generated Goa code using these structs - not the user code (as obviously that can use anything). If these assumptions are correct then the short answer is no: the Goa generated code cannot directly use runtime structs that are defined in external packages. A couple of things that might help though:

1. The CreateFrom and ConvertTo DSL functions can be used to generate functions that can convert from generated to existing structs automatically (user code still has to call them).
2. It is possible to completely override the generated transport layer when creating the gRPC server: each method is an exposed field that can be written to (e.g. https://github.com/goadesign/examples/blob/master/basic/gen/grpc/calc/server/server.go#L21). The implementation could be overridden to copy the generated logic but use pre-existing structs.

[TomTardigradeSEL]

@raphael

I came up with a possible work around by wrapping the external proto types in a new "user" type. I did this using the Meta function in the DSL with the struct:field:type and struct:field:proto special keys.

I found some inconsistencies in the code generation this way that results in code that fails to compile. I made an example repo for reference to help reproduce the behavior. https://github.com/TomTardigradeSEL/goa-type-override

The main issues are in the grpc client and server types.go files.

• Duplicate package imports due to aliased and un-aliased imports
• Invalid type conversions in the constructor functions

Responses to questions/suggestions:

The CreateFrom and ConvertTo DSL functions can be used to generate functions that can convert from generated to existing structs automatically (user code still has to call them).

Using the CreateFrom and ConvertTo functions don't work with grpc messages because they contain non-pointer nested structures (state, sizeCache, unknownFields). I included some commented out design DSL in the repo example.

It is possible to completely override the generated transport layer when creating the gRPC server: each method is an exposed field that can be written to (e.g. https://github.com/goadesign/examples/blob/master/basic/gen/grpc/calc/server/server.go#L21). The implementation could be overridden to copy the generated logic but use pre-existing structs.

Would the overriding here be dependent on being able to use the CreateFrom and ConvertTo?
Part of the reason for not redefining the external type in the DSL is due to time constrains. Ideally we would just replace the existing proto and grpc code by using goa.

Some final thoughts. It would help for projects wanting to transition into using goa, if the Meta struct:field:type could also apply to the Type DSL function. That way external types can be defined for use as payloads and results. This would allow projects to use goa for new code, while the old stuff gets migrated into a design file.

Thank you for the suggestions and response.

[raphael]

Unfortunately I don't think there is a simple way to make the above work. The Goa code generation algorithms always use pointers to define fields that point to structs. The rules for knowing when to (de)reference fields are already quite complex (see https://goa.design/learn/faq/) and making it possible to use non-pointer fields on struct would complicate code generation significantly (it's not just a matter of adding a * or a & in the right places, it's also and mostly about having code that checks whether a pointer is nil prior to de-referencing for example or assigning default values in this case etc.) It would also be quite difficult to support arbitrary runtime types as generation target (the set of basic types supported by Goa is currently a subset of all the types that one can define in Go, the problem above also applies).

Stepping back a bit, if it helps it might be easier to define a Goa design that would produce a compatible protobuf definition so that existing clients or services could be reused as is (we did something like this in my current company).