We use protobuf and the buf CLI to create and generate the shared files between a Go and a Python project (for ensuring structured data to be transmitted between the projects in a common way).

We want to build the generator commands as simple as possible, as the generation is done using GitHub workflows and taskfiles etc. and therefore do not want (to the greatest extent possible), to maintain specific file paths every time a new .proto file is added.

In addition we also use the buf linting to ensure the files are valid...

We have tried different things, but don't think we can quite get what we want (without having to do post-processing by renaming and moving files and folders)...

buf.yaml

version: v2
modules:
  - path: proto
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

As mentioned we use the .proto files as the base for both a Go and a Python project - the Go project. The file structure (output) of the Go project is not as relevant or important as it is in our other component, which uses Python.

buf.gen.yaml from the Python project (separate repo, fetching files from the other repo using the github input - this have however been removed from the below file listing as it has nothing to do with this issue)

version: v2
managed:
  enabled: true
plugins:
  - remote: buf.build/protocolbuffers/python
    out: components
  - remote: buf.build/protocolbuffers/pyi
    out: components

The remote python plugin does not support the options (opt) argument - and the python protoc_builtin plugin seems to require each file to be lisited (no wildcards etc are supported).

We have tried different structures of the proto files, but if I use a folder name with dots (like company.app1) the linter is not satisfied.

• proto/com/company/app1/entity1/v1/entity1.proto
• proto/com/company/app1/entity1/v2/entity1.proto
• proto/com/company/app2/entity2/v1/entity2.proto

This requires the packages to be something like this:

package company.app1.entity1.v1;

message Entity1 {
    string name = 2;
}

(if we add dots in a folder name, then linter says that the package name used is not correct - and you cannot use underscore or hyphen to replace the dots in the folder name)

What we want to achieve (output) is the generated folders in the python projects to be something like this:

• components/company.app1/entity1/v1/entity1.proto
• components/company.app1/entity1/v2/entity1.proto
• components/company.app2/entity2/v1/entity2.proto

The Go code has not the same requirements regarding the folder structure and can be handled by the individual imports

We use protobuf and the buf CLI to create and generate the shared files between a Go and a Python project (for ensuring structured data to be transmitted between the projects in a common way).

We want to build the generator commands as simple as possible, as the generation is done using GitHub workflows and taskfiles etc. and therefore do not want (to the greatest extent possible), to maintain specific file paths every time a new .proto file is added.

In addition we also use the buf linting to ensure the files are valid...

We have tried different things, but don't think we can quite get what we want (without having to do post-processing by renaming and moving files and folders)...

buf.yaml

version: v2
modules:
  - path: proto
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

As mentioned we use the .proto files as the base for both a Go and a Python project - the Go project. The file structure (output) of the Go project is not as relevant or important as it is in our other component, which uses Python.

buf.gen.yaml from the Python project (separate repo, fetching files from the other repo using the github input - this have however been removed from the below file listing as it has nothing to do with this issue)

version: v2
managed:
  enabled: true
plugins:
  - remote: buf.build/protocolbuffers/python
    out: components
  - remote: buf.build/protocolbuffers/pyi
    out: components

The remote python plugin does not support the options (opt) argument - and the python protoc_builtin plugin seems to require each file to be lisited (no wildcards etc are supported).

We have tried different structures of the proto files, but if I use a folder name with dots (like com.company.app1) the linter is not satisfied.

• proto/com/company/app1/entity1/v1/entity1.proto
• proto/com/company/app1/entity1/v2/entity1.proto
• proto/com/company/app2/entity2/v1/entity2.proto

This requires the packages to be something like this:

package com.company.app1.entity1.v1;

message Entity1 {
    string name = 2;
}

(if we add dots in a folder name, then linter says that the package name used is not correct - and you cannot use underscore or hyphen to replace the dots in the folder name)

What we want to achieve (output) is the generated folders in the python projects to be something like this:

• components/com.company.app1/entity1/v1/entity1.proto
• components/com.company.app1/entity1/v2/entity1.proto
• components/com.company.app2/entity2/v1/entity2.proto

The Go code has not the same requirements regarding the folder structure and can be handled by the individual imports

• python
• protocol-buffers
• buf

1 Answer 1

Protobuf and Buf expect that each dot segment in your package maps to directory names.If you truly want directory names contain dots(like com.company.app1) and also want Bufs standard lint rules to pass there is no perfect solution—either:

1.Reorganize on-disk folder structure to match standard Protobuf convention(com/company/app1/entity1/v1) so that package com.company.app1.entity1.v1; matches directory layout
2. Disable particular lint rules in buf.yaml that enforce directory/package matching
3.Post-process(rename)your generated code after Buf runs in order to get your desired `dots in folder names` layout

Because you mentioned not wanting to do post-processing your best bet is to align the folder structure with typical Protobuf/Buf conventions (1) or relax lint rules (2)