How to compile the proto definiton

[sfalahati]

Hi Everyone,

I am very new to Mimir (and Python!) trying to figure out how I can push some sample data to our Grafana instance. I did a test using curl which didn't work:

curl POST -H "Content-Type: application/x-protobuf" -H "Content-Encoding: snappy" -H "X-Prometheus-Remote-Write-Version:0.1.0" -H "X-Mimir-SkipLabelNameValidation: true" -H "X-Scope-OrgID: <id>" -d "{__name__='node_cpu_seconds_total',cpu='0',mode='idle'} 80" https://url:9009/api/v1/push

snappy: corrupt input

I was not able to work out what the issue is.

Then I thought about using Python to push the sample data using protocol buffers. however, not sure about how I could use the proto definition and generate the mimir_pb package. I presume I cannot pip install it?

I cloned the repo into /tmp/mimir and:

sfal@sfal-desktop:/tmp$ protoc --proto_path=/tmp/mimir/pkg/mimirpb/ --python_out=/home/sfal/src/ /tmp/mimir/pkg/mimirpb/mimir.proto github.com/gogo/protobuf/gogoproto/gogo.proto: File not found. mimir.proto:15:1: Import "github.com/gogo/protobuf/gogoproto/gogo.proto" was not found or had errors.

I was wondering if there is a sample command that I could use to compile the proto definition.

thanks for helping.

[56quarters]

Hi,

What specifically are you trying to do? Usually applications don't write directly to Mimir. Instead, they would expose metrics using a Prometheus client library at an HTTP endpoint (such as /metrics) and Prometheus or the Grafana Agent handles scraping each of those applications. Prometheus or the Grafana Agent would then be responsible for remote writing to Mimir.

Some links that might be helpful:

• Prometheus getting started.
• Prometheus remote storage integrations (this is what Mimir acts as in many setups).
• Grafana Cloud guides.

[sfalahati]

Hi,

thanks for reply. I need to study this more then and thanks for providing the links. I was trying to follow discussion 4009 and write data to mimir using python. There is a code sample there but it invokes methods (LabelPair, Exemplar, etc.) from - what seems to be a - package called mimir_pb2. I am not able to compile/construct this package so was wondering if I could be provided with some information as to how to do it.

[pstibrany]

Hi, you can try the following commands from inside Mimir repository:

First, fetch build image that we use for building Mimir:

make fetch-build-image

This will fetch build image and tag it with :latest label. Then we can run /usr/bin/protoc from inside this build image. We also map docker volumes so that protoc has full access to the Mimir proto files, and you can get the output from protoc after Docker finishes:

docker run -v $(pwd):/go/src/github.com/grafana/mimir:delegated,z -w /go/src/github.com/grafana/mimir --entrypoint /usr/bin/protoc grafana/mimir-build-image:latest -I /go/src:./vendor/github.com/gogo/protobuf:./vendor:./pkg/storegateway/storepb:. ./pkg/mimirpb/mimir.proto --python_out=.

After running this, you should see generated pkg/mimirpb/mimir_pb2.py file. You can use this to build WriteRequest message that Mimir is expecting (Mimir actually expects snappy-compressed version of this message).

[sfalahati]

Hi Peter,

thanks a lot!

I git cloned the repo, ran make fetch-build-image, ran the docker command and saw mimir_pb2.py created.

[ec2-user@mimirpb]$ python3
Python 3.7.16 (default, Dec 15 2022, 23:24:54)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import mimir_pb2
Traceback (most recent call last):
File "", line 1, in
File "/home/ec2-user/mimir/pkg/mimirpb/mimir_pb2.py", line 14, in
from github.com.gogo.protobuf.gogoproto import gogo_pb2 as github_dot_com_dot_gogo_dot_protobuf_dot_gogoproto_dot_gogo__pb2
ModuleNotFoundError: No module named 'github.com'

Looking inside mimir_pb2.py, it obviously complains about line 14:

from github.com.gogo.protobuf.gogoproto import gogo_pb2 as github_dot_com_dot_gogo_dot_protobuf_dot_gogoproto_dot_gogo__pb2

Would you know by any chance, how I could go about resolving this?

cheers

[Pixel-Jack]

For anyone passing by this discussion, I had the same trouble compiling the proto. What I had to do was to:

1. Copy at the root of the repo the file pkg/mimirpb/mimir.proto and the file vendor/github.com/gogo/protobuf/gogoproto/gogo.proto
2. Replace in the file mimir.proto the line import "github.com/gogo/protobuf/gogoproto/gogo.proto"; by import "gogo.proto";
3. Launch those two commands:

docker run -v $(pwd):/go/src/github.com/grafana/mimir:delegated,z -w /go/src/github.com/grafana/mimir --entrypoint /usr/bin/protoc grafana/mimir-build-image:latest -I /go/src:./vendor/github.com/gogo/protobuf:./vendor:./pkg/storegateway/storepb:.  ./pkg/mimirpb/mimir.proto --python_out=.

This will create the python files for the mimir.proto

docker run -v $(pwd):/go/src/github.com/grafana/mimir:delegated,z -w /go/src/github.com/grafana/mimir --entrypoint /usr/bin/protoc grafana/mimir-build-image:latest -I /go/src:./vendor/github.com/gogo/protobuf:./vendor:./pkg/storegateway/storepb:.  ./pkg/mimirpb/gogo.proto --python_out=.

This will create the python files for the gogo.proto

And now you can use in your python code import mimir_pb2