Multiple Bucket in bucket.yml for thanos store?

[rrraditya]

Hi, I wonder can we set multiple buckets in one bucket.yml so thanos store will recognize those all buckets that we mapped for each prometheus sites blocks?
i have a concern when we have lets say 10 prometheus sending their blocks to one buckets, is there any possibility where sometimes in a day, the block ID or we can say the ULID from 2 prometheus are identical. Is it even possible?

if external labels are included on ULID generation, then i got no concern, because I already set all external labels for each prometheus instances. So i assume with different value of external lables, then there will be no chance the ULID will ever be the same between instances.

Thank you.

[GiedriusS]

Hello! No, that's not supported. It's very easy to start a new Thanos Store instance and connect it to Thanos Query so it's probably not worth having such a feature.

ULIDs are constructed from the timestamp + entropy from crypto/rand.Reader. Here is how it looks like:

 29 /*                                                                                                                                                                                                     
 30 An ULID is a 16 byte Universally Unique Lexicographically Sortable Identifier                                                                                                                          
 31                                                                                                                                                                                                        
 32     The components are encoded as 16 octets.                                                                                                                                                           
 33     Each component is encoded with the MSB first (network byte order).                                                                                                                                 
 34                                                                                                                                                                                                        
 35     0                   1                   2                   3                                                                                                                                      
 36     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1                                                                                                                                    
 37     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                                                                                                                                  
 38     |                      32_bit_uint_time_high                    |                                                                                                                                  
 39     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                                                                                                                                  
 40     |     16_bit_uint_time_low      |       16_bit_uint_random      |                                                                                                                                  
 41     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                                                                                                                                  
 42     |                       32_bit_uint_random                      |                                                                                                                                  
 43     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                                                                                                                                  
 44     |                       32_bit_uint_random                      |                                                                                                                                  
 45     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                                                                                                                                  
 46 */                                                                                                                                                                                                     
 47 type ULID [16]byte

So, to get exact ULIDs, the timestamp would have to exactly the same + the random entropy. The blocks would have to be produced at the same millisecond and the chance for the random value to be the same is:

2 / 2^96 = 2^(-95)

Hence, it is very safe to say that you will never have equal ULID values coming from >1 Prometheus nodes.

[rrraditya]

hi @GiedriusS
thank you for replying my question.
understood. if the chance of getting same ULIDs is that complex then i have no issue on having multiple prometheus blocks in 1 bucket.

by the way, a bit out of topic.
lets say we are implementing SLO with our metrics, and we do a lot of evaluation within 30d of timerange, and we all know that it will need to process a lot of data.

By implementing thanos, we disabled the compation process within prometheus and send each 2h blocks to the s3 bucket. means that prometheus only kept raw data for certain retention time, and thanos-store will have the raw and compact data (compacted by thanos-compact).

how do you recommend us to do the evaluation?

1. do the evaluation locally by creating the recording rules on each prometheus? (lets say we extend the tsdb retention to 35d)
2. using thanos-rules to create recording rules and do the evaluation? grabbing data from prometheus<>sidecar and from thanos-store?

looking forward for some insights.
Thank you!