ParametrizedMetaModel(multivariate_optimizer=CmaFmin2) runs are not reproducible

[kvdblom]

Steps to reproduce

1. Install ioh==0.3.14 and nevergrad==1.0.0
2. Run the code below under "Relevant Code" twice
3. Observe that the output of the two runs of the ParametrizedMetaModel differ, despite providing the same seed. While the two runs of MetaModel give the exact same results.

Observed Results

• ParametrizedMetaModel(multivariate_optimizer=CmaFmin2) produced different results for two runs on the same problem, despite using the same seed.
• It does not seem to matter how the seed is set. E.g., instead setting it with np.random.seed(seed) has the same effect.

Expected Results

• When providing the same seed, two runs of the same algorithm on the same problem should give exactly the same results.
• (This is the case for all other algorithms I checked, e.g., MetaModel)

Relevant Code

#!/usr/bin/env python3
import ioh
import nevergrad as ng
from nevergrad.optimization.optimizerlib import ParametrizedMetaModel
from nevergrad.optimization.optimizerlib import MetaModel
from nevergrad.optimization.optimizerlib import CmaFmin2

class NGEvaluator:
  def __init__(self, optimizer: str, eval_budget: int) -> None:
      self.alg = optimizer
      self.eval_budget = eval_budget
      
  def __call__(self, func, seed) -> None:
      parametrization = ng.p.Array(shape=(func.meta_data.n_variables,)).set_bounds(-5, 5)
      parametrization.random_state.seed(seed)
      optimizer = eval(f"{self.alg}")(parametrization=parametrization, budget=self.eval_budget)
      print(f"Result of {self.alg}:")
      print(optimizer.minimize(func))


def run_algos(algorithm: str, problem: int, eval_budget: int, dimension: int, instance: int, seed: int) -> None:
  algorithm = NGEvaluator(algorithm, eval_budget)
  logger = ioh.logger.Analyzer()
  function = ioh.get_problem(problem, instance=instance, dimension=dimension, problem_class=ioh.ProblemClass.REAL)
  algorithm(function, seed)
  function.reset()
  logger.close()

  return


if __name__ == "__main__":
  eval_budget = 200
  dimension = 25
  problem = 11
  instance = 1
  seed = 1

  algorithm = "ParametrizedMetaModel(multivariate_optimizer=CmaFmin2)"
  run_algos(algorithm, problem, eval_budget, dimension, instance, seed)
  
  algorithm = "MetaModel"
  run_algos(algorithm, problem, eval_budget, dimension, instance, seed)

[kvdblom]

Possibly the issue is with CmaFmin2, it is listed as 'unseedable' in the tests here:

nevergrad/nevergrad/optimization/test_optimizerlib.py

Line 141 in 8442810

UNSEEDABLE: tp.List[str] = [

Ideally I think this should be fixed, so it is seedable, but I think this should at least be in the documentation and output a warning when it is used. I also note that CmaFmin2 is not listed under optimizers in the documentation at all, nor is it included in the list generated by sorted(nevergrad.optimizers.registry.keys()) which the documentation claims should list all optimisers.