Parallelized brute force, and other optimization methods

[masonyoungblood]

Hello! I have two questions related to optimization.

First, I'm having a hard time parallelizing the brute force method in lmfit, which I think was used in past versions of the package based on comments (e.g. 'workers':-1). If I set up everything for optimization using brute force, with five points across five workers...

brute = {'method':'brute', 'Ns':5, 'workers':5}
optimizer = bs.Optimizer(syllable, method_kwargs=brute)
optimal_gamma = optimizer.OptimalGamma(syllable)

Then I get the following PicklingError error, which unfortunately I can't find much info about online...

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
Cell In [12], [line 1](vscode-notebook-cell:?execution_count=12&line=1)
----> [1](vscode-notebook-cell:?execution_count=12&line=1) optimal_gamma = optimizer.OptimalGamma(syllable)
      [2](vscode-notebook-cell:?execution_count=12&line=2) syllable.p["gm"].set(value=optimal_gamma)

File [~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:103](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:103), in Optimizer.OptimalGamma(self, obj)
    [101](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:101) start = time.time()
    [102](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:102) self.obj.p["gm"].set(vary=True)
--> [103](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:103) mi    = lmfit.minimize(self.residualSCI, self.obj.p, nan_policy='omit', method=self.method, **self.kwargs) 
    [104](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:104) self.obj.p["gm"].set(value=mi.params["gm"].value, vary=False)
    [105](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:105) end   = time.time()

File [~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2583](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2583), in minimize(fcn, params, method, args, kws, iter_cb, scale_covar, nan_policy, reduce_fcn, calc_covar, max_nfev, **fit_kws)
   [2443](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2443) """Perform the minimization of the objective function.
   [2444](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2444) 
   [2445](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2445) The minimize function takes an objective function to be minimized,
   (...)
   [2577](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2577) 
   [2578](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2578) """
   [2579](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2579) fitter = Minimizer(fcn, params, fcn_args=args, fcn_kws=kws,
   [2580](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2580)                    iter_cb=iter_cb, scale_covar=scale_covar,
   [2581](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2581)                    nan_policy=nan_policy, reduce_fcn=reduce_fcn,
   [2582](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2582)                    calc_covar=calc_covar, max_nfev=max_nfev, **fit_kws)
-> [2583](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2583) return fitter.minimize(method=method)
...
     [50](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/multiprocessing/reduction.py:50)     buf = io.BytesIO()
---> [51](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/multiprocessing/reduction.py:51)     cls(buf, protocol).dump(obj)
     [52](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/multiprocessing/reduction.py:52)     return buf.getbuffer()

PicklingError: Can't pickle <function _lambdifygenerated at 0x2835a29d0>: attribute lookup _lambdifygenerated on __main__ failed

Do you know what the issue might be?

My second question is, have you had any success using the other optimization functions available in lmfit? I'm able to fit the gamma parameter using leastsq, but if I try the other parameters I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [19], [line 1](vscode-notebook-cell:?execution_count=19&line=1)
----> [1](vscode-notebook-cell:?execution_count=19&line=1) optimizer.OptimalParams(syllable)

File [~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:175](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:175), in Optimizer.OptimalParams(self, obj, Ns)
    [173](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:173) if self.method=="brute": self.kwargs["Ns"] = Ns     
    [174](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:174) start = time.time()
--> [175](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:175) print("As");   self.OptimalAs(obj);   
    [176](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:176) print("Bs");   self.OptimalBs(obj);
    [177](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:177) print("end")

File [~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:83](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:83), in Optimizer.OptimalAs(self, obj)
     [75](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:75) self.obj.p["a0"].set(vary=True)
     [77](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:77) # def ResidualCo(c):
     [78](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:78) #     print("pickling a C instance...")
     [79](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:79) #     return Syllable, (c.residualCorrelation,)
     [80](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:80) # copyreg.pickle(Syllable, ResidualCo)
---> [83](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:83) mi0    = lmfit.minimize(self.residualCorrelation, self.obj.p, nan_policy='omit', method=self.method, **self.kwargs) 
     [84](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:84) self.obj.p["a0"].set(vary=False, value=mi0.params["a0"].value)
     [85](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/birdsongs-0.1b0-py3.8.egg/birdsongs/optimizer.py:85) end0   = time.time()

File [~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2583](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2583), in minimize(fcn, params, method, args, kws, iter_cb, scale_covar, nan_policy, reduce_fcn, calc_covar, max_nfev, **fit_kws)
   [2443](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2443) """Perform the minimization of the objective function.
   [2444](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/lmfit/minimizer.py:2444) 
...
    [415](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py:415)                     f" not exceed func output vector length M={m}")
    [417](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py:417) if epsfcn is None:
    [418](https://file+.vscode-resource.vscode-cdn.net/Users/masonyoungblood/Documents/Work/Fall%202023/Song%20Extinction/~/anaconda3/envs/birdsong-phys-model/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py:418)     epsfcn = finfo(dtype).eps

TypeError: Improper input: func input vector length N=2 must not exceed func output vector length M=1

I think that the brute force method may be too slow and approximate for my application, especially without parallelization, so I'm hoping that some alternatives might be possible. Thanks in advance!