Use latest lindi, pynwb, and hdmf with aws_region

[rly]

I updated the environment file. Please remake your conda environments after merging.

I added a couple of read remote LINDI file test cases. I think the local lindi test cases still apply, though we should update those to point to the dandiset with the test NWB files that we want to use.

I'm pointing the environment to branches of HDMF and PyNWB that support the aws_region argument to NWBHDF5IO so that we can specify the AWS region in ROS3 tests.

Also we no longer need to say load_namespaces=True. That is now the default.

[rly]

nwb_benchmarks run --bench NWBLindiFileReadRemoteReferenceFileSystemBenchmark
nwb_benchmarks run --bench NWBLindiFileCreateLocalReferenceFileSystemBenchmark
nwb_benchmarks run --bench LindiFileReadLocalReferenceFileSystemBenchmark
nwb_benchmarks run --bench NWBFileReadBenchmark
nwb_benchmarks run --bench DirectFileReadBenchmark

These ran successfully. I have not yet run a full suite of benchmarks yet though.

[rly]

I defined remote benchmarks as such:

lindi_remote_rfs_parameter_cases = dict(
    EcephysTestCase=dict(
        s3_url=get_s3_url(
            is_staging=True,
            dandiset_id="213889",
            dandi_path="sub-IBL-ecephys/sub-IBL-ecephys_ses-3e7ae7c0_desc-18000000-frames-13653-by-384-chunking.lindi.json",
        ),
        object_name="ElectricalSeriesAp",
        slice_range=(slice(0, 30_000), slice(0, 384)),
    ),
    OphysTestCase=dict(
        s3_url=get_s3_url(
            is_staging=True,
            dandiset_id="213889",
            dandi_path="sub-R6_ses-20200206T210000_behavior+ophys/sub-R6_ses-20200206T210000_behavior+ophys.lindi.json",
        ),
        object_name="TwoPhotonSeries",
        slice_range=(slice(0, 3), slice(0, 796), slice(0, 512)),
    ),
    IcephysTestCase=dict(
        s3_url=get_s3_url(
            is_staging=True,
            dandiset_id="213889",
            dandi_path="sub-1214579789_ses-1214621812_icephys/sub-1214579789_ses-1214621812_icephys.lindi.json",
        ),
        object_name="data_00002_AD0",
        slice_range=(slice(0, 30_000),),
    ),

I thought that this would create three test cases. Instead, it creates 9 - the cross of all the parameters. But all of the unintended combinations will fail. I think this is an issue within BaseBenchmark. @CodyCBakerPhD is this behavior correct? and if so, how do I create three test cases instead of the cross of all these options?

[CodyCBakerPhD]

@CodyCBakerPhD is this behavior correct? and if so, how do I create three test cases instead of the cross of all these options?

Well, that's cool...

First time running that parameterized pattern and it's revealed that that is indeed the intended behavior of ASV: https://asv.readthedocs.io/en/v0.6.1/writing_benchmarks.html#parameterized-benchmarks

There's not an easy way I can think of at the moment to resolve this at the parameter_case or Basebenchmark level (attempts to use parameterized package cause issues in benchmark discovery phase)

Only way will be to break up the multilevel dict-of-dicts to separate benchmarks. I'll do that in my PR

[rly]

This fix in lindi makes the lindi ophys test case work. NeurodataWithoutBorders/lindi#71