[Bug] cellxgene_census_builder//test_builder.py failing

[ryan-williams]

Describe the bug
@ivirshup reported that cellxgene_census_builder//test_builder.py began failing when run against TileDB-SOMA@main.

To Reproduce
The issue appears to have started with #2895:

• GHA failure at 7241fef7
• GHA success at previous commit ab0a5b59

[johnkerl]

#2895 was significant enough to merit 1.13.1. Reverting that fix is a temporary step, but will not be sufficient for TileDB-SOMA 1.14.0

We need to:

• Temporarily revert 2895 if necessary
• Deep-dive the consolidate/vacuum failure CZI has been seeing
• Fix that issue
• Restore 2895, 2897, 2909 and the documented/test-cased bugs they fixed
• Only then can we release 1.14.0

I'm tagging this as a blocker for this reason.

[ivirshup]

Here's a failing test run + the conda environment I ran it in. Interestingly seeing some slight differences in errors.

Failing test run

$ pytest -n 4
============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/ubuntu/github/cellxgene-census/tools/cellxgene_census_builder
configfile: pyproject.toml
plugins: xdist-3.6.1
4 workers [160 items]   
........................................................................ [ 45%]
........................................................................ [ 90%]
...........FFFF.                                                         [100%]
=================================== FAILURES ===================================
_____________ test_base_builder_creation[True-census_build_args0] ______________
[gw3] linux -- Python 3.11.9 /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/bin/python3.11

datasets = [Dataset(dataset_id='homo_sapiens_0', dataset_asset_h5ad_uri='mock', dataset_version_id='homo_sapiens_0_v0', dataset_h...lesize=-1, cell_count=-1, mean_genes_per_cell=-1.0, schema_version='', dataset_total_cell_count=4, soma_joinid=5), ...]
census_build_args = CensusBuildArgs(working_dir=PosixPath('/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0'), co...t_prefix='census-builder-', user_agent_environment='unknown', manifest=None, test_first_n=0), state=CensusBuildState())
validate = True, setup = None

    @pytest.mark.parametrize(
        "census_build_args",
        (
            {
                "consolidate": False,
                "build_tag": "test_tag",
                "verbose": 0,
                "max_worker_processes": 1,
            },
            {
                "consolidate": True,
                "build_tag": "test_tag",
                "verbose": 1,
                "max_worker_processes": 1,
            },
        ),
        indirect=True,
    )
    @pytest.mark.parametrize("validate", (True, False))
    def test_base_builder_creation(
        datasets: list[Dataset],
        census_build_args: CensusBuildArgs,
        validate: bool,
        setup: None,
    ) -> None:
        """
        Runs the builder, queries the census and performs a set of base assertions.
        """
    
        def proxy_create_dask_client(
            *args: CensusBuildArgs,
            **kwargs: Any,
        ) -> dask.distributed.Client:
            from cellxgene_census_builder.build_soma.mp import create_dask_client
    
            kwargs["processes"] = False
            kwargs["n_workers"] = 1
            kwargs.pop("threads_per_worker")
            return create_dask_client(*args, **kwargs)
    
        # proxy psutil.virtual_memory to return 1/2 of the actual memory. There
        # are repeated cases where the test runners OOM, and this helps avoid it.
        memstats = psutil.virtual_memory()
        memstats = memstats._replace(total=int(memstats.total // 2))
    
        def proxy_psutil_virtual_memory() -> psutil._pslinux.svmem:
            return memstats
    
        with (
            patch("cellxgene_census_builder.build_soma.build_soma.prepare_file_system"),
            patch("cellxgene_census_builder.build_soma.build_soma.build_step1_get_source_datasets", return_value=datasets),
            patch(
                "cellxgene_census_builder.build_soma.build_soma.create_dask_client", side_effect=proxy_create_dask_client
            ),
            patch("psutil.virtual_memory", side_effect=proxy_psutil_virtual_memory),
        ):
            process_init(census_build_args)
>           return_value = build(census_build_args, validate=validate)

tests/test_builder.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/cellxgene_census_builder/build_soma/build_soma.py:121: in build
    assert f.result()
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/client.py:328: in result
    return self.client.sync(self._result, callback_timeout=timeout)
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/dask/bag/core.py:1868: in reify
    seq = list(seq)
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/dask/bag/core.py:2056: in __next__
    return self.f(*vals)
src/cellxgene_census_builder/logging.py:114: in wrapper
    result: R = func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    from __future__ import annotations
    
    import copy
    import dataclasses
    import gc
    import logging
    import os.path
    import pathlib
    from collections.abc import Iterable, Sequence
    from dataclasses import dataclass
    from datetime import datetime
    from typing import Any, Self, TypeVar
    
    import dask
    import numpy as np
    import numpy.typing as npt
    import pandas as pd
    import pyarrow as pa
    import tiledb
    import tiledbsoma as soma
    from dask import distributed
    from dask.delayed import Delayed
    from scipy import sparse
    
    from ..build_state import CensusBuildArgs
    from ..logging import logit
    from ..util import clamp, cpu_count, log_process_resource_status, urlcat
    from .anndata import open_anndata
    from .consolidate import list_uris_to_consolidate
    from .datasets import Dataset
    from .experiment_builder import ExperimentSpecification
    from .experiment_specs import make_experiment_specs
    from .globals import (
        CENSUS_DATA_NAME,
        CENSUS_DATASETS_NAME,
        CENSUS_DATASETS_TABLE_SPEC,
        CENSUS_INFO_NAME,
        CENSUS_OBS_STATS_COLUMNS,
        CENSUS_OBS_TABLE_SPEC,
        CENSUS_SCHEMA_VERSION,
        CENSUS_SUMMARY_CELL_COUNTS_NAME,
        CENSUS_SUMMARY_CELL_COUNTS_TABLE_SPEC,
        CENSUS_SUMMARY_NAME,
        CENSUS_VAR_TABLE_SPEC,
        CENSUS_X_LAYERS,
        CXG_OBS_COLUMNS_READ,
        CXG_OBS_TERM_COLUMNS,
        CXG_SCHEMA_VERSION,
        CXG_VAR_COLUMNS_READ,
        FEATURE_DATASET_PRESENCE_MATRIX_NAME,
        FULL_GENE_ASSAY,
        MEASUREMENT_RNA_NAME,
        SOMA_TileDB_Context,
    )
    from .mp import create_dask_client, shutdown_dask_cluster
    
    logger = logging.getLogger(__name__)
    
    
    @dataclass  # TODO: use attrs
    class EbInfo:
        """Class used to accumulate information about axis (for validation code)."""
    
        n_obs: int = 0
        vars: set[str] = dataclasses.field(default_factory=set)
        dataset_ids: set[str] = dataclasses.field(default_factory=set)
    
        def update(self: Self, b: Self) -> Self:
            self.n_obs += b.n_obs
            self.vars |= b.vars
            self.dataset_ids |= b.dataset_ids
            return self
    
        @property
        def n_vars(self) -> int:
            return len(self.vars)
    
    
    def assert_all(__iterable: Iterable[object]) -> bool:
        r = all(__iterable)
        assert r
        return r
    
    
    def open_experiment(base_uri: str, eb: ExperimentSpecification) -> soma.Experiment:
        """Helper function that knows the Census schema path conventions."""
        return soma.Experiment.open(urlcat(base_uri, CENSUS_DATA_NAME, eb.name), mode="r")
    
    
    def get_experiment_shape(base_uri: str, specs: list[ExperimentSpecification]) -> dict[str, tuple[int, int]]:
        shapes = {}
        for es in specs:
            with open_experiment(base_uri, es) as exp:
                shape = (exp.obs.count, exp.ms[MEASUREMENT_RNA_NAME].var.count)
                shapes[es.name] = shape
        return shapes
    
    
    def validate_all_soma_objects_exist(soma_path: str, experiment_specifications: list[ExperimentSpecification]) -> bool:
        """Validate all objects present and contain expected metadata.
    
        soma_path
            +-- census_info: soma.Collection
            |   +-- summary: soma.DataFrame
            |   +-- datasets: soma.DataFrame
            |   +-- summary_cell_counts: soma.DataFrame
            +-- census_data: soma.Collection
            |   +-- homo_sapiens: soma.Experiment
            |   +-- mus_musculus: soma.Experiment
        """
        with soma.Collection.open(soma_path, context=SOMA_TileDB_Context()) as census:
            assert soma.Collection.exists(census.uri)
            assert datetime.fromisoformat(census.metadata["created_on"])
            assert "git_commit_sha" in census.metadata
    
            for name in [CENSUS_INFO_NAME, CENSUS_DATA_NAME]:
                assert soma.Collection.exists(census[name].uri)
    
            census_info = census[CENSUS_INFO_NAME]
            for name in [CENSUS_DATASETS_NAME, CENSUS_SUMMARY_NAME, CENSUS_SUMMARY_CELL_COUNTS_NAME]:
                assert name in census_info, f"`{name}` missing from census_info"
                assert soma.DataFrame.exists(census_info[name].uri)
    
            assert sorted(census_info[CENSUS_DATASETS_NAME].keys()) == sorted(CENSUS_DATASETS_TABLE_SPEC.field_names())
            assert sorted(census_info[CENSUS_SUMMARY_CELL_COUNTS_NAME].keys()) == sorted(
                CENSUS_SUMMARY_CELL_COUNTS_TABLE_SPEC.field_names()
            )
            assert sorted(census_info[CENSUS_SUMMARY_NAME].keys()) == sorted(["label", "value", "soma_joinid"])
    
            census_summary = census[CENSUS_INFO_NAME][CENSUS_SUMMARY_NAME].read().concat().to_pandas()
            assert (
                census_summary.loc[census_summary["label"] == "census_schema_version"].iloc[0]["value"]
                == CENSUS_SCHEMA_VERSION
            )
            assert (
                census_summary.loc[census_summary["label"] == "dataset_schema_version"].iloc[0]["value"]
                == CXG_SCHEMA_VERSION
            )
    
            # verify required dataset fields are set
            df: pd.DataFrame = census_info[CENSUS_DATASETS_NAME].read().concat().to_pandas()
            assert (df["collection_id"] != "").all()
            assert (df["collection_name"] != "").all()
            assert (df["dataset_title"] != "").all()
            assert (df["dataset_version_id"] != "").all()
    
            # there should be an experiment for each builder
            census_data = census[CENSUS_DATA_NAME]
            for eb in experiment_specifications:
                assert soma.Experiment.exists(census_data[eb.name].uri)
    
                e = census_data[eb.name]
                assert soma.DataFrame.exists(e.obs.uri)
                assert soma.Collection.exists(e.ms.uri)
    
                # there should be a single measurement called 'RNA'
                assert soma.Measurement.exists(e.ms[MEASUREMENT_RNA_NAME].uri)
    
                # The measurement should contain all X layers where n_obs > 0 (existence checked elsewhere)
                rna = e.ms[MEASUREMENT_RNA_NAME]
                assert soma.DataFrame.exists(rna["var"].uri)
                assert soma.Collection.exists(rna["X"].uri)
    
                # layers and presence exist only if there are cells in the measurement
                if e.obs.count > 0:
                    for lyr in CENSUS_X_LAYERS:
                        assert lyr in rna.X
                        assert soma.SparseNDArray.exists(rna.X[lyr].uri)
    
                    # and a dataset presence matrix
                    assert soma.SparseNDArray.exists(rna[FEATURE_DATASET_PRESENCE_MATRIX_NAME].uri)
                    assert sum([c.non_zero_length for c in rna["feature_dataset_presence_matrix"].read().coos()]) > 0
                    # TODO(atolopko): validate 1) shape, 2) joinids exist in datsets and var
    
        gc.collect()
        log_process_resource_status()
        return True
    
    
    @logit(logger)
    def validate_axis_dataframes_schema(soma_path: str, experiment_specifications: list[ExperimentSpecification]) -> bool:
        """Validate axis dataframe schema matches spec."""
        with soma.Collection.open(soma_path, context=SOMA_TileDB_Context()) as census:
            census_data = census[CENSUS_DATA_NAME]
    
            # check schema
            for eb in experiment_specifications:
                obs = census_data[eb.name].obs
                var = census_data[eb.name].ms[MEASUREMENT_RNA_NAME].var
                assert sorted(obs.keys()) == sorted(CENSUS_OBS_TABLE_SPEC.field_names())
                assert sorted(var.keys()) == sorted(CENSUS_VAR_TABLE_SPEC.field_names())
                for field in obs.schema:
                    assert CENSUS_OBS_TABLE_SPEC.field(field.name).is_type_equivalent(
                        field.type
                    ), f"Unexpected type in {field.name}: {field.type}"
                for field in var.schema:
                    assert CENSUS_VAR_TABLE_SPEC.field(field.name).is_type_equivalent(
                        field.type
                    ), f"Unexpected type in {field.name}: {field.type}"
    
        return True
    
    
    @logit(logger)
    def validate_axis_dataframes_global_ids(
        soma_path: str,
        experiment_specifications: list[ExperimentSpecification],
        eb_info: dict[str, EbInfo],
    ) -> bool:
        """Validate axes joinid assignment, shape, etc."""
        for eb in experiment_specifications:
            with open_experiment(soma_path, eb) as exp:
                # obs
    
                census_obs_df = (
                    exp.obs.read(
                        column_names=[
                            "soma_joinid",
                            "dataset_id",
                            "tissue_type",
                            "raw_sum",
                            "nnz",
                            "raw_mean_nnz",
                            "raw_variance_nnz",
                            "n_measured_vars",
                        ]
                    )
                    .concat()
                    .to_pandas()
                )
                assert eb_info[eb.name].n_obs == len(census_obs_df) == exp.obs.count
                assert (len(census_obs_df) == 0) or (census_obs_df.soma_joinid.max() + 1 == eb_info[eb.name].n_obs)
                assert eb_info[eb.name].dataset_ids == set(census_obs_df.dataset_id.unique())
    
                # Validate that all obs soma_joinids are unique and in the range [0, n).
                obs_unique_joinids = np.unique(census_obs_df.soma_joinid.to_numpy())
                assert len(obs_unique_joinids) == len(census_obs_df.soma_joinid.to_numpy())
                assert (len(obs_unique_joinids) == 0) or (
                    (obs_unique_joinids[0] == 0) and (obs_unique_joinids[-1] == (len(obs_unique_joinids) - 1))
                )
    
                # Validate that we only contain primary tissue cells, no organoid, cell culture, etc.
                # See census schema for more info.
                assert (census_obs_df.tissue_type == "tissue").all()
    
                # Assert the stats values look reasonable
                assert all(
                    np.isfinite(census_obs_df[col]).all() and (census_obs_df[col] >= 0).all()
                    for col in ["raw_sum", "nnz", "raw_mean_nnz", "raw_variance_nnz", "n_measured_vars"]
                )
    
                del census_obs_df, obs_unique_joinids
    
                # var
                n_vars = len(eb_info[eb.name].vars)
    
                census_var_df = (
                    exp.ms[MEASUREMENT_RNA_NAME].var.read(column_names=["feature_id", "soma_joinid"]).concat().to_pandas()
                )
                assert n_vars == len(census_var_df) == exp.ms[MEASUREMENT_RNA_NAME].var.count
                assert eb_info[eb.name].vars == set(census_var_df.feature_id.array)
                assert (len(census_var_df) == 0) or (census_var_df.soma_joinid.max() + 1 == n_vars)
    
                # Validate that all var soma_joinids are unique and in the range [0, n).
                var_unique_joinids = np.unique(census_var_df.soma_joinid.to_numpy())
                assert len(var_unique_joinids) == len(census_var_df.soma_joinid.to_numpy())
                assert (len(var_unique_joinids) == 0) or (
                    (var_unique_joinids[0] == 0) and var_unique_joinids[-1] == (len(var_unique_joinids) - 1)
                )
    
                del census_var_df
    
        return True
    
    
    def validate_axis_dataframes(
        assets_path: str,
        soma_path: str,
        datasets: list[Dataset],
        experiment_specifications: list[ExperimentSpecification],
        args: CensusBuildArgs,
    ) -> Delayed[dict[str, EbInfo]]:
        @logit(logger, msg="{0.dataset_id}")
        def _validate_axis_dataframes(
            dataset: Dataset, experiment_specifications: list[ExperimentSpecification], assets_path: str, soma_path: str
        ) -> dict[str, EbInfo]:
            eb_info: dict[str, EbInfo] = {}
            for eb in experiment_specifications:
                with soma.Collection.open(soma_path, context=SOMA_TileDB_Context()) as census:
                    census_data = census[CENSUS_DATA_NAME]
                    dataset_id = dataset.dataset_id
                    ad = open_anndata(
                        dataset,
                        base_path=assets_path,
                        obs_column_names=CXG_OBS_COLUMNS_READ,
                        var_column_names=CXG_VAR_COLUMNS_READ,
                        filter_spec=eb.anndata_cell_filter_spec,
                    )
                    eb_info[eb.name] = EbInfo()
                    se = census_data[eb.name]
                    dataset_obs = (
                        se.obs.read(
                            column_names=list(CENSUS_OBS_TABLE_SPEC.field_names()),
                            value_filter=f"dataset_id == '{dataset_id}'",
                        )
                        .concat()
                        .to_pandas()
                        .drop(
                            columns=[
                                "dataset_id",
                                "tissue_general",
                                "tissue_general_ontology_term_id",
                                *CENSUS_OBS_STATS_COLUMNS,
                            ]
                        )
                        .sort_values(by="soma_joinid")
                        .drop(columns=["soma_joinid"])
                        .reset_index(drop=True)
                    )
    
                    # decategorize census obs slice, as it will not have the same categories as H5AD obs,
                    # preventing Pandas from performing the DataFrame equivalence operation.
                    for key in dataset_obs:
                        if isinstance(dataset_obs[key].dtype, pd.CategoricalDtype):
                            dataset_obs[key] = dataset_obs[key].astype(dataset_obs[key].cat.categories.dtype)
    
>                   assert len(dataset_obs) == len(ad.obs), f"{dataset.dataset_id}/{eb.name} obs length mismatch"
E                   AssertionError: homo_sapiens_0/homo_sapiens obs length mismatch

src/cellxgene_census_builder/build_soma/validate_soma.py:327: AssertionError
----------------------------- Captured stderr call -----------------------------
2024-08-21 22:27:49,190 - distributed.worker - WARNING - Compute Failed
Key:       ('_validate_axis_dataframes-478b44f9200898e3aadddedc1d475e1c', 0)
Function:  execute_task
args:      ((<function reify at 0x7e8591d6ce00>, (<function map_chunk at 0x7e8591d6d260>, <function validate_axis_dataframes.<locals>._validate_axis_dataframes at 0x7e8562b742c0>, [[Dataset(dataset_id='homo_sapiens_0', dataset_asset_h5ad_uri='/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_0.h5ad', dataset_version_id='homo_sapiens_0_v0', dataset_h5ad_path='/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_0.h5ad', dataset_title='title_homo_sapiens', citation='citation', collection_id='id_homo_sapiens', collection_name='collection_homo_sapiens', collection_doi='', collection_doi_label='', asset_h5ad_filesize=80064, cell_count=-1, mean_genes_per_cell=-1.0, schema_version='', dataset_total_cell_count=4, soma_joinid=0), Dataset(dataset_id='homo_sapiens_1', dataset_asset_h5ad_uri='/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_1.h5ad', dataset
kwargs:    {}
Exception: "AssertionError('homo_sapiens_0/homo_sapiens obs length mismatch')"

2024-08-21 22:27:49,205 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute('all-part-validate_X_layers_normalized-all-aggregate-481cabbb9d6647bcd15fc4593a9fdf00')" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:27:49,206 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute('validate_internal_consistency-8f3d8bcf-efce-4ba2-baac-1ec8dc91739e')" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:27:49,206 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-703bb73db9bb4de00bc9c6ee26839dc5', 1))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:27:49,206 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-703bb73db9bb4de00bc9c6ee26839dc5', 3))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:27:49,206 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-703bb73db9bb4de00bc9c6ee26839dc5', 0))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:27:49,206 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-703bb73db9bb4de00bc9c6ee26839dc5', 2))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
------------------------------ Captured log call -------------------------------
WARNING  py.warnings:warnings.py:110 /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 33325 instead
  warnings.warn(

INFO     cellxgene_census_builder.build_soma.mp:mp.py:58 Dask client created: <Client: 'inproc://172.31.29.102/2814823/1' processes=1 threads=32>
INFO     cellxgene_census_builder.build_soma.mp:mp.py:59 Dask client using cluster: LocalCluster(ee4baf80, 'inproc://172.31.29.102/2814823/1', workers=1, threads=32)
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:195 Build step 2 - Create root collection - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 homo_sapiens: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 mus_musculus: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:203 Build step 2 - Create root collection - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:241 Build step 3 - accumulate obs and var axes - started
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:245 Build step 3 - axis accumulation complete
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:255 Build step 3 - accumulate obs and var axes - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:94 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:266 Build step 4 - Populate X layers - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 homo_sapiens: create X layers
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 mus_musculus: create X layers
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_1.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_0.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_3.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_3.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_1.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_0.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_2.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_2.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.69s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 3.47s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.89s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 3.30s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 3.80s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 3.07s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/mus_musculus_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 3.95s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 3.33s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/h5ads/homo_sapiens_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for homo_sapiens - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for homo_sapiens - finish
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for mus_musculus - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for mus_musculus - finish
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:277 Build step 4 - Populate X layers - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:286 Build step 5 - Save axis and summary info - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 homo_sapiens: writing obs dataframe
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 homo_sapiens: writing var dataframe
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 mus_musculus: writing obs dataframe
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 mus_musculus: writing var dataframe
INFO     cellxgene_census_builder.build_soma.datasets:datasets.py:70 Creating dataset_manifest
INFO     cellxgene_census_builder.build_soma.summary_cell_counts:summary_cell_counts.py:18 Creating census_summary_cell_counts
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:17 Creating census summary
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:43 Create census organisms dataframe
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:298 Build step 5 - Save axis and summary info - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru0/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:113 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.validate_soma:validate_soma.py:1042 Validation of SOMA objects - start
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_directory_structure [exit, 0.00s]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_relative_path [exit, 0.03s]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_axis_dataframes_schema [exit, 0.03s]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_manifest_contents [exit, 0.00s]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence_general [exit, 0.09s]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.26s]: homo_sapiens, raw, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.32s]: homo_sapiens_0
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.27s]: homo_sapiens, normalized, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.24s]: mus_musculus, raw, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.49s]: homo_sapiens_1
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.29s]: mus_musculus, normalized, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.28s]: homo_sapiens_2
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.09s]: homo_sapiens_3
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.10s]: mus_musculus_0
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.09s]: mus_musculus_1
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.09s]: mus_musculus_2
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.13s]: mus_musculus_3
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 2.49s]: mus_musculus_3
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 2.57s]: mus_musculus_2
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 2.77s]: mus_musculus_0
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 2.77s]: mus_musculus_1
_____________ test_base_builder_creation[True-census_build_args1] ______________
[gw3] linux -- Python 3.11.9 /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/bin/python3.11

datasets = [Dataset(dataset_id='homo_sapiens_0', dataset_asset_h5ad_uri='mock', dataset_version_id='homo_sapiens_0_v0', dataset_h...lesize=-1, cell_count=-1, mean_genes_per_cell=-1.0, schema_version='', dataset_total_cell_count=4, soma_joinid=5), ...]
census_build_args = CensusBuildArgs(working_dir=PosixPath('/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1'), co...t_prefix='census-builder-', user_agent_environment='unknown', manifest=None, test_first_n=0), state=CensusBuildState())
validate = True, setup = None

    @pytest.mark.parametrize(
        "census_build_args",
        (
            {
                "consolidate": False,
                "build_tag": "test_tag",
                "verbose": 0,
                "max_worker_processes": 1,
            },
            {
                "consolidate": True,
                "build_tag": "test_tag",
                "verbose": 1,
                "max_worker_processes": 1,
            },
        ),
        indirect=True,
    )
    @pytest.mark.parametrize("validate", (True, False))
    def test_base_builder_creation(
        datasets: list[Dataset],
        census_build_args: CensusBuildArgs,
        validate: bool,
        setup: None,
    ) -> None:
        """
        Runs the builder, queries the census and performs a set of base assertions.
        """
    
        def proxy_create_dask_client(
            *args: CensusBuildArgs,
            **kwargs: Any,
        ) -> dask.distributed.Client:
            from cellxgene_census_builder.build_soma.mp import create_dask_client
    
            kwargs["processes"] = False
            kwargs["n_workers"] = 1
            kwargs.pop("threads_per_worker")
            return create_dask_client(*args, **kwargs)
    
        # proxy psutil.virtual_memory to return 1/2 of the actual memory. There
        # are repeated cases where the test runners OOM, and this helps avoid it.
        memstats = psutil.virtual_memory()
        memstats = memstats._replace(total=int(memstats.total // 2))
    
        def proxy_psutil_virtual_memory() -> psutil._pslinux.svmem:
            return memstats
    
        with (
            patch("cellxgene_census_builder.build_soma.build_soma.prepare_file_system"),
            patch("cellxgene_census_builder.build_soma.build_soma.build_step1_get_source_datasets", return_value=datasets),
            patch(
                "cellxgene_census_builder.build_soma.build_soma.create_dask_client", side_effect=proxy_create_dask_client
            ),
            patch("psutil.virtual_memory", side_effect=proxy_psutil_virtual_memory),
        ):
            process_init(census_build_args)
>           return_value = build(census_build_args, validate=validate)

tests/test_builder.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/cellxgene_census_builder/build_soma/build_soma.py:121: in build
    assert f.result()
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/client.py:328: in result
    return self.client.sync(self._result, callback_timeout=timeout)
src/cellxgene_census_builder/logging.py:114: in wrapper
    result: R = func(*args, **kwargs)
src/cellxgene_census_builder/build_soma/validate_soma.py:925: in validate_internal_consistency
    exp.obs.read(column_names=["soma_joinid", "nnz", "dataset_id", "n_measured_vars"]).concat().to_pandas()
pyarrow/array.pxi:883: in pyarrow.lib._PandasConvertible.to_pandas
    ???
pyarrow/table.pxi:4251: in pyarrow.lib.Table._to_pandas
    ???
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/pyarrow/pandas_compat.py:777: in table_to_dataframe
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/pyarrow/pandas_compat.py:1131: in _table_to_blocks
    result = pa.lib.table_to_blocks(options, block_table, categories,
pyarrow/table.pxi:3120: in pyarrow.lib.table_to_blocks
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowIndexError: Index 0 out of bounds

pyarrow/error.pxi:91: ArrowIndexError
----------------------------- Captured stderr call -----------------------------
2024-08-21 22:28:07,873 - distributed.worker - WARNING - Compute Failed
Key:       validate_internal_consistency-3253235b-786d-4b41-90a0-fa8685ff46b3
Function:  validate_internal_consistency
args:      ('/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma', [ExperimentSpecification(name='homo_sapiens', label='Homo sapiens', anndata_cell_filter_spec={'organism_ontology_term_id': 'NCBITaxon:9606', 'assay_ontology_term_ids': ['EFO:0003755', 'EFO:0008440', 'EFO:0008640', 'EFO:0008641', 'EFO:0008643', 'EFO:0008661', 'EFO:0008669', 'EFO:0008673', 'EFO:0008675', 'EFO:0008679', 'EFO:0008694', 'EFO:0008697', 'EFO:0008708', 'EFO:0008710', 'EFO:0008718', 'EFO:0008720', 'EFO:0008722', 'EFO:0008735', 'EFO:0008747', 'EFO:0008748', 'EFO:0008752', 'EFO:0008753', 'EFO:0008756', 'EFO:0008763', 'EFO:0008780', 'EFO:0008796', 'EFO:0008797', 'EFO:0008824', 'EFO:0008825', 'EFO:0008826', 'EFO:0008850', 'EFO:0008859', 'EFO:0008863', 'EFO:0008868', 'EFO:0008869', 'EFO:0008877', 'EFO:0008896', 'EFO:0008897', 'EFO:0008898', 'EFO:0008903', 'EFO:0008908', 'EFO:0008919', 'EFO:0008929', 'EFO:0008930', 'EFO:0008931', 'EFO:0008937', 'EFO:0008941', 'EFO:0008945', 'EFO:0008953', 'EFO
kwargs:    {}
Exception: "ArrowIndexError('Index 0 out of bounds')"

2024-08-21 22:28:07,955 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 6))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,987 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute('all-part-validate_X_layers_presence-all-aggregate-0787ae75598419ec6dd462d158354bd2')" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,987 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 7))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 0))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 5))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 4))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 3))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute('all-part-validate_X_layers_normalized-all-aggregate-17a501593321ca731d68a3cfafb9030f')" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 1))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('_validate_axis_dataframes-722439e30ab9a6069798161817cb9e77', 0))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('validate_X_layers_raw_contents-all-part-c3a3f9bfb238dcdbca04a5d3608eb998', 2))" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
2024-08-21 22:28:07,988 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute('all-part-validate_X_layers_has_unique_coords-all-aggregate-0a582d5e309ceaeb75defcf2387841aa')" coro=<Worker.execute() done, defined at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/distributed/worker_state_machine.py:3615>> ended with CancelledError
------------------------------ Captured log call -------------------------------
INFO     cellxgene_census_builder.build_soma.mp:mp.py:58 Dask client created: <Client: 'inproc://172.31.29.102/2814823/11' processes=1 threads=32>
INFO     cellxgene_census_builder.build_soma.mp:mp.py:59 Dask client using cluster: LocalCluster(1b8ed91c, 'inproc://172.31.29.102/2814823/11', workers=1, threads=32)
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:195 Build step 2 - Create root collection - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 homo_sapiens: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 mus_musculus: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:203 Build step 2 - Create root collection - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:241 Build step 3 - accumulate obs and var axes - started
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_3/homo_sapiens - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_1/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_2/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_2
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_0/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_3/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_3
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_1/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_1/homo_sapiens - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_3/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_2/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_2
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_0/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_0/homo_sapiens - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_3/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_3
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_2/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_2/homo_sapiens - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_0/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_1/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_1
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:245 Build step 3 - axis accumulation complete
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:255 Build step 3 - accumulate obs and var axes - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:94 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:266 Build step 4 - Populate X layers - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 homo_sapiens: create X layers
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 mus_musculus: create X layers
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_1.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_3.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_0.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_0.h5ad, 0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 0.53s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_3.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_0.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 0.65s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_3.h5ad, 0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_2.h5ad, 0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_2.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_1.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_2.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.64s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.03s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.70s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.84s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.83s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/mus_musculus_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.42s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/h5ads/homo_sapiens_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for homo_sapiens - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for homo_sapiens - finish
DEBUG    cellxgene_census_builder.util:util.py:144 Resource use: pid=2814823, threads=393 (max=393), maps=2798 (max=2798), page-faults(cumm)=0 uss=278646784, rss=428601344
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for mus_musculus - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for mus_musculus - finish
DEBUG    cellxgene_census_builder.util:util.py:144 Resource use: pid=2814823, threads=457 (max=457), maps=2926 (max=2926), page-faults(cumm)=0 uss=279449600, rss=429293568
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:277 Build step 4 - Populate X layers - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:286 Build step 5 - Save axis and summary info - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 homo_sapiens: writing obs dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:177 experiment homo_sapiens obs = (16, 30)
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 homo_sapiens: writing var dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:203 experiment homo_sapiens var = (5, 6)
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 mus_musculus: writing obs dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:177 experiment mus_musculus obs = (16, 30)
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 mus_musculus: writing var dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:203 experiment mus_musculus var = (5, 6)
INFO     cellxgene_census_builder.build_soma.datasets:datasets.py:70 Creating dataset_manifest
INFO     cellxgene_census_builder.build_soma.summary_cell_counts:summary_cell_counts.py:18 Creating census_summary_cell_counts
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:17 Creating census summary
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:43 Create census organisms dataframe
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:298 Build step 5 - Save axis and summary info - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:113 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:86 Consolidate: found 24 TileDB objects to consolidate
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:89 Consolidate: 24 consolidation jobs queued
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 1.68 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 1.68 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 1.71 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.03 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.06 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.08 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/homo_sapiens/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 1.68 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 3.29 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.07 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.04 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.03 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.04 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data/mus_musculus/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_data
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/datasets
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/organisms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.03 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/datasets
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.03 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/organisms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/summary
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/summary
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/summary_cell_counts
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.04 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Tru1/test_tag/soma/census_info/summary_cell_counts
INFO     cellxgene_census_builder.build_soma.validate_soma:validate_soma.py:1042 Validation of SOMA objects - start
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 validate_directory_structure [enter]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_directory_structure [exit, 0.00s]
DEBUG    cellxgene_census_builder.util:util.py:144 Resource use: pid=2814823, threads=329 (max=457), maps=2678 (max=2926), page-faults(cumm)=0 uss=294547456, rss=444731392
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 validate_relative_path [enter]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_relative_path [exit, 0.03s]
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 validate_axis_dataframes_schema [enter]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_axis_dataframes_schema [exit, 0.04s]
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 validate_manifest_contents [enter]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 validate_manifest_contents [exit, 0.00s]
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: mus_musculus_3
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: mus_musculus_0
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: homo_sapiens_1
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: homo_sapiens_2
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: mus_musculus_1
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: mus_musculus_2
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: homo_sapiens_0
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_raw_contents [enter]: homo_sapiens_3
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence_general [enter]
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 validate_internal_consistency [enter]
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence_general [exit, 0.09s]
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_axis_dataframes [enter]: homo_sapiens_0
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: homo_sapiens_0
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_normalized [enter]: homo_sapiens rows [0, 32000)
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_has_unique_coords [enter]: homo_sapiens, raw, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.20s]: homo_sapiens, raw, rows [0, 96000)
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_has_unique_coords [enter]: homo_sapiens, normalized, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.33s]: homo_sapiens_0
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: homo_sapiens_1
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.25s]: homo_sapiens, normalized, rows [0, 96000)
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_has_unique_coords [enter]: mus_musculus, raw, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.19s]: mus_musculus, raw, rows [0, 96000)
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_has_unique_coords [enter]: mus_musculus, normalized, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.39s]: homo_sapiens_1
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: homo_sapiens_2
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_has_unique_coords [exit, 0.18s]: mus_musculus, normalized, rows [0, 96000)
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.20s]: homo_sapiens_2
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: homo_sapiens_3
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.08s]: homo_sapiens_3
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: mus_musculus_0
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.08s]: mus_musculus_0
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: mus_musculus_1
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.08s]: mus_musculus_1
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: mus_musculus_2
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.08s]: mus_musculus_2
DEBUG    cellxgene_census_builder.build_soma.validate_soma:logging.py:112 _validate_X_layers_presence [enter]: mus_musculus_3
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_presence [exit, 0.08s]: mus_musculus_3
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 1.83s]: mus_musculus_3
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 1.88s]: mus_musculus_2
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 2.00s]: mus_musculus_0
INFO     cellxgene_census_builder.build_soma.validate_soma:logging.py:116 _validate_X_layers_raw_contents [exit, 2.05s]: mus_musculus_1
_____________ test_base_builder_creation[False-census_build_args0] _____________
[gw3] linux -- Python 3.11.9 /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/bin/python3.11

datasets = [Dataset(dataset_id='homo_sapiens_0', dataset_asset_h5ad_uri='mock', dataset_version_id='homo_sapiens_0_v0', dataset_h...lesize=-1, cell_count=-1, mean_genes_per_cell=-1.0, schema_version='', dataset_total_cell_count=4, soma_joinid=5), ...]
census_build_args = CensusBuildArgs(working_dir=PosixPath('/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0'), co...t_prefix='census-builder-', user_agent_environment='unknown', manifest=None, test_first_n=0), state=CensusBuildState())
validate = False, setup = None

    @pytest.mark.parametrize(
        "census_build_args",
        (
            {
                "consolidate": False,
                "build_tag": "test_tag",
                "verbose": 0,
                "max_worker_processes": 1,
            },
            {
                "consolidate": True,
                "build_tag": "test_tag",
                "verbose": 1,
                "max_worker_processes": 1,
            },
        ),
        indirect=True,
    )
    @pytest.mark.parametrize("validate", (True, False))
    def test_base_builder_creation(
        datasets: list[Dataset],
        census_build_args: CensusBuildArgs,
        validate: bool,
        setup: None,
    ) -> None:
        """
        Runs the builder, queries the census and performs a set of base assertions.
        """
    
        def proxy_create_dask_client(
            *args: CensusBuildArgs,
            **kwargs: Any,
        ) -> dask.distributed.Client:
            from cellxgene_census_builder.build_soma.mp import create_dask_client
    
            kwargs["processes"] = False
            kwargs["n_workers"] = 1
            kwargs.pop("threads_per_worker")
            return create_dask_client(*args, **kwargs)
    
        # proxy psutil.virtual_memory to return 1/2 of the actual memory. There
        # are repeated cases where the test runners OOM, and this helps avoid it.
        memstats = psutil.virtual_memory()
        memstats = memstats._replace(total=int(memstats.total // 2))
    
        def proxy_psutil_virtual_memory() -> psutil._pslinux.svmem:
            return memstats
    
        with (
            patch("cellxgene_census_builder.build_soma.build_soma.prepare_file_system"),
            patch("cellxgene_census_builder.build_soma.build_soma.build_step1_get_source_datasets", return_value=datasets),
            patch(
                "cellxgene_census_builder.build_soma.build_soma.create_dask_client", side_effect=proxy_create_dask_client
            ),
            patch("psutil.virtual_memory", side_effect=proxy_psutil_virtual_memory),
        ):
            process_init(census_build_args)
            return_value = build(census_build_args, validate=validate)
    
            # return_value = 0 means that the build succeeded
            assert return_value == 0
    
            # Query the census and do assertions
            with soma.Collection.open(
                uri=census_build_args.soma_path.as_posix(),
                context=soma.options.SOMATileDBContext(tiledb_config={"vfs.s3.region": "us-west-2"}),
            ) as census:
                # There are 16 cells in total (4 in each dataset). They all belong to homo_sapiens
>               human_obs = census[CENSUS_DATA_NAME]["homo_sapiens"]["obs"].read().concat().to_pandas()

tests/test_builder.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/array.pxi:883: in pyarrow.lib._PandasConvertible.to_pandas
    ???
pyarrow/table.pxi:4251: in pyarrow.lib.Table._to_pandas
    ???
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/pyarrow/pandas_compat.py:777: in table_to_dataframe
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/pyarrow/pandas_compat.py:1131: in _table_to_blocks
    result = pa.lib.table_to_blocks(options, block_table, categories,
pyarrow/table.pxi:3120: in pyarrow.lib.table_to_blocks
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowIndexError: Index 0 out of bounds

pyarrow/error.pxi:91: ArrowIndexError
------------------------------ Captured log call -------------------------------
INFO     cellxgene_census_builder.build_soma.mp:mp.py:58 Dask client created: <Client: 'inproc://172.31.29.102/2814823/21' processes=1 threads=32>
INFO     cellxgene_census_builder.build_soma.mp:mp.py:59 Dask client using cluster: LocalCluster(bbaaa0f6, 'inproc://172.31.29.102/2814823/21', workers=1, threads=32)
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:195 Build step 2 - Create root collection - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 homo_sapiens: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 mus_musculus: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:203 Build step 2 - Create root collection - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:241 Build step 3 - accumulate obs and var axes - started
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:245 Build step 3 - axis accumulation complete
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:255 Build step 3 - accumulate obs and var axes - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:94 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:266 Build step 4 - Populate X layers - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 homo_sapiens: create X layers
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 mus_musculus: create X layers
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_2.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_3.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 0.68s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_3.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_0.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_1.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_0.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 0.89s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_2.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_1.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.78s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.95s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.82s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.16s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/mus_musculus_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.57s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.87s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/h5ads/homo_sapiens_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for homo_sapiens - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for homo_sapiens - finish
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for mus_musculus - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for mus_musculus - finish
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:277 Build step 4 - Populate X layers - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:286 Build step 5 - Save axis and summary info - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 homo_sapiens: writing obs dataframe
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 homo_sapiens: writing var dataframe
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 mus_musculus: writing obs dataframe
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 mus_musculus: writing var dataframe
INFO     cellxgene_census_builder.build_soma.datasets:datasets.py:70 Creating dataset_manifest
INFO     cellxgene_census_builder.build_soma.summary_cell_counts:summary_cell_counts.py:18 Creating census_summary_cell_counts
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:17 Creating census summary
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:43 Create census organisms dataframe
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:298 Build step 5 - Save axis and summary info - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal0/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:113 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:124 Validation & consolidation complete.
INFO     cellxgene_census_builder.build_soma.mp:mp.py:86 Dask cluster shut down
_____________ test_base_builder_creation[False-census_build_args1] _____________
[gw3] linux -- Python 3.11.9 /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/bin/python3.11

datasets = [Dataset(dataset_id='homo_sapiens_0', dataset_asset_h5ad_uri='mock', dataset_version_id='homo_sapiens_0_v0', dataset_h...lesize=-1, cell_count=-1, mean_genes_per_cell=-1.0, schema_version='', dataset_total_cell_count=4, soma_joinid=5), ...]
census_build_args = CensusBuildArgs(working_dir=PosixPath('/tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1'), co...t_prefix='census-builder-', user_agent_environment='unknown', manifest=None, test_first_n=0), state=CensusBuildState())
validate = False, setup = None

    @pytest.mark.parametrize(
        "census_build_args",
        (
            {
                "consolidate": False,
                "build_tag": "test_tag",
                "verbose": 0,
                "max_worker_processes": 1,
            },
            {
                "consolidate": True,
                "build_tag": "test_tag",
                "verbose": 1,
                "max_worker_processes": 1,
            },
        ),
        indirect=True,
    )
    @pytest.mark.parametrize("validate", (True, False))
    def test_base_builder_creation(
        datasets: list[Dataset],
        census_build_args: CensusBuildArgs,
        validate: bool,
        setup: None,
    ) -> None:
        """
        Runs the builder, queries the census and performs a set of base assertions.
        """
    
        def proxy_create_dask_client(
            *args: CensusBuildArgs,
            **kwargs: Any,
        ) -> dask.distributed.Client:
            from cellxgene_census_builder.build_soma.mp import create_dask_client
    
            kwargs["processes"] = False
            kwargs["n_workers"] = 1
            kwargs.pop("threads_per_worker")
            return create_dask_client(*args, **kwargs)
    
        # proxy psutil.virtual_memory to return 1/2 of the actual memory. There
        # are repeated cases where the test runners OOM, and this helps avoid it.
        memstats = psutil.virtual_memory()
        memstats = memstats._replace(total=int(memstats.total // 2))
    
        def proxy_psutil_virtual_memory() -> psutil._pslinux.svmem:
            return memstats
    
        with (
            patch("cellxgene_census_builder.build_soma.build_soma.prepare_file_system"),
            patch("cellxgene_census_builder.build_soma.build_soma.build_step1_get_source_datasets", return_value=datasets),
            patch(
                "cellxgene_census_builder.build_soma.build_soma.create_dask_client", side_effect=proxy_create_dask_client
            ),
            patch("psutil.virtual_memory", side_effect=proxy_psutil_virtual_memory),
        ):
            process_init(census_build_args)
            return_value = build(census_build_args, validate=validate)
    
            # return_value = 0 means that the build succeeded
            assert return_value == 0
    
            # Query the census and do assertions
            with soma.Collection.open(
                uri=census_build_args.soma_path.as_posix(),
                context=soma.options.SOMATileDBContext(tiledb_config={"vfs.s3.region": "us-west-2"}),
            ) as census:
                # There are 16 cells in total (4 in each dataset). They all belong to homo_sapiens
>               human_obs = census[CENSUS_DATA_NAME]["homo_sapiens"]["obs"].read().concat().to_pandas()

tests/test_builder.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/array.pxi:883: in pyarrow.lib._PandasConvertible.to_pandas
    ???
pyarrow/table.pxi:4251: in pyarrow.lib.Table._to_pandas
    ???
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/pyarrow/pandas_compat.py:777: in table_to_dataframe
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
../../../../miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/pyarrow/pandas_compat.py:1131: in _table_to_blocks
    result = pa.lib.table_to_blocks(options, block_table, categories,
pyarrow/table.pxi:3120: in pyarrow.lib.table_to_blocks
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowIndexError: Index 0 out of bounds

pyarrow/error.pxi:91: ArrowIndexError
----------------------------- Captured stderr call -----------------------------
2024-08-21 22:28:39,460 - distributed.scheduler - WARNING - Removing worker 'inproc://172.31.29.102/2814823/34' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'_consolidate_tiledb_object-13b93bb0c981b2836a512c1339dbc2d5'} (stimulus_id='handle-worker-cleanup-1724279319.4603403')
------------------------------ Captured log call -------------------------------
INFO     cellxgene_census_builder.build_soma.mp:mp.py:58 Dask client created: <Client: 'inproc://172.31.29.102/2814823/31' processes=1 threads=32>
INFO     cellxgene_census_builder.build_soma.mp:mp.py:59 Dask client using cluster: LocalCluster(980f3439, 'inproc://172.31.29.102/2814823/31', workers=1, threads=32)
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:195 Build step 2 - Create root collection - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 homo_sapiens: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:147 mus_musculus: create experiment at /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:203 Build step 2 - Create root collection - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:241 Build step 3 - accumulate obs and var axes - started
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_1/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_3/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_3
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_3/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_3
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_1/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_2/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_2
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_2/homo_sapiens - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_0/homo_sapiens - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_2/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_0/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_3/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_1/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_2/homo_sapiens - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 homo_sapiens - H5AD has no data after filtering, skipping mus_musculus_2
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_0/mus_musculus - found 0 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:289 mus_musculus - H5AD has no data after filtering, skipping homo_sapiens_0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_3/homo_sapiens - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 mus_musculus_0/mus_musculus - found 4 cells
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:285 homo_sapiens_1/homo_sapiens - found 4 cells
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:245 Build step 3 - axis accumulation complete
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:255 Build step 3 - accumulate obs and var axes - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:94 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:266 Build step 4 - Populate X layers - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 homo_sapiens: create X layers
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:209 mus_musculus: create X layers
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_3.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_3.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_3.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 0.59s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_3.h5ad, 0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_0.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 0.76s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_3.h5ad, 0
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_2.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_1.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_1.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_2.h5ad, 0, chunk 0 of 1
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:logging.py:112 dispatch_X_chunk [enter]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:536 processing X /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_0.h5ad, 0, chunk 0 of 1
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 0.75s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.31s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.58s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/mus_musculus_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.68s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_1.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 1.79s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_2.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:logging.py:116 dispatch_X_chunk [exit, 2.04s]: /tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/h5ads/homo_sapiens_0.h5ad, 0
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for homo_sapiens - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for homo_sapiens - finish
DEBUG    cellxgene_census_builder.util:util.py:144 Resource use: pid=2814823, threads=396 (max=457), maps=2803 (max=2926), page-faults(cumm)=0 uss=313647104, rss=462872576
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:231 Save presence matrix for mus_musculus - start
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:259 Save presence matrix for mus_musculus - finish
DEBUG    cellxgene_census_builder.util:util.py:144 Resource use: pid=2814823, threads=460 (max=460), maps=2931 (max=2931), page-faults(cumm)=0 uss=314449920, rss=463548416
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:277 Build step 4 - Populate X layers - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:286 Build step 5 - Save axis and summary info - started
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 homo_sapiens: writing obs dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:177 experiment homo_sapiens obs = (16, 30)
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 homo_sapiens: writing var dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:203 experiment homo_sapiens var = (5, 6)
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:159 mus_musculus: writing obs dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:177 experiment mus_musculus obs = (16, 30)
INFO     cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:183 mus_musculus: writing var dataframe
DEBUG    cellxgene_census_builder.build_soma.experiment_builder:experiment_builder.py:203 experiment mus_musculus var = (5, 6)
INFO     cellxgene_census_builder.build_soma.datasets:datasets.py:70 Creating dataset_manifest
INFO     cellxgene_census_builder.build_soma.summary_cell_counts:summary_cell_counts.py:18 Creating census_summary_cell_counts
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:17 Creating census summary
INFO     cellxgene_census_builder.build_soma.census_summary:census_summary.py:43 Create census organisms dataframe
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:298 Build step 5 - Save axis and summary info - finished
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:329 tiledb_soma_1969_work_around: deleting bounding box from file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:113 Scaling cluster to 1 workers.
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:86 Consolidate: found 24 TileDB objects to consolidate
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:89 Consolidate: 24 consolidation jobs queued
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 1.69 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 1.72 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 3.34 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.05 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.04 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.05 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/homo_sapiens/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 3.44 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/normalized
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 3.42 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X/raw
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/var
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.05 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/X
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.04 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA/feature_dataset_presence_matrix
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms/RNA
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/obs
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.05 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus/ms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.02 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data/mus_musculus
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_data
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/datasets
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/organisms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.03 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/organisms
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/summary
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.05 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/datasets
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/summary_cell_counts
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.03 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/summary
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.03 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info/summary_cell_counts
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:178 Consolidate[vacuum=True] start, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info
INFO     cellxgene_census_builder.build_soma.consolidate:consolidate.py:187 Consolidate[vacuum=True] finish, 0.01 seconds, uri=file:///tmp/pytest-of-ubuntu/pytest-62/popen-gw3/test_base_builder_creation_Fal1/test_tag/soma/census_info
INFO     cellxgene_census_builder.build_soma.build_soma:build_soma.py:124 Validation & consolidation complete.
INFO     cellxgene_census_builder.build_soma.mp:mp.py:86 Dask cluster shut down
=============================== warnings summary ===============================
tests/anndata/test_anndata.py::test_empty_estimated_density
tests/anndata/test_anndata.py::test_empty_estimated_density
tests/anndata/test_anndata.py::test_open_anndata_raw_X
  /home/ubuntu/miniforge3/envs/census-local-tiledbsoma/lib/python3.11/site-packages/anndata/_core/aligned_df.py:67: ImplicitModificationWarning: Transforming to str index.
    warnings.warn("Transforming to str index.", ImplicitModificationWarning)

tests/test_axis_stats.py::test_get_obs_stats[csr]
tests/test_axis_stats.py::test_get_obs_stats[csc]
  /home/ubuntu/github/cellxgene-census/tools/cellxgene_census_builder/src/cellxgene_census_builder/build_soma/stats.py:22: RuntimeWarning: invalid value encountered in divide
    raw_mean_nnz = raw_sum / nnz

tests/test_axis_stats.py::test_get_obs_stats[csr]
tests/test_axis_stats.py::test_get_obs_stats[csc]
  /home/ubuntu/github/cellxgene-census/tools/cellxgene_census_builder/tests/test_axis_stats.py:19: RuntimeWarning: invalid value encountered in divide
    raw_mean_nnz = X.sum(axis=1, dtype=np.float64).A1 / X.getnnz(axis=1)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_builder.py::test_base_builder_creation[True-census_build_args0] - AssertionError: homo_sapiens_0/homo_sapiens obs length mismatch
FAILED tests/test_builder.py::test_base_builder_creation[True-census_build_args1] - pyarrow.lib.ArrowIndexError: Index 0 out of bounds
FAILED tests/test_builder.py::test_base_builder_creation[False-census_build_args0] - pyarrow.lib.ArrowIndexError: Index 0 out of bounds
FAILED tests/test_builder.py::test_base_builder_creation[False-census_build_args1] - pyarrow.lib.ArrowIndexError: Index 0 out 

Conda env info

$ conda list
# packages in environment at /home/ubuntu/miniforge3/envs/census-local-tiledbsoma:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aiobotocore               2.13.2                   pypi_0    pypi
aiohappyeyeballs          2.4.0                    pypi_0    pypi
aiohttp                   3.10.2                   pypi_0    pypi
aioitertools              0.11.0                   pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
anndata                   0.10.6                   pypi_0    pypi
array-api-compat          1.8                      pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
botocore                  1.34.131                 pypi_0    pypi
bottleneck                1.4.0                    pypi_0    pypi
bzip2                     1.0.8                h4bc722e_7    conda-forge
ca-certificates           2024.7.4             hbcca054_0    conda-forge
cellxgene-census          1.15.0                   pypi_0    pypi
cellxgene-census-builder  1.15.1.dev17+g8ed87c4.d20240819          pypi_0    pypi
cellxgene-ontology-guide  1.0.0                    pypi_0    pypi
certifi                   2024.7.4                 pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
contourpy                 1.2.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
cython                    3.0.11                   pypi_0    pypi
dask                      2024.3.1                 pypi_0    pypi
distributed               2024.3.1                 pypi_0    pypi
execnet                   2.1.1                    pypi_0    pypi
fonttools                 4.53.1                   pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.3.1                 pypi_0    pypi
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.42                   pypi_0    pypi
h5py                      3.11.0                   pypi_0    pypi
idna                      3.7                      pypi_0    pypi
importlib-metadata        8.3.0                    pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.4.2                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
legacy-api-wrap           1.4                      pypi_0    pypi
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 14.1.0               h77fa898_0    conda-forge
libgomp                   14.1.0               h77fa898_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
llvmlite                  0.42.0                   pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.9.2                    pypi_0    pypi
msgpack                   1.0.8                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
natsort                   8.4.0                    pypi_0    pypi
ncurses                   6.5                  h59595ed_0    conda-forge
networkx                  3.3                      pypi_0    pypi
numba                     0.59.1                   pypi_0    pypi
numexpr                   2.10.1                   pypi_0    pypi
numpy                     1.26.4                   pypi_0    pypi
openssl                   3.3.1                h4bc722e_2    conda-forge
owlready2                 0.44                     pypi_0    pypi
packaging                 24.1                     pypi_0    pypi
pandas                    2.2.1                    pypi_0    pypi
partd                     1.4.2                    pypi_0    pypi
patsy                     0.5.6                    pypi_0    pypi
pillow                    10.4.0                   pypi_0    pypi
pip                       24.2               pyhd8ed1ab_0    conda-forge
pluggy                    1.5.0                    pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
pyarrow                   15.0.2                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pynndescent               0.5.13                   pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pytest                    8.3.2                    pypi_0    pypi
pytest-xdist              3.6.1                    pypi_0    pypi
python                    3.11.9          hb806964_0_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h8228510_1    conda-forge
requests                  2.32.0                   pypi_0    pypi
s3fs                      2024.3.1                 pypi_0    pypi
scanpy                    1.10.2                   pypi_0    pypi
scikit-learn              1.5.1                    pypi_0    pypi
scipy                     1.12.0                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
session-info              1.0.0                    pypi_0    pypi
setuptools                72.1.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0                   pypi_0    pypi
smmap                     5.0.1                    pypi_0    pypi
somacore                  1.0.14                   pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
statsmodels               0.14.2                   pypi_0    pypi
stdlib-list               0.10.0                   pypi_0    pypi
tblib                     3.0.0                    pypi_0    pypi
threadpoolctl             3.5.0                    pypi_0    pypi
tiledb                    0.31.1                   pypi_0    pypi
tiledbsoma                1.12.0rc0.post94.dev1270126378          pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4.1                    pypi_0    pypi
tqdm                      4.66.5                   pypi_0    pypi
typing-extensions         4.10.0                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
umap-learn                0.5.6                    pypi_0    pypi
urllib3                   2.2.2                    pypi_0    pypi
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.16.0                   pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.9.4                    pypi_0    pypi
zict                      3.0.0                    pypi_0    pypi
zipp                      3.20.0                   pypi_0    pypi

[johnkerl]

Internal tracking [sc-47660]