Accessing parquet files from Planetary Computer Docker image

[glaroc]

Hi, I'm trying to access GBIF data locally using Dask from the Planetary Computer Docker image. If I run the code on the Jupyter Hub on the PC, it works. However, running the same code within the Docker gives me an error:

import pystac_client
from dask_gateway import GatewayCluster
import planetary_computer
import dask.dataframe as dd
from dotenv import load_dotenv
import os

load_dotenv() 
catalog = pystac_client.Client.open(
    "https://planetarycomputer.microsoft.com/api/stac/v1",
)

cluster = GatewayCluster(
 "https://pccompute.westeurope.cloudapp.azure.com/compute/services/dask-gateway",
 proxy_address="gateway://pccompute-dask.westeurope.cloudapp.azure.com:80",
 auth="jupyterhub",
 image="mcr.microsoft.com/planetary-computer/python:latest"
)
cluster.scale(20)
client = cluster.get_client()

search = catalog.search(collections=["gbif"])
items = search.get_all_items()
items = {x.id: x for x in items}
item = list(items.values())[0]

signed_asset = planetary_computer.sign(item).assets["data"]
#signed_asset.extra_fields["table:storage_options"]

filters=[('species', '==', 'Acer saccharum'), ('decimallatitude', '<', 62.3), ('decimallatitude', '>', 45.2), ('decimallongitude', '>', -77.3), ('decimallongitude', '<', -60.3)]

df = dd.read_parquet(
    signed_asset.href,
    storage_options=signed_asset.extra_fields["table:storage_options"],
    filters=filters,
    dataset={"require_extension": None},
    engine="pyarrow"
)

res=df.compute()

This gives me:

TypeError: dataset() got an unexpected keyword argument 'require_extension'

The variable JUPYTERHUB_API_TOKEN is set properly and the Dask versions seem to the same within the Docker and on the Hub. Is there something different we should do to run this from the Docker image ?

[TomAugspurger]

Can you check what version of Dask you're using locally? That was added somewhere in the 2021.11 or 2021.12 range IIRC. See dask/dask#8491.

Edit: Oh, sorry, I should have read more closely. You say you're using the docker image. Can you share your docker run and the version of the image you're using?

[glaroc]

I'm setting the environment variables with docker-compose. Here is the Dockerfile.

FROM mcr.microsoft.com/planetary-computer/python:latest
SHELL ["conda", "run", "-n", "notebook", "/bin/bash", "-c"]
RUN pip3 install appdirs geojson_rewind geomet requests_cache matplotlib python-dotenv

# The code to run when container is started:
RUN python
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "notebook"]

compose.yml

version: '2.3'
services:
  pc-dock:
    build: .
    image: "planetary-computer-pygbif"
    user: root
    command:
      - python
    volumes:
      - ${HOST_PATH}/scripts/:/home/jovyan/scripts/
      - ${HOST_PATH}/output:/output/:rw
    environment:
      - GRANT_SUDO=yes
      - JUPYTERHUB_API_TOKEN=${JUPYTERHUB_API_TOKEN}
      - DASK_GATEWAY__AUTH__TYPE="jupyterhub"
      - DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE="mcr.microsoft.com/planetary-computer/python:latest"
      - DASK_GATEWAY__ADDRESS="https://pccompute.westeurope.cloudapp.azure.com/compute/services/dask-gateway"
      - DASK_GATEWAY__PROXY_ADDRESS="gateway://pccompute-dask.westeurope.cloudapp.azure.com:80"
    ports:
      - "8888:8888"

And then I just do.

docker-compose run pc-dock python

[glaroc]

Dask within the Docker is the same as on the hub (2021.11.2).

[TomAugspurger]

Thanks for the reproducible example. That did finish for me

>>> df = dd.read_parquet(
...     signed_asset.href,
...     storage_options=signed_asset.extra_fields["table:storage_options"],
...     filters=filters,
...     dataset={"require_extension": None},
...     engine="pyarrow"
... )
>>>
>>> res=df.compute()
>>>
>>> res.head()
       gbifid                            datasetkey                                       occurrenceid  kingdom        phylum  ... typestatus establishmentmeans           lastinterpreted                                         mediatype                 issue
0  1883499846  50c9509d-22c7-4a22-a47d-8c48425ef4a7  https://www.inaturalist.org/observations/14254831  Plantae  Tracheophyta  ...       None               None  2022-02-25T19:32:40.522Z                          [StillImage, StillImage]  [COORDINATE_ROUNDED]
0  2901723277  50c9509d-22c7-4a22-a47d-8c48425ef4a7  https://www.inaturalist.org/observations/63204188  Plantae  Tracheophyta  ...       None               None  2022-02-25T19:33:56.779Z  [StillImage, StillImage, StillImage, StillImage]  [COORDINATE_ROUNDED]
1  3455262240  50c9509d-22c7-4a22-a47d-8c48425ef4a7  https://www.inaturalist.org/observations/91582103  Plantae  Tracheophyta  ...       None               None  2022-02-25T19:47:21.872Z                                      [StillImage]  [COORDINATE_ROUNDED]
2  3455449765  50c9509d-22c7-4a22-a47d-8c48425ef4a7  https://www.inaturalist.org/observations/10284...  Plantae  Tracheophyta  ...       None               None  2022-02-25T19:58:30.468Z                                      [StillImage]  [COORDINATE_ROUNDED]
0  3455162018  50c9509d-22c7-4a22-a47d-8c48425ef4a7  https://www.inaturalist.org/observations/77849863  Plantae  Tracheophyta  ...       None               None  2022-02-25T20:12:04.870Z                                      [StillImage]  [COORDINATE_ROUNDED]

[5 rows x 50 columns]

I did have to do a docker pull ... to get the latest version (I had an earlier version locally). You might want to try explicit version for the tag, which are available at https://mcr.microsoft.com/v2/planetary-computer/python/tags/list.

[glaroc]

Thank you. I thought it was pulling the latest image but I guess it wasn't. It works now after specifying the version explicitly!