Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for downloading google buildings #550

Merged
merged 1 commit into from
Sep 15, 2023
Merged

Add support for downloading google buildings #550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 56 additions & 8 deletions docs/notebooks/81_buildings.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@
"[![image](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/opengeos/leafmap/blob/master/examples/notebooks/01_leafmap_intro.ipynb)\n",
"[![image](https://mybinder.org/badge_logo.svg)](https://gishub.org/leafmap-binder)\n",
"\n",
"**Downloading Microsoft Building Footprints**\n",
"**Downloading Microsoft and Google Building Footprints**\n",
"\n",
"This notebook demonstrates how to download Microsoft Building Footprints and merge them into a single vector file.\n",
"This notebook demonstrates how to download Microsoft and Google Building Footprints and merge them into a single vector file.\n",
"\n",
"https://github.com/microsoft/GlobalMLBuildingFootprints\n",
"- Microsoft Global Building Footprints: https://github.com/microsoft/GlobalMLBuildingFootprints\n",
"- Google Open Buildings: https://sites.research.google/open-buildings\n",
"\n",
"Uncomment the following line to install [leafmap](https://leafmap.org) if needed."
]
Expand Down Expand Up @@ -66,14 +67,14 @@
"metadata": {},
"outputs": [],
"source": [
"head = 3"
"head = 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the building footprints."
"Download the Microsoft building footprints."
]
},
{
Expand All @@ -82,14 +83,19 @@
"metadata": {},
"outputs": [],
"source": [
"leafmap.download_ms_buildings(country, out_dir=\"buildings\", merge_output=f\"{country}.shp\", head=head)"
"leafmap.download_ms_buildings(\n",
" country, \n",
" out_dir=\"buildings\", \n",
" merge_output=f\"{country}_ms.shp\", \n",
" head=head\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Display the building footprints."
"Display the Microsoft building footprints."
]
},
{
Expand All @@ -99,7 +105,49 @@
"outputs": [],
"source": [
"m = leafmap.Map()\n",
"m.add_vector(f\"{country}.shp\", layer_name=\"Buildings\")\n",
"m.add_basemap(\"SATELLITE\")\n",
"m.add_vector(f\"{country}_ms.shp\", layer_name=\"MS Buildings\")\n",
"m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the Google building footprints."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"leafmap.download_google_buildings(\n",
" country, \n",
" out_dir=\"buildings\", \n",
" merge_output=f\"{country}_google.shp\", \n",
" head=head, \n",
" overwrite=True\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Display the Google building footprints."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m = leafmap.Map()\n",
"m.add_basemap(\"SATELLITE\")\n",
"m.add_vector(f\"{country}_google.shp\", layer_name=\"Google Buildings\")\n",
"m"
]
}
Expand Down
2 changes: 1 addition & 1 deletion docs/tutorials.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@
78. Interactive Extraction and Visualization of AWS Open Geospatial Data ([notebook](https://leafmap.org/notebooks/78_read_raster))
79. Visualizing time series images interactively with a time slider ([notebook](https://leafmap.org/notebooks/79_timeseries))
80. Visualizing solar radiation data from Google Solar API ([notebook](https://leafmap.org/notebooks/80_solar))
81. Downloading Microsoft Building Footprints ([notebook](https://leafmap.org/notebooks/81_buildings))
81. Downloading Microsoft and Google Building Footprints ([notebook](https://leafmap.org/notebooks/81_buildings))

## Demo

Expand Down
2 changes: 1 addition & 1 deletion examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@
78. Interactive Extraction and Visualization of AWS Open Geospatial Data ([notebook](https://leafmap.org/notebooks/78_read_raster))
79. Visualizing time series images interactively with a time slider ([notebook](https://leafmap.org/notebooks/79_timeseries))
80. Visualizing solar radiation data from Google Solar API ([notebook](https://leafmap.org/notebooks/80_solar))
81. Downloading Microsoft Building Footprints ([notebook](https://leafmap.org/notebooks/81_buildings))
81. Downloading Microsoft and Google Building Footprints ([notebook](https://leafmap.org/notebooks/81_buildings))

## Demo

Expand Down
64 changes: 56 additions & 8 deletions examples/notebooks/81_buildings.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@
"[![image](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/opengeos/leafmap/blob/master/examples/notebooks/01_leafmap_intro.ipynb)\n",
"[![image](https://mybinder.org/badge_logo.svg)](https://gishub.org/leafmap-binder)\n",
"\n",
"**Downloading Microsoft Building Footprints**\n",
"**Downloading Microsoft and Google Building Footprints**\n",
"\n",
"This notebook demonstrates how to download Microsoft Building Footprints and merge them into a single vector file.\n",
"This notebook demonstrates how to download Microsoft and Google Building Footprints and merge them into a single vector file.\n",
"\n",
"https://github.com/microsoft/GlobalMLBuildingFootprints\n",
"- Microsoft Global Building Footprints: https://github.com/microsoft/GlobalMLBuildingFootprints\n",
"- Google Open Buildings: https://sites.research.google/open-buildings\n",
"\n",
"Uncomment the following line to install [leafmap](https://leafmap.org) if needed."
]
Expand Down Expand Up @@ -66,14 +67,14 @@
"metadata": {},
"outputs": [],
"source": [
"head = 3"
"head = 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the building footprints."
"Download the Microsoft building footprints."
]
},
{
Expand All @@ -82,14 +83,19 @@
"metadata": {},
"outputs": [],
"source": [
"leafmap.download_ms_buildings(country, out_dir=\"buildings\", merge_output=f\"{country}.shp\", head=head)"
"leafmap.download_ms_buildings(\n",
" country, \n",
" out_dir=\"buildings\", \n",
" merge_output=f\"{country}_ms.shp\", \n",
" head=head\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Display the building footprints."
"Display the Microsoft building footprints."
]
},
{
Expand All @@ -99,7 +105,49 @@
"outputs": [],
"source": [
"m = leafmap.Map()\n",
"m.add_vector(f\"{country}.shp\", layer_name=\"Buildings\")\n",
"m.add_basemap(\"SATELLITE\")\n",
"m.add_vector(f\"{country}_ms.shp\", layer_name=\"MS Buildings\")\n",
"m"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the Google building footprints."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"leafmap.download_google_buildings(\n",
" country, \n",
" out_dir=\"buildings\", \n",
" merge_output=f\"{country}_google.shp\", \n",
" head=head, \n",
" overwrite=True\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Display the Google building footprints."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"m = leafmap.Map()\n",
"m.add_basemap(\"SATELLITE\")\n",
"m.add_vector(f\"{country}_google.shp\", layer_name=\"Google Buildings\")\n",
"m"
]
}
Expand Down
108 changes: 103 additions & 5 deletions leafmap/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10574,19 +10574,19 @@ def download_ms_buildings(
dataset_links = pd.read_csv(
"https://minedbuildings.blob.core.windows.net/global-buildings/dataset-links.csv"
)
greece_links = dataset_links[dataset_links.Location == location]
country_links = dataset_links[dataset_links.Location == location]

if not quiet:
print(f"Found {len(greece_links)} links for {location}")
print(f"Found {len(country_links)} links for {location}")
if head is not None:
greece_links = greece_links.head(head)
country_links = country_links.head(head)

filenames = []
i = 1

for _, row in greece_links.iterrows():
for _, row in country_links.iterrows():
if not quiet:
print(f"Downloading {i} of {len(greece_links)}: {row.QuadKey}.geojson")
print(f"Downloading {i} of {len(country_links)}: {row.QuadKey}.geojson")
i += 1
filename = os.path.join(out_dir, f"{row.QuadKey}.geojson")
filenames.append(filename)
Expand All @@ -10605,3 +10605,101 @@ def download_ms_buildings(
merge_vector(filenames, merge_output, quiet=quiet)

return filenames


def download_google_buildings(
location: str,
out_dir: Optional[str] = None,
merge_output: Optional[str] = None,
head: Optional[int] = None,
keep_geojson: bool = False,
overwrite: bool = False,
quiet: bool = False,
**kwargs,
) -> List[str]:
"""
Download Google Open Building dataset for a specific location. Check the dataset links from
https://sites.research.google/open-buildings.

Args:
location: The location name for which to download the dataset.
out_dir: The output directory to save the downloaded files. If not provided, the current working directory is used.
merge_output: Optional. The output file path for merging the downloaded files into a single GeoDataFrame.
head: Optional. The number of files to download. If not provided, all files will be downloaded.
keep_geojson: Optional. If True, the GeoJSON files will be kept after converting them to CSV files.
overwrite: Optional. If True, overwrite the existing files.
quiet: Optional. If True, suppresses the download progress messages.
**kwargs: Additional keyword arguments to be passed to the `gpd.to_file` function.

Returns:
A list of file paths of the downloaded files.

"""

import pandas as pd
import geopandas as gpd
from shapely import wkt

building_url = "https://sites.research.google/open-buildings/tiles.geojson"
country_url = (
"https://naciscdn.org/naturalearth/110m/cultural/ne_110m_admin_0_countries.zip"
)

if out_dir is None:
out_dir = os.getcwd()

if not os.path.exists(out_dir):
os.makedirs(out_dir)

building_gdf = gpd.read_file(building_url)
country_gdf = gpd.read_file(country_url)

country = country_gdf[country_gdf["NAME"] == location]

if len(country) == 0:
country = country_gdf[country_gdf["NAME_LONG"] == location]
if len(country) == 0:
raise ValueError(f"Could not find {location} in the Natural Earth dataset.")

gdf = building_gdf[building_gdf.intersects(country.geometry.iloc[0])]
gdf.sort_values(by="size_mb", inplace=True)

print(f"Found {len(gdf)} buildings for {location}.")
if head is not None:
gdf = gdf.head(head)

if len(gdf) > 0:
links = gdf["tile_url"].tolist()
download_files(links, out_dir=out_dir, quiet=quiet, **kwargs)
filenames = [os.path.join(out_dir, os.path.basename(link)) for link in links]

gdfs = []
for filename in filenames:
# Read the CSV file into a pandas DataFrame
df = pd.read_csv(filename)

# Create a geometry column from the "geometry" column in the DataFrame
df["geometry"] = df["geometry"].apply(wkt.loads)

# Convert the pandas DataFrame to a GeoDataFrame
gdf = gpd.GeoDataFrame(df, geometry="geometry")
gdf.crs = "EPSG:4326"
if keep_geojson:
gdf.to_file(
filename.replace(".csv.gz", ".geojson"), driver="GeoJSON", **kwargs
)
gdfs.append(gdf)

if merge_output:
if os.path.exists(merge_output) and not overwrite:
print(f"File {merge_output} already exists, skip merging...")
else:
if not quiet:
print("Merging GeoDataFrames ...")
gdf = gpd.GeoDataFrame(
pd.concat(gdfs, ignore_index=True), crs="EPSG:4326"
)
gdf.to_file(merge_output, **kwargs)

else:
print(f"No buildings found for {location}.")