Batch parse and geocode addresses from followthemoney entities. Simply geocoding just address strings works as well, of course.
There are as well some parsing / normalization helpers.
- Parse/normalize addresses via libpostal
- Geocoding via geopy
- Cache geocoding results in a sql database (using dataset)
- Optional fallback geocoders when preferred geocoder doesn't match
- Create, update and merge
Addressentities for ftm data
ftmgeo --help
The command line interface is designed for piping input / output streams, but
for each command a -i <input_file> and -o <output_file> can be used as well.
Geocode an input stream of ftm entities with nominatim and google maps as fallback (geocoders are tried in the given order):
cat entitis.ftm.ijson | ftmgeo geocode -g nominatim -g google > entities_geocoded.ftm.ijson
This looks for the address prop on input entities and creates address entities with reference to the input entities. The output contains all entities from the input stream plus newly created addresses.
If an input entity is itself an Address entity, it will be geocoded as well and their props (country, city, ...) will be merged with the geocoder result.
During the process, addresses are parsed and normalized and looked up in the address cache database before actual geocoding. After geocoding, new addresses are added to the cache database.
Address ids will be rewritten based on normalization (addressEntity refs are updated on other entities),
to keep the original ids, add the flag --no-rewrite-ids
Geocoders can be set via GEOCODERS and default to nominatim
More information:
ftmgeo geocode --help
csv format (for all csv input streams)
first column address, optional second column country (name or code) and
third language for postal context
To ftm address entities:
cat addresses.csv | ftmgeo geocode --input-format=csv > addresses.ftm.ijson
To csv:
cat addresses.csv | ftmgeo geocode --input-format=csv --output-format=csv > addresses.csv
Get a cleaned address line from messy input strings.
cat addresses.txt | ftmgeo format-line > clean_addresses.csv
Get a csv with all the parsed components from libpostal.
cat addresses.txt | ftmgeo parse-components > components.csv
Generate address entities from input stream (without geocoding):
cat entities.ftm.ijson | ftmgeo map > entities.ftm.ijson
cat addresses.csv | ftmgeo map --input-format=csv > addresses.ftm.ijson
ftmgeo cache --help
During geocoding, addresses are first looked up in the local cache, and new geocoding results are added.
Ignore cache during geocoding (new results are still written to it):
ftmgeo geocode --no-cache ...
Export cache:
ftmgeo cache iterate > geocoded_addresses.ftm.ijsonl
ftmgeo cache iterate --output-format=csv > geocoded_addresses.csv
Populate cache:
csv input:
address_id,canonical_id,original_line,result_line,country,lat,lon,geocoder,geocoder_place_id
optional field: geocoder_raw - json of geocoder response
cat geocoded_addresses.csv | ftmgeo cache populate
ftmgeo cache apply-csv --help
To get addresses from cache without geocoding from a csv input stream, passing through additional csv data from input:
cat addresses.csv | ftmgeo cache apply-csv --output-format csv > results.csv
Only get missing to re-geocode (e.g. with a different geocoder):
cat addresses.csv | ftmgeo cache apply-csv --output-format csv --get-missing | ftmgeo geocode
geocoders
Default geocoders: env var GEOCODERS
They are used in the given order
Make sure to configure the geocoders as needed for geopy (endpoints, api keys, ...):
export FTMGEO_<GEOCODERNAME>_<SETTING>=...
Persistent cache
The cache database is set via FTM_STORE_URI (so it is the same as the
ftm store, if any,
otherwise it defaults to sqlite:///cache.db
Required external is libpostal, see installation instructions there.
Once libpostal is installed on your system, you can install:
pip install ftm-geocoder[postal]
Verify that this works without errors:
ftmgeo --help
echo "Cowley Road, Cambridge, UK" | ftmgeo geocode --input-format=csv --no-header
make install
make test