2023.12.0

What's Changed

• Update file regex by @jdddog in #94
• Save JSON index for the filtering API and COKI Open Access Dataset release by @jdddog in #89
• Add DOI badge by @aroelo in #96
• Fix/country table schema by @jdddog in #95
• Create CITATION.cff by @bechandcock in #98
• Create .zenodo.json by @jdddog in #97
• Added Kathryn's middle initial by @bechandcock in #99
• Update telescope summaries for readthedocs by @aroelo in #93
• Fix error for Orcid telescope by @aroelo in #100
• Updated crossref metadata schema by @aroelo in #102
• Fix gsutil bug for OpenAlex by @aroelo in #103
• Fix empty line in OpenAlex file by @aroelo in #105
• Fix wiki descriptions by @aroelo in #106
• Update create_bigquery_table_from_query call by @aroelo in #109
• MEL-800 added data trust zenodo community by @kathrynnapier in #108
• Remove largest remainder rounding by @jdddog in #107
• Rename 'metrics_green_only' to 'metrics_percent_green_only' by @aroelo in #111
• Publish new version of dataset to Zenodo automatically by @jdddog in #110
• Feature/oa web acronyms by @jdddog in #104
• Fix/zenodo date version by @jdddog in #112
• Inf 326/dataset release creation by @tuanchien in #84
• Fix/api update by @jdddog in #113
• Download pre-generated twitter cards by @jdddog in #114
• Unpaywall: use merged daily feeds to update by @tuanchien in #115
• Update fieldname "metrics_citations_per_cited_output" by @aroelo in #117
• Update OpenAlex by @aroelo in #116
• Inf 406/seed ao workflows db by @tuanchien in #120
• Update OpenAlex to use transfer manifest file by @aroelo in #121
• Use find_free_port in tests by @tuanchien in #122
• Add dag tag by @tuanchien in #123
• Update elastic_import_workflow.py by @jdddog in #125
• Compute histograms and change country and institution inclusion thresholds by @jdddog in #126
• Closes COK-322 Added All Access Type Counts to Generic and Discipline Exports by @cameronneylon in #128
• Add COKI open access outputs and repositories to the DOI and aggregate tables by @jdddog in #129
• OpenAlex: set start_date to end_date of recent release by @tuanchien in #134
• INF-521 DOI Workflow: Remove DOI table being output to coki-dashboards by @alexmassen-hane in #132
• INF-525: Removing MAG and GRID telescopes, workflows and unit tests by @alexmassen-hane in #137
• OpenAlex: add extra release date management tests by @tuanchien in #136
• OA web workflow other platform open updates by @jdddog in #133
• Pre-commit update by @keegansmith21 in #138
• Replace MAG with OpenAlex by @aroelo in #87
• Migrate aggregate/export doi metrics to openalex by @tuanchien in #140
• INF-557: Removed authors from the list of aggregations by @alexmassen-hane in #141
• Check for invalid lat lng values by @jdddog in #143
• DOI Workflow: add genealogical institutions by @jdddog in #144
• OA Web Open Alex by @jdddog in #142
• Bad 308 schema updates by @keegansmith21 in #145
• Additional kibana spaces by @jdddog in #148
• Fix space name by @jdddog in #151
• Update github unit tests in github workflow/actions by @keegansmith21 in #149
• Updated regex pattern match by @keegansmith21 in #153
• Remove ORCID sensor by @jdddog in #154
• INF-588: HTTP request updates by @keegansmith21 in #155
• Feature/oa web d1 database by @jdddog in #152
• Added crossref events download throttling by @keegansmith21 in #157
• Add descriptions to DOI and aggregate tables by @jdddog in #159
• Fix/doi ror affiliation matcher errors by @jdddog in #161
• Revert changes for re-raising non 500 errors by @jdddog in #163
• Fix Crossref Metadata release date by @jdddog in #162
• Feature/workflow config file by @jdddog in #158
• Added .env to gitignore by @keegansmith21 in #165
• Fix/deploy may 2023 by @jdddog in #166
• Filter Crossref type and fix release_date naming in oa web workflow by @jdddog in #167
• Fix/oa web repo dispatch by @jdddog in #168
• Fix write disposition by @jdddog in #170
• Fix/Add s3:// prefix to OpenAlex AWS blob URL if not present by @alexmassen-hane in #172
• Fix/OpenAlex "abstract_inverted_index" field is sometimes a string, not a dictionary by @alexmassen-hane in #173
• Refactor DOI table by @jdddog in #171
• Feature/airflow2.6.3 by @jdddog in #175
• Remove Elastic mappings by @jdddog in #176
• Convert updated_date into timestamp when it is supplied as a date by @jdddog in #177
  ...

2022.03.0

What's Changed

• Inf 40/create unpaywall data feed telescope by @tuanchien in #39
• Rename snapshot to unpaywall_snapshot, datafeed to unpaywall by @tuanchien in #44
• Update StreamTelescope telescopes by @aroelo in #45
• INF-166/airflow-2.2 by @jdddog in #49
• Inf 170/unpaywall download changes by @tuanchien in #51
• ROR Telescope by @aroelo in #50
• Inf 60/add operator by @jdddog in #53
• Port open citations to telescope template. by @tuanchien in #48
• Inf 164/port wos telescope template by @tuanchien in #55
• port scopus telescope by @tuanchien in #56
• INF-181 ORCID records not being correctly matched by @rhosking in #58
• Added new oa fields and repo names by @rhosking in #57
• Remove Unpaywall history by @jdddog in #54
• Update ror_telescope.py by @jdddog in #63
• INF-65: MAG updates by @jdddog in #60
• INF-191 Add discipline to affiliation list by @rhosking in #61
• INF-182 use issn-l for journal ids by @rhosking in #59
• INF-60: Open Access Website Workflow by @jdddog in #52
• Add xcom cleanup to mag telescope by @tuanchien in #65
• Inf 183 - Move to ROR by @rhosking in #64
• Inf 304/scopus deployment fix extra documentation by @tuanchien in #70
• Feature/crossref metadata schema update by @jdddog in #71
• Ignore unknown fields by @jdddog in #69
• INF-193 Commenting the SQL scripts by @rhosking in #62
• Test fixes after bq quota checks feature by @tuanchien in #72
• INF-208 Updated DOI workflow page in readthedocs by @rhosking in #68
• Remove MAG sensor by @jdddog in #73
• INF-320: Fix duplicate DOIs by @aroelo in #76
• Update config instance for elastic import workflow by @aroelo in #75
• Fix crossref metadata invalid timestamp by @aroelo in #77
• INF-328: Ignore unknown values on BigQuery load by @jdddog in #79
• INF-287: Update generated data to work with Next.js web application by @jdddog in #66
• Fix ORCID telescope gcloud error by @aroelo in #85
• OpenAlex telescope by @aroelo in #80
• Add wikipedia descriptions during oa web workflow by @aroelo in #82
• Inf 333/publish oa web data by @jdddog in #83
• OpenAlex fixes by @aroelo in #88
• Small fixes OA web workflow wiki descriptions by @aroelo in #86
• Fix error getting wiki description oa_web_workflow by @aroelo in #90
• Fix transform for cases where the transformed field does not exist. by @aroelo in #92
• Refactor to use new country and region table by @jdddog in #91

New Contributors

• @rhosking made their first contribution in #58

Full Changelog: 2021.09.1...2022.03.0

2021.09.1

This release includes the following changes / new features:

• Add black to pre-commit config.
• Added a standardised make_release_date function to observatory-platform, which is now used in Onix Workflow. It returns a pendulum.DateTime rather than a pendulum.Date.
• Remove upload_downloaded from SnapshotTelescope subclasses.
• Refactor relevant workflows to use XComless StreamTelescope.
• Ported Unpaywall workflow to use the SnapshotTelescope.
• Removed get_file and _hash_file.
• Crossref Metdata:
  • Can transform JSON input files whether pretty printed or not.
  • Refactored transform functions to use Python rather than mawk as it is easier to understand what the transformation does and it is faster.
  • Added explicit tests for transform_file and transform_item.

This release includes the following bugfixes:

• Remove RUN statement from requirements shell script.
• Crossref Metadata now needs the API key to make a head request, which is now added to the head request.
• Fix bug in Fundref get_release_info, which was passing a string to bigquery_sharded_table_id rather than a DateTime.

2021.09.0

This release includes telescope workflows for:

• Crossref Events
• Crossref Fundref
• Crossref Metadata
• Geonames
• GRID
• Microsoft Academic Graph
• Open Citations
• ORCID
• Scopus
• Unpaywall
• Web of Science

And the following analytical workflows:

• DOI Workflow
• Elastic Import Workflow setup to export output from the DOI Workflow into Elasticsearch and Kibana.