BigQueryCache: connect without service account

[4sushi]

Hello,

I'm trying to use BigQueryCache without service account.
I don't manage to generate a valid credential file. Do you have any idea how I can do it?
I follow this example run_bigquery_faker.py.

SCOPES = ['https://www.googleapis.com/auth/bigquery']
credentials, project = google.auth.default(scopes=SCOPES)
print(credentials)
# {"refresh_token": "XXX", "token_uri": "https://oauth2.googleapis.com/token", "client_id": "YYY", "client_secret": "ZZZZ", "universe_domain": "googleapis.com", "account": "", "expiry": "2024-03-14T14:33:18.294693Z"}

with tempfile.NamedTemporaryFile(mode="w", delete=False) as temp:
    temp.write(credentials.to_json())
    temp.flush()
    temp.close()
    bq_cache = BigQueryCache(project_name=project, credentials_path=temp.name)
    source.select_all_streams()
    read_result: ab.ReadResult = source.read(cache=bq_cache)

The structure of the credential is not good, I get this error:

MalformedError: Service account info was not in the expected format, missing fields client_email.

Usually, when I'm using BigQuery I don't use credential file.
The authentication is handle by Google Cloud client.
It seems it used application_default_credentials.json file, but there is no client_email field inside.

from google.cloud import bigquery
cli = bigquery.Client()

Thanks for your work.
PyAirbyte looks really promising.

[aaronsteers]

@4sushi - Thanks for trying out PyAirbyte and thanks for leaving this feedback. I have briefly looked into what would be required to use the Google CLI Client instead of a service account credentials file.

The place that most likely would need to be updated is _get_credentials(), which currently expects to initialize auth for a service account:

PyAirbyte/airbyte/_processors/sql/bigquery.py

Lines 141 to 149 in 86722f4

	def _get_credentials(self) -> service_account.Credentials:
	"""Return the GCP credentials."""
	if self._credentials is None:
	self._credentials = service_account.Credentials.from_service_account_file(
	self.cache.credentials_path
	)
	return self._credentials

We may also need to update get_sql_alchemy_url():

https://github.com/airbytehq/PyAirbyte/blob/86722f438f0a79e5b101518e3a3cfb986fdfcd61/airbyte/caches/bigquery.py#L53-L58

And we might want to modify the BigQueryCache constructor (if different inputs are expected).

Let us know if this is something you are interested in contributing to. Otherwise, we will add to the project board for prioritization.

[4sushi]

@aaronsteers Thanks for your answer. I will try to contribute on this feature

[aaronsteers]

Circling back here for future readers.

This feature has been released! Thanks again to @4sushi for contributing. 🙏🎉