Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1774037: OCSP check fails when execption is not of type RevocationCheckError #2094

Open
knowhoper opened this issue Oct 30, 2024 · 8 comments
Assignees
Labels
bug status-triage_done Initial triage done, will be further handled by the driver team

Comments

@knowhoper
Copy link

knowhoper commented Oct 30, 2024

Python version

3.10.3

Operating system and processor architecture

macOS-14.7-arm64-arm-64bit

Installed packages

asn1crypto==1.5.1
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
cryptography==43.0.3
filelock==3.16.1
idna==3.10
packaging==24.1
platformdirs==4.3.6
pycparser==2.22
PyJWT==2.9.0
pyOpenSSL==24.2.1
pytz==2024.2
requests==2.32.3
sortedcontainers==2.4.0
tomlkit==0.13.2
typing_extensions==4.12.2
urllib3==2.2.3
snowflake-connector-python==3.12.3

What did you do?

OCSP errors fail when the Exception thrown is not of type RevocationCheckError. This is on any invocation of the connector in our infrastructure - currently trying to diagnose why we have OCSP issues.


Fails with error

 File "/opt/pysetup/.venv/lib/python3.10/site-packages/snowflake/connector/ocsp_snowflake.py", line 1147, in verify_fail_open
    if ex_obj.errno is ER_OCSP_RESPONSE_CERT_STATUS_REVOKED:
AttributeError: 'TypeError' object has no attribute 'errno'


### What did you expect to see?

A more concise OCSP error or results from the Snowflake service.

https://github.com/snowflakedb/snowflake-connector-python/blob/main/src/snowflake/connector/ocsp_snowflake.py#L1147

This line fails when the type passed is of type Exception as Exception has no property errno. This is causing issues in our stack due to some OCSP error in our infrastructure we can't diagnose due to the error routine being incorrect in the connector

### Can you set logging to DEBUG and collect the logs?

```bash
import logging
import os

for logger_name in ('snowflake.connector',):
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)
@github-actions github-actions bot changed the title OCSP check fails when execption is not of type RevocationCheckError SNOW-1774037: OCSP check fails when execption is not of type RevocationCheckError Oct 30, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka self-assigned this Nov 1, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka added status-triage Issue is under initial triage and removed needs triage labels Nov 1, 2024
@sfc-gh-dszmolka
Copy link
Contributor

sfc-gh-dszmolka commented Nov 1, 2024

hi - thanks for raising this issue with us. taking a look. do you think it would be possible to post a full(er) stack of the error thrown? we supposed to raise a RevocationCheckError when the OCSP server is not reachable, so really curious what else is going on here.

edit: re: to diagnose why you're having OCSP errors, this could be helpful:

  • run SYSTEM$ALLOWLIST() (or ALLOWLIST_PRIVATELINK() if you're using PL) in your snowflake account, and take note of every OCSP-related endpoint. Note that they indeed do operate on port 80, and they should.
  • work with the infra/cloud people to allow connectivity to these endpoints over port 80.
    Nothing user-related or sensitive is going over the wire unencrypted, only information which is already public (certificates and their validity/not validity)

@sfc-gh-dszmolka sfc-gh-dszmolka added the status-information_needed Additional information is required from the reporter label Nov 1, 2024
@knowhoper
Copy link
Author

knowhoper commented Nov 2, 2024

Hi @sfc-gh-dszmolka , thank you for the follow up. Yes, we are currently in the process of diagnosing the OCSP issue within our environment, suspect its a timeout due to some missing whitelisting. Errors started appearing with no code changes on our side so assuming something in either our SF tenant or GCP infra has changed. Will report back findings.

Regardless, I suspect the error routine there is incorrect, which is making pin pointing this issue harder.

Thanks again.

@sfc-gh-dszmolka sfc-gh-dszmolka added status-triage_done Initial triage done, will be further handled by the driver team and removed status-information_needed Additional information is required from the reporter status-triage Issue is under initial triage labels Nov 4, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka removed their assignment Nov 4, 2024
@knowhoper
Copy link
Author

Here are the results from the OCSP check, noting the failure on

OCSP_RESPONDER: ocsp.pki.goog:80: URL Check: Failed: HTTP/1.1 404 Not Found

=========Connectivity diagnostic report================================
INITIAL: Specified snowflake account: redacted
INITIAL: Host based on specified account: redacted.us-central1.gcp.snowflakecomputing.com

=========Proxy information - These are best guesses, not guarantees====
PROXY: Proxies with Env vars removed(SYSTEM PROXIES): {}
PROXY: Proxies with Env vars restored(ENV PROXIES): {}
PROXY: There is likely a proxy because the issuer for www.google.com is not correct. Got C=US; O=Google Trust Services; CN=WE2 and expected one of ('C=US; O=Google Trust Services LLC', 'C=US; O=Amazon', 'C=US; O=DigiCert Inc')
PROXY: Proxy check using invalid URL did not show proxy: Review result, but you can probably ignore: Result: HTTPSConnectionPool(host='ireallyshouldnotexistatallanywhere.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<snowflake.connector.vendored.urllib3.connection.HTTPSConnection object at 0x3e16d64b8b80>: Failed to establish a new connection: [Errno -2] Name or service not known'))

=========Snowflake URL information=====================================
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: nslookup results: public ip: 34.70.63.80
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com:443: URL Check: Connected Successfully
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: Cert info:
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: subject: {'C': 'US', 'ST': 'California', 'L': 'San Mateo', 'O': 'Snowflake Inc.', 'CN': '*.us-central1.gcp.snowflakecomputing.com'}
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: issuer: {'C': 'US', 'O': 'DigiCert Inc', 'CN': 'DigiCert Global G2 TLS RSA SHA256 2020 CA1'}
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: serialNumber: 2619509689619176879997026077985058025
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: version: 2
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: notBefore: 2024-01-29 00:00:00
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: notAfter: 2025-01-28 23:59:59
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: subjectAltName: DNS:*.us-central1.gcp.snowflakecomputing.com, DNS:*.snowflake.app, DNS:*.gcpuscentral1.snowflake.app, DNS:*.snowflakecomputing.com, DNS:*.global.snowflakecomputing.com, DNS:us-central1.gcp.snowflakecomputing.com, DNS:snowflake.app, DNS:gcpuscentral1.snowflake.app, DNS:snowflakecomputing.com, DNS:global.snowflakecomputing.com
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
SNOWFLAKE_URL: redacted.us-central1.gcp.snowflakecomputing.com: ocspURLs: ['ocsp.digicert.com', 'cacerts.digicert.com']

=========Snowflake Stage information===================================
We retrieved stage info from the allowlist
STAGE: storage.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: storage.googleapis.com:443: URL Check: Connected Successfully
STAGE: storage.googleapis.com: Cert info:
STAGE: storage.googleapis.com: subject: {'CN': 'storage.googleapis.com'}
STAGE: storage.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: storage.googleapis.com: serialNumber: 45079917743581688039704620458248828968
STAGE: storage.googleapis.com: version: 2
STAGE: storage.googleapis.com: notBefore: 2024-10-21 08:40:59
STAGE: storage.googleapis.com: notAfter: 2025-01-13 08:40:58
STAGE: storage.googleapis.com: subjectAltName: DNS:storage.googleapis.com
STAGE: storage.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: storage.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com:443: URL Check: Connected Successfully
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: Cert info:
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: version: 2
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: storage-upload.googleapis.com: nslookup results: ['142.250.66.187', '172.217.24.59']
STAGE: storage-upload.googleapis.com:443: URL Check: Connected Successfully
STAGE: storage-upload.googleapis.com: Cert info:
STAGE: storage-upload.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: storage-upload.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: storage-upload.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: storage-upload.googleapis.com: version: 2
STAGE: storage-upload.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: storage-upload.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: storage-upload.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: storage-upload.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: storage-upload.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com:443: URL Check: Connected Successfully
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: Cert info:
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: version: 2
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-upload.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: storage-download.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: storage-download.googleapis.com:443: URL Check: Connected Successfully
STAGE: storage-download.googleapis.com: Cert info:
STAGE: storage-download.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: storage-download.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: storage-download.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: storage-download.googleapis.com: version: 2
STAGE: storage-download.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: storage-download.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: storage-download.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: storage-download.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: storage-download.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: nslookup results: ['172.217.24.59']
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com:443: URL Check: Connected Successfully
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: Cert info:
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: subject: {'CN': '*.storage.googleapis.com'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WE2'}
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: serialNumber: 189390029780162745962145226958905017814
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: version: 2
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: notBefore: 2024-10-21 08:37:54
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: notAfter: 2025-01-13 08:37:53
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: subjectAltName: DNS:*.storage.googleapis.com, DNS:*.googleapis.com, DNS:commondatastorage.googleapis.com, DNS:*.commondatastorage.googleapis.com, DNS:storage.googleapis.com, DNS:storage-p2.googleapis.com, DNS:*.storage-p2.googleapis.com, DNS:storage.mtls.googleapis.com, DNS:*.appspot.com.storage.googleapis.com, DNS:*.content-storage.googleapis.com, DNS:*.content-storage-p2.googleapis.com, DNS:*.content-storage-upload.googleapis.com, DNS:*.content-storage-download.googleapis.com, DNS:*.storage-upload.googleapis.com, DNS:*.storage-download.googleapis.com
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: gcpuscentral1-5fd82fl-stage.storage-download.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']
STAGE: www.googleapis.com: nslookup results: ['142.251.221.74', '142.250.71.74', '142.250.66.234', '142.250.76.106', '172.217.167.74', '172.217.167.106', '142.250.66.202', '142.250.67.10', '142.250.204.10']
STAGE: www.googleapis.com:443: URL Check: Connected Successfully
STAGE: www.googleapis.com: Cert info:
STAGE: www.googleapis.com: subject: {'CN': 'upload.video.google.com'}
STAGE: www.googleapis.com: issuer: {'C': 'US', 'O': 'Google Trust Services', 'CN': 'WR2'}
STAGE: www.googleapis.com: serialNumber: 249523738985507380678070046713553759211
STAGE: www.googleapis.com: version: 2
STAGE: www.googleapis.com: notBefore: 2024-10-07 08:25:41
STAGE: www.googleapis.com: notAfter: 2024-12-30 08:25:40
STAGE: www.googleapis.com: subjectAltName: DNS:upload.video.google.com, DNS:*.clients.google.com, DNS:*.docs.google.com, DNS:*.drive.google.com, DNS:*.gdata.youtube.com, DNS:*.googleapis.com, DNS:*.photos.google.com, DNS:*.youtube-3rd-party.com, DNS:upload.google.com, DNS:*.upload.google.com, DNS:upload.youtube.com, DNS:*.upload.youtube.com, DNS:uploads.stage.gdata.youtube.com, DNS:bg-call-donation.goog, DNS:bg-call-donation-alpha.goog, DNS:bg-call-donation-canary.goog, DNS:bg-call-donation-dev.goog
STAGE: www.googleapis.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
STAGE: www.googleapis.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']

=========Snowflake OCSP information====================================
We were able to retrieve system allowlist.
These OCSP hosts came from the certificate and the allowlist.
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.47
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.5
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.122
OCSP_RESPONDER: ocsp.snowflakecomputing.com: nslookup results: public ip: 108.158.32.11
OCSP_RESPONDER: ocsp.snowflakecomputing.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: cacerts.digicert.com: nslookup results: ['152.195.38.76']
OCSP_RESPONDER: cacerts.digicert.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.digicert.com: nslookup results: ['152.195.38.76']
OCSP_RESPONDER: ocsp.digicert.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: e5.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: e5.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: e7.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: e7.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: e6.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: e6.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.r2m01.amazontrust.com: nslookup results: ['18.67.98.168']
OCSP_RESPONDER: ocsp.r2m01.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: e8.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: e8.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.rootg2.amazontrust.com: nslookup results: ['13.35.146.220']
OCSP_RESPONDER: ocsp.rootg2.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: e9.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: e9.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.rootca1.amazontrust.com: nslookup results: ['13.35.146.220']
OCSP_RESPONDER: ocsp.rootca1.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: r10.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: r10.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.r2m03.amazontrust.com: nslookup results: ['18.67.98.168']
OCSP_RESPONDER: ocsp.r2m03.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.pki.goog: nslookup results: ['172.217.24.35']
OCSP_RESPONDER: ocsp.pki.goog:80: URL Check: Failed: HTTP/1.1 404 Not Found
Date: Thu, 07 Nov 2024 23:04:58 GMT
Content-Type: text/html; charset=UTF-8
Server: ocsp_responder
Content-Length: 1561
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/</code> was not found on this server.  <ins>That’s all we know.</ins>

OCSP_RESPONDER: ocsp.sca1b.amazontrust.com: nslookup results: ['18.67.95.191', '18.67.95.118', '18.67.95.61', '18.67.95.7']
OCSP_RESPONDER: ocsp.sca1b.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: r11.o.lencr.org: nslookup results: ['23.1.240.137', '23.46.179.226']
OCSP_RESPONDER: r11.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: o.pki.goog: nslookup results: ['172.217.24.35']
OCSP_RESPONDER: o.pki.goog:80: URL Check: Failed: HTTP/1.1 404 Not Found
Date: Thu, 07 Nov 2024 23:04:58 GMT
Content-Type: text/html; charset=UTF-8
Server: ocsp_responder
Content-Length: 1561
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/</code> was not found on this server.  <ins>That’s all we know.</ins>

OCSP_RESPONDER: r14.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: r14.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: ocsp.r2m02.amazontrust.com: nslookup results: ['18.67.98.168']
OCSP_RESPONDER: ocsp.r2m02.amazontrust.com:80: URL Check: Connected Successfully
OCSP_RESPONDER: r12.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: r12.o.lencr.org:80: URL Check: Connected Successfully
OCSP_RESPONDER: r13.o.lencr.org: nslookup results: ['23.46.179.226', '23.1.240.137']
OCSP_RESPONDER: r13.o.lencr.org:80: URL Check: Connected Successfully

=========Snowflake Out of bound telemetry check========================
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: nslookup results: public ip: 54.191.121.111
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: nslookup results: public ip: 44.235.223.145
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: nslookup results: public ip: 44.229.173.185
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com:443: URL Check: Connected Successfully
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: Cert info:
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: subject: {'C': 'US', 'ST': 'California', 'L': 'San Mateo', 'O': 'Snowflake Inc.', 'CN': 'client-telemetry.snowflakecomputing.com'}
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: issuer: {'C': 'US', 'O': 'DigiCert Inc', 'CN': 'DigiCert Global G2 TLS RSA SHA256 2020 CA1'}
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: serialNumber: 14635881134541535973514408807947601738
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: version: 2
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: notBefore: 2024-03-22 00:00:00
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: notAfter: 2025-03-22 23:59:59
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: subjectAltName: DNS:client-telemetry.snowflakecomputing.com
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: crlUrls: ['crl3.digicert.com', 'crl4.digicert.com']
OUT_OF_BAND_TELEMETRY: client-telemetry.snowflakecomputing.com: ocspURLs: ['ocsp.snowflakecomputing.com', 'cacerts.digicert.com', 'ocsp.digicert.com']

@sfc-gh-dszmolka
Copy link
Contributor

sfc-gh-dszmolka commented Nov 11, 2024

this issue seems to be coming from SnowCD itself, and is unrelated to the problem. (it looks to be trying to check http://ocsp.pki.goog OCSP Responder, but there's no OCSP Responder endpoint on / path of the said server.
Please ignore this error line from SnowCD.

do you think it would be possible to post a full(er) stack of the error thrown?

Is it possible to share it, from the original issue you saw? the stack of exceptions from the the python application, not just a single error line?

@flekkk
Copy link

flekkk commented Nov 11, 2024

Hello, everyone. On our project there was exactly the same problem described above knowhoper. The problem appeared also unexpectedly and without any changes in code/infrastructure. The error in the logs fully corresponds to the error attached by knowhoper.

So far it turned out that everything works fine with insecure_mode=True.

I would be very grateful for any help and information related to this issue.

All the logs we were able to get:

DEFAULT 2024-11-11T11:44:41.372083Z [ERROR 14 2024-11-11 11:44:41,370] - snowflake.connector.file_transfer_agent 606 An exception was raised in <bound method SnowflakeStorageClient.prepare_upload of <snowflake.connector.gcs_storage_client.SnowflakeGCSRestClient object at 0x3e9339ee1990>>
ERROR 2024-11-11T11:44:41.372115Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1125, in validate_by_direct_connection self.process_ocsp_response(issuer, cert_id, ocsp_response) File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 310, in process_ocsp_response self.verify_signature( File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 372, in verify_signature public_key.verify( TypeError: _EllipticCurvePublicKey.verify() takes 4 positional arguments but 5 were given
DEFAULT 2024-11-11T11:44:41.372120Z During handling of the above exception, another exception occurred:
ERROR 2024-11-11T11:44:41.372129Z Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/snowflake/connector/file_transfer_agent.py", line 603, in function_and_callback_wrapper work(*args, **kwargs),
DEFAULT 2024-11-11T11:44:41.372132Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372135Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/storage_client.py", line 229, in prepare_upload
DEFAULT 2024-11-11T11:44:41.372138Z self.preprocess()
DEFAULT 2024-11-11T11:44:41.372141Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/storage_client.py", line 195, in preprocess
DEFAULT 2024-11-11T11:44:41.372144Z file_header = self.get_file_header(
DEFAULT 2024-11-11T11:44:41.372147Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372150Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/gcs_storage_client.py", line 344, in get_file_header
DEFAULT 2024-11-11T11:44:41.372153Z response = self._send_request_with_retry(
DEFAULT 2024-11-11T11:44:41.372156Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372160Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/storage_client.py", line 294, in _send_request_with_retry
DEFAULT 2024-11-11T11:44:41.372162Z response = session.request(verb, url, **rest_kwargs)
DEFAULT 2024-11-11T11:44:41.372165Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372168Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/requests/sessions.py", line 589, in request
DEFAULT 2024-11-11T11:44:41.372171Z resp = self.send(prep, **send_kwargs)
DEFAULT 2024-11-11T11:44:41.372174Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372177Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/requests/sessions.py", line 703, in send
DEFAULT 2024-11-11T11:44:41.372180Z r = adapter.send(request, **kwargs)
DEFAULT 2024-11-11T11:44:41.372182Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372186Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/requests/adapters.py", line 486, in send
DEFAULT 2024-11-11T11:44:41.372188Z resp = conn.urlopen(
DEFAULT 2024-11-11T11:44:41.372191Z ^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372196Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 715, in urlopen
DEFAULT 2024-11-11T11:44:41.372199Z httplib_response = self._make_request(
DEFAULT 2024-11-11T11:44:41.372202Z ^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372204Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 404, in _make_request
DEFAULT 2024-11-11T11:44:41.372207Z self._validate_conn(conn)
DEFAULT 2024-11-11T11:44:41.372211Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 1058, in _validate_conn
DEFAULT 2024-11-11T11:44:41.372214Z conn.connect()
DEFAULT 2024-11-11T11:44:41.372217Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connection.py", line 419, in connect
DEFAULT 2024-11-11T11:44:41.372220Z self.sock = ssl_wrap_socket(
DEFAULT 2024-11-11T11:44:41.372223Z ^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372226Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ssl_wrap_socket.py", line 91, in ssl_wrap_socket_with_ocsp
DEFAULT 2024-11-11T11:44:41.372229Z ).validate(server_hostname, ret.connection)
DEFAULT 2024-11-11T11:44:41.372232Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372235Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 999, in validate
DEFAULT 2024-11-11T11:44:41.372238Z return self._validate(
DEFAULT 2024-11-11T11:44:41.372241Z ^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372244Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1012, in _validate
DEFAULT 2024-11-11T11:44:41.372247Z results = self._validate_certificates_sequential(
DEFAULT 2024-11-11T11:44:41.372249Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372252Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1205, in _validate_certificates_sequential
DEFAULT 2024-11-11T11:44:41.372255Z r = self.validate_by_direct_connection(
DEFAULT 2024-11-11T11:44:41.372258Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372261Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1140, in validate_by_direct_connection
DEFAULT 2024-11-11T11:44:41.372264Z err = self.verify_fail_open(ex, telemetry_data)
DEFAULT 2024-11-11T11:44:41.372266Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.372269Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1159, in verify_fail_open
DEFAULT 2024-11-11T11:44:41.372272Z if ex_obj.errno is ER_OCSP_RESPONSE_CERT_STATUS_REVOKED:
DEFAULT 2024-11-11T11:44:41.372275Z ^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.488608Z AttributeError: 'TypeError' object has no attribute 'errno'


ERROR 2024-11-11T11:44:41.490209Z Traceback (most recent call last): File ... in _write_dataframe success, nchunks, nrows, _ = write_pandas(con, df, self.datapoint_table_name)
DEFAULT 2024-11-11T11:44:41.490212Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.490216Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/pandas_tools.py", line 389, in write_pandas
DEFAULT 2024-11-11T11:44:41.490219Z cursor.execute(upload_sql, _is_internal=True)
DEFAULT 2024-11-11T11:44:41.490222Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/cursor.py", line 1059, in execute
DEFAULT 2024-11-11T11:44:41.490225Z data = sf_file_transfer_agent.result()
DEFAULT 2024-11-11T11:44:41.490228Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.490231Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/file_transfer_agent.py", line 740, in result
DEFAULT 2024-11-11T11:44:41.490240Z Error.errorhandler_wrapper(
DEFAULT 2024-11-11T11:44:41.490243Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/errors.py", line 284, in errorhandler_wrapper
DEFAULT 2024-11-11T11:44:41.490247Z handed_over = Error.hand_to_other_handler(
DEFAULT 2024-11-11T11:44:41.490250Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-11-11T11:44:41.490253Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/errors.py", line 339, in hand_to_other_handler
DEFAULT 2024-11-11T11:44:41.490256Z cursor.errorhandler(connection, cursor, error_class, error_value)
DEFAULT 2024-11-11T11:44:41.490259Z File "/usr/local/lib/python3.11/site-packages/snowflake/connector/errors.py", line 215, in default_errorhandler
DEFAULT 2024-11-11T11:44:41.490262Z raise error_class(
DEFAULT 2024-11-11T11:44:41.524363Z snowflake.connector.errors.OperationalError: 253003: While putting file(s) there was an error: 'AttributeError("'TypeError' object has no attribute 'errno'")', this might be caused by your access to the blob storage provider, or by Snowflake.

Some important dependencies:

python 3.11

asn1crypto==1.5.1
attrs==23.2.0
Babel==2.15.0
beautifulsoup4==4.12.2
certifi==2023.11.17
cryptography==41.0.7
filelock==3.13.1
flake8==6.1.0
httplib2==0.22.0
numpy==1.26.3
packaging==23.2
pandas==2.1.4
pyOpenSSL==23.3.0
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.11.4
snowflake-connector-python==3.12.3
snowflake-snowpark-python==1.11.1
snowflake-sqlalchemy==1.5.1
SQLAlchemy==1.4.51
sqlparse==0.4.4
tomlkit==0.12.3
urllib3==2.1.0

@sfc-gh-dszmolka sfc-gh-dszmolka self-assigned this Nov 12, 2024
@sfc-gh-dszmolka
Copy link
Contributor

hi @flekkk thank you, this response was very useful! looks like, at least in your case, something causes the public key verification to somehow get 5 arguments:

ERROR 2024-11-11T11:44:41.372115Z Traceback (most recent call last): 
File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_snowflake.py", line 1125, in validate_by_direct_connection self.process_ocsp_response(issuer, cert_id, ocsp_response) 
File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 310, in process_ocsp_response self.verify_signature( 
File "/usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py", line 372, in verify_signature public_key.verify( TypeError: _EllipticCurvePublicKey.verify() takes 4 positional arguments but 5 were given

despite only four should be passed:

            public_key.verify(
                signature, digest, padding.PKCS1v15(), utils.Prehashed(chosen_hash)
            )

and we don't have error handling for this situation as it should never happen.

Anyways; if you're up for a little more debugging and gathering more info, I created a patched version of ocsp_asn1crypto.py with some more logging and a dumb but hopefully effective error handling, attaching here:
patched_ocsp_asn1crypto.py.txt

0.1 make sure you're running Snowflake PythonConnector 3.12.3, as the patch was made for this version. Should work with other 3.12.x too. I see you already have it.
0.2 remove insecure_mode or set it to False

  1. optionally, take a backup of /usr/local/lib/python3.11/site-packages/snowflake/connector/ocsp_asn1crypto.py
  2. replace the above file with the patched version
  3. either use this article to enable DEBUG logging in your original Python application

or you can just use this very simple one to connect to Snowflake and PUT a file (I see the error happened in the PUT), if using the test app please make sure to replace the account details, database, stage, etc:

import snowflake.connector

import logging
import os

for logger_name in ['snowflake','botocore']:
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.FileHandler('python_connector.log')
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)

conn = snowflake.connector.connect(
    user='REPLACE_USER',
    password='REPLACE_PASSWORD',
    account='REPLACE-ACCOUNT',
    database='REPLACE_DB',
    schema='REPLACE_SCHEMA'
    )

conn.cursor().execute("PUT file:///PATH/TO/test.txt @REPLACE_STAGE OVERWRITE=TRUE")

Main thing is, that either way, the DEBUG level logs need to be turned on.
4. Reproduce the issue, i.e. run the test script above or run your original one which has DEBUG logs enabled. It will log tons of events into python_connector.log where it's executed.
5. Now, I will need to see the logs somehow :) So either share it here (and then make sure to sanitize it, remove any sensitive data) or raise a Support Case with us and reference this GH Issue.
6. You can revert DEBUG logs or even insecure_mode, but if you decide to keep the patched version of the module, then it hopefully the simple error handling will prevent the unhandled exception and allow your app to run, until we fix this properly.

Thanks for your help here !

@opqpop
Copy link

opqpop commented Nov 13, 2024

Hi, this seems to still happen for me even when upgraded to 3.12.3, started happening 11/8/24

any thoughts on what's wrong? it only happens for a very specific query that i do, only fetches 400 rows and I'm not sure why it happens

using insecure_mode=True solves the issue

def snowflake_conn(keep_alive=False, **kwargs):
    """
    Connect to Snowflake
        account: looks like "{}.us-central1.gcp"
    """

    keys = {
        "account": cfgutil.get_env("SNOWFLAKE_ACCOUNT"),
        "user": cfgutil.get_env("SNOWFLAKE_USER"),
        "password": cfgutil.get_env("SNOWFLAKE_PASSWORD"),
        "warehouse": cfgutil.get_env("SNOWFLAKE_WAREHOUSE", "COMPUTE_WH"),
        "database": cfgutil.get_env("SNOWFLAKE_DB_NAME", "EDW"),
    }

    # useful debug
    # print(keys)

    logger.info("Creating snowflake connection")

    conn = snowflake.connector.connect(
        account=keys["account"],
        user=keys["user"],
        password=keys["password"],
        warehouse=keys["warehouse"],
        database=keys["database"],
        client_session_keep_alive=keep_alive,
        # fixes weird 11/14/24 issue causing nftpulse fetches to not
        # work for 
        # https://github.com/snowflakedb/snowflake-connector-python/issues/2094
        insecure_mode=True,
        **kwargs,
    )
    return conn
    
    # Acquire a connection from the pool within each task
def run_query_with_pool(
    sql_query: str,
    sql_values: Optional[Union[Tuple, List[Dict]]] = None,
    convert_to_df=True,
) -> Any:
    if not POOL_INITIALIZED:
        init_snowflake_conn_pool()

    conn = connections.get()
    try:
        cursor = conn.cursor()
        cursor.execute(sql_query, sql_values)
        rows = cursor.fetchall()

        if convert_to_df:
            df = pd.DataFrame(
                rows,
                columns=[
                    # lowercase col names makes things easier
                    desc[0].lower()
                    for desc in cursor.description
                ],
            )
            return df
        else:
            return rows
    except Exception as err:
        breakpoint()
        print(1)
    finally:
        connections.put(conn)  # Return the connection to the pool
    ```

@sfc-gh-dszmolka
Copy link
Contributor

@opqpop & folks, this looks to be some issue in GCP US CENTRAL, when verifying the certificate during Snowflake/GCP Bucket connection. So far, could not reproduce with an account in GCP US CENTRAL, so if any of you has the possibility to try with the method described in this comment in the environment where the issue reproduces for you, and shares the debug logs, that could potentially massively help. Thanks in advance !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug status-triage_done Initial triage done, will be further handled by the driver team
Projects
None yet
Development

No branches or pull requests

4 participants