Pipedrive custom fields not moved to S3 and Postgres

[moligslagers]

I am running Airbyte in an EC2 instance on AWS. I have connected our Pipedrive CRM to an RDS Postgres Instance and to S3. Neither of the destinations include custom fields in the raw data. For S3 I have also tried the S3 Glue Connector with the same result.

I found this: https://discuss.airbyte.io/t/pipedrive-normalization-custom-fields-don-t-show-on-entities/2138 and it indicates that the fields should be available at least before normalization, but even that is not the case.

I am running Airbyte 0.50.19 and the Pipedrive Connector version is 1.0.0. Any ideas?

[moligslagers]

I found two possile routes to go:

use octavia cli

• install the octavia cli as shown here
• mkdir airbyt_config
• octavia import all
• now go to connections/<connection>/configuration.yaml
• find your stream and add custom fields to the properties
• save the file
• octavia apply

airbyte does not show the additional fields in the UI, but when I rerun the sync, the fields show up

use the builder to create your own connector

You can create your own pipedrive connector in the "Builder". Its pretty straight forward. The connector will simply dump the entire Pipedrive API response into the S3 Bucket or the raw data tables. From here on out they are in your DWH and you can put it together yourself.

I will go with option 1 after all, since the result smoothly works with the S3 Glue connector and makes the data queriable in Athena immediately.

What I could not figure out is if there is a way to assign an alias to the the custom fields so that the custom fields are directly added as human readable fields to the source instead of the cryptic keys.