Fix: Resolve conflicting schema issues by removing dependency on PyArrow

[aaronsteers]

Resolves: #89

We've discovered through user testing that the PyArrow dependency, and PyArrow itself, are inherently fragile when it comes to JSON schemas with variable typing. PyArrow exceeds at handling large volumes of typed data very quickly, but there's no way to process ragged and potentially conflicting types without reverting json types to strings.

Paqruet, similarly, has problems rendering json structures without child nodes. So, something like "extra_properties: {}" would cause a runtime error.

We've recently moved our file writer to a JSONL file writer, in order to better handle ragged, non-populated objects, and unpredictable types. However, we still have a dependency on pyarrow, which is problematic for some users' workloads.

This PR will remove the pyarrow dependency and replace it with a flow that simply writes JSONL lines to local files as quickly as possible.

[aaronsteers]

Performance snapshot, prior to changes (approx 15K records per second):

AJ-Steers-MacBook-Pro---GGHWM7QWPJ:airbyte-lib-private-beta ajsteers$ poetry run python ./examples/run_faker.py

Enter the value for secret 'DUMMY_SECRET': 
Installing Faker source...
Faker source installed.
Connection check succeeded for `source-faker`.
Started `source-faker` read operation at 09:23:55...

                                                    Read Progress                                                     

Started reading at 17:23:55.                                                                                          

Read 500,100 records over 33 seconds (15,154.5 records / second).                                                     

Wrote 500,100 records over 53 batches.                                                                                

Finished reading at 17:24:28.                                                                                         

Started finalizing streams at 17:24:28.                                                                               

Finalized 53 batches over 3 seconds.                                                                                  

Completed 3 out of 3 streams:                                                                                         

 • purchases                                                                                                          
 • products                                                                                                           
 • users                                                                                                              

Completed writing at 17:24:32. Total time elapsed: 36 seconds                                                         

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Completed `source-faker` read operation at 09:24:32.
Stream products: 100 records
Stream users: 250000 records
Stream purchases: 250000 records
AJ-Steers-MacBook-Pro---GGHWM7QWPJ:airbyte-lib-private-beta ajsteers$ 

[aaronsteers]

Refactoring is functionally complete - although still some cleanup to do.

Performance is boosted by approximately 30% - from 15K records per second to 20K records per second.

AJ-Steers-MacBook-Pro---GGHWM7QWPJ:PyAirbyte ajsteers$ poetry run python ./examples/run_faker.py

Enter the value for secret 'DUMMY_SECRET': 
Installing Faker source...
Faker source installed.
Connection check succeeded for `source-faker`.
Started `source-faker` read operation at 10:51:43...
Completed `source-faker` read operation at 10:52:13.
Stream products: 100 records
Stream users: 250000 records
Stream purchases: 250000 records

                                               Read Progress                                               

Started reading at 18:51:43.                                                                               

Read 500,100 records over 25 seconds (20,004.0 records / second).                                          

Wrote 500,100 records over 51 batches.                                                                     

Finished reading at 18:52:09.                                                                              

Started finalizing streams at 18:52:09.                                                                    

Finalized 51 batches over 3 seconds.                                                                       
                                                    ...                                                    AJ-Steers-MacBook-Pro---GGHWM7QWPJ:PyAirbyte ajsteers$ 

[aaronsteers]

@bindipankhudi - Tests are passing locally and lint cleanup is complete.

I am taking another pass through the code myself but wanted to ping you of the latest status.

[bindipankhudi]

Sounds great! I am taking a look also, but don't wait on it. Feel free to merge.

…

On Thu, Mar 7, 2024 at 1:17 PM aaronsteers ***@***.***> wrote: @bindipankhudi <https://github.com/bindipankhudi> - Tests are passing locally and lint cleanup is complete. I am taking another pass through the code myself but wanted to ping you of the latest status. — Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BF2EJSTJIV6LA2G3D4EU2FDYXDKNVAVCNFSM6AAAAABDZAGBPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBUGUYDONRXGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bindipankhudi]

Quite some refactoring. Looks great and cleaner than before!