Flexible switching #1

mikecmpbll · 2017-06-26T16:29:34Z

No description provided.

When the gem is initialised and a schema adapter is being used, the adapter will initialise the schema_search_path which involves touching the database, and thus checking out a connection for the current thread. For threaded web servers, it is not expected that the main thread has a database connection, so you typically set the connection pool size equal to the number of web server threads. This patch checks the connection back in to the pool after init so that a web server thread can pick it up and use it.

* Use public_suffix for smarter subdomain parsing * Check validity of domain and add tests for localhost and IP * Remove unnecessary TLD length test

This is a complete rewrite of the adapters & of the test suite. - Test suite rewritten in minitest - Adapters handle separate physical tenant hosts seamlessly - Supports Rails 5.1 and above only - Dropped all adapters other than PG and MySQL (didn't get time, will accept PRs) - Added 'tenant resolvers' for generating full database configurations from tenant names - Added `tenant_decorator` option for using a proc to decorate your tenant database/schema names with Rails.env or anything else - All excluded models share a connection pool - All adapters & switching strategies are threadsafe

Still some issues around multithreading, I'm not confident it's correct. Need to improve tests and fix postgresql database drop problem :/.

Use a Thread local in the connection_specification_name to isolate each thread's connection config.

rubendinho · 2019-09-27T17:47:43Z

@mikecmpbll - is this PR meant to resolve issues like influitive#615 and influitive#311?

Looking to use Apartment with Puma and Postgres connection-switching, and running into issues running tests.

mikecmpbll · 2019-09-27T18:48:02Z

That right, yep. A lot has happened recently actually. We’ve been using this branch in production across multiple hosts, using threads, and serving approximately 50-100 requests per second, for about three weeks now. We’ve not experienced any major issues so far. We’ve fixed a number of small bugs over the last week or two.

cgratigny · 2019-10-09T05:00:31Z

@mikecmpbll this is really hopeful for what I'm doing, having threadsafe issues with Rails 6 + Postgres.

Do you have any example for what you are running instead of rake db:drop?

On development we often will do db:drop, db:create, db:migrate but the drop of course says not implemented.

mikecmpbll · 2019-10-09T10:24:08Z

hey @cgratigny . the postgres adapter isn't as well tested as the mysql which i use in production, although i did attempt to replicate all the existing tests for pg. because there's two ways to separate tenents on pg (schemas and databases), there's two methods drop_database and drop_schema.

there's no config option anymore for whether you're using schemas or databases and you can use both at the same time, so i guess there's no way for apartment to know whether you're trying to drop the tenant's database or schema without explicitly calling one of those methods.

not sure what the best approach should be for the drop method.

rubendinho · 2019-10-14T14:58:45Z

@mikecmpbll - Thanks so much for putting in so much work into this project. Do you have plans to eventually merge this PR into the main Apartment repo? If so, any idea when that might happen? We're deciding between schema-based and connection-switching approaches (Rails 6 + Postgres + Puma), so any ETA would be super helpful before we commit to one of the approaches.

cgratigny · 2019-10-14T21:08:04Z

@rubendinho I have a very strong interest in making this work with Postgres and database switching, we are limping along the best we can with some hacks.

mikecmpbll · 2019-10-14T22:36:00Z

Given there's demand I'll do a proper fork of this and release it to rubygems this week. I plan to merge it in bit by bit to apartment but seeing as it's a complete rewrite I don't think it'd be right by existing users to merge wholesale, especially in a domain as important as tenant data separation.

…

________________________________ From: Chris Gratigny <[email protected]> Sent: Monday, October 14, 2019 10:08:05 PM To: mikecmpbll/apartment <[email protected]> Cc: Mike Campbell <[email protected]>; Mention <[email protected]> Subject: Re: [mikecmpbll/apartment] Flexible switching (#1) @rubendinho<https://github.com/rubendinho> I have a very strong interest in making this work with Postgres and database switching, we are limping along the best we can with some hacks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1?email_source=notifications&email_token=AAO7GMNSIH5JNE7D47ZYM7DQOTNTLA5CNFSM4DQWGPP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBGRJVY#issuecomment-541922519>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAO7GMNLLET7IEQGYG4JBOLQOTNTLANCNFSM4DQWGPPQ>.

cgratigny · 2019-10-14T22:52:11Z

@mikecmpbll I'm willing to contribute to the Postgres database side of things, test etc.

mikecmpbll · 2019-10-14T22:55:41Z

Super. I'll get that done tomorrow.

…

________________________________ From: Chris Gratigny <[email protected]> Sent: Monday, October 14, 2019 11:52:11 PM To: mikecmpbll/apartment <[email protected]> Cc: Mike Campbell <[email protected]>; Mention <[email protected]> Subject: Re: [mikecmpbll/apartment] Flexible switching (#1) @mikecmpbll<https://github.com/mikecmpbll> I'm willing to contribute to the Postgres database side of things, test etc. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1?email_source=notifications&email_token=AAO7GMP5WHKLKKDNPSMC2X3QOTZZXA5CNFSM4DQWGPP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBG3TUI#issuecomment-541964753>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAO7GMLTFUW5YC2PVD3H5ULQOTZZXANCNFSM4DQWGPPQ>.

rubendinho · 2019-10-19T18:18:42Z

@mikecmpbll and @cgratigny - our use-case requires physical data separation (PG schemas are not sufficient), and I'd also be happy to contribute to the PG side of things, including using Heroku's API to manage database provision, if that's of interest to folks.

mikecmpbll · 2019-10-21T14:20:54Z

@mikecmpbll Any reason the first_subdomain elevator was removed here?

because it was no different to the other subdomain elevator as far as i could tell.

i've forked our active branch here; https://github.com/cpoms/innkeeper and released to rubygems https://rubygems.org/gems/innkeeper. i've not fully tested it after renaming the whole thing.

cgratigny · 2019-10-21T14:29:33Z

@rubendinho great! The of side needs some love on this for sure. I'll be physically separating in the future as well, just not there yet.

I'm not on Heroku but know a lot of people use it.

NeelTandel-BTC · 2020-02-24T10:15:00Z

@mikecmpbll any update regarding this #PR?

pradeepibz · 2020-07-29T07:10:37Z

Hi @mikecmpbll - I'm using this gem from your branch 'flexible-switching' in my old application got some compatible version issue.

My Dependencies:
Rails - 5.1.4
Ruby - 2.3.3

My errors when tried to do the bundle update:

Bundler could not find compatible versions for gem "actionmailer":
In Gemfile:
exception_notification was resolved to 4.4.3, which depends on
actionmailer (>= 4.0, < 7)

rails (~> 5.1.5.rc1) was resolved to 5.1.5.rc1, which depends on
  actionmailer (= 5.1.5.rc1)

Bundler could not find compatible versions for gem "actionpack":
In Gemfile:
axlsx_rails was resolved to 0.6.1, which depends on
actionpack (>= 3.1)

haml-rails was resolved to 2.0.1, which depends on
  actionpack (>= 5.1)

rails (~> 5.1.5.rc1) was resolved to 5.1.5.rc1, which depends on
  actionpack (= 5.1.5.rc1)

rspec-rails (~> 3.6) was resolved to 3.9.1, which depends on
  actionpack (>= 3.0)

Bundler could not find compatible versions for gem "activerecord":
In Gemfile:
apartment was resolved to 2.0.0, which depends on
activerecord (>= 5.2.0)

rails (~> 5.1.5.rc1) was resolved to 5.1.5.rc1, which depends on
  activerecord (= 5.1.5.rc1)

Bundler could not find compatible versions for gem "acts_as_list":
In Gemfile:
acts_as_list

Bundler could not find compatible versions for gem "json":
In Gemfile:
json (~> 1.8, >= 1.8.3)

instagram_graph_api was resolved to 0.0.10, which depends on
  koala was resolved to 3.0.0, which depends on
    json (>= 1.8)

link_thumbnailer was resolved to 3.4.0, which depends on
  json (>= 1.7.7)

sendinblue was resolved to 2.4, which depends on
  json

Bundler could not find compatible versions for gem "listen":
In Gemfile:
listen (>= 3.0.5, < 3.2)

spring-watcher-listen (~> 2.0.0) was resolved to 2.0.1, which depends on
  listen (>= 2.7, < 4.0)

Bundler could not find compatible versions for gem "rails":
In Gemfile:
rails (~> 5.1.5.rc1)

browser-timezone-rails was resolved to 1.1.0, which depends on
  rails (>= 3.1)

Bundler could not find compatible versions for gem "redis":
In Gemfile:
redis (< 4)

capistrano-resque (~> 0.2.2) was resolved to 0.2.3, which depends on
  resque-scheduler was resolved to 4.4.0, which depends on
    redis (>= 3.3)

sidekiq-scheduler was resolved to 3.0.1, which depends on
  redis (>= 3, < 5)

Bundler could not find compatible versions for gem "ruby":
In Gemfile:
ruby (~> 2.4.4.0)

acts_as_list was resolved to 1.0.1, which depends on
  ruby (>= 2.4.7)

Bundler could not find compatible versions for gem "web-console":
In Gemfile:
web-console (>= 3.3.0)

Help me to solve this issue

When the initial connection is to an invalid tenant (non-existant db, for example), the app process can never successfully reconnect to a valid tenant.

Ludicrous amount of rescues.

@current

If it does, we lose the consistency of @current, which seems to be what's happening.

When the process is forked or a new thread is started, set-up our ActiveRecord connection_specification_name overrides again because programs like Passenger will now call `ActiveRecord::Base.establish_connection` when they spawn new app processes, which in turn cause `ActiveRecord::Base.connection_specification_name` to be reset to whatever is specified in database.yml (or :primary as default). I can't see that this could have any adverse affects. If the app sets `connection_specification_name` on anything other than `Apartment.connection_class` (`ActiveRecord::Base` by default) that should be unaffected and preserved. This is a more generic solution that hooking into some Passenger-specific after-spawn hook.

hamza-costiq · 2020-12-15T09:48:09Z

When the threading issue with Puma will be resolved in Apartment Gem ?

Mike Campbell and others added 11 commits June 15, 2017 12:45

Accept DB config

959432f

Use public_suffix for smarter subdomain parsing (influitive#309)

77f8e65

* Use public_suffix for smarter subdomain parsing * Check validity of domain and add tests for localhost and IP * Remove unnecessary TLD length test

Removing deprecated methods.

3061e99

Add Rails ~5.0.0 to travis and remove ~3.2.0

194b289

Fix excluded_model reset during tests & railtie test

7007f0f

Update readme & generator template

724940b

Improve connection pooling

17edd0e

Still some issues around multithreading, I'm not confident it's correct. Need to improve tests and fix postgresql database drop problem :/.

Fixed connection handling to be threadsafe

323fdfc

Use a Thread local in the connection_specification_name to isolate each thread's connection config.

Make threads test more stressful

d242773

derosm2 mentioned this pull request Mar 18, 2019

Flexible switching with rails 5.2 derosm2/apartment#1

Closed

Ryan Brunner and others added 4 commits May 28, 2019 13:05

Run migrations in parallel

b742ad0

Don't fail when default tenant doesn't exist

2d9e7e1

Fix method missing in abstract adapter

57e65ef

Fix creating tenants on other hosts

75a68f6

Use AR 5.2 migration_context

184cbce

mikecmpbll added 10 commits August 18, 2020 11:56

Fix unrecoverable invalid tenant loop

89db3a0

When the initial connection is to an invalid tenant (non-existant db, for example), the app process can never successfully reconnect to a valid tenant.

Ensure even resetting doesn't fail

84fa477

Ludicrous amount of rescues.

Try to only reconnect if broken connection

eaa572e

Tidy up logic ...

1ff7849

A switch should never totally fail

da67f09

If it does, we lose the consistency of @current, which seems to be what's happening.

Turn logging up to 10...

ebeb824

Make connection switching resilient

7b00853

Prevent invalid tenant names from getting through

4730352

Access config with_indifferent_access

c79c8a4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flexible switching #1

Flexible switching #1

mikecmpbll commented Jun 26, 2017

rubendinho commented Sep 27, 2019

mikecmpbll commented Sep 27, 2019

cgratigny commented Oct 9, 2019

mikecmpbll commented Oct 9, 2019

rubendinho commented Oct 14, 2019

cgratigny commented Oct 14, 2019

mikecmpbll commented Oct 14, 2019 via email

cgratigny commented Oct 14, 2019

mikecmpbll commented Oct 14, 2019 via email

rubendinho commented Oct 19, 2019

mikecmpbll commented Oct 21, 2019

cgratigny commented Oct 21, 2019

NeelTandel-BTC commented Feb 24, 2020

pradeepibz commented Jul 29, 2020

hamza-costiq commented Dec 15, 2020

Flexible switching #1

Are you sure you want to change the base?

Flexible switching #1

Conversation

mikecmpbll commented Jun 26, 2017

rubendinho commented Sep 27, 2019

mikecmpbll commented Sep 27, 2019

cgratigny commented Oct 9, 2019

mikecmpbll commented Oct 9, 2019

rubendinho commented Oct 14, 2019

cgratigny commented Oct 14, 2019

mikecmpbll commented Oct 14, 2019 via email

cgratigny commented Oct 14, 2019

mikecmpbll commented Oct 14, 2019 via email

rubendinho commented Oct 19, 2019

mikecmpbll commented Oct 21, 2019

cgratigny commented Oct 21, 2019

NeelTandel-BTC commented Feb 24, 2020

pradeepibz commented Jul 29, 2020

hamza-costiq commented Dec 15, 2020