Feature: Module that allows configuration overrides via environment variables and/or system properties

[jdimeo]

This is a feature suggestion to allow overrides on an arbitrary object via environment variables and system properties.

Given a configuration bean (or any bean) that has been de serialized by a configuration YAML or (H)JSON file, this allows ad-hoc/at runtime overrides of the values in the persistent data source. It stores any env var or prop starting with a prefix in a hash map (for example MYAPP_FOO_BAR=1 or -Dmyapp.foo.bar=1) and then it turns the deserialized object back into a JsonTree, injects any discovered values from the environment, and recreates the object.

It's not ready for a pull request yet because I'm not sure this is in scope and I'm not sure about my implementation.
Please see it here:
https://github.com/ElderResearch/commons-jvm/blob/master/core/src/main/java/com/elderresearch/commons/lang/config/EnvironmentTree.java
specifically the applyOverrides()

[cowtowncoder]

I really do not like use of environment variables or system properties -- not using any is a conscious decision, not oversight.
My main problem is that Jackson components are often used by other libraries, frameworks, and given setup may have multiple instances. Sysprops/env are global settings and can and do affect all such usage, and my experiences with libraries that require such settings in the past have not been positive (but I only object to these being the only config mechanism -- if there are alternatives that's fine).

But this only means that I don't think core should add any support. It does not mean that one could not create a module or extension that offered such functionality.
I would be happy to help anyone who might want to build a system that uses extension points to offer configuration-from-env.

[jdimeo]

How would I begin building an extension e.g. jackson-environment? does that fit in an existing repo, or should I build it under my company's namespace and send it to you and the other maintainers for review?

The use case I'm envisioning is OK with global state because it's for overriding configuration values. Let me try again to illustrate the use case (how I'm using this IRL). Say for example my config was:

---
database:
  url: db-host:1234/db
  user: user
  pass: *****
otherConfig:
  otherKey: otherValue

and I had that saved in a file myapp-config.yaml. When myapp runs, it uses Jackson's awesomeness to load a Java bean from the config as given in the file.

Now, say I'm testing from my IDE. It's a pain to either a) temporarily change the saved YAML file on disk to go back and forth between hitting localhost for testing and the real server or b) managing two separate config files. Instead, using the EnvironmentTree, I can define the following env var in my Eclipse run configuration:

MYAPP_DATABASE_URL=localhost:1234/db

Now, the base configuration is loaded as normal, but the environment variable overrides the real DB server only when I run Eclipse- the run configuration is able to set that one configuration value only without changing the file itself.

Likewise, say I want to test something once it's compiled into an executable. Instead of changing the file, since this is a tentative/practice change, I can do:

> ./myapp -Dmyapp.otherConfig.otherKey=testValue start ...

and CLI support is "built in"/"for free".

The way I've implemented it, only environment values that start with a prefix (in this case myapp) are considered, and because it scans the object tree, it's robust enough to figure out when myapp_key1_key2 is bean.getKey1().setKey2(...) vs. bean.setKey1Key2(...).

All of this creates the "best of all worlds" configuration system I've seen to date. The closest is stuff like HOCON that has the ability to specify {env:KEY} as part of the file, but then that still has to be in the file itself and doesn't allow ad-hoc env override at anywhere in the object tree. Most other systems are config.getString("key") which misses out on all the powerful typing we get from Jackson.

If you don't see the use case/vision, I totally understand! thanks for your time :-)

Also, using this environment-aware extension would be 100% opt-in, but perhaps could be as seamless as a call like mapper.enable(Feature.ENVIRONMENT_...). It can be clearly communicated that the environment overrides will impact every single instance of an overridden class.

[cowtowncoder]

I think that an extension would typically start in its own repo, under whatever personal or org account is easiest to access.
It may even be initially published from there (if publishing is desired). If and when it becomes mature enough it could be moved to under FasterXML org as separate repo or merged into one of existing multi-project repos.

Thank you for explaining the use case -- this makes a lot of sense, and I have no objections to using env/system property values into modifying content, as they are often used for services in that way. My objections would only relate to configuring (most?) aspects of Jackson itself (although if opt-in, via extensions, that should also be fine).

Now as to implementation, on modifying content when reading/writing (if I understand it correctly): that is trickier, just due to lack of extension points. There are some long-standing wishes to expose extension points, f.ex:

FasterXML/jackson-core#355

which I thought could be defined for this kind of use case (on reader-side).

I would be open to adding extension points, and those could/would then be used by actual module to plug-in sources for values, I think.

(apologies if I misunderstood the ideas... I sometimes skim things too fast :) )