$feature_flag_called only reports bootstrap values if bootstrapping is used

[danbim]

Bug description

Without bootstrapping, the system works as expected: the client retrieves a feature flag value from the server and
reports accordingly, using the $feature_flag_called event with the value previously retrieved from the server.

With bootstrapping, the value reported with the $feature_flag_called event equals the bootstrapped value. There is however no
second $feature_flag_called event sent to PostHog containing the actual value later on retrieved from the server. As a
result, if you're using bootstrapping in your application you will only ever see the bootstrap values being reported in
the usages tab of the PostHog dashboard. AFAIU this will also cause wrong (exposure) numbers in A/B test experiments.

How to reproduce

I have created a "minimal" reproducible example for verification in form of an executable React Native application with activated PostHog debug logging. Please go to https://gitlab.holi.team/danbim/posthog-bug-demo and follow the instructions in the README.

Related sub-libraries

• All of them
• posthog-web
• posthog-node
• posthog-react-native

Our real application uses posthog-react-native and sees this problem. We also have a web version of our application which uses posthog-js for which we have not observed (nor validated) this problem.

I have previously contacted PostHog support (reference e776bf96-ec43-4d4f-8ec9-94a61b730592) but after initial back and forth I haven't received an answer for quite some time. To support the effort on your side, I have created the reproducible example.

Additional context

I'm suspecting

posthog-js-lite/posthog-core/src/index.ts

Line 1186 in ca271b8

if (this.sendFeatureFlagEvent && !this.flagCallReported[key]) {

to be the root cause of the issue. Once a $feature_flag_called event has been emitted for a given feature flag key, it will never be emitted again. However, I believe that in presence of bootstrapping, this is exactly what should happen. Therefore I guess that resetting flagCallReported after receiving a response from the server could be a potential solution.

Thank you for your bug report – we love squashing them!

[marandaneto]

Hello @danbim
This is a known issue, See PRs #259 and #260
I am still waiting for @Phanatic and @neilkakkar since they are the ones who should know how this should work, comment here.

[danbim]

Hi @marandaneto , thanks for the response. If I understand the MRs correctly and your message, then this means that the issue I see is caused by bootstrap values overwriting previously persisted (on device) values that were originally retrieved from PostHogs API. Is that correct?

If that's the case I fear there could still be an issue if the device has no values persisted yet, e.g. on fresh app installations. In this case I would guess that the $feature_flag_called event would still be emitted with the bootstrap value and not again within the same app session. But that's theoretical, not verified...

[danbim]

Just to add what (I believe) this bug means to us: we can currently not run A/B tests because the insights generated for the A/B test believe that every user is in the 'control' group while in fact 50% are in a 'test' group. So for us this blocks the usage of the A/B testing feature altogether.

[marandaneto]

@danbim

If I understand the MRs correctly and your message, then this means that the issue I see is caused by bootstrap values overwriting previously persisted (on device) values that were originally retrieved from PostHogs API. Is that correct?

correct

f that's the case I fear there could still be an issue if the device has no values persisted yet, e.g. on fresh app installations. In this case I would guess that the $feature_flag_called event would still be emitted with the bootstrap value and not again within the same app session. But that's theoretical, not verified...

in this case $feature_flag_called would be reported with the bootstrap values, every time that the flag is read/used.
Yes, there's some caching to not report the very same flag again.

posthog-js-lite/posthog-core/src/index.ts

Lines 1186 to 1192 in ca271b8

	if (this.sendFeatureFlagEvent && !this.flagCallReported[key]) {
	this.flagCallReported[key] = true
	this.capture('$feature_flag_called', {
	$feature_flag: key,
	$feature_flag_response: response,
	})
	}

Apparently the flagCallReported is never cleared out if the feature flags response changed, do you mean that this is the issue?

[danbim]

Yes that's precisely what I believe could be a (follow-up) issue if not addressed in the MRs right away :)

[marandaneto]

@danbim thanks for the issue, I think #281 should address the other issue we discussed.