Add own service for memcached #8818

EHJ-52n · 2022-02-17T10:36:33Z

Description

Goal: "one service per container"

Background

Docker documentation: "Run multiple services in a container"
devops.stackexchange: "Why it is recommended to run only one process in a container?"

Changes

add own service for memcached
remove memcached from geonode image
add MEMCACHED_OPTIONS variable to env files
~~enable memcached by default~~

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

Confirm you have read the contribution guidelines
You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
The issue connected to the PR must have Labels and Milestone assigned
PR for bug fixes and small new features are presented as a single commit
Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented
This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
This PR passes the QA checks: flake8 geonode
Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates
Commits adding new texts do use gettext and have updated .po / .mo files (without location infos)

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

cla-bot · 2022-02-17T10:36:35Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @EHJ-52n on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

gitguardian · 2022-02-17T10:37:09Z

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}}

codecov · 2022-02-17T11:09:16Z

Codecov Report

Merging #8818 (2ed1236) into master (006332b) will increase coverage by 0.00%.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8818   +/-   ##
=======================================
  Coverage   61.95%   61.95%           
=======================================
  Files         867      867           
  Lines       51577    51577           
  Branches     6467     6467           
=======================================
+ Hits        31954    31956    +2     
+ Misses      18084    18081    -3     
- Partials     1539     1540    +1

gannebamm · 2022-02-17T11:29:54Z

Welcome @EHJ-52n
I like the idea of this PR a lot.

There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)

I see this is a change that shall not be visible to end-users but nonetheless, I would like to have a proper improvement description as an issue.

@afabiani @giohappy Do you confirm my demand, or do you guys think this can be a PR without issue?

Same for #8817

afabiani

So, I know this is a bit confusing, but this option MEMCACHED_LOCATION is not currently driving the memcached lock, this one setting the frontend cache instead

MEMCACHED_ENABLED = ast.literal_eval(os.getenv('MEMCACHED_ENABLED', 'False'))
MEMCACHED_BACKEND = os.getenv('MEMCACHED_BACKEND', 'django.core.cache.backends.memcached.PyMemcacheCache')
MEMCACHED_LOCATION = os.getenv('MEMCACHED_LOCATION', '127.0.0.1:11211')
MEMCACHED_LOCK_EXPIRE = int(os.getenv('MEMCACHED_LOCK_EXPIRE', 3600))
MEMCACHED_LOCK_TIMEOUT = int(os.getenv('MEMCACHED_LOCK_TIMEOUT', 10))

if MEMCACHED_ENABLED:
    CACHES['default'] = {
        'BACKEND': MEMCACHED_BACKEND,
        'LOCATION': MEMCACHED_LOCATION,
    }

The celery tasks will use only the MEMCACHED_LOCATION variable instead

try:
    import pylibmc
    import sherlock
    from sherlock import MCLock as Lock

    sherlock.configure(
        expire=settings.MEMCACHED_LOCK_EXPIRE,
        timeout=settings.MEMCACHED_LOCK_TIMEOUT)
    memcache_client = pylibmc.Client(
        [settings.MEMCACHED_LOCATION],
        binary=True)
    lock_type = "MEMCACHED"
except Exception:
    from django.core.cache import cache
    from contextlib import contextmanager
    lock_type = "MEMCACHED-LOCAL-CONTEXT"
    memcache_client = None

It would be probably better to change this and rename the variable so that it is very clear which one enables what, and also having the MEMCACHED_ENABLED driving the tasks behavior.

We need to revise completely this one I'm afraid.

EHJ-52n · 2022-03-01T13:45:41Z

...this option MEMCACHED_LOCATION is not currently driving the memcached lock...

@afabiani I am unsure what you are referring to by this?

I see this is a change that shall not be visible to end-users...

@gannebamm what do you want to say by this? Should the memcached configuration not be visible to "end-users"? Is an admin operating GeoNode an end-user here?

afabiani · 2022-03-02T10:58:59Z

@EHJ-52n please take a look here

as you can see the locking mechanism is created only taking into account two conditions:

sherlock is installed
it is able to create a lock

    import pylibmc
    import sherlock
    from sherlock import MCLock as Lock

    sherlock.configure(
        expire=settings.MEMCACHED_LOCK_EXPIRE,
        timeout=settings.MEMCACHED_LOCK_TIMEOUT)
    memcache_client = pylibmc.Client(
        [settings.MEMCACHED_LOCATION],
        binary=True)
    lock_type = "MEMCACHED"

no check is done against the MEMCACHED_ENABLED boolean

gannebamm · 2022-03-03T15:35:34Z

I see this is a change that shall not be visible to end-users...

@gannebamm what do you want to say by this? Should the memcached configuration not be visible to "end-users"? Is an admin operating GeoNode an end-user here?

I was referring to the checklist item

There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)

And was thinking out loud, that we do not need an issue for that PR, since it is not something end-users will face.

cla-bot · 2022-04-07T11:11:43Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @EHJ-52n on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

EHJ-52n · 2022-04-07T11:34:21Z

@afabiani

Are you recommending to rephrase MEMCACHED_LOCK_EXPIRE to SHERLOCK_LOCK_EXPIRE or CACHE_LOCK_EXPIRE and MEMCACHED_LOCK_TIMEOUT, too?

In addition, surrounding the memcache_client initialization with a check for MEMCACHE_ENABLED?

try:
    import pylibmc
    import sherlock
    from sherlock import MCLock as Lock

    sherlock.configure(
        expire=settings.SHERLOCK_LOCK_EXPIRE,
        timeout=settings.SHERLOCK_LOCK_TIMEOUT)
    if settings.MEMCACHED_ENABLED:
        memcache_client = pylibmc.Client(
            [settings.MEMCACHED_LOCATION],
            binary=True)
        lock_type = "MEMCACHED"
    else:
        raise Exception
except Exception:
    from django.core.cache import cache
    from contextlib import contextmanager
    lock_type = "MEMCACHED-LOCAL-CONTEXT"
    memcache_client = None

I am not sure if the else case is required because I did not fully check the task module.

giohappy · 2023-09-12T08:52:00Z

@EHJ-52n can you please take a look at the conflicts?

Goal: one service per container - add own service for memcached - remove memcached from geonode image - add MEMCACHED_OPTIONS variable to env files - enable memcached by default

EHJ-52n · 2023-09-12T15:09:00Z

@giohappy Fixed the conflicts. Are you okay with the changes?

giohappy · 2023-09-13T09:03:53Z

@EHJ-52n I have gone through the PR and comments.

IMHO the thing about MEMCACHE_ENABLED driving the Django cache but not the locking mechanism is outside this PR.
I agree that mixing different mechanisms with variables sharing the same prefix can be confusing and error-prone, but it's something that should be addressed in another task.

Possible solutions (for another task):

make MEMCACHE_ENABLED dive both (as in the last example from @EHJ-52n)
rename MEMCACHE_ENABLED to something like DJANGO_MEMCACHE_ENABLED

giohappy · 2023-09-13T09:42:45Z

@EHJ-52n please align geonode-project once this is merged

EHJ-52n · 2023-09-13T14:21:47Z

@giohappy working on this - see GeoNode/geonode-project#479.

fixes GeoNode#479 see GeoNode/geonode#8818

fixes #479 see GeoNode/geonode#8818

gannebamm self-assigned this Feb 17, 2022

gannebamm self-requested a review February 17, 2022 16:17

afabiani requested changes Feb 18, 2022

View reviewed changes

Add own service for memcached

2ed1236

Goal: one service per container - add own service for memcached - remove memcached from geonode image - add MEMCACHED_OPTIONS variable to env files - enable memcached by default

EHJ-52n force-pushed the feature/memcached-own-service branch from dbc8702 to 2ed1236 Compare September 12, 2023 14:40

cla-bot bot added the cla-signed CLA Bot: community license agreement signed label Sep 12, 2023

giohappy requested a review from afabiani September 13, 2023 09:04

afabiani approved these changes Sep 13, 2023

View reviewed changes

giohappy merged commit 9626fbd into GeoNode:master Sep 13, 2023
18 checks passed

giohappy added this to the 4.2.0 milestone Sep 13, 2023

giohappy added the docker Issues specific to GeoNode docker or GeoNode SPC label Sep 13, 2023

giohappy mentioned this pull request Sep 13, 2023

Revert to using localhost for memcached inside development .env files #11477

Closed

EHJ-52n mentioned this pull request Sep 13, 2023

Add own service for memcached (see GN#8818) GeoNode/geonode-project#479

Closed

EHJ-52n added a commit to 52North/geonode-project that referenced this pull request Sep 13, 2023

Add memcached as own service

8171c96

fixes GeoNode#479 see GeoNode/geonode#8818

EHJ-52n mentioned this pull request Sep 13, 2023

Add memcached as own service GeoNode/geonode-project#480

Merged

giohappy pushed a commit to GeoNode/geonode-project that referenced this pull request Sep 13, 2023

Add memcached as own service (#480)

51a3781

fixes #479 see GeoNode/geonode#8818

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add own service for memcached #8818

Add own service for memcached #8818

EHJ-52n commented Feb 17, 2022 •

edited

Loading

cla-bot bot commented Feb 17, 2022

gitguardian bot commented Feb 17, 2022 •

edited

Loading

codecov bot commented Feb 17, 2022 •

edited

Loading

gannebamm commented Feb 17, 2022 •

edited

Loading

afabiani left a comment

EHJ-52n commented Mar 1, 2022

afabiani commented Mar 2, 2022

gannebamm commented Mar 3, 2022

cla-bot bot commented Apr 7, 2022

EHJ-52n commented Apr 7, 2022

giohappy commented Sep 12, 2023

EHJ-52n commented Sep 12, 2023

giohappy commented Sep 13, 2023 •

edited

Loading

giohappy commented Sep 13, 2023

EHJ-52n commented Sep 13, 2023

Add own service for memcached #8818

Add own service for memcached #8818

Conversation

EHJ-52n commented Feb 17, 2022 • edited Loading

Description

Checklist

cla-bot bot commented Feb 17, 2022

gitguardian bot commented Feb 17, 2022 • edited Loading

️✅ There are no secrets present in this pull request anymore.

codecov bot commented Feb 17, 2022 • edited Loading

Codecov Report

gannebamm commented Feb 17, 2022 • edited Loading

afabiani left a comment

Choose a reason for hiding this comment

EHJ-52n commented Mar 1, 2022

afabiani commented Mar 2, 2022

gannebamm commented Mar 3, 2022

cla-bot bot commented Apr 7, 2022

EHJ-52n commented Apr 7, 2022

giohappy commented Sep 12, 2023

EHJ-52n commented Sep 12, 2023

giohappy commented Sep 13, 2023 • edited Loading

giohappy commented Sep 13, 2023

EHJ-52n commented Sep 13, 2023

EHJ-52n commented Feb 17, 2022 •

edited

Loading

gitguardian bot commented Feb 17, 2022 •

edited

Loading

codecov bot commented Feb 17, 2022 •

edited

Loading

gannebamm commented Feb 17, 2022 •

edited

Loading

giohappy commented Sep 13, 2023 •

edited

Loading