Celery won't start on MacOs

[AndyMik90]

Self Checks

• This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.3

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

Start Celery with "celery -A app.celery worker -P gevent -c 1 -Q dataset,generation,mail --loglevel INFO" on latest version

✔️ Expected Behavior

Celery has worked just fine up until now, don't know if its something I've done? Tried with fresh Enviroment in python aswell

❌ Actual Behavior

`Usage: celery [OPTIONS] COMMAND [ARGS]...
Try 'celery --help' for help.

Error: Invalid value for '-A' / '--app':
Unable to load celery application.
While trying to load the module app.celery the following error occurred:
Traceback (most recent call last):
File ".../site-packages/celery/bin/celery.py", line 58, in convert
return find_app(value)
File ".../site-packages/celery/app/utils.py", line 383, in find_app
sym = symbol_by_name(app, imp=imp)
File ".../site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File ".../site-packages/celery/utils/imports.py", line 109, in import_from_cwd
return imp(module, package=package)
File ".../importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File ".../api/app.py", line 8, in
import grpc.experimental.gevent
File ".../site-packages/grpc/init.py", line 22, in
from grpc import _compression
File ".../site-packages/grpc/_compression.py", line 20, in
from grpc._cython import cygrpc
ImportError: dlopen(.../site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_CFDataGetBytes'
`

[dosubot[bot]]

The error you're encountering with Celery not starting on macOS seems to be related to an issue with the grpc package, specifically with loading the grpc._cython.cygrpc module. This could be due to a variety of reasons such as a corrupted installation of grpc, compatibility issues between the versions of Celery, gevent, and grpcio, or problems related to compiling and running native extensions on macOS.

Here's a consolidated approach to troubleshoot and potentially resolve the issue:

1. Reinstall gRPC: Start by ensuring your grpcio package is correctly installed and up to date. Use pip install --upgrade grpcio to update or reinstall it.

2. Verify Compatibility: Check that the versions of Celery, gevent, and grpcio are compatible. Incompatibilities between these packages can lead to the kind of error you're seeing.

3. Reinstall gevent: Make sure gevent is installed with Cython support, which could resolve issues related to native extensions. Use pip install --no-cache-dir --force-reinstall gevent to reinstall it.

4. Run Outside of VS Code: If you're using VS Code, try running the Celery command directly in a terminal outside of VS Code. This can help identify if the issue is related to VS Code's environment.

5. Install Xcode Command Line Tools: On macOS, compiling and running native extensions sometimes requires Xcode Command Line Tools. Install them using xcode-select --install.

6. Check Your Virtual Environment: Ensure you're using the correct virtual environment where all dependencies are installed, especially if you're working within VS Code.

If these steps do not resolve the issue, consider checking the issue trackers for Celery, gevent, and grpcio for similar issues or posting a detailed question with your setup and the error message for further assistance.

Sources

• api/.vscode/launch.json
• api/requirements.txt

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[takatost]

trying pip install --upgrade -r requirements.txt?

[AndyMik90]

trying pip install --upgrade -r requirements.txt?

Yes, still same error.

Also tried with clean conda environment.

The Api server is running fine.

[AndyMik90]

Works after installing grpcio

[AndyMik90]

pip install --upgrade grpcio

[AndyMik90]

Note that the issue arised again, but since we use qdrant I made the following change:

api/app.py

monkey.patch_all() if os.environ.get("VECTOR_STORE") == 'milvus': import grpc.experimental.gevent grpc.experimental.gevent.init_gevent()