AIL Framework not starting v4.2 (same like #442)

[lobojc]

Hello AIL Framework team, I have the same problem running AIL Framework, the Website doesn't start.

I installed pybgpranking manually because it failed and I updated jsonschema to 4.9.1

ERROR: pymisp 2.4.160.1 has requirement jsonschema<5.0.0,>=4.9.1, but you'll have jsonschema 3.2.0 which is incompatible.

After check the screens I detected:

1. Flask_AIL ERROR:

Flask_base_template.py  create_default_user.py    server.key
Flask_server.py         create_new_web_module.py  static
__pycache__             modules                   templates
blueprints              server.crt                update_thirdparty.sh
Traceback (most recent call last):
  File "./Flask_server.py", line 24, in <module>
    from pytaxonomies import Taxonomies
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
    from .api import Taxonomies, Taxonomy, Predicate, Entry
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
    import requests
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
    from . import packages, utils
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/utils.py'

2. Script_AIL ERROR:

Flask_base_template.py  create_default_user.py    server.key
Flask_server.py         create_new_web_module.py  static
__pycache__             modules                   templates
blueprints              server.crt                update_thirdparty.sh
Traceback (most recent call last):
  File "./Flask_server.py", line 24, in <module>
    from pytaxonomies import Taxonomies
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
    from .api import Taxonomies, Taxonomy, Predicate, Entry
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
    import requests
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
    from . import packages, utils
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/sugar/poll.py'

3. Redis_AIL info:

52383:M 10 Aug 2022 22:27:17.193 # Server initialized
52383:M 10 Aug 2022 22:27:17.193 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
52383:M 10 Aug 2022 22:27:17.193 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
52383:M 10 Aug 2022 22:27:17.193 * Ready to accept connections

4. Other screen without error messages !

PS: AIL Starting with ./LAUNCH.sh -l output without ERRORS:

    * Launching Redis servers
    * Launching ARDB servers
    * Launching logging process
    * Launching all the queues
    * Checking configuration

Config File: Nothing to update
Config File: Nothing to update
* Configuration up-to-date
* Launching core scripts ...
* Launching scripts
* Launching Flask server

Thank you

[lobojc]

BTW the errors change every time when I run LAUNCH -k and LAUNCH -l

1. SOME STOP + START ->

Flask_base_template.py  create_new_web_module.py  static
Flask_server.py         modules                   templates
blueprints              server.crt                update_thirdparty.sh
create_default_user.py  server.key
Traceback (most recent call last):
  File "./Flask_server.py", line 16, in <module>
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/__init__.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/json/__init__.py'

2. Another STOP + START ->

Flask_base_template.py  create_new_web_module.py  static
Flask_server.py         modules                   templates
blueprints              server.crt                update_thirdparty.sh
create_default_user.py  server.key
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/usr/lib/python3.8/site.py", line 597, in <module>
    main()
  File "/usr/lib/python3.8/site.py", line 580, in main
    known_paths = venv(known_paths)
  File "/usr/lib/python3.8/site.py", line 498, in venv
    with open(virtual_conf, encoding='utf-8') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/pyvenv.cfg'

[lobojc]

And to finish ./LAUNCH -t command output:

ubuntu@ailframework:~/ailframework/bin$ ./LAUNCH.sh -t
EEE
======================================================================
ERROR: Failure: PermissionError ([Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/ailframework/tests/testApi.py", line 17, in <module>
    from Flask_server import app
  File "/home/ubuntu/ailframework/var/www/Flask_server.py", line 160, in <module>
    importlib.import_module(name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/ubuntu/ailframework/var/www/modules/Tags/Flask_Tags.py", line 39, in <module>
    clusters = Clusters(skip_duplicates=True)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/api.py", line 294, in __init__
    with open(cluster_file, 'r') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json'
-------------------- >> begin captured logging << --------------------
pymisp: DEBUG: pymisp loaded properly
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/ailframework/tests/testHelper.py", line 9, in <module>
    from Helper import Process
  File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
    import zmq
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
    from zmq import backend
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
    raise original_error from None
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
    _ns = select_backend(first)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
    mod = import_module(name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory

======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/ailframework/tests/test_modules.py", line 15, in <module>
    from modules.ApiKey import ApiKey
  File "/home/ubuntu/ailframework/bin/modules/ApiKey.py", line 21, in <module>
    from modules.abstract_module import AbstractModule
  File "/home/ubuntu/ailframework/bin/modules/abstract_module.py", line 17, in <module>
    from Helper import Process
  File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
    import zmq
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
    from zmq import backend
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
    raise original_error from None
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
    _ns = select_backend(first)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
    mod = import_module(name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory

Name                                   Stmts   Miss  Cover
----------------------------------------------------------
bin/BankAccount.py                        72     72     0%
bin/Crawler.py                           281    281     0%
bin/Cryptocurrencies.py                   78     78     0%
bin/Cve.py                                26     26     0%
bin/DumpValidOnion.py                     23     23     0%
bin/Duplicates.py                        116    116     0%
bin/Helper.py                            158    153     3%
bin/IPAddress.py                          50     50     0%
bin/Languages.py                          21     21     0%
bin/MISP_The_Hive_feeder.py              156    156     0%
bin/Mail.py                              135    135     0%
bin/Mixer.py                             114    114     0%
bin/ModuleStats.py                        80     80     0%
bin/ModulesInformationV2.py              549    549     0%
bin/NotificationHelper.py                 52     52     0%
bin/PgpDump.py                           164    164     0%
bin/QueueIn.py                            13     13     0%
bin/QueueOut.py                           14     14     0%
bin/Release.py                            46     46     0%
bin/SourceCode.py                         39     39     0%
bin/Tools.py                              62     62     0%
bin/Update-conf.py                        59     59     0%
bin/Update.py                            329    329     0%
bin/__init__.py                            0      0   100%
bin/ailleakObject.py                      90     90     0%
bin/core/ail_2_ail.py                    852    697    18%
bin/core/screen.py                       122    105    14%
bin/empty_queue.py                        18     18     0%
bin/export/AILObjects.py                  39     22    44%
bin/export/Export.py                      49     26    47%
bin/export/MispExport.py                 331    276    17%
bin/export/MispImport.py                 172    140    19%
bin/import/importer.py                    61     34    44%
bin/indexer_lookup.py                     60     60     0%
bin/launch_queues.py                      49     49     0%
bin/lib/ConfigLoader.py                   41     11    73%
bin/lib/Correlate_object.py              371    327    12%
bin/lib/Decoded.py                       260    210    19%
bin/lib/Domain.py                        573    463    19%
bin/lib/Investigations.py                294    219    26%
bin/lib/Language.py                       27     18    33%
bin/lib/Screenshot.py                    161    120    25%
bin/lib/Tracker.py                       944    799    15%
bin/lib/Username.py                       13      1    92%
bin/lib/btc_ail.py                        40     31    22%
bin/lib/crawlers.py                     1009    833    17%
bin/lib/d4.py                             58     39    33%
bin/lib/domain_basic.py                   16      5    69%
bin/lib/exceptions.py                      8      2    75%
bin/lib/item_basic.py                    161    117    27%
bin/lib/objects/CryptoCurrencies.py       34     19    44%
bin/lib/objects/Decodeds.py               40     20    50%
bin/lib/objects/Domains.py                33     17    48%
bin/lib/objects/Items.py                 426    319    25%
bin/lib/objects/Pgps.py                   30     15    50%
bin/lib/objects/Screenshots.py            24      8    67%
bin/lib/objects/Usernames.py              31     16    48%
bin/lib/objects/abstract_object.py        57     31    46%
bin/lib/objects/ail_objects.py            54     27    50%
bin/lib/queues_modules.py                 53     34    36%
bin/modules/ApiKey.py                     50     44    12%
bin/modules/Categ.py                      49     49     0%
bin/modules/Credential.py                111    111     0%
bin/modules/CreditCards.py                40     40     0%
bin/modules/Decoder.py                    99     99     0%
bin/modules/DomClassifier.py              51     51     0%
bin/modules/Global.py                    122    122     0%
bin/modules/Hosts.py                      31     31     0%
bin/modules/Indexer.py                    89     89     0%
bin/modules/Keys.py                      116    116     0%
bin/modules/LibInjection.py               57     57     0%
bin/modules/Onion.py                      61     61     0%
bin/modules/Phone.py                      40     40     0%
bin/modules/SQLInjectionDetection.py      48     48     0%
bin/modules/SentimentAnalysis.py          98     98     0%
bin/modules/Tags.py                       26     26     0%
bin/modules/Telegram.py                   53     53     0%
bin/modules/Urls.py                       40     40     0%
bin/modules/Zerobins.py                   30     30     0%
bin/modules/__init__.py                    0      0   100%
bin/modules/abstract_module.py            52     46    12%
bin/modules/submit_paste.py              189    189     0%
bin/packages/Correlation.py              266    213    20%
bin/packages/Cryptocurrency.py            68     49    28%
bin/packages/Date.py                     127    101    20%
bin/packages/Hash.py                      30     19    37%
bin/packages/HiddenServices.py           231    190    18%
bin/packages/Import_helper.py             56     43    23%
bin/packages/Item.py                     401    298    26%
bin/packages/Paste.py                    195    146    25%
bin/packages/Pgp.py                       17      5    71%
bin/packages/Tag.py                      483    396    18%
bin/packages/Term.py                     392    325    17%
bin/packages/User.py                      40     24    40%
bin/packages/__init__.py                   0      0   100%
bin/packages/git_status.py               110     95    14%
bin/packages/lib_refine.py                94     94     0%
bin/packages/lib_words.py                 51     51     0%
bin/template.py                           18     18     0%
bin/tor_fetcher.py                        46     46     0%
bin/update-background.py                  44     44     0%
----------------------------------------------------------
TOTAL                                  13359  11447    14%
----------------------------------------------------------------------
Ran 3 tests in 0.002s

FAILED (errors=3)

Thank you

[lobojc]

And I have had to comment the next lines 126 to 130 of installing_deps.sh file to run the script successfully: git fetch failed -> [email protected]: Permission denied (publickey)

git fetch --depth=500 --tags --prune
if [ ! -z "$TRAVIS" ]; then
    echo "Travis detected"
    git fetch --unshallow
fi

[lobojc]

Other error details in the installating_steps.sh step:

Successfully built pymispgalaxies
Installing collected packages: jsonschema, pymispgalaxies
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 4.9.1
    Uninstalling jsonschema-4.9.1:
      Successfully uninstalled jsonschema-4.9.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pymisp 2.4.160.1 requires jsonschema<5.0.0,>=4.9.1, but you have jsonschema 3.2.0 which is incompatible.
Successfully installed jsonschema-3.2.0 pymispgalaxies-0.2

And later ->

Installing collected packages: jsonschema
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 3.2.0
    Uninstalling jsonschema-3.2.0:
      Successfully uninstalled jsonschema-3.2.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pymispgalaxies 0.2 requires jsonschema<4.0,>=3.2, but you have jsonschema 4.9.1 which is incompatible.
Successfully installed jsonschema-4.9.1
Requirement already satisfied: thehive4py in /home/ubuntu/ailframework/AILENV/lib/python3.9/site-packages (1.8.1)
Requirement already satisfied: python-magic in /home/ubuntu/ailframework/AILENV/lib/python3.9/site-packages (from thehive4py) (0.4.27)

It looks that some package require one jsonschema version and another package requires a different version

[lobojc]

FYI, the same results using python 3.8 and python 3.9 with Ubuntu 20.04 OS