Downgrade python, update grafana and elasticsearch to fix the pie charts

[surfen]

Hello,

Recently I needed a way to visualiise my DMARC reports and wanted to use Docker. Thank you for sharing your repository, it helped me a lot.

Unfortunately, I encountered several issues when trying to use dmarc-visualizer, including #30, #31, #32

Here's the summary of those issues and proposed solution included in my PR:

• Python 3.12 produces ERROR: Could not build wheels for aiohttp, had to downgrade to 3.11
• New dashboard template (Grafana-DMARC_Reports.json) updated in Updated Grafana query template to support Elasticsearch 8.0 and higher domainaware/parsedmarc#366 did not work anymore, I had to upgrade elasticsearch and grafana and also apply workaround mentioned in Clean OS, clean docker image #41. The root cause is a depracated pie chart plugin (https://github.com/grafana/piechart-panel) which in the long run should be replaced with pie chart included in Grafana.

Please follow steps below to prevent possible issues with old files, missing files or docker cache

1. remove any output files
2. make sure there are DMARC report input files in /files/ folder (you may need to create it)
3. docker-compose build --pull --no-cache
4. docker-compose up
5. Observe the logs and wait for the containers to spin up and process the inputs. Parsedmarc container will log errors when trying to connect to elasticsearch, these errors stop after a while. After all files are processed no more lines will be added to the log and you may proceed to next step.
6. Navigate to localhost:3000 (or refresh the browser if you had the tab open)

Closes #31
Closes #32
Closes #41

[wakko4]

I tried Sufren suggested changes and is working again.

[wakko4]

Update: After working with it 24-48 noticed the filtering (clicking on any field thru the visual tables) doesn't work as expected...like IP, country, etc.. It adds the filter on top but data doesn't filter down to that. Filtering by domain works perfectly.

the other thing doesn't seem to work for me is the reverse DNS lookup. Reverse DNS table just shows only IP's. I added nameservers under general config but still not resolving IP's (added Cloud fare thou is default and G but still). Nice to have but not a show stopper.

Other than that @surfen did an awesome job

[surfen]

@wakko4 Well spoted :) Indeed the "Ad hoc filter" feature does not work in grafana/grafana:latest (i.e. 10.1.5)

Apparently there have been some recent changes which caused the bug, and the issue is expected to be resolved in 10.2
See this issue for more details
grafana/grafana#76183

You may want to switch to some older version for now, e.g. in grafana/grafana:9.5.13 ad hoc filtering on tables works properly.
It seems that in 9.5.13 the Dashboard can only be accessed after logging into Grafana using default username/password (admin:admin).

I was not able to reproduce issue with reverse DNS lookup. I can see domain name for my servers under different tables with "Reverse DNS" column.

[wakko4]

@surfen thanks for the info about downgrading grafana for the filters, let me test that. DNS issue, I will join that Ubnutu VM to domain, maybe something in domain is blocking a not joined pc in AD.

Do you have it reading from mailbox? (with MS Graph) I set everything as instructions but don't see anything in archive yet(default folder to move). I did create a python script to connect to sharemailbox using MS Graph and download all attachments to the 'files' folder in the dmarc-visualizer folder(input file). Can shared it later if anyone need it.

another thing was exploring setting smtp in grafana for alert report but says to create grafana.ini,

• set it up inside grafana folder with smtp setting - didn't work
• moved to root folder - didn't work
• I was able to change smtp host inside the yaml docker file under environment:

grafana:
build: ./grafana/
ports:
- 3000:3000
environment:
GF_INSTALL_PLUGINS: grafana-piechart-panel,grafana-worldmap-panel
GF_AUTH_ANONYMOUS_ENABLED: 'true'
GF_SMTP_ENABLED = 'true'
GF_SMTP_HOST = mysmtpserver.com:25

[wakko4]

@surfen thanks for the info about downgrading grafana for the filters, let me test that. DNS issue, I will join that Ubnutu VM to domain, maybe something in domain is blocking a not joined pc in AD.

Do you have it reading from mailbox? (with MS Graph) I set everything as instructions but don't see anything in archive yet(default folder to move). I did create a python script to connect to sharemailbox using MS Graph and download all attachments to the 'files' folder in the dmarc-visualizer folder(input file). Can shared it later if anyone need it.

another thing was exploring setting smtp in grafana for alert report but says to create grafana.ini,

• set it up inside grafana folder with smtp setting - didn't work
• moved to root folder - didn't work
• I was able to change smtp host inside the yaml docker file under environment:

grafana: build: ./grafana/ ports: - 3000:3000 environment: GF_INSTALL_PLUGINS: grafana-piechart-panel,grafana-worldmap-panel GF_AUTH_ANONYMOUS_ENABLED: 'true' GF_SMTP_ENABLED = 'true' GF_SMTP_HOST = mysmtpserver.com:25

I was able to read from a grafana ini file for smtp setting or to modify other grafana settings with this answer:

grafana.ini with docker

so main yaml file neds to add volumes line like:

volumes:
- ./grafana/custom.ini:/etc/grafana/grafana.ini

[surfen]

@wakko4
Yes, I configured it to read from e-mail via IMAP based on parsedmarc instructions here.

I simply edited parsedmarc\parsedmarc.ini file which already exists in dmarc-visualiser repository. I didn't use any additional python scripts. Sadly, I had to restart for it to pick up new e-mail(s).

It created automatically three folders on the mailbox ("Aggregate"/"Forensic"/"Invalid") under "Archive".
I see that there is separate config file section [msgraph] which you might find interesting.

[wakko4]

@wakko4 Yes, I configured it to read from e-mail via IMAP based on parsedmarc instructions here.

@surfen , thanks , I did read those top/bottom few times. Issue is co have MFA in M365, so IMAP dont work I have all the necessary permission on MS Graph as required. Actually my code use that appid, tenant, sec etc. so is not permission. I list emails and move to other folder, so has the Write.

btw, VM froze had to do full restart. Now when I run docker-compose up . does all the load but at end keeps in loop for 10 mins already. token not found. By chance you know what could be that? if image and guest PC (VM) restart too. I tried build again. I can see grafana on 3000. but visuals dont load 😞

[surfen]

@wakko4 "token not found" suggests authentication issue. Try logging into grafana. If you did not modify the username/password then the defaults are admin/admin.

[RousselJulien]

@wakko4 "token not found" suggests authentication issue. Try logging into grafana. If you did not modify the username/password then the defaults are admin/admin.

Hi! I've been having the same issue, at first I thought it was elastisearch setting up but it does not seem to work. Is there any fix for this issue?