From f0fc08371fdd95991231d154e066ed0885bc7050 Mon Sep 17 00:00:00 2001 From: Eric Eisenhart Date: Tue, 6 Feb 2024 15:40:32 -0800 Subject: [PATCH] Update docs based on feedback about what's confusing from #12 --- README.md | 64 ++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index f7d417c..d1d8445 100644 --- a/README.md +++ b/README.md @@ -40,8 +40,8 @@ If there's interest, could update Zabbix for "delegated", "escalated", "status_update_published". I'm not sure how to even cause some of those, and the others didn't seem important to have show up in Zabbix. -The Priority/severity mappings are currently hard-coded, but if there's -interest in making those configurable it shouldn't be hard to do. +The Priority/severity mappings are configurable, but default to a config +that should work well in a stock PagerDuty setup. ### Requirements - Zabbix 6.4+ (might work with older, I'm testing on 6.4) @@ -63,12 +63,20 @@ servers are purely internal/VPN-only. 1. Install a web server. 2. Configure web server to be able to run CGIs. -3. Either `git clone` this yum repo into someplace you can run CGIs from +3. Open firewall holes so that (at least) PagerDuty's WebHook IPs can hit + the http or https port that you're using. https://developer.pagerduty.com/docs/9a349b09b87b7-webhook-i-ps +4. If you're using https, make sure you have a valid certificate. +5. Either `git clone` this yum repo into someplace you can run CGIs from _or_ copy pd2zabbix.cgi to someplace you can run CGIs from. ```bash git clone https://github.com/sonic-com/pagerduty2zabbix.git ``` -4. Install perl and the modules needed by pd2zabbix.cgi: + * On a RHEL/Rocky/Alma 9 server, this should work: + ```bash + cd /var/www/cgi-bin + git clone https://github.com/sonic-com/pagerduty2zabbix.git + ``` +6. Install perl and the modules needed by pd2zabbix.cgi: On RHEL/CentOS/Rocky/Alma/Fedora, this probably looks like: ```bash @@ -88,13 +96,13 @@ servers are purely internal/VPN-only. cd pagerduty2zabbix cpanm --installdeps . # This should read cpanfile ``` -5. Verify you have appropriate perl modules installed with `perl -c pd2zabbix.cgi` -6. Verify your CGI config is correct with web browser or `curl` on the URL for +7. Verify you have appropriate perl modules installed with `perl -c pd2zabbix.cgi` +8. Verify your CGI config is correct with web browser or `curl` on the URL for pd2zabbix.cgi. If it's working right, it should give an error that includes: ``` No json_payload from webhook POSTDATA ``` -7. Configure Zabbix to send alerts to PagerDuty with the Zabbix WebHook included with recent Zabbix versions. +9. Configure Zabbix to send alerts to PagerDuty with the Zabbix WebHook included with recent Zabbix versions. If you've updated Zabbix, this may need to be updated to a version of the script that sets pagerduty "dedup_key" to zabbix "eventid". @@ -102,9 +110,12 @@ servers are purely internal/VPN-only. I recommend setting the `token` in the `Media type` to `{ALERT.SENDTO}` and putting your PagerDuty API token into "Send to" of the user's media configuration. (so you have the easy option of additional PD integrations for different teams, etc) -6. Copy pagerduty2zabbix.conf.example to ./pagerduty2zabbix.conf or /etc/pagerduty2zabbix.conf + If you don't have any plans to have multiple groups or to route Zabbix + alerts to different PagerDuty integrations/services/automation, don't + worry about this. +10. Copy pagerduty2zabbix.conf.example to ./pagerduty2zabbix.conf or /etc/pagerduty2zabbix.conf Make sure not accessible to public, since needs a secret (zabbix API key). -7. Edit pagerduty2zabbix.conf: +11. Edit pagerduty2zabbix.conf: - Get an API token from PagerDuty that can update the relevant PagerDuty events and set `pdtoken` to that. (profile pic > User Settings > Create API User Token) - Make a random string and set `pdauthtoken` to that. @@ -116,9 +127,25 @@ servers are purely internal/VPN-only. - If you want multiple retries for each zabbix URL, set `zabbixretries` > 1. - If you don't want PD event urls as a comment on new zabbix events, set `triggeredupdate=0`. - If you don't want clicking "resolve" in PD to close Zabbix events, set `resolvedupdate=0`. -8. In PagerDuty, go to the service Zabbix is sending events to, and: +12. Optional Testing: + - You should be able to access the URL from a web-browser or with `curl`. + It will return a "Software error" error about "No json_payload from webhook POSTDATA" + when tested this way. + - You should _not_ have to bypass any certificate warnings. + - If everything is configured appropriately, you should be able to test a + ping with curl like this (change the authentication token to your pdauthtoken + value and the URL to the URL of your copy of pd2zabbix.cgi) + ``` + curl --header 'Authentication: changeme' \ + --json '{"event": {"id": "01CH754SM17TWPE2V2H4VPBRO7","event_type": "pagey.ping","resource_type": "pagey","occurred_at": "2021-12-08T22:58:53.510Z","agent": null,"client": null,"data": {"message": "Hello from your friend Pagey!","type": "ping"}}}' + https://zabbix.example.com/cgi-bin/pagerduty2zabbix/pd2zabbix.cgi + ``` +13. In PagerDuty, go to the service Zabbix is sending events to, and: 1. Add a webhook - 2. For webhook URL, your URL + 2. For webhook URL, your fully-qualified URL for external access + - If you did a `git clone` into `/var/www/cgi-bin` on + RHEL/Rocky/Alma/CentOS, then this will probably look like + `https://zabbix.example.com/cgi-bin/pagerduty2zabbix/pd2zabbix.cgi` 3. Webhook Status: Active 4. Event Subscription: incident.acknowledged, incident.annotated, incident.priority_updated, incident.resolved, incident.triggered, @@ -127,8 +154,19 @@ servers are purely internal/VPN-only. - Name: `Authentication` - Value: The value from `pdauthtoken` earlier 6. Save. - 7. If you're able to watch error logs (where STDERR of CGIs go), do a "Send Test Event". - You should see `pagey.pong` in that log. + 7. Do a "Send Test Event". + - The access logs should show a POST to the CGI from one of the + PagerDuty WebHook IPs and a status code of 202, like: + ``` + 54.213.187.133 - - [06/Feb/2024:15:27:14 -0800] "POST /cgi-bin/pagerduty2zabbix/pd2zabbix.cgi HTTP/1.1" 202 87 + ``` + - If you're able to watch error logs (where STDERR of CGIs go), you should see `pagey.pong` in that + log. Something like: + ``` + [Tue Feb 06 15:27:14.129996 2024] [cgid:error] [pid 1887:tid 1945] [client 54.213.187.133:53658] [Tue Feb 6 15:27:14 2024] pd2zabbix.cgi: pagey.pong: /var/www/cgi-bin/pagerduty2zabbix/pd2zabbix.cgi + ``` + Exact formatting depends quite a bit on exact versions of httpd, + perl, and perl libraries. ## FAQ/Common Problems/Likely Problems: