mac maintenance

Mac Maintenance

These are the manual steps to maintain the macs. Until we fully automate setup and maintenance, we need to ssh into each mac to do maintenance.

The list of machines is here. Please keep the statuses updated on that page.

For each of these, consider quarantining the machine before performing maintenance. Don't forget the notarization poller worker on the non-dep workers!

SSHing into the macs

We now have duo set up on the macs themselves, rather than using a jumphost. You will need to be added to the duo allowlist to ssh to the mac signers to ssh to them.

If you had the jumphost enabled, remove all mentions of the rejh hosts from your .ssh/config. If you want to cache your duo setup for 10min per host, you can add this to your .ssh/config:

Host mac-v3-signing* tb-mac-v3-signing* dep-mac-v3-signing*
    ControlMaster auto
    ControlPath ~/.ssh/ssh-%C
    ControlPersist 10m

Updating python packages

There's three steps here:

1. Update the ronin_puppet repo with the changes.
2. Merge the changes into the production-mac-signing branch.

Both of these are documented below.

Step 1: Creating the PR

Dependency changes

Dependencies are hash pinned in Puppet (todo, add link). To update these, you must create a PR to the ronin_puppet repo as follows:

cd ronin_puppet/modules/signing_worker/files
# choose which worker type(s) you will update: ff-prod, tb-prod, dep
# run this script for each one that needs to be updated
bash update-requirements.sh dep > requirements.dep.txt

Then commit the changes and create a PR. .

Updating scriptworker or scriptworker-scripts

These repositories are revision pinned for each worker type in the scriptworker_config section of common.yaml. Update them and create a ronin-puppet PR to change them.

Updating the widevine repository

This repository is not revision pinned, but is also not updated automatically. To pick up changes to it, update that repository, and then delete the widevine directory on the signing workers, eg:

# ff-prod or tb-prod
rm -rf /builds/scriptworker/widevine
# dep
rm -rf /builds/{dep1,dep2,tb-dep}/widevine

When you run Puppet (see below), it will repopulate and reinstall widevine with the latest code.

Step 2: Merge ronin-puppet to production-mac-signing

First, be careful! If a change has landed that breaks puppet badly enough, it's possible to brick the entire pool with a merge. Inspect the changes. If unsure, it's safest to either cherry pick your changes, or reimage a machine to make sure that the master branch will puppetize into a working mac signer.

Second, if you merge changes into the ronin-puppet production-mac-signing branch, the mac signers will pick that up within a half hour or so.

Pausing periodic puppet

If you're working on a mac (did you quarantine it and its notarization poller?), you may want to disable periodic puppet. Either change /usr/local/bin/periodic-puppet.sh to exit out before puppetizing, or unload the com.mozilla.periodic.plist launch daemon. (Make sure puppet isn't running while you're doing this, otherwise it may undo what you just did.)

Wiping secrets

Wipe secrets if someone may gain access to the box: when moving the machine or retiring the machine.

When wiping secrets, you want to nuke ed25519_privkey (prod only), *.keychain, and widevine*.crt. Use rm -P to overwrite them multiple times.

Obsolete instructions

Running puppet

As root on each signing worker:

cd ~
git clone https://github.com/mozilla-platform-ops/ronin_puppet/ ronin_puppet
cd ronin_puppet
mkdir -p data/secrets
cp ~/vault.yaml data/secrets/vault.yaml
FQDN=$(hostname)
cat <<EOF > manifests/nodes/nodes.pp
node '${FQDN}' {
    include ::roles_profiles::roles::mac_v3_signing
}
EOF
cd /
puppet apply --modulepath=/var/root/ronin_puppet/modules/:/var/root/ronin_puppet/r10k_modules/ --hiera_config=/var/root/ronin_puppet/hiera.yaml --logdest=console /var/root/ronin_puppet/manifests

restarting the daemos

As root on the signing worker:

# for ff-prod or tb-prod
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.cltbld.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.cltbld.plist
launchctl unload /Library/LaunchDaemons/org.mozilla.notarization_poller.poller.plist && launchctl load /Library/LaunchDaemons/org.mozilla.notarization_poller.poller.plist
# for dep
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.depbld1.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.depbld1.plist
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.depbld2.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.depbld2.plist
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.tbbld.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.tbbld.plist