Working for systems integrator means a lot of experience with different environments. Every infrastructure is different even when composed of similar ingredients. Every implementation is different, even when you deploy similar solution multiple times.
Sometimes you can face really funny and horrible things at once.
Expect unexpected – Master Key (Palo Alto Networks)
Every firewall and Panorama management server has a default master key that encrypts all the private keys and passwords in the configuration to secure them (such as the private key used for SSL Forward Proxy Decryption). As a best practices, it’s recommended to change default master key under device settings. Master key has lifetime, with Pan-OS 10.x it is 18250 days, before Pan-OS 8.x it was up to 2 years. Expired key is a “little” disaster, firewall go to Maintenance Mode and only one way to recover is reset to factory default.
Story from the field, happend couple years ago.
Customer called me Sunday morning (I thought, ooh, interesting). He has cluster of PA-3020 (now EoL), he said that both firewalls are dead, every port was down. I asked him to connect via console and… both firewalls were in Maintenance Mode. Really interesting…
What happened… Somebody during deployment, configure master and forgot to say about that to Customer. Master key expired and firewalls end up in Maintenance Mode. We had backup file, but recover wasn’t easy. We cannot just restore configuration without knowing Master Key – firewall cannot decrypt things like passwords, private keys etc.
Palo Alto Networks firewalls has capability to restore configuration only partially (load partial config), I used that to recover only parts which were in clear text, like objects, policies etc.
Generally Customer was lucky. Three factories operation were dependent on PAN cluster, but they had technological break during weekend when interesting things happened:)