Azure PostgreSQL backup failing intermittently

We are backing up an Azure PostgreSQL instance we have and are experiencing intermittent failures recently.

Backups were fine until Jan 28th and after that we have only 25% chance of it working.

Error log is:

14/02/2022 7:00:12 PM [Info] Backing up XXXXX.postgres.database.azure databases with pg_dump.exe: pg_dump (PostgreSQL) 11.15
14/02/2022 7:00:04 PM [Info] Starting scheduled job Backup Job - XXXXX - Full. Server XXXXX. System account SYSTEM@XXXXXX. App v.12.6.14.
14/02/2022 7:00:04 PM [Info] Connecting to PostgreSQL Server: XXXXXX.postgres.database.azure.
14/02/2022 7:00:10 PM [Info] Connected to PostgreSQL 11.12, compiled by Visual C++ build 1800, 64-bit.
14/02/2022 7:00:11 PM [Info] The temporary folder E:\SQLBackupAndFTP\Pranas\SBF\XXXXXX\backup has 830.546GB free space.
14/02/2022 7:00:12 PM [Info] Backing up XXXXXX with Full backup type.

14/02/2022 7:07:09 PM [Error] Failed to backup “XXXXX” database with “Full” backup type: C:\Program Files\PostgreSQL\11\pgsql\bin\pg_dump.exe process has finished with “1” code. The error message: “pg_dump: Dumping the contents of table “XXXXXXX” failed: PQgetCopyData() failed.
pg_dump: Error message from server: SSL error: decryption failed or bad record mac
pg_dump: The command was: COPY XXXXXX (id, “timestamp”, value, XXXXXX) TO stdout;”.

|14/02/2022 7:07:09 PM|[Info]|Cleaning up backups of Backup Job - XXXXX job.|
|14/02/2022 7:07:10 PM|[Info]|Deleting temporary files.|
|14/02/2022 7:07:10 PM|[Error]|Job Backup Job - XXXXX finished with 1 errors.|

Originally it was using the supplied pgdump version 13.4, I tried changing to latest version 11.15 as azure is on 11.12 but that did not resolve it.

If you can help to resolve this please.

Hi Rod_Whitcombe,

Could you please let us know if you are able to perform a backup manually, via pg_dump tool?

Thank you and sorry for the inconvenience.

Hi Alexander

We have found the issue, our firewall IPS function was breaking it. It took a while to realise because the firewall IPS logs didnt record anything so looks like it wasnt blocking the traffic, more like a bug. We have made an exception and it has worked ok over the weekend.

Kind Regards

Hi Rod_Whitcombe,

Good news, thank you for your reply.

Please feel free to contact us on any issues or concerns.