I solved a problem today that’s been plaguing me for a two weeks now. Exchange transaction logs are a record of every single operation that changes the state of any data in the database. Adding a new item, deleting an old item, modifying an existing item - all of these are recorded in a transaction log at the same time they’re applied to the database itself. When an Exchange database is mounted, the transaction logs are reprocessed to make sure there aren’t any discrepancies or database errors. If there are, the transaction logs are used to rebuild the database state. This same process applied to the DAG as well. The entire purpose of transaction logs in Exchange is to provide information on the transactions that occurred since the last time you ran a complete backup of your Exchange environment.
If you’re using Exchange-aware backup software, Exchange will ‘know’ when a full backup occurs and then it’ll delete all transaction logs prior to that point. They’re no longer needed for a database rebuild because there’s a backup.
The problem I ran into is that the transaction logs were not being purged despite a successful backup (by our Exchange-aware software) and filled up the volume, which caused a database dismount… which led to some unhappy employees. Fortunately, it was just 1 of our 6 databases and I was able to fix it within 15 minutes, so not many people even noticed an issue. I wrote about workarounds and solutions for that in Exchange DAGs and Cluster Ownership. As I mentioned in that post, I believed that the root cause was that the Owner of our DAG cluster objects wasn’t the proper host. However, this wasn’t the case because the transaction logs continued to grow!
The Transaction Logs for Journaling are causing the problem. Despite no old logs (they’re all timestamped AFTER our daily backup occurs, which tells me that the backup is working properly and Exchange is aware of it), they’re consuming 10+ GB more disk space each passing day. Yesterday morning they occupied 60GB, and 24 hours later they’re already consuming 84GB.
I checked the Queues and found dozens of messages in the Journaling queue:
[PS] C:\Windows\system32>Get-QueueDigest -Dag Exchange2016 | ft -a
GroupByValue MessageCount DeferredMessageCount LockedMessageCount StaleMessageCount Details
------------ ------------ -------------------- ------------------ ----------------- -------
journaling 85 85 0 0 {Exchange06\4, EXCHANGE04\4, Exchange05\5}
[PS] C:\Windows\system32>get-queue -Identity Exchange04\4
Identity DeliveryType Status MessageCount Velocity RiskLevel OutboundIPPool NextHopDomain
-------- ------------ ------ ------------ -------- --------- -------------- -------------
EXCHANGE04\4 SmtpDeliveryToMailbox Ready 29 0 Normal 0 journaling
I took a look at one of these messages via. the Exchange Toolbox\Queue Viewer GUI. This message has been stuck in the Journaling queue for almost 30 days (since August 13/21):
Identity: EXCHANGE04\4\75801877807477
Subject: Fw:'RE: 137396- Ray & Rachel Application'.
Internet Message ID: <59d5803a-273b-4123-a41f-dfb67fc6cf1e@journal.report.generator>
From Address: <>
Status: Retry
Size (KB): 11170
Message Source Name: Journaling
Source IP: 255.255.255.255
SCL: 0
Date Received: 8/13/2021 10:48:12 AM
Expiration Time:
Last Error: 400 4.4.7 The server responded with: 554 5.2.0 STOREDRV.Deliver.Exception:MessageSubmissionExceededException.MapiExceptionMaxSubmissionExceeded; Failed to process message due to a permanent exception with message Cannot save changes made to an item to store. 16.55847:09020000, 17.43559:0000000090000000000000000100000000000000, 20.52176:010F04890E00000000000000, 20.50032:010F04897E17103100000000, 0.35180:010F0489, 255.23226:00000000, 255.27962:0E000000, 255.31418:21000000, 16.55847:E3000000, 17.43559:0000000070010000000000000000000000000000, 20.52176:010F04890E00001080030400, 20.50032:010F04897E17F01F010F0489, 0.35180:2C000000, 255.23226:2C000000, 255.27962:0A000000, 255.27962:0C000000, 255.17082:DA040000, 0.18273:00000000, 4.21921:DA040000, 255.27962:FA000000, 255.1494:010F0489, 5.59176:0000A0004C696D69746174696F6E001005000780, 5.34600:A418B00043757272656E7453697A65000F010480, 4.42792:DA040000, 7.40748:000000000000000031343163, 7.57132:000000000000000005000780, 1.63016:0C000000, 4.39640:DA040000, 8.45434:7EFF3D197915D443BAF1677741440BB605000780, 5.10786:0000000031352E30312E323330382E3031343A45786368616E676530353A36623134316338652D663135382D346465612D616564652D3837656131656238336534310080, 255.1750:2C000000, 255.31418:2C000000, 0.21457:010F0489, 4.19665:DA040000, 0.37632:DA040000, 4.37888:DA040000 [Stage: CreateMessage]. The failure was replaced by a retry response because the message was marked for retry if rejected.
Queue ID: EXCHANGE04\4
Recipients: Tier3M.Journaling@Company.com;3;3;[{LED=400 4.4.7 The server responded with: 554 5.2.0 STOREDRV.Deliver.Exception:MessageSubmissionExceededException.MapiExceptionMaxSubmissionExceeded; Failed to process message due to a permanent exception with message Cannot save changes made to an item to store. 16.55847:09020000, 17.43559:0000000090000000000000000100000000000000, 20.52176:010F04890E00000000000000, 20.50032:010F04897E17103100000000, 0.35180:010F0489, 255.23226:00000000, 255.27962:0E000000, 255.31418:21000000, 16.55847:E3000000, 17.43559:0000000070010000000000000000000000000000, 20.52176:010F04890E00001080030400, 20.50032:010F04897E17F01F010F0489, 0.35180:2C000000, 255.23226:2C000000, 255.27962:0A000000, 255.27962:0C000000, 255.17082:DA040000, 0.18273:00000000, 4.21921:DA040000, 255.27962:FA000000, 255.1494:010F0489, 5.59176:0000A0004C696D69746174696F6E001005000780, 5.34600:A418B00043757272656E7453697A65000F010480, 4.42792:DA040000, 7.40748:000000000000000031343163, 7.57132:000000000000000005000780, 1.63016:0C000000, 4.39640:DA040000, 8.45434:7EFF3D197915D443BAF1677741440BB605000780, 5.10786:0000000031352E30312E323330382E3031343A45786368616E676530353A36623134316338652D663135382D346465612D616564652D3837656131656238336534310080, 255.1750:2C000000, 255.31418:2C000000, 0.21457:010F0489, 4.19665:DA040000, 0.37632:DA040000, 4.37888:DA040000 [Stage: CreateMessage]. The failure was replaced by a retry response because the message was marked for retry if rejected.};{MSG=};{FQDN=};{IP=};{LRT=}];0;CN=Journaling,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Company,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=corp,DC=Company,DC=com;0
The error is MessageSubmissionExceededException, which made me suspect a size limit problem. Indeed, all 28 messages in the Journaling queue have a size > 10MB. Our standard maximum message size is 10GB across the organization (to discourage people from using email as a file transfer/file storage mechanism), but last month an exception was granted for one of our customer service mailboxes. That mailbox has a maximum message size of 30MB.
I realized that this current problem is probably stemming from the changes made several weeks ago to allow large messages to this mailbox - they arrive there, but the Journaling process cannot handle them because the Journal mailbox limit is still 10MB. The Journaling process continually tries to add them to the Journaling mailbox which causes the retry process to generate a huge volume of transaction logs. I checked the mailbox message limits to be sure the sizes matched my hypothesis:
[PS] C:\Windows\system32>get-mailbox contact | select name,*size*
Name MaxSendSize MaxReceiveSize
---- ----------- --------------
contact 30 MB (31,457,280 bytes) 30 MB (31,457,280 bytes)
[PS] C:\Windows\system32>get-mailbox Tier* | select name,*size*
Name MaxSendSize MaxReceiveSize
---- ----------- --------------
Tier3A Journaling 10 MB (10,485,760 bytes) 10 MB (10,485,760 bytes)
Tier3B Journaling 10 MB (10,485,760 bytes) 10 MB (10,485,760 bytes)
Tier3C Journaling 10 MB (10,485,760 bytes) 10 MB (10,485,760 bytes)
Tier3D Journaling 10 MB (10,485,760 bytes) 10 MB (10,485,760 bytes)
Tier3L Journaling 10 MB (10,485,760 bytes) 10 MB (10,485,760 bytes)
Tier3M Journaling 10 MB (10,485,760 bytes) 10 MB (10,485,760 bytes)
I increased the max message size that Journaling mailboxes can handle:
[PS] C:\Windows\system32>get-mailbox Tier* | Set-mailbox -MaxReceiveSize 30MB -MaxSendSize 30MB
This flushed the 28 queued messages within 15 minutes.
I then ran a pair of consecutive diskshadow backups on Exchange05, which is the server that has mounted the Journaling database. This cleared out the logs on disk within 30 minutes. Instead of taking 84GB, the logs now consume a much-more-reasonable 2GB.