Windows Server 2008 R2 System State backup

Question

Running a member server 2008R2 server. Backups have been running fine for some time but recently ceased to run. We have been using CloudBerry Lab Enterprise Backup to store backups in cloud storage.

I started looking at CloudBerry logs and did not get much information.

---

 2015-06-25 08:18:54,230 [UI] [1] NOTICE - **********************************************************************************************************
 2015-06-25 08:18:54,232 [UI] [1] NOTICE - CloudBerry Backup Enterprise Edition Console started. Version: 3.9.6.31
 2015-06-30 09:02:00,595 [PL] [32] WARN  - Communication channel faulted. Will be recreated.
 2015-06-30 09:02:01,521 [PL] [48] WARN  - Communication channel faulted. Will be recreated.
 2015-06-25 08:18:54,559 [PL] [4] INFO  - Creating plan status monitor communication channel.
 2015-06-25 08:18:55,955 [PL] [1] INFO  - Refreshing plan list
 2015-06-25 08:18:56,432 [PL] [1] INFO  - Plan created: Plan name: Backup Bare Metal / System State on 10/31/2014 4:33:01 PM, plan id: 8ea94349-1d1d-4035-b9a1-e3e9c8831358
 2015-06-25 08:18:56,433 [PL] [1] INFO  - Plan created: Plan name: Backup plan on 11/12/2014 12:50:32 PM, plan id: 61e2bcae-caa6-41f7-852f-ab61bdd716b5
 2015-06-25 08:18:56,433 [PL] [1] INFO  - Plan created: Plan name: Backup plan on 11/17/2014 9:50:39 AM, plan id: 5ad1d6fd-7c83-451d-9639-edd5e6237a95
 2015-06-25 08:18:56,825 [PL] [4] INFO  - Database file accessed: c:\programdata\cloudberry backup enterprise edition\data\cbbackup.db
 2015-06-25 08:18:57,236 [PL] [4] INFO  - Repository version: 3.9.3.6, created by product version: 3.9.6.31, date: 10/31/2014 16:30:13
 2015-06-25 08:18:59,205 [Base] [1] INFO  - MemoryManager instance created
 2015-06-25 08:18:59,834 [UI] [10] INFO  - Start checking for a new version
 2015-06-25 08:18:59,834 [PL] [10] INFO  - VersionCheckWebServiceBased initialized
 2015-06-25 08:19:00,409 [UI] [10] INFO  - ForceCheckForUpdate=False
 2015-06-25 08:19:00,409 [UI] [10] INFO  - SilentMode=True
 2015-06-25 08:19:00,409 [UI] [10] INFO  - CurrentVersion=3.9.6.31
 2015-06-25 08:19:00,410 [UI] [10] INFO  - NewVersion=4.1.0.54
 2015-06-25 08:19:15,352 [PL] [4] INFO  - SQL query 'SELECT SUM(ver.size) as size FROM cloud_files as fl INNER JOIN cloud_file_versions as ver ON ver.file_id=fl.id AND fl.destination_id = ?'. Parameters: '1' takes in total: 00:00:18
 2015-06-30 08:55:58,183 [PL] [1] INFO  - Starting plan Backup Bare Metal / System State on 10/31/2014 4:33:01 PM(8ea94349-1d1d-4035-b9a1-e3e9c8831358)...
 2015-06-30 09:02:00,595 [PL] [32] WARN  - Communication channel faulted. Will be recreated.
 2015-06-30 09:02:01,210 [PL] [48] INFO  - Creating plan status monitor communication channel.
 2015-06-30 09:02:01,521 [PL] [48] WARN  - Communication channel faulted. Will be recreated.
 2015-06-30 09:02:31,223 [PL] [20] INFO  - Creating plan status monitor communication channel.
 2015-06-30 09:02:54,799 [PL] [1] INFO  - Starting plan Backup Bare Metal / System State on 10/31/2014 4:33:01 PM(8ea94349-1d1d-4035-b9a1-e3e9c8831358)...
 2015-06-30 09:08:24,294 [PL] [1] INFO  - Saving plan: Plan name: Backup Bare Metal / System State on 10/31/2014 4:33:01 PM, plan id: 8ea94349-1d1d-4035-b9a1-e3e9c8831358
 2015-06-30 09:08:24,394 [PL] [77] INFO  - Plan changed: Plan name: Backup Bare Metal / System State on 10/31/2014 4:33:01 PM, plan id: 8ea94349-1d1d-4035-b9a1-e3e9c8831358
 2015-06-30 09:08:27,302 [UI] [1] INFO  - Wizard enginesettings save time: 00:00:00.0857669
 2015-06-30 09:08:27,319 [PL] [1] INFO  - Starting plan Backup Bare Metal / System State on 10/31/2014 4:33:01 PM(8ea94349-1d1d-4035-b9a1-e3e9c8831358)...
 2015-06-30 09:15:59,058 [UI] [1] INFO  - Check Network shares step: Fill shares list: Selected shares:
 2015-06-30 09:15:59,063 [UI] [1] INFO  - \\<server name removed>\h. Read/Write: True
 2015-06-30 09:15:59,091 [UI] [1] INFO  - Check Network shares step: Shares list filled
 2015-06-30 09:16:29,368 [PL] [1] INFO  - Saving plan: Plan name: Backup Bare Metal / System State on 10/31/2014 4:33:01 PM, plan id: 8ea94349-1d1d-4035-b9a1-e3e9c8831358
 2015-06-30 09:16:29,376 [PL] [33] INFO  - Plan changed: Plan name: Backup Bare Metal / System State on 10/31/2014 4:33:01 PM, plan id: 8ea94349-1d1d-4035-b9a1-e3e9c8831358
 2015-06-30 09:16:31,958 [UI] [1] INFO  - Wizard enginesettings save time: 00:00:00.0053946
 2015-06-30 09:16:31,971 [PL] [1] INFO  - Starting plan Backup Bare Metal / System State on 10/31/2014 4:33:01 PM(8ea94349-1d1d-4035-b9a1-e3e9c8831358)...
 2015-06-30 09:33:55,837 [PL] [1] INFO  - Starting plan Backup Bare Metal / System State on 10/31/2014 4:33:01 PM(8ea94349-1d1d-4035-b9a1-e3e9c8831358)...

---

So I started looking into the Event Viewer application logs and noticed some errors from the Volume Shadow Copy Service. I found four VSS Errors with EventID 8194 corresponding to backup failures.

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="VSS" /> 
  <EventID Qualifiers="0">8194</EventID> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2015-06-23T23:03:36.000000000Z" /> 
  <EventRecordID>39938</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>COMPUTERNAME.DOMAINNAME.local</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>0x80070005, Access is denied.</Data> 
  <Data>Operation: Gathering Writer Data Context: Writer Class Id:     {5382579c-98df-47a7-ac6c-98a6d7106e09} Writer Name: TermServLicensing Writer     Instance ID: {c4179ee4-dcf0-4868-a9b1-5815067704df}</Data> 
  <Binary>2D20436F64653A20575254575254494330303030313236302D2043616C6C3A2057525457 5254494330303030313231342D205049443A202030303030333433362D205449443A202030303030 303439322D20434D443A2020433A5C57696E646F77735C73797374656D33325C737663686F737420 2D6B2054534C6963656E73696E672020202020202D20557365723A204E616D653A204E5420415554484F524954595C4E4554574F524B20534552564943452C205349443A532D312D352D3230</Binary    > 
  </EventData>
  </Event>

---

There are four errors, two each from the System Writer, and two from TermServiceLicensing. I checked out the writers with vssadmin list writers

C:\Windows\System32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {e714901a-2c74-4a21-a75d-2a972a17ea4d}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {a68be00b-b8fc-47c6-8318-67a9eded4081}
   State: [1] Stable
   Last error: No error

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {a84447a4-b489-49ad-8091-df30e5292191}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {74c93069-57c7-448f-a9b6-eb79ba2119bc}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {8adb4bc8-4dd8-4470-ad63-88c0ae1a0821}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {3ac7d9a9-2006-4a75-90d3-0796975c2738}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {ed1ac904-9dea-444b-abe6-dbe8a13977fa}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'TermServLicensing'
   Writer Id: {5382579c-98df-47a7-ac6c-98a6d7106e09}
   Writer Instance Id: {c4179ee4-dcf0-4868-a9b1-5815067704df}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {7cb6910c-e9af-4371-911d-f38a6eab0308}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {836b0e10-a5cb-4e1e-b9ef-3cd629e365c5}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {1f8f224f-2101-41db-b71a-8e130b0d320f}
   State: [1] Stable
   Last error: No error

I saw that some of them were hung in State: [ 5 ] Waiting on completion and found a batch file that re-registered the VSS components and restarted some services, which took care of everything but the 'IIS Config Writer' but that cleared after running another manually.

cd /d %windir%\system32
net stop vss
net stop swprv
net stop bits
net stop iisadmin
net stop solarwindsagent64
net stop MSSQL$MSSMLBIZ
net stop cryptosvc
regsvr32 /s ole32.dll
regsvr32 /s oleaut32.dll
regsvr32 /s vss_ps.dll
vssvc /register
regsvr32 /s /i swprv.dll
regsvr32 /s /i eventcls.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s vssui.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml4.dll
vssvc /register
net start cryptosvc
net start MSSQL$MSSMLBIZ
net start solarwindsagent64
net start iisadmin
net start bits
net start swprv
net start vss

I did a little digging and found some information about a registry key controlling callback access into VSS. I added REG_DWORD values underneath HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\VssAccessControl for several accounts.

I attempted to add domain accounts but received the following.

I followed the instructions and tried several times using the FQDN, NetBIOS domain name, NetBIOS domain name in capitals, UPN. Never could get the domain lookup to work, but I suspect that's due to the NetGetLocalGroup() call. Local group accounts do work ( I added the local Administrators for good measure ) but the articles that I found all specified creating a domain account.

The next step in the article was to check permissions on the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\Diag Key. I noticed that there were no permissions for the Nt Authority\LocalService and Nt Authority\NetworkService "identities." I immediately ran into the problem of not being able to add those using the GUI. I wasn't sure if it could be done using a command line utility, WMI, or if I had to do it using C++. Oddly enough, once I added permissions to the local Administrators group and started the Volume Shadow Copy service again, the permissions magically showed up. ~Great~

That seems to have resolved the warnings and errors on the startup of the Volume Shadow Copy service so I moved on to the next step. (NOTE: Exactly a week later we were right back to 0x80000005 Access Denied errors)

I thought I would cut down on some of the overhead by running a system state from the command line with wbadmin start systemstatebackup -backupTarget:\computername\temporraryshare.
Technet Wbadmin.exe Running the backup to an external hard drive connected to another server continually failed. Checking the Event Viewer underneath Microsoft\Windows\Backup\Operational yields:

Looking up information on Event ID 5 here: Windows Server Backup Events and looking up HRESULT information specific to same here: Windows Server Backup HRESULTs

Checked Local Group Policy to make sure nothing was set. Started looking at event details.

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" /> 
  <EventID>5</EventID> 
  <Version>2</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x4000000000000000</Keywords> 
  <TimeCreated SystemTime="2015-06-30T13:21:14.567755700Z" /> 
  <EventRecordID>497</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="11340" ThreadID="5280" /> 
  <Channel>Microsoft-Windows-Backup</Channel> 
  <Computer>COMPUTERNAME.DOMAINNAME.local</Computer> 
  <Security UserID="S-1-5-18" /> 
  </System>
- <EventData>
  <Data Name="BackupTemplateID">{E1E1D6B2-EEF1-45D7-A7B6-477F343B4D45}</Data> 
  <Data Name="HRESULT">2155347997</Data> 
  <Data Name="BackupState">12</Data> 
  <Data Name="BackupTarget">\\COMPUTERNAME\TEMPSHARE</Data> 
  <Data Name="NumOfVolumes">2</Data> 
  <Data Name="BackupTime">2015-06-30T13:16:43.380755700Z</Data> 
  <Data Name="HRESULT2">2155347997</Data> 
  <Data Name="VolumesInfo"><VolumeInfo><VolumeInfoItem Name="C:" OriginalAccessPath="C:" State="7" HResult="-2139619299" DetailedHResult="0" PreviousState="15" IsCritical="1" IsIncremental="0" BlockLevel="0" HasFiles="0" HasSystemState="1" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="2" DataTransferred="0" NumUnreadableBytes="0" TotalSize="0" TotalNoOfFiles="0" Flags="1574" BackupTypeDetermined="1" SSBTotalNoOfFiles="122303" SSBTotalSizeOnDisk="17221751022" /><VolumeInfoItem Name="E:" OriginalAccessPath="E:" State="15" HResult="-2139619228" DetailedHResult="0" PreviousState="0" IsCritical="1" IsIncremental="0" BlockLevel="0" HasFiles="0" HasSystemState="1" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="2" DataTransferred="0" NumUnreadableBytes="0" TotalSize="0" TotalNoOfFiles="0" Flags="548" BackupTypeDetermined="1" SSBTotalNoOfFiles="122067" SSBTotalSizeOnDisk="17018290228" /></VolumeInfo></Data> 
  <Data Name="DetailedHRESULT">2147942487</Data> 
  <Data Name="SourceSnapStartTime">2015-06-30T13:16:43.353755700Z</Data> 
  <Data Name="SourceSnapEndTime">2015-06-30T13:17:27.586755700Z</Data> 
  <Data Name="PrepareBackupStartTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  <Data Name="PrepareBackupEndTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  <Data Name="BackupWriteStartTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  <Data Name="BackupWriteEndTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  <Data Name="TargetSnapStartTime">1601-01-01T00:00:00.000000000Z</Data> 
  <Data Name="TargetSnapEndTime">1601-01-01T00:00:00.000000000Z</Data> 
  <Data Name="DVDFormatStartTime"><TimesList></TimesList></Data> 
  <Data Name="DVDFormatEndTime"><TimesList></TimesList></Data> 
  <Data Name="MediaVerifyStartTime"><TimesList></TimesList></Data> 
  <Data Name="MediaVerifyEndTime"><TimesList></TimesList></Data> 
  <Data Name="BackupPreviousState">8</Data> 
  <Data Name="ComponentStatus"><ComponentStatus></ComponentStatus></Data> 
  <Data Name="SSBEnumerateStartTime">2015-06-30T13:17:31.002755700Z</Data> 
  <Data Name="SSBEnumerateEndTime">2015-06-30T13:21:13.354755700Z</Data> 
  <Data Name="SSBVhdCreationStartTime">2015-06-30T13:21:13.354755700Z</Data> 
  <Data Name="SSBVhdCreationEndTime">1601-01-01T00:00:00.000000000Z</Data> 
  <Data Name="SSBBackupStartTime">1601-01-01T00:00:00.000000000Z</Data> 
  <Data Name="SSBBackupEndTime">1601-01-01T00:00:00.000000000Z</Data> 
  <Data Name="SystemStateBackup"><SystemState IsPresent="1" HResult="-2139619299" DetailedHResult="-2147024809" /></Data> 
  <Data Name="BMR">false</Data> 
  <Data Name="VssFullBackup">false</Data> 
  <Data Name="UserInputBMR">false</Data> 
  <Data Name="UserInputSSB">true</Data> 
  <Data Name="BackupSuccessLogPath">C:\Windows\Logs\WindowsServerBackup\Backup-30-06-2015_09-16-43.log</Data> 
  <Data Name="BackupFailureLogPath">C:\Windows\Logs\WindowsServerBackup\Backup_Error-30-06-2015_09-16-43.log</Data> 
  <Data Name="EnumerateBackupStartTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  <Data Name="EnumerateBackupEndTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  <Data Name="PruneBackupStartTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  <Data Name="PruneBackupEndTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data> 
  </EventData>
  </Event>

Both logfiles were completely empty 1K files.

Referencing MSDN HRESULT structure information: MSDN HRESULT MSDN HRESULT values NTSTATUS values Win32 Error Codes

Checking Detailed HRESULT properties and using the Windows Calculator to translate into HEX from DEC yields

HRESULT = 2155347997
hex = 8078001D

HRESULT = -2139619299
HEX = FFFFFFFF8078001D


HRESULT = -2139619228
HEX = FFFFFFFF80780064

which I couldn't locate in the referenced information. I thought it was weird that a built in didn't have regular facility code but instead looked like it was all flagged on, etc., but okay.

What's strange is that there's plenty of disk space ( 1.5 TB free ) and write permissions are okay because backup files are created in the folder ( .xml, etc. ) including the VHD, all files are enumerated and indexed prior to the error taking place. I can also successfully backup system state from that sever that the external drive is connected to locally, and I can run system state backup from the server I'm working on over the network to a UNC share on my desktop, but not one to the other. With the exception of a single file error logged in %SystemRoot%\Logs\WindowsServerBackup when backing up to the desktop share.

 Error in backup of C:\Program Files (x86)\Common Files\Adobe\ARM\1.0\ during write: Error [0x80070003] The system cannot find the path specified.

I traced WBADMIN.EXE and WBENGINE.EXE using process monitor and the exact moment that the backup fails there is a WriteFile call that returns an INVALID PARAMETER result.

Additionally strange is that the backup had been running to the 2nd volume on the drive and then suddenly it was flagged as having a portion of the system state ( IsCritical=1 ) present and was no longer a valid path to write. I can't disable Volume Shadow Copy on that volume as it's needed for CloudBerry Lab and that seems to be what's preventing the System State Backup from running to the local disk ( I know there is a registry key that can be set to force this KB Article Number(s): 944530, but it seems like there are way too many problems with doing that in this scenario ).

That being said, this is what I see from the disk properties:

But using vssadmin list shadows shows:

C:\Windows\System32>vssadmin list shadows
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Contents of shadow copy set ID: {bb50cbc9-9d0b-4bed-87f4-6bb4393d60d7}
   Contained 1 shadow copies at creation time: 3/24/2015 12:19:15 AM
      Shadow Copy ID: {820ea33d-2659-422f-a9ab-1cdeb2ec4b13}
         Original Volume: (E:)\\?\Volume{ab756977-74c0-11e0-85c2-b499ba011334}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy3
         Originating Machine: COMPUTERNAME
         Service Machine: COMPUTERNAME
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: DataVolumeRollback
         Attributes: Persistent, No auto release, No writers, Differential

Contents of shadow copy set ID: {e4c5af0d-7624-4506-b120-4f10bbcdef31}
   Contained 1 shadow copies at creation time: 3/25/2015 12:19:17 AM
      Shadow Copy ID: {dc45986e-7753-4345-a29a-bcf2d508ff24}
         Original Volume: (E:)\\?\Volume{ab756977-74c0-11e0-85c2-b499ba011334}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5

         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: DataVolumeRollback
         Attributes: Persistent, No auto release, No writers, Differential

   ... plus about 75 more of these...

Running the backup using wbadmin to the other server still gives this:

C:\Windows\System32>wbadmin start systemstatebackup -backupTarget:\\COMPUTERNAME\SHARENAME
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.

Starting to back up the system state [7/2/2015 3:54 PM]...
Retrieving volume information...
This will back up the system state from volume(s) Local Disk(C:),Data(E:) to \\COMPUTERNAME\SHARENAME
Do you want to start the backup operation?
[Y] Yes [N] No y

Creating a shadow copy of the volumes specified for backup...
Creating a shadow copy of the volumes specified for backup...
Creating a shadow copy of the volumes specified for backup...
Creating a shadow copy of the volumes specified for backup...
Windows Server Backup is updating the backup for deleted items.
This might take a few minutes.
Found (116) files.
Found (5634) files.
Found (10787) files.
Found (13859) files.
Found (18345) files.
Found (23126) files.
Found (27218) files.
Found (42094) files.
Found (48699) files.
Found (52586) files.
Found (56519) files.
Found (60455) files.
Found (67646) files.
Found (74117) files.
Found (80317) files.
Found (84052) files.
Found (91931) files.
Found (100599) files.
Found (108703) files.
Found (117027) files.
Found (122210) files.
Summary of the backup operation:
------------------

The backup of the system state failed [7/2/2015 3:58 PM].
Log of files successfully backed up:
C:\Windows\Logs\WindowsServerBackup\Backup-02-07-2015_15-54-35.log

Log of files for which backup failed:
C:\Windows\Logs\WindowsServerBackup\Backup_Error-02-07-2015_15-54-35.log

The operation ended before completion.
The parameter is incorrect.

But I can do the same thing to the share on my desktop successfully. It also seems that fixing these issues only lasts for a few days before the same errors crop back up. I found a specific hotfix for the 'parameter is incorrect' issue, but on installation I received a message that it was not applicable to my system. ( I believe it is 417386_intl_x64_zip ) KB Article Number(s): 2182466

I still get this when attempting to start a new backup to the same location previously attempted, making me think that the problem is somehow related to the CreateFile call.

- System 

  - Provider 

   [ Name]  Microsoft-Windows-Backup 
   [ Guid]  {1DB28F2E-8F80-4027-8C5A-A11F7F10F62D} 

   EventID 24 

   Version 0 

   Level 4 

   Task 0 

   Opcode 0 

   Keywords 0x4000000000000000 

  - TimeCreated 

   [ SystemTime]  2015-07-02T19:55:15.033279700Z 

   EventRecordID 527 

   Correlation 

  - Execution 

   [ ProcessID]  13804 
   [ ThreadID]  4868 

   Channel Microsoft-Windows-Backup 

   Computer COMPUTERNAME.DOMAINNAME.local 

  - Security 

   [ UserID]  S-1-5-18 


- EventData 

  VolumeGUID {6B432740-3984-11E0-8EA6-806E6F6E6963} 
  VolumeFriendlyName \\?\Volume{6b432740-3984-11e0-8ea6-806e6f6e6963}\ 
  VhdDeleteReason Backup VHD is corrupt 

So I'm running out of ideas for methods of attack. I can't seem to find very specific information on the HRESULTs. Sometimes I end up with an EventID of 5 and an EventID of 517. Occasionally I've seen other values for the HRESULT as well, such as the infamous 'Error enumeration files'. Below are the Process Monitor ( from SysInternals ) tabs for the WriteFile event.

Anyone have any thoughts on this? Help is greatly appreciated.

Solution

Seen such errors with various backup apps, usually it's a COM permission issue.

1. Start > Run > dcomcnfg
2. On the right-side of the newly-opened windows, expand the Component Services > Computers > My Computer nodes.
3. Right-click on My Computer and select Properties from the contextual menu.
4. In the newly-opened window, select the COM Security tab. look up for the Access Permissions options panel and click to the Edit Default button.
5. In the newly-opened window, add the SYSTEM and Network Service users (if not already there) and grant them the Local Access permission by activating the proper checkbox.
6. close everything and restart the machine.

Comments

[Stuart Smith]

May be similar to: serverfault.com/questions/514648/…

[yagmoth555]

Good detail for your post. +1 (but sorry can't help, never seen that problem)

[Vesper]

Isn't 200GB on E: just too much? Your backup may fail on the shadow copying of E: due to insufficient space. Also run memory diagnostics on your server, there might be a single address which does not hold data stored in it, therefore various weird errors might occur if that physical address is used for something.

[Stuart Smith]

You're correct, that's a lot of space being used. From what I can tell, Cloudberry uses the shadow copies as part of the backup process. Then it copies the files out to cloud storage. The result being that the disk space is not really used, just allocated ( I know that amounts to the same thing ). I can't figure out how to get around that limitation. It looks like no new shadow copies have been created since April, however, so I question whether or not I can just remove those and recover disk space. I'd welcome any input on that.

[bshacklett]

I hate to leave a "me too" comment, but I'm having a very similar issue. Have you made any progress on this? I'm using Cloudberry Enterprise Backup as well.

[Stuart Smith]

Still backing up to "local" removable hard drives and then up to cloud. So, no successful resolution yet. Having an additional issue that all PDF backups from the last year or so are corrupted...