How increase write speed of raid1 mdadm?

linux performance mdadm raid1 performance-tuning

7,979

This variable you are referring to is related to RAID rebuild speed. If your array is in normal mode (not rebuilding), it should not affect the system performance.

I can notice slow HD (or too busy one) by looking at running processes. You can look for processes that you know are accessing disk with D state (uninterruptible sleep). If you got some, this may indicate a slow disk. Otherwise, your processes may not be doing IO at expected rate.

7,979

Marvin

Updated on September 18, 2022

Comments

Marvin over 1 year

CentOS, Xeon 1230, 16 Gb RAM, 2x1TB SSD in raid1 (mdadm).

Read from raid1 vary from 130 to 250 MB/s depends of files etc.

But write speed very slow: 15-20 MB/s (iotop, mc etc.)! You know the real write speed to SSD. CPU load not more than 23-25% (only one core). So situation isn't normal.

What should I do to reach hiest speed?

I try increase speed_limit_min:

# sysctl -w dev.raid.speed_limit_min=20000

but without result.

What should I do in order to increase write speed?

Update 1:

Raid details

# mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Mon Dec  7 14:23:49 2015
     Raid Level : raid1
     Array Size : 999547904 (953.24 GiB 1023.54 GB)
  Used Dev Size : 999547904 (953.24 GiB 1023.54 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Feb 12 13:26:17 2017
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : localhost:pv00
           UUID : 0fe1355e:edd9d104:e2db49ec:bc3bb9b6
         Events : 7865

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       33        1      active sync   /dev/sdc1

Parted output for sda:

# parted /dev/sda print
Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sda: 1024GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  538MB   537MB   primary  ext4         boot
 2      538MB   1024GB  1024GB  primary               raid

Parted output for sdc

# parted /dev/sdc print
Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sdc: 1024GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1024GB  1024GB  primary               raid

mdstat file contents:

# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdd1[1] sdb1[0]
      976630464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

md127 : active raid1 sdc1[1] sda2[0]
      999547904 blocks super 1.2 [2/2] [UU]
      bitmap: 3/8 pages [12KB], 65536KB chunk

It is impossible...

dmesg | grep -i ahci

return nothing... the end... Can it lead to my problem?

Update 1: Turn on (bios, initramfs). No affect. Copying 20 Gbytes: firstly 130-160 MB/s and liearly slowing down to 17 MB/s after minute or two.

Update 2:

# hdparm -I /dev/sda

/dev/sda:

ATA device, with non-removable media
        Model Number:       Samsung SSD 850 PRO 1TB
        Serial Number:      S252NXAG905316P
        Firmware Revision:  EXM02B6Q
        Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Used: unknown (minor revision code 0x0039)
        Supported: 9 8 7 6 5
        Likely used: 9
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors: 2000409264
        Logical  Sector size:                   512 bytes
        Physical Sector size:                   512 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:      976762 MBytes
        device size with M = 1000*1000:     1024209 MBytes (1024 GB)
        cache/buffer size  = unknown
        Nominal Media Rotation Rate: Solid State Device
Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 32
        Standby timer values: spec'd by Standard, no device specific minimum
        R/W multiple sector transfer: Max = 1   Current = 1
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
                SET_MAX security extension
           *    48-bit Address feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
           *    General Purpose Logging feature set
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
           *    64-bit World wide name
                Write-Read-Verify feature set
           *    WRITE_UNCORRECTABLE_EXT command
           *    {READ,WRITE}_DMA_EXT_GPL commands
           *    Segmented DOWNLOAD_MICROCODE
           *    Gen1 signaling speed (1.5Gb/s)
           *    Gen2 signaling speed (3.0Gb/s)
           *    Gen3 signaling speed (6.0Gb/s)
           *    Native Command Queueing (NCQ)
           *    Phy event counters
           *    unknown 76[15]
           *    DMA Setup Auto-Activate optimization
                Device-initiated interface power management
           *    Asynchronous notification (eg. media change)
           *    Software settings preservation
                unknown 78[8]
           *    SMART Command Transport (SCT) feature set
           *    SCT Write Same (AC2)
           *    SCT Error Recovery Control (AC3)
           *    SCT Features Control (AC4)
           *    SCT Data Tables (AC5)
           *    reserved 69[4]
           *    DOWNLOAD MICROCODE DMA command
           *    SET MAX SETPASSWORD/UNLOCK DMA commands
           *    WRITE BUFFER DMA command
           *    READ BUFFER DMA command
           *    Data Set Management TRIM supported (limit 8 blocks)
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
                frozen
        not     expired: security count
                supported: enhanced erase
        2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 500253884009b0f0
        NAA             : 5
        IEEE OUI        : 002538
        Unique ID       : 84009b0f0
Checksum: correct


# hdparm -I /dev/sdc

/dev/sdc:

ATA device, with non-removable media
        Model Number:       WDC WD1004FBYZ-01YCBB1
        Serial Number:      WD-WMC6M0D492LN
        Firmware Revision:  RR04
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Used: unknown (minor revision code 0x006d)
        Supported: 10 9 8 7 6 5
        Likely used: 10
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors: 1953525168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                   512 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:      953869 MBytes
        device size with M = 1000*1000:     1000204 MBytes (1000 GB)
        cache/buffer size  = unknown
        Form Factor: 3.5 inch
        Nominal Media Rotation Rate: 7200
Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 32
        Standby timer values: spec'd by Standard, with device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Advanced power management level: 96
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
           *    Advanced Power Management feature set
                Power-Up In Standby feature set
           *    SET_FEATURES required to spinup after power up
           *    48-bit Address feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
           *    General Purpose Logging feature set
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
           *    64-bit World wide name
           *    IDLE_IMMEDIATE with UNLOAD
           *    WRITE_UNCORRECTABLE_EXT command
           *    {READ,WRITE}_DMA_EXT_GPL commands
           *    Segmented DOWNLOAD_MICROCODE
                unknown 119[6]
                unknown 119[7]
                unknown 119[8]
                unknown 119[9]
           *    Gen1 signaling speed (1.5Gb/s)
           *    Gen2 signaling speed (3.0Gb/s)
           *    Gen3 signaling speed (6.0Gb/s)
           *    Native Command Queueing (NCQ)
           *    Phy event counters
           *    Idle-Unload when NCQ is active
           *    NCQ priority information
           *    unknown 76[14]
           *    unknown 76[15]
           *    DMA Setup Auto-Activate optimization
                Device-initiated interface power management
           *    Software settings preservation
                unknown 78[7]
           *    SMART Command Transport (SCT) feature set
           *    SCT Write Same (AC2)
           *    SCT Error Recovery Control (AC3)
           *    SCT Features Control (AC4)
           *    SCT Data Tables (AC5)
           *    reserved 69[3]
           *    DOWNLOAD MICROCODE DMA command
           *    Long physical sector diagnostics
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
                frozen
        not     expired: security count
                supported: enhanced erase
        118min for SECURITY ERASE UNIT. 118min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee0aef2a46c
        NAA             : 5
        IEEE OUI        : 0014ee
        Unique ID       : 0aef2a46c
Checksum: correct

Update 3: smartctl

# smartctl -a /dev/sda

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   097   097   000    Old_age   Always       -       15006
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       34
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       18
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   067   054   000    Old_age   Always       -       33
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       12
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       22922238691

Update 4: Think I've found. Option "discard" in /etc/fstab. It increase write speed on testing server up to 70-80 MB/s, on target server I can't check it right now.

Thank you all, I think problem is solved.

Thomas about 7 years

How did you built your md device? Add the output of mdadm --detail /dev/<md-device>. Did you use partitions or the whole disk. In case of partition, are those partitions properly aligned? Add the output of parted /dev/<disks> print. Did you test each disk individually to check if both drives are OK and perform as expected?
shodanshok about 7 years

Please add the output of cat /proc/mdstat
Marvin about 7 years

ps: I impressed with auto commenting on command listing!
shodanshok about 7 years

You have two RAID array. Are both slow when writing? Can you show the output of hdparm -I /dev/sda (and all other devices)?
Spooler about 7 years

You've not said much about these SSDs themselves. Are they new? Have you looked at SMART status both when idle and when your data transfer slows down on both drives? This can be done with something like smartctl -a /dev/sda
Marvin about 7 years

SSD: 2xSamsung 850 Pro, new (3 month worked under slow testing).
Marvin about 7 years

hdparm very big to paste in comments. Add to post update 2.
Marvin about 7 years

Yes, slow write speed on both raid: from md127 to md2, from md2 to md127, from external esata ssd disk to md127,md2. That why when I see that use IDE but not IHCI, I had idea that it can be the source. But this time mode is IHCI.
Marvin about 7 years

The same result if run "dd if=/dev/zero bs=1M count=5000 | pv | dd of=./zero.dd" -> about 12-20 MB/s!!!

Marvin about 7 years

Thank you. I saw it and try it. Even more: sysctl -w dev.raid.speed_limit_min=20000. No result :( But I still didn't reboot because of working process (copying). May be I must reboot?
Marvin about 7 years

Sorry, dont notice your idea. Everything clear.