When Data Disks in a RAID Group can be Smaller than Other Data Disks in the RAID Group

Something I noticed recently that I didn’t know (not sure this knowledge is actually that useful), is that you can have smaller data disks in a RAID group with larger data disks, and the larger data disks are not right-sized down to the size of the smaller disk!

The rule is that you can have data disks in the RAID group smaller that the largest data disk, and not cause the larger data disks to be down-sized, as-long-as the parity disks are the size of the largest data disk. The smaller disk just get truncated.

Here are a few tests/examples:

TEST 1: Create an aggregate with 5 * 113.4MB SSD and usable size = 233.8MB

TEST 2: Add a 527.6MB SSD to the aggregate created in TEST 1, and - as expected - the disk would be downsized

TEST 3: Create an aggregate with 6 * 527.6MB SSD + 1 * 113.4MB SSD and usable size = 1.83GB (not 389.5MB as might have been expected)

TEST 4: Create an aggregate with 7 * 527.6MB SSD and add 1 * 113.4MB SSD

TEST 5: What happens when we try to replace the 113.4MB SSD with a 527.6MB SSD?

Test Results

TEST 1: Create an aggregate with 5 * 86.56MB SSD and usable size = 233.8MB

CLU01::> aggr create -aggregate TEST_SSD2 -disklist NET-1.37,NET-1.38,NET-1.39,NET-1.40,NET-1.41

Info: The layout for aggregate "TEST_SSD2" on node "CLU01-01" would be:

      First Plex

        RAID Group rg0, 5 disks (block checksum, raid_dp)

                                                            Usable Physical

          Position   Disk                      Type           Size     Size

          ---------- ------------------------- ---------- -------- --------

          dparity    NET-1.37                  SSD               -        -

          parity     NET-1.38                  SSD               -        -

          data       NET-1.39                  SSD         86.56MB  113.4MB

          data       NET-1.40                  SSD         86.56MB  113.4MB

          data       NET-1.41                  SSD         86.56MB  113.4MB

      Aggregate capacity available for volume use would be 233.8MB.

TEST 2: Add a 500MB SSD to the aggregate created in TEST 1, and - as expected - the disk would be downsized

CLU01::> aggr add-disk -aggregate TEST_SSD2 -disklist NET-1.8 -raidgroup rg0

Info: Disks would be added to aggregate "TEST_SSD2" on node "CLU01-01" in the following manner:

      First Plex

        RAID Group rg0, 6 disks (block checksum, raid_dp)

                                                            Usable Physical

          Position   Disk                      Type           Size     Size

          ---------- ------------------------- ---------- -------- --------

          data       NET-1.8                   SSD         86.56MB  527.6MB

      Aggregate capacity available for volume use would be increased by 77.93MB.

      WARNING: One or more disks to be added will be downsized.

TEST 3: Create an aggregate with 6 * 527.6MB SSD + 1 * 113.4MB SSD and usable size = 1.83GB (not 389.5MB as might have been expected)

CLU01::> aggr create -aggregate TEST_SSD -disklist NET-1.1,NET-1.46,NET-1.47,NET-1.48,NET-1.49,NET-1.50,NET-1.51

Info: The layout for aggregate "TEST_SSD" on node "CLU01-01" would be:

      First Plex

        RAID Group rg0, 7 disks (block checksum, raid_dp)

                                                            Usable Physical

          Position   Disk                      Type           Size     Size

          ---------- ------------------------- ---------- -------- --------

          dparity    NET-1.46                  SSD               -        -

          parity     NET-1.47                  SSD               -        -

          data       NET-1.1                   SSD         86.56MB  113.4MB

          data       NET-1.48                  SSD           500MB  527.6MB

          data       NET-1.49                  SSD           500MB  527.6MB

          data       NET-1.50                  SSD           500MB  527.6MB

          data       NET-1.51                  SSD           500MB  527.6MB

      Aggregate capacity available for volume use would be 1.83GB.

CLU01::> aggr show

Aggregate     Size Available Used% State   RAID Status

--------- -------- --------- ----- ------- ------------

TEST_SSD    1.83GB    1.83GB    0% online  raid_dp,normal

Image: The smaller data disk doesn’t affect the usable size of the larger data disks, it just truncates!

TEST 4: Create an aggregate with 7 * 527.6MB SSD and add 1 * 113.4MB SSD

CLU01::> aggr create -aggregate TESTSSD3 -disklist NET-1.6,NET-1.7,NET-1.52,NET-1.53,NET-1.54,NET-1.55,NET-1.56

Info: The layout for aggregate "TESTSSD3" on node "CLU01-01" would be:

      First Plex

        RAID Group rg0, 7 disks (block checksum, raid_dp)

                                                            Usable Physical

          Position   Disk                      Type           Size     Size

          ---------- ------------------------- ---------- -------- --------

          dparity    NET-1.6                   SSD               -        -

          parity     NET-1.7                   SSD               -        -

          data       NET-1.52                  SSD           500MB  527.6MB

          data       NET-1.53                  SSD           500MB  527.6MB

          data       NET-1.54                  SSD           500MB  527.6MB

          data       NET-1.55                  SSD           500MB  527.6MB

          data       NET-1.56                  SSD           500MB  527.6MB

      Aggregate capacity available for volume use would be 2.20GB.

CLU01::> aggr add-disk -aggr "TESTSSD3" -disklist NET-1.4 -raidgroup rg0

Info: Disks would be added to aggregate "TESTSSD3" on node "CLU01-01" in the following manner:

      First Plex

        RAID Group rg0, 8 disks (block checksum, raid_dp)

                                                            Usable Physical

          Position   Disk                      Type           Size     Size

          ---------- ------------------------- ---------- -------- --------

          data       NET-1.4                   SSD         86.56MB  113.4MB

      Aggregate capacity available for volume use would be increased by 77.93MB.

CLU01::> aggr show

Aggregate     Size Available Used% State   RAID Status

--------- -------- --------- ----- ------- ------------

TESTSSD3    2.27GB    2.27GB    0% online  raid_dp,normal

TEST 5: What happens when we try to replace the 113.4MB SSD with a 527.6MB SSD?

CLU01::> disk replace -disk NET-1.4 -replacement NET-1.51 -action start

Warning: Replacement 520.5MB disk will be downsized to 107.1MB. Continue? {y|n}: n