Saturday, 18 July 2015

Search-NcFileSystem & Search-NaFileSystem

Introduction

In the next two posts, I’ll post the PowerShell code for:

Search-NcFileSystem.ps1
Search-NaFileSystem.ps1

The first one (Nc) is for Clustered Data ONTAP. And the second (Na) is for 7-Mode. These are PowerShell tools for scanning a filesystem (volume) for particular file/files and/or folder/folders, or you can just get a list of all the folders and files on that volume.

They are straightforward to run and can be parameterized on the CLI as desired.

The code is adapted from the code in this post from the start of the year:


FAQ

Q: Can I scan RO volumes (the SnapMirror-ed DR) without breaking the mirror?
A: YES

Q: What impact does running this have on the system?
A: It will consume a significant amount of CPU and disk resources; hence it is recommended to run this on DR or test systems.

IMPORTANT: Please do not run these tools against production volumes. If you do then please at least keep an eye on CPU and disk utilization!

Q: Does this need the DR volume to be mounted (for cDOT)?
A: NO. The tool uses APIs so it’s more efficient than file scanning tools that need to use NFS/CIFS access protocols - this tool does not even need NFS or CIFS to be running.

Performance

All these tests were run on a VSIM (either cDOT or 7-Mode) with 2 vCPUs, 2.9GB memory, and VMDK’s on SSD.

Test 1: cDOT 8.2.3 2-Node Cluster - first run

Processed 1 volume with 10’411 files and 1767 folders
Time = 157 seconds

Test 2: cDOT 8.2.3 2-Node Cluster - second run

Processed 1 volume with 10’411 files and 1767 folders
Time = 84 seconds

Note: The reason why it was faster the second time is probably because more of the file/folder information data was in memory.

Test 3: 7-Mode 8.1.2P4 - first run

Processed 1 volume with 10’361 files and 1729 folders
Time = 728 seconds

Note: The APIs are improved and more efficient in cDOT, hence the longer running time on 7-Mode is no surprise!

Test 4: 7-Mode 8.1.2P4 - second run

Processed 1 volume with 10’411 files and 1767 folders
Time = 716 seconds

Note: Interesting that the second run wasn’t much quicker.

Performance Estimation

On the basis of very little performance analysis (apologies, simply don’t have the time to do a thorough analysis) and on the hardware used in this lab:

Search-NcFileSystem (cDOT) can process:

10’000 files in 100 seconds
1 Million files in less than 3 hours.

Search-NaFileSystem (7-Mode) can process:

10’000 files in 700 seconds
1 Million files in ~ 20 hours

2 comments:

  1. i ran a few test runs of your search-ncfilesystem script against a 4 node 2240 (8.3P2 - all SATA and it is only a snapvault destination):

    ##### SEARCH STATISTCS #####

    Processed 186859 files and 6492 folders.

    Started at Fri Jul 24 08:01:45 CDT 2015
    Finished at Fri Jul 24 08:27:32 CDT 2015

    ##### SEARCH STATISTCS #####

    Processed 6077492 files and 804694 folders.

    Started at Fri Jul 24 08:49:13 CDT 2015
    Finished at Sun Jul 26 02:22:24 CDT 2015

    thanks for posting all your NetApp scripts/content
    great stuff!
    mark

    ReplyDelete
    Replies
    1. Thank you very much for the reply Mark. It's great to hear you've used the tool, and your statistics are very interesting. Cheers, VC

      Delete