Rice Enterprise Storage Certification

This document outlines the certification processes and result of those processes for the Rice Enterprise Storage.  Mosts tests were performed with a Macintosh laptop verifying access to CIFS mounts during specific tests.  On some occasions we also used a Windows XP laptop.  For NFS testing we used Linux and Macintosh.

Off-campus Storage Network

• User Access: Users will generally access their data via the CIFS protocol, authenticating with their NetID and Password.  Access controls are managed via Microsoft Active Directory.  NFS is available as well for servers or clients with access controls provided by the Enterprise LDAP.  iSCSI is available on a limited basis by special request to authenticated servers.  IT is planning on providing access via https and possibly WebDAV in the future.
  
• Storage characterization:  The disk storage has been designed for optimum scalability and performance.  In order to achieve this goal, a virtual file system is created called an EVS.  The EVS is an abstraction of a number of raid sets that are distributed across controllers, shelves, and disks.  The raid sets are made up of 10 drives each with 2 hot spare drives allocated from a total of 42 drives. By presenting a single EVS for a large amount of storage, the data can be striped over 80 spindles providing very fast and efficient read/write speeds.  Because the EVS is virtualized storage, it can be dynamically grown for efficient utilization of the disk space.  Additional storage can be added to any EVS without any interruption of services.  The trade off to this design is that the occurrence of 2 disks failing in any raid set can make the entire EVS unusable. This is considered a low probability event.  In this scenario, Enterprise Class data would be accessed on the replicated storage via a manual operation.
  
• NAS Server characteristics:  The presentation of the storage is provided  by the Network Attached Storage Server (NAS).  The architecture of the Bluearc Titan NAS is a modular design with an ASICs matrix for protocol access to the storage resources.  Application Specific Integrated Circuits (ASIC) perform much better than software protocol stacks due to the speed with which information can flow through the system and the speed of access to memory.  This results in very high scalability (greater than 50 Peta bytes with 4Gb/s fiber channel) behind a single  Titan.  Due to the way that the Titan is engineered, the server can be clustered to provide high availability in the event of a hardware failure as well a load sharing. This means that one NAS server can take over in the event of a failure or can be brought down for maintenance without effecting service AND the network load can be distributed between 2-4 NAS servers. 
• Performance test results ( Click on graphic for larger image. )

• iozone -K -r 32 -s 250M -i 0 -i 1 -b io.wks                
  32k records, 250MB file               
                 
                              cifs_win      cifs_lin        nfs_lin        iscsi_lin
  Writer Report       30416          2564            338298        943973
  Re-writer Repo     951762        2423            937149       1002608
  Reader Report      21609         2479            2103894      2080641
  Re-reader Report 1656913      1939482       2113660      2096298       
  
                               cifs_win      cifs_lin       nfs_lin         iscsi_lin
• Writing                  3802            320.5         42287.25      117996.625
  Re-writing             118970.25    302.875     117143.625   125326
  Reading                2701.125      309.875      262986.75    260080.125
  Re-reading            207114.125   242435.25  264207.5     262037.25

ISCSI testing (by Trey Rouse)  Load testing with MS SQL and  a 100 User license test.  All tests with 100 contiguous user connections over 1 ea GB Ethernet link shared with network access and ZERO errors.  Server was equipped with a TCP Offload Engine NIC.

• UPDATE INTENSIVE TESTS
  
• Maximum Transactions per second (TPS)  Sustained Peak Performance ~> 12K
• Maximum Bytes per second (BPS) Sustained Peak Performance ~> 2.2M

• Replication of Enterprise Class Storage
  • The storage mechanism for data identified as Enterprise Class which consists of user home directories and departmental shared folders is replicated to a set of lower cost disks on 4 hour intervals.  This is to minimize the interruption to work flow in the event of a file system failure.  The replication was tested by creating data on the primary data storage, replicating the data and testing for the existence of the newly created files on the replicated storage.  Replication is accomplished on a scheduled based defined every 4 hours.  This provides a worse case failure scenario with a maximum delta in data loss for enterprise storage of 4 hours.
• Backup/Restore
  • Snapshots:  Enterprise and Commodity class storage are protected from data loss every 4 hours by a process known as snapshots.  Every 4 hours, data that has changed is preserved by reserving the file table making any new changes written to new and existing files occur on unused disk space.  This provides a copy of work that has occurred within the last 4 hours.  Each day, the last snapshot of the day is preserved and all others are discarded.  A full weeks worth of daily snapshots, plus the current days 6 x 4 hours snapshots are available for data recovery by arc admins and the help desk.  The snapshots have been verified by testing the copies of files that have been preserved at all levels.
  •  NDMP backups and restorations:  The NAS Enterprise and Commodity data are incrementally backed up to the COPAN MAID (disk based Virtual Tape Library) every day via NDMP on a schedule defined in the Commvault backup software.  NDMP backups and restores of the data have been successfully completed and the data has been verified.
    
  • Tape:  Tape media is still required in order to provide removable media for longer term storage and disaster recovery (DR).  The Commvault software can create full backups from the incrementals stored on the Copan system and write them to removable tape media.  Tapes have been written, verified and restored from using the Commvault software.
• Failure Testing
  
  •  Cluster failures: The NAS server can be clustered to provide high availability.  This means that up to 4 NAS servers can be joined together and automatically assume the responsibility of a failed server.  In order to test this function, we have certified this process by taking one server off line to perform an OS upgrade while testing access to the file services.
              o Client response results were momentary disruption (90 seconds) of client access as the secondary server took over.
  • Storage Failures
              o Disk Failure:  Individual disk failures have been tested 8 times.  4 Sata and 4 Fiber Channel disks have failed and been replaced over the last 6 months.  When a disk fails, the NAS server alerts the mfg, and IT.  The server automatically utilizes one of the hot spares available to the system to replace the failed drive and operations replaces the failed disk with a shelf spare.  The failed drive is returned to the manufacturer when the shelf spare replacement arrives at Rice.
              o EVS Failure:  An EVS can fail if 2 drives fail within any RAID set.  This is considered an extremely unlikely occurrence, but we have had such an event occur during pre-deployment testing.  In this scenario, the Enterprise class storage must be manually redirected to the  replicated storage.  Upon redirection, clients will need to re-establish connectivity to their shares. Any Commodity class storage would have to be recovered from the Copan Systems.  Due to the failure we experienced, we have halved the probability of such a failure by reducing the number of drives per raid set from 20 to 10.  Reducing the number of disk drives in a raid set has a negative effect on the total amount of available storage, but increases the reliability of the subsystems.
    
  • Restoration from replication:  If an EVS fails, the restoration for Enterprise class data will be restored by reverse replication from the mirrored data maintained on a lower performance file system. Commodity class storage will be restored from the Copan or Tape systems  This scenario has been tested 3 times since the system has been installed.
  •  iSCSI test: See test results section . Tests will be need to be completed regarding expected results from nas fail over to specific servers.
  •  Ethernet Failure:  The Nas has 8 bonded Ethernet ports.  We have tested the failure of any single or multiple physical network connections by removing the connections from the network interfaces.  If a user's connection is being presented through a failed or disconnected, the user must reestablish the connection so that it can occur over a remaining valid connection.  Currently due to limitations of bonding and architecture a single network device failure can result in an access failure.
  •  Management Failure:  The management servers (SMU) can be rebooted without any effect to the storage or the NAS server.  We have a redundant management server in place that is on hot standby.
  • Fiber Chanel switch failure:  This was tested by powering down a switch and testing access to the storage via a client share.  The switch failure was reported successfully but access was not interrupted.
  • FC Controller Failure:  This was tested by removing a FC controller and testing access to the storage via a client share.  The failure was reported successfully but access was not interrupted.
    

• Security
  • Default Access Controls provided by NIS for NFS currently - This will move to LDAP by 7/06 
  • Default User Access controls provided via  MSAD for  CIFS
    • User folders:
      • User access inherited defaults:
        • Full Control = BlueArc Admins, User
          
        • Modify,Read & Execute, List, Read Write = ARC Admins
      • Shared folders inherited defaults:
        • Full Control = BlueArc Admins
        • List = High level Group
        • Modify,Read & Execute, List, Read Write = Specified Group Members
        • FUTURE = Web management over https via NFS mounted webdav service
          

• Administrative processes
  • Controls are currently in place via change management
  • Security is scheduled to review password policies for admin access.

• Certification check list and results
  • Replication  - Complete
  • Backup & Restore - Complete
  • Clustering - Complete
  • Storage Failure Recovery - Complete
  • Replication Recovery - Complete
  • iSCSI Testing - Completed with the exception of server failure testing
  • Network Failure Recovery - Complete
  • Management Failure Recovery - Complete
  • Fiber Chanel Failure Recovery - Complete
  • Administrative Management Procedures and Documentation - located on the Nas (DR needs to be on CD)
    

•  Things to consider
  • costs (initial software license costs and ongoing maintenance)
    
  • planning (what kind of backups? what are the retention cycles? Software documentation?)
    
  • an evolution cycle from the old backups to the new
•  IT has a team from Windows, Unix and Mac that can help you plan this migration.  See Omar regarding scheduling. 
  

• Enterprise - Replicated for critical systems data, snapshots, backed up to disk, backed up to tape, archiving available
• Commodity - Snapshots, backed up to disk, backed up to tape, archiving available - data is not replicated to redundant storage.
  
• Near line - This is a disk backup that is archived to tape and is not user accessible
• Backup - Data written to LTO3 tape ~ 400 GB / tape
• Archive - Written to CD or DVD