NetApp Deduplication

NetApp Deduplication

Netapp supports deduplication where only unique blocks in the flex volume is stored and it creates a small amount of additional metadata in the de-dup process. The NetApp deduplication technology allows duplicate 4KB blocks anywhere in the flexible volume to be deleted and stores a unique one.

The core enabling technology of deduplication is fingerprints. These are unique digital signatures for every 4KB data block in the flexible volume.

When deduplication runs for the first time on a flexible volume with existing data, it scans the blocks in the flexible volume and creates a fingerprint database, which contains a sorted list of all fingerprints for used blocks in the flexible volume. After the fingerprint file is created, fingerprints are checked for duplicates and if found, first a byte-by-byte comparison of the blocks is done to make sure that the blocks are indeed identical. If they are found to be identical, the block’s pointer is updated to the already existing data block and the duplicate data block is released and inode is updated.

The below link will make you to understand the detailed information about how de-dup works - Source NetApp Community

https://communities.netapp.com/community/netapp-blogs/drdedupe/blog/2010/04/07/how-netapp-deduplication-works--a-primer

Deduplication refers to the elimination of redundant data in the storage. In the deduplication process, duplicate data is deleted, leaving only one copy of the data to be stored. However, indexing of all data is still retained should that data ever be required. Deduplication is able to reduce the required storage capacity since only the unique data is stored.

1. Enable dedup (asis) license:

filer> license add <Code for NearStore> filer> license add <Code for De-dup> filer> sis on /vol/bali

2. If you have a new flex volume which was just created, follow this step to enable ASIS deduplication

filer> sis on /vol/bali Deduplication for "/vol/bali" is enabled. Already existing data could be processed by running "sis start -s /vol/bali” 3. If you have already existing flex volume with data in it, follow this step.

filer> sis start -s /vol/bali

4. If you have already existing flex volume with data in it, follow this step.

filer> vol status bali Volume State Status Options bali online raid_dp, flex nosnap=on sis Containing aggregate: 'aggrSATA'

5. Check de-dupe status

filer> sis status /vol/bali Path State Status Progress /vol/demovol Enabled Idle Idle for 00:02:12

6. Check the storage space saved due to deduplication

filer> df -s /vol/bali Filesystem used saved %saved /vol/bali/ 9316052 0 0%

7. If you have to run de-dupe at a later point of time on this volume, use “sis status /vol/bali”

8. Deduplication can be scheduled by using "sis config" with below options sis config [ [ [ -s schedule ] | [ -m minimum_blocks_shared ] ] <path> ...] - Sets up, modifies and retrieves schedule and minimum blocks shared value of SIS volumes

9. If you wish to stop and delete the de-dupe entries on a specific volume, follow the below steps

filer> sis stop /vol/bali Operation is currently idle: /vol/bali filer> sis off /vol/bali SIS for "/vol/bali" is disabled. filer> priv set advanced Warning: These advanced commands are potentially dangerous; use them only when directed to do so by Network Appliance personnel. filer*> sis undo /vol/bali filer*> priv setfiler> sis status /vol/bali Disabled Undoing 6810 MB Processed filer> sis status /vol/bali No status entry found.