EDocDeDup: Electronic Document Data Deduplication Towards Storage Optimization

doi:10.23940/ijpe.23.07.p6.471480

Abstract

Abstract: Understanding data deduplication in storage is essential for investigating the optimization of various data storage issues. For detecting and removing duplicate data, data deduplication has become an important and cost-effective optimization technique. Storage issues in the storage area for organizations exist, and if not conveyed to optimize there and then, a slower rate of storage capacity is expected. The proposed system (EdocDedup) addresses the aforementioned issue by applying data deduplication technique and implementing SHA-256 for hash value calculation and only keeping the unique hash values on an electronic document’s dataset containing word files, text files, html files, excel files, zip files, pdf files, and PowerPoint presentation files. By demanding the proposed technique there is a benefit in storage saved and a variety of duplicate files are explored efficiently. EdocDedup's performance is achieved through the use of user-uploaded files.

Key words: data deduplication, EdocDedup, electronic document files, SHA-256, hash, storage optimization

Me Me Khaing and N. Jeyanthi. EDocDeDup: Electronic Document Data Deduplication Towards Storage Optimization [J]. Int J Performability Eng, 2023, 19(7): 471-480.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

1. Xia W., Jiang H., Feng D., Douglis F., Shilane P., Hua Y., Fu M., Zhang Y. and Zhou Y., 2016. A comprehensive study of the past, present, and future of data deduplication. Proceedings of the IEEE,104(9), pp.1681-1710.
2. He Q., Li Z. and Zhang X., 2010, October. Data deduplication techniques. In 2010 international conference on future information technology and management engineering (Vol. 1, pp. 430-433). IEEE.
3. Sujatha, G. and Raj, J.R., 2022. A Comprehensive Study of Different Types of Deduplication Technique in Various Dimensions. International Journal of Advanced Computer Science and Applications,13(3).
4. Singh, S. and Singh, R., 2018. A Viewpoint on Different Data Deduplication Systems and Allied Issues. In Progress in Advanced Computing and Intelligent Engineering: Proceedings of ICACIE 2016, Volume 2(pp. 385-393). Springer Singapore.
5. Gowthaman, A. and Sumathi, M., 2015. Performance study of enhanced SHA-256 algorithm. International Journal of Applied Engineering Research,10(4), pp.10921-10932.
6. Zhang, X. and Deng, M., 2017. An overview on data deduplication techniques. In Information Technology and Intelligent Transportation Systems: Volume 2, Proceedings of the 2015 International Conference on Information Technology and Intelligent Transportation Systems ITITS 2015, held December 12-13, 2015, Xi’an China (pp. 359-369). Springer International Publishing.
7. Sharma N., Prasad A.K. and Kakulapati V., 2021, January. File-level Deduplication by using text files-Hive integration. In 2021 International Conference on Computer Communication and Informatics (ICCCI)(pp. 1-6). IEEE.
8. Krishina Sankar Pet al., “Identification of duplicate files using SHA3 algorithm”, International Journal of Applied Engineering Research, Vol. 10, No. 9, pp8112-8116, (2015).
9. G. Kalyaniet al., “Data de-duplication using SHA (Secure Hash Algorithm)”, International Journal of Scientific Research in Computer Science, Engineering and Information Technology, Vol. 5, No. 3, pp1-5, (2019).
10. Rajeev, S. and Geetha, G., 2012. Cryptographic hash functions: a review international. Journal of Computer Science Issues,9(2).
11. Sahu, A. and Ghosh, S.M., 2017. Review paper on secure hash algorithm with its variants. International Journal of Technical Innovation in Modern Engineering & Science,3(05).
12. Purohit R., Mishra U. and Bansal A., 2013. A survey on recent cryptographic hash function designs. International Journal of Emerging Trends & Technology in Computer Science (IJETTCS),2(1), pp.2278-6856.
13. Thwel, T.T. and Sinha, G.R., 2019. Efficient Data Deduplication Mechanism for Genomic Data.CSVTU International Journal of Biotechnology, Bioinformatics and Biomedical 4.2, pp.52-58.
14. Thwel, T.T. and Thein, N.L., Data Deduplication using B+ Tree Indexing(Doctoral dissertation, MERAL Portal).
15. Ninni Singhet al., “A Novel Encryption Technique for Data De- Duplication”, IEEE, AMITY, (2014).
16. Benita Musabiyinemaet al., “Deduplication storage optimization for data management”, International Journal of Scientific & Engineering and Research, Vol. 9, No. 9, pp328-334, (2018).
17. Shaikh D.A.,2022. Implementation of Data Deduplication Detection and Removal on Cloud Using 64-Bit Architecture of SHA-256.Available at SSRN 4109062.
18. Shay Shay Gureron, “SHA512/256”, The International Association for Cryptographic Research (IACR), (2010).
19. Anita Kundgiret al., “A Survey on secure deduplication schemes”, International Journal of Advanced Research, Vol. 5 No. 6, pp1787-1792, (2017).
20. Meister D.,2013. Advanced data deduplication techniques and their application (Doctoral dissertation, Mainz, Univ., Diss., 2013).

[1]	Sagnik Pal, Rutvik Patel, Vijayasherly V., and Ramani Selvanambi. Hashtag Recommendation System for Instagram Posts using Transfer Learning with EfficientNet and ALS Model [J]. Int J Performability Eng, 2022, 18(8): 552-558.
[2]	Yong Gan, Yuan Zhuang, and Lei He. RFID Tag Ownership Transfer Protocol using Blockchain [J]. Int J Performability Eng, 2019, 15(9): 2544-2552.
[3]	Xiaomei Tian, Huihuang Zhao, Yaqi Sun, and Xiaoman Liang. Efficiently Retrieving Differences Between Remote Sets using Counting Bloom Filter [J]. Int J Performability Eng, 2019, 15(7): 1947-1954.
[4]	Pinle Qin, Qi Li, Jianchao Zeng, Haiyan Liu, and Yuhao Cui. Fully Convolutional-based Dense Network forLungNodule Image Retrieval Algorithm [J]. Int J Performability Eng, 2019, 15(1): 326-336.
[5]	Shan Zhao and Yongsi Li. Locality Preserving Hashing based on Random Rotation and Offsets of PCA in Image Retrieval [J]. Int J Performability Eng, 2018, 14(11): 2601-2611.