(BigData 2020) Entropy-based Approach to Efficient Cleaning of Big Data in Hierarchical Databases

Abstract: When databases are at risk of containing erroneous, redundant, or obsolete data, a cleaning procedure is used to detect, correct or remove such undesirable records. We propose a methodology for improving data cleaning efficiency in a large hierarchical database. The methodology relies on Shannon's information entropy for measuring the amount of information stored in databases. This approach, which builds on previously-gathered statistical data regarding the prevalence of errors in the database, enables the decision maker to determine which components of the database are likely to have undergone more information loss, and thus to prioritize those components for cleaning. In particular, in cases where the cleaning process is iterative (from the root node down), the entropic approach produces a scientifically motivated stop-ping rule that determines the optimal (i.e. minimally required) number of tiers in the hierarchical database that need to be examined. This stopping rule defines a more streamlined representation of the database, in which less in-formative tiers are eliminated.

Authors: Eugene Levner (Holon Institute of Technology, Israel); Boris Kriheli (Ashkelon Academic College, Israel); Arriel Benis (Holon Institute of Technology, Israel); Alexander Ptuskin (Moscow Bauman Technical University, Kaluga Branch, Russia); Amir Elalouf and Sharon Hovav (Bar Ilan University, Israel); Shai Ashkenazi (Ariel University, Israel)

Email: levner@hit.ac.il, borisk@hit.ac.il, arrielb@hit.ac.il, aptuskin@mail.ru, amir.elalouf@biu.ac.il, sharonh@clalit.org.il, shai.ashkenazi7@gmail.com