Mar 15, 2025

Looking at the most optimal filtering of the point table for duplicates, it appears that Bloom filtering, while logically the simplest approach - basically what its designed for, is very inefficient. This is because the data is necessarily stored in a hashmap (for at least allowing efficient resolution of false positives), and a global bloom filter would not be taking advantage of the prefiltering by hashmap bucket. Using bloom filters within a bucket, for the same overall size of the filters, is also functional and inefficient. The filter becomes quickly saturated at higher k values and at lower values has a very high false positive rate.