The Curious Case of High-Dimensional Indexing as a File Structure: A Case Study of eCP-FS

While approximate nearest-neighbor (ANN) search is an integral part of the modern multimedia analytics pipeline, the ever-hungry AI-models may frequently starve the ANN structures of resources, particularly memory. We present a novel white-box implementation of the disk-based hierarchical eCP index, called eCP-FS, that extends the disk-based strategy beyond merely storing data on disk, and instead implements the entire structure as an overlay file system using Zarr. This maps the (normally complex) index structure intuitively to a familiar hierarchical folder structure, which in turn makes the index much easier to visualize and analyse than the typical in-memory black-box structures of other algorithms. We furthermore implement incremental retrieval over eCP-FS, which benefits even more from file-system caching. Using an experimental benchmark inspired by live retrieval competitions, we show that despite trading raw speed for reduced memory footprint, eCP-FS is still a competitive option in the modern day analytics pipeline.