Yeah, doing some preferred cloud Data Warehouse with an indexing layer seems fine for this sort of thing. That has an advantage over something specialized like this of still being able to easily do stream processing / Spark / etc, plus probably saves some money.
Maybe Quickwit is that indexing layer in this case? I haven't dug too much into the general state of cloud dw indexing.
Quickwit is designed to do full-text search efficiently with an index stored on an object storage.
There are no equivalent technology, apart maybe:
- Chaossearch but it is hard to tell because they are not opensource and do not share their internals. (if someone from chaossearch wants to comment?)
- Elasticsearch makes it possible to search into an index archived on S3. This is still a super useful feature as a way to search punctually into your archived data, but it would be too slow and too expensive (it generates a lot of GET requests) to use as your everyday "main" log search index.
Maybe Quickwit is that indexing layer in this case? I haven't dug too much into the general state of cloud dw indexing.