So I have this extra large query. It’s eating up the memory on my 3 nodes. Is there anyway we can cap memory on specific queries and force Starburst to batch the processing of that query? Any other ideas?
This is a great use case for fault-tolerant execution. Starburst | Enable fault-tolerant execution for queries in SEP
You can configure SEP to be more resilient against query failure by enabling fault-tolerant execution. This allows SEP to handle larger queries such as batch operations without worker node interruptions causing the query to fail.