A question about auto-scaling

Trino
1

Hey guys, I have a k8s trino cluster deployed more or less manually on AWS using their tutorial, give or take touching a few parameters on the values file
One of which is

autoscaling:
enabled: true

min_workers is 1 and max is 8
I’ve been getting the error:

TrinoQueryError(type=INSUFFICIENT_RESOURCES, name=EXCEEDED_LOCAL_MEMORY_LIMIT, message="Query exceeded per-node memory limit

and was wondering, why is there no attempt to scale up when this happens? There was 1 worker during that whole process for the entire time.

Note that auto-scaling does work on more complicated queries, this one just happens to be a rather large and simple “select * from very_big_table” type

Thanks to anyone that can help explain what I did wrong or need to know about auto-scaling :smile:

2

Your error message indicates, that your worker instance does not have enough memory to execute the query.
Autoscaling scales the workers horizontally.

What you can do is:

  • Check jvm config, if the memory settings are correct
  • Scale workers vertically. E.g. more ram.

On the query side, what you can do is:

  • partition your table
  • use filters with your partitions

And I think this is an amazing video from Starburst to understand cluster sizing and optimization:
Tuning Starburst and Trino Performance