Why is Trino is faster than Spark?

Trino
1

Originally posted on the Trino Slack.

Why is Trino is faster than Spark?

1 Like
2

Answers were sourced from this Stack Overflow Question and the answer in Trino Slack.

In general, it is hard to say if Trino is definitely faster or slower than Spark SQL. It really depends on the type of query you’re executing, environment and engine tuning parameters. However, what I see in the industry(Uber, Neflix examples) Trino is used as ad-hoc SQL analytics whereas Spark for ETL/ML pipelines.

  • There is no much overhead for scheduling a query for Trino. The Trino coordinator is always up and waits for a query. On the other hand, Spark is doing lazy approach. It takes time for the driver to negotiate with the cluster manager the resources, copy jars and start processing.

  • Trino uses a pipelined execution model. i.e. each operation is given a Page of data to work on, once that Page is done it can be sent to next operator. This means that multiple parts of a query plan can execute concurrently as long as there is data to be read. While Spark has disconnected stages. Until one stage finishes processing the next stage cannot do anything. i.e. the execution is very linear.

  • Trino is 100% in-memory and all stages of a plan start executing at the same time while in case of Spark the driver requests resources for each stage separately (this was true the last time I checked but may have changed now).

  • Trino was built from the ground up as an MPP SQL engine so most of the code and architecture is specifically tailored towards SQL queries while Spark on the other hand started as a general purpose distributed execution framework so it can be used to do almost anything while Trino focuses on SQL and does it really well.

  • Trino sacrifices mid-query fault-tolerance because most people running SQL queries are doing it in an interactive manner and speed is more important in interactive use-cases. Spark suffers here in terms of performance because it has to do a lot of bookkeeping to provide the fault tolerance even when most of the queries don’t fail.

  • Trino tries to push-down some operations to the remote system which can be done more efficiently there. e.g. filtering on a column that has an index in the remote RDBMS will be faster than pulling all data and then filtering in Trino. Spark doesn’t pushdown any operations at all and has to pull most of the raw data and then apply processing on top of it.

2 Likes
3

Thank you for pointing this out. It is often overlooked and taken for granted.

A key trade-off is this:
In order to add resiliency to a Spark operation, it has to spend resources (memory/cpu) with the notion that if it has to “replay” something, there is a way to recover. But this can (in many cases) reduce performance and increase compute requirements.

Instead of allocating additional resources for that overhead of “protection”, Trino instead utilizes it for execution and completes the job more reliably in the first place.

Spark is more “forgiving” from a sense that it can “spill” oversubscription of memory to storage (if needed), but this can dramatically impact performance. This alone can be attributed to query failures due to timeouts or hardware misconfigurations. There is no “free lunch” with this approach in Spark… it has costs associated with it that have to be taken into acount.

1 Like