Awesome post, Vu. Have you taken a look at the Apache Gluten project? It can offload spark operations to native vectorized engines like Meta's Velox, similar to the databricks photon engine but open source. I think it's a promising project, especially with some people looking to move away from databricks after recent changes to their pricing models
Thank you for reaching out, Ibrahim; I've heard about Apache Gluten but have not dived deep into it yet. Thank you for your suggestion. For the point " especially with some people looking to move away from data bricks after recent changes to their pricing models," this is quite interesting. Could you share more about this?
Awesome post, Vu. Have you taken a look at the Apache Gluten project? It can offload spark operations to native vectorized engines like Meta's Velox, similar to the databricks photon engine but open source. I think it's a promising project, especially with some people looking to move away from databricks after recent changes to their pricing models
Thank you for reaching out, Ibrahim; I've heard about Apache Gluten but have not dived deep into it yet. Thank you for your suggestion. For the point " especially with some people looking to move away from data bricks after recent changes to their pricing models," this is quite interesting. Could you share more about this?
There was a reddit post on the data engineering subreddit that blew up, regarding the end of life of standard workspaces in databricks: https://www.reddit.com/r/dataengineering/comments/1c0uf21/and_so_it_begins_databricks_just_couldnt_help/
Will give it a look right away, thank you Ibrahim