That’s cool and I tested it via the kubeflow’s spark-operator, it was a bit difficult to make it work with connection to my iceberg catalog. One thing that would be so cool is if the spark-connect driver could scale down to 0 instead of being idle, and have a kubernetes operator to add/remove spark-connect drivers!
From my knowledge, the spark-operator only create driver and executor when the spark job is submitted. How it work with spark-connect? Please let me know
That’s cool and I tested it via the kubeflow’s spark-operator, it was a bit difficult to make it work with connection to my iceberg catalog. One thing that would be so cool is if the spark-connect driver could scale down to 0 instead of being idle, and have a kubernetes operator to add/remove spark-connect drivers!
From my knowledge, the spark-operator only create driver and executor when the spark job is submitted. How it work with spark-connect? Please let me know