Snowflake is changing the game for businesses that rely on analytics workloads. The company is now allowing users to run Apache Spark directly in its cloud, eliminating the need to host separate instances elsewhere and reducing data transfer delays.

Imagine being able to access all your analytics tools from a single platform, without worrying about transferring data back and forth between systems. That’s exactly what Snowflake has achieved with its new offering, called Snowpark Connect for Apache Spark. This innovative solution uses the same technology as Apache Spark’s Spark Connect feature, introduced in version 3.4.

With Snowpark Connect, users can run their Spark code on Snowflake’s vectorized engine in Data Cloud. This not only saves developers time and effort but also reduces costs associated with maintaining separate infrastructure. Sanjeev Mohan, chief analyst at SanjMo, believes that this new capability will make it easier for developers to move their code to Snowpark, offering a combination of Spark’s familiarity and Snowflake’s simplicity.

Other benefits include faster processing times due to Snowflake’s vectorized engine and the elimination of challenges such as finding staff with Spark expertise. Shubham Yadav, senior analyst at Everest Group, sees this launch as timely, given the growing demand for simplified infrastructure and cost reduction amidst AI and ML adoption.

It’s worth noting that there may be some confusion between Snowpark Connect for Apache Spark and Snowflake’s existing Snowflake Connector for Spark. Think of the Connector like a bridge connecting two cities, allowing data to travel back and forth. In contrast, Snowpark Connect is more like relocating the entire Spark city into Snowflake.

Enterprises can migrate from the existing Connector to the newer Snowpark Connect without needing to convert any code. The new capability is currently in public preview and works with Spark 3.5. Rival Databricks also offers similar capabilities via its Databricks Connect offering.

Snowflake’s move into direct cloud access for analytics workloads marks a significant shift in the industry, one that could change the way businesses approach data analysis. As more companies seek to simplify their infrastructure and reduce costs, Snowflake is poised to be at the forefront of this trend.

Inspired by

• https://www.infoworld.com/article/4030595/snowflake-brings-analytics-workloads-into-its-cloud-with-snowpark-connect-for-apache-spark.html

Sources

• shutterstock.com
• apache.org
• snowflake.com
• snowflake.com