The Apache Flink team has just released a major upgrade to its real-time data processing engine, and it’s a big deal for anyone interested in AI and real-time decision-making. Apache Flink 2.1.0 is now available for download from flink.apache.org, and it adds some serious firepower to the platform.

The new release introduces support for defining and managing AI models, as well as invoking them in real time within Flink SQL. This is a game-changer for building end-to-end, real-time AI workflows. With Apache Flink 2.1.0, users can define and manage AI models programmatically via the Table API in both Java and Python, providing a flexible, code-driven alternative to SQL.

The ML_PREDICT table-valued function (TVF) has also been expanded to perform real-time model inference in SQL queries. This means that machine learning models can be applied to data streams seamlessly, without needing to write custom code. The implementation supports both Flink built-in model providers and interfaces for users to define custom model providers, making it easier to integrate AI into real-time data processing.

But that’s not all – Apache Flink 2.1.0 also introduces Process Table Functions (PTFs), which are the most powerful kind of function for Flink SQL and Table API. PTFs are a superset of all other user-defined functions, mapping zero, one, or multiple tables to zero, one, or multiple rows. This enables implementing user-defined operators that can be as feature-rich as built-in operations.

New Features and Improvements

Apache Flink 2.1 also adds VARIANT as a data type for semi-structured data such as JSON. This new type supports storing any semi-structured data, including ARRAY, MAP (with STRING keys), and scalar types, while preserving field type information in a JSON-like structure. Users can use PARSE_JSON or TRY_PARSE_JSON to convert JSON-formatted VARCHAR data to VARIANT.

Other improvements in the new release include a DeltaJoin operator for stream processing jobs, optimizations for a simple streaming join pipeline, Smile binary format support for compiled plans, and a pluggable batching mechanism for Async Sink. A new connector for keyed state also allows users to query keyed state directly from a checkpoint or savepoint using Flink SQL.

Overall, Apache Flink 2.1.0 is a significant upgrade that will make it easier to build real-time AI workflows and integrate AI into data processing pipelines. With its new features and improvements, the platform is taking a major step forward in its evolution from a real-time data processing engine to a unified real-time AI platform.

Inspired by

• https://www.infoworld.com/article/4033040/apache-flink-integrates-ai-for-real-time-decision-making.html

Sources

• apache.org
• apache.org