RapidAPI announced it has added support for the open source Kafka messaging software as another type of application programming interface (API) that DevOps teams can centrally access.
An open source library provided by RapidAPI, available in beta, makes it possible to discover Kafka clusters and topics, view schemas and test and consume records from a browser via either the RapidAPI Marketplace or the RapidAPI Enterprise Hub. The library is based on Apache Avro and Confluent wire formats found in the open source confluent-schema-registry library. RapidAPI extended that library to add support for JSON and a protocol buffer, which is now being shared with the open source Kafka community.
RapidAPI CEO Iddo Gino said DevOps teams can now discover all available Kafka clusters and instances in the same location where they find, test and connect to microservices based on REST, SOAP and GraphQL APIs. That effort is designed to complement initiatives, such as the AsyncAPI specification, that is now being advanced under the auspices of the Linux Foundation, Gino noted.
RapidAPI is adding this capability because usage of Kafka to enable asynchronous communication across event-driven applications has significantly increased, Gino said. As the number of digital business transformation initiatives increased in the wake of the COVID-19 pandemic, Gino said reliance on asynchronous messaging platforms such as Kafka became a requirement. In many cases, synchronous communications across highly distributed applications simply isn’t feasible, Gino added.
That doesn’t mean asynchronous requirements will replace the need for traditional synchronous APIs. Rather, developers will find themselves incorporating a greater mix of both as enterprise IT environments become more distributed, noted Gino. Applications are increasingly become one large puzzle that developers need to construct using various classes of APIs, added Gino.
The library created by RapidAPI is expected to be generally available in the third quarter. In the meantime, adoption of Kafka continues to accelerate. The open source community that oversees development of Kafka, under the auspices of the Apache Software Foundation, claims more than 80% of all Fortune 100 companies are using the platform for everything from constructing data pipelines to running streaming analytics applications. In some cases, data is being queried directly while in transit versus waiting for it to be stored on-premises or in the cloud.
Regardless of how Kafka is employed, developers are rediscovering event-driven architectures, that can trace their lineage to the 1980s, for building applications. The difference now is there an open source framework for handling the asynchronous communications required to build and deploy those applications at scale.
It’s not quite clear how the rise of event-driven applications will impact DevOps workflows, as event-driven applications become more commonly built and deployed. However, as the APIs become more widely employed within the context of these applications, it should become simpler to incorporate them.
The challenge is event-driven architectures typically requires developers to think differently about how their applications are structured, so it might be a while before Kafka – and other similar platforms – are pervasively employed across the enterprise.