Teradata Listener #TD3PI

Continuing at the Teradata Influencers event with a discussion of Teradata Listener. Teradata Listener provides real time streaming data support., intending to make it easy to allow multiple projects in an enterprise to listen to streaming data, capture it and make it available for later analysis. It’s based on open source and software only. It is designed to support different user groups:

• IT, allowing them to reduce complexity by simplifying the data integration architecture
• For developers it is designed to provide APIs that are easy to use.
• For data scientists its all about self service integration.

The product is designed to deliver four key features:

• No ETL when capturing data at original fidelity
• High volume streams
• Volume metrics
• Publish and collect data

It is designed for open access and self service. As such it supports throttles and locks as well as simple pause and locks. Data can be shared into multiple places, allowing sandboxes (like Teradata Data Labs for instance) to listen to a subset of data as well as production environments.  It is also designed as an enterprise platform so it is highly scalable and fault tolerant as well as API-centric.

Use cases include:

• Web click data collection
• Application and system data log collection
• Data enrichment
• IoT/Sensor
• Log/ETL buffer

Listener sits between data sources and target systems. Sources are ingested through a REST API then through a kafka queue to a spark router that splits it into multiple streams based also on kafka that then write to target systems. The architecture decouples how the sources produce data from how the systems listen to it – a burst or batch of data can be ingested and cached before being fed to listening systems.

Listener will also allow customers to pick up data within the environment and integrate it with other stream processing environments. Everything in Listener is parallelized and this includes external integrations.

Listener is still under development and is in internal beta. Mid to late Q3 Teradata are hoping to get some early customer engagements and GA in early Q4. Public announcements to come.