Middleware Beta Version: Advancing Unified Data Management
Maria Tassi, Nikos Gkevrekis, Evaggelos Foufikos and Konstatina Tsioli | February 9th 2025
Middleware Beta Version: Advancing Unified Data Management
Empowering interoperable and secure data ecosystems in line with CORE IC’s innovation vision.
Following the successful alpha release, CORE IC in the framework of CARDIMED project released the beta version of the unified middleware platform. This new release significantly enhances and expands the platform's capabilities, strengthening its role as a flexible and robust solution for managing heterogeneous data flows and ensuring secure, interoperable data sharing across diverse data sources and demonstration sites.
The beta version builds upon the layered architecture with critical enhancements aimed at improving performance, robustness and stability, expanding functionality,ensuring even greater data storage and sharing quality and reliability.
Key Feature Enhancements in the Beta Release
The beta version introduces several key advancements compared to the alpha release that are being summarized in the following table.
In more detail, the Beta Version includes the following enhancements:
Ingestion Layer Enhancement
The ingestion layer, which serves as the entry point for all incoming data, has been significantly upgraded:
Expanded Data Format and Transfer Support: The middleware now offers enhanced support for reading and processing data in common formats, specifically JSON, ensuring broader compatibility with various data sources. In parallel, the ability to send and receive data from data sources via MQTT / ΟPC-UA and HTTP has been improved, offering a more robust and reliable communication mechanism.
Advanced Data Preparation: Enhanced features for error handling and retry mechanisms have been implemented to increase data ingestion reliability, while enhancements have been made to ensure low latency in data processing, guaranteeing timely updates and minimal delay. In addition, new functionalities have been added for data cleansing and preparation to ensure that all ingested data is accurate and usable for downstream analysis and parallelization, allowing it to load data faster by handling multiple data ingestion and processing tasks simultaneously.
Storage Layer Optimization
The storage layer has achieved full deployment and received functional enhancements:
Fully Deployed Object Storage and Enhanced Timeseries Functionality: The MinIO S3 object storage is now fully deployed, providing a dedicated and scalable solution for handling large files efficiently, while the Timeseries Database has been enhanced, providing additional functionalities in PostgreSQL + Timescale for better time-series data management.
Updated Fileserver (Fileserver 2.0): The fileserver functionalities have been updated to support features like organizational-project context and improved document management and data organization.
Consumption Layer and Security Updates
The consumption layer, responsible for secure data access, has been expanded:
Expanded Interoperability: The support for REST APIs has been expanded across multiple layers, facilitating greater interoperability and integration options throughout the platform.
Enhanced permission handling and search options: Users can have access to multiple organizations, projects and their devices without needing more than one account and can search for specific devices and their metadata with optional searching queries before downloading the device’s data.
Monitoring and Operational Improvements
To ensure operational coherence and traceability, the beta version includes:
Expanded Event Logging and System Monitoring: Event logging capabilities have been expanded to keep minimum records of actions and monitor activities, which is crucial for identifying and fixing issues promptly.
