Analytical store for streaming data with huge volume
A concept for organizing an analytical data warehouse has been developed, which includes a method of interaction between data producers and a repository, a method of data circuit control, a method of data streaming, a method of storing initial data, a method of data processing and a method of provid...
Gespeichert in:
| Datum: | 2022 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Artikel |
| Sprache: | Ukrainian |
| Veröffentlicht: |
Інститут програмних систем НАН України
2022
|
| Schlagworte: | |
| Online Zugang: | https://pp.isofts.kiev.ua/index.php/ojs1/article/view/489 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Назва журналу: | Problems in programming |
Institution
Problems in programming| Zusammenfassung: | A concept for organizing an analytical data warehouse has been developed, which includes a method of interaction between data producers and a repository, a method of data circuit control, a method of data streaming, a method of storing initial data, a method of data processing and a method of providing secure data access. Other concepts on the market are discussed, namely: SDLF as the leading standard recommended by AWS, IronSource DL using Upsolver, SimilarWeb DL using Upsolver. A comparative analysis was conducted (mostly with SDLF, as its implementation is open, and the implementation by private companies is hidden). The advantages of the proposed concept over the existing ones are examined in detail. Recommendations on how to integrate the concept with data schema control applications are given. A service for streaming data using Apache Beam in Java has been developed. A repository architecture for analytics was designed and developed. A data schema management model was developed as well as a data schema management model and a model for secure access to data. The research that has been conducted can be improved by the experience of implementing the concept in business, as well as by collecting and systematizing knowledge about other standards that will be created.Problems in programming 2022; 1: 67-74 |
|---|