Izdelava programske opreme za preverjanje, ocenjevanje in urejanje podatkovnih tokov in model kakovosti podatkovnega toka in vira : študija št. 2377 /
V študijski nalogi je razvit pregleden model kazalnikov kakovosti podatkov, ki omogoči naročniku spremljanje kakovosti podatkov in posledično pripomore k natančnejšim rezultatom pri obdelavi teh podatkov. Model je implementiran v knjižnici za validacijo, ocenjevanje in urejanje podatkov merilnih sis...
Shranjeno v:
| Main Authors: | , , , , , |
|---|---|
| Format: | Knjiga |
| Jezik: | Slovenian |
| Izdano: |
Ljubljana :
Elektroinštitut Milan Vidmar,
2018.
|
| Teme: | |
| Oznake: |
Označite
Brez oznak, prvi označite!
|
| Izvleček: | V študijski nalogi je razvit pregleden model kazalnikov kakovosti podatkov, ki omogoči naročniku spremljanje kakovosti podatkov in posledično pripomore k natančnejšim rezultatom pri obdelavi teh podatkov. Model je implementiran v knjižnici za validacijo, ocenjevanje in urejanje podatkov merilnih sistemov, poleg tega pa je implementiran sistem sporočilnih vrst, ki omogoča prenos merilnih podatkov od virov do podatkovnega skladišča SODO ob hkratnem izvajanju procesov validacije, ocenjevanja in urejanja ter spremljanju kakovosti tako pridobljenih podatkov. Model kakovosti podatkov temelji na njihovi hierarhični organiziranosti v časovne vrste (npr. merilne kanale), ki se paketno prenašajo iz vsakega podatkovnega toka (npr. merilnega mesta) do podatkovnega vira (npr. skladišča podatkov izbranega EDP). Ta podatke posreduje v naročnikovo podatkovno skladišče. Kakovost je opazovana s štirimi skupinami kazalnikov kakovosti podatkovnega toka. Kazalniki popolnosti opazujejo razliko med pričakovanimi in dejanskimi količinami podatkov, kazalniki rednosti obravnavajo pravočasnost prispetja podatkov, kazalniki usklajenosti obravnavajo skladnost z regulatomimi in fizikalnimi zakonitostmi, katerim naj bi izmerjeni podatki ustrezali, kazalniki natančnosti pa obravnavajo ujemanje podatkov s predvidenimi modeli procesov, ki podatke generirajo. Predstavitvi abstraktnega modela kakovosti podatkov in definiciji povezanih kazalnikov sledi opis implementacije prvih treh skupin kazalnikov, ki so ciljne skupine za dani nabor merilnih podatkov iz elektrodistribucijskega omrežja, saj spremljanje in vrednotenje natančnosti meritev z razpoložljivimi podatki ni izvedljivo. Sledi opis arhitekture aplikacije za izvajanje procesa VEE in nato še opis sistema za ,Y.alidacijo in estimacijo podatkov. Za izdelovalce komponent za izvajanje procesa VEE so opisani poteki celotnega procesa validacije in estimacije, prehodi statusov meritev ter potrebni vhodni podatki za posamezna preverjanja. Iz mednarodnega standarda IEC 61968-9 in standarda podjetja Elhub so povzeti procesi validiranja in ocenjevanja. Sledi podrobnejši opis namestitve vseh potrebnih sistemov in storitev, opis vseh uporabljenih metod za validiranje in estimiranje podatkov, opis funkcij in procesa generiranja poročila o kakovosti podatkov ter opis knjižnic in strukture storitve obdelave sporočila iz sporočilne vrste. Za lažje razumevanje poročila o kakovosti podatkov so podrobneje opisani posamezni vidiki kazalnikov kakovosti, ki dopolnjujejo primer poročila, izdelanega na vzorčnih podatkih. The tehnical report presents a transparent data quality model, which enables monitoring of the quality of data and, consequently, contributes to accuracy of results in the processing ot these data. The model is implemented as a library for validation, estimation and editing of metering data. In addition, a system of message queues is implemented that enables the transmission of measurement data from its sources to the SODO data warehouse, while simultaneously carrying out validation, estimation and editing processes and monitoring of the quality of the obtained data. The data quality model is based on its hierarchical organization into tirne series (eg. measuring channels) that are transmitted as XML meassages from each data stream (eg. measuring point) to the data source (eg. EDP database). It transmitts data to clients database. Quality is observed with four grups of data quality indicators. The indicators of completenes observe the difference between the expected and the actual amount of data, the indicators of regularity consider the timeliness of the arrival of data, the indicators of consistency consider the regulatory and physical laws which the measured data should adhere to, and the indicators of accuracy consider the matching of data with the predicted models of processes that generate data. The presentation of the abstract data quality model and the definition of related indicators are folowed by the description of the implementation of the first three groups of indicators. These are the target groups for a given set of measurements from the elecrticity distribuion network, since monitoring and evaluation of the accuracy of the measurements would require supplementary information and processing that is not available in the given context. The tehnical implementation of the VEE processes is first described through the application architecture of the implement VEE process, as well as a detailed description of the workflow for validation and estimation of data. For component manufacturers to implement the VEE process, we described the processes of the validation,and estimation, the changing of measurement statuses and the necessary input data for individual verifications. Details of the VEE process, including formulae, were summarized from international standards IEC 61968-9 and from Elhub 2016 standard. Installation and maintenance of the developed system are facilitated by instructions on docker, services and libraries installation, and a description of all methods used to validate and evaluate data, a description of the functions and process of generating a data quality report, and a description of the libraries and the structure of the message processing service form the message type. In order to facilitate the understanding of the sample quality report prepared on given test data, individual aspects of the quality indicators are described in a one-page explanatory summary that is to supplement the quality report andThe tehnical report presents a transparent data quality model, which enables monitoring of the quality of data and, consequently, contributes to accuracy of results in the processing ot these data. The model is implemented as a library for validation, estimation and editing of metering data. In addition, a system of message queues is implemented that enables the transmission of measurement data from its sources to the SODO data warehouse, while simultaneously carrying out validation, estimation and editing processes and monitoring of the quality of the obtained data. % The data quality model is based on its hierarchical organization into tirne series ( eg. measuring channels) that are transmitted as XML meassages from each data stream (eg. measuring point) to the data source (eg. EDP database). It transmitts data to clients database. Quality is observed with four grups of data quality indicators. The indicators of completenes observe the difference between the expected and the actual amount of data, the indicators of regularity consider the timeliness of the arrival of data, the indicators of consistency consider the regulatory and physical laws which the measured data should adhere to, and the indicators of accuracy consider the matching of data with the predicted models of processes that generate data. The presentation of the abstract data quality model and the definition of related indicators are folowed by the description of the implementation of the first three groups of indicators. These are the target groups for a given set of measurements from the elecrticity distribuion network, since monitoring and evaluation of the accuracy of the measurements would require supplementary information and processing that is not available in the given context. The tehnical implementation of the VEE processes is first described through the application architecture of the implement VEE process, as well as a detailed description of the workflow for validation and estimation of data. For component manufacturers to implement the VEE process, we described the processes of the validation,and estimation, the changing of measurement statuses and the necessary input data for individual verifications. Details of the VEE process, including formulae, were summarized from international standards IEC 61968-9 and from Elhub 2016 standard. Installation and maintenance of the developed system are facilitated by instructions on docker, services and libraries installation, and a description of all methods used to validate and evaluate data, a description of the functions and process of generating a data quality report, and a description of the libraries and the structure of the message processing service form the message type. In order to facilitate the understanding of the sample quality report prepared on given test data, individual aspects of the quality indicators are described in a one-page explanatory summary that is to supplement the quality report and point to detailed descriptions. |
|---|---|
| Opis knjige/članka: | Podatek o odgovornosti na začetnih str. Soavtorji: Amadeja Bratuša, Tadej Žerak, Robert Goljat, Andrej Souvent, Leon Maruša. |
| Fizični opis: | XVII, 84 str. : ilustr. ; 30 cm. |
| Bibliografija: | Povzetek v slov. in angl. |