OpenDSO Data Flow
Data flow diagram
The above diagram shows the high level data flow between the Open Distributed Systems Operations (OpenDSO) modules that compose the OpenDSO framework. OpenDSO communications are handled by both RPCs (gRPC, HTTP/JSON APIs), and the NATS message bus. With that said, most communication and data exists on the NATS message bus, a result of the distributed nature of OpenDSO.
The majority of data on the NATS message bus is either OpenFMB Profiles (protocol buffers) or OpenDSO Data Models (ODM). OpenFMB Adapters produce the majority of OpenFMB Profile data, and the Topology services generate the ODM Data.
The data on the NATS bus is not limited to ODM and OpenFMB Profiles. Services like the historian provide their own data models (DTOs), which are documented on the DTOs section of this website.
The below sections break down each main area found in the Data Flow diagram above. Each section explains some of the data types flowing in-and-out of those areas, but more in depth implementation details relating to each service and application can found throughout this website.
Topology Services / ODM Models
The ODM model is a flexible data model, that can contain all or some of the equipment on a circuit depending on the context of when and where that data was generated. ODM holds equipment-model information (power limits, capabilities, etc) and their topological information (open-or-closed, and connected-devices).
- Topology Nodes produce "active/connected" state ODM models, showing what devices are currently connected on a given circuit.
- Topology Genesis (CIMEX) only produces "as designed" ODM models (explained more in the CIM section below).
As seen on the above diagram, both Topology services populate the NATS Bus with ODM data. The specifics of the ODM model can be found in the DTOs/Cimex section of this website.
OpenDSO Data
Including the ODM model described above, there is a wide-variety of data available to applications in an arbitrary OpenDSO setup. Below is a table of each type of data, a brief description of it, it's sources and the protocol/data-format for each source. A source is defined as either a service that emits data, or a service that captures the data for later-access. Note: data not currently available has been marked with an asterisk.
| Type | Description | Sources:(Protocol|Data-format) |
|---|---|---|
| OpenFMB Message | Protocol buffer representation of an OpenFMB Profile | openfmb-adapter:NATS, citus:JSON, historian:GRPC |
| OpenDSO Event | Protocol buffer representation of a CloudEvent containing generic data | openfmb-event-creator:NATS, citus:JSON,historian:GRPC |
| OpenDSO Log | Log message emitted from an OpenDSO service such as topology-nodes, topology-genesis, historian, etc. | <opendso-service>:STDOUT, fluentd:JSON, citus:JSON |
| *System Log | System log message emitted from system hosting an OpenDSO service; captured with fluentd and stored in citus | syslog:UDP, fluentd:JSON, citus:JSON |
| OpenFMB Rule | Defines a rule for an OpenFMB device; when rule is broken an OpenDSO Event containing the rule is published via NATS | openfmb-event-creator:NATS, citus:JSON, historian:GRPC |
| Cimex-Equipment | DTOs for describing various types of equipment and their properties | topology-nodes:NATS, topology-genesis:NATS |
| GMS-Equipment | A modified version of Cimex-Equipment for use in GMS UI applications | topology-nodes:NATS, topology-genesis:NATS |
| Historian Request | Protocol buffer message for requesting OpenFMB/OpenDSO Event data persisted via historian | <3rd-party-service>:GRPC |
| *OpenDSO-CoreDNS Request | Protocol buffer message for registering the necessary DNS records for accessing an arbitrary service via some domain | <3rd-party-service>:NATS |
Below is a diagram that provides a basic diagram for the data flow of OpenDSO Logs, Events, and OpenFMB Profiles:
Applications and Services
As seen in the diagram above, each Service and Application ingests ODM and OpenFMB data types for their own needs. Each service and application may produce OpenFMB Control Profiles, often in response to the state reflected in the ingested ODM and Proto data. But no service, other than the Topology services, produce ODM data at this time.
CIM / Circuit Model Ingestion
For circuit modeling, OpenDSO topology ingests CIM XML files that implement the CIM IEC format. We have demo circuits (e.g. IEEE13 Feeder model) that showcase how CIM XML files are structured and how the resulting model is displayed and used in OpenDSO HMI's and services.
These CIM XML files are ingested by OpenDSO's Topology Service, the "Genesis Service". This service ingests the data by doing a circuit traversal and validation of the CIM XML data, resulting in an electrically connected node structure. This data then becomes accessible via Topology API calls. This 'electrically connected' data type is the "OpenDSO Data Model (ODM)", or simply "CIMEX" (CIM-expanded).
The current CIM spec is very nuanced. There is often an effort needed to correct or refine CIM files during the ingestion into OpenDSO. If you're finding the ingestion process to have issues, your OpenDSO team will be able to point you in the right direction.
OpenFMB Data and Simulation
The above diagram assumes a DI/Algorithm development environment, in which a Power Flow Simulator is producing DNP3 data that is mapped to OpenFMB Adapters.
The data of a simulated environment when compared to a real world environment is intentionally designed to look the same. This is an intentional goal of OpenDSO, aimed to provide transportability of the platform between testing environments and real world environments. The transportable nature of OpenDSO allows engineers to develop on their own PCs, gaining a level of QA and trust of their applications, before deploying and testing on more time consuming & expensive equipment.
OpenDSO is not strictly tied to any simulation software. To ease our own internal testing, we created a simple simulator "Maverick", which is strictly limited to quasi-real data population and basic control responses on our demo circuits. But any simulation software that can be mapped to OpenFMB Adapters can be used within the OpenDSO framework. This includes Hardware-In-The-Loop (HIL) test beds.
Adding a Power Simulator, as indicated in the above diagram, is not necessarily trivial and the efforts depend heavily on features of the given simulator. If the simulator can produce OpenFMB mappable protocols (ex. DNP3/Modbus), then the efforts required should mostly be the provisioning of OpenFMB adapters, steps of which are found on the OpenFMB Docs website
If you are planning for DI/Algorithm development, it's highly recommended you implement an appropriate simulator for your needs. We have experimented with OpenDSS and Typhon in our internal development, but are still working on the distribution of software simulators (OpenDSS) and other simulators in our deployment process. We plan to provide more on this topic in the near future.