Body
Currently, when an adapter is updated, its data stream schema is automatically propagated to downstream pipelines. While this update mechanism works, it can introduce runtime errors if the schema change is incompatible with existing pipeline components, particularly data sets.
A common example is a change in the data type of a dataset column that is used within a measurement. Such changes can break the measurement logic and only surface as runtime errors, making them hard to detect early.
To improve robustness and user experience, the system should detect potentially incompatible dataset schema changes and surface a clear warning within affected pipelines. At minimum, users should be notified that a schema change may lead to issues, enabling them to review and adjust their configurations proactively.
StreamPipes Committer
I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.
Body
Currently, when an adapter is updated, its data stream schema is automatically propagated to downstream pipelines. While this update mechanism works, it can introduce runtime errors if the schema change is incompatible with existing pipeline components, particularly data sets.
A common example is a change in the data type of a dataset column that is used within a measurement. Such changes can break the measurement logic and only surface as runtime errors, making them hard to detect early.
To improve robustness and user experience, the system should detect potentially incompatible dataset schema changes and surface a clear warning within affected pipelines. At minimum, users should be notified that a schema change may lead to issues, enabling them to review and adjust their configurations proactively.
StreamPipes Committer
I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.