From mvasko at cesnet.cz Mon Apr 1 06:51:51 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Mon, 01 Apr 2019 08:51:51 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <577c-5ca1b500-9-12b8f000@181427425> Hi Henry, firstly, I am not part of NMDA working group or any other for that matter :) I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? Regards, Michal On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > Hi Michal? > > Thanks for all the insights. > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > IMHO, an implementation that allows such flexibility seems have huge value. > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Friday, March 29, 2019 1:04 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > Regards, > Michal > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > Hi Michal, > > > > Thanks for all the insight. > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Thursday, March 28, 2019 12:35 AM > > To: Huan Le (huanle) > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Huan, > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > Regards, > > Michal > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > Hello, > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > Thank you in advance for your helps. > > > Best, > > > Huan > > > > > > > > > From hesong at cisco.com Mon Apr 1 06:57:54 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Mon, 1 Apr 2019 06:57:54 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <577c-5ca1b500-9-12b8f000@181427425> References: <577c-5ca1b500-9-12b8f000@181427425> Message-ID: Hi Michal: Thanks again. The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Sunday, March 31, 2019 11:52 PM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, firstly, I am not part of NMDA working group or any other for that matter :) I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? Regards, Michal On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > Hi Michal? > > Thanks for all the insights. > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > IMHO, an implementation that allows such flexibility seems have huge value. > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Friday, March 29, 2019 1:04 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > Regards, > Michal > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > Hi Michal, > > > > Thanks for all the insight. > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Thursday, March 28, 2019 12:35 AM > > To: Huan Le (huanle) > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Huan, > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > Regards, > > Michal > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > Hello, > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > Thank you in advance for your helps. > > > Best, > > > Huan > > > > > > > > > From hesong at cisco.com Mon Apr 1 17:24:33 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Mon, 1 Apr 2019 17:24:33 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: References: <577c-5ca1b500-9-12b8f000@181427425> Message-ID: Yeah, Michal, I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) ? Thanks Henry -----Original Message----- From: Henry Song (hesong) Sent: Sunday, March 31, 2019 11:58 PM To: 'Michal Va?ko' Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Michal: Thanks again. The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Sunday, March 31, 2019 11:52 PM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, firstly, I am not part of NMDA working group or any other for that matter :) I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? Regards, Michal On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > Hi Michal? > > Thanks for all the insights. > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > IMHO, an implementation that allows such flexibility seems have huge value. > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Friday, March 29, 2019 1:04 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > Regards, > Michal > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > Hi Michal, > > > > Thanks for all the insight. > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Thursday, March 28, 2019 12:35 AM > > To: Huan Le (huanle) > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Huan, > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > Regards, > > Michal > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > Hello, > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > Thank you in advance for your helps. > > > Best, > > > Huan > > > > > > > > > From mvasko at cesnet.cz Tue Apr 2 07:23:09 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Tue, 02 Apr 2019 09:23:09 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <2245-5ca30e00-11-595ba880@158505146> Hi Henry, right, only YANG validation is performed in that case. We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? Regards, Michal On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > Yeah, Michal, > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > ?? > > Thanks > Henry > > -----Original Message----- > From: Henry Song (hesong) > Sent: Sunday, March 31, 2019 11:58 PM > To: 'Michal Va?ko' > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Michal: > > Thanks again. > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Sunday, March 31, 2019 11:52 PM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > firstly, I am not part of NMDA working group or any other for that matter :) > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > Regards, > Michal > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > Hi Michal? > > > > Thanks for all the insights. > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Friday, March 29, 2019 1:04 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > Regards, > > Michal > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > Hi Michal, > > > > > > Thanks for all the insight. > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Thursday, March 28, 2019 12:35 AM > > > To: Huan Le (huanle) > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Huan, > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > Regards, > > > Michal > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > Hello, > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > Thank you in advance for your helps. > > > > Best, > > > > Huan > > > > > > > > > > > > > > > > > > From hesong at cisco.com Tue Apr 2 07:43:27 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Tue, 2 Apr 2019 07:43:27 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <2245-5ca30e00-11-595ba880@158505146> References: <2245-5ca30e00-11-595ba880@158505146> Message-ID: Hi Michal: Thanks. What you described is certainly a good way to address it. I have seen people call it "preverify", ? The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. Thanks again for taking our requirements into consideration. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Tuesday, April 2, 2019 12:23 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, right, only YANG validation is performed in that case. We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? Regards, Michal On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > Yeah, Michal, > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > ?? > > Thanks > Henry > > -----Original Message----- > From: Henry Song (hesong) > Sent: Sunday, March 31, 2019 11:58 PM > To: 'Michal Va?ko' > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Michal: > > Thanks again. > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Sunday, March 31, 2019 11:52 PM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > firstly, I am not part of NMDA working group or any other for that matter :) > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > Regards, > Michal > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > Hi Michal? > > > > Thanks for all the insights. > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Friday, March 29, 2019 1:04 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > Regards, > > Michal > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > Hi Michal, > > > > > > Thanks for all the insight. > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Thursday, March 28, 2019 12:35 AM > > > To: Huan Le (huanle) > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Huan, > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > Regards, > > > Michal > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > Hello, > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > Thank you in advance for your helps. > > > > Best, > > > > Huan > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Tue Apr 2 07:59:56 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Tue, 02 Apr 2019 09:59:56 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <321c-5ca31680-3d-5de32e00@237708273> Hi Henry, the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. Regards, Michal On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Thanks. > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > Thanks again for taking our requirements into consideration. > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 12:23 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > right, only YANG validation is performed in that case. > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > Regards, > Michal > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > Yeah, Michal, > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > ?? > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Henry Song (hesong) > > Sent: Sunday, March 31, 2019 11:58 PM > > To: 'Michal Va?ko' > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Michal: > > > > Thanks again. > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Sunday, March 31, 2019 11:52 PM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > Regards, > > Michal > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > Hi Michal? > > > > > > Thanks for all the insights. > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Friday, March 29, 2019 1:04 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > Regards, > > > Michal > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal, > > > > > > > > Thanks for all the insight. > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > To: Huan Le (huanle) > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Huan, > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > Hello, > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > Thank you in advance for your helps. > > > > > Best, > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Tue Apr 2 18:16:42 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Tue, 2 Apr 2019 18:16:42 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <321c-5ca31680-3d-5de32e00@237708273> References: <321c-5ca31680-3d-5de32e00@237708273> Message-ID: Hi Michal: Understood all the points. A side note, can you please elaborate more on this: "Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway" If interface got disconnected between VERIFY and APPLY, how "operational datastore" help? Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Tuesday, April 2, 2019 1:00 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. Regards, Michal On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Thanks. > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > Thanks again for taking our requirements into consideration. > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 12:23 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > right, only YANG validation is performed in that case. > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > Regards, > Michal > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > Yeah, Michal, > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > ?? > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Henry Song (hesong) > > Sent: Sunday, March 31, 2019 11:58 PM > > To: 'Michal Va?ko' > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Michal: > > > > Thanks again. > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Sunday, March 31, 2019 11:52 PM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > Regards, > > Michal > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > Hi Michal? > > > > > > Thanks for all the insights. > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Friday, March 29, 2019 1:04 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > Regards, > > > Michal > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal, > > > > > > > > Thanks for all the insight. > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > To: Huan Le (huanle) > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Huan, > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > Hello, > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > Thank you in advance for your helps. > > > > > Best, > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From huanle at cisco.com Tue Apr 2 18:19:03 2019 From: huanle at cisco.com (Huan Le (huanle)) Date: Tue, 2 Apr 2019 18:19:03 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <321c-5ca31680-3d-5de32e00@237708273> References: <321c-5ca31680-3d-5de32e00@237708273> Message-ID: <3B004CFB-3E7B-4D26-BDBC-14F9BB1E874C@cisco.com> Thank you Michal and Henry. @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? Best, Huan ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: Hi Henry, the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. Regards, Michal On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Thanks. > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > Thanks again for taking our requirements into consideration. > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 12:23 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > right, only YANG validation is performed in that case. > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > Regards, > Michal > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > Yeah, Michal, > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > ?? > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Henry Song (hesong) > > Sent: Sunday, March 31, 2019 11:58 PM > > To: 'Michal Va?ko' > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Michal: > > > > Thanks again. > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Sunday, March 31, 2019 11:52 PM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > Regards, > > Michal > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > Hi Michal? > > > > > > Thanks for all the insights. > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Friday, March 29, 2019 1:04 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > Regards, > > > Michal > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal, > > > > > > > > Thanks for all the insight. > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > To: Huan Le (huanle) > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Huan, > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > Hello, > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > Thank you in advance for your helps. > > > > > Best, > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Tue Apr 2 22:26:58 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Tue, 2 Apr 2019 22:26:58 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <3B004CFB-3E7B-4D26-BDBC-14F9BB1E874C@cisco.com> References: <321c-5ca31680-3d-5de32e00@237708273> <3B004CFB-3E7B-4D26-BDBC-14F9BB1E874C@cisco.com> Message-ID: I would believe MOCK_CHANGE is on candidate datastore. Thanks Henry -----Original Message----- From: Huan Le (huanle) Sent: Tuesday, April 2, 2019 11:19 AM To: Michal Va?ko ; Henry Song (hesong) Cc: sysrepo-devel at sysrepo.org Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question Thank you Michal and Henry. @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? Best, Huan ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: Hi Henry, the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. Regards, Michal On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Thanks. > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > Thanks again for taking our requirements into consideration. > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 12:23 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > right, only YANG validation is performed in that case. > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > Regards, > Michal > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > Yeah, Michal, > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > ?? > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Henry Song (hesong) > > Sent: Sunday, March 31, 2019 11:58 PM > > To: 'Michal Va?ko' > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Michal: > > > > Thanks again. > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Sunday, March 31, 2019 11:52 PM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > Regards, > > Michal > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > Hi Michal? > > > > > > Thanks for all the insights. > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Friday, March 29, 2019 1:04 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > Regards, > > > Michal > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal, > > > > > > > > Thanks for all the insight. > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > To: Huan Le (huanle) > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Huan, > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > Hello, > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > Thank you in advance for your helps. > > > > > Best, > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Wed Apr 3 06:55:06 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 03 Apr 2019 08:55:06 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <796f-5ca45900-1d-73aa8880@528753> Hi Henry, what I meant is that with operational datastore the whole verify-then-apply principle should be abandoned to avoid data races such as the one I mentioned. In cases such as the example one (having specific constraints inexpressible in YANG) the ideal plugin should probably do the following. It should first subscribe for providing operational data for the whole module (or at least the configuration nodes that have these constraints and may or may not be actually in use. Then, when a configuration change occurs, they can safely ignore SR_EV_EVENT and react only to SR_EV_DONE (there is even a subscription flag to skip the first event). And, if the new configuration cannot be applied, the plugin will simply not apply it. It is fine, because when then a user issues some sr_get*() function on operational datastore, this new configuration will not appear there which means it is not in use. Hence consistency is kept and the whole problem in the current sysrepo is avoided. What is more, if the configuration can be applied at a later time, it is possible to apply the configuration and from then on it will appear in the operational datastore. I hope you understand now what I mean. Regards, Michal On Tuesday, April 2, 2019 20:16 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Understood all the points. > > A side note, can you please elaborate more on this: "Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway" > > If interface got disconnected between VERIFY and APPLY, how "operational datastore" help? > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 1:00 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > Regards, > Michal > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Thanks. > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > Thanks again for taking our requirements into consideration. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 12:23 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > right, only YANG validation is performed in that case. > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > Regards, > > Michal > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > Yeah, Michal, > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > ?? > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Henry Song (hesong) > > > Sent: Sunday, March 31, 2019 11:58 PM > > > To: 'Michal Va?ko' > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Michal: > > > > > > Thanks again. > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Sunday, March 31, 2019 11:52 PM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > Regards, > > > Michal > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal? > > > > > > > > Thanks for all the insights. > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal, > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > To: Huan Le (huanle) > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Huan, > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > Hello, > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > Best, > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Wed Apr 3 06:55:48 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 03 Apr 2019 08:55:48 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: <3B004CFB-3E7B-4D26-BDBC-14F9BB1E874C@cisco.com> Message-ID: <606c-5ca45900-87-47672e00@155471938> Hi Huan, yes, your understanding is correct. Regards, Michal On Tuesday, April 2, 2019 20:19 CEST, "Huan Le (huanle)" wrote: > Thank you Michal and Henry. > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > Best, > Huan > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > Hi Henry, > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > Regards, > Michal > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Thanks. > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > Thanks again for taking our requirements into consideration. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 12:23 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > right, only YANG validation is performed in that case. > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > Regards, > > Michal > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > Yeah, Michal, > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > ?? > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Henry Song (hesong) > > > Sent: Sunday, March 31, 2019 11:58 PM > > > To: 'Michal Va?ko' > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Michal: > > > > > > Thanks again. > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Sunday, March 31, 2019 11:52 PM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > Regards, > > > Michal > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal? > > > > > > > > Thanks for all the insights. > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal, > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > To: Huan Le (huanle) > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Huan, > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > Hello, > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > Best, > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Wed Apr 3 06:59:42 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 03 Apr 2019 08:59:42 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <2245-5ca45a00-73-595ba880@7927850> Hi Henry, there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. Regards, Michal On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > I would believe MOCK_CHANGE is on candidate datastore. > > Thanks > Henry > > -----Original Message----- > From: Huan Le (huanle) > Sent: Tuesday, April 2, 2019 11:19 AM > To: Michal Va?ko ; Henry Song (hesong) > Cc: sysrepo-devel at sysrepo.org > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > Thank you Michal and Henry. > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > Best, > Huan > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > Hi Henry, > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > Regards, > Michal > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Thanks. > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > Thanks again for taking our requirements into consideration. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 12:23 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > right, only YANG validation is performed in that case. > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > Regards, > > Michal > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > Yeah, Michal, > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > ?? > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Henry Song (hesong) > > > Sent: Sunday, March 31, 2019 11:58 PM > > > To: 'Michal Va?ko' > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Michal: > > > > > > Thanks again. > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Sunday, March 31, 2019 11:52 PM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > Regards, > > > Michal > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal? > > > > > > > > Thanks for all the insights. > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal, > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > To: Huan Le (huanle) > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Huan, > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > Hello, > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > Best, > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Wed Apr 3 07:09:05 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Wed, 3 Apr 2019 07:09:05 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <2245-5ca45a00-73-595ba880@7927850> References: <2245-5ca45a00-73-595ba880@7927850> Message-ID: Hi Michal: Removing candidate datastore concept is a quite big change, I am wondering following use case: Step 1. app1 connect and modify "running", without doing any sr_validate/sr_commit Step 2. app2 connect to running, will app2 see whatever app1 has changed in running? Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Wednesday, April 3, 2019 12:00 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. Regards, Michal On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > I would believe MOCK_CHANGE is on candidate datastore. > > Thanks > Henry > > -----Original Message----- > From: Huan Le (huanle) > Sent: Tuesday, April 2, 2019 11:19 AM > To: Michal Va?ko ; Henry Song (hesong) > Cc: sysrepo-devel at sysrepo.org > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > Thank you Michal and Henry. > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > Best, > Huan > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > Hi Henry, > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > Regards, > Michal > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Thanks. > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > Thanks again for taking our requirements into consideration. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 12:23 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > right, only YANG validation is performed in that case. > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > Regards, > > Michal > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > Yeah, Michal, > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > ?? > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Henry Song (hesong) > > > Sent: Sunday, March 31, 2019 11:58 PM > > > To: 'Michal Va?ko' > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Michal: > > > > > > Thanks again. > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Sunday, March 31, 2019 11:52 PM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > Regards, > > > Michal > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal? > > > > > > > > Thanks for all the insights. > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal, > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > To: Huan Le (huanle) > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Huan, > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > Hello, > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > Best, > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Wed Apr 3 07:18:41 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Wed, 3 Apr 2019 07:18:41 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <796f-5ca45900-1d-73aa8880@528753> References: <796f-5ca45900-1d-73aa8880@528753> Message-ID: Hi Michal: I am quite confused to be honest. 1. There are two concept: configuration data, and operational data, they in general, do not bother with each other. You can have a YANG model with configuration only, and no operational data ( config false in YANG). So are we requiring all YANG model that have constraints beyond YANG expressiveness to have operational data? 2. are we removing concept of "verify and then apply", replacing with " SR_EV_CHANGE and SR_EV_DONE"? is SR_EV_CHANGE equivalent to VERIFY? 3. if as you described EV_DONE just don't apply the config, will the config stay inside running datastore and visible to every application and hoping it can be "applied" later? And why you continue using English word "apply" when you are trying to replace it with something called "DONE"? Sorry I am quite confused at this stage. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Tuesday, April 2, 2019 11:55 PM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, what I meant is that with operational datastore the whole verify-then-apply principle should be abandoned to avoid data races such as the one I mentioned. In cases such as the example one (having specific constraints inexpressible in YANG) the ideal plugin should probably do the following. It should first subscribe for providing operational data for the whole module (or at least the configuration nodes that have these constraints and may or may not be actually in use. Then, when a configuration change occurs, they can safely ignore SR_EV_EVENT and react only to SR_EV_DONE (there is even a subscription flag to skip the first event). And, if the new configuration cannot be applied, the plugin will simply not apply it. It is fine, because when then a user issues some sr_get*() function on operational datastore, this new configuration will not appear there which means it is not in use. Hence consistency is kept and the whole problem in the current sysrepo is avoided. What is more, if the configuration can be applied at a later time, it is possible to apply the configuration and from then on it will appear in the operational datastore. I hope you understand now what I mean. Regards, Michal On Tuesday, April 2, 2019 20:16 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Understood all the points. > > A side note, can you please elaborate more on this: "Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway" > > If interface got disconnected between VERIFY and APPLY, how "operational datastore" help? > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 1:00 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > Regards, > Michal > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Thanks. > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > Thanks again for taking our requirements into consideration. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 12:23 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > right, only YANG validation is performed in that case. > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > Regards, > > Michal > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > Yeah, Michal, > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > ?? > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Henry Song (hesong) > > > Sent: Sunday, March 31, 2019 11:58 PM > > > To: 'Michal Va?ko' > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Michal: > > > > > > Thanks again. > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Sunday, March 31, 2019 11:52 PM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > Regards, > > > Michal > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal? > > > > > > > > Thanks for all the insights. > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal, > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > To: Huan Le (huanle) > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Huan, > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > Hello, > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > Best, > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Wed Apr 3 07:21:05 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 03 Apr 2019 09:21:05 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <2dc9-5ca45f00-5-5dca2680@262557473> Hi Henry, no, in your use-case app2 will not see app1 changes. It will function exactly as it does with the current sysrepo and netopeer2. As for sysrepo itself, it already caches all the changes a session makes and will apply them (and make them visible for other sessions) only once sr_commit() is issued. This is also how the candidate datastore works in the current sysrepo meaning it does not add any functionality so it was removed. Regards, Michal On Wednesday, April 3, 2019 09:09 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Removing candidate datastore concept is a quite big change, I am wondering following use case: > > Step 1. app1 connect and modify "running", without doing any sr_validate/sr_commit > Step 2. app2 connect to running, will app2 see whatever app1 has changed in running? > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Wednesday, April 3, 2019 12:00 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. > > Regards, > Michal > > On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > > > I would believe MOCK_CHANGE is on candidate datastore. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Huan Le (huanle) > > Sent: Tuesday, April 2, 2019 11:19 AM > > To: Michal Va?ko ; Henry Song (hesong) > > Cc: sysrepo-devel at sysrepo.org > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Thank you Michal and Henry. > > > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > > > Best, > > Huan > > > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > > > Hi Henry, > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > Regards, > > Michal > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > Hi Michal: > > > > > > Thanks. > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > Thanks again for taking our requirements into consideration. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > right, only YANG validation is performed in that case. > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > Regards, > > > Michal > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > Yeah, Michal, > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > ?? > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Henry Song (hesong) > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > To: 'Michal Va?ko' > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Michal: > > > > > > > > Thanks again. > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > Regards, > > > > Michal > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal? > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Henry, > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > To: Huan Le (huanle) > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > Hello, > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > Best, > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Wed Apr 3 07:27:05 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Wed, 3 Apr 2019 07:27:05 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <2dc9-5ca45f00-5-5dca2680@262557473> References: <2dc9-5ca45f00-5-5dca2680@262557473> Message-ID: Hi Michal: This is acceptable change. Now I am wondering why NMDA has "candidate datastore" concept in RFC now, ? Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Wednesday, April 3, 2019 12:21 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, no, in your use-case app2 will not see app1 changes. It will function exactly as it does with the current sysrepo and netopeer2. As for sysrepo itself, it already caches all the changes a session makes and will apply them (and make them visible for other sessions) only once sr_commit() is issued. This is also how the candidate datastore works in the current sysrepo meaning it does not add any functionality so it was removed. Regards, Michal On Wednesday, April 3, 2019 09:09 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Removing candidate datastore concept is a quite big change, I am wondering following use case: > > Step 1. app1 connect and modify "running", without doing any > sr_validate/sr_commit Step 2. app2 connect to running, will app2 see whatever app1 has changed in running? > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Wednesday, April 3, 2019 12:00 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. > > Regards, > Michal > > On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > > > I would believe MOCK_CHANGE is on candidate datastore. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Huan Le (huanle) > > Sent: Tuesday, April 2, 2019 11:19 AM > > To: Michal Va?ko ; Henry Song (hesong) > > > > Cc: sysrepo-devel at sysrepo.org > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Thank you Michal and Henry. > > > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > > > Best, > > Huan > > > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > > > Hi Henry, > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > Regards, > > Michal > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > Hi Michal: > > > > > > Thanks. > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > Thanks again for taking our requirements into consideration. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() > > question > > > > > > > > Hi Henry, > > > right, only YANG validation is performed in that case. > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > Regards, > > > Michal > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > Yeah, Michal, > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Henry Song (hesong) > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > To: 'Michal Va?ko' > > > > Cc: Huan Le (huanle) ; > > sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Michal: > > > > > > > > Thanks again. > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; > > sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > Regards, > > > > Michal > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal? > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Henry, > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > To: Huan Le (huanle) > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > Hello, > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > Best, > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Wed Apr 3 07:31:18 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Wed, 3 Apr 2019 07:31:18 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: References: <796f-5ca45900-1d-73aa8880@528753> Message-ID: Maybe a description of a most common use case can help here, ? Thanks Henry -----Original Message----- From: Henry Song (hesong) Sent: Wednesday, April 3, 2019 12:19 AM To: 'Michal Va?ko' Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Michal: I am quite confused to be honest. 1. There are two concept: configuration data, and operational data, they in general, do not bother with each other. You can have a YANG model with configuration only, and no operational data ( config false in YANG). So are we requiring all YANG model that have constraints beyond YANG expressiveness to have operational data? 2. are we removing concept of "verify and then apply", replacing with " SR_EV_CHANGE and SR_EV_DONE"? is SR_EV_CHANGE equivalent to VERIFY? 3. if as you described EV_DONE just don't apply the config, will the config stay inside running datastore and visible to every application and hoping it can be "applied" later? And why you continue using English word "apply" when you are trying to replace it with something called "DONE"? Sorry I am quite confused at this stage. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Tuesday, April 2, 2019 11:55 PM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, what I meant is that with operational datastore the whole verify-then-apply principle should be abandoned to avoid data races such as the one I mentioned. In cases such as the example one (having specific constraints inexpressible in YANG) the ideal plugin should probably do the following. It should first subscribe for providing operational data for the whole module (or at least the configuration nodes that have these constraints and may or may not be actually in use. Then, when a configuration change occurs, they can safely ignore SR_EV_EVENT and react only to SR_EV_DONE (there is even a subscription flag to skip the first event). And, if the new configuration cannot be applied, the plugin will simply not apply it. It is fine, because when then a user issues some sr_get*() function on operational datastore, this new configuration will not appear there which means it is not in use. Hence consistency is kept and the whole problem in the current sysrepo is avoided. What is more, if the configuration can be applied at a later time, it is possible to apply the configuration and from then on it will appear in the operational datastore. I hope you understand now what I mean. Regards, Michal On Tuesday, April 2, 2019 20:16 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > Understood all the points. > > A side note, can you please elaborate more on this: "Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway" > > If interface got disconnected between VERIFY and APPLY, how "operational datastore" help? > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 1:00 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > Regards, > Michal > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Thanks. > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > Thanks again for taking our requirements into consideration. > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 12:23 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > right, only YANG validation is performed in that case. > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > Regards, > > Michal > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > Yeah, Michal, > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > ?? > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Henry Song (hesong) > > > Sent: Sunday, March 31, 2019 11:58 PM > > > To: 'Michal Va?ko' > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Michal: > > > > > > Thanks again. > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Sunday, March 31, 2019 11:52 PM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > Regards, > > > Michal > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal? > > > > > > > > Thanks for all the insights. > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal, > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > To: Huan Le (huanle) > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Huan, > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > Hello, > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > Best, > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Wed Apr 3 07:35:58 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 03 Apr 2019 09:35:58 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <7701-5ca46280-6d-524ffb00@166532097> Hi Henry, firstly, I did not want to start the discussion we are having because, as you can see, the issues are quite complex and not easy to explain. Too late now, I guess. 1. I do not think your description is fitting if NMDA is supported. Operational data are no longer limited to "config false" data nodes, when I say operational I mean "data that appear in the operational datastore" [1]. 2. From the functional point of view, nothing was changed except renaming SR_EV_VERIFY to SR_EV_CHANGE and SR_EV_APPLY to SR_EV_DONE. What I was discussing was how to react to these events from a general point of view having some abstract concepts in mind. If you have a working plugin implemented, if you simply rename the events, it will continue to work. 3. This is, again, a concept from the NMDA which allows to have certain configuration in datastore and some other configuration in datastore. If there is configuration in that could not be applied (is not in-use), it will not appear in datastore. This is the main datastore change between the current sysrepo and the new one. Also, this is what I generally mean when I say that it now supports NMDA. Regards, Michal [1] https://tools.ietf.org/html/rfc8342#section-5.3 On Wednesday, April 3, 2019 09:18 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > I am quite confused to be honest. > > 1. There are two concept: configuration data, and operational data, they in general, do not bother with each other. You can have a YANG model with configuration only, and no operational data ( config false in YANG). > > So are we requiring all YANG model that have constraints beyond YANG expressiveness to have operational data? > > 2. are we removing concept of "verify and then apply", replacing with " SR_EV_CHANGE and SR_EV_DONE"? is SR_EV_CHANGE equivalent to VERIFY? > > 3. if as you described EV_DONE just don't apply the config, will the config stay inside running datastore and visible to every application and hoping it can be "applied" later? And why you continue using English word "apply" when you are trying to replace it with something called "DONE"? > > Sorry I am quite confused at this stage. > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 11:55 PM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > what I meant is that with operational datastore the whole verify-then-apply principle should be abandoned to avoid data races such as the one I mentioned. > > In cases such as the example one (having specific constraints inexpressible in YANG) the ideal plugin should probably do the following. It should first subscribe for providing operational data for the whole module (or at least the configuration nodes that have these constraints and may or may not be actually in use. Then, when a configuration change occurs, they can safely ignore SR_EV_EVENT and react only to SR_EV_DONE (there is even a subscription flag to skip the first event). And, if the new configuration cannot be applied, the plugin will simply not apply it. It is fine, because when then a user issues some sr_get*() function on operational datastore, this new configuration will not appear there which means it is not in use. Hence consistency is kept and the whole problem in the current sysrepo is avoided. What is more, if the configuration can be applied at a later time, it is possible to apply the configuration and from then on it will appear in the operational datastore. > > I hope you understand now what I mean. > > Regards, > Michal > > On Tuesday, April 2, 2019 20:16 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Understood all the points. > > > > A side note, can you please elaborate more on this: "Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway" > > > > If interface got disconnected between VERIFY and APPLY, how "operational datastore" help? > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 1:00 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > Regards, > > Michal > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > Hi Michal: > > > > > > Thanks. > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > Thanks again for taking our requirements into consideration. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > right, only YANG validation is performed in that case. > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > Regards, > > > Michal > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > Yeah, Michal, > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > ?? > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Henry Song (hesong) > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > To: 'Michal Va?ko' > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Michal: > > > > > > > > Thanks again. > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > Regards, > > > > Michal > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal? > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Henry, > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > To: Huan Le (huanle) > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > Hello, > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > Best, > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Wed Apr 3 07:41:28 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 03 Apr 2019 09:41:28 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <7701-5ca46380-6f-524ffb00@166532130> Hi Henry, the concept makes sense from the NETCONF point of view, the whole all session changes caching is only a sysrepo-specific mechanism. It enables, for example, to prepare changes for as well. Regards, Michal On Wednesday, April 3, 2019 09:27 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > This is acceptable change. > > Now I am wondering why NMDA has "candidate datastore" concept in RFC now, ?? > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Wednesday, April 3, 2019 12:21 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > no, in your use-case app2 will not see app1 changes. It will function exactly as it does with the current sysrepo and netopeer2. As for sysrepo itself, it already caches all the changes a session makes and will apply them (and make them visible for other sessions) only once sr_commit() is issued. This is also how the candidate datastore works in the current sysrepo meaning it does not add any functionality so it was removed. > > Regards, > Michal > > On Wednesday, April 3, 2019 09:09 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Removing candidate datastore concept is a quite big change, I am wondering following use case: > > > > Step 1. app1 connect and modify "running", without doing any > > sr_validate/sr_commit Step 2. app2 connect to running, will app2 see whatever app1 has changed in running? > > > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Wednesday, April 3, 2019 12:00 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. > > > > Regards, > > Michal > > > > On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > > > > > I would believe MOCK_CHANGE is on candidate datastore. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Huan Le (huanle) > > > Sent: Tuesday, April 2, 2019 11:19 AM > > > To: Michal Va?ko ; Henry Song (hesong) > > > > > > Cc: sysrepo-devel at sysrepo.org > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Thank you Michal and Henry. > > > > > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > > > > > Best, > > > Huan > > > > > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > > > > > Hi Henry, > > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > > > Regards, > > > Michal > > > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal: > > > > > > > > Thanks. > > > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", > > > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > > > Thanks again for taking our requirements into consideration. > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() > > > question > > > > > > > > > > > Hi Henry, > > > > right, only YANG validation is performed in that case. > > > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > > > Regards, > > > > Michal > > > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > > > Yeah, Michal, > > > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Henry Song (hesong) > > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > > To: 'Michal Va?ko' > > > > > Cc: Huan Le (huanle) ; > > > sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Michal: > > > > > > > > > > Thanks again. > > > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; > > > sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Henry, > > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > Hi Michal? > > > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. > > > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > > To: Henry Song (hesong) > > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Henry, > > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > Henry > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Michal Va?ko > > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > > To: Huan Le (huanle) > > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > > > Regards, > > > > > > > Michal > > > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > > Best, > > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Wed Apr 3 07:55:56 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Wed, 3 Apr 2019 07:55:56 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <7701-5ca46280-6d-524ffb00@166532097> References: <7701-5ca46280-6d-524ffb00@166532097> Message-ID: Hi Michal: Thanks for fixing me on #1, ( I missed it , need to go back and read MNDA again ). #2 is great ! #3, if I understand correctly, sysrepo will have a "running datastore ( which contains whatever user changed) " and a "operational datastore ( which contain whatever used changed as well as approved by the application callback via SR_EV_CHANGE )"? and sysrepo now support NMDA, ? Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Wednesday, April 3, 2019 12:36 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, firstly, I did not want to start the discussion we are having because, as you can see, the issues are quite complex and not easy to explain. Too late now, I guess. 1. I do not think your description is fitting if NMDA is supported. Operational data are no longer limited to "config false" data nodes, when I say operational I mean "data that appear in the operational datastore" [1]. 2. From the functional point of view, nothing was changed except renaming SR_EV_VERIFY to SR_EV_CHANGE and SR_EV_APPLY to SR_EV_DONE. What I was discussing was how to react to these events from a general point of view having some abstract concepts in mind. If you have a working plugin implemented, if you simply rename the events, it will continue to work. 3. This is, again, a concept from the NMDA which allows to have certain configuration in datastore and some other configuration in datastore. If there is configuration in that could not be applied (is not in-use), it will not appear in datastore. This is the main datastore change between the current sysrepo and the new one. Also, this is what I generally mean when I say that it now supports NMDA. Regards, Michal [1] https://tools.ietf.org/html/rfc8342#section-5.3 On Wednesday, April 3, 2019 09:18 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > I am quite confused to be honest. > > 1. There are two concept: configuration data, and operational data, they in general, do not bother with each other. You can have a YANG model with configuration only, and no operational data ( config false in YANG). > > So are we requiring all YANG model that have constraints beyond YANG expressiveness to have operational data? > > 2. are we removing concept of "verify and then apply", replacing with " SR_EV_CHANGE and SR_EV_DONE"? is SR_EV_CHANGE equivalent to VERIFY? > > 3. if as you described EV_DONE just don't apply the config, will the config stay inside running datastore and visible to every application and hoping it can be "applied" later? And why you continue using English word "apply" when you are trying to replace it with something called "DONE"? > > Sorry I am quite confused at this stage. > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Tuesday, April 2, 2019 11:55 PM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > what I meant is that with operational datastore the whole verify-then-apply principle should be abandoned to avoid data races such as the one I mentioned. > > In cases such as the example one (having specific constraints inexpressible in YANG) the ideal plugin should probably do the following. It should first subscribe for providing operational data for the whole module (or at least the configuration nodes that have these constraints and may or may not be actually in use. Then, when a configuration change occurs, they can safely ignore SR_EV_EVENT and react only to SR_EV_DONE (there is even a subscription flag to skip the first event). And, if the new configuration cannot be applied, the plugin will simply not apply it. It is fine, because when then a user issues some sr_get*() function on operational datastore, this new configuration will not appear there which means it is not in use. Hence consistency is kept and the whole problem in the current sysrepo is avoided. What is more, if the configuration can be applied at a later time, it is possible to apply the configuration and from then on it will appear in the operational datastore. > > I hope you understand now what I mean. > > Regards, > Michal > > On Tuesday, April 2, 2019 20:16 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Understood all the points. > > > > A side note, can you please elaborate more on this: "Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway" > > > > If interface got disconnected between VERIFY and APPLY, how "operational datastore" help? > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Tuesday, April 2, 2019 1:00 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > Regards, > > Michal > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > Hi Michal: > > > > > > Thanks. > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", ?? > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > Thanks again for taking our requirements into consideration. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > right, only YANG validation is performed in that case. > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > Regards, > > > Michal > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > Yeah, Michal, > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > ?? > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Henry Song (hesong) > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > To: 'Michal Va?ko' > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Michal: > > > > > > > > Thanks again. > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Hi Henry, > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > Regards, > > > > Michal > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal? > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ?? > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Henry, > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > To: Huan Le (huanle) > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > Hello, > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > Best, > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From hesong at cisco.com Thu Apr 4 17:38:12 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Thu, 4 Apr 2019 17:38:12 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <7701-5ca46380-6f-524ffb00@166532130> References: <7701-5ca46380-6f-524ffb00@166532130> Message-ID: Hi Michal: I really like everything for next version of sysrepo, when can we have it? ? Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Wednesday, April 3, 2019 12:41 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, the concept makes sense from the NETCONF point of view, the whole all session changes caching is only a sysrepo-specific mechanism. It enables, for example, to prepare changes for as well. Regards, Michal On Wednesday, April 3, 2019 09:27 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > This is acceptable change. > > Now I am wondering why NMDA has "candidate datastore" concept in RFC > now, > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Wednesday, April 3, 2019 12:21 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > no, in your use-case app2 will not see app1 changes. It will function exactly as it does with the current sysrepo and netopeer2. As for sysrepo itself, it already caches all the changes a session makes and will apply them (and make them visible for other sessions) only once sr_commit() is issued. This is also how the candidate datastore works in the current sysrepo meaning it does not add any functionality so it was removed. > > Regards, > Michal > > On Wednesday, April 3, 2019 09:09 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > Removing candidate datastore concept is a quite big change, I am wondering following use case: > > > > Step 1. app1 connect and modify "running", without doing any > > sr_validate/sr_commit Step 2. app2 connect to running, will app2 see whatever app1 has changed in running? > > > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Wednesday, April 3, 2019 12:00 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. > > > > Regards, > > Michal > > > > On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > > > > > I would believe MOCK_CHANGE is on candidate datastore. > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Huan Le (huanle) > > > Sent: Tuesday, April 2, 2019 11:19 AM > > > To: Michal Va?ko ; Henry Song (hesong) > > > > > > Cc: sysrepo-devel at sysrepo.org > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Thank you Michal and Henry. > > > > > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > > > > > Best, > > > Huan > > > > > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > > > > > Hi Henry, > > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > > > Regards, > > > Michal > > > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > > > Hi Michal: > > > > > > > > Thanks. > > > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", > > > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > > > Thanks again for taking our requirements into consideration. > > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Michal Va?ko > > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > > To: Henry Song (hesong) > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() > > > question > > > > > > > > > > > Hi Henry, > > > > right, only YANG validation is performed in that case. > > > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > > > Regards, > > > > Michal > > > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > > > Yeah, Michal, > > > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Henry Song (hesong) > > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > > To: 'Michal Va?ko' > > > > > Cc: Huan Le (huanle) ; > > > sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Michal: > > > > > > > > > > Thanks again. > > > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; > > > sysrepo-devel at sysrepo.org > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > Hi Henry, > > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > Hi Michal? > > > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. > > > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > > To: Henry Song (hesong) > > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Henry, > > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > Henry > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Michal Va?ko > > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > > To: Huan Le (huanle) > > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > > > Regards, > > > > > > > Michal > > > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > > Best, > > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mvasko at cesnet.cz Fri Apr 5 07:09:38 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Fri, 05 Apr 2019 09:09:38 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <2427-5ca6ff00-33-4f8ad900@96776527> Hi Henry, strictly speaking, you can already have it [1]. However, if you decide to look at it, there are several important thing to keep in mind. This is only sysrepo, there is no netopeer2 (NETCONF interface) that would work with this sysrepo yet, so you are limited to working with it locally. Then, there is little documentation (currently working on it) and also stability is a problem as, at this point, I am still the only one that have ever used the new sysrepo (but some other guys are also starting to use it as we speak). Other than that all the features (of the current sysrepo) are implemented and there are even several more. Naturally, I also have some TODO list (with the SR_EV_MOCK_CHANGE event, for example) but that is not a priority right now. Let me know if you decide to actually try it and encounter any problems. Regards, Michal [1] https://github.com/michalvasko/sysrepo/tree/sysrepo2 On Thursday, April 4, 2019 19:38 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > I really like everything for next version of sysrepo, when can we have it? ?? > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Wednesday, April 3, 2019 12:41 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > the concept makes sense from the NETCONF point of view, the whole all session changes caching is only a sysrepo-specific mechanism. It enables, for example, to prepare changes for as well. > > Regards, > Michal > > On Wednesday, April 3, 2019 09:27 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > This is acceptable change. > > > > Now I am wondering why NMDA has "candidate datastore" concept in RFC > > now, > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Wednesday, April 3, 2019 12:21 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > no, in your use-case app2 will not see app1 changes. It will function exactly as it does with the current sysrepo and netopeer2. As for sysrepo itself, it already caches all the changes a session makes and will apply them (and make them visible for other sessions) only once sr_commit() is issued. This is also how the candidate datastore works in the current sysrepo meaning it does not add any functionality so it was removed. > > > > Regards, > > Michal > > > > On Wednesday, April 3, 2019 09:09 CEST, "Henry Song (hesong)" wrote: > > > > > Hi Michal: > > > > > > Removing candidate datastore concept is a quite big change, I am wondering following use case: > > > > > > Step 1. app1 connect and modify "running", without doing any > > > sr_validate/sr_commit Step 2. app2 connect to running, will app2 see whatever app1 has changed in running? > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Wednesday, April 3, 2019 12:00 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. > > > > > > Regards, > > > Michal > > > > > > On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > > > > > > > I would believe MOCK_CHANGE is on candidate datastore. > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Huan Le (huanle) > > > > Sent: Tuesday, April 2, 2019 11:19 AM > > > > To: Michal Va?ko ; Henry Song (hesong) > > > > > > > > Cc: sysrepo-devel at sysrepo.org > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > Thank you Michal and Henry. > > > > > > > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > > > > > > > Best, > > > > Huan > > > > > > > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > > > > > > > Hi Henry, > > > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal: > > > > > > > > > > Thanks. > > > > > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", > > > > > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > > > > > Thanks again for taking our requirements into consideration. > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() > > > > question > > > > > > > > > > > > > > Hi Henry, > > > > > right, only YANG validation is performed in that case. > > > > > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > > > > > Yeah, Michal, > > > > > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Henry Song (hesong) > > > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > > > To: 'Michal Va?ko' > > > > > > Cc: Huan Le (huanle) ; > > > > sysrepo-devel at sysrepo.org > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Michal: > > > > > > > > > > > > Thanks again. > > > > > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > > > To: Henry Song (hesong) > > > > > > Cc: Huan Le (huanle) ; > > > > sysrepo-devel at sysrepo.org > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Henry, > > > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > > > Hi Michal? > > > > > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. > > > > > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > Henry > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Michal Va?ko > > > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > > > To: Henry Song (hesong) > > > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > > > > Hi Henry, > > > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > > > > > Regards, > > > > > > > Michal > > > > > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > Henry > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Michal Va?ko > > > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > > > To: Huan Le (huanle) > > > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Michal > > > > > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > > > Best, > > > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From rkrejci at cesnet.cz Fri Apr 5 14:48:53 2019 From: rkrejci at cesnet.cz (=?UTF-8?B?UmFkZWsgS3JlasSNw60=?=) Date: Fri, 5 Apr 2019 16:48:53 +0200 Subject: [sysrepo-devel] libyang 1.0 release Message-ID: <0e68ea78-f9a5-52aa-3d89-711546a92f61@cesnet.cz> Hi all, I would like to announce that there is release of libyang project with version 1.0 [1] (after the initial release earlier today, we found and fixed a bug, which is the reason why the referenced release has number 2). With this release we close active development of new features on the current codebase. We are currently working on libyang2 branch to rewrite problematic parts. The main issue, we see, is maintainability of the current codebase. We were trying to bring so many features for very different purposes, which are often opposite, that certain parts of the code are quite complex and it is more and more challenging to maintain or extend it. We still plan to maintain this stable version 1.0 for at least 1 year, we just stop adding new features there to have more time for the development of libyang2. Best regards, Radek Krejci [1] - https://github.com/CESNET/libyang/releases From hesong at cisco.com Fri Apr 5 19:40:55 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Fri, 5 Apr 2019 19:40:55 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <2427-5ca6ff00-33-4f8ad900@96776527> References: <2427-5ca6ff00-33-4f8ad900@96776527> Message-ID: Thanks Michal. Will keep you posted if we start trying it. Thanks again. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Friday, April 5, 2019 12:10 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, strictly speaking, you can already have it [1]. However, if you decide to look at it, there are several important thing to keep in mind. This is only sysrepo, there is no netopeer2 (NETCONF interface) that would work with this sysrepo yet, so you are limited to working with it locally. Then, there is little documentation (currently working on it) and also stability is a problem as, at this point, I am still the only one that have ever used the new sysrepo (but some other guys are also starting to use it as we speak). Other than that all the features (of the current sysrepo) are implemented and there are even several more. Naturally, I also have some TODO list (with the SR_EV_MOCK_CHANGE event, for example) but that is not a priority right now. Let me know if you decide to actually try it and encounter any problems. Regards, Michal [1] https://github.com/michalvasko/sysrepo/tree/sysrepo2 On Thursday, April 4, 2019 19:38 CEST, "Henry Song (hesong)" wrote: > Hi Michal: > > I really like everything for next version of sysrepo, when can we have > it? > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Wednesday, April 3, 2019 12:41 AM > To: Henry Song (hesong) > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Henry, > the concept makes sense from the NETCONF point of view, the whole all session changes caching is only a sysrepo-specific mechanism. It enables, for example, to prepare changes for as well. > > Regards, > Michal > > On Wednesday, April 3, 2019 09:27 CEST, "Henry Song (hesong)" wrote: > > > Hi Michal: > > > > This is acceptable change. > > > > Now I am wondering why NMDA has "candidate datastore" concept in RFC > > now, > > > > Thanks > > Henry > > > > -----Original Message----- > > From: Michal Va?ko > > Sent: Wednesday, April 3, 2019 12:21 AM > > To: Henry Song (hesong) > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > Hi Henry, > > no, in your use-case app2 will not see app1 changes. It will function exactly as it does with the current sysrepo and netopeer2. As for sysrepo itself, it already caches all the changes a session makes and will apply them (and make them visible for other sessions) only once sr_commit() is issued. This is also how the candidate datastore works in the current sysrepo meaning it does not add any functionality so it was removed. > > > > Regards, > > Michal > > > > On Wednesday, April 3, 2019 09:09 CEST, "Henry Song (hesong)" wrote: > > > > > Hi Michal: > > > > > > Removing candidate datastore concept is a quite big change, I am wondering following use case: > > > > > > Step 1. app1 connect and modify "running", without doing any > > > sr_validate/sr_commit Step 2. app2 connect to running, will app2 see whatever app1 has changed in running? > > > > > > > > > Thanks > > > Henry > > > > > > -----Original Message----- > > > From: Michal Va?ko > > > Sent: Wednesday, April 3, 2019 12:00 AM > > > To: Henry Song (hesong) > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > Hi Henry, > > > there actually is no candidate datastore in the new sysrepo itself because all the (useful) functionality is already there without it. MOCK_CHANGE would be called on all sr_validate() calls on all the datastores that can be subscribed to (running and startup). Nevertheless, netopeer2 will, naturally, support the candidate datastore but will only emulate it. In effect, if you issue validate from NETCONF on the candidate datastore, SR_EV_MOCK_CHANGE callbacks set for the running datastore will be called. > > > > > > Regards, > > > Michal > > > > > > On Wednesday, April 3, 2019 00:26 CEST, "Henry Song (hesong)" wrote: > > > > > > > I would believe MOCK_CHANGE is on candidate datastore. > > > > > > > > Thanks > > > > Henry > > > > > > > > -----Original Message----- > > > > From: Huan Le (huanle) > > > > Sent: Tuesday, April 2, 2019 11:19 AM > > > > To: Michal Va?ko ; Henry Song (hesong) > > > > > > > > Cc: sysrepo-devel at sysrepo.org > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() > > > > question > > > > > > > > > > > Thank you Michal and Henry. > > > > > > > > @Michal, if SR_EV_MOCK_CHANGE on sr_validate() returned error, I assume sr_commit would still send SR_EV_CHANGE (and SR_EV_DONE if all SR_EV_CHANGE passes). And all the events are on running data-store only, right? > > > > > > > > Best, > > > > Huan > > > > > > > > ?On 4/2/19, 1:00 AM, "Michal Va?ko" wrote: > > > > > > > > Hi Henry, > > > > the suggested event SR_EV_MOCK_CHANGE would be called ONLY on sr_validate(). For sr_commit() we want to keep the current way of getting SR_EV_CHANGE first and then SR_EV_DONE (with guaranteed no changes in-between in the configuration). Naturally, nothing prevents you from doing the same work for both SR_EV_MOCK_CHANGE and SR_EV_CHANGE. > > > > > > > > The data race I mentioned is, for example, a use-case that was often discussed when the original sysrepo callback events were designed. You may want to configure a new network interface but it must be connected. Such a constraint is not expressible in YANG so you could detect that in SR_EV_VERIFY and refuse the configuration change. However, like I said, the interface could be disconnected before SR_EV_APPLY was generated, in which case you can no longer refuse the change but cannot apply it either. Hence the changes in new sysrepo but with operational datastore this is no longer a problem, anyway. > > > > > > > > Regards, > > > > Michal > > > > > > > > On Tuesday, April 2, 2019 09:43 CEST, "Henry Song (hesong)" wrote: > > > > > > > > > Hi Michal: > > > > > > > > > > Thanks. > > > > > > > > > > What you described is certainly a good way to address it. I have seen people call it "preverify", > > > > > > > > > > The backend needs to fully understand that for such callback, there is no guarantee a "sr_commit" will happen, and even in the case of a "sr_commit" happen later, we don't know what would have changed in-between, I think this is the data race condition you refereed, unless the whole datastore were locked all the time, making everything serialized. > > > > > > > > > > Thanks again for taking our requirements into consideration. > > > > > > > > > > > > Thanks > > > > > Henry > > > > > > > > > > -----Original Message----- > > > > > From: Michal Va?ko > > > > > Sent: Tuesday, April 2, 2019 12:23 AM > > > > > To: Henry Song (hesong) > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() > > > > question > > > > > > > > > > > > > > Hi Henry, > > > > > right, only YANG validation is performed in that case. > > > > > > > > > > We have thought about it and came up with an acceptable solution. Firstly, it will only work this way in new sysrepo, the current one stays the way it is, no new features will be implemented. One of the changes made was renaming the events, now there are SR_EV_CHANGE and SR_EV_DONE. The purpose was to make them completely general so that callbacks are not pushed into doing some verification first and then applying the changes because there are data races occuring in that approach. > > > > > > > > > > The solution would be to add another event SR_EV_MOCK_CHANGE that could optionally be subscribed to and this event would be generated when one would call sr_validate() on a datastore. That should cover your use-case, right? > > > > > > > > > > Regards, > > > > > Michal > > > > > > > > > > On Monday, April 1, 2019 19:24 CEST, "Henry Song (hesong)" wrote: > > > > > > > > > > > Yeah, Michal, > > > > > > > > > > > > I did the experiment, after sr_validate(), the configuration owner's subscribed verify callback was not called, it was called only after sr_commit(). > > > > > > > > > > > > So that is exactly the opposite of what I want, I think I tried the above with netconf-cli last week, this time I tried with example code (sr_set_item_example, application_change_ example) > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Henry Song (hesong) > > > > > > Sent: Sunday, March 31, 2019 11:58 PM > > > > > > To: 'Michal Va?ko' > > > > > > Cc: Huan Le (huanle) ; > > > > sysrepo-devel at sysrepo.org > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Michal: > > > > > > > > > > > > Thanks again. > > > > > > > > > > > > The way you described solved my use case perfectly. However it was not observed by my testing code, let me check further and get back to you on this. > > > > > > > > > > > > Thanks > > > > > > Henry > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michal Va?ko > > > > > > Sent: Sunday, March 31, 2019 11:52 PM > > > > > > To: Henry Song (hesong) > > > > > > Cc: Huan Le (huanle) ; > > > > sysrepo-devel at sysrepo.org > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > Hi Henry, > > > > > > firstly, I am not part of NMDA working group or any other for that matter :) > > > > > > > > > > > > I understand that validating could simplify certain use-cases as would many other features. However, every feature increases complexity of the whole code and if we indulge all requests such as yours, we would end up with a bloated code that is difficult to maintain. > > > > > > > > > > > > To the matter at hand, I asked for your specific use-case for a reason. Validating configuration without impacting the device is already possible. You simply perform the changes on the but you do not call sr_commit() but sr_validate() instead. That will validate the current with your prepared changes and then you can call sr_discard_changes() to discard them. That should cover your use-case without even needing datastore, right? > > > > > > > > > > > > Regards, > > > > > > Michal > > > > > > > > > > > > On Friday, March 29, 2019 22:31 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > > > Hi Michal? > > > > > > > > > > > > > > Thanks for all the insights. > > > > > > > > > > > > > > Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. > > > > > > > > > > > > > > last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. > > > > > > > > > > > > > > So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. > > > > > > > > > > > > > > What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. > > > > > > > > > > > > > > The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. > > > > > > > > > > > > > > IMHO, an implementation that allows such flexibility seems have huge value. > > > > > > > > > > > > > > I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > Henry > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Michal Va?ko > > > > > > > Sent: Friday, March 29, 2019 1:04 AM > > > > > > > To: Henry Song (hesong) > > > > > > > Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org > > > > > > > > > > > > Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > > > > Hi Henry, > > > > > > > I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. > > > > > > > > > > > > > > Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. > > > > > > > > > > > > > > Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. > > > > > > > > > > > > > > As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. > > > > > > > > > > > > > > I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). > > > > > > > > > > > > > > Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. > > > > > > > > > > > > > > Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. > > > > > > > > > > > > > > Regards, > > > > > > > Michal > > > > > > > > > > > > > > On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > > > > > > > > > > > > > > > Hi Michal, > > > > > > > > > > > > > > > > Thanks for all the insight. > > > > > > > > > > > > > > > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > > > > > > > > > > > > > > > > > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > > > > > > > > > > > > > > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > > > > > > > > > > > > > > > 1. brings a behavior inconsistency on treating running/startup and candidate. > > > > > > > > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > > > > > > > > > > > > > > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > > > > > > > > > > > > > > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > > > > > > > > > > > > > > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > > > > > > > > > > > > > > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > > > > > > > > > > > > > > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > > > > > > > > > > > > > > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > Henry > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Michal Va?ko > > > > > > > > Sent: Thursday, March 28, 2019 12:35 AM > > > > > > > > To: Huan Le (huanle) > > > > > > > > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > > > > > > > > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > > > > > > > > > > > > > > > > > > > > > > Hi Huan, > > > > > > > > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > > > > > > > > > > > > > > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Michal > > > > > > > > > > > > > > > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > > > > > > > > > > > > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > > > > > > > > > > > > > > > Thank you in advance for your helps. > > > > > > > > > Best, > > > > > > > > > Huan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From Kaigen.YANG at hsmoptics.com Thu Apr 25 01:55:45 2019 From: Kaigen.YANG at hsmoptics.com (YANG Kaigen) Date: Thu, 25 Apr 2019 01:55:45 +0000 Subject: [sysrepo-devel] about the subscription Message-ID: Hi, we found there have two types of asynchronous events in sysrepo, * change in the running datastore (Running datastore subscriptions), * request for state data (Data provider subscriptions). Sometimes, we need a function to initiative notify the client the change of the state data, such as the alarm, temperature etc. Whether there have some solution to implement it? Thanks a lot! Br. kaigeny -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvasko at cesnet.cz Thu Apr 25 11:26:33 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Thu, 25 Apr 2019 13:26:33 +0200 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBhYm91dCB0aGUgc3Vi?= =?utf-8?q?scription?= In-Reply-To: Message-ID: <4d8f-5cc19980-51-1cf4b920@8821554> Hi, best suitable would be notifications for this purpose. You can subscribe to them using sr_event_notif_subscribe() and ten send (generate) them using sr_event_notif_send(). Regards, Michal On Thursday, April 25, 2019 03:55 CEST, YANG Kaigen wrote: > Hi, we found there have two types of asynchronous events in sysrepo, > > * change in the running datastore (Running datastore subscriptions), > * request for state data (Data provider subscriptions). > > Sometimes, we need a function to initiative notify the client the change of the state data, such as the alarm, temperature etc. Whether there have some solution to implement it? Thanks a lot! > > > > > > Br. > kaigeny > > >