From huanle at cisco.com Wed Mar 27 18:13:49 2019 From: huanle at cisco.com (Huan Le (huanle)) Date: Wed, 27 Mar 2019 18:13:49 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question Message-ID: <5FE2D4A7-5673-4E46-9E84-D4BD9ED33A97@cisco.com> Hello, We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. Thank you in advance for your helps. Best, Huan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvasko at cesnet.cz Thu Mar 28 07:34:38 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Thu, 28 Mar 2019 08:34:38 +0100 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: <5FE2D4A7-5673-4E46-9E84-D4BD9ED33A97@cisco.com> Message-ID: <69e2-5c9c7900-17-121eb7a0@85602678> Hi Huan, as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. Regards, Michal On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > Hello, > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > Thank you in advance for your helps. > Best, > Huan From hesong at cisco.com Thu Mar 28 19:27:13 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Thu, 28 Mar 2019 19:27:13 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <69e2-5c9c7900-17-121eb7a0@85602678> References: <5FE2D4A7-5673-4E46-9E84-D4BD9ED33A97@cisco.com> <69e2-5c9c7900-17-121eb7a0@85602678> Message-ID: Hi Michal, Thanks for all the insight. netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: 1. brings a behavior inconsistency on treating running/startup and candidate. 2. application only get a chance to reject when candidate moves to running, by then it might be too late. Perhaps it is a lot better design to give application callback a chance for candidate datastore. If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). This bring the first question: verify and apply really are two separate concept and orthogonal to each other. Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Thursday, March 28, 2019 12:35 AM To: Huan Le (huanle) Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question Hi Huan, as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. Regards, Michal On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > Hello, > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > Thank you in advance for your helps. > Best, > Huan From mvasko at cesnet.cz Fri Mar 29 08:04:11 2019 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Fri, 29 Mar 2019 09:04:11 +0100 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBTZXNzaW9uLmNvbW1p?= =?utf-8?b?dCgpL3NyX2NvbW1pdCgpIHF1ZXN0aW9u?= In-Reply-To: Message-ID: <2498-5c9dd180-73-4dd1940@173729732> Hi Henry, I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. Regards, Michal On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > Hi Michal, > > Thanks for all the insight. > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > 1. brings a behavior inconsistency on treating running/startup and candidate. > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Thursday, March 28, 2019 12:35 AM > To: Huan Le (huanle) > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Huan, > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > Regards, > Michal > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > Hello, > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > Thank you in advance for your helps. > > Best, > > Huan > > > From jan.kundrat at cesnet.cz Fri Mar 29 15:10:22 2019 From: jan.kundrat at cesnet.cz (=?iso-8859-1?Q?Jan_Kundr=E1t?=) Date: Fri, 29 Mar 2019 16:10:22 +0100 Subject: [sysrepo-devel] =?iso-8859-1?q?=3F=3D=3D=3Futf-8=3Fq=3F__Session?= =?iso-8859-1?q?=2Ecommit=28=29/sr=5Fcommit=28=29_question?= In-Reply-To: <2498-5c9dd180-73-4dd1940@173729732> References: <2498-5c9dd180-73-4dd1940@173729732> Message-ID: <1fe5fa10-76f0-48e3-9ca3-c2ba4c0257c0@cesnet.cz> On p?tek 29. b?ezna 2019 9:04:11 CET, Michal Va?ko wrote: > I want to add that VERIFY verification is mainly meant for some > use-cases that are not expressible in YANG and depend only on > the device itself such as configuring a network interface that > is not connected, for instance. However, with the introduction > of NMDA, I believe the whole VERIFY callback becomes a > completely redundant feature, which may simplify applying the > configuration changes to the device but should not be strictly > needed (provided that the current sysrepo would support NMDA, of > course). Another use case for VERIFY, and a reason why VERIFY cannot actually be removed, in my opinion, is interoperability with third-party YANG Models. It's just a sad reality that plenty of standard organizatons produce YANG models, and a significant part of these models have non-trivial semantics constraints which are, unfortunately, *not* represented via XPath-ish validation via `must` statements, etc. I suspect that one of the reasons for these "out-of-band" validation constraints which are not included in the machine-readable part of YANG model are restrictions and limitations in some popular NETCONF server implementations :), but that could be just my imagination. TL;DR: VERIFY will have to stay, anyway, because users of sysrepo cannot augment their YANG models. With kind regards, Jan From hesong at cisco.com Fri Mar 29 21:31:05 2019 From: hesong at cisco.com (Henry Song (hesong)) Date: Fri, 29 Mar 2019 21:31:05 +0000 Subject: [sysrepo-devel] Session.commit()/sr_commit() question In-Reply-To: <2498-5c9dd180-73-4dd1940@173729732> References: <2498-5c9dd180-73-4dd1940@173729732> Message-ID: Hi Michal? Thanks for all the insights. Rob Wilton is my colleague and he was in NMDA working group, I am guessing you are part of NMDA working group as well. last year, we launched a project to do the "validation offline" with pure .yang constraints, it turned out to be a very hard job, for obvious reasons. ? So we still continue to validate on the real box. Now question is : should we validate candidate before pushing it to running, or validate against running. The RFC never stated clearly even with RFC8342. What I found out is that if an implementation allow the flexibility to validate against candidate, there may not be a need to validate against running, or better yet, an implementation offer both validation options for candidate and running and let use case pick. The use case: I have several configuration sets that I want to know if they can pass validation 100% on a box without impacting box, without being pushed to running. IMHO, an implementation that allows such flexibility seems have huge value. I am not sure I convinced you either way. it is my personal opinion that such flexibility would add lots of value to sysrepo. Thanks Henry -----Original Message----- From: Michal Va?ko Sent: Friday, March 29, 2019 1:04 AM To: Henry Song (hesong) Cc: Huan Le (huanle) ; sysrepo-devel at sysrepo.org Subject: RE: [sysrepo-devel] Session.commit()/sr_commit() question Hi Henry, I will try to elaborate more on the YANG concepts the way I understand them and the overall callback design of sysrepo. Generally, the core principle is that you have configuration that is the current configuration of a device. Whenever changes it means that the device configuration should also change, hence the callbacks. That is all, in a way. This is also expressed by the fact that the other datastores have their support optional based on features. Additional datastores just try to add some functionality and improve user experience. There is so that it is possible to have a separate persistent configuration that can be different from . There is also that should make it easier to make larger changes to so that they can all be applied atomically. That is the main use-case I believe. This is the purpose of these datastore, anything beyond that may or may not be supported. As for sysrepo itself, it reflects this in the fact that you can only set callbacks for changes (not possible for , why do you think so?). The 2 events for callbacks already extend the original functionality. Callbacks first process VERIFY when the they can still refuse the configuration change and it will not be written into . This allows for some additional verification of the changes by the application even though it should, in the perfect case, be completely validated by YANG constraints already. Then, after all the changes pass VERIFY, the callbacks are called with APPLY when they can no longer refuse the change and should simply apply the change to the device. That was the design. I want to add that VERIFY verification is mainly meant for some use-cases that are not expressible in YANG and depend only on the device itself such as configuring a network interface that is not connected, for instance. However, with the introduction of NMDA, I believe the whole VERIFY callback becomes a completely redundant feature, which may simplify applying the configuration changes to the device but should not be strictly needed (provided that the current sysrepo would support NMDA, of course). Finally, do you understand the datastores differently or do you think they have some other purpose? Now to react to what you wrote. Yes, there are "behavior inconsistencies" between different datastores. In other words, they behave differently since they have different purpose. Why would you want to have several datastores that behave the same way, that would be completely redundant and useless. Also what do you mean it may be too late to refuse changes? If a callback does that, the changes are not applied and does not change meaning consistency between and the device configuration is kept. Just a final thought, what and how confd does things is of no particular interest. Actually, it is not even that relevant how sysrepo does things, they are both just an implementation of the whole YANG datastore concept. We need to agree what those concepts are and whether the implementations support them or not. Regards, Michal On Thursday, March 28, 2019 20:27 CET, "Henry Song (hesong)" wrote: > Hi Michal, > > Thanks for all the insight. > > netconf RFC has a "validate" on datastore ( running, startup, candidate ), RFC does not clear say how validate should be done. > > There are different level of validate, involving a callback from the configuration owner/application is most complicated one, just do a .yang semantic check (with range and MUST key word check) is simpler without needing to talk to application. > > Clearly sysrepo allows "validate via application callback" on running and startup, but not candidate. This suggest two things: > > 1. brings a behavior inconsistency on treating running/startup and candidate. > 2. application only get a chance to reject when candidate moves to running, by then it might be too late. > > Perhaps it is a lot better design to give application callback a chance for candidate datastore. > > If we believe above, we really should call "verify (the validity)" for candidate and "apply ( to make it effective)" when candidate transit to "running" ( since already approved ). There may not be much point to do verify on "running" and "startup" ( they already verified/validated early, there is no reason to fail during different boot ). > > This bring the first question: verify and apply really are two separate concept and orthogonal to each other. > > Confd allow all the above flexibility ( including call application callback to validate a candidate datastore ): > > https://discuss.tail-f.com/t/how-to-skip-validating-netconf-edit-config-s-when-writing-to-the-candidate/302 > > perhaps at least we can have an sysrepo option to allow application verify callback on candidate datastore , that would be a small improvement that will go a long way. > > > Thanks > Henry > > -----Original Message----- > From: Michal Va?ko > Sent: Thursday, March 28, 2019 12:35 AM > To: Huan Le (huanle) > Cc: sysrepo-devel at sysrepo.org; Henry Song (hesong) > Subject: Re: [sysrepo-devel] Session.commit()/sr_commit() question > > Hi Huan, > as for your first question, VERIFY and APPLY events and tied together and cannot be sent separately. What I am wondering about is what exactly are you trying to achieve, what is your use-case? > > Regarding second question, yes, candidate changes should not trigger any callbacks. The reason is quite simple, the callbacks are supposed to be called on configuration changes so that the configured device can react to these changes. Candidate datastore is just a mechanism for preparing a configuration change, it does not affect the configuration unless it is committed (written into running) so it would be wrong to call any callbacks. > > Regards, > Michal > > On Wednesday, March 27, 2019 19:13 CET, "Huan Le (huanle)" wrote: > > > Hello, > > We understand sr_commit() operation in a running data-store session will deliver a Verify and Apply notification event to subscribed sysrepo client. Is it possible to separate the controlling action that trigger Verify and Apply operation? For example, would it be possible to add sr_verify() to deliver Verify notification of the changes in a data-store only? > > > > On a separate note, I notice (module) changes subscription in a candidate data-store does not return error. However, the callback is not invoked when a change in the candidate data-store is committed. It is invoked when a change in the running data-store is committed. Is this expected behavior? If yes, please help us understand the reasons. > > > > Thank you in advance for your helps. > > Best, > > Huan > > >