From marco.augello at italtel.com Tue Nov 14 08:16:08 2017 From: marco.augello at italtel.com (Augello Marco) Date: Tue, 14 Nov 2017 08:16:08 +0000 Subject: [sysrepo-devel] sysrepo performance with large datastore In-Reply-To: <82e2522ef49c4411a1e41983f8d8fbeb@ITMI01VW253.corp.dom> References: <82e2522ef49c4411a1e41983f8d8fbeb@ITMI01VW253.corp.dom> Message-ID: <41288f58591c4cf0960fb074ef2a4640@ITMI01VW253.corp.dom> Hi all, we would use netopeer 2 for Italtel device configuration in cloud oriented applications We are testing sysrepod performance for our scopes. We noticed a significant performance degradation after the creation of thousands of data nodes. We always adding data nodes without deleting the old ones. It seems that the problem depends on the generation and the handling of the persistence file. When datastore file is large, also a single add operation is slow and more data nodes we adding more slow is the operation. For example, when the datastore is empty the first add is very fast, but after 4000 data nodes added, the next add takes 7 seconds and so on. Have you notice of this degradation? Maybe is our wrong configuration (In attach there is the yang model that we are testing at this moment)? Is in your roadmap an eventually change of the persistence handling? (different from single xml file) tell me if you need any other information. Thanks in advance, Sorry for my English :) Marco Internet Email Confidentiality Footer ******************************************************************************************************************************************** La presente comunicazione, con le informazioni in essa contenute e ogni documento o file allegato, e' rivolta unicamente alla/e persona/e cui e' indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non siete i destinatari/autorizzati siete avvisati che qualsiasi azione, copia, comunicazione, divulgazione o simili basate sul contenuto di tali informazioni e' vietata e potrebbe essere contro la legge (art. 616 C.P., D.Lgs n. 196/2003 Codice in materia di protezione dei dati personali). Se avete ricevuto questa comunicazione per errore, vi preghiamo di darne immediata notizia al mittente e di distruggere il messaggio originale e ogni file allegato senza farne copia alcuna o riprodurne in alcun modo il contenuto. ***************** This e-mail and its attachments are intended for the addressee(s) only and are confidential and/or may contain legally privileged information. If you have received this message by mistake or are not one of the addressees above, you may take no action based on it, and you may not copy or show it to anyone; please reply to this e-mail and point out the error which has occurred. ******************************************************************************************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: routingManipulation.yang Type: application/octet-stream Size: 5067 bytes Desc: routingManipulation.yang URL: From mislav.novakovic at sartura.hr Wed Nov 15 10:33:00 2017 From: mislav.novakovic at sartura.hr (Mislav Novakovic) Date: Wed, 15 Nov 2017 11:33:00 +0100 Subject: [sysrepo-devel] sysrepo performance with large datastore In-Reply-To: <41288f58591c4cf0960fb074ef2a4640@ITMI01VW253.corp.dom> References: <82e2522ef49c4411a1e41983f8d8fbeb@ITMI01VW253.corp.dom> <41288f58591c4cf0960fb074ef2a4640@ITMI01VW253.corp.dom> Message-ID: Hi Marco, we are aware of the issues you are facing, the design philosophy by now was to prioritize memory usage instead of performance because Sysrepo/Netopeer2 was intended to be used on low memory devices. Currently we are making some changes to the code base in order to make performance improvements but I'm afraid the biggest bottleneck will remain, which is the communication between Sysrepo and Netopeer2. There are some options to fix this, like creating a new program which combines Sysrepo and Netopeer2, and there is interest for fixing this issue but at the moment we lack the resources for a project of this scope. There are some things you could do in order to improve performance: 1) combine NETCONF requests if possible, like add 10 list entries in one NETCONF request instead of 10 separate ones. 2) make a tmpfs for the location /etc/sysrepo Also can you make a Sysrepo feature ticket about this, there you can explain your use case more in detail. Regards, Mislav On Tue, Nov 14, 2017 at 9:16 AM, Augello Marco wrote: > Hi all, > > > > we would use netopeer 2 for Italtel device configuration in cloud oriented > applications > > > > We are testing sysrepod performance for our scopes. > > > > We noticed a significant performance degradation after the creation of > thousands of data nodes. > > We always adding data nodes without deleting the old ones. > > > > It seems that the problem depends on the generation and the handling of the > persistence file. > > When datastore file is large, also a single add operation is slow and more > data nodes we adding more slow is the operation. > > > > For example, when the datastore is empty the first add is very fast, but > after 4000 data nodes added, the next add takes 7 seconds and so on. > > > > Have you notice of this degradation? Maybe is our wrong configuration (In > attach there is the yang model that we are testing at this moment)? > > > > Is in your roadmap an eventually change of the persistence handling? > (different from single xml file) > > > > tell me if you need any other information. > > > > Thanks in advance, > > Sorry for my English J > > > > Marco > > > > > > > > Internet Email Confidentiality Footer > ******************************************************************************************************************************************** > La presente comunicazione, con le informazioni in essa contenute e ogni > documento o file allegato, e' rivolta unicamente alla/e persona/e cui e' > indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non siete > i destinatari/autorizzati siete avvisati che qualsiasi azione, copia, > comunicazione, divulgazione o simili basate sul contenuto di tali > informazioni e' vietata e potrebbe essere contro la legge (art. 616 C.P., > D.Lgs n. 196/2003 Codice in materia di protezione dei dati personali). Se > avete ricevuto questa comunicazione per errore, vi preghiamo di darne > immediata notizia al mittente e di distruggere il messaggio originale e ogni > file allegato senza farne copia alcuna o riprodurne in alcun modo il > contenuto. ***************** This e-mail and its attachments are intended > for the addressee(s) only and are confidential and/or may contain legally > privileged information. If you have received this message by mistake or are > not one of the addressees above, you may take no action based on it, and you > may not copy or show it to anyone; please reply to this e-mail and point out > the error which has occurred. > ******************************************************************************************************************************************** > > _______________________________________________ > sysrepo-devel mailing list > sysrepo-devel at sysrepo.org > http://lists.sysrepo.org/listinfo/sysrepo-devel > From marco.augello at italtel.com Thu Nov 16 14:32:29 2017 From: marco.augello at italtel.com (Augello Marco) Date: Thu, 16 Nov 2017 14:32:29 +0000 Subject: [sysrepo-devel] R: sysrepo performance with large datastore In-Reply-To: References: <82e2522ef49c4411a1e41983f8d8fbeb@ITMI01VW253.corp.dom> <41288f58591c4cf0960fb074ef2a4640@ITMI01VW253.corp.dom> Message-ID: <8a64704d50b24d7a840e1d0bd3e09f86@ITMI01VW253.corp.dom> Hi Mislav, thanks for your answer. We already tried to do a tmpfs for the persistence file of our tests, more precisely the folder /etc/sysrepo/data/, but the improvement was too poor for our scopes. When we have disabled the persistence file of our module (with some code base modification in lyd_print_fd) we found out a consistent improvement. Here some useful stats to understand the size of improvement: With persistence file enabled: First 1000 edit-config, incrementally, one by one netconf request: summary = 1000 in 00:33:19 = 0,5/s Avg: 1995 Min: 459 Max: 21571 Err: 0 (0,00%) Without persistence file enabled: First 1000 edit-config, incrementally, one by one netconf request: summary = 1000 in 00:02:58 = 5,6/s Avg: 174 Min: 129 Max: 3095 Err: 0 (0,00%) I'll make a sysrepo feature ticket with other detailed informations Thanks for all and we appreciate your works! Hi, Marco -----Messaggio originale----- Da: Mislav Novakovic [mailto:mislav.novakovic at sartura.hr] Inviato: mercoled? 15 novembre 2017 11:33 A: Augello Marco Cc: sysrepo-devel at sysrepo.org Oggetto: Re: [sysrepo-devel] sysrepo performance with large datastore Hi Marco, we are aware of the issues you are facing, the design philosophy by now was to prioritize memory usage instead of performance because Sysrepo/Netopeer2 was intended to be used on low memory devices. Currently we are making some changes to the code base in order to make performance improvements but I'm afraid the biggest bottleneck will remain, which is the communication between Sysrepo and Netopeer2. There are some options to fix this, like creating a new program which combines Sysrepo and Netopeer2, and there is interest for fixing this issue but at the moment we lack the resources for a project of this scope. There are some things you could do in order to improve performance: 1) combine NETCONF requests if possible, like add 10 list entries in one NETCONF request instead of 10 separate ones. 2) make a tmpfs for the location /etc/sysrepo Also can you make a Sysrepo feature ticket about this, there you can explain your use case more in detail. Regards, Mislav On Tue, Nov 14, 2017 at 9:16 AM, Augello Marco wrote: > Hi all, > > > > we would use netopeer 2 for Italtel device configuration in cloud > oriented applications > > > > We are testing sysrepod performance for our scopes. > > > > We noticed a significant performance degradation after the creation of > thousands of data nodes. > > We always adding data nodes without deleting the old ones. > > > > It seems that the problem depends on the generation and the handling > of the persistence file. > > When datastore file is large, also a single add operation is slow and > more data nodes we adding more slow is the operation. > > > > For example, when the datastore is empty the first add is very fast, > but after 4000 data nodes added, the next add takes 7 seconds and so on. > > > > Have you notice of this degradation? Maybe is our wrong configuration > (In attach there is the yang model that we are testing at this moment)? > > > > Is in your roadmap an eventually change of the persistence handling? > (different from single xml file) > > > > tell me if you need any other information. > > > > Thanks in advance, > > Sorry for my English J > > > > Marco > > > > > > > > Internet Email Confidentiality Footer > ********************************************************************** > ********************************************************************** > La presente comunicazione, con le informazioni in essa contenute e > ogni documento o file allegato, e' rivolta unicamente alla/e persona/e cui e' > indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non > siete i destinatari/autorizzati siete avvisati che qualsiasi azione, > copia, comunicazione, divulgazione o simili basate sul contenuto di > tali informazioni e' vietata e potrebbe essere contro la legge (art. > 616 C.P., D.Lgs n. 196/2003 Codice in materia di protezione dei dati > personali). Se avete ricevuto questa comunicazione per errore, vi > preghiamo di darne immediata notizia al mittente e di distruggere il > messaggio originale e ogni file allegato senza farne copia alcuna o > riprodurne in alcun modo il contenuto. ***************** This e-mail > and its attachments are intended for the addressee(s) only and are > confidential and/or may contain legally privileged information. If you > have received this message by mistake or are not one of the addressees > above, you may take no action based on it, and you may not copy or > show it to anyone; please reply to this e-mail and point out the error which has occurred. > ********************************************************************** > ********************************************************************** > > _______________________________________________ > sysrepo-devel mailing list > sysrepo-devel at sysrepo.org > http://lists.sysrepo.org/listinfo/sysrepo-devel > Internet Email Confidentiality Footer ******************************************************************************************************************************************** La presente comunicazione, con le informazioni in essa contenute e ogni documento o file allegato, e' rivolta unicamente alla/e persona/e cui e' indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non siete i destinatari/autorizzati siete avvisati che qualsiasi azione, copia, comunicazione, divulgazione o simili basate sul contenuto di tali informazioni e' vietata e potrebbe essere contro la legge (art. 616 C.P., D.Lgs n. 196/2003 Codice in materia di protezione dei dati personali). Se avete ricevuto questa comunicazione per errore, vi preghiamo di darne immediata notizia al mittente e di distruggere il messaggio originale e ogni file allegato senza farne copia alcuna o riprodurne in alcun modo il contenuto. ***************** This e-mail and its attachments are intended for the addressee(s) only and are confidential and/or may contain legally privileged information. If you have received this message by mistake or are not one of the addressees above, you may take no action based on it, and you may not copy or show it to anyone; please reply to this e-mail and point out the error which has occurred. ******************************************************************************************************************************************** From liuai_finch at 163.com Tue Nov 28 12:14:28 2017 From: liuai_finch at 163.com (liuai) Date: Tue, 28 Nov 2017 20:14:28 +0800 (CST) Subject: [sysrepo-devel] How to install yang with different revision? Message-ID: <5bbd11b0.14e86.160028cc667.Coremail.liuai_finch@163.com> Hi All, I have a question about import yang with different revision. root at 5fed04cb28dc:/# sysrepoctl -i -g test-system.yang Installing a new module from file 'test-system.yang'... Installing the YANG file to '/etc/sysrepo/yang/test-system at 2016-06-01.yang'... Installing data files for module 'test-system'... Notifying sysrepo about the change... Install operation completed successfully. root at 5fed04cb28dc:/# root at 5fed04cb28dc:/# sysrepoctl -i -ig test-system1.yang Installing a new module from file 'test-system1.yang'... Installing the YANG file to '/etc/sysrepo/yang/test-system at 2017-08-01.yang'... [ERR] Module 'test-system' is already implemented in revision '2016-06-01'. Error: Unable to insert the module into the dependency graph. Reverting the install operation... Deleted the schema file '/etc/sysrepo/yang/test-system at 2017-08-01.yang'. Install operation failed. root at 5fed04cb28dc:/# Does sysrepo support different revision for yang? Thanks, AiPing -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvasko at cesnet.cz Tue Nov 28 13:19:20 2017 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Tue, 28 Nov 2017 14:19:20 +0100 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBIb3cgdG8gaW5zdGFs?= =?utf-8?q?l_yang_with_different_revision=3F?= In-Reply-To: <5bbd11b0.14e86.160028cc667.Coremail.liuai_finch@163.com> Message-ID: <1d72-5a1d6280-73-15c76640@215987219> Hello, sysrepo supports more revisions of one module, but only one can always be implemented (installed) as per [1]! So, using sysrepoctl install only modules that you actually want to be implemented, imported modules get loaded automatically. You can see the current state of each module by executing 'sysrepoctl -l'. Regards, Michal [1] https://tools.ietf.org/html/rfc7950#section-5.6.5 On Tuesday, November 28, 2017 13:14 CET, liuai wrote: > Hi All, > > > I have a question about import yang with different revision. > > > root at 5fed04cb28dc:/# sysrepoctl -i -g test-system.yang > Installing a new module from file 'test-system.yang'... > Installing the YANG file to '/etc/sysrepo/yang/test-system at 2016-06-01.yang'... > Installing data files for module 'test-system'... > Notifying sysrepo about the change... > Install operation completed successfully. > root at 5fed04cb28dc:/# > root at 5fed04cb28dc:/# sysrepoctl -i -ig test-system1.yang > Installing a new module from file 'test-system1.yang'... > Installing the YANG file to '/etc/sysrepo/yang/test-system at 2017-08-01.yang'... > [ERR] Module 'test-system' is already implemented in revision '2016-06-01'. > Error: Unable to insert the module into the dependency graph. > Reverting the install operation... > Deleted the schema file '/etc/sysrepo/yang/test-system at 2017-08-01.yang'. > Install operation failed. > root at 5fed04cb28dc:/# > > > Does sysrepo support different revision for yang? > > > Thanks, > AiPing From mvasko at cesnet.cz Wed Nov 29 07:27:55 2017 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 29 Nov 2017 08:27:55 +0100 Subject: [sysrepo-devel] =?utf-8?b?Pz09P3V0Zi04P3E/ICBIb3cgdG8gaW5zdGFs?= =?utf-8?q?l_yang_with_different_revision=3F?= In-Reply-To: <791aedf2.5d27.16005c31ce1.Coremail.liuai_finch@163.com> Message-ID: <7024-5a1e6180-8d-521a0e80@82545994> Hi Aiping, I presume you do not want these 2 devices to have the same configuration. Then you have a flawed YANG module. Whenever it is reasonable to have more certain devices on one machine, the model should accommodate this and the whole device configuration should be in a list. In short, you cannot do this, intentionally. Regards, Michal On Wednesday, November 29, 2017 04:12 CET, liuai wrote: > Thanks. Another question, i have two devices that use same yang module, how can i use sysrepo as the datastore to store two devices's data? > > > Thanks, > Aiping > At 2017-11-28 21:19:20, "Michal Va?ko" wrote: > >Hello, > >sysrepo supports more revisions of one module, but only one can always be implemented (installed) as per [1]! So, using sysrepoctl install only modules that you actually want to be implemented, imported modules get loaded automatically. You can see the current state of each module by executing 'sysrepoctl -l'. > > > >Regards, > >Michal > > > >[1] https://tools.ietf.org/html/rfc7950#section-5.6.5 > > > >On Tuesday, November 28, 2017 13:14 CET, liuai wrote: > > > >> Hi All, > >> > >> > >> I have a question about import yang with different revision. > >> > >> > >> root at 5fed04cb28dc:/# sysrepoctl -i -g test-system.yang > >> Installing a new module from file 'test-system.yang'... > >> Installing the YANG file to '/etc/sysrepo/yang/test-system at 2016-06-01.yang'... > >> Installing data files for module 'test-system'... > >> Notifying sysrepo about the change... > >> Install operation completed successfully. > >> root at 5fed04cb28dc:/# > >> root at 5fed04cb28dc:/# sysrepoctl -i -ig test-system1.yang > >> Installing a new module from file 'test-system1.yang'... > >> Installing the YANG file to '/etc/sysrepo/yang/test-system at 2017-08-01.yang'... > >> [ERR] Module 'test-system' is already implemented in revision '2016-06-01'. > >> Error: Unable to insert the module into the dependency graph. > >> Reverting the install operation... > >> Deleted the schema file '/etc/sysrepo/yang/test-system at 2017-08-01.yang'. > >> Install operation failed. > >> root at 5fed04cb28dc:/# > >> > >> > >> Does sysrepo support different revision for yang? > >> > >> > >> Thanks, > >> AiPing > > From liuai_finch at 163.com Wed Nov 29 03:21:00 2017 From: liuai_finch at 163.com (liuai) Date: Wed, 29 Nov 2017 11:21:00 +0800 (CST) Subject: [sysrepo-devel] How to install yang with different revision? In-Reply-To: <791aedf2.5d27.16005c31ce1.Coremail.liuai_finch@163.com> References: <1d72-5a1d6280-73-15c76640@215987219> <791aedf2.5d27.16005c31ce1.Coremail.liuai_finch@163.com> Message-ID: <1b62f2f0.60f7.16005caba84.Coremail.liuai_finch@163.com> Thanks. Another question, i have two devices that use same yang module, how can i use sysrepo as the datastore to store two devices's data? Is it possible? Thanks, Aiping At 2017-11-28 21:19:20, "Michal Va?ko" wrote: >Hello, >sysrepo supports more revisions of one module, but only one can always be implemented (installed) as per [1]! So, using sysrepoctl install only modules that you actually want to be implemented, imported modules get loaded automatically. You can see the current state of each module by executing 'sysrepoctl -l'. > >Regards, >Michal > >[1] https://tools.ietf.org/html/rfc7950#section-5.6.5 > >On Tuesday, November 28, 2017 13:14 CET, liuai wrote: > >> Hi All, >> >> >> I have a question about import yang with different revision. >> >> >> root at 5fed04cb28dc:/# sysrepoctl -i -g test-system.yang >> Installing a new module from file 'test-system.yang'... >> Installing the YANG file to '/etc/sysrepo/yang/test-system at 2016-06-01.yang'... >> Installing data files for module 'test-system'... >> Notifying sysrepo about the change... >> Install operation completed successfully. >> root at 5fed04cb28dc:/# >> root at 5fed04cb28dc:/# sysrepoctl -i -ig test-system1.yang >> Installing a new module from file 'test-system1.yang'... >> Installing the YANG file to '/etc/sysrepo/yang/test-system at 2017-08-01.yang'... >> [ERR] Module 'test-system' is already implemented in revision '2016-06-01'. >> Error: Unable to insert the module into the dependency graph. >> Reverting the install operation... >> Deleted the schema file '/etc/sysrepo/yang/test-system at 2017-08-01.yang'. >> Install operation failed. >> root at 5fed04cb28dc:/# >> >> >> Does sysrepo support different revision for yang? >> >> >> Thanks, >> AiPing > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mvasko at cesnet.cz Wed Nov 29 08:47:33 2017 From: mvasko at cesnet.cz (=?utf-8?q?Michal_Va=C5=A1ko?=) Date: Wed, 29 Nov 2017 09:47:33 +0100 Subject: [sysrepo-devel] =?utf-8?q?sysrepo_release_0=2E7=2E2?= Message-ID: <1d8e-5a1e7400-6b-1a798ca0@248047002> Hello everyone, we are planning to release a new sysrepo version 0.7.2 in the next several days. There are no major changes, the release will include mostly bugfixes and minor adjustments based on libyang API changes. Naturally, new libyang and consequently also libnetconf2 and netopeer2 will also be released to keep the compatibility. Kind regards, Michal