[sysrepo-devel] R: sysrepo performance with large datastore

[Augello Marco]

Hi Mislav,

thanks for your answer.
We already tried to do a tmpfs for the persistence file of our tests, more precisely the folder /etc/sysrepo/data/, but the improvement was too poor for our scopes.

When we have disabled the persistence file of our module (with some code base modification in lyd_print_fd) we found out  a consistent improvement.

Here some useful stats to understand the size of improvement:

With persistence file enabled:
First 1000 edit-config, incrementally, one by one netconf request:
summary =   1000 in 00:33:19 =    0,5/s Avg:  1995 Min:   459 Max: 21571 Err:     0 (0,00%)

Without persistence file enabled:
First 1000 edit-config, incrementally, one by one netconf request:
summary =   1000 in 00:02:58 =    5,6/s Avg:   174 Min:   129 Max:  3095 Err:     0 (0,00%)

I'll make a sysrepo feature ticket with other detailed informations

Thanks for all and we appreciate your works!

Hi,
Marco

-----Messaggio originale-----
Da: Mislav Novakovic [mailto:mislav.novakovic at sartura.hr]
Inviato: mercoledì 15 novembre 2017 11:33
A: Augello Marco
Cc: sysrepo-devel at sysrepo.org
Oggetto: Re: [sysrepo-devel] sysrepo performance with large datastore

Hi Marco,

we are aware of the issues you are facing, the design philosophy by now was to prioritize memory usage instead of performance because
Sysrepo/Netopeer2 was intended to be used on low memory devices.

Currently we are making some changes to the code base in order to make performance improvements but I'm afraid the biggest bottleneck will remain, which is the communication between Sysrepo and Netopeer2.

There are some options to fix this, like creating a new program which combines Sysrepo and Netopeer2, and there is interest for fixing this issue but at the moment we lack the resources for a project of this scope.

There are some things you could do in order to improve performance:
1) combine NETCONF requests if possible, like add 10 list entries in one NETCONF request instead of 10 separate ones.
2) make a tmpfs for the location /etc/sysrepo

Also can you make a Sysrepo feature ticket about this, there you can explain your use case more in detail.

Regards,
Mislav

On Tue, Nov 14, 2017 at 9:16 AM, Augello Marco <marco.augello at italtel.com> wrote:
> Hi all,
>
>
>
> we would use netopeer 2 for Italtel device configuration in cloud
> oriented applications
>
>
>
> We are testing sysrepod performance for our scopes.
>
>
>
> We noticed a significant performance degradation after the creation of
> thousands of data nodes.
>
> We always adding data nodes without deleting the old ones.
>
>
>
> It seems that the problem depends on the generation and the handling
> of the persistence file.
>
> When datastore file is large, also a single add operation is slow and
> more data nodes we adding more slow is the operation.
>
>
>
> For example, when the datastore is empty the first add is very fast,
> but after 4000 data nodes added, the next add takes 7 seconds and so on.
>
>
>
> Have you notice of this degradation? Maybe is our wrong configuration
> (In attach there is the yang model that we are testing at this moment)?
>
>
>
> Is in your roadmap an eventually change of the persistence handling?
> (different from single xml file)
>
>
>
> tell me if you need any other information.
>
>
>
> Thanks in advance,
>
> Sorry for my English J
>
>
>
> Marco
>
>
>
>
>
>
>
> Internet Email Confidentiality Footer
> **********************************************************************
> **********************************************************************
> La presente comunicazione, con le informazioni in essa contenute e
> ogni documento o file allegato, e' rivolta unicamente alla/e persona/e cui e'
> indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non
> siete i destinatari/autorizzati siete avvisati che qualsiasi azione,
> copia, comunicazione, divulgazione o simili basate sul contenuto di
> tali informazioni e' vietata e potrebbe essere contro la legge (art.
> 616 C.P., D.Lgs n. 196/2003 Codice in materia di protezione dei dati
> personali). Se avete ricevuto questa comunicazione per errore, vi
> preghiamo di darne immediata notizia al mittente e di distruggere il
> messaggio originale e ogni file allegato senza farne copia alcuna o
> riprodurne in alcun modo il contenuto. ***************** This e-mail
> and its attachments are intended for the addressee(s) only and are
> confidential and/or may contain legally privileged information. If you
> have received this message by mistake or are not one of the addressees
> above, you may take no action based on it, and you may not copy or
> show it to anyone; please reply to this e-mail and point out the error which has occurred.
> **********************************************************************
> **********************************************************************
>
> _______________________________________________
> sysrepo-devel mailing list
> sysrepo-devel at sysrepo.org
> http://lists.sysrepo.org/listinfo/sysrepo-devel
>
Internet Email Confidentiality Footer ******************************************************************************************************************************************** La presente comunicazione, con le informazioni in essa contenute e ogni documento o file allegato, e' rivolta unicamente alla/e persona/e cui e' indirizzata ed alle altre da questa autorizzata/e a riceverla. Se non siete i destinatari/autorizzati siete avvisati che qualsiasi azione, copia, comunicazione, divulgazione o simili basate sul contenuto di tali informazioni e' vietata e potrebbe essere contro la legge (art. 616 C.P., D.Lgs n. 196/2003 Codice in materia di protezione dei dati personali). Se avete ricevuto questa comunicazione per errore, vi preghiamo di darne immediata notizia al mittente e di distruggere il messaggio originale e ogni file allegato senza farne copia alcuna o riprodurne in alcun modo il contenuto. ***************** This e-mail and its attachments are intended for the addressee(s) only and are confidential and/or may contain legally privileged information. If you have received this message by mistake or are not one of the addressees above, you may take no action based on it, and you may not copy or show it to anyone; please reply to this e-mail and point out the error which has occurred. ********************************************************************************************************************************************