[sysrepo-devel] new generation of sysrepo

[Jan Kundrát]

Hi,
here's a bunch of comments from my part. We're using sysrepo (and the rest 
of the NETCONF stack) as an exclusive management interface to our optical 
devices.

I will start with a personal wishlist related to general SW development, 
testing and cross-compiling. I wish sysrepo supported self-contained, 
portable repository locations. That would be super-useful for unit tests. 
Each test could then run in complete isolation from other parallel 
executions. There would be no extra setup and teardown (maybe just a `cp` 
and `rm`). In other words, please allow programmers to specify a "prefix" 
or "context" when initializing the first connection, and locate *all* data 
underneath that prefix. It is OK to have a system-wide default for 
"production use", but please treat another prefix as a first-class citizen.

Lack of SR_EV_VERIFY sounds like a blocker for us. We are trying to push as 
much stuff as possible to XPath-based `must` conditions in our own models, 
but we've been hitting performance problems in the past, and the models are 
only getting more and more complex. Also, there are existing models written 
by people who put the validation logic into their NETCONF servers, i.e., 
there are rules not visible to clients via YANG. We must implement these 
models, and SR_EV_VERIFY is an important tool for that. I see in the 
updated presentation that it is probably not going to be removed, just 
renamed -- that's great. We "just" need to be able to reject a proposed 
change.

I am a bit worried by the proposed use of SHM and zero-copy, but perhaps 
I'm just not understanding the slides. The current/legacy design uses GPB, 
a library for checking whether the data read from clients pass some 
rudimentory validation. How will different clients access SHM in the 
proposed next-gen design? Will they have direct access to, e.g., shared 
data structures? If the in-memory structures contain offsets or pointers, 
which part will check whether they are valid? What happens when a malicious 
client changes the data in an attempt to trigger a race condition (time of 
check vs time of use)?

Right now, notifications leave much to be desired. Either they are one-off, 
or they are stored in an extremely inefficient format which basically 
requires an O(n^2) parsing overhead [1]. I wonder if sysrepo is actually a 
correct place for storing high-volume notifications. Perhaps looking at 
some existing message-bus system might make sense?

Ops data subscriptions -- currently there are some limitations on how the 
code can be structured and how the YANG model can look like, especially 
when it comes to lists. Our implementation requests data from some 
low-level HW modules, and batching requests provides a notable performance 
boost. Therefore I prefer a single request for all ops data for a given 
module (or perhaps even modules?) with a list of requested XPath subtrees. 
That way our low-level code can use the most efficient method for obtaining 
the data.

Changes to the startup datastore. Right now it requires a convoluted setup 
[2] where we're using `inotify`, listening for changes to the `foo.startup` 
data file. This means that we cannot reject such a modification, and that 
we have to use an external tool rather than hooking into sysrepo. Our use 
case is simple, e.g., a plugin which handles the `ietf-interfaces` model 
and pregenrates some persistent config to be applied during next reboot.

Regarding locking, I don't have much to contribute here because our changes 
have so far only targeted a single DS. That said, it "feels wrong" if the 
commits are not actually atomic. If we can end up with a transaction which 
originally touches modules A and B, and it ends up only persisteng changes 
to, say, B, then that would be a bug in my opinion.

Hope this helps,
Jan

[1] https://github.com/sysrepo/sysrepo/issues/735
[2] https://github.com/sysrepo/sysrepo/issues/954