The libkea-yang library was developed to handle base YANG operations, such as retrieving YANG schema and configuration and translating data between YANG and JSON that is understandable by Kea.

Translator between YANG and JSON

An essential concept is the idea of translator. It is a primitive that is able to convert certain data structure between YANG and JSON. It is envisaged that more complex translators will use other translators to handle more complex data structures. For details, see isc::yang::Translator. It is also envisioned that translators could do the translation automatically by relying on capabilities of iterating through data, retrieving data type information and value information from nodes through the libyang and sysrepo APIs.

Note that although the initial focus is on translation from YANG to JSON (so Kea can retrieve its configuration), the opposite translation direction - from JSON to YANG - is also very useful, for at least three reasons. First, in many cases we can use it in tests to check that conversion back and forth doesn't lose anything: yang = toYang(toJson(yang)). Secondly, YANG modules cover two major types of data: configuration and run-time state. While we're initially focusing on getting the configuration, the run-time state is something that Kea is expected to provide. Kea uses JSON internally in many places and that data will have to be exported in YANG format. Thirdly, the Event::Update (SR_EV_UPDATE prior to sysrepo v2) callback allows mid-flight configuration changes before data is committed to the sysrepo datastore. If it ever will be used in the future, changes applied during this step will most likely come from Kea's current JSON configuration. As such, JSON to YANG translation will be necessary. One application for this is reverting stuff that is necessary for Kea - Sysrepo communication like the unix socket.

All translators take a Session pointer (a structure provided by Sysrepo that is responsible for maintaining a connection) in constructors and derive from the basic / base class and recursively from translators for embedded parts.

isc::yang::Translator provides several public methods:

isc::yang::Translator::checkAndGet() is able to retrieve a YANG node that can be retrieved through complex logic which is abstracted through a lambda.
isc::yang::Translator::checkAndGetLeaf() is a convenience wrapper over isc::yang::Translator::getItem().
isc::yang::Translator::checkAndGetAndJsonifyLeaf() retrieves elements that are strings in the YANG schema, but that require passing through isc::data::Element::fromJSON() when translating to ElementPtr.
isc::yang::Translator::checkAndStringifyAndSetLeaf() does the opposite of isc::yang::Translator::checkAndGetAndJsonifyLeaf(). It takes an arbitrary Element and sets it as a YANG string in sysrepo.
isc::yang::Translator::checkAndSetLeaf() is a convenience wrapper over isc::yang::Translator::setItem().
isc::yang::Translator::checkAndSetLeafList() is able to push multiple leaf list nodes to the same xpath to form a leaf list.
isc::yang::Translator::checkAndSetUserContext() is specifically tailored for setting user context in sysrepo. It's use is frequent enough to have earned its own function.
isc::yang::Translator::deleteItem() deletes the data node found at given xpath.
isc::yang::Translator::findXPath() retrieves any data node found at any xpath. It is computationally intensive. Use sparingly.
isc::yang::Translator::forAll() iterates over the node found at given xpath and all its descendants and calls the given function.
isc::yang::Translator::getData() retrieves any data node found at any xpath. The difference from isc::yang::Translator::findXPath() is that it does not throw if the data node is not found and instead returns nullopt.
isc::yang::Translator::getItem() retrieves and translates a leaf from YANG to Element.
isc::yang::Translator::getList() retrieves a list from sysrepo and translates it from YANG to Element.
isc::yang::Translator::getMandatoryLeaf() fetches a leaf that is expected to be present in the YANG data node, most of the time a YANG key.
isc::yang::Translator::schemaNodeExists() checks if an xpath is valid from the YANG schema point of view. Not used anywhere, but it's here to substitute logic that had been previously removed.
isc::yang::Translator::setItem() translates a leaf from Element to YANG and sets it in sysrepo.
isc::yang::Translator::setMandatoryLeaf() sets a leaf that is expected to be present in the Element node, most of the time a YANG key.
isc::yang::Translator::translateFromYang(optional<DataNode>, string) translates a YANG leaf to an Element node based on YANG type. All YANG types are explicitly handled.
isc::yang::Translator::translateToYang(ConstElementPtr, LeafBaseType) translates an Element leaf to a string based on the YANG type. All YANG types are explicitly handled.

Some of these methods have a counterpart that have "Diverging" in their name. They are exceptionally used in the case where YANG xpath and Element map key are different. This facilitates identifying these diverging nodes.

Pool translator

isc::yang::TranslatorPool is the standard example of a translator for a structured value. Its constructor takes a module name: the code implements some variants to accommodate the module with shared code moved into a common private routine. When called with an unsupported module, generic methods of all structure translators throw isc::NotImplemented.

Note pools show two shortcomings in IETF modules:

option sets make to track changes nearly impossible: the only easy code is to translate the whole configuration.
prefix and start - end forms of pool ranges are both mandatory. (reported to authors' so should be fixed in the next version).

All structure translators depend on isc::yang::Translator and some of them depend on other structures, for instance isc::yang::TranslatorPool depends on isc::yang::TranslatorOptionDataList which itself, as all list translators, depends on the corresponding list item translator isc::yang::TranslatorOptionData. This multiple inheritance forms a graph with the basic and the configuration translators at the two ends. Multiple inheritance and its diamond issue are handled by C++ with the virtual inheritance: depending classes must be virtually inherited and explicitly constructed.

Subnet translator

The new thing here is the use of adaptors to move timers from subnets to pools and back.

Adapting JSON configuration

Adaptors are tools which adapt JSON complete or partial configuration before translation to YANG to ease this translation or after translation from YANG to follow the Kea syntax, for instance by adding static components which are not in the module.

Methods provided by adaptors are class methods (i.e. declared static). Specific adaptors can be derived from the isc::yang::Adaptor base class.

There are a few basic adaptors and per structure adaptors. The second category of adaptors are divided into:

from JSON to YANG adaptors or pre-processing which adapt a JSON configuration to make it acceptable by a from JSON to YANG (setXXX) translators. For a Kea module this kind of adaptors fill some required but missing fields, or only transform a configuration into a canonical form. Note for a Kea module and a configuration taken from config-get or config-write it likely does nearly nothing but the code must handle any hand written configuration so these adaptors are always applied.
from YANG to JSON adaptors or post-processing which adapt translated YANG configuration (by getXXX) to make it acceptable by a Kea server. By definition, they are not defined for Kea modules.

Running unit tests with Sysrepo

To run YANG/NETCONF/Sysrepo tests you need to compile Kea with Sysrepo support:

./configure --with-libyang --with-sysrepo --with-libyang-cpp --with-sysrepo-cpp

For details, see Section "YANG/NETCONF support" in the Kea Administrator Reference Manual: https://kea.readthedocs.io/en/latest/arm/integrations.html#yang-netconf.

You also need to install YANG modules, so the unit tests are able to retrieve, add, update and generally interact with the sysrepo information. There are several Kea modules (*.yang in src/share/yang/modules/), mostly usable in production, but one called keatest-module is only used in unit tests. To be able to run unit tests as a non-root user, which is the recommended way, make sure the sysrepo repository and /dev/shm/sr* are owned by said user. One way to prevent sporadic chown-ing is to install sysrepo and the Kea modules as non-root.

To install all the modules, run the following script:

./src/share/yang/modules/utils/reinstall.sh

If the YANG modules are already installed, they will be upgraded. To avoid any upgrade hastle, the -u flag may be passed to uninstall the modules before installing again.

./src/share/yang/modules/utils/reinstall.sh -u

Alternatively to install each module, issue the following command:

sysrepoctl -i "src/share/yang/modules/${module}.yang"

To verify that you have the schemas installed, do this:

sysrepoctl -l

Make sure that keatest-module and other necessary modules are on the list.

As DHCP modules are still being developed, if the revision has been bumped, reinstalling it will update the module automatically. Otherwise, it can be useful to uninstall them before reinstalling a more recent version:

sysrepoctl -u <module-name>

Tests use these modules which you can find in src/share/yang/modules in addition to keatest-module:

ietf-dhcpv6-server
kea-ctrl-agent
kea-dhcp-ddns
kea-dhcp4-server
kea-dhcp6-server

Those modules depend on the following modules:

ietf-inet-types
ietf-yang-types
ietf-interfaces
kea-types
kea-dhcp-types

The following modules are extracted from the IETF DHCPv6 YANG draft:

ietf-dhcpv6-client
ietf-dhcpv6-options
ietf-dhcpv6-relay
ietf-dhcpv6-types

All are available in the src/share/yang/modules directory using the <module-name>[<revision>].yang syntax for file names. src/share/yang/modules/utils provides a few utilities for developers:

check-revisions.sh which verifies if the revision in the file name and in the file content matches
check-hashes.sh which detects updates in the file content without a revision change using the SHA-256 hash of the to YIN translation. Updates hashes automatically if -a is passed to the script.
gen-revisions.sh which produces the module / revision table of the yang_revisions.h header file.
reinstall.sh which installs all the modules.

You can run this tool:

src/lib/yang/pretests/sysrepo_setup_tests

to verify that your environment is ready. If there is anything wrong, it will enumerate the problems and will suggest how to solve them.

Multi-Threading Consideration for YANG Utilities

The YANG utilities are not thread-safe. Note as they are used only in a configuration context it is not a problem, and the libyang and sysrepo libraries are multi-threaded so their APIs are thread-safe.