Cisco : YANG Opensource Tools for Data Modeling-driven Management

Cisco Blog >Architect & DE Discussions

This blog describes some of the opensource tools around YANG. While there exist some tools around the YANG language validation, I want to cover the bigger landscape of data modeling-driven management tools.

Data modeling-driven management is built on the idea of applying modeling languages to formally describe data sources and APIs. For some applications, this includes the ability to generate code from the models. YANG, as THE data model language of choice to create data models, is initially published in [RFC6020] as YANG 1.0, and recently updated in YANG 1.1 [RFC7950]. This new RFC does NOT obsolete YANG 1.0 [RFC6020]. What does it mean for YANG model designers: We should start using YANG 1.1 in YANG models if we need the extra YANG 1.1 capabilities. Once the YANG models are specified and implemented, a network management system (NMS) can select a particular encoding (XML or JSON) and a particular protocol (NETCONF or RESTCONF) for transport. (Yes, there are differences in capabilities between the different protocols, but that will be the topic of a future blog.) Based on the encoding and protocol selections, we can generate code, for example in Python, to access the YANG objects on the managed device, also known as the NETCONF or RESTCONF server. As we can see, the primary source of information to generate code is the set of YANG models, or YANG modules, to use the proper terminology. This is the reason why the IETF and the industry in general have been spending so much energy to specify those YANG modules as precisely as possible.

The first step in creating YANG modules is the YANG syntax validation. We have the famous pyang YANG validator, lead by Martin Bjorklund, and constantly improved by the community , especially now that the YANG 1.1 specifications [RFC 7950] are published. At the IETF, pyang is now part of the submission tool (thanks to Qin Wu and Dapeng Liu): when posting an IETF draft containing a YANG module, the YANG module language is validated. Note that no warnings or errors will prevent you from posting your draft.

In order to extract the YANG module(s) out of IETF draft for the validation, Jan Medved created the xym.py, the eXtracting Yang Module Python script for IETF drafts and RFCs. Over time, Jan improved xym.py to be more in line with rfcstrip regarding the line numbers of extracted modules.

During the IETF 97 hackathon, I added two extra validators, part of the cron jobs on my web site. Next to the pyang and confdc validators, we now have yanglint (thanks to Radek Krejčí) and the commercial yangdump-pro (thanks to Andy Bierman, from Yumaworks) and. The daily cron jobs extract all YANG modules from the IETF drafts and RFCs, with the xym tool. In some cases, it would be more appropriate to write that the script attempts to extract the YANG modules. This is a good reminder for IETF draft authors: the conventions from the 'Guidelines for Authors and Reviewers of YANG Data Model Documents' RFC6087 must be followed, otherwise we can't validate your YANG modules. The results of the validation are updated daily here. From that page, we can review the four validator outcomes for each YANG module, and also look up the draft name, email the draft authors, and download the YANG module. Downloading the YANG module is useful as the most compilers provide the YANG module line number of the specific warning or error. Indeed, it might happen that in the process of inserting the YANG module in the IETF draft (which adds headers and footers) and extracting it with xym.py, the YANG module line numbers might not match exactly. As a side note, the IETF process will need to improve to accommodate the proper treatment of YANG module and code in general in IETF drafts. Finally, this page offers the 'compilation field', which is computed based on the results of the four validators, to the best of my knowledge, as there is a little bit of analytics involved here: the possible compilation results are PASSED, PASSED WITH WARNINGS, FAILED, and NOTE.

Based on the introduction of those new YANG validators, the number of YANG modules that pass compilation recently dropped. This is fine, as the quality of the YANG modules will eventually improve, as a result. The comparison of the different validators helped tremendously in finding bugs in the validators - with many solved already in the current or future release - and helped clarifying the Guidelines for Authors and Reviewers of YANG Data Model Documents RFC 6087bis document, currently in the final phase at the IETF. On a daily basis, I contact the authors of the newly posted draft, and provide pointers to the YANG compilation errors, if any.

Note that the cron jobs do not only focus on improving the IETF YANG modules, but also cover the validation of IEEE, BBF, MEF, ONF, ODL, openconfig and sysrepo YANG modules, as you can see here. Finally, from the same page, you can download all the IETF YANG modules.

If you want to directly test your YANG modules or IETF drafts in an online tool, before posting them, the YANG validator, developed by Carl Moberg is your friend. Right now, the YANG validation is based on pyang and confdc, but the yanglint validator will be added soon.

During one of the previous IETF hackaton, Jan Medved developed the symd.py, which generates a variety of yang module dependency graphs and output suitable for visualization with D3.js tools. A typical example, on the right, shows the importance of the ietf-routing YANG module, as it is imported by many other YANG modules. This ietf-routing YANG module is flagged as red as it's now published as a RFC [RFC8022], as opposed to green for IETF drafts. (See the list of YANG modules issued from RFCs here). The symd.py tool was later improved by Einar Nilsen-Nygaard and then by Hari Ananthakrishnan. Hari added an extra option to contact all authors of dependent YANG models, with a message « new imported YANG model updated, please update yours». This feature is especially important for the import-by-revision YANG feature: indeed, when the imported module is updated, all dependent YANG modules must be updated.

During the IETF 97 hackathon, Joe Clarke created an interesting visual dependency tool based on the output of symd. Here is a comparison with the previous figure. In this tool, you can add any number of the YANG modules (drafts or RFCs), you can select the level of recursion, and it will generate the YANG module dependencies. While the symd output was either purely textual or static pictures, this tool has more functionalities: the ability to display the IETF draft or RFC from which the YANG module is extracted by moving the mouse over the YANG module, the ability to move the YANG modules around (for example to group the YANG modules stemming from the same document), and the ability to discover the next YANG module the community should focus on (the bottleneck is highlighted with a black circle). Finally, note that the RFCs are in red and the drafts are in green. See all options under the 'graph options' pull down menu. Let's look at this 'Client/Server Configuration Model' package, which consists of those drafts, currently developed in the NETFCONF WG :

• draft-ietf-netconf-keystore-00 Keystore Model
• draft-ietf-netconf-ssh-client-server-00 SSH Client and Server Models
• draft-ietf-netconf-tls-client-server-00 TLS Client and Server Models
• draft-ietf-netconf-netconf-client-server-00 NETCONF Client and Server Models
• draft-ietf-netconf-restconf-client-server-00 RESTCONF Client and Server Models

Features:

• Upload / Compile yang models from User Interface Or Command Line
• Build NetConf RPC
• Generate Python example code [new]
• Search yang xpaths [new]
• Execute RPC against real netconf server
• Save created RPCs to collections for later use
• Build dependency graph for models
• Browse data model tree and inspect yang properties

The ultimate goal in data model-driven management is the YANG Development Kit (YDK ): a Software Development Kit that provides API's that are modeled in YANG. The main goal of YDK is to reduce the learning curve of YANG data models by expressing the model semantics in an API and abstracting protocol/encoding details. YDK is composed of a core package that defines services and providers, plus one or more module bundles that are based on YANG models. Each module bundle is generated using a bundle profile and the ydk-gen tool. YDK-Py provides Python APIs for several model bundles. Similarly, YDK-Cpp includes C++ APIs for the same bundles. There are hundreds of YDK-Py samples and YDK-Cpp samples that you can use to get started. You can find additional resources at the YDK portal.

What are the conclusions of this long blog? Data model-driven management is a reality, and there are many open source tools to support its automation. Keeping in mind that 'automation is as good as your tool chain', we need to spend some more efforts to integrate all these tools for the perfect automation - for example the integration of the yangcatalog with the YANG explorer and the YANG developement kit. And finally, I would conclude by telling that the only way to significantly reduce significantly the OPEX via automation is to focus on standard YANG modules, as the source of information in data model-driven management.

Regards, Benoît Claise