--- title: From docs to Schema description: 'This article was originally featured on Chad Smith’s blog Is there an echo in here? When looking through cloud-config modules it seemed there was a lot of boilerplate documentation and logic in each module to document and validate accepted configuration keys for the module. Houston, we have a problem Problem 1: Doc rot Cloud-init has […]' url: https://canonical-com-2950.demos.haus/blog/from-docs-to-schema?format=md --- 1. [Blog](https://canonical-com-2950.demos.haus/blog) 2. Article --- [Canonical](https://canonical-com-2950.demos.haus/blog/author/canonical "More about Canonical") 24 August 2017 # From docs to Schema --- Share the article *This article was originally featured [on Chad Smith’s blog](https://blackboxsw.github.io/)* Is there an echo in here? When looking through cloud-config modules it seemed there was a lot of boilerplate documentation and logic in each module to document and validate accepted configuration keys for the module. ## Houston, we have a problem ### Problem 1: Doc rot Cloud-init has 51 python modules which define the configuration functions for cloud-config features. Each module has a set of supported YAML configuration options which are documented @ http://cloudinit.readthedocs.io. Documentation of new configuration options need to be updated with changes to module supported options. We’re all (mostly) human, and here’s where our friend “doc rot” enters our project. It is easy to forget to update documentation to match changed features. ### Problem 2: Repetitive docs and configuration option parsing Each cloud-config module has a boilerplate reStructured text docstring describing all configuration option for the module. Most modules also check presence of a top-level configuration key before parsing or skipping a given config. This key definition could be encoded in a simple structure which can be sourced for both documentation and initial config parse. Let’s observe a DRY approach to docs and module configuration definitions. ### Problem 3: Absent config validation Most cloud-config modules do little validation on the configuration options provided to each module. While appearing flexible, the lack of validation ultimately costs the user time and clarity due to terse KeyError or ValueError tracebacks which could be better handled if more strict validation were performed. ## Solution: One schema to rule them all Performing validation using a strict declarative schema has the following benefits: * a declared schema is an **explicit API contract** between the module and the configuration making it easier to consume due to reduced ambiguity * a schema definition **improves automated testing coverage** by describing all supported options which can be exercised * **avoid stale docs** by tightly coupling documentation to our config validation * strict validation versus permissive acceptance **reduces cost of failures** by addressing invalid configuration errors exlicitly and early instead of at deployment time. * performing upfront schema validation on the entire config allows for **reporting multiple errors in one pass** instead of individually hitting them at runtime. ### Step 1: Add JSONchema definitions for each cloud-config module from which documentation can also be generated. The ntp module, which supports optional **servers** and **pools** keys, shows and easy schema which codifies each property name, type and expected format: ``` schema = { 'id': 'cc_ntp', 'name': 'NTP', 'title': 'enable and configure ntp', 'description': 'Something with ntp', 'distros': ['centos', 'ubuntu',...], 'examples': [...], 'properties': { 'ntp': { 'properties': { 'pools': { 'type': 'array', 'items': { 'type': 'string', 'format': 'hostname' }, 'uniqueItems': True, }, 'servers': { 'type': 'array', 'items': { 'type': 'string', 'format': 'hostname' }, 'uniqueItems': True, } }, 'required': [], # No required properties 'additionalProperties': False # Error on unregistered properties } } } ``` ### Step 2: Add simple helper functions to generate sphinx docs from schema dict instead of module docstrings The magic in sphinx doc generation is overriding the default module-level docstring behavior to make use of docs rendered from schema definition. This docstring generating callback needs to be added to your conf.py in the directory where you run sphinx: ``` def generate_docstring_from_schema(app, what, name, obj, options, lines): """Override module docs from schema when present.""" if what == 'module' and hasattr(obj, "schema"): del lines[:] lines.extend(get_schema_doc(obj.schema).split('\n')) def setup(app): app.connect('autodoc-process-docstring', generate_docstring_from_schema) ``` And the simple doc-generation from schema function is below: ``` SCHEMA_DOC_TMPL = """ {name} --- **Summary:** {title} {description} **Internal name:** ``{id}`` **Module frequency:** {frequency} **Supported distros:** {distros} **Config schema**: {property_doc} {examples} """ def get_schema_doc(schema): """Return reStructured text rendering the provided jsonschema. @param schema: Dict of jsonschema to render. @raise KeyError: If schema lacks an expected key. """ schema['property_doc'] = _get_property_doc(schema) schema['examples'] = _get_schema_examples(schema) schema['distros'] = ', '.join(schema['distros']) return SCHEMA_DOC_TMPL.format(**schema) ``` ### Step 3: The module handler function will iterate over schema errors with jsonschema.Validator and log collected warnings for all schema infractions. ``` from jsonschema import Draft4Validator, FormatChecker validator = Draft4Validator(schema, format_checker=FormatChecker()) for error in sorted(validator.iter_errors(config), key=lambda e: e.path): path = '.'.join([str(p) for p in error.path]) errors += ((path, error.message),) if errors: raise SchemaValidationError(errors) ``` ### Step 4: Simple cmdline tools to validate cloud-config files against known schema to avoid costly errors during instance deployment. Already included with cloud-init 0.7.9 is a minimal schema validation development tool: ``` python3 -m cloudinit.config.schema --help ``` [Get in touch Interested in running Ubuntu in your organization?](https://ubuntu.com/about/contact-us/form) ## Sign up for our newsletter Get the latest Canonical news and updates in your inbox. Work email: \*I agree to receive information about Canonical's products and services. By submitting this form, I confirm that I have read and agree to [Canonical's Privacy Policy](https://canonical-com-2950.demos.haus/legal/dataprivacy). Sign up ## Share on --- ## Related posts [### From raw data to intelligent actions: inside our next-gen AI analytics data lake stack](https://canonical-com-2950.demos.haus/blog/inside-our-next-gen-ai-analytics-data-lake-stack) At Canonical, we believe organisations should be able to unlock the full value of their data without giving up control. That principle is at the heart of our next-generation... [Rob Gibbon](https://canonical-com-2950.demos.haus/blog/author/robgibbon) 18 September 2026 [### Beyond the 10-year mark: Extending Ubuntu Pro 16.04 LTS security coverage](https://canonical-com-2950.demos.haus/blog/extending-ubuntu-pro-16-04-coverage) A decade ago, Canonical launched Ubuntu 16.04 LTS (codenamed “Xenial Xerus”). As a Long-Term Support (LTS) release, it comes with 5 years of standard security coverage, which... [Carlos Bravo](https://canonical-com-2950.demos.haus/blog/author/carlos-bravo) 17 September 2026 [### Network disaggregation in telecommunication transport networks](https://canonical-com-2950.demos.haus/blog/network-disaggregation) Telecommunications networks have traditionally been built from tightly integrated systems. A router vendor supplies the forwarding hardware, network operating system (NOS),... [Benjamin Ryzman](https://canonical-com-2950.demos.haus/blog/author/benjaminryzman) 16 September 2026 [### Bring Zenoh to ROS 2 with snaps](https://canonical-com-2950.demos.haus/blog/bring-zenoh-to-ros-2-with-snaps) ROS 2 gives robotics developers the freedom to choose the middleware that fits their system. As a communication protocol for ROS, Zenoh has gained strong traction. It delivers... [gbeuzeboc](https://canonical-com-2950.demos.haus/blog/author/gbeuzeboc) 15 September 2026