Skip to content

GitLab

Commit 0922e92a authored by Phil Shafer's avatar Phil Shafer
Browse files

Import libxo 1.3.0

parent 9e479b34
Expand all Hide whitespace changes
Inline Side-by-side
Makefile.am
View file @ 0922e92a
......@@ -32,7 +32,6 @@ errors:
docs:
@(cd doc ; ${MAKE} docs)
DIST_FILES_DIR = ~/Dropbox/dist-files/
GH_PAGES_DIR = gh-pages/
GH_PAGES_DIR_VER = gh-pages/${PACKAGE_VERSION}
......@@ -49,18 +48,20 @@ upload: dist upload-docs upload-xohtml-files
@echo "Remember to run:"
@echo " gt tag ${PACKAGE_VERSION}"
upload-docs: docs
@echo "Uploading libxo-manual.html ... "
@-[ -d ${GH_PAGES_DIR} ] \
&& echo "Updating manual on gh-pages ..." \
&& mkdir -p ${GH_PAGES_DIR_VER} \
&& cp doc/libxo-manual.html ${GH_PAGES_DIR} \
&& cp doc/libxo-manual.html ${GH_PAGES_DIR_VER} \
upload-docs: docs upload-html
upload-html:
@echo "Uploading html ... "
@-[ -d ${GH_PAGES_DIR} -a -d doc/html ] \
&& echo "Updating html on gh-pages ..." \
&& mkdir -p ${GH_PAGES_DIR_VER}/html \
&& cp doc/top-link.html ${GH_PAGES_DIR}/libxo.html \
&& cp -r doc/html/* ${GH_PAGES_DIR_VER}/html/ \
&& (cd ${GH_PAGES_DIR} \
&& git add ${PACKAGE_VERSION} \
&& git add libxo-manual.html \
&& git add libxo.html \
&& git add ${PACKAGE_VERSION}/html \
&& git commit -m 'new docs' \
libxo-manual.html ${PACKAGE_VERSION} \
libxo.html ${PACKAGE_VERSION}/html \
&& git push origin gh-pages ) ; true
upload-xohtml-files:
......
README.md
View file @ 0922e92a
......@@ -10,6 +10,9 @@ application calls a function "xo_emit" to product output that is
described in a format string. A "field descriptor" tells libxo what
the field is and what it means.
Imagine a simplified ``wc`` that emits its output fields in a single
xo_emit call:
```
xo_emit(" {:lines/%7ju/%ju} {:words/%7ju/%ju} "
"{:characters/%7ju/%ju}{d:filename/%s}\n",
......
configure.ac
View file @ 0922e92a
......@@ -12,7 +12,7 @@
#
AC_PREREQ(2.2)
AC_INIT([libxo], [1.0.4], [phil@juniper.net])
AC_INIT([libxo], [1.3.0], [phil@juniper.net])
AM_INIT_AUTOMAKE([-Wall -Werror foreign -Wno-portability])
# Support silent build rules. Requires at least automake-1.11.
......@@ -452,6 +452,7 @@ AC_CONFIG_FILES([
libxo/add.man
encoder/Makefile
encoder/cbor/Makefile
encoder/csv/Makefile
encoder/test/Makefile
xo/Makefile
xolint/Makefile
......@@ -459,6 +460,7 @@ AC_CONFIG_FILES([
xopo/Makefile
packaging/libxo.pc
doc/Makefile
doc/top-link.html
tests/Makefile
tests/core/Makefile
tests/gettext/Makefile
......
doc/Makefile.am
View file @ 0922e92a
......@@ -8,68 +8,22 @@
# using the SOFTWARE, you agree to be bound by the terms of that
# LICENSE.
if HAVE_OXTRADOC
OXTRADOC_DIR = ${SLAX_OXTRADOCDIR}
OXTRADOC_PREFIX = ${OXTRADOC_DIR}
OXTRADOC = ${OXTRADOC_DIR}/oxtradoc
SLAXPROC_BINDIR = ${SLAX_BINDIR}
doc docs: xolint.rst html
XML2RFC = ${OXTRADOC_DIR}/xml2rfc.tcl
XML2HTMLDIR = ${OXTRADOC_DIR}
XML2HTMLBIN = ${XML2HTMLDIR}/rfc2629-to-html.slax
SLAXPROC = ${SLAX_BINDIR}/slaxproc
SLAXPROC_ARGS = \
-a oxtradoc-dir ${OXTRADOC_DIR} \
-a oxtradoc-install-dir ${OXTRADOC_DIR} \
-a anchor-prefix docs
SLAXPROC_ARGS_INLINE = \
-a oxtradoc-inline yes
SLAXPROC_ARGS += ${SLAXPROC_ARGS_INLINE}
XML2HTML = \
${SLAXPROC} -g -e -I ${OXTRADOC_DIR} -I . \
${SLAXPROC_ARGS} \
${XML2HTMLBIN}
OX_ARGS = -P ${OXTRADOC_PREFIX} -L ${OXTRADOC_PREFIX}
OX_ARGS += -S ${SLAXPROC} -p doc
OX_CMD = ${PERL} ${PERLOPTS} ${OXTRADOC} ${OX_ARGS}
OXTRADOC_CMD = ${OX_CMD}
OUTPUT = libxo-manual
INPUT = libxo
EXTRA_DIST = \
${INPUT}.txt \
${OUTPUT}.html \
${OUTPUT}.txt
doc docs: ${OUTPUT}.txt ${OUTPUT}.html
${OUTPUT}.txt: ${INPUT}.txt ${OXTRADOC} xolint.txt
${OXTRADOC_CMD} -m text -o $@ $<
${OUTPUT}.html: ${INPUT}.txt ${OXTRADOC} ${XML2HTMLBIN} xolint.txt
${OXTRADOC_CMD} -m html -o $@ $<
xolint.txt: ${top_srcdir}/xolint/xolint.pl
perl ${top_srcdir}/xolint/xolint.pl -D > xolint.txt
#
# The contents of xolint.rst is generated based on xolint.pl, since we
# really want this to be self-documenting. But readthedocs.org needs this
# data to be _in_ repo. So we generate this file on command only, and
# the developer needs to commit any changes.
#
CLEANFILES = \
xolint.txt \
${INPUT}.xml \
${INPUT}.fxml \
${OUTPUT}.txt \
${OUTPUT}.html
else
doc docs:
@${ECHO} "The 'oxtradoc' tool is not installed; see libslax.org"
endif
xolint.rst: ${top_srcdir}/xolint/xolint.pl
perl ${top_srcdir}/xolint/xolint.pl -D > ${top_srcdir}/doc/xolint.rst
SPHINX = python3.4 -msphinx
SPHINX = python3 -msphinx
html sphinx sphinx-html:
${SPHINX} -M html ${srcdir} .
${SPHINX} -M html ${srcdir} . -N -E
singlehtml:
${SPHINX} -M singlehtml ${srcdir} . -N -E
doc/api.rst
View file @ 0922e92a
.. index: API
.. index:: API
The libxo API
=============
......@@ -155,14 +155,14 @@ Output Styles (XO_STYLE\_\*)
The libxo functions accept a set of output styles:
=============== =========================
Flag Description
=============== =========================
XO_STYLE_TEXT Traditional text output
XO_STYLE_XML XML encoded data
XO_STYLE_JSON JSON encoded data
XO_STYLE_HTML HTML encoded data
=============== =========================
=============== =========================
Flag Description
=============== =========================
XO_STYLE_TEXT Traditional text output
XO_STYLE_XML XML encoded data
XO_STYLE_JSON JSON encoded data
XO_STYLE_HTML HTML encoded data
=============== =========================
The "XML", "JSON", and "HTML" output styles all use the UTF-8
character encoding. "TEXT" using locale-based encoding.
......@@ -256,26 +256,26 @@ Flags (XOF\_\*)
The set of valid flags include:
=================== =========================================
Flag Description
=================== =========================================
XOF_CLOSE_FP Close file pointer on `xo_destroy`
XOF_COLOR Enable color and effects in output
XOF_COLOR_ALLOWED Allow color/effect for terminal output
XOF_DTRT Enable "do the right thing" mode
XOF_INFO Display info data attributes (HTML)
XOF_KEYS Emit the key attribute (XML)
XOF_NO_ENV Do not use the :ref:`libxo-options` env var
XOF_NO_HUMANIZE Display humanization (TEXT, HTML)
XOF_PRETTY Make "pretty printed" output
XOF_UNDERSCORES Replaces hyphens with underscores
XOF_UNITS Display units (XML, HMTL)
XOF_WARN Generate warnings for broken calls
XOF_WARN_XML Generate warnings in XML on stdout
XOF_XPATH Emit XPath expressions (HTML)
XOF_COLUMNS Force xo_emit to return columns used
XOF_FLUSH Flush output after each `xo_emit` call
=================== =========================================
=================== =========================================
Flag Description
=================== =========================================
XOF_CLOSE_FP Close file pointer on `xo_destroy`
XOF_COLOR Enable color and effects in output
XOF_COLOR_ALLOWED Allow color/effect for terminal output
XOF_DTRT Enable "do the right thing" mode
XOF_INFO Display info data attributes (HTML)
XOF_KEYS Emit the key attribute (XML)
XOF_NO_ENV Do not use the :ref:`libxo-options` env var
XOF_NO_HUMANIZE Display humanization (TEXT, HTML)
XOF_PRETTY Make "pretty printed" output
XOF_UNDERSCORES Replaces hyphens with underscores
XOF_UNITS Display units (XML, HMTL)
XOF_WARN Generate warnings for broken calls
XOF_WARN_XML Generate warnings in XML on stdout
XOF_XPATH Emit XPath expressions (HTML)
XOF_COLUMNS Force xo_emit to return columns used
XOF_FLUSH Flush output after each `xo_emit` call
=================== =========================================
The `XOF_CLOSE_FP` flag will trigger the call of the *close_func*
(provided via `xo_set_writer`) when the handle is destroyed.
......@@ -300,12 +300,12 @@ regardless of whether warnings are enabled.
If the style is `XO_STYLE_HTML`, the following additional flags can be
used:
=============== =========================================
Flag Description
=============== =========================================
XOF_XPATH Emit "data-xpath" attributes
XOF_INFO Emit additional info fields
=============== =========================================
=============== =========================================
Flag Description
=============== =========================================
XOF_XPATH Emit "data-xpath" attributes
XOF_INFO Emit additional info fields
=============== =========================================
The `XOF_XPATH` flag enables the emission of XPath expressions detailing
the hierarchy of XML elements used to encode the data field, if the
......@@ -317,11 +317,11 @@ output. See :ref:`field-information` for details.
If the style is `XO_STYLE_XML`, the following additional flags can be
used:
=============== =========================================
Flag Description
=============== =========================================
XOF_KEYS Flag "key" fields for XML
=============== =========================================
=============== =========================================
Flag Description
=============== =========================================
XOF_KEYS Flag "key" fields for XML
=============== =========================================
The `XOF_KEYS` flag adds "key" attribute to the XML encoding for
field definitions that use the "k" modifier. The key attribute has
......@@ -1308,52 +1308,52 @@ These values are defined in <syslog.h>.
The priority value indicates the importance and potential impact of
each message:
============= =======================================================
Priority Description
============= =======================================================
LOG_EMERG A panic condition, normally broadcast to all users
LOG_ALERT A condition that should be corrected immediately
LOG_CRIT Critical conditions
LOG_ERR Generic errors
LOG_WARNING Warning messages
LOG_NOTICE Non-error conditions that might need special handling
LOG_INFO Informational messages
LOG_DEBUG Developer-oriented messages
============= =======================================================
============= =======================================================
Priority Description
============= =======================================================
LOG_EMERG A panic condition, normally broadcast to all users
LOG_ALERT A condition that should be corrected immediately
LOG_CRIT Critical conditions
LOG_ERR Generic errors
LOG_WARNING Warning messages
LOG_NOTICE Non-error conditions that might need special handling
LOG_INFO Informational messages
LOG_DEBUG Developer-oriented messages
============= =======================================================
The facility value indicates the source of message, in fairly generic
terms:
=============== =======================================================
Facility Description
=============== =======================================================
LOG_AUTH The authorization system (e.g. :manpage:`login(1)`)
LOG_AUTHPRIV As LOG_AUTH, but logged to a privileged file
LOG_CRON The cron daemon: :manpage:`cron(8)`
LOG_DAEMON System daemons, not otherwise explicitly listed
LOG_FTP The file transfer protocol daemons
LOG_KERN Messages generated by the kernel
LOG_LPR The line printer spooling system
LOG_MAIL The mail system
LOG_NEWS The network news system
LOG_SECURITY Security subsystems, such as :manpage:`ipfw(4)`
LOG_SYSLOG Messages generated internally by :manpage:`syslogd(8)`
LOG_USER Messages generated by user processes (default)
LOG_UUCP The uucp system
LOG_LOCAL0..7 Reserved for local use
=============== =======================================================
=============== =======================================================
Facility Description
=============== =======================================================
LOG_AUTH The authorization system (e.g. :manpage:`login(1)`)
LOG_AUTHPRIV As LOG_AUTH, but logged to a privileged file
LOG_CRON The cron daemon: :manpage:`cron(8)`
LOG_DAEMON System daemons, not otherwise explicitly listed
LOG_FTP The file transfer protocol daemons
LOG_KERN Messages generated by the kernel
LOG_LPR The line printer spooling system
LOG_MAIL The mail system
LOG_NEWS The network news system
LOG_SECURITY Security subsystems, such as :manpage:`ipfw(4)`
LOG_SYSLOG Messages generated internally by :manpage:`syslogd(8)`
LOG_USER Messages generated by user processes (default)
LOG_UUCP The uucp system
LOG_LOCAL0..7 Reserved for local use
=============== =======================================================
In addition to the values listed above, xo_open_log accepts a set of
addition flags requesting specific logging behaviors:
============ ====================================================
Flag Description
============ ====================================================
LOG_CONS If syslogd fails, attempt to write to /dev/console
LOG_NDELAY Open the connection to :manpage:`syslogd(8)` immediately
LOG_PERROR Write the message also to standard error output
LOG_PID Log the process id with each message
============ ====================================================
============ ====================================================
Flag Description
============ ====================================================
LOG_CONS If syslogd fails, attempt to write to /dev/console
LOG_NDELAY Open the connection to :manpage:`syslogd(8)` immediately
LOG_PERROR Write the message also to standard error output
LOG_PID Log the process id with each message
============ ====================================================
.. index:: xo_syslog
......@@ -1588,26 +1588,26 @@ processing model of libxo. Content is formatted within libxo, and
callbacks are made to the encoder's handler function when data is
ready to be processed:
======================= =======================================
Operation Meaning (Base function)
======================= =======================================
XO_OP_CREATE Called when the handle is created
XO_OP_OPEN_CONTAINER Container opened (xo_open_container)
XO_OP_CLOSE_CONTAINER Container closed (xo_close_container)
XO_OP_OPEN_LIST List opened (xo_open_list)
XO_OP_CLOSE_LIST List closed (xo_close_list)
XO_OP_OPEN_LEAF_LIST Leaf list opened (xo_open_leaf_list)
XO_OP_CLOSE_LEAF_LIST Leaf list closed (xo_close_leaf_list)
XO_OP_OPEN_INSTANCE Instance opened (xo_open_instance)
XO_OP_CLOSE_INSTANCE Instance closed (xo_close_instance)
XO_OP_STRING Field with Quoted UTF-8 string
XO_OP_CONTENT Field with content
XO_OP_FINISH Finish any pending output
XO_OP_FLUSH Flush any buffered output
XO_OP_DESTROY Clean up resources
XO_OP_ATTRIBUTE An attribute name/value pair
XO_OP_VERSION A version string
======================= =======================================
======================= =======================================
Operation Meaning (Base function)
======================= =======================================
XO_OP_CREATE Called when the handle is created
XO_OP_OPEN_CONTAINER Container opened (xo_open_container)
XO_OP_CLOSE_CONTAINER Container closed (xo_close_container)
XO_OP_OPEN_LIST List opened (xo_open_list)
XO_OP_CLOSE_LIST List closed (xo_close_list)
XO_OP_OPEN_LEAF_LIST Leaf list opened (xo_open_leaf_list)
XO_OP_CLOSE_LEAF_LIST Leaf list closed (xo_close_leaf_list)
XO_OP_OPEN_INSTANCE Instance opened (xo_open_instance)
XO_OP_CLOSE_INSTANCE Instance closed (xo_close_instance)
XO_OP_STRING Field with Quoted UTF-8 string
XO_OP_CONTENT Field with content
XO_OP_FINISH Finish any pending output
XO_OP_FLUSH Flush any buffered output
XO_OP_DESTROY Clean up resources
XO_OP_ATTRIBUTE An attribute name/value pair
XO_OP_VERSION A version string
======================= =======================================
For all the open and close operations, the name parameter holds the
name of the construct. For string, content, and attribute operations,
......
doc/conf.py
View file @ 0922e92a
......@@ -21,6 +21,14 @@
# import sys
# sys.path.insert(0, os.path.abspath('.'))
import subprocess
#
# Instead of hardcoding the version number here, we read it from the
# project's configure script
#
vers_cmd = "grep AC_INIT ../configure.ac | awk '{ print substr($2, 2, length($2) - 3);}'"
version = subprocess.check_output(vers_cmd, shell=True).decode("utf-8")
# -- General configuration ------------------------------------------------
......@@ -47,7 +55,7 @@
# General information about the project.
project = 'libxo'
copyright = '2017, Juniper Networks'
copyright = '2017-2019, Juniper Networks Inc'
author = 'Phil Shafer'
default_role = 'code'
primary_domain = 'c'
......@@ -58,9 +66,9 @@
# built documents.
#
# The short X.Y version.
version = '0.8.4'
#version = 'develop'
# The full version, including alpha/beta/rc tags.
release = '0.8.4'
release = version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
doc/encoders.rst 0 → 100644
View file @ 0922e92a
.. index:: encoder
Encoders
========
This section gives an overview of encoders, details on the encoders
that ship with libxo, and documentation for developers of future
encoders.
Overview
--------
The libxo library contains software to generate four "built-in"
formats: text, XML, JSON, and HTML. These formats are common and
useful, but there are other common and useful formats that users will
want, and including them all in the libxo software would be difficult
and cumbersome.
To allow support for additional encodings, libxo includes a
"pluggable" extension mechanism for dynamically loading new encoders.
libxo-based applications can automatically use any installed encoder.
Use the "encoder=XXX" option to access encoders. The following
example uses the "cbor" encoder, saving the output into a file::
df --libxo encoder=cbor > df-output.cbor
Encoders can support specific options that can be accessed by
following the encoder name with a colon (':') and one of more options,
separated by a plus sign "+"::
df --libxo encoder=csv:path=filesystem+leaf=name+no-header
This example instructs libxo to load the "csv" encoder and pass the
following options::
path=filesystem
leaf=name
no-header
Each of these option is interpreted by the encoder, and all such
options names and semantics are specific to the particular encoder.
Refer to the intended encoder for documentation on its options.
.. _csv_encoder:
CSV - Comma Separated Values
----------------------------
libxo ships with a custom encoder for "CSV" files, a common format for
comma separated values. The output of the CSV encoder can be loaded
directly into spreadsheets or similar applications.
A standard for CSV files is provided in :RFC:`4180`, but since the
format predates that standard by decades, there are many minor
differences in CSV file consumers and their expectations. The CSV
encoder has a number of options to tailor output to those
expectations.
Consider the following XML::
% list-items --libxo xml,pretty
<top>
<data test="value">
<item test2="value2">
<sku test3="value3" key="key">GRO-000-415</sku>
<name key="key">gum</name>
<sold>1412</sold>
<in-stock>54</in-stock>
<on-order>10</on-order>
</item>
<item>
<sku test3="value3" key="key">HRD-000-212</sku>
<name key="key">rope</name>
<sold>85</sold>
<in-stock>4</in-stock>
<on-order>2</on-order>
</item>
<item>
<sku test3="value3" key="key">HRD-000-517</sku>
<name key="key">ladder</name>
<sold>0</sold>
<in-stock>2</in-stock>
<on-order>1</on-order>
</item>
</data>
</top>
This output is a list of `instances` (named "item"), each containing a
set of `leafs` ("sku", "name", etc).
The CSV encoder will emit the leaf values in this output as `fields`
inside a CSV `record`, which is a line containing a set of
comma-separated values::
% list-items --libxo encoder=csv
sku,name,sold,in-stock,on-order
GRO-000-415,gum,1412,54,10
HRD-000-212,rope,85,4,2
HRD-000-517,ladder,0,2,1
Be aware that since the CSV encoder looks for data instances, when
used with :ref:`xo`, the `--instance` option will be needed::
% xo --libxo encoder=csv --instance foo 'The {:product} is {:status}\n' stereo "in route"
product,status
stereo,in route
.. _csv_path:
The `path` Option
~~~~~~~~~~~~~~~~~
By default, the CSV encoder will attempt to emit any list instance
generated by the application. In some cases, this may be
unacceptable, and a specific list may be desired.
Use the "path" option to limit the processing of output to a specific
hierarchy. The path should be one or more names of containers or
lists.
For example, if the "list-items" application generates other lists,
the user can give "path=top/data/item" as a path::
% list-items --libxo encoder=csv:path=top/data/item
sku,name,sold,in-stock,on-order
GRO-000-415,gum,1412,54,10
HRD-000-212,rope,85,4,2
HRD-000-517,ladder,0,2,1
Paths are "relative", meaning they need not be a complete set
of names to the list. This means that "path=item" may be sufficient
for the above example.
.. _csv_leafs:
The `leafs` Option
~~~~~~~~~~~~~~~~~~
The CSV encoding requires that all lines of output have the same
number of fields with the same order. In contrast, XML and JSON allow
any order (though libxo forces key leafs to appear before other
leafs).
To maintain a consistent set of fields inside the CSV file, the same
set of leafs must be selected from each list item. By default, the
CSV encoder records the set of leafs that appear in the first list
instance it processes, and extract only those leafs from future
instances. If the first instance is missing a leaf that is desired by
the consumer, the "leaf" option can be used to ensure that an empty
value is recorded for instances that lack a particular leaf.
The "leafs" option can also be used to exclude leafs, limiting the
output to only those leafs provided.
In addition, the order of the output fields follows the order in which
the leafs are listed. "leafs=one.two" and "leafs=two.one" give
distinct output.
So the "leafs" option can be used to expand, limit, and order the set
of leafs.
The value of the leafs option should be one or more leaf names,
separated by a period (".")::
% list-items --libxo encoder=csv:leafs=sku.on-order
sku,on-order
GRO-000-415,10
HRD-000-212,2
HRD-000-517,1
% list-items -libxo encoder=csv:leafs=on-order.sku
on-order,sku
10,GRO-000-415
2,HRD-000-212
1,HRD-000-517
Note that since libxo uses terminology from YANG (:RFC:`7950`), the
data modeling language for NETCONF (:RFC:`6241`), which uses "leafs"
as the plural form of "leaf". libxo follows that convention.
.. _csv_no_header:
The `no-header` Option
~~~~~~~~~~~~~~~~~~~~~~
CSV files typical begin with a line that defines the fields included
in that file, in an attempt to make the contents self-defining::
sku,name,sold,in-stock,on-order
GRO-000-415,gum,1412,54,10
HRD-000-212,rope,85,4,2