Architecture for a Semantic Web of Smart Objects, part 2, Constraints and API

Data Models for the Internet of Things Part 4, Smart Object API and Architecture

8/30/2012
Michael J Koster

The goal is to create a web ecosystem of sensing, reasoning, and action around the Internet of Things.

In the first post I talked about the current Internet of Things architecture as consisting of sensor nets connected to web services in a mostly vertical fashion. That is, there are systems consisting of sensors, a gateway, and a cloud service where the gateway interacts with the cloud service in a specific or even proprietary fashion. Likewise the API for adding applications to the service is either private to that service or at least a service-specific API. There is a different API for each service.

Yet the interactions between gateway and service, and the affordances the service provides to applications are mostly the same. These gateways interact with the cloud service using web capable RESTful interfaces. There are common patterns emerging for these interfaces and APIs

There are many new sensors, from personal fitness trackers to buddy lamps that turn on and off together halfway around the world. To bring each new idea to market requires a lot of plumbing to be reinvented, mostly from standard parts and patterns. When they're done, they don't talk to each other. There's practically no way to second source a service or to create an application that can arbitrarily mash up data streams. A connector is needed for each service and the only access is to the stored datastream. Cosm, Nimbits, Sen.se, ThingWorx have different APIs that all do the same things. There is no standard way for gateways to interact directly with each other or with user devices like smartphones.

The opportunity is for a simple, open set of constraints and conventions around the existing pattern using sensor nets, gateways, and web services. A standard web API and M2M protocol will allow gateways, smart sensors, services, and user devices to interact with each other at the granular, scale-less level of "things". 

The Smart Object is a Semantic Web application for the Internet of Things

In the earlier posts, I describe a Smart Object as a self-describing internet representation of a thing of interest. It's a generalization of the concept of a sensor that can interact directly with services. 

The Semantic Web is a set of tools that enable semantic discovery and linkage of web objects pointed to by URLs. These tools can be used to enable content-based discovery and linkage, which is necessary to support interaction at a granular scale within a large system.

In the context of this discussion, a Smart Object is an entity, pointed to by a URL, that encapsulates some Observable Properties along with their Semantic Web descriptions, and application software agents with a RESTful API. An Observable Property can be anything from a sensor reading to an arbitrary document.

It's all about the API

The Smart Object API enables semantic discovery and linkage, access to Observable Properties, and control of application agents. 

Observable Properties from one Smart Object to another are linked by a subscribe/notify method that optionally generates callbacks to application handlers. 

The API uses named and described elements inside the object to interact with the Smart Object using a simple RESTful interface. In this way an agent or service proxy wishing to interact with an object's Observable Properties can discover enough information to connect Observable Properties to algorithms and actuators. 

The Smart Object API looks the same to an agent inside the object as it does on the RESTful web interface. M2M interactions between an agent at one node and an object on another over http consist of simply generating, exchanging, and scanning document fragments. Agent software can in this way be plugged into data streams based on semantic compatibility and linkage criteria.

Architected Constraints for a base Smart Object Semantic Model

This is a description of a semantic reference model for a Smart Object, and a list of API methods based on that reference model. This defines the API and reference architecture of a Smart Object.

The model is object oriented and extensible. The elements define their own methods and interfaces. The elements are chosen to allow simple idempotent operations with storage semantics, i.e. a CRUD style interface using REST principles.


I'm using a form of declarative constraints to define the model where the basic form is a graph showing linked entity-relationship triples to define patterns. Here's an example:

An object "Has" a element (one or more) which "Is" a Pattern is more precisely "An object has contained within it a top level named element, which is an instance of class "Pattern"

A shorthand is Object has Pattern meaning one or more instances of a Pattern.

Smart Object Pattern

Here is the current proposed Smart Object Pattern:

This shows the constrained relationships between the SmartObject's top level elements. The Smart Object is pointed to by a URL and a path to each method is exposed e.g. :

URL = Object.Description.ADDTRIPLES(triples);

The Observable Properties are the representations of individual things at any granular scale level, from say a temperature reading to an entire document. They are also self-describing objects.

The Description contains RDF triples describing the Object's top level elements; The Observable Properties and Agents, and overall description of the Object itself.

The Agent pattern shows how Application Agents and the Web API agent (web proxy) see the same underlying methods, thus have the same API, local or remote. An Agent can be either the Web API proxy for Web clients, plus any number of separate Application Agents that can be created within the scope of the Smart Object. 

Agents have a Web client they use for interacting with Smart Objects on other services, gateways, user devices, etc. using a RESTful M2M protocol over http. The client and proxy exchange document fragments representing the remote API method invocations and responses.

Observable Properties and Agents can be dynamically created and deleted after the base instance of a Smart Object is created.

The Observable Property is itself a self-describing Object with the following structural constraints:

The Observable Property has the same Description pattern as the top level object. 

The Property of Interest is the container for the actual data or data stream, and allows custom implementations and methods to be defined by each Property of Interest. There is exactly one instance of Property Of Interest for each Observable Property, but it can be of any type including compound types or documents.

The Subscription is how data push on change or other asynchronous notification is performed. SUBSCRIBE adds a URL to a set of handlers to call back on updates to the Property of Interest. The handlers can be to the local agent, or to a URL pointing to an Observable Property on some other Smart Object to push data or notification to.

API Summary

Mapping to CRUD methods

Since each element has a unique path to it, e.g. 

Object.ObservablePropertyX.GET()

Object.Description.DESCRIBE()

A form of the GET verb can be used for both without confusion:

Object.ObservablePropertyX.GET()

Object.Description.GET()

ADDTRIPLES and REMOVETRIPLES can map to CREATE and DELETE respectively.

Likewise, SUBSCRIBE can be though of as the CREATE of a subscription, and UNSUBSCRIBE is a DELETE.

Each method in the Smart Object API can be mapped to one of the four CRUD methods relative to it's element. 

Summary

The Smart Object is a prototype Semantic Web API and application framework for the Internet of Things. 

It is a common data model and API for Internet Gateways, Web Services, and User Devices to use for interacting with each other spontaneously at the granular scale of connected Things.

Semantic discovery and linkage is enabled by an interface that attaches semantic tags dynamically to named resources within the object.

Every element instance is self-describing. The "data" element resolves to a self-describing instance of a type, allowing complete flexibility in data representation. 

An instance of a Smart Object is pointed to by a URL and includes the object's Observable Properties, it's Description, and it's Application Agents. 

Observable properties of a Smart Object can be Subscribed to by other other Smart Objects for Notify-On-Change or Data Push operation.

The Smart Object API looks the same to a local Agent as to a Web-connected Agent, allowing transparency of location. Only the URLs are differently routed.

Web services can be second-sourced, or mashed up with other services at the data algorithm level. 

Gateways and Smart Sensors can interact directly with actuators and user devices when it makes sense, using the same agents and APIs. Smart Objects and their services can be migrated and replicated easily for total system ruggedness. 

Next up

As I build a prototype Smart Object and deploy a test service, I'll be looking at the high level ecosystem again. A general tools framework is needed around Smart Objects to make plumbing a new web service as easy as making an instance of a class. 

I also will be looking at prototyping Arduino sensors, super-structed gateways, user informing devices, and smartphone/tablet controls. 

Architecture Features for a Semantic Web of Smart Objects

Architecture Features for a Semantic Web of Smart Objects

Michael J. Koster
August 21st, 2012

In this post I'd like to take a brief architectural look at the Semantic Web of Smart Objects I described in the first 2 posts. This provides a framework from which to understand the choices and direction.

Architecture as a set of constraints

Here I am looking at the notion of architecture as broadly consisting of a set of constraints imposed in order to achieve a particular goal.

The goal is to enable the creation of a web ecosystem of sensing, reasoning, and action around the internet of things. 

The mechanism is broad interoperability across sensors, services, and devices.

I propose to do this by creating a semantic web template (set of constraints) for the internet of things, and a set of common practices that can also be semantically defined.


In particular, the architecture of a semantic web of smart objects is a set of constraints imposed on the use of http/REST and semantic web standards in order to achieve broad interoperability at a granular scale of things and collections of things. (These constraints can themselves be semantically defined, making this a sort of "declarative architecture").

Seven Layer Model

The first organizing principle I'd like to invoke is the well-known seven layer model for networked applications. Given the predefined set of constraints (http/REST and semantic web) the session state management layer is well established by the RESTful interface.

Thus we need to define the constraints necessary to make interoperable the layers for data presentation and the interface to the application layer. This architecture therefore constrains the data presentation layer and API. 

Constraints will be applied to the RESTful interface to enable an interoperable service laver which provides data presentation services and a harmonized application API to enable data sources and applications to interact seamlessly and semantically with each other. In other words, data sources may be plugged into applications and applications may provide pluggable data sources using a uniform set of mechanisms.

Smart Objects


The next organizing principe I'd like to invoke is object encapsulation. The definition of an object also becomes the de facto definition for the representation of a thing or set of things on the Semantic Web and on the Internet.

The basic object encapsulation is an important architectural choice, as this constraint becomes a constraint on the way applications interact with data and metadata.

I propose to encapsulate observable property data and metadata together under a RESTful interface model that can be semantically shared between the web interface to an object and the API to application software executing within the context of the object.

Object state, object metadata, software agents, and software agent metadata share a RESTful API, with a single instance of each object pointed to by a URL. The entity this URL points to is referred to as a Smart Object for the purpose of this discussion.

Semantic discovery and linkage

The constraints needed to define the architecture here are related to semantic web operations. Some well defined method for getting the semantic metadata from an object to provide for indexing and discovery is needed. Fortunately the metadata are unstructured RDF triples and a simple list-all mechanism is sufficient. All that is needed is the URL to point to the object and a well known type to retrieve the triples. The triples can then be indexed for semantic linkage.

Semantic linkage can be accommodated by adding methods to return the URLs of resources in the object whose metadata properties satisfy semantic queries. A type for semantic queries and a semantically defined subset of SPARQL is indicated. An additional mtype for embedded semantic linkage is also possible and may prove useful to improve efficiency.

Additional constraints and use cases

It's important for several use cases to provide a mechanism to conditionally push linked data from one Smart Object to another. This should not impact architectural constraints, since the push operation can be semantically defined, as can the application agent call back handler linkage. There may be a need to push metadata describing the push operation itself. Also of note is that metadata triples may also be conditionally pushed.

The use case for Smart Objects includes bridging to constrained network endpoints. In this case, the bridge may create a Smart Object proxy for resources on the constrained network, and record associations of resource metadata with resource addresses on the constrained network, functioning as a semantic reverse proxy router. The bridge can also function as metadata repository and agent execution environment.

A simple bridge endpoint can also function as a standalone smart sensor and interact directly with other Smart Objects on the Internet. The sensing/actuating process is performed by an agent with well defined properties.

The other endpoint is at a user device. Rather than generate HTML for a web page or XML for AJAX controls, a Smart Object proxy can run directly on the user interface device and provide the Smart Object agent API directly to a user information/control app. A Smart Object proxy also allows the easy connection of a broad range of user interface devices, including the use of simple devices on constrained networks.

Standards 

The architectural constraints apply to a set of existing web and internet standards. No new methods or protocols need to be invented. The standards drawn from include:

http(s)
REST
HTML
XML
JSON
Xpointer/Xlink
RDF/Turtle
SPARQL
SSN ontology
DUL ontology
OAuth

In addition it is expected to use and interoperate with:

6LoWPAN
CoRE
CoAP
Arduino
Processing
Python/RESTlite

Summary

The goal is to enable the creation of a web ecosystem of sensing, reasoning, and action around the internet of things. 

The above describes what may be a minimum set of constraints and models needed to enable broad interoperability. It also defines an architectural approach to achieving a Semantic Web of Things. 

The architecture described above defines a data presentation layer and API which use a common RESTful interface model.

A Smart Object abstraction encapsulates an instance of observable data properties, semantic metadata, application agent processing, and agent process metadata under a URL.

Smart Objects can be semantically linked together through a metadata query interface or dynamic  semantic linkage language.

Endpoints of constrained sensor nets, Smart Sensors, and diverse user interface devices are enabled through Smart Object proxies.

No new methods or protocols need be invented. The architecture consists mostly of declarative constraints.

Additional constraints may be found as the investigation proceeds toward a prototype Smart Object framework.

Smart Objects and Semantic Linkage, toward a prototype Smart Object

Smart Objects and Semantic Linkage 

Toward a prototype Smart Object

Michael J Koster
August 17, 2012

This is the second post in a series documenting my investigation into data models for pervasive interoperability on the Internet of Things. Here I'm looking into the information architecture inside the data model.

Previously, I mentioned that many sensors and devices already have the ability to connect to the internet. There is emerging a common way for devices and sensors to present themselves on the internet, which is through a RESTful interface. In short, Representational State Transfer is a style of communication that allows interfaces to be stateless and self-describing. Using REST allows two entities to exchange information is a well defined way without them having to know in advance what the other endpoint is going to do internally. The result is a standard and stable API that can be widely shared.

Smart Objects 

Within the internet of things there is emerging the concept of a Smart Object. This is often described as a sensor or actuator which interacts using a REST interface on a network or on the internet.

Here is a somewhat more general concept of a smart object. In this document I'm going to use the term "Smart Object" to refer to a hardware device or software entity, pointed to by a URL, that implements the data models for the Semantic Web of Things. This is a logical broadening of the existing definition of a smart object from a component of a sensor network to a data node in the Semantic Web of Things.

A Smart Object observes or controls some feature of interest and interacts using a RESTful interface on a network such as the internet. The feature of interest can be practically any information source as long as there is a well known representation. The feature of interest may be a mashup of observable properties of other Smart Objects.

The Smart Object is a resource pointed to by a URL. It can be a smart sensor, a sensor observation or actuator control on a smarthome gateway,  a resource on a cloud service that provides an information stream, or a device that informs the user e.g. an "orb". It can also be a data connector to a system that uses it's own proprietary data model. Practically anything from documents to real time data streams can be made into Smart Objects. Smart Objects can have arbitrary internal processes that filter, aggregate, or mashup data streams, process documents, or drive user controls. 

There is not a current strong use-case for the extended functionality of the Smart Object interface, in that no large scale Internet of Things applications exist today. Common themes in the vision for the Internet of Things are location awareness, real time data consistency, granular scale (scale-less interactions), information mash-up, semantic discovery and linkage, self-aware systems. 

The well known RESTful API operations of GET/SET/CREATE/DELETE for resources have been extended in CoAP to support data push using a subscribe/publish model. I believe publish-on-change data push is a very important mode in order to support both high volume and granular scale while maintaining real time data consistency.

I propose to add both semantic metadata operations and a push/subscription data transfer mode to the familiar REST interface for smart objects. I believe this can be the basis of a scalable, effective data model for a semantic web of Smart Objects.

Semantic Linkage of Smart Objects - A Semantic Web of Things

Smart Objects can be considered data sources. The capability of Smart Objects can be extended by providing an interface allowing the smart object to describe itself. A user, service, or agent can ask the Smart Object to describe itself and thus obtain a catalog of the observable properties of the object and their semantic descriptions.

This semantic metadata interface allows a crawler or application composer tool to discover the observable properties and other resources of a Smart Object, thus facilitating the chaining, aggregation, mash-up, etc. of the observable properties and resources from a specific collection of Smart Objects.

Data Push Capability - Data driven compute model

Data push is an optional method to propagate updates of observable properties or actuator states between Smart Objects. When a qualifying event such as a significant change in an observable property occurs, the information will be pushed, literally multicast, to other Smart Objects.

Smart Objects need the capability to respond to changes pushed to their input properties without having to arbitrarily poll resources. This requires some kind of callback to a handler or agent in response. The Smart Object proposed contains agent code and a framework to link handlers to property updates pushed from other Smart Objects. 

A data driven, semantically linked web of things

The combination of semantic capability and data push allows Smart Objects to be assembled into a semantically linked data flow graph to construct larger systems. Smart Objects can be instantiated and connected to data sources, and outputs routed to other smart objects for filtering, aggregation, display, etc.

A Prototype Smart Object

A simple prototype can be constructed to create a testbed for the Smart Object and the Semantic Web of Things. Here is a simplified block diagram of the components for a prototype Smart Object.

Smart Object

Data and metadata (semantic tags) are accessed by a set of RESTful methods which are both mapped onto the http interface and available to a local software agent. 

The software agent contains the application software such as timers, PID controllers, filters, aggregators and the callback handlers for data push processing. The software agent is also responsible for sensor and actuator control in Smart Sensors, user interface connections, and connectors to other data models e.g. legacy databases.

Semantic metadata are stored in the form of RDF triples that describe the resources and observable properties of the Smart Object. CREATE, DELETE, SET, and GET should be able to operate on the metadata, and the DESCRIBE method is proposed to point to a resource containing the full RDF description of the object or one of it's observable properties.

Smart Object Registry for semantic discovery and linkage

A Smart Object can also function as a registry for itself and other Smart Objects, providing a resource in the network that can support discovery and linkage through a well known URL. Registry objects would provide additional RDF query functions, e.g. SPARQL interface and index to the triple store, to support semantic discovery and linking.

Crawler software on the registry nodes can obtain the RDF from Smart Objects using the DESCRIBE method, and populate and index a triplestore. A SPARQL query engine on the triplestore indices returns URLs of Smart Objects that contain observable properties or resources that satisfy the query predicates. Any additional semantic metadata necessary to perform a particular mash-up can be obtained from the registry that describes the selected objects and resources. A new Smart Object is created that has as it's inputs properties of other Smart Objects, and as it's output the new composite properties resulting from the mash-up algorithm. If real time push updates of observable properties is desired, subscriptions are created to the Smart Objects that are the data sources.

Data models

The data models for interoperability consist of the semantic definitions of the Smart Object REST interface including the push and metadata operations, the use of embedded RDF to describe the observable properties and resources of a Smart Object, and the Smart Object registry interface for semantic discovery and linking.

The choice of data types, representations, sizes, formats, etc. is left entirely to the domain of interest to use the applicable ontologies. The choice of object granularity vs. complexity is entirely up to the system designer and information architect.

The location of registries and system of well known registry paths in larger deployments is as yet unspecified and the subject of investigation. It's expected that some design patterns will emerge and some conventions and protocols refined as a result.

Related Work

The SPITFIRE project associates semantic metadata with RESTful smart sensors, and provides a triplestore with SPARQL interface for semantic discovery and linkage.

https://www.iti.uni-luebeck.de/fileadmin/sensornw/paper/IEEEComMag.pdf

CoAP is a constrained application protocol meant to use CoRE, a REST interface for constrained environments. It's emerging in the set of 6LoWPAN related standards. The interesting thing about CoAP is the inclusion of a subscription/publish-on-change mode.

Here's a concise feature analysis of CoAP:

https://datatracker.ietf.org/doc/draft-shelby-core-coap/

Next Up

I'm currently looking at tools for manually building and maintaining registry information and beginning implementation of the prototype. 

The Internet of Things

Data Models for the Internet of Things

Michael J Koster
August 15, 2012

The title of this is "Data Models for the Internet of Things". Perhaps it should be "Data Models for a Semantic Web of Things" because that describes what I believe is going to be the next need for a broad interoperability standard. First I need to set the background and state some assumptions.

What is the Internet of Things?

Here is a broad definition: People connecting to things, things connecting to things, and people connecting to people at a granular scale.

A thing can be a sensor that encodes observations, or some other source of information about the physical world around us, or a display to inform us, or it may be an actuator to control some feature of our environment. For example, a thing might be a measurement from a temperature sensor, or a weather forecast for tomorrow, or the current location of a friend.

Connecting at a granular scale means that the information can be used at either large scale or small scale, or anywhere in between. It's also known as scale-less. It means, for example, that single sensor data can be used as a single observation, or data from multiple sensors can be aggregated over time for a larger view of the phenomenon. Mash-ups are examples of data interaction on a granular scale.

Many of these things are currently observable on the internet, but they don't easily connect in meaningful ways to people or to each other. Nor do they always function well on a granular scale.

The Internet of Things is, more specifically, a set of tools and technologies which enable the internet to be used to connect people to things, things to things, and people to people on a granular scale.

The World Wide Web

This isn't meant to be a history lesson, but more of an exposition of a process.

In 1990, a number of tools and techniques were integrated to create a World Wide Web of documents on top of the existing internet. The first prototype consisted of a document server (http) transmitting HTML documents to a simple client on a NeXT workstation which was able to display the HTML in outline format with images and other hyperlinked resources rendered in separate windows.

The prototype code was openly published for people to play with and, over time, the system of rough consensus and running code was adopted for creating and extending internet and particularly world wide web standards.

The platform that is "the web" is today far richer than the HTML + http + URL hyperlinks of the first prototype; there are multiple revisions of HTML, dozens of other markup standards, WS, REST, LAMP, AJAX, Ruby On Rails..., all of which use http and the underlying network stack to basically accomplish the same thing: wiring user controls and displays in the web browser to application logic on the server or more recently "in the cloud". Now we have smartphones and tablets with skinny GUI tools as a very nice alternative to bloated browserware, but in the end it's still controls wired to server or cloud applications as the main user interaction paradigm, with back end communication between servers (Machine to Machine, or M2M) doing most of the work of integrating and filtering of data streams.

Web pages for light switches. Yay.

I believe that we are at a point in the development of the Internet of Things which is analogous to where the Internet, specifically the World Wide Web, was in 1990. At that time, the Internet existed and there were a number of applications hosted including email (POP, IMAP), relay chat (IRC), and document file transmission (UUCP).

The Internet of Things today looks a lot like the good old World Wide Web. Analogous to using uucp for sharing documents on the internet of 1989 (it was better than quarter inch tape...) we now have systems where networks of sensors e.g. smarthomes, are interacted with on web pages and smartphone apps hosted by cloud servers. Sure it's fun and useful, but the connectivity and ultimately the ability of the system to evolve is limited by use of the legacy interaction mechanism.

The sensor and low level network technology is also available today. It's in the early adopter phase waiting for the number of connections to increase to the point where Metcalfe's law becomes relevant. Low power radio, 6LoWPAN, CoRE, CoAP, and related efforts are creating a broad base for a low level infrastructure of sensing and control. Personal sensors like Nike Fuel are on the market. Kickstarter is flush with gadgets that connect to the internet, gadgets that sense and also gadgets that passively inform through soft lights or lava-lamp style readouts.

The problem is that each new gadget comes with it's own back end cloud hosted service that only connects to a select set of other things on the same service (and of course twitter). There is no easy way for a user of these gadgets to mash up their own set of interactions or even have their displays and controls all in one place. We might say that the interfaces are in silos, stuck in vertical enclosures that hide the contents from other silos.

We could think about some intermediary aggregators on the web to connect data sources from multiple cloud hosted services, but each service has it's own coding and semantics, quickly becoming a big data flavor of problem if we want to solve it in a general way.

Toward A Semantic Web of Things

The technology to accomplish the higher level goals, connecting people to things and things to things at a granular scale, exists today. It exists in a component form and in the form of hundreds of separately designed proprietary back end systems that all use common tools and design patterns but lack the common data models needed to successfully interoperate.

What's missing is a set of tools and techniques, analogous to the early web tools, for creating a system that can connect anything to anything else that makes sense. Note that this is not about connecting everything to everything else. Though we want to allow large scale deployments, this is more about connecting things in a very selective manner.