Scaling telemetry monitoring with InfluxDB

Person expectations for software program purposes maintain rising. These days, companies are anticipated to be extremely dependable and carry out properly 24/7. Any form of downtime goes to lead to annoyed customers and damage your enterprise long-term.

A key part in enhancing reliability is monitoring your software. Whereas organising primary monitoring is straightforward, being able to scale monitoring effectively as site visitors to your service grows is a serious problem. You additionally need visibility into each essential metric on your service and the flexibility to make the info you might be amassing helpful and actionable with the flexibility to question and analyze it effectively in actual time on demand.

In brief, there’s a giant distinction between the issues you run into throwing collectively one thing for a facet undertaking or small scale system vs. deploying telemetry monitoring at scale in a manufacturing setting.

One crew at Cisco experimented with InfluxDB to create an instance of a scalable telemetry monitoring structure that different firms with large-scale manufacturing environments may draw on, with out having to start out from scratch. This setup allowed Cisco to scale up its telemetry knowledge ingestion to 3TB per day (or round 16GB per minute). On the core of this structure is Cisco IOS-XR and InfluxDB.

Cisco telemetry monitoring structure overview

There are three primary elements in Cisco’s telemetry structure. The primary half is the Cisco {hardware} working IOS-XR, which produces the telemetry knowledge. The second half is the collector agent that takes in that knowledge after which sending it to the ultimate part for storage, which is completed with InfluxDB.

InfluxData

Cisco IOS-XR

IOS-XR is the working system utilized by Cisco for its high-end, carrier-grade routers such because the CRS collection, 12000 collection, and ASR 9000 collection community routers. In comparison with different community working techniques, IOS-XR supplies improved availability, higher scalability for giant {hardware} configurations, the flexibility to put in upgrades or patches whereas the router stays in service, and quite a few different options not obtainable in rivals.

One significantly related function is that IOS-XR supplies built-in streaming of telemetry knowledge to extend community visibility and has APIs obtainable for engineers to take motion primarily based on telemetry knowledge.

For this structure, Cisco streamed knowledge from three totally different IOS-XR platforms: the NCS 5500, ASR 9000, and the 8000 collection router. Cisco had the gadgets configured to run in dial-out mode, with self-describing GPBs (Google Protocol Buffers), over a TCP connection. One of many key elements in a telemetry monitoring structure at this stage is ensuring it doesn’t acquire extra knowledge than it wants when it comes to general metrics in addition to the frequency of metric assortment.

Collector agent

The telemetry knowledge from the IOS-XR {hardware} was despatched to a load balancer, which then forwarded the info between three totally different collector brokers. At giant scale, single-threaded collector techniques will be unable to deal with the quantity of knowledge being despatched to them. Multi-threaded collectors even have points as a result of they’re all importing to the database with separate connections, which creates one other set of issues.

To get round these issues Cisco wrote a multi-processing collector agent, with the code being open supply on GitHub. The collector agent’s primary course of is decoupled from the employee pool, which parses the info and uploads it to InfluxDB. The primary course of provides knowledge to a queue as it’s streamed in after which sends the telemetry knowledge to the employee pool in batches. The collector agent is ready to deal with gigabytes of knowledge per second, whereas remaining dependable resulting from this decoupled structure. This may be seen within the diagram under.

InfluxData

InfluxDB

The ultimate piece of the telemetry structure is InfluxDB, which is used to retailer the info. For this experiment, InfluxDB was deployed with two knowledge nodes and three meta nodes to kind a cluster to help improved reliability and efficiency.

InfluxDB is a purpose-built time collection database designed to deal with huge volumes of time-stamped knowledge, which made it an ideal match for Cisco’s telemetry monitoring use case. InfluxDB additionally works nice for any workload that requires having the ability to write giant quantities of knowledge and having the ability to question that knowledge in real-time. Widespread use instances embody IoT, analytics, and software monitoring.

InfluxDB is open supply and may be deployed by yourself infrastructure or arrange in minutes utilizing InfluxData’s cloud providing, InfluxDB Cloud. InfluxDB Cloud is a fully-managed, elastic time collection knowledge platform that enables customers to get began shortly after which simply scale to satisfy their necessities. Ingested knowledge may be displayed utilizing InfluxDB Cloud’s in-built dashboards and knowledge may be queried utilizing Flux, InfluxData’s composable, practical question language designed for time collection workloads.

For Cisco’s use case, it made a couple of modifications to InfluxDB’s normal configuration to optimize it for his or her particular wants. The primary was adjusting the default cache (buffer) reminiscence measurement. As a result of they had been writing knowledge in batches from the collector agent, InfluxDB wanted a bigger quantity of reminiscence put aside so it could persist that knowledge whereas it was being written. On the cluster degree, Cisco additionally selected to permit out-of-order duplicate writes to be made between nodes. This allowed extra flexibility within the relationship between knowledge arrival order and the factors’ accompanying timestamps.

Scaling telemetry knowledge is a troublesome job that many firms have tried to unravel on their very own. Cisco’s objective on this experiment was to supply a blueprint structure for different firms to comply with in order that they don’t should reinvent the wheel for their very own use case. A core a part of Cisco’s resolution was InfluxDB due to its efficiency, ease of use, and open supply code base.

Sam Dillard is senior product supervisor of IoT and enterprise at InfluxData.

—

New Tech Discussion board supplies a venue to discover and talk about rising enterprise expertise in unprecedented depth and breadth. The choice is subjective, primarily based on our decide of the applied sciences we imagine to be essential and of biggest curiosity to InfoWorld readers. InfoWorld doesn’t settle for advertising and marketing collateral for publication and reserves the proper to edit all contributed content material. Ship all inquiries to newtechforum@infoworld.com.