Deploying Telemetry at Bloomberg

Bloomberg offers a product called Bloomberg Terminal that has
thousands of components that are all monitored and updated. They
have 25% engineering staff.

Lots of disparate systems, and not a good way to correlate the
incoming data streams.  They needed a way to centralize and unify
all the services being offered.

Researched time series databases to find the best fit and to plan
how to deprecate the old stuff.

Collectd-win was an open source windows agent for collectd

Lessons
  - Understand what you are trying to solve - Routinely revisit
  those decisions - Make on-boarding flexible - Software should be
  interchangeable

"APIs enforce consistency, and documentation explains it"

Staging and production are different clusters, Grafana is the front
end interface with some Grafana, and metric-tank magic with Cassandra


Spinnaker, Helm, and a Development Life-cycle were experimented
with.

The kubernetes scheduler was a bit of a headache.

PR84405 - Kubernetes Kafka - Problems with them

Questions

Q: In the diagram Kafka was the ingestion engine that are then
cached by metric tank. What happened after it got into metric tank
before it goes to Cassandra ? A: We sanitize the data and clean it
up.