How to use ApplicationInsights with the vFunction OpenTelemetry Agent for a Distributed Application
Overview
Azure Monitor uses ApplicationInsights for OpenTelemetry observability. ApplicationInsights is built on top of OpenTelemetry but does not use the Open-Source OpenTelemetry Java Agent to collect its metrics. Instead, ApplicationInsights uses its own version of the OpenTelemetry Java Agent to collect the data needed for Azure Monitor. Unlike the Open-Source OpenTelemetry Agent, ApplicationInsights does not allow external products to use the OpenTelemetry data collected through the ApplicationInsights Java Agent.
There are multiple paths forward to be able to use both ApplicationInsights and additional OpenTelemetry products. This article is intended to explain some of these workflows as well as the Pros and Cons to those workflows.
Using the OpenTelemetry Agent to send data to an OpenTelemetry Collector and onto Azure Monitor as well as vFunction
The Open-Source OpenTelemetry Agent and Collector can be configured to send the trace information that Azure Monitor needs onto the Azure Monitor service. Take the following steps to install the OpenTelemetry Agent, install the OpenTelemetry Collector and add the configuration settings into the OpenTelemetry Collector’s Configuration files.
- Download the OpenTelemetry Java Agent. Note that this needs to be downloaded to the filesystem where the application runs
- Download the OpenTelemetry Collector. Note that this could be running alongside the Application or on a separate Container / Host
- Retrieve the ApplicationInsights Connection String. This can be retrieved, for example, from the Environment Variable ApplicationInsight_Connection_String in Azure App Services if ApplicationInsights is already enabled. This looks like “InstrumentationKey=aceb30b6-1111-1111-1111-8f2b6858a47f;IngestionEndpoint=https://eastus2-3.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus2.livediagnostics.monitor.azure.com/;ApplicationId=3d74d811-1111-1111-1111-4bc846fc833d”
- Modify the OpenTelemetry Collector’s configuration files to add Azure Monitor
vi /etc/otelcol/config.yaml
- Add references to send OpenTelemetry traffic to Azure Monitor
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
otlp:
endpoint: otelcol:4317
azuremonitor:
connection_string: "<CONNECTION_STRING>"
# maxbatchsize is the maximum number of items that can be
# queued before calling to the configured endpoint
maxbatchsize: 100
# maxbatchinterval is the maximum time to wait before calling
# the configured endpoint.
maxbatchinterval: 10s
extensions:
health_check:
endpoint: 0.0.0.0:13133
service:
extensions: [health_check]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp,azuremonitor]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
- Modify the OpenTelemetry Collector configuration to add vFunction
vi /etc/otelcol/config.yaml
- Add references to send OpenTelemetry traffic to Azure Monitor
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
metadata_keys:
- X-VF-APP
exporters:
otlp:
endpoint: otelcol:4317
azuremonitor:
connection_string: "<CONNECTION_STRING>"
# maxbatchsize is the maximum number of items that can be
# queued before calling to the configured endpoint
maxbatchsize: 100
# maxbatchinterval is the maximum time to wait before calling
# the configured endpoint.
maxbatchinterval: 10s
otlphttp/vf:
endpoint: $PROTOCOL://$VFSERVER/api/unauth/otlp
auth:
authenticator: headers_setter
# If the VF Server was configured to use HTTP instead of HTTPS, you will need to enable HTTP via the "tls.insecure: true" Key and Value
# tls:
# insecure: true
extensions:
health_check:
endpoint: 0.0.0.0:13133
headers_setter:
headers:
- action: insert
key: X-VF-APP
from_context: X-VF-APP
service:
extensions: [health_check,headers_setter]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp,azuremonitor,otlphttp/vf]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
- Restart the OpenTelemetry Collector
sudo systemctl restart otelcol-contrib
- Add the OpenTelemetry Java Agent’s Startup Parameters to the Application
### Modify $PATH_TO to be the location of the downloaded file
### Modify $VF_APP_ID to be the UUID provided in the VF Server UI's Installation Instructions
### Modify $MY_SERVICE to be the name of the Application in the ApplicationInsights and vFunction UI
-javaagent:/$PATH_TO/opentelemetry-javaagent.jar
-Dotel.exporter.otlp.endpoint=http://collector.example.com:4318
-Dotel.exporter.otlp.traces.headers=X-VF-APP=$VF_APP_ID
-Dotel.service.name=$MY_SERVICE
- Restart the Application
- Confirm functionality in the vFunction Server UI and Azure Monitor
Using the OpenTelemetry Agent and ApplicationInsight Side-by-Side
As can be seen in the OpenTelemetry Collector configuration, vFunction and Azure Monitor both use Trace Exporter details. By using the OpenTelemetry Collector and the OpenTelemetry Agent outlined in the prior option above, the overhead of processing this information is offloaded to the Collector and the details are sent from there onto both vFunction and Azure Monitor. That workflow requires less overhead to be used by any Agent. But, if ApplicationInsights is already in place and cannot be adjusted, the OpenTelemetry Agent can be added to run side-by-side with ApplicationInsights. Obviously, this redundancies between these two Agents will add more overhead to the application.
- Download the OpenTelemetry Java Agent. Note that this needs to be downloaded to the filesystem where the application runs
- Add the OpenTelemetry Java Agent’s Startup Parameters to the Application
### Modify $PATH_TO to be the location of the downloaded file
### Modify the $PROTOCOL and $VF_SERVER_ADDRESS to be relevant to your VF Server installation
### Modify $VF_APP_ID to be the UUID provided in the VF Server UI's Installation Instructions
### Modify $MY_SERVICE to be the name of the Application in the vFunction UI
-javaagent:/$PATH_TO/opentelemetry-javaagent.jar
-Dotel.exporter.otlp.protocol=http/protobuf
-Dotel.exporter.otlp.endpoint=$PROTOCOL://$VF_SERVER_ADDRESS/api/unauth/otlp
-Dotel.traces.exporter=otlp
-Dotel.exporter.otlp.traces.headers=X-VF-APP=$VF_APP_ID
-Dotel.metrics.exporter=none
-Dotel.logs.exporter=none
-Dotel.service.name=$MY_SERVICE
- Restart the Application
- Confirm functionality in the vFunction Server UI and Azure Monitor