Enhancing Performance and Visibility in Phoenix with Spandex and Datadog

Elixir and Phoenix Tracing

Tracing is an essential part of monitoring and understanding the behaviour of distributed systems. By capturing and analyzing traces, we can gain valuable insights into the performance and behaviour of our applications. Spandex is a tracing library for Elixir that provides instrumentation and tracing capabilities for various frameworks and libraries, including Phoenix.

Spandex integrates with popular tracing platforms such as Datadog, allowing you to collect and analyze traces from your Phoenix applications. Traces provide detailed information about requests and their associated operations, enabling you to identify performance bottlenecks, diagnose errors, and optimize your application’s behaviour.

In this tutorial, we will guide you through the process of integrating Spandex with Phoenix and configuring it to send traces to Datadog.

Step 1: Adding Dependencies

To begin, we need to add the necessary dependencies to our Phoenix project. Open the mix.exs file and locate the deps function. Add the following dependencies:

{:spandex, "~> 3.0.3"},
{:telemetry_metrics_statsd, "~> 0.6.0"},
{:spandex_datadog, "~> 1.2"},
{:spandex_phoenix, "~> 1.0"},
{:spandex_ecto, "~> 0.7"},
{:decorator, "~> 1.2"},
{:logger_json, "~> 5.0"}

Save the file and run mix deps.get it, in your terminal to fetch the new dependencies.

Step 2: Configuring Telemetry Metrics StatsD

Next, let’s configure the Telemetry Metrics StatsD library, which Spandex uses for metric reporting. Open the lib/my_app_web/telemetry.ex file and locate the init/1 function. Add the following code to the children list:

{
  TelemetryMetricsStatsd,
  metrics: metrics(),
  host: Application.fetch_env!(:my_app, :datadog) |> Keyword.fetch!(:host),
  port: Application.fetch_env!(:my_app, :datadog) |> Keyword.fetch!(:listning_port),
  prefix: "elixir",
  formatter: :datadog
}

This configuration sets up Telemetry Metrics StatsD to send metrics to the specified host and port. The :formatter option is set to :datadog is to ensure compatibility with Datadog's metrics format.

Step 3: Modifying Telemetry Event Definitions

To capture additional information in our traces, we need to modify the telemetry event definitions. In the same lib/my_app_web/telemetry.ex file, find the event definition for "phoenix.endpoint.stop.duration". Modify it as follows:

summary(
  "phoenix.endpoint.stop.duration",
  tags: [:method, :request_path, :query_string],
  tag_values: &tag_method_and_other_information/1,
  unit: {:native, :millisecond}
)

This change adds the :query_string tag to the event, allowing us to include the query string in our traces.

Additionally, add the following telemetry event definitions below the modified event definition:

summary(
  "phoenix.router_dispatch.exception.duration",
  tags: [:method, :request_path, :query_string],
  tag_values: &tag_method_and_other_information/1,
  unit: {:native, :millisecond}
),

summary(
  "phoenix.error_rendered.duration",
   tags: [:method, :request_path, :query_string],
   tag_values: &tag_method_and_other_information/1,
   unit: {:native, :millisecond}
 ),
# custom web metric
counter(
  "web.fallback.error",
  tags: [:method, :request_path, :query_string],
  tag_values: &tag_method_and_other_information/1
),

These event definitions capture additional metrics related to routing, error rendering, and fallback errors in Phoenix.

Step 4: Updating the `tag_method_and_other_information/1` Function

Now, we need to update the `tag_method_and_other_information/1` function in the same `lib/my_app_web/telemetry.ex` file. Replace the existing function with the following code:

defp tag_method_and_other_information(metadata) do
 Map.take(metadata.conn, [:method, :request_path, :query_string])
end

This function extracts the required information from the conn structure and includes it in the trace tags.

Step 5: Adding SpandexPhoenix and LoggerJSON to the Endpoint

Open the lib/my_app_web/endpoint.ex file and add the following lines at the top of the file:

use SpandexPhoenix
plug LoggerJSON.Plug

The use SpandexPhoenix line includes the Spandex Phoenix integration, enabling automatic tracing of incoming HTTP requests. The plug LoggerJSON.Plug line adds the LoggerJSON plug, which formats logs as JSON for better compatibility with Datadog's logging format.

Step 6: Configuring Application Start

In the lib/my_app/application.ex file, locate the start/2 function. Add the following code at the beginning of the children list:

{SpandexDatadog.ApiServer, spandex_datadog_options()},

This configuration sets up the Spandex Datadog API server, which allows communication between your Phoenix application and Datadog’s tracing agent.

Additionally, add the following lines to attach telemetry events for Spandex Ecto and LoggerJSON:

:telemetry.attach(
  "spandex-query-tracer",
  [:my_app, :repo, :query],
  &SpandexEcto.TelemetryAdapter.handle_event/4,
  nil
)

:telemetry.attach(
  "logger-json-ecto",
  [:my_app, :repo, :query],
  &LoggerJSON.Ecto.telemetry_logging_handler/4,
  :info
)

These configurations ensure that Spandex captures Ecto database queries and LoggerJSON logs as part of the tracing process.

Step 7: Updating Application Configuration

Open the config/config.exs file and locate the configuration for MyApp.Repo. Add the following options to the config block:

config :my_app, MyApp.Repo,
  loggers: [
    {Ecto.LogEntry, :log, [:info]},
    {SpandexEcto.EctoLogger, :trace, ["my_app_repo"]}
  ]

This configuration sets up Ecto to use the SpandexEcto logger for database query tracing.

Next, add the following configuration to enable SpandexPhoenix instrumentation for the endpoint:

config :my_app, MyAppWeb.Endpoint,
  instrumenters: [SpandexPhoenix.Instrumenter]

This configuration ensures that Phoenix requests are instrumented by Spandex.

Step 8: Configuring Logger and LoggerJSON

In the same config/config.exs file, find the configuration for :logger. Replace the existing configuration with the following code:

config :logger,
  backends: [LoggerJSON],
  format: "$dateT$time [$level]$levelpad $metadata $message\n",
  metadata: [:id, :request_id, :mfa, :line, :trace_id, :span_id]

and remove this existing code.

config :logger, :console,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id, :module, :line]

This configuration sets up the Logger to use the LoggerJSON backend and formats log entries according to the specified pattern. It also includes additional metadata fields such as `trace_id` and `span_id`, which will be included in the logs.

Finally, add the following configuration for LoggerJSON:

config :logger_json, :my_app,
  formatter: LoggerJSON.Formatters.DatadogLogger,
  metadata: [:id, :request_id, :mfa, :line, :trace_id, :span_id],
  json_encoder: Jason

This configuration specifies the LoggerJSON formatter, includes the required metadata fields, and sets the JSON encoder to Jason.

Step 9: Additional Configuration

In the config/config.exs file, add the following configuration for :my_app and MyApp.Datadog.Tracer at the end of the file:

config :my_app, :datadog,
  host: "localhost",
  listning_port: 8125, # Datadog agent listens for traces
  default_port: 8126 # Datadog agent sends traces

config :my_app, MyApp.Repo,
  pool_size: 10,
  log: false,
  adapter: Ecto.Adapters.Postgres,
  loggers: [
    {SpandexEcto.EctoLogger, :trace, ["my_app_repo"]}
  ]

config :my_app, MyApp.Datadog.Tracer,
  adapter: SpandexDatadog.Adapter,
  service: :my_app,
  type: :web

config :spandex, :decorators, tracer: MyApp.Datadog.Tracer

config :spandex_ecto, SpandexEcto.EctoLogger,
  service: :my_app_ecto,
  tracer: MyApp.Datadog.Tracer,
  otp_app: :my_app

config :spandex_phoenix, tracer: MyApp.Datadog.Tracer

These configurations specify various settings for Datadog, Ecto, and Spandex. Ensure that the values are set according to your specific environment and requirements.

Step 10: Environment-Specific Configuration

If you have a config/releases.exs file, add the following code to set environment-specific configuration:

config :my_app,
  spandex_batch_size: "SPANDEX_BATCH_SIZE" |> System.fetch_env!() |> String.to_integer(),
  spandex_sync_threshold: "SPANDEX_SYNC_THRESHOLD" |> System.fetch_env!() |> String.to_integer()

config :my_app, MyApp.Datadog.Tracer,
  env: System.fetch_env!("ENV_NAME")

Make sure to replace "SPANDEX_BATCH_SIZE", "SPANDEX_SYNC_THRESHOLD", and "ENV_NAME" with appropriate environment variables or default values for batch size, sync threshold, and environment name.

For the config/dev.exs file, add the following configuration:

config :my_app,
spandex_batch_size: "SPANDEX_BATCH_SIZE" |> System.get_env("10") |> String.to_integer(),
spandex_sync_threshold: "SPANDEX_SYNC_THRESHOLD" |> System.get_env("100") |> String.to_integer()

config :my_app, MyApp.Datadog.Tracer,
  disabled?: true, #Disable tracing in development environment

This configuration disables tracing in the development environment and sets the appropriate batch size and sync threshold values.

Step 11: Starting the Tracing Agent Finally,

start the Datadog tracing agent by running the following command in your terminal:

DATADOG_TRACE_AGENT_ENABLED=true DATADOG_APM_ENABLED=true DATADOG_TRACE_AGENT_PORT=8126 mix phx.server

This command enables the Datadog tracing agent and starts your Phoenix server with tracing enabled. In order to make it works on the server you need to set up an agent on your server, you can find the agent and the instructions to set up your server here.

Conclusion

Congratulations! You have successfully integrated the Spandex tracing library into your Phoenix application and configured it to send traces to Datadog. Traces provide valuable insights into the behaviour and performance of your application, allowing you to identify and resolve issues effectively. Explore the features and capabilities of Spandex and Datadog to gain deeper visibility into your distributed systems and improve the overall performance of your Phoenix application.

Thank you for taking the time to read this post about the impact of Elixir on software development. As an experienced Elixir engineer, I am always excited to share my knowledge and insights with the community.
If you are interested in learning more about Elixir, or if you are in need of an experienced Elixir developer, please don’t hesitate to reach out to me. I am currently seeking new opportunities in the field and would love to hear from you.

Thank you again for your time and attention, and I look forward to hearing from you soon. 🙏