Log Drains

You can use log drains to send your application’s logs to an external server with the following command.

clever drain create DRAIN_TYPE DRAIN_URL

Where DRAIN-TYPE is one of:

• betterstack: for Better Stack endpoint (note that this endpoint needs your source token)
• datadog: for Datadog endpoint (note that this endpoint needs your Datadog API Key in its URL)
• elasticsearch: for Elasticsearch endpoint (note that this endpoint needs an index prefix, and has optional username/password parameters as HTTP Basic Authentication)
• newrelic: for New Relic endpoint (note that this endpoint needs your New Relic API Key)
• ovh-tcp: for OVH TCP syslog endpoint (note that this endpoint has an optional sd-params parameter)
• raw-http: for HTTP endpoint (note that this endpoint has optional username/password parameters as HTTP Basic Authentication)
• sekoia: for Sekoia.io HTTPS intake endpoint (note that this endpoint needs your intake key)
• splunk: for Splunk HTTP Event Collector endpoint (note that this endpoint needs your HEC token)
• syslog-tcp: for TCP syslog endpoint
• syslog-udp: for UDP syslog endpoint

Each drain type has its own options, listed in the Clever Tools logs drains documentation. Add --alias ALIAS or --app APP_ID_OR_NAME to target a specific application. You can list the currently activated drains with this command, and add --type to only list the drains of a given type. To list the drains of every application and add-on of your organisations at once, use clever drain list:

clever drain
clever drain --type datadog
clever drain list

Check that the recipient of a drain is reachable and accepts deliveries, and remove it if needed:

clever drain check DRAIN_ID
clever drain remove DRAIN_ID

If the status of your drain appears as DISABLED without you disabling it, it may be because it haven’t been able to send your application logs to your drain endpoint or because the requests timed out after 25 seconds.

Use the logs drain to send your add-on’s logs by using --addon flag, the value must be the add-on ID starting by addon_, or its real ID such as postgresql_xxxxxxxx.

Elasticsearch

Elasticsearch drains use the Elastic bulk API. To match this endpoint, specify /_bulk at the end of your Elasticsearch endpoint, and set an index prefix with --index-prefix. For example, with the credentials of a Clever Cloud Elasticsearch add-on:

clever drain create elasticsearch "https://$ES_ADDON_HOST/_bulk" --index-prefix logstash --username "$ES_ADDON_USER" --password "$ES_ADDON_PASSWORD"

Each day, the drain creates an index named after this prefix, logstash-YYYY-MM-DD in this example, and pushes logs to it.

Index Lifecycle Management

Depending on the amount of logs generated by your application, you might want to manage the lifecycle of your log indexes to prevent your Elasticsearch instance from running out of storage space.

To do so, Elasticsearch provides a feature called Index Lifecycle management that allows you to create a policy to delete indexes based on their creation date.

With our Elasticsearch add-on, you can choose to create a Kibana application in which you can create the policy and apply it to your indexes with an index template, but you can create them manually through API requests.

Here is an example that will create a policy to delete indexes older than 30 days:

curl -X PUT "https://username:password@xxx-elasticsearch.services.clever-cloud.com/_ilm/policy/logs_drain?pretty" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
'

An index template example to apply the policy based on an index pattern:

curl -X PUT "https://username:password@xxx-elasticsearch.services.clever-cloud.com/_index_template/logs_drain?pretty" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logstash-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs_drain"
    }
  }
}
'

For more information, please refer to the official documentation.

Datadog

To create a Datadog drain, use your Datadog API key in the intake URL:

clever drain create datadog "https://http-intake.logs.datadoghq.com/v1/input/$DATADOG_API_KEY?ddsource=clevercloud&service=myapp"

NewRelic

To create a NewRelic drain, use:

clever drain create newrelic https://log-api.eu.newrelic.com/log/v1 --api-key "$NEW_RELIC_API_KEY"

Better Stack

To create a Better Stack drain, use the ingesting host of your source, along with its source token:

clever drain create betterstack "https://$BETTERSTACK_INGESTING_HOST" --source-token "$BETTERSTACK_SOURCE_TOKEN"

Sekoia.io

To send logs to Sekoia.io, use the /jsons endpoint of the HTTPS intake matching the region of your workspace, along with the intake key. On regions other than FRA1, the URL is https://intake.REGION.sekoia.io/api/v1/intake-http/jsons:

clever drain create sekoia https://intake.sekoia.io/jsons --intake-key "$SEKOIA_INTAKE_KEY"

Sekoia.io has no built-in intake format for Clever Cloud logs: create the intake with a custom format that parses the fields of the forwarded logs.

Splunk

To send logs to a Splunk HTTP Event Collector (HEC), use the full collector endpoint and the token bound to your HEC input:

clever drain create splunk https://splunk.example.com:8088/services/collector/event --hec-token "$SPLUNK_HEC_TOKEN"

The --index and --sourcetype options override the values configured on the HEC token. If your self-hosted Splunk instance still uses its default self-signed certificate, add --tls-verification trustful.

OVHcloud Logs Data Platform

To export logs from an application or an add-on to OVHcloud Logs Data Platform, use the following setup:

• A TCP drain log with clever drain create ovh-tcp
• Your Logs Data Platform host with port 514 (SSL ports aren’t supported for TCP drains)
• The write token for your stream (provided on your Logs Data Platform console)

On your terminal, use the following command:

clever drain create ovh-tcp "tcp://$LDP_HOST:514" --app APP_ID_OR_NAME --sd-params "X-OVH-TOKEN=\"$LDP_WRITE_TOKEN\""

clever drain create ovh-tcp "tcp://$LDP_HOST:514" --addon ADDON_ID --sd-params "X-OVH-TOKEN=\"$LDP_WRITE_TOKEN\""

Manage drains through the API

Clever Tools relies on the /v4/drains endpoints of the Clever Cloud API, which you can call directly from your own tools. Drains belong to a resource: an application ID, or the real ID of an add-on such as postgresql_xxxxxxxx. To create one, send a POST request to /v4/drains/organisations/{ownerId}/resources/{resourceId}/drains:

{
  "kind": "LOG",
  "recipient": {
    "type": "RAW_HTTP",
    "url": "https://logs.example.com/ingest",
    "username": "drain-user",
    "password": "drain-password"
  }
}

Set kind to LOG for application logs or ACCESSLOG for access logs. The recipient.type value sets which fields the recipient accepts, besides type and url:

tlsVerification is either DEFAULT, which verifies the certificate against the default trust store, or TRUSTFUL, which skips certificate and hostname verification. Use a tcps:// URL to enable TLS for OVH_TCP and SYSLOG_TCP recipients: a tcp:// URL stays unencrypted, and the API refuses TRUSTFUL on it. rfc5424StructuredDataParameters holds RFC 5424 structured data, such as X-OVH-TOKEN="…".

The API probes the recipient before it saves the drain: an unreachable recipient fails the creation with a 412 status. Add ?skipCheck=true to the request to create the drain without this probe.

The other endpoints act on existing drains, under the same /v4/drains/organisations/{ownerId} prefix:

Both list endpoints accept the optional status, executionStatus and executionStatusNotIn query parameters to filter the results. The check result uses a 200 status even when the probe fails: read its ok field, then its code and message fields to diagnose a failure. A SYSLOG_UDP recipient sends no acknowledgement, so its check returns ok with the skipped code.

Community software

HTTPS-based solution

Some tools available on GitHub enable to create a drain to collect logs through an HTTPS endpoint. This project, for example, is fully compatible with Clever Cloud.

You could host it as an app and an add-on on Clever Cloud. A complete README explains all the features.