Instrument your application with OpenTelemetry

lab

This procedure is part of a lab that teaches you how to instrument your application with OpenTelemetry.

Each procedure in the lab builds upon the last, so make sure you've completed the last procedure, Set up your lab environment, before starting this one.

Your Python application is running and getting a lot of traffic, so it's time to instrument it. Using OpenTelemetry provides freedom for you to choose your preferred observability platform—New Relic, of course. Also, because it's open source and supported by many actors in the observability space, you can trust its development and authority as a standard, knowing that it will be broadly supported into the future.

Here, you manually instrument your application to generate traces with the OpenTelemetry API. Then, you configure the SDK to send them to New Relic so you can analyze the results later.

Instrument your application

Step 1 of 8

In the terminal window that's running your simulator, press <CTRL-C>.

You should see your simulator shut down. Now you can add some dependencies and update your app logic.

Step 2 of 8

Install the OpenTelemetry SDK and supporting packages into your virtual environment:

bash

$pip install opentelemetry-api
$pip install opentelemetry-sdk
$pip install opentelemetry-exporter-otlp-proto-grpc

Now that you've installed your dependencies, you need to use those dependencies to instrument your application.

Step 3 of 8

Instrumenting your application begins with a tracer provider. A tracer provider is used for holding configurations and for building tracers. Tracers are then used for creating spans. Spans collect information about an operation or process.

In db.py, create a tracer provider:

1
import logging
2
from opentelemetry import trace
3
from opentelemetry.sdk.resources import Resource
4
from opentelemetry.sdk.trace import TracerProvider
5

6
provider = TracerProvider(
7
    resource=Resource.create({"service.name": "speedeedeebee"})
8
)
9
trace.set_tracer_provider(provider)
10

11
class DuplicateKeyError(Exception):
12
    pass
13

14
class KeyDoesNotExistError(Exception):
15
    pass
16

17
db = {}
18

19
def read(key):
20
    """Read key from the database."""
21
    global db
22

23
    try:
24
        value = db[key]
25
        logging.debug("Successful read")
26
        return value
27
    except KeyError as ke:
28
        msg = f"Key `{key}` doesn't exist"
29
        logging.debug(msg)
30
        raise KeyDoesNotExistError(msg)
31

32
def create(key, value):
33
    """Write key:value to the database."""
34
    global db
35

36
    if key in db:
37
        msg = f"Key `{key}` already exists"
38
        logging.debug(msg)
39
        raise DuplicateKeyError(msg)
40

41
    db[key] = value
42
    logging.debug("Successful create")
43
    return value
44

45
def update(key, value):
46
    """Update key in the database."""
47
    global db
48

49
    if key in db:
50
        db[key] = value
51
        logging.debug("Successful update")
52
        return value
53

54
    msg = f"Key `{key}` doesn't exist"
55
    logging.debug(msg)
56
    raise KeyDoesNotExistError(msg)
57

58
def delete(key):
59
    """Delete key from the database."""
60
    global db
61

62
    if key in db:
63
        del db[key]
64
        logging.debug("Successful delete")
65
        return True
66

67
    return False

db.py

Here, you created a tracer provider with a resource. A resource describes a service as a collection of attributes. In this resource, you specified name of your service as "speedeedeebee". You also configured your API to use your new tracer provider.

Step 4 of 8

Add a span processor, which processes span data before exporting it to a telemetry consumer:

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
class DuplicateKeyError(Exception):
20
    pass
21

22
class KeyDoesNotExistError(Exception):
23
    pass
24

25
db = {}
26

27
def read(key):
28
    """Read key from the database."""
29
    global db
30

31
    try:
32
        value = db[key]
33
        logging.debug("Successful read")
34
        return value
35
    except KeyError as ke:
36
        msg = f"Key `{key}` doesn't exist"
37
        logging.debug(msg)
38
        raise KeyDoesNotExistError(msg)
39

40
def create(key, value):
41
    """Write key:value to the database."""
42
    global db
43

44
    if key in db:
45
        msg = f"Key `{key}` already exists"
46
        logging.debug(msg)
47
        raise DuplicateKeyError(msg)
48

49
    db[key] = value
50
    logging.debug("Successful create")
51
    return value
52

53
def update(key, value):
54
    """Update key in the database."""
55
    global db
56

57
    if key in db:
58
        db[key] = value
59
        logging.debug("Successful update")
60
        return value
61

62
    msg = f"Key `{key}` doesn't exist"
63
    logging.debug(msg)
64
    raise KeyDoesNotExistError(msg)
65

66
def delete(key):
67
    """Delete key from the database."""
68
    global db
69

70
    if key in db:
71
        del db[key]
72
        logging.debug("Successful delete")
73
        return True
74

75
    return False

db.py

The BatchSpanProcessor you used here batches spans before exporting them. This reduces the number of requests you send to New Relic.

Within this span processor, you also configured a span exporter. The exporter is in charge of serializing and sending spans to the consumer. Here, you used OTLP, OpenTelemetry's exchange protocol, and Gzip compression to efficiently transport your telemetry data to New Relic.

You configure your tracer provider with this processor logically prior to setting your tracer provider in the API.

Step 5 of 8

Create two environment variables that you use to configure your OpenTelemetry pipelines. Don't forget to replace the license key placeholder with your real one:

bash

$export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net:4317
$export OTEL_EXPORTER_OTLP_HEADERS="api-key=<YOUR-LICENSE-KEY>"

The OpenTelemetry Protocol (OTLP) endpoint is the url of our OpenTelemetry receiver. Your service sends data directly to New Relic through this endpoint.

Important

https://otlp.nr-data.net:4317 is our US endpoint. If you're in the EU, use https://otlp.eu01.nr-data.net/ instead.

There are several different types of API keys to choose from in New Relic that each serve a different purpose. To instrument your application with OpenTelemetry, you need a license key.

The span exporter you configured in the last step automatically uses these standard environment variables.

Step 6 of 8

Create a tracer:

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
tracer = trace.get_tracer(__name__)
20

21
class DuplicateKeyError(Exception):
22
    pass
23

24
class KeyDoesNotExistError(Exception):
25
    pass
26

27
db = {}
28

29
def read(key):
30
    """Read key from the database."""
31
    global db
32

33
    try:
34
        value = db[key]
35
        logging.debug("Successful read")
36
        return value
37
    except KeyError as ke:
38
        msg = f"Key `{key}` doesn't exist"
39
        logging.debug(msg)
40
        raise KeyDoesNotExistError(msg)
41

42
def create(key, value):
43
    """Write key:value to the database."""
44
    global db
45

46
    if key in db:
47
        msg = f"Key `{key}` already exists"
48
        logging.debug(msg)
49
        raise DuplicateKeyError(msg)
50

51
    db[key] = value
52
    logging.debug("Successful create")
53
    return value
54

55
def update(key, value):
56
    """Update key in the database."""
57
    global db
58

59
    if key in db:
60
        db[key] = value
61
        logging.debug("Successful update")
62
        return value
63

64
    msg = f"Key `{key}` doesn't exist"
65
    logging.debug(msg)
66
    raise KeyDoesNotExistError(msg)
67

68
def delete(key):
69
    """Delete key from the database."""
70
    global db
71

72
    if key in db:
73
        del db[key]
74
        logging.debug("Successful delete")
75
        return True
76

77
    return False

db.py

You use this to create spans.

Step 7 of 8

Wrap the logic in each of your database functions with a span:

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
tracer = trace.get_tracer(__name__)
20

21
class DuplicateKeyError(Exception):
22
    pass
23

24
class KeyDoesNotExistError(Exception):
25
    pass
26

27
db = {}
28

29
def read(key):
30
    """Read key from the database."""
31
    global db
32

33
    with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER) as span:
34
        try:
35
            value = db[key]
36
            logging.debug("Successful read")
37
            return value
38
        except KeyError as ke:
39
            msg = f"Key `{key}` doesn't exist"
40
            logging.debug(msg)
41
            raise KeyDoesNotExistError(msg)
42

43
def create(key, value):
44
    """Write key:value to the database."""
45
    global db
46

47
    with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER) as span:
48
        if key in db:
49
            msg = f"Key `{key}` already exists"
50
            logging.debug(msg)
51
            raise DuplicateKeyError(msg)
52

53
        db[key] = value
54
        logging.debug("Successful create")
55
        return value
56

57
def update(key, value):
58
    """Update key in the database."""
59
    global db
60

61
    with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER) as span:
62
        if key in db:
63
            db[key] = value
64
            logging.debug("Successful update")
65
            return value
66

67
        msg = f"Key `{key}` doesn't exist"
68
        logging.debug(msg)
69
        raise KeyDoesNotExistError(msg)
70

71
def delete(key):
72
    """Delete key from the database."""
73
    global db
74

75
    with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
76
        if key in db:
77
            del db[key]
78
            logging.debug("Successful delete")
79
            return True
80

81
        return False

db.py

To capture data about the operations in your database functions, you used the tracer.start_as_current_span() context manager. In it, you specified the name of the span and the kind of span it is. Because it's a database server, you specify trace.SpanKind.SERVER.

The API populates some data about the span for you. You'll see the data it captures when you look at your spans in New Relic.

Step 8 of 8

In the success cases, capture the key that was used for each operation:

1
import logging
2
from grpc import Compression
3
from opentelemetry import trace
4
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
5
from opentelemetry.sdk.resources import Resource
6
from opentelemetry.sdk.trace import TracerProvider
7
from opentelemetry.sdk.trace.export import BatchSpanProcessor
8

9
provider = TracerProvider(
10
    resource=Resource.create({"service.name": "speedeedeebee"})
11
)
12
provider.add_span_processor(
13
    BatchSpanProcessor(
14
        OTLPSpanExporter(compression=Compression.Gzip)
15
    )
16
)
17
trace.set_tracer_provider(provider)
18

19
tracer = trace.get_tracer(__name__)
20

21
class DuplicateKeyError(Exception):
22
    pass
23

24
class KeyDoesNotExistError(Exception):
25
    pass
26

27
db = {}
28

29
def read(key):
30
    """Read key from the database."""
31
    global db
32

33
    with tracer.start_as_current_span("read", kind=trace.SpanKind.SERVER) as span:
34
        try:
35
            value = db[key]
36
            logging.debug("Successful read")
37
            span.set_attribute("key", key)
38
            return value
39
        except KeyError as ke:
40
            msg = f"Key `{key}` doesn't exist"
41
            logging.debug(msg)
42
            raise KeyDoesNotExistError(msg)
43

44
def create(key, value):
45
    """Write key:value to the database."""
46
    global db
47

48
    with tracer.start_as_current_span("create", kind=trace.SpanKind.SERVER) as span:
49
        if key in db:
50
            msg = f"Key `{key}` already exists"
51
            logging.debug(msg)
52
            raise DuplicateKeyError(msg)
53

54
        db[key] = value
55
        logging.debug("Successful create")
56
        span.set_attribute("key", key)
57
        return value
58

59
def update(key, value):
60
    """Update key in the database."""
61
    global db
62

63
    with tracer.start_as_current_span("update", kind=trace.SpanKind.SERVER) as span:
64
        if key in db:
65
            db[key] = value
66
            logging.debug("Successful update")
67
            span.set_attribute("key", key)
68
            return value
69

70
        msg = f"Key `{key}` doesn't exist"
71
        logging.debug(msg)
72
        raise KeyDoesNotExistError(msg)
73

74
def delete(key):
75
    """Delete key from the database."""
76
    global db
77

78
    with tracer.start_as_current_span("delete", kind=trace.SpanKind.SERVER) as span:
79
        if key in db:
80
            del db[key]
81
            logging.debug("Successful delete")
82
            span.set_attribute("key", key)
83
            return True
84

85
        return False

db.py

Here, you use the span you created with the context manager to capture the key used in the operation as an attribute.

Restart your simulator

Now that you've changed the application logic, you need to restart your simulator. Make sure you do this in the same terminal window where you set your environment variables:

bash

$python simulator.py

You've instrumented your application to send traces to New Relic using OTLP. You've also restarted your simulator. Now, it's time to view your data.

lab

This procedure is part of a lab that teaches you how to instrument your application with OpenTelemetry. Now that you've instrumented your app, view your telemetry data in New Relic.