Integrating Sentry#

Sentry is an exception reporting and tracing observability service. It has great out-of-the-box integrations with many of the Python libaries that we use, including FastAPI, SQLAlchemy, and arq. Most apps can get a lot of value out of Sentry by doing nothing other than calling the init function early in their app and using some of the helpers described here.

Instrumenting your application#

The simplest instrumentation involves calling sentry_sdk.init as early as possible in your app’s main.py file. You will need to provide at least:

  • A Sentry DSN associated with your app’s Sentry project

  • An environment name with which to tag Sentry events

You can optionally provide:

The initialize_sentry function will parse the DSN, environment, and optionally the traces sample rate from environment variables. It will then call sentry_sdk.init with those values and the before_send_handler before_send handler. The config values are taken from enviroment variables and not the main app config class because we want to initialize Sentry before the app configuration is initialized, especially in apps that load their config from YAML files.

Your app’s Kubernetes workload template needs to specify these environment variables and their values, which might look like this:

deployment.yaml#
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "myapp"
spec:
  template:
    spec:
      containers:
        - name: "app"
          env:
            {{- if .Values.sentry.enabled }}
            - name: "SENTRY_DSN"
              valueFrom:
                secretKeyRef:
                  name: "myapp"
                  key: "sentry-dsn"
            - name: "SENTRY_ENVIRONMENT"
              value: {{ .Values.global.environmentName }}
            - name: "SENTRY_RELEASE"
              value: {{ .Values.image.tag | default .Chart.AppVersion }}
            {{- end }}

And your main.py might look like this:

src/myapp/main.py#
import sentry_sdk

from safir.sentry import initialize_sentry
from .config import config

initialize_sentry()


@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator: ...


app = FastAPI(title="My App", lifespan=lifespan, ...)

Special Sentry exceptions#

You can use Reporting an exception to a Slack webhook to create custom exceptions that will send specific Sentry tags, contexts, and attachments with any events that arise from them. You need to use the before_send_handler handler for this to work.

You can define a to_sentry method on any custom exception that inherit from SlackException. This method returns a SentryEventInfo object with tags, contexts, ant attachments attributes. These attributes can be set from attributes on the exception object, or any other way you want. If Sentry sends an event that arises from reporting one of these exceptions, the event will have those tags, contexts, and attachments attached to it.

Note

Tags are short key-value pairs that are indexed by Sentry. Use tags for small values that you would like to search by and aggregate over when analyzing multiple Sentry events in the Sentry UI. Contexts can hold more text, and are for more detailed information related to single events. Attachments can hold the most text, but are the hardest to view in the Sentry UI. You can not search by context or attachment values, but you can store more data in them. You should use a tag for something like "query_type": "sync", a context for something like "query_info": {"query_text": text}, and an attachment for something like an HTTP response body.

from safir.sentry import initialize_sentry
from safir.slack.blockkit import SlackException


initialize_sentry()


class SomeError(SlackException):
    def __init__(
        self, message: str, short: str, medium: str, long: str
    ) -> None:
        super.__init__(message)
        self.short = short
        self.medium = medium
        self.long = long

    @override
    def to_sentry(self):
        info = super().to_sentry()
        info.tags["some_tag"] = self.short
        info.contexts["some_context"] = {"some_item": self.medium}
        info.attachments["some_attachment"] = self.long


raise SomeError(
    "Some error!",
    short="some_value",
    medium="some longer value...",
    long="A large bunch of text...............",
)

Testing#

Safir includes some functions to build pytest fixtures to assert you’re sending accurate info with your Sentry events.

  • sentry_init_fixture will yield a function that can be used to initialize Sentry such that it won’t actually try to send any events. It takes the same arguments as the normal sentry init function.

  • capture_events_fixture will return a function that will patch the sentry client to collect events into a container instead of sending them over the wire, and return the container.

These can be combined to create a pytest fixture that initializes Sentry in a way specific to your app, and passes the event container to your test function, where you can make assertions against the captured events.

conftest.py#
@pytest.fixture
def sentry_items(monkeypatch: pytest.MonkeyPatch) -> Generator[Captured]:
    """Mock Sentry transport and yield a list of all published events."""
    with sentry_init_fixture() as init:
        init(traces_sample_rate=1.0, before_send=before_send)
        events = capture_events_fixture(monkeypatch)
        yield events()
my_test.py#
def test_spawn_timeout(sentry_items: Captured) -> None:
    do_something_that_generates_an_error()

    # Check that an appropriate error was posted.
    (error,) = sentry_items.errors
    assert error["contexts"]["some_context"] == {
        "foo": "bar",
        "woo": "hoo",
    }
    assert error["exception"]["values"][0]["type"] == "SomeError"
    assert error["exception"]["values"][0]["value"] == (
        "Something bad has happened, do something!!!!!"
    )
    assert error["tags"] == {
        "some_tag": "some_value",
        "another_tag": "another_value",
    }
    assert error["user"] == {"username": "some_user"}

    # Check that an appropriate attachment was posted with the error.
    (attachment,) = sentry_items.attachments
    assert attachment.filename == "some_attachment"
    assert "blah" in attachment.bytes.decode()

    transaction = sentry_items.transactions[0]
    assert transaction["spans"][0]["op"] == "some.operation"

On a Captured container, errors and transactions are dictionaries. Their contents are described in the Sentry docs. You’ll probably make most of your assertions against the keys:

  • tags

  • user

  • contexts

  • exception

attachments is a list of Attachment.