Erstes Cloud Native Linz Meetup

Constantin Winkler:
Welcome everybody. This will be a talk about an open source approach to multi-tenant label-based access control.

My name is Constantin Winkler. I’m a DevOps engineer and 21 years old. I started one and a half years ago at GCH and primarily joined the GCH Run team, which is our cloud offering.

Let’s do a short history of OpenShift.
This is OpenShift 4.10, specifically the monitoring stack.
And this is OpenShift 4.11.

Can you spot the difference?

Once again:
4.10 … and 4.11.

There’s something missing – Grafana.
Effectively, Red Hat removed Grafana and instead offered metrics directly in the OpenShift console. These metrics are feature-rich, but they’re missing something crucial: user-customizable dashboards.

This is powerful because application-specific dashboards can immediately show what’s wrong with your application. Right now, you cannot do that in OpenShift.

So we took Red Hat’s advice to “DIY” – or, as engineers usually say: deploy it yourself.
We took Grafana and pointed it to Prometheus – technically Thanos – inside OpenShift.

We didn’t want to give everyone access to Grafana, so we put a proxy in front of it.
For most applications, the OAuth2 proxy is enough, but we faced an issue: OAuth2 proxy didn’t work well with Grafana group names.

Here you can see how Grafana maps groups.
We wanted:

• GCH Run admins → Grafana admins

• Developers → editors

• Everyone else → viewers

So we switched out the OAuth2 proxy for Keycloak.

Everything worked well… but not so fast.

By doing this, we basically gave everyone access to all metric information in the OpenShift cluster.
If you like living dangerously, that has some spice to it.

Everyone could search alerts, and if you have multiple customers on one cluster, seeing alerts from other customers is definitely not ideal. With this solution, we effectively reduced our security to this.

Grafana offers something to help with that: label-based access control.
But you have to pay for it.

So what is LBAC?
It stands for label-based access control. I’ll give you a short deep dive based on a Prometheus query.

Here we see a normal metric in Prometheus:
currently allocated bytes from a Go application.

This is the metric name, but you probably have many Go applications in your cluster. So you specify which one using a label selector. Labels are key-value pairs, for example container=mailer, combined with operators like equals or matches. These are called label matchers.

To secure queries sent by Grafana, we append another label, for example namespace=tenant1.
Now users only see information from their own namespace.

If a user tries to query another tenant, it should return an error.

OpenShift already has this capability with the Prometheus label proxy, part of the monitoring stack. A potential solution is configuring Grafana data sources with HTTP headers or URL parameters that define enforced labels.

But this didn’t solve the problem – we again had multiple data sources visible to everyone.

So our first real iteration was this:

• Take the request

• Identify the user

• Look up allowed labels from a label provider

• Append the labels

• Forward the request to the Prometheus label proxy

The label provider can be:

• A YAML file (for example via ConfigMap), or

• A MySQL database

We originally built this proxy together with a partner operator whose permission system was stored in MySQL.

Multina looks up which labels a user is allowed to see, inserts them into the request header, sends the query to the Prometheus label proxy, which enforces the labels.

But we had another problem: multiple namespaces.

For a long time, the or operator was not supported. You couldn’t query multiple namespaces like dev OR qa. There was an open pull request for almost two years, but it never got merged.

So we forked the fork and deployed a modified Prometheus label proxy that supports multiple namespaces.
This is very useful because you can compare, for example, CPU usage between QA and production.

If a user tries to access a namespace they’re not allowed to, the proxy returns an error:
“Label tenant1-prod is not in your allowed label list.”

That was the first iteration of Multina.
It worked well for metrics.

But with OpenShift 4.11, Red Hat didn’t only remove Grafana – they also removed the ELK stack. Kibana disappeared and logging moved into the OpenShift console. They introduced Vector and Grafana Loki.

This is great – Vector is written in Rust, and Loki is one of the best log search engines in my opinion. We got aggregated logs and pod-level logs, which is very helpful in large microservice landscapes.

But again: no customizable dashboards.

So we extended Multina.

After research, we found the Observatorium project. They also build a multi-tenant gateway for observability data. Interestingly, they use the same approach as the Prometheus label proxy: parsing queries using a YAK grammar – Yet Another Compiler.

Observatorium builds a full abstract syntax tree.
We used Observatorium as a dependency, pulled it into Multina, and inspected all label selectors for invalid or forbidden queries.

That’s how we achieved Multina support for both Prometheus and Loki.

Recently, DeepL reached out to us because they have similar multi-tenant challenges and were interested in Tempo support. We’re considering supporting Tempo in the future and removing some code dependencies.

How does Multina know who the user is?

We use Keycloak in front. Grafana has a “forward identity” feature that sends a signed JWT. We verify the JWT, retrieve the certificate, and validate the user identity.

That’s how we achieved secure multi-tenancy.

The last part is the Multina RBAC collector.

You don’t want to manually maintain label mappings. Initially, we tried Kubernetes SubjectAccessReviews, but with dashboards containing many panels and namespaces, this resulted in dozens of API calls and very slow dashboards.

So instead, we retrieve all roles, role bindings, and cluster roles once, resolve permissions ourselves, and cache them.

This reduced runtime from about 6 minutes to 1.4 seconds.

Thank you for your attention.

Are there any questions?

Question: How would you get rid of the Observatorium dependency?

Constantin Winkler:
The main problem is that Tempo doesn’t yet provide a parser for this. But Grafana Tempo is open source and already defines a YAK grammar. We could reuse that grammar and build our own parser so that enforcement works consistently across Prometheus, Thanos, and Tempo.

Question: Is Multina open source?

Constantin Winkler:
Yes, of course.

Question: Why was DeepL interested?

Constantin Winkler:
They have similar multi-tenant challenges across their clusters, especially with Tempo and Loki deployments.

Thank you.

[Applause]

Katharina Sick:
So, welcome to my talk, which is about “Pathways, not pitfalls: building platforms with developers in mind.” But before we jump into that, I actually want to talk about what platforms are, because the term platform is quite overloaded.

Today we are going to focus on so-called developer platforms or productivity platforms. These platforms have one main goal: to reduce the cognitive load of developers, and by that to allow them to focus on innovation.

How is this done? In a very small nutshell: you pull all the repetitive tasks around building, testing, deploying, and observing applications into one central hub—one central platform. Then developers can consume those building blocks in a self-service manner.

Basically, what we want to do is create a self-service heaven for developers. And I really like the definition from Thoughtworks, who put it much nicer than me:

Platforms are a means of centralizing expertise whilst decentralizing innovation to the user or customer.
This means we centralize the expertise about building, deploying, observing, and so on into the platform, but we decentralize innovation to the developers—so they can still work in their own way.

Today we’re going to take a look at developer platforms: how they can be built, at least on a high level, and how users can interact with them.

But before we do that—who am I to talk about this? My name is Katharina, or actually Katti. According to my job title, I’m a software engineer with “dino trees”—but I have fully merged into the fascinating world of DevOps, cloud-native, and platform engineering. And I have to say: I really enjoy what I’m doing. But that’s enough about me.

Let’s go back to platforms.

When we build platforms, we offer self-service capabilities for developers. So what’s the catch? Platform engineers are very often people with operations and DevOps backgrounds—but suddenly we are offering an interface to developers so they can consume the platform’s capabilities.

So suddenly, we as platform engineers also need UI/UX skills, because in the end every platform offers a user interface—somehow.

This can be:

• a command line interface,

• an HTTP API,

• a Git repository (if you’re doing GitOps),

• a developer portal like Backstage (or Port, or Cortex),

• or a combination of those, or anything else that makes sense for your organization.

And yes: a command line interface is a user interface too.

By building that interface, we decide over the success of our platform—because getting the UI right is crucial for adoption. If you get it right, more developers will use your platform and it becomes successful. If you get it wrong, you might end up like those people on Reddit who spent months building their platform… and developers hate it—and they don’t know why.

That’s platform engineering: you can’t just sit in your room for months and build what you think is useful. You really have to talk with developers, talk to your users, get their requirements, and build something they actually want to use.

So how can you build a successful platform?

The first thing I have to tell you is the thing we all hate to hear: it depends. You need to find your own needs because every platform is different. If it didn’t need to be different, I don’t think so many companies would be building platforms right now.

A developer platform needs to respect:

• the domain it’s built in,

• the use cases of your company,

• your needs, requirements, and stakeholders.

Find that—then get focused and define a mission. Think about:

• which problem you want to solve,

• how you want to solve it,

• and also which problems you do not want to solve (or not yet).

And when thinking about the mission, focus on the problems, not the tools, especially in the beginning. I’ve had many platform discussions where after two or three minutes it quickly becomes about tools—because we all like tools. But we should step back and think about what we want to do first, and only then about how we want to do it.

Next: design for a good user experience. I’m not talking about pretty UIs—although I like them. I’m talking about UX in the sense that if a user wants to deploy an application, they shouldn’t have to read pages and pages of manuals. The platform should guide them toward deploying the workload. The manual should only be an addition.

Next point: start small. We’re in the lucky position that developers have built products for years or decades—so why not learn from them?

In product development there is the concept of the MVP, the minimum viable product: start small, ship, get feedback, iterate, improve, repeat.

The same idea exists for platform engineering: the FVP, the fittest viable platform. You start with something very small. It could even be just a wiki page describing how to use a cloud provider. Get people to use it, get feedback, improve, iterate. You will get much better feedback than if you build something for months in isolation.

Last: make your platform observable. Platforms are complex, so you will run into errors sooner or later. If it’s observable, you realize problems faster—ideally before users tell you—find the root cause, and fix it.

Also, observability helps you understand usage:

• how many people use the platform,

• which features are used,

• which features are useless,

• and where people drop off, so you know what to improve.

So we have five points:

1. find your needs,

2. define a clear mission,

3. design for good UX,

4. start small,

5. make it observable.

All of them sum up to one core idea: treat your developer platform as a product.

You don’t have external customers; your customers are developers. But you should still treat them like external customers—because they decide the success of the platform by using it or not using it.

And the nice thing: you can talk to them. You can schedule coffee calls—hard to do sometimes, but easy to get started. Talk to them, ask for their needs, and your platform will get much better.

If you’re interested, I’ve also done a longer talk that contains this introduction in 30–40 minutes. There’s a QR code; I’ll also upload the slides later.

That’s it about platforms.

Now, let’s talk about abstraction.

Users need to interact with the platform in a self-service way, so we need to provide interfaces. And to provide these interfaces, we can use abstraction.

Imagine you go to a restaurant, order a burger—and instead of getting a burger, you get bread, lettuce, meat, tomato, cheese… maybe a stove and a pan, and you have to prepare it yourself. Not great.

Now imagine another restaurant: you order a burger and you get a burger. Much nicer.

Platforms are similar. A platform is not just the ingredients; it’s how the components are connected into something meaningful.

As Gregor Hohpe puts it: the value of the platform doesn’t only derive from the scope—the elements it contains—but also from how these pieces come together to form a meaningful whole.

For me, this meaningful whole is the most important thing. It’s better to have fewer features that are connected well than a huge set of ingredients that users don’t know how to use.

So how can you connect components into one whole? This is where abstraction helps.

Abstraction can be done with many tools. At Dynatrace we heavily use Crossplane and Backstage for that. There are alternatives: CUE and the Application Model, Helm (I personally don’t like Helm that much), or other portals like Port or Cortex. Today I’ll focus on Crossplane and Backstage—both open source and widely used—but the concepts apply to other tools too.

Let’s start with Crossplane.

Many people know Crossplane as “infrastructure as code.” That’s the first superpower: it’s a Kubernetes operator you install in the cluster. Then you can create custom resources that represent infrastructure from different providers: Google Cloud, Azure, AWS, but also GitHub repositories, and much more. You can even manage on-prem infrastructure via your own provider.

How does it look to create a database in GCP?

First, you create a database instance. The API version comes from the provider, the kind is DatabaseInstance, it has a name. Then you configure things like: PostgreSQL, deletion protection, replication, disk size, instance size, backups, networking, and much more.

Then you create a database resource that references the instance.

After you kubectl apply those resources, Crossplane reconciles them and creates the real infrastructure in Google Cloud.

But: even though this is “as code,” you still need lots of GCP knowledge (networking, backups, instance types) and some Kubernetes knowledge (how to apply, where to push, etc.).

We can make this simpler.

If a developer wants to create an application with a database, they also need deployment, service, ingress, DNS, etc. That’s a lot to understand, especially if their main focus is software development. They need to understand cloud internals and Kubernetes internals—and how everything connects. And they need to troubleshoot.

We as platform engineers can do better.

We can combine all of that into a single type called Application. That’s the second, even more important superpower: composition.

You compose a set of resources into one simple resource. For the user it looks like: a small YAML that says:

• kind: Application

• name: my fancy app

• domain: where it should be reachable

• database: yes

• disk size: X

The developer does not need to know service/ingress/networking details in most cases. This is the “burger” instead of the ingredients.

Now: how can abstraction be done in Backstage?

A disclaimer: it’s arguable if this is abstraction, but it simplifies the user interface.

Backstage is a developer portal that aims to be the UI of your platform. It has:

• a service catalog,

• documentation support,

• lots of plugins,

• and the scaffolder.

In the scaffolder, you provide templates. Users select a template, enter inputs, and Backstage runs steps: generate files, publish to GitHub, register it in the catalog—done in a few seconds.

That’s nice because templates are discoverable and searchable.
But it can still be painful later: if you template all the low-level resources into the repo, users still have to deal with all the “ingredients” when changing things later.

So what I really like is using Crossplane and Backstage together:

• Backstage gives discoverability and UI for creation,

• but instead of templating everything into the repo, you deploy the Crossplane “Application” composition.

That way: creation is easy, and updating stays easy.

Now, enough about tools. Let’s go back to abstraction itself.

Making things simpler isn’t always simple. When I first heard about Crossplane and Backstage, I wanted to abstract everything—but I soon noticed I did too much.

So here are five important points when doing abstraction:

1. Focus on your user.
   Talk to developers. Find needs. Build for their problems and use cases.

2. Respect the domain.
   Use the language of the domain—like domain-driven design. Naming is hard, but we should try to create good names. Build outside-in: focus on use cases, not components.

3. Provide opinions, not restrictions.
   If you hide too much, you create one thin “path” that works only for one use case. A little left or right, the experience drops to zero. Instead, provide defaults (opinions) but allow overrides. And you can offer multiple opinions for different teams.

4. Be transparent.
   If you restrict something, document and communicate why. It’s easier to accept constraints when you understand the reason. Also: if you create an abstraction like “Application,” allow developers to understand what’s underneath if they want to.

5. Show the right information at the right time.
   At creation time you want inputs. When running, you want observability and performance. If something fails, you must propagate errors. Worst case: hide errors and users don’t know what happened. Better: propagate raw errors. Best: provide good, actionable error messages—maybe even prevent leaks. But abstractions will leak sometimes, so be prepared.

If I had known those points when I started, I wouldn’t have tried to restrict everybody.

I come from mobile engineering. Back then I just wanted to write code; I didn’t care what happens after my pull request. I used that mental model for all developers—and that’s not how it works. We’re all different. We have different workflows and expectations. Platform engineers should respect that and keep it up to developers how they want to use the platform.

So: how do you do abstraction?

Long story short: it depends. You need to find out about your users. Start small with the fittest viable platform, improve iteratively. Then if you do something wrong, you notice early and can adjust direction.

We covered:

• what platform engineering is,

• fittest viable platform,

• abstraction,

• and how abstraction can become too much.

Resources:

• Platform Strategy by Gregor Hohpe (highly recommended; surprisingly fun to read)

• Team Topologies (fittest viable platform mentioned)

• Domain-Driven Design

• DevOps Toolkit (Victor Farcic on YouTube)

• Thoughtworks blog and podcasts

• and most importantly: conversations with people like you.

Keep it up—let’s grab a drink afterwards. If you have questions, feel free to ask. Thank you.

---

Q&A

Audience:
You mentioned Backstage a lot. I tried to look into it. It looks nice and promising, but it seems expensive to maintain. How are you doing that?

Katharina Sick:
It needs quite a lot of people to maintain it, yes. We’re a team of seven people, and about half of our focus is on Backstage: building features, testing plugins, integrating them. It needs a lot of focus, especially if you want to customize it to your company, which you should do in my opinion.

Audience:
How do you deal with updates if you customize it? You’re not really forking it, but you use lots of Node dependencies including Backstage itself, and you update them.

Katharina Sick:
We update most of that with Renovate. If we have problems, Backstage provides nice CLI commands and you can use them. It works quite well to keep it updated.

Audience:
Do you know how many developers have an idea of everything behind it?

Katharina Sick:
Unfortunately not. We are not that far yet with abstraction. A lot of the points I mentioned are from books. We’re also not as far as we should be with observability, so we’re still in the process of finding that out. Maybe in the next few meetups I can give a better answer.

Audience:
You mentioned Crossplane applications. Can I think of those like Terraform modules?

Katharina Sick:
Yes, pretty much. You compose a set of resources into a smaller thing. The output isn’t Terraform, but a Kubernetes YAML / custom resource that you apply. Conceptually it’s very similar.

Audience:
I saw you talk a few months back at Dynatrace. Back then Crossplane was still kind of beta. Do you see improvements?

Katharina Sick:
A lot. Especially now, 1.15 was released—one, two, three weeks ago—and 1.14 already had many improvements. It feels more stable and mature. There are still some problems we fight from time to time, but it improved a lot since last year.

Audience:
Are you using it for production deployments?

Katharina Sick:
Not yet, but we’re getting there—getting close.

Audience:
You talked about opinions and opening them to users. Can you measure which opinions or paths are taken?

Katharina Sick:
We haven’t done it yet. But ideas: with Backstage you could track how many people use a template, and how many complete creation. With Crossplane you could track how often each composition is deployed. That’s what comes to my mind, but we still need experience there.

Audience:
In your example, an “application with database” seems like something everyone wants. Are there higher-level abstractions already provided by Upbound or the community, or does everyone define their own applications?

Katharina Sick:
There are examples, especially from Upbound and from the DevOps Toolkit channel—like for EKS clusters. They have reference architectures that are similar across companies, and you can use them or get inspiration. For microservice applications it’s harder: there are apps out there, but because you need to build for your domain, I think you’ll need custom applications. For clusters like EKS, it’s complex, but possible—there are good examples. For smaller apps with database and observability, it’s much easier to write compositions, and the documentation is very good.

Audience:
You said you’re not far yet on monitoring. Do Crossplane or Backstage integrate with OpenTelemetry?

Katharina Sick:
Crossplane relies on Prometheus metrics right now. There are GitHub issues discussing a migration to OpenTelemetry, but I think it’s discussion only so far. Crossplane itself and many providers, especially those by Upbound, expose Prometheus metrics that you can scrape.

For Backstage, they also offer some metrics, but I don’t remember which kind right now—sorry. For us, standard Kubernetes monitoring works well: events, errors, HTTP requests failing, CPU, memory, things like that.

Audience:
Awesome, thank you.

Katharina Sick:
Cool, thank you.