Self-service platforms run on context

Hello! Today I’m writing to you from the Compass AI and Tech Summit in Budapest, where yesterday I spoke about Implementing a self-service data platform. I think it went well :) Video should be online in the next couple of weeks.

This weeks newsletter comes from that talk, and looks at the context required to build a self-service data platform.

There’s also links to semantic layers and headless BI, BigQuery graph as a semantic layer, and the easiest tracking setup in the world.

Self-service platforms run on context

The goal of a self-service platform is to allow its users to request the capabilities they need through a self-serve interface so they can deliver at their own pace, without waiting for someone to action a ticket and manually provide that capability.

But for the platform to deliver the capability, it needs to know a bit about what you’re building and what its requirements are.

For example, an internal developer platform (IDP) might provide the capability to deploy a service to a compute cluster with autoscaling and metric collection. Before it can provide that capability, it needs to know:

• What the CPU and memory requirements are
• What metric to monitor to trigger autoscaling, and how many nodes to scale to
• What metric collector to use

These requirements are often provided in a YAML file, such as the following:

flaskapp:
  name: flask-app-api
  resources:
    limits:
      cpu: 128m
      memory: 500Mi
    requests:
      cpu: 128m
      memory: 500Mi
loadbalancer:
  enabled: true
autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
prometheus:
  enabled: true

This can then be used by a tool such as Helm to deploy that service to Kubernetes.

To build a self-service data platform we also need to know about the data that is being created and its requirements.

For example, for our platform to provision a table in a data warehouse, run some data quality checks, and anonymise data when it is past its retention period, it would need to know:

• The schema, including data types
• The data quality checks to run, and where to send alerts
• Which fields are sensitive and how to anonymise them

These requirements and context are provided by the data contract, such as in the following example, which is based on ODCS:

version: 1.0.0
name: Customer
schema:
  - name: customer
    description: A customer of our e-commerce website.
    properties:
      - name: id
        logicalType: string
        description: The unique identifier for the customer.
        required: true
      - name: name
        logicalType: string
        description: The name of the customer.
        required: true
        classification: restricted
        customProperties:
          - property: anonymizationStrategy
            value: hex
      - name: email
        logicalType: string
        description: The email address of the customer.
        required: true
        quality:
          - metric: invalidValues
            mustBe: 0
            unit: rows
            arguments:
              pattern: "^[a-zA-Z0-9._%+-]+@...$"
        classification: restricted
        customProperties:
          - property: anonymizationStrategy
            value: email
      - name: language
        logicalType: string
        description: The language preference of the customer.
        quality:
          - metric: invalidValues
            mustBe: 0
            unit: rows
            arguments:
              validValues: [en, fr, es]

It’s that context, provided by the data producer, which allows us to build out the data platform capabilities to support the creation, governance, and consumption of data.

Interesting links

Your Data Needs A User Manual: Semantic Layers and Headless BI by Nathan Derave and Pascal Knapen (Dataminded) (YouTube)

I enjoyed this webinar earlier this week. Balanced look at the options for semantic layers and their pros/cons.

BigQuery Graph as a semantic layer by Eugen Sopot

Also on semantic layers, here’s an example building one with the relatively new BigQuery Graph feature.

The easiest tracking setup in the world by Timo Dechau

Not new but new to me. Focus on what you need, not what you can collect.

Being punny 😅

Solar power is the future, but it won’t happen overnight.

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew