IaC is a prerequisite for automation
Hey, hope your week’s been good!
In this week’s newsletter I write about how infrastructure/configuration as code is a prerequisite for automation, whether you are automating with platform tooling or with agents.
There’s also links to articles on improving agentic analytics, building an Agentic Development Platform, and Apache Ossie.
IaC is a prerequisite for automation
One of the things I’m thinking about at work is how we can use more automation to reduce the support load on the data team, freeing up the time we can spend on delivery.
Some of the solutions I’m thinking about involve more platform tooling and self-serve capabilities, while others could use AI agents to perform multi-step workflows.
A realisation I’ve had while thinking about all of this is how the ability to define infrastructure/configuration as code is a prerequisite for automation.
For example, one of the more common support requests we get is to manage access to our data warehouse. We did that by logging into the admin UI, copy/pasting an email address into an Add User form, and telling them when it was done.
One way to reduce the steps required by the data team for these onboarding/offboarding tasks would be to manage this access in code in a Git repository, and allow managers to create pull requests to define the required access for their team members. If needed, someone from the data team can approve the change, then a pipeline can run to apply the change in the data warehouse.
Having this managed as code gives us:
- (Optionally) an approval step
- Easy roll back
- An audit trail
If we didn’t have those, we would not be able to automate the task.
This is equally true if we’re using an AI agent to further automate some of these steps.
In this case, the user can make the request to the agent (e.g. via Slack, by creating a ticket, etc), and the agent will ensure it has all the context it needs (asking the user for more as needed), make the changes to the configuration and create the pull request. When that pull request is merged, either automatically when validation passes or after a review by a member of the data team, they can let the user know and provide them with links to user documentation.
Again, it is the ability to include approval steps, to roll back erroneous changes, to view the audit trail, that enables us to deploy an agent to implement this automation.
For various reasons we’re not currently using infrastructure/configuration as code as much as we could, and that’s something we will start addressing as we look to deploy more automation with tooling and agents.
Interesting links
Agentic Analytics: How Much Does Grounding Actually Buy You? by Dmitry Ustimov
Interesting analysis on what makes an impact when trying to improve agentic analytics.
What is an Agentic Development Platform (ADP) by Kaspar von Grünberg
I’m also starting to think about what kind of platform would support faster and safer development of agents. I like a lot of what is being suggested in here.
Apache Ossie (Incubating): The New Name for Open Semantic Interchange by Josh Klahr
Good to see the project come under the Apache Software Foundation.
Being punny 😅
My friend Sid was a sudden ID theft victim. Now they’re just S.
Upcoming events
- Data Community Conference, September, Online
Thanks! If you’d like to support my work…
Thanks for reading this weeks newsletter — always appreciated!
If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.
Enjoy your weekend.
Andrew