Increasing data accessibility

Hello 👋 Sorry for the lack of newsletter recently, I’ve been on holiday (Southwold 🏖️) and then lots of childcare as we came to the end of the summer holidays leaving very little energy for writing about data!

I’m back now and this week writing about how we increase data accessibility.

There’s also links to articles on RAG architectures, LLMs in search, and agentic data.

Increasing data accessibility

One of the key goals of a data team is to increase the amount of accessible data to the organisation.

We generally do that by having data engineers build (often complex) ETL pipelines to take raw data, clean and refine it, and then make it available to others either directly or through dashboards and other data applications.

But as demand for data grows, particularly as more users are taking advantage of AI, we find we can’t build enough pipelines quickly enough to satisfy all use cases. For example, there are datasets that would not have been worth building pipelines for before, but now that many more people can do some basic analysis, these datasets start to increase in value.

Most of us cannot hire more and more data engineers to meet this demand, and so these data engineers become a bottleneck. We can continue working on the most important datasets as they have the highest priority, but we cannot invest in making the others more accessible.

Having a bottleneck like this prevents the application of data, which of course is not what we want to do! So, how do we remove this bottleneck?

We remove it by reducing the effort to make data accessible, such that it can be done by many more people across the organisation.

For example, we could provide a self-service data platform that makes it easy for a product engineering team to make their data available to PMs and other stakeholders to analyse, by providing capabilities that enable the production, storage, and management of data. That would unlock local use cases such as product/feature analytics, A/B test results, and so on, which may never have got to the top of our priority list, but which have real value for their part of the company.

This federated approach can impact the application of our data governance rules and policies, but again we can invest in platform capabilities to fill this gap. For example, if some of this data is sensitive, you can ensure the platform provides the tools necessary to handle sensitive data, without requiring all data owners to become experts in data regulation.

This is a different approach to the centralised data engineering-led approach we had before, but only by opening up these capabilities can we increase the amount of accessible data, unlocking the long tail of use cases that AI assistance is making possible.

Interesting links

6 RAG Architectures — and How to Avoid Over-Engineering by Rafael Pierre

Nice overview of different RAG architectures and when to reach for them.

Why DoorDash, Instacart, and Uber Eats Integrated LLMs Into Search Three Different Ways

Really interesting to see the different architectures in use to solve similar problems.

Institutional knowledge doesn’t scale: Building an agentic data analyst by Navo Das (Incident.io)

Half the company as weekly active users is impressive.

Being punny 😅

Which country’s capital is the fastest growing? Ireland. Every year, it’s Dublin.

Upcoming events

🆕 Compass AI & Tech Summit, 1-2 October, Budapest. Use the code AndrewSentMe for 20% off!

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew