Table of contents
Web3 is more than smart contract platforms. It is a full suite of technologies that enable us to create decentralized platforms. In this post, let's talk about the data layer.
Imagine a way where users can own/control their data. The application no longer exists to collect data but instead provide the best experience and most value to the users. This is what a decentralized data platform promises.
So let's get to understanding data in web3, decentralized data platforms and creating your user-centric data streams with Ceramic.
Why are apps failing us?
We generate data every second. However, we don't own our data. Most applications exist to collect data; that's how the data economy came to be.
WhatsApp has my messages; Twitter has my followers; Amazon has my shopping history. Each of these platforms have their own database(s) where they add my data. This creates several issues and challenges.
I'd highly encourage you to checkout this article by Danny, its a great breakdown of the issue of apps today and an intro to user data in web3, I've referred to it many times. I'll try to explain my view here as well.
Data Economy
The data economy refers to gathering, using and monetizing data. All your actions, messages, decisions are entered into the application's database. As a result, the application gathers immense insights and can use it from targeted ads to showing you certain news.
In many cases, data is the main value proposition of the application.
User Experience
Some people say, "so what if Amazon knows what I buy?" and I'm not here to talk about this, although it could lead to unresolved trust issues. But giving up control and ownership over your data extends beyond these platforms getting to know us; it affects the user experience.
For example, let's consider a typical online shopping experience. You go to Amazon and add your name, phone, and address. You have now set up your account on Amazon and can start shopping. You need to repeat the same sign-up procedure when you want to shop from another website, say eBay. (Now you have some options such as logging in with Google, FB etc., to carry your identity however, not all your data is tied to your identity (eg: home address), and you may need to make some adjustments. Each app and its database has its own rules.) So basically, you need to add the same home address information to eBay if you want to start shopping over there.
This creates a burden for the user.
App Development
When each application has its database, this is inefficient for both the user and the developers. The developers need to plan and create databases specifically for their applications, even if it's the same data that another app uses. This is valuable time that can be used to focus on user experience instead.
What do we need?
We mentioned some of the issues of today: the current data economy, less user-centric design and burden for app developers.
Now let's try and understand what we need to overcome these challenges:
- User Owned Data: users control/own their data. There needs to be a user-owned identity(aka self-sovereign identity); the user's data is then tied to this digital identity, which they have full control over. This approach would remove the centralized intermediaries who can monetize over data.
- Portable Data: users can carry their data from app to app. The user owns all the data; applications then read from there. It's like taking your luggage when you go on holiday; you basically take your data when you go to different apps.
- Ease of use: apps today spend a lot of time on database design and implementation. This is a huge piece of the software development cycle. Changing the focus from data to providing a better user experience seems like a better approach. Developers need an easy way to interact with user data for this to happen.
Users controlling their own data would completely reshape the way applications are built today. We need to redesign the application structure at its core.
Web3
Introduction
Let's make a short recap. Web3 is mainly described as the decentralized web; it's still a term under construction. The goal is to remove the intermediaries and have protocols over central entities.
I talked about dApp's (decentralized applications) in my post over here; I mentioned that a dApp has its backend running on a decentralized network (e.g. Ethereum, Solana). It's a start but not enough, we need ways to decentralize the full tech stack and not only a particular piece. This includes the backend, storage, database, API layer etc.
Nader has an article on the Web3 Tech Stack, which I've found to be the most comprehensive and well-explained guide on the topic. I'd highly recommend checking it out.
In this article we will be covering the Data layer (& identity — the two are strongly tied as we talk about user-centric data).
Decentralized Data Layer
Storing large data on the blockchain can be expensive and inefficient, especially with mutable data, when you want to make updates. That's where a decentralized data platform comes in; it's a network optimized for user owned and mutable data.
Users are the owner of their data; they can carry it from app to app and make changes. Applications can use the data associated with the user. There are no intermediaries; the logic is in the protocol.
There are a few off-chain data solutions and identity services out there. Let's dive into Ceramic Network, one that's worth keeping a close eye on.
Ceramic Network
Ceramic is a decentralized, open-source platform for user data. It's a community-driven and open-source project that provides the data & identity layer for the web3 tech stack.
The data layer is a decentralized network of nodes storing and updating the data. All data published onto the ceramic network is shared between applications without an intermediary or any relationship between these apps.
How does it work?
Let's break it down. The building blocks consist of three parts:
- Decentralized data network: a network of nodes running the ceramic protocol
- Authentication: understand which users are adding and reading the data.
- Client-side Library: perform operations on the data.
Each entry on the Ceramic Network is added to an append-only list, called "stream."
Each stream has a stream type. Streams support various Decentralized Identifiers (DIDs); Ceramic also has its decentralized identity protocol. The identity piece allows us to create decentralized user profiles; this makes it possible to have users own/update their data and getting data associated to a certain user.
All the data on the ceramic network is open and available for applications to use. Once a data stream is created, it can be updated by the same user. An update reflects to all the apps using this data stream. Client-side libraries provide the interfaces to perform operations onto the data streams.
The user controls their data, and applications can benefit from users' data which they bring to the application.
Identity
Ceramic has an identity protocol called IDX protocol. Self-id is an application to showcase how it works.
Creating Data Models
We mentioned that entries on the ceramic network are data streams, essentially a list of data. Alongside having data streams, Data models create a standard way to group related data.
You can think it of it as a pre-defined packet of data that users can take to different applications.
With ceramic data models, users can define the type of data structure they want. Creating such a schema is important to standardize the data structures among applications. The data model registry is a community-driven collection of data models on the ceramic network.
Use Cases
The ecosystem is at its early stages, and many use cases are yet to be discovered. Here are two that I'd like to touch upon and link for your reference because they seem to get a lot of attention, and I found them both very interesting.
Orbis a Decentralized Social Network: it's decentralized twitter. I found it pretty cool and easy to use. (Here is also their twitter 😀)
DAOhaus: member profiles on seem like a pretty interesting use case.
Ceramic has curated a list of the projects across the Ceramic Network ecosystem. The Ceramic Blog is also a good place to follow the projects.
Challenges w/Decentralized data
It's all good so far, but I feel the challenges are missing with many web3 posts, so I'm going to try and illustrate them as far as I understand.
- App-specific data: today, there seems to be many data that are app-specific. We need to have standard data structures to get the most out of composable data. These models are being built right now.
- Scalability: This is a challenge with many platforms in web3.
- Shared Data: User-centric data means that only a specific user can change the data stream. The case for shared data is yet to be defined.
Building with Ceramic
Let's add some code in here. I especially want to include this section because when I first discovered Ceramic, I thought it would be pretty hard and complicated to use. The reality was far from this.
So, here you go, a demo to show how to read/write to the ceramic network 🙌
Tools
- Playgound App: Mini browser extension created by the 3box team (core builders of Ceramic) that allows you to experiment with the full stack of ceramic network components.
- Metamask Wallet: crypto wallet.
You won't be needing anything more than your browser.
Creating Data Streams on Ceramic
We will authenticate via the crypto wallet in this demo and then start writing and reading data.
Open the Playgound App and connect your wallet.
The app will authenticate with 3ID connect. It checks if the user already has a profile, or can create one(=identity record). Click continue and sign the transaction in order to create/login to a user profile.
Once you have logged into the user profile. Right click on the browser and select "Inspect," click on the "Console" tab.
Create a data stream
On the console enter the command below. It will create a Tile Document in Ceramic Clay Testnet. A tile document is a stream type which stores a JSON document. With the command below we are creating a data stream on Ceramic that has a key: "hi" and a value: "eda"
const dataStream = await TileDocument.create(ceramic, {"hi": "eda"})Get the data stream details
dataStream.id.toString()You can use the data stream id to get the data stream details by connecting to the Ceramic network(we are on the clay testnet).
Update the data stream
await dataStream.update({"hi":"tom"})The data stream is an append-only list which only the certain wallet id can append to. You can update the value via the command. It will be added to the data stream and the latest value will be "tom."
Your turn
We will not be going to detail in this post however its worth mentioning how to create schemas as it too is way easier than it sounds.
Basically, creating a schema allows you to create a standard data stream. It allows you to enforce certain fields, types etc. Here is an example of how you can create a schema.
const mySchema = await TileDocument.create(ceramic, { #THE_SCHEMA_DETAILS})Once you have created a schema you can then create entries to match the style. It's a piece of structured portable data.
Head over to the Ceramic Developer Docs to give it a go.
Next Steps
The playground app is very convenient way to test and start building. You can also use ceramic from your cli. You can install the js files and run it from your command line. Check out the Ceramic developers guide for more information.
This was a very basic demo to give some ideas on how to read/write to the ceramic network. Let me know if you're interested in building more; I think it's pretty cool and a technology we should be talking more about!
The promise of user owned and composable data is fascinating. More use cases are yet to be created. I'll be keeping a close eye. 👀
If you have any questions feel free to drop them below or reach out to me from Twitter.