How I optimized Firestore read/write by 2000 times
By using sharding for Firestore
If you use Firebase and Firestore for long enough, you gonna realize that the read/write times are the biggest impact on your invoice every month.
I have this problem for my pet project https://boostmequotes.vercel.app/
Basically, I have about 65k quotes that gonna send to the browser for running offline mode. The app is doesn't have many users, except me, and let's think about this. If 1 user takes 65k read every day, what if I have 50 users?
65k * 50 * 30 days = 97 500k
Which cost me about 17.5$ month
That's a lot for just 50 users and a pet project, if I use Cloud computing, it is a more affordable solution.
How can I reduce read/wite to 15 times
When I learn NEAR Protocal I found that they used sharding - which is an approach to optimization that comes from databases.
Understanding Database Sharding | DigitalOcean
This article is great at explaining what is Sharding
By changing the structure of #FireStore from each item into a new doc, we can combine multiple items into a doc and saving thousand of reading/writing time.
I store about 200 quotes in each document, you can think that each document is the same as a pagination query.
So that instead of costing reading 2000 records, It just cost 1 read. 2000 times decreasing!
What does it cost?
Well, the most cost is, you need to programing to get your desire result, no more query, filter, or order supported by the SDK, you need to do it by yourself
But believe me, you gonna do that by yourself if you use Firestore, it's suck 🙂
Ref: https://firebase.google.com/docs/firestore/query-data/queries#query_limitations
Firestore looks awesome when we first start and then it's suck when you found out those limitations. You can't even count how many documents you have in a collection.
How should I split my documents?
It depends, this is the hardest question when you implement sharding. For my application, I just separate each shard by 200 quotes. You have more about your application context, so you're the best to answer on How to separate your documents
After having a strategy on separating your document, you also need some effort when inserting/updating your document. Which is hard or easy is based on your strategy
When should I use sharding for Firestore?
- You are happy on coding to query documents
- You have a strategy on sharding that can answers
- How can I separate my document?
- By your separate strategy, how can I find the document I desire
- How can I insert/update new documents? Which shard should I put it in
If you have a clear answer to those questions? Get ready on saving your billing by 2000 times
P/S: If you have any solutions, let's share in the comment bellow 👇👇👇