At Dimona, a leading Latin American apparel company founded 55 years ago in Brazil, our business is t-shirts. We design them, manufacture them, and sell them to consumers online and through our five retail stores in Rio de Janeiro. We also supply B2B companies for their customers in Brazil and the United States.
We’ve come a long way since 2011 when I joined Dimona to launch our first website. Today, our API enables our B2B customers to upload custom designs, and automatically route orders from their e-commerce sites to us. We then make the shirts on demand and ship them in as little as 24 hours.
Both APIs and fast-turnaround drop shipping were major innovations for the Latin American apparel industry, and it enabled us to grow very quickly. Today, we have more than 80,000 B2B customers supplied by our factories in Rio de Janeiro and South Florida. We can dropship on behalf of our B2B customers anywhere in Brazil and the U.S. and help them avoid the hassle and cost of import taxes.
Our business is thriving. However, we almost didn’t get here due to growing pains with our data technology.
Off-the-Shelf ERP Systems Too Limited
Due to our vertically-integrated business model, our supply chain is longer than most clothing makers. We need to track raw fabric as it arrives in our factories, the t-shirts as they move through the cutting, sewing and printing phases, and the finished products as they travel from factory to warehouse to retail store or mail carrier before finally reaching customers.
Not only is our supply chain longer than normal, so is the size and diversity of our inventory. We have up to one million t-shirts in stock depending on the season. And due to the many custom designs, colors, fabrics and sizes that we offer, the number of unique items is also higher than other apparel makers.
We tried many off-the-shelf ERP systems to manage our inventory end-to-end but nothing proved up to the task. In particular, limitations in these systems meant we could only store the end-of-day inventory counts by location, rather than a full record of each individual item as it traveled through our supply chain.
Tracking only inventory counts minimized the amount of data we had to store. However, it also meant that when we tried to compare these counts with the inventory movements we did have on file, mysterious errors emerged that we could not reconcile. That made it hard for us to trust our own inventory data.
MySQL Crumbles Under Analytic Load
In 2019, we deployed our own custom-built inventory management system to our main warehouse in Rio de Janeiro. Having had experience with AWS, we built our inventory management system around Amazon Aurora, AWS’s version of MySQL-as-a-service. Rather than just record end-of-day inventory totals, we recorded every inventory movement using three pieces of data: the item ID, its location ID, and the quantity of that item at that location.
In other words, we created a ledger that tracked every t-shirt as it moved from raw fabric to finished goods into the hands of a customer. Every single barcode scan was recorded, whether it was a pallet of t-shirts shipped from the warehouse to a store, or a single shirt moved from one store shelf to another.
This created an explosion in the amount of data we were collecting in real time. Suddenly, we were uploading 300,000 transactions to Aurora every two weeks. But it also enabled us to query our data to discover the exact location of a particular t-shirt at any given time, as well as view high-level inventory totals and trends.
At first, Aurora was able to handle the task of both storing and aggregating the data. But as we brought more warehouses and stores online, the database started bogging down on the analytics side. Queries that used to take tens of seconds started taking more than a minute or timing out altogether. After a reboot, the system would be fine for a short while before becoming sluggish and unresponsive again.
Pandemic-Led Expansion
Compounding the issue was the COVID-19’s arrival in early 2020. Suddenly we had many international customers clamoring for the same drop shipment services we provided in Brazil in other markets. In mid-2020, I moved to Florida and opened our U.S. factory and warehouse.
By that point, our inventory management system had slowed down to the point of being unusable. And our bills from doing even simple aggregations in Aurora were through the roof.
We were faced with several options. Going back to an error-ridden inventory-count system was out of the question. Another option was to continue recording all inventory movements but use them only to double-check our separately-tracked inventory counts, rather than generating our inventory totals from the movement records themselves. That would avoid overtaxing the Aurora database’s meager analytical capabilities. But it would force us to maintain two separate datasets – datasets that would have to be constantly compared against each other with no guarantee that it would improve accuracy.
We needed a better technology solution, one that could store massive data sets and query them in fast, automated ways as well as make quick, simple data aggregations. And we needed it soon.
Finding Our Solution
I looked at several disparate options. I considered a blockchain-based system for our ledger before quickly dismissing it. Within AWS, I looked at DynamoDB as well as another ledger database offered by Amazon. We couldn’t get DynamoDB to ingest our data, while the ledger database was too raw and would have required too much DIY effort to make work. I also looked at Elasticsearch, and came to the same conclusion – too much custom engineering effort to deploy.
I learned about Rockset from a company that also was looking to replace query-challenged Aurora with a faster managed cloud alternative.
It took us just two months to test and validate Rockset before deploying it in September 2021. We continued to ingest all of our inventory transactions into Aurora. But using Amazon’s Database Migration Service (DMS), we now continuously replicate data from Aurora into Rockset, which does all of the data processing, aggregations and calculations.
“Where Rockset really shines is its ability to deliver precise, accurate views of our inventory in near-real time.”
– Igor Blumberg, CTO, Dimona
This connection was extremely easy to set up due to Rockset’s integration with MySQL. And it is fast: DMS replicates updates from one million+ Aurora documents to Rockset every minute, becoming available to users instantly.
Where Rockset really shines is its ability to deliver precise, accurate views of our inventory in near-real time. We use Rockset’s Query Lambda capability to pre-create named, parameterized SQL queries that can be executed from a REST endpoint. This avoids having to use application code to execute SQL queries, which is easier to manage and track performance, as well as more secure.
Using Rockset’s Query Lambdas and APIs also shrank the amount of data we needed to process. This accelerates the speed at which we can deliver answers to customers browsing our website, and to store employees and corporate workers internally searching our inventory management system. Rockset also completely eliminated database timeouts.
Rockset also gives us full confidence in the ongoing accuracy of our inventory management system without having to constantly double-check against daily inventory counts. And it allows us to track our supply chain in real time and predict potential spikes in demand and shortages.
Rockset has been in production for us for more than half a year. Though we are not yet leveraging Rockset’s capabilities in complex analytics or deep data explorations, we are more than satisfied with the near real-time, highly-accurate views of our inventory we have now – something that MySQL could not deliver.
In the future we are thinking of monitoring DMS to guard against hiccups or replication errors, though there have been none to date. We are also considering using Rockset’s APIs to create objects as we ingest inventory transactions.
Rockset has had a massive effect on our business. Its speed and accuracy give us unprecedented visibility into our inventory and supply chain, which is mission critical for us.
Rockset helped us thrive during Black Friday and Christmas 2021. For the first time, I was able to get some sleep during the holiday season!
“Rockset gives us full confidence in the ongoing accuracy of our inventory management system without having to constantly double-check against daily inventory counts. And it allows us to track our supply chain in real time and predict potential spikes in demand and shortages.”
– Igor Blumberg, CTO, Dimona
Rockset is the real-time analytics database in the cloud for modern data teams. Get faster analytics on fresher data, at lower costs, by exploiting indexing over brute-force scanning.