Being part of a massive foundational shift, like the one currently occurring in the blockchain and distributed systems spaces, is a double-edged sword. It’s a fun, career-building opportunity but can be daunting, as you are often solving many problems for the first time. Even a simple bug can be a huge task to overcome.
Those building decentralized applications have experienced this time and time again. A challenge my team and I have encountered is achieving scalability on the Ethereum blockchain, which we use for payments on our platform. Others have discussed Ethereum’s scalability in recent weeks. As alluded to here, what many overlook is that every emerging platform struggles to scale early on. In spite of its problems, Ethereum is still in the best position to scale long term.
Discovering Challenges Looming In The Ether
Transaction fees are one of the biggest community concerns. For individuals sending Ethereum, fees in the $1-$2 range don’t seem high. Odds are you’re sending a high-value transaction, so the fee makes up a small percentage of the overall payment. When you’re sending large volumes of transactions required for real-world applications, that $2 fee across 150,000 payments quickly grows to $300,000.
Our decentralized cloud storage network is composed of individuals sharing their unused hard drive space in exchange for STORJ, our Ethereum-based utility token. We currently run monthly payouts to our community.
Last March, our team experienced a huge payment volume milestone. We sent more than 145,000 payments in one week. Ethereum processes about 15 payments per second, meaning if the network only handled our payments, it would have taken over four hours to complete. Because of this volume, we lean toward sending transactions when fees are low.
Problems aside, we’ve been impressed that we can run so many payments with just regular, off-the-shelf laptops, a script and the payout data.
Finding Solutions To Scaling Problems One at a Time
When you are solving multiple issues within a platform, it’s best to solve them one by one to see how each change affects your entire ecosystem. Our payment issues began to compound when our Ethereum transaction volume began to exceed 100,000 per month. The hurdles were tied to several things, including the web3.js library used in our node.js payment application, the Parity Ethereum nodes that handled the interaction with the blockchain, OS-level configuration and Ethereum chain-specific issues.
We began by upgrading our web3.js library to fix a number of bugs and take advantage of new features. We were initially using a simple node.js script to process payments, which didn’t attempt to confirm transactions or handle many error cases. The latest library allowed us to utilize new capabilities to overcome many of our issues, and the rest were relatively easy to work around.
Our issues with Parity were some of the most challenging we encountered. We ran our own Parity nodes to securely relay all of the transactions we wanted to broadcast. However, the local queue would fill up and cause the node to drop local transactions in favor of other transactions, making it difficult to check their status. We overcame this by carefully managing the nonces – a number the network uses to identify transactions per account, increasing by one each time a different transaction is sent per account. This is highly recommended if you’re pushing out notable transaction volume and is relatively easy to implement using Parity.
Out of the box, Parity also struggled with large numbers of concurrent transactions. This took some fine-tuning of the command line parameters to resolve. This gave rise to a new bottleneck: One node could not relay transactions to the network fast enough. Running multiple nodes solved this by increasing the maximum throughput of transactions.
Several other issues have now been fixed on Parity’s side as well. In previous versions, there were delays between submitting the transaction and when it would show up in queries. Things in this industry are changing so rapidly it can be difficult to remain apprised of the latest updates and innovations.
The last area where we experienced issues was related to OS-level configuration and tuning. When processing large volumes of transactions, it is essential to increase the number of available TCP connections as well as the max number of open files for the user of the Parity process. After tuning these appropriately, we experienced great results.
Ethereum Network Scalability
The future of the Ethereum network depends on effectively scaling as demand grows. There are two considerations to keep in mind when solving Ethereum’s transaction throughput scaling challenges. The first is the fact that every transaction must be completed by every node. This inherently means that adding more nodes to the network will never help scalability, only security. The second is that all transactions, no matter how small, must go through the main network.
The first possible solution is sharding, which breaks the Ethereum network into logical chunks that can independently and securely process more transactions. As demand increases and the network grows, this would dynamically, and potentially indefinitely, scale.
The second is the Raiden Network, which enables off-chain, fast, low-fee transactions based on a predefined Ethereum smart contract between two parties. At any time, either party can resolve the contract, resulting in a settlement to the payee in the amount that they’re owed, returning the remaining funds from escrow to the payer. Ultimately, I think the answer is a combination of both solutions.
It is often said that necessity is the mother of innovation. The more real-world Ethereum use cases that emerge, the faster we will achieve scalability. There are about 2.6 million seconds in a month, implying the network can handle about 26 million transactions during that time frame. That means it would take fewer than 200 companies of our size to completely saturate the Ethereum blockchain. With the entire blockchain and decentralization communities working on this together, I feel confident that we will soon have the solutions that we all need.