Applications are currently used by people all over the world. The applications have the following features.

• Have millions of users
• Store a large amount of data that amounts from petabytes to exabytes(TB-EB)
• Require performance from ms to μs
• Handle millions of requests per second

The number TB-EB may be difficult to understand because it is a number of digits that is not often used in daily life.

Hardware is always required to run the software. It is necessary to select the hardware that is suitable for the features of the software to be run. Understanding the amount of data will help you choose the memory and storage of your hardware. Hardware selection ranges from on-premises server devices and virtual server instance types to PCs used at home.

Today, the advent of the cloud computing blurs the border between application engineers and infrastructure engineers. Many teams of application engineers build infrastructure using the cloud. Understanding the amount of data makes you choose the right hardware for yourself.

In this article, we'll give you a sense of the amount of data by showing the amount of data of various things.

Data size unit

The unit of data size is a byte. Currently, 1 byte is defined as 8 bits.

Since the data size handles a large number of digits, add a prefix to omit the number of digits. In the International System of Units(SI), the prefix is as follows:

On the other hand, the data size prefix is as follows:

The data is treated as a binary number in computer, so the prefix is every 1024, which is the 8th power of 2, instead of every 1000. There is also a notation that uses KiB instead of KB to distinguish it from the International System of Units, but this article uses KB.

Website data size

According to Page Weight, the data size of a website component is as follows:

The target period for aggregation is from January 2017/1 to 2022/1, and the target is mobile sites.

Note that it is the total size per page, not the size per file. Some people may find the size of JavaScript larger than expected. The reason is that it includes not only own code but also the code of external packages such as frameworks and libraries.

File size

The size varies depending on the contents of the file, so it is for your reference. The size and format were converted from the reference as needed. Also, these data are not a comparison of good and bad file formats. This is because the appropriate file format depends on the features of the file.

Hardware capacity

The data size of the hardware memory and storage are following table. If there is no standard, the value is shown as a guide.

Real application data volume

According to Data Never Sleeps, the data created by the actual application is following:

Next, let's look at it in bytes. Assuming to tweet on Twitter has 100 characters and one character is 1 byte, 1tweet is 0.1KB. Instagram 1photo is assumed to be 0.1MB. Assuming to message in Slack has 50 characters and one character is 1 byte, 1message is 0.05KB. Based on these things, it is as follows.

If you operate the service for one year, more than terabytes of data are accumulated. It is difficult to store this amount of data on a single database server, and you need to scale it out and store it on a distributed database server. We call this a distributed database.

There are two ways to build a distributed database: Master/Slave method and partitioning method. The Master/Slave method is an approach to high traffic, not a large amount of data volume. We should use partitioning for a large amount of data.

RDBs are not designed for partitioning. Therefore, maintaining the partitioned RDB is costly. If you want to build a partitioned distributed database, consider a database designed for partitioning, such as DynamoDB.