Why SQLite may become foundational for digital progress

If every dog ​​has a day, well, maybe the same is true for every database. Judging from the recent news, SQLite is spending its day in the sun. Over the past few weeks, many companies have announced that they are building or endorsing new projects built around the venerable open-source database.

Is SQLite one of the foundations of the next generation of Internet? Some think so. CloudFlare announced that they are using a new database service built around backend tools. Meanwhile, Fly announced that it is hiring one of the developers of Litestream, an open-source project that enhances the basic version of SQLite by adding the ability to copy data to enhance performance and survival.

At first glance, the idea may seem a little strange. This project is over 20 years old and is written in plain, old C. It is not a standalone application that deserves its name as much as a library that can be linked to your code. In front of that house, the forgotten servant works as ungrateful as the marquee software option is not. Many developers may start using code when they are sketching a project or creating a prototype, but they often move on to other, full-featured options like Oracle or PostgreSQL.

But the ads suggest that companies are looking for something more. CloudFlare, for example, is launching a new database service called D1 to give developers another way to store data generated by their Workers Serverless apps. They already offer a key-value store and a bucket product (R2), but developers want to rely more frequently on the design and power of SQLite to simplify their workload.

“It’s a database for the same model,” said Rita Kozlov, Cloudflare’s senior director of production. “Its embedded nature also makes a lot of sense, where the whole purpose of D1 is to make it really easy for our developers to be able to spin the database along with their calculations.”

Cloudflare has been expanding rapidly since its inception as a worldwide static cache. They have built hundreds of data centers close to end users for quick response. Recently, they have been adding products like Workers or Pages that offer a serverless model of application deployment. Developers can write a few basic functions, pay only for the time the function runs, and even provide lightning-fast feedback as the code runs on machines close to the user.

Adding SQLite helps developers provide more sophisticated applications. Data can be stored locally on edge nodes and then eventually copied worldwide. Developers who are more ambitious and need to keep an eye on more user states can adopt the platform.

“We asked many of our interior developers, ‘How can we build you more with workers?'” Kozlov said. “Their answer is, ‘Give me the database. That’s the tool I use.’ I can probably figure out how to do what I need to do [Key-Value] But that’s not where people are today. We always want to meet the developers wherever they are. “

The fly is jumping with the same goal. They have announced that they are supporting work on Litestream, an open-source project that adds background processing to SQLite. It will stream updates to various object stores and FTP sites, so developers can trust that SQLite’s data will still be available and can be recovered after an injury.

It’s easy to see where Fly came from to improve and expand the open-source database. One of their main products is fully supported clusters of PostgrassQL. Developers can set up a scalable, flexible version of PostgreSQL in just a few clicks.

Many other companies are doing the same thing with open-source databases. Companies like PlanetScale, Yugabyte, Amazon, Oracle and Google are starting with MySQL or PostgreSQL and then adding additional layers of features to improve reliability, scalability and more.

Just last week, Google announced AlloyDB, their version of PostgrassQL that offers full compatibility with some additional enhancements, such as the Column Store, which could dramatically improve some workloads.

Single-threaded, but multidimensional

However, there are many differences between SQLite and other projects. SQLite is a basic, single-threaded system. Other databases are designed with multiple threads to juggle more complex constellations of users.

For many small projects, this is not much of a limitation and some developers see it as a feature.

“I run a database company before this and I think people like me never want to talk that everyone has a few sub-10 gigabytes of databases.” Said Fly CEO Kurt McKee. “If you’re really into that range, you know this is very interesting because it’s SQL and it’s awesome for 10 Gig databases.”

Developers can often get what they want from basic core functions without the complexity of supporting a full-featured database.

“The documentation for Postgrass 14 is about 3,000 pages.” Said Ben Johnson, one of Fly’s developers. “And if you don’t need Postgres features, that’s a liability. For example, if you do not use multiple user accounts, you will still need to configure and debug host-based authentication. You need to turn off the firewall from your postgrace server. “

The Litestream open-source project supported by Fly enhances SQLite by adding the option to add more resilience to hardware failures as well as adds more consent. It addresses the most serious concerns developers have about using the tool with more serious, server-side projects.

“It’s really nice during development.” A developer who frequently uses SQLite in projects, said Kent Dodds. ,[There’s] No need for database server up and running (or docker container). It’s just a file. If you need help with anything, you can send a database file to a colleague. “

What are the shortcomings and beyond

However, while many Fly customers successfully use SQLite for data storage for some simple applications running on the service, Fly’s Mackie reports that there are some rough edges. The software runs very fast without performance glitch, but the number of tools that can help support it is not the same.

“I think the biggest complication for us is that there is no tool for that.” Said Mackie. “We have people using apps. They are like, ‘How do I connect to my database and select query items? How do I import data? ‘”

Many rely on folklore and third-party tools that are common. The code has been widely adopted over the years and many have written their tools that can support it. Even if the tools do not directly target new server-side operations, they can still be adapted.

“One of the things I like about it as a product is that it’s very stable.” Said David Croso, chief technical officer of Tilescale. The company uses SQL to support many network operations. “It simply came to our notice then. It means that when I come back to it, what I have learned is still useful.

The second is that it is not exactly open source. Dwayne Richard Hipp, the original developer of SQLite, put it in the public domain. In general, this means that there are no legal restrictions on the use of the code, although there are some questions as to whether each country recognizes this view.

For developers it is a good stepping stone or starting point, and sometimes just as necessary. This independence has encouraged many developers in the past. It is common to find SQLite running inside many devices. Many smartphones and tablets use it for default storage.

Still, it makes some wonder how much of a real trend this is and how much of a step it is for companies. This was further underscored by Kozlov, who noted that the project in Cloudflare was named “D1” for a number of reasons: increasing the number is easy and said, “I don’t think this is – or rather, I know. That’s not it – our last stop in database space. I think we will find ways to extend it or we will extend our offer. “

Venturebeat’s mission Digital Town Square is set to become a place for technical decision makers to gain knowledge about the changing enterprise technology and practices. Learn more about membership.

Similar Posts

Leave a Reply

Your email address will not be published.