The GPU Banana Stand

How SQLite Scales Read Concurrency

Fixed Partitions

Issue #266



Sorry for no issue yesterday, was traveling all day.
Let me know what you think of the more substantial summaries today. Is it better? Or does it take away the mystique of what hidden gems may be found?


The GPU Banana Stand

Published: 20 August 2022
Tags: gpu, web

GPU programming is notorious for not being very practical. This is even more-so the case when you're trying to use it on the web. In this article, Steven Wittens presents some of the functionality and features of his "declarative/reactive WebGPU library", Use.GPU.
Some key takeaways are:

  • The React derived runtime, Live, is completely self-hosted and is fully rewindable and memoized
  • Compute shaders are fully supported
  • All of the elements are just components
  • WebGPU has made things that were impractical with WebGL practical
  • By sacrificing some performance, the implementation and use of the library has become far simpler
  • The library has quality-of-life components out of the box, but you can still "crack them open and use the raw parts if they're not right for you"
  • Copying how popular software in one domain (this case React) works to another domain can lead to intuitive and easily adopted interfaces


How SQLite Scales Read Concurrency

Published: 23 August 2022
Tags: sqlite

SQLite has historically been considered unusable for anything beyond a single user hobby project. That's no longer the case though, thanks to the introduction of a write-ahead log (WAL) in 2010, as an option instead of the rollback journal. Ben Johnson's article goes deeply into how the journaling method works.
Some key takeaways are:

  • Rollback journal works "by copying the old version of changed pages to another file so that they can be copied back to the main database file if the transaction rolls back"
  • WAL "writes the new version of a page to another file and leaves the original page in-place in the main database file"
  • WAL has to be enabled via config option PRAGMA journal_mode = wal;
  • For WAL, write transactions can occur without interfering with reads as a result of pages not being overwritten
  • Checkpointing (when SQLite copies the latest version of each page in the WAL back into the main database file) ensures the WAL doesn't get too long
  • SQLite uses a "shared memory" file to store an index for speeding up the look up of the latest version of a page for every transaction
  • Thanks to WAL, SQLite can be used for more than just hobby projects


Fixed Partitions

Published: 22 August 2022
Tags: distributed systems

A fairly complicated problem with distributed systems is keeping track of where all of the data is stored, while also keeping things performant. This becomes an even bigger issue when there's the potential for a lot of data to be added in the system's lifetime. Unmesh Joshi presents a method of setting up a distributed system with data mapped to node (logical) partitions, keeping the necessary copying of data to a minimum when new nodes are added.
Some key takeaways are:

  • Look up keys are the hash of the data item key and its partition
  • Instantiate the system with a large number of partitions (3-10x more than the number of nodes)
  • Data storage/retrieval is a two step process: find partition of data item -> find cluster node storing partition
  • Key hash function should be runtime independent (MD5 or Murmur)
  • Redistributing data among the nodes becomes a matter of just copying relatively small partitions
  • Use a "dedicated Consistent Core as a coordinator which keeps track of all nodes in the cluster and maps partitions to nodes"
  • Meta data should be kept on all cluster nodes so the Consistent Core controller isn't the bottleneck for querying

How did I do?

1 2 3 4 5


Want to help and get cool stuff?

Thank you for reading! If you enjoy the newsletter, I would really appreciate you helping me spread the word by forwarding this to your friends and colleagues or sharing it on social media! Get cool stuff for your referrals using your link or the buttons below.

Your referrals:

If you want to discuss or comment on this issue, head on over to this page at A Byte of Coding. You can also subscribe there if you're new!

Have comments or feedback? Just reply to this email or hit me up on Twitter @AByteOfCoding.

Email landed in your promotions tab? Please move it over to primary so you don't miss the latest issues in the future.
Thanks for your Support! 

Thanks to sponsors and supporters like Євген Грицай, Scott Munro, zturak, pek, Emil Hannesbo, Joe Hill, Astrid Sapphire, Gregory Mazzola, moki scott, Michael, Matt Braun, Tim Nash, Christoffer, and Mike Rhodes this newsletter is provided to you for free. If you'd like to also show your support and buy me a monthly meal, you can donate on the Patreon page. It's not necessary, but it lets me know that I'm doing a good job and that you're finding value in the content.

Stats (updated daily)

Sent: 2945

Opens: 1402

Clicks: 358

Link Clicks Clicks % Unique Clicks Unique Clicks %
The GPU Banana Stand 90 36.89% 97 37.74
How SQLite Scales Read Concurrency 78 31.97% 81 31.52
Fixed Partitions 76 31.15% 79 30.74


Back to Issues