Earlier in the week I wrote a post on how, when executing long-running sub-processes in your Ruby scripts, it might be useful to show some kind of progress to your user.
But the progress displayed there was fake: it bore no resemblance to the actual progress of the sub-process. In many respects this is better than nothing, but in others it’s worse; although we’re displaying an indeterminate progress bar, so we’re not actually lying to the user, we’re still not distinguishing adequately between the importing state and the hung state.
Surely there must be a way to improve on this, and show real progress to the user? Turns out, there is. With the application of a little more Unix knowledge, we can do just that.
For this example, I’m going to revisit the use case of importing a database dump into mySQL that I discussed in my first post. For reference, it was equivalent to a system call of:
mysql some_database < dump.sql
Showing real progress
This solution, while perhaps involving more conceptual understanding of Unix fundamentals, is actually simpler than our previous “fake” solution.
Here’s the code in full. I’ll then break it down and discuss it.
require "ruby-progressbar" total_lines = `wc -l dump.sql`.to_i File.open("dump.sql", "r") do |dump| progress = ProgressBar.create(title: "Importing", total: total_lines) IO.popen("mysql some_database", "w") do |stdin| dump.each do |line| progress.increment stdin.puts(line) end end end
What’s going on here, then?
total_lines = `wc -l dump.sql`.to_i
First, we use the Unix
wc command to get the total number of lines in
the file — it’s much quicker than trying to do this calculation ourselves
File.open("dump.sql", "r") do |dump| progress = ProgressBar.create(title: "Importing", total: total_lines)
Next, we open the
dump.sql file for reading, and create a new progress
bar. We set the total value of the progress bar to the number of lines
in the dump; this will allow us to increment once per line and have the
progress bar finish when we’ve finished reading the file.
IO.popen("mysql some_database", "w") do |stdin|
This is where the magic happens. We use the
popen system call to
execute our mySQL command as a sub-process. We tell
popen that we’re
interested in writing to it by passing a mode of
w (just like when we
open a file for writing).
This is just like opening a file: if we passed a block to
the block would be passed a handle that would allow it to write to the
file. We’re doing the same here, except the handle we have will write to
mysql command’s standard input stream, rather than to a file.
dump.each do |line| progress.increment stdin.puts(line) end
Now we loop over the lines in the dump. For each one, we increment the
progress bar by one, and then pass the line to the mySQL process.
stdin here refers to the handle that we opened with
Now, this code might be a bit hard to wrap your head around, especially if you’re not familiar with Unix pipelines. But if you’ve ever used a shell, you’ve used this idiom: it’s what happens under the hood when we pipe between two processes in our shell. In essence, what we’re doing here is:
cat dump.sql | mysql some_database
The only difference is that our Ruby script sits in the middle, orchestrating the pipeline and displaying progress to the user in their shell.
This illustrates neatly one of the foundational principles of the Unix
philosophy: everything is a file. Swap out
open, and we’re
doing exactly the same things that we’d do if we wanted to write the
database dump to another file; it’s also the same as we’d do if we
wanted to write to a network socket. Across all of these disparate
interfaces, Unix presents the same, consistent abstraction: the file.
That’s it! We should now see a proper progress bar on the command-line that increments as the file is imported into mySQL.
Since the file is never read into memory at any point — it’s processed line-by-line — it should scale to any size of input file that you can throw at it.
There aren’t really many downsides to this approach. I’ve never found it to be measurably slower than simply shelling out the command — unsurprising, really, given that the shell is making the same system calls — and even if there was a performance hit, there are many use-cases where a slightly slower import with accurate feedback would be the preferable option.
Ta-da; another way that a little use of Unix processes can help us out in our day-to-day development.