Software.next

A Retrospective. Part 1

Wed, 23 Sep 2015 00:00:00 -0400

I’ve had a lot of things to say about the way both “the web” and native platforms are built. The web, as its name suggests, is an unstructured, fast-moving, often-breaking wild west. Native is a hyper-specialized, peformant, but often an archaic relic. Native is still super compelling for mobile right now, since the flashy UI and smooth user interaction is harder to build in a manner that matches native performance and smoothness. However you want to argue, you and I both know when we’re using a web app on mobile vs. a native app.

At the same time, there is a massive shift towards building anything and everything using web technology. React Native, Electron (formerly Atom Shell), among others, are all allowing people to use a common HTML/CSS/JavaScript stack to build “native” apps, that is, apps that run in a web container that feels and interacts with an OS more like we’re used to with native apps.

My main complaint against the web is that it is so fragmented and disoriented in its goals. In a way, the web is having an identity crisis. Partly this is due to an evolving landscape in the world of web “applications”. The web, and therefore browsers, are being expected to deliver more functionality than ever before and this doesn’t help provide a stable, reliable platform for applications to operate from. Major corperations are now pushing agendas to make the web more friendly towards their goals. And with all this fluctuation, user experience suffers. Application performance and reliablity suffers. Mobile usefulness suffers. So we write more libraries, create more standards, etc. This cartoon illustrates this problem.

But here’s the kicker. If it were me, I’d still rather write an app using an HTML/CSS/JS stack. Why?

Because it’s so easy to do it.

While it may be a mess of fragile hacks, browser compatability quirks, feature detection, screen resolution “responsitivity”, and the list goes on, it’s still takes me far less time to get something that I can see and tweak on the fly than it does with any current native solutions. The problem with building software for the web is one of its greatest strengths. Native platform vendors take note.

While this is entirely opinion, here are a few things I think native vendors should learn from the web:

Your build/deploy cycle needs to be simple, or at least abstracted enough to make first-time developers comfortable getting from nothing to running app. For the web, this is creating an .html file and opening it in a browser. done. Publish that file to a static file server and again, you’ve launched your product to the world.

Notice, I’m not saying anything about cross-platform compatibility. I realize this is a huge draw to things like Electron, but if you’re truly duing native, your app generally has to adopt the look, feel and mannerisms of the ecosystem it is running on, and I think per-platform development is a worthwhile effort. And if the nothing-to-something pipeline were smoothed out for differing platforms, your startup and development costs would be cut down immensely.

Ruby Methods

Tue, 25 Aug 2015 00:00:00 -0400

I rediscovered a nice way to give Ruby methods a more first-class feel when wanting to pass them around them without having to worry about knowledge of where the methods are defined. Lots of languages have this concept. Take JavaScript for example:

myFunc = function addTwo(arg) {
  return (arg + 2);
}

typeof myFunc;
//=> 'function'

myFunc(5);
//=> 7

In this way, we could pass a function to someone that wanted to use it to operate on something else

function caller(arg, func) {
  func.call(this, arg)
}

caller(2, myFunc)
//=> 4

Most Ruby code I’ve seen doesn’t use this same mechanism of passing functions to other functions. We, as superior beings to JS developers, have blocks after all.

def caller(arg, &block)
  block.call arg
end

caller do |arg|
  puts (arg + 2)
end

#=> 2

But what if we want to inject behavior into a caller from another class or use the same code for different blocks? Typically this involves needing to send across an object and a symbol representing the method to call via send or wrapping things in lambdas or Procs. Let’s set up an example:

class ZipUploader
  def initialize(zip)
    @zip = zip
  end

  def upload
    Unzipper.open(@zip).each_file do |file|
      TheCloud.upload file
    end
  end
end

Obviously, you don’t need to care how Unzipper or TheCloud work. Unzipper opens the zip files and allows for iterating over each file in the archive. TheCloud.upload uploads them to your favorite cloud storage provider.

Now imagine you want to do some additional processing for specific files in this archive but don’t want to pollute the uploader with knowledge outside of its main responsibility of getting files to the cloud. We could do this a few different ways:

class ZipUploader
  def initialize(zip)
    @zip = zip
    @callbacks = {}
  end

  def upload
    Unzipper.open(@zip).each_file do |file|
      run_callbacks(file)

      TheCloud.upload file.read
    end
  end

  # Option 1: pass a block
  def file_callback(file, &block)
    @callbacks[file] = block
  end

  private

  def run_callbacks(file)
    callback = @callbacks[file.name]
    callback.call(file) if callback
  end

  # Option 2: Pass a class and a symbol to call via `send`
  def file_callback(file, processor, action)
    @callbacks[file] = { runner: processor, method: action }
  end

  def run_callbacks(file)
    callback_defn = @callbacks[file.name]
    callback_defn[:runner].send(callback[:method], file)
  end
end

class Runner
  def initialize
    @uploader = ZipUploader.new("my_archive.zip")
  end

  def run
    # Using option 1
    uploader.file_callback("my_face.jpg") do |image|
      image = ImageMagic.superheroize(image)
    end

    # Or option 2
    uploader.file_callback("my_face.jpg", self.class, :superize)

    uploader.upload
  end

  def superize(image)
    image = ImageMagic.superheroize(image)
  end
end

So now, using either of the two options, as the uploader is uploading files, whenever it encounters a file called “my_face.jpg”, it will make it look like I’m a superhero. Awesome!

Obviously, option 2 is more complex and not as readable. The main reason we might do something like this when dealing with a system like Resque or Sidekiq, where you need jobs to be able to serialize callback info into Redis. You’d be better off storing class/method pairs over trying to serialize actual Proc or block calls.

So back to our example, in a real world application, we might have much more complex processing or a whole bunch of processors:

uploader.file_callback("*_avatar.jpg") do |image|
  image = ImageMagic.resize(image, width: 80, height: 80)
end

uploader.file_callback("*_retina.tif") do |image|
  image = ImageMagic.enhance(image)
end

# and so on...

That adds a lot of block code to wherever we’re setting up the uploader callbacks. What if we could just pass functions to our callback definitions? We could define these all as lamdas or Procs:

RESIZER = -> (img) { img = ImageMagic.resize(img, width: 80, height: 80) }
RETINIZER = Proc.new {|img| image = ImageMagic.enhance(image) }

uploader.file_callback("*_avatar.jpg", RESIZER)
uploader.file_callback("*_retina.tif", RETINIZER)

But that feels kind of ugly and un-Ruby. Well with method you can actually extract methods from a class and pass them to a function.

def resize(image)
  image = ImageMagic.resize(image, width: 80, height: 80)
end

def retinize(image)
  image = ImageMagic.enhance(image)
end

uploader.file_callback "*_avatar.jpg", &method(:resize)
uploader.file_callback "*_retina.tif", &method(:retinize)

The cool thing about this is you can even extract your image processing methods to a class of some kind, responsible for different processing types and still extract the functions and pass them to a call requiring a block.

class ImageHandlers
  def resize(image)
    image = ImageMagic.resize(image, width: 80, height: 80)
  end

  def retinize(image)
    image = ImageMagic.enhance(image)
  end
end

uploader.file_callback "*_avatar.jpg", &ImageHandlers.instance_method(:resize)
uploader.file_callback "*_retina.tif", &ImageHandlers.instance_method(:retinize)

What this method… uh… method returns is an instance of Ruby’s Method class. This class has a number of useful methods that you can use to build out much more functional-esque code. instance_method does the same thing for a class’s instance methods, only it returns an UnboundMethod instance, which is mostly the same thing, but with the added abilities to check things like super_method references and rebind to a different context or class/module.

If we wanted to use these Method objects to serialize to Redis, we could kinda fake it in the callback definition method, instead of storing the actual Method instance, we could serialize it using info we have available:

# callback is the Method we extract
def file_callback(file, callback)
  # holds the callback reference in a string: "Class#method"
  @callbacks[file] = "#{callback.owner}##{callback.original_name}"
end

def run_callbacks(file)
  callback_defn = @callbacks[file.name]
  klass, meth = callback_defn.split('#')

  callback = klass.constantize.instance_method(meth)
  callback.send(file)

  # or just
  klass.constantize.send(meth, file)
end

This would allow you to serialize callbacks if need be. This will still allow you to pass methods as parameters in the calling code, allowing you to simply define methods on your class and pass them to the processor.

Subnotes

Like subtweeting, but for blog posts

class Processor
  def plus_two(arg)
    arg + 2
  end
end

class Runner
  def block_run(arg, &block)
    yield arg
  end

  def sender(arg, cls, mth)
    cls.send mth, arg
  end
end

lambda = -> (arg) { arg + 2 }
proc = Proc.new {|arg| arg + 2 }

processor = Processor.new
runner = Runner.new

meth = processor.method(:plus_two)
rebinder = Processor.new
rebound = meth.unbind.bind rebinder

Benchmark.ips do |test|
  test.report("Direct calls") { processor.plus_two(2) }
  test.report("Via send") { runner.sender(2, processor, :plus_two) }
  test.report("Blocks") { runner.block_run(2) {|arg| arg + 2 } }
  test.report("Lambda") { runner.block_run(2, &lambda) }
  test.report("Proc") { runner.block_run(2, &proc) }
  test.report("Method") { runner.block_run(2, &meth) }
  test.report("Rebound direct call") { rebinder.plus_two(2) }
  test.report("Rebound method") { runner.block_run(2, &rebound) }

  test.compare!
end

Calculating -------------------------------------
        Direct calls   135.058k i/100ms
            Via send   125.582k i/100ms
              Blocks    79.429k i/100ms
              Lambda   129.758k i/100ms
                Proc   122.336k i/100ms
              Method    60.091k i/100ms
 Rebound direct call   139.934k i/100ms
      Rebound method    59.337k i/100ms
-------------------------------------------------
        Direct calls      7.723M (± 6.9%) i/s -     38.492M
            Via send      5.190M (± 6.6%) i/s -     25.870M
              Blocks      1.515M (± 7.4%) i/s -      7.546M
              Lambda      5.252M (± 7.2%) i/s -     26.211M
                Proc      5.446M (± 5.3%) i/s -     27.159M
              Method    998.451k (± 6.2%) i/s -      4.988M
 Rebound direct call      8.097M (± 5.7%) i/s -     40.441M
      Rebound method      1.018M (± 5.7%) i/s -      5.103M

Comparison:
 Rebound direct call:  8096886.8 i/s
        Direct calls:  7722969.7 i/s - 1.05x slower
                Proc:  5445994.0 i/s - 1.49x slower
              Lambda:  5251876.4 i/s - 1.54x slower
            Via send:  5190066.6 i/s - 1.56x slower
              Blocks:  1515285.7 i/s - 5.34x slower
      Rebound method:  1017916.6 i/s - 7.95x slower
              Method:   998451.0 i/s - 8.11x slower

So obviously there are some performance concerns with a number of these approaches. I’m unsure as to why, other than possibly internal Ruby VM optimizations for regular method calls. Surprisingly, a block is actually slower than passing procs/lambdas or using send, though probably for the same reasons that using Method instances are slower too. Maybe I’ll dig on that in my next post!

The Great Sqlizer

Tue, 30 Jun 2015 00:00:00 -0400

After a fun bit of traipsing all over the Rails codebase, I found a few things out about bot Postgresql, Rails’ schema format and Rails’ convention around primary keys that I thought would be good to document for future me.

Rails has a schema format that looks similar to its migration format. Example:

create_table "users", force: true do |t|
  t.citext   "email"
  t.datetime "created_at"
  t.datetime "updated_at"
  # ... etc ...
end

This is pretty cool unless you’re doing something different from the standard conventions, like naming the primary key something different and changing its type. For example, Postgres has the uuid field type. This is a pretty standard key format convention for global unique identifiers (GUIDs), and, hey, let’s use them!

Rails provides for a means of doing this out-of-the-box with Postgres:

create_table "users", id: :uuid force: true do |t|
  # ... etc ...
end

Great! What does that do? Inspecting the database reveals two things:

It creates the column id of type uuid and makes it a PRIMARY KEY
It sets the default to be auto-generated via the uuid_generate_v4() method, which is part of the uuid-ossp extention in Postgres. This allows PKs to be generated on the fly uniquely, similarly to the auto-incrementing sequence field type.

So that’s cool, and works generally for most people’s use-case. But let’s say for example we want to change the PK’s name. Something like my_cool_pk. Rails provides at least three ways of doing this via migrations:

create_table "users", id: false, primary_key: "my_cool_pk"

# Or...

create_table "users", id: false do |t|
  t.primary_key :my_cool_pk, :uuid
  # ...
end

# Or...

create_table "users", id: false do |t|
  t.uuid :my_cool_pk, primary_key: true
  # ...
end

What you may notice from the first is that the PK column type gets dropped from the definition. Super important detail. The other two work just fine though. HOWEVER, both of the second two mechanisms, while they create the table just fine when running migrations, the schema.rb that all three methods generate all end up looking exactly like the first statement:

create_table "users", id: false, primary_key: "my_cool_pk"

Which means, if you load this schema into the database, Rails will use the default PK convention of setting the column type as sequence. Why is this a problem?

The first problem is repeatability. The only way to create the same DB you have in development is to run all of the migrations. The schema.rb file has this big disclaimer at the top of the file:

Note that this schema.rb definition is the authoritative source for your database schema. If you need to create the application database on another system, you should be using db:schema:load, not running all the migrations from scratch. The latter is a flawed and unsustainable approach (the more migrations you’ll amass, the slower it’ll run and the greater likelihood for issues).

It’s strongly recommended that you check this file into your version control system.

The glaring problem with this statement is that, because of the way this schema is generated for our special case, the schema file no longer represents our database structure. So what’s a database to do?

There are two options, a good one, and a better one. The good one involves creating a table with no primary key, a uuid column, and a mix of constraints and an index that mimic the properties of PRIMARY KEY. Here’s what that looks like in Rails migration land”

create_table "users", id: false "my_cool_pk" do |t|
  t.uuid :my_cool_pk, null: false   # you can add the default of uuid_generate_v4() if you like
end

create_index :users, :my_cool_pk, unique: true

This basically makes my_cool_pk follow all the same conventions as a PRIMARY KEY field. You can see from the Postgres documentation that this is basically what PRIMARY KEY does (though some sources tell me that it is implemented with some differences). “But Luke”, you say, as a responsible developer should, “what about the performance?”. I’m glad you asked. I ran (albiet through Rails ActiveRecord) some quick benchmarks to see how selection over an index faired over a PRIMARY KEY field. The following is my test code and benchmark results.

RECORD_COUNT = 10_000
user_ids = []
other_user_ids = []

RECORD_COUNT.times do
  id = SecureRandom.uuid

  user_ids << User.create.id
  other_user_ids << OtherUser.create.id
end

Benchmark.ips do |x|
  x.time = 10

  x.report('index') { User.find user_ids.sample }
  x.report('pk') { OtherUser.find other_user_ids.sample }
  x.compare!
end

# Calculating -------------------------------------
#               index    20.000  i/100ms
#                  pk    20.000  i/100ms
# -------------------------------------------------
#               index    214.927  (± 7.0%) i/s -      2.140k
#                  pk    217.380  (± 9.2%) i/s -      2.160k
#
# Comparison:
#                  pk:      217.4 i/s
#               index:      214.9 i/s - 1.01x slower

So the resulting performance between the two field types (PRIMARY KEY vs index + constraints) have negligible performance impact. So that’s an option.

However, there’s one other option to get Rails to cooperate: Ditch the troublesome schema.rb. This option is helpful if you want to still want to follow good DB practice while still using Rails’ db sync mechanisms. The main one being test db preperation. Rails has/had/has again a db:test:prepare task which loads the test database from the schema if there are any migrations pending on the test db.

task :prepare => %w(environment load_config) do
  unless ActiveRecord::Base.configurations.blank?
    db_namespace['test:load'].invoke
  end
end

And the test:load action:

task :load => %w(db:test:purge) do
  case ActiveRecord::Base.schema_format
    when :ruby
      db_namespace["test:load_schema"].invoke
    when :sql
      db_namespace["test:load_structure"].invoke
  end
end

So Rails can use an actual SQL structure file instead of the schema file, so long as you’ve set config.active_record.schema_format = :sql in your application.rb config file. This, from all the investigation I’ve made so far, does the Right Thing™ in generating the correct PRIMARY KEY field with the correct uuid type.

CREATE TABLE users (
    my_cool_pk uuid DEFAULT uuid_generate_v4() NOT NULL
    # ... etc ...
);

ALTER TABLE ONLY users
    ADD CONSTRAINT users_pkey PRIMARY KEY (my_cool_pk);

This means the db:test:load/prepare tasks will work normally. The only thing that breaks is you can no longer use db:schema:load. However, the db:structure:load task will work just as well.

Update

Short Twitter convo with a few Heroku engineers on the core PG team:

@plukevdh yep http://t.co/9uTDYwNiZr
— Craig Kerstiens (@craigkerstiens) June 30, 2015

@craigkerstiens do you know of any reason the performance would differ between the two?
— Luq (@plukevdh) June 30, 2015

@plukevdh you mean creating a PK vs. a unique index not null ?
— Craig Kerstiens (@craigkerstiens) June 30, 2015

@craigkerstiens @plukevdh They're equivalent in how they perform.
— Peter Geoghegan (@petervgeoghegan) June 30, 2015

Date, DateTime and Time, Oh My!

Wed, 17 Jun 2015 00:00:00 -0400

TIL DateTime / Time.now does not respect app time zone settings. TheDateTime / Time.current appears to be the correct (and documented) means of accessing the application’s TZ-adjusted time.

Time.now.zone
#=> "EDT"

Time.current.zone
#=> "UTC"

The “equivalent” Date.today is also unpredictable:

Time.now
#=> 2015-06-17 23:17:33 -0400

Time.current
#=> Thu, 18 Jun 2015 03:17:38 UTC +00:00

Time.current.to_date
#=> Thu, 18 Jun 2015

Date.today.send :zone
#=> "+00:00"

Date.today
#=> Wed, 17 Jun 2015

This seems to be the TZ respecting way to get just date for today:

Time.current.to_date
#=> Thu, 18 Jun 2015

The x.days.from_now etc helpers do appear to respect TZ out of the box.