Chances are, some of you have run into the issue with the
invalid byte sequence
in UTF-8 error when dealing with user-submitted data. A Google
that my hunch isn’t off.
Among the search results are plenty of answers—some using the deprecated iconv library—that might lead you to a sufficient fix. However, among the slew of queries are few answers on how to reliably replicate and test the issue.
In developing the Griddler gem we ran into some cases where the data being posted back to our controller had invalid UTF-8 bytes. For Griddler, our failing case needs to simulate the body of an email having an invalid byte, and encoded as UTF-8.
What are valid and invalid bytes? This table on Wikipedia tells us bytes 192, 193, and 245-255 are off limits. In ruby’s string literal we can represent this by escaping one of those numbers:
> "hi \255" => "hi \xAD"
There’s our string with the invalid byte! How do we know for sure? In that IRB
session we can simulate a comparable issue by sending a message to the string it
won’t like - like
> "hi \255".split(' ') ArgumentError: invalid byte sequence in UTF-8 from (irb):9:in `split' from (irb):9 from /Users/joel/.rvm/rubies/ruby-1.9.3-p125/bin/irb:16:in `<main>'
Yup. It certainly does not like that.
Let’s create a very real-world, enterprise-level, business-critical test case:
require 'rspec' def replace_name(body, name) body.gsub(/joel/, name) end describe 'replace_name' do it 'removes my name' do body = "hello joel" replace_name(body, 'hank').should eq "hello hank" end it 'clears out invalid UTF-8 bytes' do body = "hello joel\255" replace_name(body, 'hank').should eq "hello hank" end end
The first test passes as expected, and the second will fail as expected but not with the error we want. By adding that extra byte we should see an exception raised similar to what we simulated in IRB. Instead it’s failing in the comparison with the expected value.
1) replace_name clears out invalid UTF-8 bytes Failure/Error: replace_name(body, 'hank').should eq "hello hank" expected: "hello hank" got: "hello hank\xAD" (compared using ==) # ./invalid_byte_spec.rb:17:in `block (2 levels) in <top (required)>'
Why isn’t it failing properly? If we pry into our running test we find out that
inside our file the strings being passed around are encoded as
 pry(#<RSpec::Core::ExampleGroup::Nested_1>)> body.encoding => #<Encoding:ASCII-8BIT>
As a result we’ll have to force that string’s encoding to UTF-8:
it 'clears out invalid UTF-8 bytes' do body = "hello joel\255".force_encoding('UTF-8') replace_name(body, 'hank').should_not raise_error(ArgumentError) replace_name(body, 'hank').should eq "hello hank" end
By running the test now we will see our desired exception
1) replace_name clears out invalid UTF-8 bytes Failure/Error: body.gsub(/joel/, name) ArgumentError: invalid byte sequence in UTF-8 # ./invalid_byte_spec.rb:4:in `gsub' # ./invalid_byte_spec.rb:4:in `replace_name' # ./invalid_byte_spec.rb:17:in `block (2 levels) in <top (required)>' Finished in 0.00426 seconds 2 examples, 1 failure
Now that we’re comfortably in the red part of red/green/refactor we
can move on to getting this passing by updating our
def replace_name(body, name) body .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '') .gsub(/joel/, name) end
And the test?
Finished in 0.04252 seconds 2 examples, 0 failures
For such a small piece of code we admittedly had to jump through some hoops. Through that process, however, we learned a bit about character encoding and how to put ourselves in the right position—through the red/green/refactor cycle—to fix bugs we will undoubtedly run into while writing software.