Ruby – Strange \n in base64 encoded string in Ruby

base64encoderuby

The inbuilt Base64 library in Ruby is adding some '\n's. I'm unable to find out the reason. For this special example:

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'base64'
=> true
irb(main):003:0> str =  "1110--ad6ca0b06e1fbeb7e6518a0418a73a6e04a67054"
=> "1110--ad6ca0b06e1fbeb7e6518a0418a73a6e04a67054"
irb(main):004:0> Base64.encode64(str)
=> "MTExMC0tYWQ2Y2EwYjA2ZTFmYmViN2U2NTE4YTA0MThhNzNhNmUwNGE2NzA1\nNA==\n"

The \n's are at the last and 6th position from end. The decoder (Base64.decode64) returns back the old string perfectly. Strange thing is, these \n's don't add any value to the encoded string. When I remove the newlines from the output string, the decoder decodes it again perfectly.

irb(main):005:0> Base64.decode64(Base64.encode64(str).gsub("\n", '')) == str
=> true

More of this, I used an another JS library to produce the base64 encoded output of the same input string, the output comes without the \n's.

Is this a bug or anything else? Has anybody faced this issue before?

FYI,

$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

Best Answer

Edit: Since I wrote this answer Base64.strict_encode64() was added, which does not add newlines.

The docs are somewhat confusing, the b64encode method is supposed to add a newline for every 60th character, and the example for the encode64 method is actually using the b64encode method.

It seems the pack("m") method for the Array class used by encode64 also adds the newlines. I would consider it a design bug that this is not optional.

You could either remove the newlines yourself, or if you're using rails, there's ActiveSupport::CoreExtensions::Base64::Encoding with the encode64s method.

Related Solutions

Ruby – How to call shell commands from Ruby

This explanation is based on a commented Ruby script from a friend of mine. If you want to improve the script, feel free to update it at the link.

First, note that when Ruby calls out to a shell, it typically calls /bin/sh, not Bash. Some Bash syntax is not supported by /bin/sh on all systems.

Here are ways to execute a shell script:

cmd = "echo 'hi'" # Sample string that can be used

Kernel#` , commonly called backticks – `cmd`

This is like many other languages, including Bash, PHP, and Perl.

Returns the result (i.e. standard output) of the shell command.

Docs: http://ruby-doc.org/core/Kernel.html#method-i-60
```
value = `echo 'hi'`
value = `#{cmd}`
```
Built-in syntax, %x( cmd )

Following the x character is a delimiter, which can be any character. If the delimiter is one of the characters (, [, {, or <, the literal consists of the characters up to the matching closing delimiter, taking account of nested delimiter pairs. For all other delimiters, the literal comprises the characters up to the next occurrence of the delimiter character. String interpolation #{ ... } is allowed.

Returns the result (i.e. standard output) of the shell command, just like the backticks.

Docs: https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-Percent+Strings
```
value = %x( echo 'hi' )
value = %x[ #{cmd} ]
```
Kernel#system

Executes the given command in a subshell.

Returns true if the command was found and run successfully, false otherwise.

Docs: http://ruby-doc.org/core/Kernel.html#method-i-system
```
wasGood = system( "echo 'hi'" )
wasGood = system( cmd )
```
Kernel#exec

Replaces the current process by running the given external command.

Returns none, the current process is replaced and never continues.

Docs: http://ruby-doc.org/core/Kernel.html#method-i-exec
```
exec( "echo 'hi'" )
exec( cmd ) # Note: this will never be reached because of the line above
```

Here's some extra advice: $?, which is the same as $CHILD_STATUS, accesses the status of the last system executed command if you use the backticks, system() or %x{}. You can then access the exitstatus and pid properties:

$?.exitstatus

For more reading see:

Javascript – How to you encode a string to Base64 in JavaScript

You can use btoa() and atob() to convert to and from base64 encoding.

There appears to be some confusion in the comments regarding what these functions accept/return, so…

btoa() accepts a “string” where each character represents an 8-bit byte – if you pass a string containing characters that can’t be represented in 8 bits, it will probably break. This isn’t a problem if you’re actually treating the string as a byte array, but if you’re trying to do something else then you’ll have to encode it first.
atob() returns a “string” where each character represents an 8-bit byte – that is, its value will be between 0 and 0xff. This does not mean it’s ASCII – presumably if you’re using this function at all, you expect to be working with binary data and not text.

Best Answer

Related Solutions

Ruby – How to call shell commands from Ruby

Javascript – How to you encode a string to Base64 in JavaScript

See also:

Related Topic