All posts by james

Inline parsing JSON objects in 3 languages

JavaScript Object Notation commonly known as JSON, is a convenient format for many reasons and it is no wonder that many APIs and web services today support returning (in some cases exclusively) JSON.  Many times I have had a need to parse values from large JSON Objects in a shell script environment or a command line shell, and use them as inputs to another program.  This has put me on the look out for easy ways to handle inline parsing of JSON Objects using languages commonly available on modern Linux systems.

Below are examples in three languages (Node JS, Python and Perl) to accomplish this task.

Example JSON Object:

{"product": {"builds": {"1234": "IT WORKS!"}, "default": "1234"}}

Inline Node JS:

node -e "s=''; i=process.stdin; i.on('data', function(d) { s += d }); i.on('end', function() { j=JSON.parse(s).product; console.log(j.builds[j.default]) })"

Inline Python:

python -c 'import sys, json; j=json.load(sys.stdin)["product"]; print j["builds"][j["default"]]'

Inline Perl:

perl -e 'use JSON; local $/; $d=decode_json(<>)->{product}; print $d->{builds}->{$d->{default}}'
  • The JSON module is required in the above example. If not already available it can easily be installed via CPAN with:
sudo perl -MCPAN -e 'install JSON'

Full example for Python:

echo '{"product": {"builds": {"1234": "IT WORKS!"}, "default": "1234"}}' |python -c 'import sys, json; j=json.load(sys.stdin)["product"]; print j["builds"][j["default"]]'

Scoop – From life to work, and back again.

I have departed VMware after 5 years and am now at a new company based in San Francisco that specializes in providing a service for scheduling carpools. Very excited about this!

On a side note, a friend and I were talking the other day and decided that we need to write more posts about all the cool stuff we do. So with that in mind I will be making a point to be more active around here as I am working with an entirely different software stack at the new company.

SureTrak 3.0 Printing Issues

Recently a client called me in to investigate an issue they were having with SureTrak 3.0 by Primavera. The program would crash whenever they tried to print a schedule. This is legacy software and no longer being updated or supported. Primavera product line was recently bought by Oracle but backward compatability is not guaranteed and this particular client frequently works with old data files so backward compatibility is an absolute must.

The Suretrak software states that it is compatible with Windows 98 and ME but not XP. My initial review of the issue indicated a software incompatibility with XP however the software had been working fine and the issue had appeared out of nowhere and was causing Suretrak to crash on all systems when attempting to print. The solution was not apparent but after a number of tests I was able to conclusively determine that the problem was due to path length limits from the MS DOS era.

(TLDR; Shorten your path and/or filename so it does not exceed the maximum length of 79 characters.)

It has been nearly 20 years since I was developing software for MS DOS so I needed to refresh my memory on the subject. DOS paths are limited to 64 characters; 66 if you count the storage designation (drive letter and colon) I’m sure more seasoned DOS coders will remember the magic number as being 80 characters for allocating string buffers. The following helpful excerpt was taken from:

The longest filename string including the drive letter and colon is
79 characters (many programmers remember the magic number to be 80
 which includes the terminating "nul" character at the end).

Another limit many users overlook is the 64-character limit on the
pathname.  In this context, the 64-character limit starts with the
first backslash which represents the root directory.  If you add
the common volume specifier (a drive letter plus a colon), the
maximum length for the pathname will be 66.   Now, the longest name
(the "lastname") in the so-called 8.3 DOS naming convention is 12
characters.  Therefore, the total is

   66 + 1 + 12 = 79

The "+ 1" in the middle is for the last backslash.  Adding the
terminating nul character at the end will make up the magic number
of 80 bytes which most people remember.

The better way to remember the limits is to remember that the
longest subdirectory name allowed in DOS is 66 characters.  The
80-character limit commonly sited should be only for programmers
who need to allocate a buffer of 80 bytes.  If you remember the
66-character limit, then the 79 character limit can be derived.

That says it all right there. Needless to say identifying the issue was the hard part, the solution was easy. Keep your path lengths reasonably short and you will never have to deal with this elusive and obscure bug!

Get more free space on large ext3 partitions

Storage capacities have grown exponentially over the past few years and it is not uncommon when building new systems to configure file systems on volumes larger than a 1TB. By default ext3 partitions reserve 5% of the space for the system. On large partitions this is much more than necessary and can be safely reduced to 1% with “tune2fs -m 1 /dev/hd??”

3ware RAID controllers and the command prompt

3ware makes some great RAID products however when it comes down to configuring them things get a little confusing as their website isn’t completely clear what is offered in the packages available for download. Suffice to say that if you are looking to manage a 3ware RAID controller from a Linux command prompt then the little gem you’re looking for is called ‘tw_cli’ and NOT the 3dm2 package which is some sort of web interface frontend package; completely useless to shell guys like me.

If you are reading this and have already went down the 3dm2 road like I did, you can get rid of it by running “install.3dm uninstall” which isn’t that intuitive or obvious but works great! Just make sure you kill the 3dm2 processes first if they’re running.

Then head over to 3ware and get the goods:

Free RADIUS patch

Submitted a bug fix / patch to the rlm_unix module which was merged with the main branch of the Free Radius project on June 30th 2003.

Revision 1.57 / (download) – annotate – [select for diffs], Mon Jun 30 15:22:45 2003 UTC by aland
Branch: MAIN
Changes since 1.56: +3 -3 lines
Diff to previous 1.56 (colored)
Use the ‘canonical user name’ for Unix group lookups, not the
User-Name attribute.

Patch from James J. Williams