Product and corporate development

A post written by Tor started me thinking about business and product development, especially in this age of online products. Classically, and not so many years ago, a company would have years in which to develop and maintain a new product (such as friends-reunited) and then have years to recoup any investment and of course profit!

However in the current markets, there are lots of people, with easy access to both the talent and technologies (developers and already pre-scaled technologies such as Amazon Web Services), that allows for almost any small startup to be able to deliver a globally accessible and scalable web application in essentially a very much shorter development cycle than many “classical” companies can manage.

This is less important in markets where the company is the sole provider of goods and services, such as Satellite TV in the UK (BSkyB), where as an effective monopoly the drive for innovation is less extreme, and that new entrants to the markets cannot compete so directly, and hence normal free-market conditions cannot happen.

Of course, all of this leads to the further development of “lean” development methods, and driving fast development of new products and features, where you are in a strongly competitive market.

java log files

So debugging java can be a mighty pain, a little few lines of ways to make it simpler.


# grep -m 1 -n '^2010-06-30 14:20:' catalina.out
11746233:2010-06-30 14:20:01,011 DEBUG com.beginning.of.line
# grep -m 1 -n '^2010-06-30 14:21:' catalina.out
11747788:2010-06-30 14:21:00,161 WARN org.apache.commons.httpclient.HttpMethodBase - Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended
# sed -n '11746233,11747788p' < catalina.out

This gives start line and end line between any two greps and then the body of the log file from those two lines.

Snow Leopard and the jvm


So in a startling case of “Apple knows best” they have gone and removed the 1.5 JVM – I know it’s pretty old and people were complaining about not having a 1.6 JVM for a long time – but really!

So the following post steps you through re-enabling the 1.5 VM and now semi-happy days

SNMP over SSH tunnels

Sometimes you just need to tunnel UDP based protocols – such as SNMP – and the easiest ways is to use socat


socat tcp4-listen:6667,reuseaddr,fork UDP:DESTINATION:161
socat udp4-listen:161,reuseaddr,fork tcp:localhost:6667

And in combination with your normal SSH tunnel


ssh -L6667:localhost:6667 BASTION_HOST

Smokeping and measuring SOAP requests

We’ve had an issue with performance of a SOAP interface, and here’s how you go about setting up smokeping to time it:-

extraargs = -H Content-Type:text/xml --data @/srv/scripts/soap_check/soap-test.xml
urlformat = http://server.name.com/url/soap_url

The only annoying problem is that the SOAP payload cannot be included as part of the command line, so any slaves would require the file manually copied into the same location

UTF-8 and MySQL databases

Nice bugs that no one seems to want to fix:-


mysql> create table test5 (
wibble varchar(500), PRIMARY KEY (wibble)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes


mysql> create table test5 (
wibble varchar(500), PRIMARY KEY (wibble)
) ENGINE=myisam DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
ERROR 1071 (42000): Specified key was too long; max key length is 1332 bytes

And even with MySQL 6.0.9 (using the Falcon engine which is still brand new)

mysql> create table test5 (
wibble varchar(500), PRIMARY KEY (wibble)
) ENGINE=falcon DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
ERROR 1071 (42000): Specified key was too long; max key length is 1100 bytes