25 March 2010

CodeSnip problems, problems, problems

Arrgh!

I've been having a lot of problems with the CodeSnip program's database update code - it's been working for some and not for others. All this has happened since I converted the program to Unicode and compiled with Delphi 2010.

I'm posting this for two reasons:

  1. To try to explain to long suffering users what has been going on with program lately - it's not usually this flaky.
  2. To forewarn anyone about to fall down the same hole as I have.

The problem never raised its head on my system, but did on some that don't use the Windows-1252 code page. The problem was caused by a checksum failure, which in turn was caused by the downloaded data being converted to the wrong code page before the checksum got calculated: result - bang - bad checksum.

My code relies on Indy components to translate downloaded content into text - and Indy and I make different assumptions about how this should be done. And I'm not sure either of us was correct, but they are a much more experienced bunch than I am so I'm prepared to take the blame!

The upshot of all this is that a proper fix insn't going to be trivial. I produced a temporary fix in v3.5.4 by the simple expedient of not checking the checksum. Then I went and commented out a crucial piece of code that meant that downloaded failed for everyone. At least that's fair I suppose because it screwed up UK and USA users just as much as those in the rest of the world!

Next up is a total rewrite of the web service management code - I'm going to handle the raw data myself and perform my own translations rather than let Indy do it. And I'm going to do the checksum properly using HTTP headers rather than my own custom approach.

I may write some of this up here once I've worked it all out.

And the moral of this story - be damn careful about character encoding when interacting with web services, and be sure you understand what your components are up to.

Some news for CodeSnip users: I've hacked the update web service to recognise v3.5.4 and feed it data that doesn't make it fall over. And I've released v3.5.5 that does the temporary fix that v3.5.4 was supposed to do. Please update. Again!

No comments: