Tuesday, 5 August 2014

More weird adventures in Cache SQL - SOLVED

Well made a new interesting discovery today. Combining TOP with ORDER BY can severely slow down a query.

I was looking into a slow query which was taking around 10 seconds to return 69 results. It was getting the "TOP 100" from a specific date range and then ordering by ID descending. It was fascinating to me to discover that removing the TOP 100 returned the results in 0.005 seconds. Removing the ORDER BY instead of the TOP 100 also resulted in again producing the results in 0.005 seconds.

So the moral of the story is don't use TOP and ORDER BY if you want your results back quickly.

I have no idea why this kills performance right now, but it is perhaps even more interesting that even though the query TOP 100 ORDER BY ID ASC results in the same set of results as TOP 100 without an ORDER BY command, it takes 8.8 seconds rather than 0.005 seconds. There are definitely some huge optimisations available in respect to these queries.

Update

Well after reporting this to Intersystems I was directed to %NOTOPOPT query optimisation method. It states that without this TOP is optimised for first row retrieval rather than the full result set. This implies to me that a TOP 1 should be faster without %NOTOPOPT and that returning more than 1 has the chance to be faster with it. However, when I was testing across various queries whether it was TOP 1 or TOP 100 with an ORDER BY I always retrieved the results much faster with %NOTOPOPT. If you have a TOP query running slowly I recommend testing with this optimisation option, as you may see much better performance as I did.

Saturday, 2 August 2014

So which is the best mobile keyboard 2014

PhotographerCindy J Grady
I have been lucky enough to experienced using keyboards on iOS, Android and Windows Phone (8 and 8.1) which has really shown me how important a good input method is in the mobile experience.

Each keyboard has unique features which makes choosing the best keyboard a bit of a struggle. Until recently I had completely dismissed iOS keyboard for the following reasons

1. 4" display means that the keys are harder to press accurately
2. Selecting & deleting words is trickier than both Android and Windows Phone not just due to the smaller target area, but also the implementation requires a "long press" to activate the magnifying glass to achieve accurate cursor location.
3. Lack of swype / shape writing style input.

While options 1 and 2 were of just minor inconvenience option 3 is a major issue.

Swyping / Shape writing

There are three great features of shape writing

1. It is a much faster input method on the whole than key pressing as demonstrated by the numerous keyboards of this style achieving Guiness World Records.

2. It allows a more generalised area of keypress which is of great advantage on a smaller screen, or even allows you to cope with a smaller keyboard on a larger screen giving the benefit of seeing more of what you have typed so far.

3. The input can be done at full speed with a single digit. Rather than the common double thumb entry for iOS to generate speed shape style writing can be performed one handed.

What Does iOS have up its sleeve

Historical Autocorrect

Obviously we have all had a laugh at damn you autocorrect and probably have received a couple of weird messages due to autocorrection errors. However, autocorrection is pretty clever, but on iOS it actually tracks back up to 4 words based on what you are entering and can correct the 4th word. Contextually autocorrecting 4 words back will tend to help weed out a huge number of errors, and is certainly a clever feature to have,

Cross Platform Support

Apple has always integrated its devices well, anyone with an iPad, Apple TV and Mac will know that so many things transfer relatively seamlessly across without the need to be tech-savvy. iMessages synchronising with your devices is a good example of this. Well the iOS keyboard has another one of these clever features, simply create keyboard macros on your Mac and they are straight away available on your iPhone and iPad.

While the stock Android keyboard offers keyboard macros (as well as some less popular alternatives) these macros are limited in length. If for example you had a long HTML sign off signature you were particularly proud of this would probably be too big for the dictionary. However, Kii keyboard will store macros that are large enough, but the dictionary does not sync across devices, so you need to re-enter any macros on all devices you want to use.

Is this enough?

I can understand that these features can be sufficient to turn the tide for some, but shape writing is such a massive time saver for 99% of the text I input that synced macros is simply not important enough. So sadly iOS keyboard is ruled out of the running, leaving Google Keyboard, Windows 8.1, Swype, Swiftkey, SlideIT & Kii.

Google Keyboard

Google have built a very good basic keyboard. It is fast and responsive, even on lower end devices, which is not true of some of the other options. I have found it to be quite accurate and fast, and is my recommendation if you have a slower device as there is nothing quite as frustrating as a sluggish keyboard response.

SwiftKey

SwiftKey recently became free, which is quite a bonus in its favour. The next work predictions are quite scary, it makes you realise just how much you repeat yourself and various phrases. I find this aspect a bit creepy almost as if it knows what you are about to say. It does however, inspire me to try and vary my language a bit more almost like a challenge to say, "No you are wrong! I was actually going to use a 16th century old English word, take that dictionary!"

This is probably the second best keyboard for non-shape writing input, with iOS beating it and Windows 8.1 coming in a very close third. I know that some people cannot get on with shape writing and if you are one of those then SwiftKey is probably your best bet on Android.#

SlideIT

I will be brief about SlideIT to save you sometime, the predictions are not a good as the other options, it does seem to be relatively low on resources, but overall the other options including the stock keyboard are superior.

Kii

Kii tries to be a combination of SwiftKey and Swype, the options available are staggering it definitely has the best options for personalisation. I have found that its predictions are good, but not the best and so I am not placing it at the top, however, there are some relatively unique features such as dual language input to allow seemless corrections for bi-lingual users (no dictionary switching required) and the ability to customise the size and placement of the keyboard. Kii is certainly worth a try and you may find that the personalisation options make it the best option for you.

Swype

Swype continues to offer the best predictions and best corrections. By comparison I rarely find the need to go back and make corrections. It is nearly perfect, but there are a couple of let downs. I always turn off Dragon Speak as I find it is very bad at managing to identify anything I say, this means there is no quite way of activating Google voice for dictation. This is a minor inconvience for me, but I could see others finding it a deal breaker. I would also love to be able to customise symbol key placement and perhaps create gestures like Kii offers. It does over some very useful customisation in the form of keyboard size and placement, which can allow you to perform one handed typing on a much larger device. I have found it possible to shape write one handed on a Nexus 7 without issue and with many devices pushing 6" it could be of great benefit to many.

Fundamentally the accuracy and the best autocorrection leads me to choose Swype as the best Android Keyboard, but is it the best keyboard of all?

Windows 8.1

Windows 8 has very good correction algorithms for corrections making it one of the best keyboards for typing, but 8.1 introduced shape writing. The accuracy of the shape writing is similar to that of Swype, leading to rarely requiring post writing corrections. It also has a couple of unique features, you can for example add or remove capitalisation from an already complete word just by placing the cursor in it and pressing shift. Additionally the voice recognition is of a similar quality to that of Google's and with this minor addition I would rate the keyboard as a fraction above Swype's offering, making the Windows 8.1 keyboard the best mobile keyboard. Sadly it is only available on Windows Phone, and the Windows Phone platform while great in many ways is not the equal of Android right now.

iOS 8

One of the features that I am really interested in is iOS 8's opening up of the keyboards. I am really curious how many people will switch away to Swype or Swift key.

Sunday, 27 July 2014

How to improve the speed of your web page, the basics

Is saving a few seconds really important?

Image by FracFx
While practically any change, no matter how sure you are that it is a positive improvement, can be met with anger from the user base performance improvements are not one of them.

In an ideal world all webpages would appear the instant we demanded them, however achieving good performance has costs, and those costs need to be offset against the benefits. Features and complexity sells and performance is just one of those features which can be easily ignored and tough to sell, especially if it is say the difference between 10 seconds and 5 seconds.

The benefits of speed are often underrated and can be difficult to appreciate. For example a fast application is easier to use, not just because of the reduced frustration of waiting, but because it is easier to explore. If you click in the wrong place as the next piece of information appears instantly you can easily explore the options on the page. If every option takes ages not only are you more careful in your decisions taking long to complete any interaction, you also will not explore and perhaps find new better pathways to complete your task.

Another aspect of speed is not only does it allow you to explore, because it does not take long to get back to a known pathway, but it gives you further confidence that the application is well made. Confidence in the application again has unseen benefits such as the user is more likely to assume that the problem they are encountering is of their own making and not the application, reducing support calls as they try to resolve their issues themselves.

Also people can start using more optimised work practices to save themselves time, often what the developer sees as the "correct" pathway can be ignored because the "incorrect" pathway is faster.

Saving time, even seconds can add up to a lot more than the some of its parts.

What do I do next?

Well done you have managed to secure resource to improve performance, but where do you start? There is only one place to start measuring. Always start by monitoring and measuring how long the page takes to display. There is an issue that by logging time taken does actually slow down the page which might be considered an issue in a live environment. However, it is important to remember that the performance bottlenecks spotted in a test environment may not full reflect your live platform.

For the remainder of this article I am going to focus purely on the HTML, CSS and JavaScript and the page rendering performance. To truly improve performance you need to consider all factors and in fact some of the best gains can be achieved through a more holistic approach. For example a great performance improvement pattern is to anticipate the next request and cache this pathway. This works extremely well with a string find UI, before the user clicks "Find Next" the system has already calculated where the next string match is and jumps to it immediately. Simple approaches like this can get around the requirements for massive amounts of power or super efficient algorithms. Of course most of my best gains have been working on the "back end" processes, but there is plenty that can be done at the "front end" too.

Minimise

It is not exactly rocket surgery to realise that reducing the amount of data sent to the browser will improve performance. This is often the simplest change to make and most of the time gives you the biggest boost in performance.

TCP/IP connections initially send a packet of 14kb followed by the rest of the data. If you are accessing the page via cellular data or perhaps through a satellite connection then latency of the connection becomes an issue. Latency can be in the hundreds of milliseconds so if you are aiming for great responsiveness then the initial 14kb becomes important. If at all possible it would be optimal to provide all of the data necessary to display the visible portion of the page in the first 14kb. To achieve this it is vital to minimise everything that is being sent

Lets take a look at a corporate website example. City Link is a large UK logistics company. On a decent broadband connection their website pops up pretty quickly, but, it could be faster.

Images

The home page comes in at a total of 567 kb. 256 kb of the total are the images. Most of these are compressed appropriately, it is possible to compress these without any loss to image quality and achieve a reduction of around 153 kb. Now of course you could argue that some of the images could be replaced with HTML and CSS for a much larger reduction in size, if there is opportunity for this then it would obviously be a good place to investigate next.

HTML

The HTML for the page is 29 kb, by just removing carriage returns and unnecessary space this page drops to 22kb. Now the results are not quite so impressive if you consider that this will be sent over the line compressed, but we are still able to see a drop of 7kb to 6kb.

CSS

Again none of the CSS files are compressed, while the built in server compression again helps hide the benefits of compression it is still possible to save 3kb off the 30.6 kb total without the need to try and combine CSS rules, or cull any unnecessary rules.

JavaScript

The page loads 17 JavaScript resources totalling 245 kb of data. Again a lot of this is not compressed and it is easy to save 8kb of data just by "minifying" the JS.

Requests

The page makes a total of 52 request to get all of the required files. Each request has quite an overhead and has quite an impact on how quickly the data gets to the user as well as the amount of traffic required to pass this data in. It is possible to combine a number of the images and use image slicing to produce the images. As the images are almost entirely green, yellow and white as the pallet is not diverse combining the images will compress well and can substantially reduce the overall image requirements on the page.

Combining the CSS will have a similar effect. There are 4 css files, although one is served from Google the other 3 can be combined quite easily to reduce the number of requests.

Load the JavaScript as late as possible

JavaScript blocks the rendering of the page, moving the JavaScript from the header to the end of the body can often get the visuals up and running before the event control in the JavaScript is required.

Do not load what you do not need

I might be missing something in the code, but I cannot see where the Jquery.mousewheel.min.js is used. If this is indeed not in use then removing this code gets rid of an unnecessary http request, 1 kb of data and also stops the browser delaying render while it parses this file. Try to load JavaScript or any other resource as conditionally as possible, yes your site will Cache resources so the overall effects might not be too bad, but getting rid of an unnecessary parsing will improve the responsiveness even just a little.

Fix any broken HTML

When the HTML is in error the browser triggers its error handling, this is only a fraction slower, but may trigger various quirks modes, this probably leads to extra CSS and HTML to "fix" the problem. Running a validator such as HTML Tidy is invaluable in preventing this path. City Link only triggers 7 warnings, this is far fewer than the majority of pages I have looked at, but these 7 could quickly be addressed for a perfect score.

Sadly I have used technologies which prevent you from getting a perfect score for example they force you to use lang or type='javascript' when you are working in HTML5 because the source will not compile otherwise. fixing broken HTML is not always possible, but when it is try to use HTML and rid your page of those warnings.

Do not use XHTML

Some websites use XHTML such as City Link. Besides the fact that support for this version of HTML within browser is poor and they tend to just parse as if it were HTML, it is not compatible the with XHTML 2 draft and perhaps more importantly it requires more verbose HTML than HTML 5. Save precious bandwidth and choose HTML 5.

Friday, 25 July 2014

The big constraint in Development and how it can bring down a team

I always consider the primary constraint in software development to be time. While technically this constraint could be considered a function of money, it is difficult for a developer to assert direct influence over budget. Most organisations require developers to go quite high up the chain of influence in order to try affect the proportion of budget development have access to. Supply and demand factors for the product that development are supplying is difficult for them to directly influence, in most corporate structures sales and marketing are the real influencers of this factor. Yes they rely on the quality of product that development provide, but it is unlikely that and improvement in the product quality will have an equivalent impact on sales. The ability of sales and marketing to reach new customers and alter demand is a much more influential factor.

It is one of the great features of software development that you can essentially do anything the customer asks for, but only if you have enough time. Developers realise how precious this commodity is. There will always be bugs in the product that can be fixed, there will always be ways you can enhance the existing features and there will always be brand new features which can benefit the customer. As your team is not infinite in size and your system resources are finite you are always unable to write super efficient bug free code covering every desired feature.

All the time a developer is working on a given section of work they will be performing cost benefit decisions, how much time do they spend optimising their changes for performance, how much time do they spend reviewing/testing it for bugs or coding defensively against them. They weigh these factors up against what they perceive as the risks of the code and against the relative importance of the change.

Software developers have normally entered the field due to a natural aptitude for logic. Logical reasoning will commonly be a very important to how they perceive the world. This is critical in understanding how best to manage a team of developers. A manager must be able to provide logical reasoning for the decisions he makes and more importantly communicate these reasons effectively to their team.

The development team will often know what processes appear to be impeding their time, they will be aware of many of the issues the product they are working on, and have a good idea of how long each will take to fix. They will probably even be able to provide you with an ordered list based on their cost benefit analysis of what should be done. Now if that list tallies with the managers scheduling plans then the team will be functioning smoothly, the developers will naturally respect the managers decisions. However, if that list does not tally then there is potential for a break down within the team dynamic.

The manager will need to explain their priority order clearly and provide rational explanations for the differences in opinion. I have not yet been in a team where the developer has not altered their opinion in the face of rational argument. It is vital that the manager should be open to change in the same way. If the manager cannot provide a consistent rational argument to influence a change in opinion of the developer they should strongly consider if their ideas are the best way forward. If the decision making is obviously particularly contentious then it is important for the manager to review their decisions at the end of the process and if they were wrong admit to this, if they were right they should provide this evidence to the team. Everyone has some level of ego so obviously this should not be presented in a gloating manner, however, as I have stated developers tend to work from a logical stand point, if the manager adequately proves their decision making process was effective then the developer will be able to maintain their respect and the team should continue to function well.

The more the developers see the decisions of their manager as irrational the more respect they will lose, the lose of respect will lead to reduced communication and a drop in the effectiveness of the team.

Developers normally want to release the best products possible and realise that the product quality and breadth is influenced by the time available to code any seemingly irrational scheduling or processes eat into this time and are seen by the developer as barriers thrown up to stop them producing a great product. I have not spoken to a developer yet who does not agree with the agile manifesto and I believe it is clear that this document was borne out of experiencing frustratingly irrational management processes that the developers saw as restricting the precious resource of time.

Sunday, 20 July 2014

Intersystems Cache - Write Performance

In previous blog posts I have mentioned that Cache has good write performance, in fact in certain circumstances it is faster than many other databases. However, it is rarely the case that optimising performance is not desirable. Every one loves a performance boost, it is the most universally loved product change. The closest I have ever experienced to a complaint about a performance boost is "its so fast I didn't believe it had worked".

Any way I was working on improving query performance on a server when I noticed some intermittent slow down. I traced the performance issue to a task which performs a huge burst of write operations. The write operations were so heavy that the rest of the server was experiencing slow down.

Obviously the code is under review to see if the amount of data being written is necessary, but I was curious about optimising the write operation itself. With read operations I have noticed that dynamic SQL is slower than compiled SQL which is in turn slower than direct global access, in certain circumstances this performance difference can be 10-100 times difference. I wanted to determine if this is the case with write operations as well.

The Test

I thought a simple test should be sufficient, just write to the database 10,000 times 3 small strings. 

Test 1 Dynamic SQL

For better code reuse and readability it is sometimes superior to use dynamic sql, however, there is often quite a performance penalty. I assumed this would be the slowest method and it did not disappoint. My laptop managed to complete the operation in 2.173 seconds.

Test 2 Compiled SQL

If possible use compiled SQL over the dynamic variety, not only is the syntax highlighting of benefit, the performance gain is substantial. The test ran through in 0.238 seconds, nearly a 10 times performance improvement and the code is just as readable. The only downside of compiled SQL is that occasionally in a production environment I have found that queries can stop performing correctly and you need to clear the cached queries, this is relatively uncommon though.

Test 3 Direct Global Write

Unsurprisingly writing globals was dramatically faster at 0.042 seconds. Writing directly to globals has many issues in terms of code re-use and clarity, additionally it requires a lot of extra code around it to make sure the data matches the requirements for the fields, and its not exactly ACID compliant without substantial work. That being said if the performance is insufficient for the task and hardware upgrade is out of the question it can become necessary to use global access. 

Conclusion

Avoid writing dynamic SQL if you can, compiled SQL will really boost the performance and without any real cost. If you have to move lots of data about really quickly and are happy to deal with the limitations then globals can really help get that application performing well. 


Wednesday, 16 July 2014

When will the S5 get a working CyanogedMod 11?

As of today (17th July) CyanogenMod 11 is only available in a semi-functional version. With both GPS and the camera not working it could not be considered a daily driver, and of course there are plenty of other minor niggles.

Samsung has introduced lots of great features in its latest version of TouchWiz, and I have always liked the swipe left and right on contacts to send a message or initiate a phone call since it was on the original Galaxy S, but this single likeable feature is does not compare to stock Android.

The cleaner look, the instant responsiveness and now with Android L the promise of better battery life we all know that TouchWiz cannot compete.

So far I have turned off most of Samsung's proprietary functionality, and found the finger print scanner and heart rate monitor to be next to useless.

The Galaxy S5 is still the best phone I have ever owned, but a Google Play edition or CyanogenMod would be a massive step up in usability, speed and enjoyment.

Dearth of Intersystems Code Examples

When I first started my current job I was told that the database was an Intersystems database, I had not heard of this before on researching it I found out a number of things.

1. It was ranked number 63 on DB-Engines most popular databases
2. Cache is not a good name for a piece of software if you want to find information in Google
3. There was a distinct lack of examples and people discussing real code

This 3rd issue is perhaps the most important. In programming most of what you want to do has been done by thousands of people before you. Want to parse a CSV file, well if your language does not have a built in function someone will have posted a solution on the internet. Not only will you have a working solution it will probably have been commented on by a number of programmers and improved to a level which would have probably taken you several live iterations to achieve.

The code examples in the documentation are basic, out of context and ignore even simple advise on field filtering or error handling.

Cache is one of those languages which does not appear to offer native CSV handling. Additionally when experimenting with the SQL Import/Export manager code I found that it did not appear to allow more complex CSV files where for example there are carriage return line feeds in the delimited fields. Of course Excel also breaks with these files, but then it is not exactly the best thing for dealing with CSV files...

So rather than complaining I thought I would post an example. Please if you notice any bugs or good ideas for improvement please advise so that I can improve the code.

This is a simple Parsed CSV File object. It takes a file and parses the data and stores the extracted data in a global. It also stores a total row count and a longest row. I found it useful to store the longest row when displaying the results in an HTML table so that I could set up the table with the appropriate width of empty cells.

Class User.CSVParser Extends %Persistent
{

Property Name As %String [ Required ];

Index Name On Name [ Unique ];

Property MaxColLength As %Integer;

Property TotalRows As %Integer;

ClassMethod StripDotsAndCommas(String As %String) As %String
{
 Set String=$Replace(String,",","")
 Set String=$Replace(String,".","")
 Quit String
}

ClassMethod CSVFileToGlobal(FileName, Output Name, FieldsEnclosedBy As %String = """", FieldsEscapedBy As %String = """", FieldsDelimitedBy As %String = ",") As %Status
{
 if '##class(%File).Exists(FileName)
 {
  Quit $SYSTEM.Status.Error(5001,"File "_FileName_" Does not exist")
 }
  
 Set Stream=##class(%FileBinaryStream).%New()
 Set Stream.Filename=FileName
 set lineNo=1
   
 Set ParsedCSV=..%New()
 Set FileName=$Piece(FileName,"/",$Length(FileName,"/"))
 Set FileName=$Piece(FileName,"\",$Length(FileName,"\"))
 Set FileName=..StripDotsAndCommas(FileName)
 Set Name=FileName_..StripDotsAndCommas($ZH)
 
 Set ParsedCSV.Name=Name
 Set Status=ParsedCSV.%Save()
 If $$$ISERR(Status)
 {
  Set i=0
  While $$$ISERR(Status)
  {
   Set i=i+1
   //Try 10 times to generate a unique name
   If i=10
   {
    Quit  
   }
   Set Name=FileName_..StripDotsAndCommas($ZH)
   Set ParsedCSV.Name=Name
   Set Status=ParsedCSV.%Save()
  }
  If $$$ISERR(Status)
  {
   Quit Status 
  }
 }
 Set global="^"_Name
 Set subscript="CSV"
  
 Set colLengthArray=##class(%Library.ArrayOfDataTypes).%New()
 Set maxColNo=0
 Set state=1
 Set Data=""
 Set colNo=1
 Set fullLine=""
While 'Stream.AtEnd { //Do not assume CRFL are end of lines Set line=Stream.Read() if lineLength=0 { Set Data="" } else { //Loop through each character //State 1 - initial state (ready for start of a new field) //State 2 - Inside enclosed string //State 3 - Possibly at end of enclosed string //State 4/5 - Next character should be an escaped character //State 6 - Carriage return for i=1:1:lineLength { set char=$Extract(line,i) If ((char=FieldsEnclosedBy) && (state=1)) { //opening encapsulating character Set state=2 } ElseIf ((char=FieldsEnclosedBy) && (state=2)) { //possible closing characer Set state=3 } ElseIf ((char=FieldsEnclosedBy) && (state=3)) { //escape char / enclosing char are the same Set state=2 Set Data=Data_char } ElseIf ((char=FieldsEscapedBy)) { If state=1 { Set state=4 } ElseIf state=2 { Set state=5 } ElseIf state=4 { Set Data=Data_char Set state=1 } ElseIf state=5 { Set Data=Data_char Set state=2 } } ElseIf ((char=FieldsDelimitedBy) && (state'=2)) { //delimiter Set @global@(lineNo,colNo)=Data Set colNo=colNo+1 Set state=1 Set Data="" } ElseIf ((char=$Char(13)) && (state'=2)) { //Probably start of newline - set new line on LF Set state=6 } ElseIf ((char=$Char(10)) && (state'=2)) { //New line Set @global@(lineNo,colNo)=Data If colNo>maxColNo { Set maxColNo=colNo } Set state=1 Set Data="" Set colNo=1 Set lineNo=lineNo+1 Set fullLine="" } ElseIf ((char'=FieldsDelimitedBy) && (state=3)) { Set Data=Data_FieldsEnclosedBy_char Set state=2 } Else { Set Data=Data_char If state=4 { Set state=1 } ElseIf state=5 { Set state=2 } } } }
 Set ParsedCSV.MaxColLength=maxColNo
 Set ParsedCSV.TotalRows=lineNo-1
 Set Status = ParsedCSV.%Save()
 Quit Status
} }