TotalCross: everything that SuperWaba always wanted to be

TotalCross a new product that is being developed.

Actual status: the vm is fully functional, all libraries are implemented and tested. Tasks to finish for first release: fix the few converter problems, implement native threads (windows ce 90% implemented and tested, palm os 50%), update all documentation. Tasks that will be implemented after the first release: Symbian support, reflection/serialization, PIM. Estimated release date: mid August, 2008.

In this wiki we will present the changes between SuperWaba and TotalCross. For the programmer, at a first glance, TotalCross may be just a repackaging of the Waba and SuperWaba packages. But TotalCross is much more than that: it has a new virtual machine, written from scratch, faster and much more reliable than SuperWaba's. We also tried to fix in TotalCross everything that bugs SuperWaba programmers. For example, the deployment process is incredibly easy.

The packages were changed to match the new product name and also to look closer to Java. For example, waba.ui.MainWindow is now placed at totalcross.ui; the MultiEdit class was moved from superwaba.ext.xplat.ui to totalcross.ui. The Image class is now at totalcross.ui.image (which resembles java.awt.image). Some class names were changed, like Catalog (which is now PDBFile), SerialPort (now PortConnector), PopupMenu and PopList (now MenuBarDropDown and ComboBoxDropDown), RadioGroup (RadioGroupController, to clarify that it is not a Control) and many methods were renamed to improve correctness.

Here is the table of contents of this wiki:

Supported Platforms

These are the platforms that TotalCross runs on:

• iPhone (using jailbrake)
• Blackberry 4.2.1 and up
• Palm OS 5 and up (Zire 21 not supported)
• Windows CE: Pocket PC ARM/SH3/MIPS (3.0, .Net, Windows Mobile 5 and 6), PPC 2.11 ARM, HPC2000 ARM
• Symbian S60, S60V3, S80
• Linux desktop *
• Windows 98 and beyond
• Browser as Java applet

Note: Palm OS 3 and 4 were dropped to allow real multithread.
* May be ported to linux smartphones as demand grows.

New opcodes

SuperWaba interprets Java Bytecodes. TotalCross uses a proprietary set of bytecodes to improve program´s security and performance: TotalCross is about two times faster than SuperWaba. The translation between the java bytecodes to our opcodes is done automatically when the application is deployed.

Regarding security, using SuperWaba is very easy to recover the sources from the application's PDB file. We can extract the .class files from the PDB and then decompile them to the .java files. In TotalCross this is IMPOSSIBLE: there are no decompilers. So, don't forget to take backups of your source files, because it will be impossible to recover them. Don't trust developers, trust only your set of backups!

Virtual Machine features

The TotalCross Virtual Machine (TCVM) was written from scratch. Not even a single line of code was taken from the original Waba. The TCVM has the following features:

• It interprets a proprietary set of opcodes, not Bytecodes.
• It is a register-based VM, not stack-based as Java.
• Its Garbage Collector is 10x faster than SuperWaba's.
• It has support for real multi-threading. Note that the TotalCross API does not supports concurrency, which must be implemented by your own.
• It does not have the 64KB limit for Objects; now an Object (like a String) can have any size. These (lack of) limits also apply to class files.
• We adopted the little endian for storing internal information in the tclass files, since its the most widely used format of actual microprocessors.
• The tclass files are highly optimized to save space. For instance, the constant pool (where Strings, constants and identifiers are stored) is shared among all deployed classes, and each class entry is compressed using zlib.
• The virtual machine is now a shared libray and not an executable like SuperWaba. This fixes some problems in Windows CE (the icon in the last used applications shows the SuperWaba icon instead of the application's one) and in Palm OS (exiting the app sometimes does not return to the caller application), and will also make the creation of native libraries much easier.
• Supports headless applications (like daemon applications, without user interface): just implement the interface totalcross.MainClass and this class will be loaded by the TCVM.
• It has special opcodes for array manipulation. If the converter detects that the array reference is never changed outside the loop and that the loop never gets out-of-bounds, a special opcode will be used to iterate through the array without checking for NullPointerException and ArrayIndexOutOfBoundsException.

And a few drawbacks:

• It does not support the float type, only double. This option was adopted because all actual PDA processors have a math co-processor, and also because the vast majority of SuperWaba applications are not scientific programs. During our research, we found that the float types are 2x faster than double, but the virtual machine overhead is much greater than this small performance difference. The change from float to double will be done by the translator, so there's no need to change your source code.
• TotalCross no longer uses the Simple Directmedia Layer (SDL). The SDL, introduced in SuperWaba version 5.0, was important for us to increase Graphics performance, but introduced many problems with the input devices, such as pen and keyboard inputs in foreign languages like Chinese and Japanese. We created a faster Graphics engine, and will now use the default input routines of the PDAs, resolving these problems. Getting rid of SDL will also shrink the TCVM in about 150Kb.

Deployment process

The deploying process is much easier in TotalCross. There are no more Warp and Exegen programs, now the program tc.Deploy must be used instead. The basic format is: tc.Deploy <what to deploy> <platforms to deploy>. The what to deploy can be:

1. The main window file name (the .class, not the .java); you can pass the full path to the main window file, it does not have to be with the correct package. For example, if the class that extends MainWindow is my.sample.App, located in c:\myprogram\classes\my\sample\App.class, you can use:

> cd c:\myprogram\classes
> tc.Deploy my/sample/App.class

Another option is:

> tc.Deploy c:\myprogram\classes\my\sample\App.class

The correct package is automatically detected from the class file.

2. A jar or zip file with everything that must be packaged, for example: "tc.Deploy myfile.jar" or "tc.Deploy myfile.zip" (a Jar file is basically a zip file with a new extension). This is the only valid option to create libraries (which usually do not contain a MainWindow class). You must name the jar/zip file with the same name of the class that extends MainWindow.

3. The folder that contains a single class that directly extends from MainWindow or TestSuite (or implements MainClass). This option will not work if your main class extends a class which in turn extends MainWindow (ie, MainWindow <- A works, but MainWindow <- B <- A won't, assuming A is the main class).

These commands can be typed in the prompt, placed in a bat file or in an ANT build script. Classes that implement totalcross.MainClass are also considered in the search.

The deployer automatically detects Class.forName references and include them.

The platforms to deploy can be:

• palm or palmos - creates prc+pdb files (if a MainWindows is detected) or pdb file only (otherwise), and also an installer.
• ce or wince - creates the cab files and the bat to invoke ActiveSync
• win32 - creates an exe file to run the application in win32
• linux - creates an application file
• symbian - creates the sis files for S60, S80 and S60v3
• bb or blackberry - creates the cod file
• applet or html - creates the html files and a jar file with all needed classes (including TotalCross dependencies). This allow you to use a single and tiny jar file to visualize your application in the browsers
• all - creates all the above in a single option

So, if you're at the folder where the class files are, you may type: "tc.Deploy . -all" and everything for all platforms will be created.

You may have noted that there's no need to pass the creator id, nor many other parameters, such as the icon's title. All these are inferred from the default constructor of the class that extends MainWindow. The following patterns are used:

Code inferred	Resulting property
super("My application", border_type)	"My application" is used as icon's title
super("My application "+appVersion, border_type)	"My application" is used as icon's title. (this example would show, during program execution, something like: "My application 6.0")
Settings.applicationId = "Crtr";	"Crtr" will be the creator id (which in TotalCross is renamed to application id, which is the correct name that Palm should have used).
Settings.appVersion = "1.3"	"1.3" will be used to set the version of the application in the stubs.
Settings.symbianUID = 0x12435705;	0x12435705 will be used as the Symbian UID when creating Symbian installation files (SIS).
Settings.companyInfo = "This is my company information";	"This is my company information" will be used as the company's information when creating SIS files. You may also use this in an About box in your program.

If you don't specify these in the application's MainWindow constructor (or declare them static somewhere else), default values will be assumed. If you want to specify the icon's title as one thing and your application's title as another thing, use super("My icon title", border_type); setTitle("My app title");.

There are other options that are set by the programmer in the MainWindow's constructor and read directly by the TCVM at runtime:

• Settings.minimizeOnClose: for WinCE only, uses a x button to minimize the application, instead of the ok button that closes it.
• Settings.fullScreen: makes it full screen at startup

The use of the information stored in the program instead of passing them in the command line is important to avoid many (and many) problems that occurred at Warp and Exegen when the parameters used by the program was not the ones passes to them.

The tc.Deploy program will, starting in the working directory, create folders with the installation files for each platform. A .tcz file is outputted with all classes and its dependencies, already translated to the target opcodes, and compressed with ZLib to save space. Bmp and Gif files are automatically converted to 24bpp Png files (JPeg are kept in its original format - read more about images below). The tcz file does not complain with the .zip format, although ZLib is used, because we need to store more information in the tcz file. Note that, since we translate the programs to our opcodes, there's no need to run obfuscators in the class file, because obfuscation is already done by the converter (if requested). Actually, tcz files are 1/4 of the original .class size, 50% due to our tclass file format and 50% due to the zlib compression.

Now, instead of providing all the 11 icon formats with the web safe palette, you may optionally provide a single appicon.gif image, with any palette and any size. This image will be resized to the target size and dithered to the web safe palette. This greatly simplifies the application's icon creation. We removed the non-8bpp images, so now there are only 7 images with the various sizes used by the platforms.

Now is possible to create installation files of all platforms for library-only archives (without an executable).

SuperWaba to TotalCross conversion

How to proceed in the convertion:

1. In order to make the conversion easier, we created a program, tc.tools.SW2TC, which reads recursively a set of folders and change the names of the old SuperWaba packages, classes and methods, to the new ones. This program does not create a backup, so, you must do a backup before running the program, since the original files are overwritten. The program takes care about 99% of the problems, remaining only 1% for you to manually change them.
2. SW2TC can't change some method names, so, after running it, if an error occors in a method, check the javadoc to see the new name of this method.
3. Some methods now return void instead of boolean.
4. Many methods now throw a checked Exception, which now must be correctly handled. In SuperWaba, there were almost no check of invalid parameters being passed to the methods; in TotalCross, all parameters are checked, and exceptions are thrown accordingly.
5. Some packages and classes were removed, like the DateRGF package and the IntVector(DataStream) method. If you use them, grab their code from SuperWaba, migrate them to TotalCross and add them directly to your project.
6. Color is now an int, in format 0xRRGGBB. Change all Color to int. Instead of comparing or initializing the color to null, use -1.
7. The PIMAL api will be stripped from version 1.0, and will be completely revised, fixed, and placed back in a future version.
8. The isOpen method was removed from all streams. Now they all throw an IOException (or FileNotFoundException) if an open error occurs. We had some meetings to decide about this change, but we implemented it to make TotalCross programs more robust (in SuperWaba, users often forgot to check if a file was really open).
9. The onStart method was renamed to initUI, and SW2TC does the rename for you. But, if for some reason a screen appears blank, you should check if the method is still called onStart. onAdd was also renamed to onAddAgain.

In this table we summarize the package, class and method changes.

The Launcher

To launch a program at desktop with SuperWaba you use the waba.applet.Applet applet/application. In TotalCross, you will use totalcross.Launcher program instead. This Launcher accepts the following parameters:

Possible Arguments (in any order and case insensitive). Default is marked as *
   /scr WIDTHxHEIGHT     : sets the width and height
   /scr WIDTHxHEIGHTxBPP : sets the width, height and bits per pixel
*  /scr PalmLo     : Palm OS low     (same of /scr 160x160x8)
   /scr PalmHI     : Palm OS high    (same of /scr 320x320x16)
   /scr PalmTall   : Palm OS tall    (same of /scr 320x480x16)
   /scr PalmWide   : Palm OS wide    (same of /scr 480x320x16)
   /scr S60        : Symbian S60     (same of /scr 176x189x16)
   /scr S80        : Symbian S80     (same of /scr 640x200x16)
   /scr WinCE      : Windows CE      (same of /scr 240x320x16)
   /scr bbLo       : BlackBerry low  (same of /scr 240x160x16)
   /scr bbHi       : BlackBerry high (same of /scr 320x240x16)
   /scr iPhone     : iPhone (same of /scr 320x480x24)
   /pos x,y        : Sets the openning position of the application
*  /uiStyle WinCE  : Windows CE user interface style
   /uiStyle PalmOS : Palm OS user interface style
   /uiStyle Flat   : Flat user interface style
   /uiStyle Vista  : Vista user interface style
   /penlessDevice  : acts as a device that has no touch screen.
   /keypadOnly     : acts as a device that has only the 0-9*# keys
   /bpp 4          : emulates 4  bits per pixel screens (16    colors)
   /bpp 8          : emulates 8  bits per pixel screens (256   colors)
   /bpp 16         : emulates 16 bits per pixel screens (64K colors)
   /bpp 24         : emulates 24 bits per pixel screens (16M colors)
   /scale <1 to 4> : scales the screen, magnifying the contents
   /dataPath <path>: sets where the PDB and media files are stored
   /cmdLine <...>  : the rest of arguments-1 are passed as the command line
The class name that extends MainWindow or MainClass must always be the last argument

Now the Launcher correctly emulates 4, 8, 16 and 24 bpp, changing to the colors that are really used at the device. Due to the changes in the Graphics class and also in the Launcher class, the screen refresh at desktop is 2 times faster when comparing with SuperWaba.

User Interface changes

There were lots of important changes in the user interface, which are listed below:

• Button: now supports text with images. Text can be placed at LEFT, RIGHT, TOP or BOTTOM (the image is placed at the opposite position).
• Edit: now supports mask for DATE and CURRENCY modes. To use the mask, you can use something like:

      Edit ed;
      ed = new Edit("999.999.999,99"); 
      ed.setMode(Edit.CURRENCY,true); 
      add(ed,LEFT,CENTER);
      
      ed = new Edit(); 
      ed.setMode(Edit.DATE,true); 
      add(ed,LEFT,AFTER+2);

The mask of the DATE will use the format defined in the Settings class, and getText will return the displayed text (E.G.: 27/08/2007), while getTextWithoutMask will return only the numbers (27082007). For the CURRENCY mode, getText() will return the masked text (E.G.: 1.234,99) and getTextWithoutMask will return the number with . in the place of , (if , is the default decimal separator), like 1234.99. In CURRENCY, numbers are aligned at RIGHT during edition.

• The Graphics class now supports real clipping, so creating a Graphics for an Edit that is placed beyond its parent container, drawing on it will not overwrite the other container's area. Due to this, we added...
• ScrollContainer: a Container that automatically adds vertical and horizontal scrollbars if the components are placed beyond its limits. CAUTION: in the hands of badly intentioned programmers, this can lead to very poor user interfaces. Use it with care. Note that you cannot use RIGHT/BOTTOM and FIT placements when adding controls to a ScrollableContainer: the controls must be always added left to right, top to bottom, in sequence.
• Added method add(ctrl, x,y,w,h) and its variants to the Container class. This allow you to use: add(ed = new Edit(), LEFT,TOP,FILL,PREFERRED).
• Added support for screen rotation and collapsible input area. If you implement the user interface using ONLY relative coordinates, the user interface will reposition itself automatically when the screen resolution changes. Screen rotation can now be tested in the Launcher using the F9 key. Note that one cannot use xxx.setRect(getClientRect()), which must be replaced by the equivalent xxx.setRect(LEFT,TOP,FILL,FILL), resulting in the same placement but now supporting the automatic repositioning.
• There are new classes, like Whiteboard, used to input signatures and drawings; ImageControl, to show an image as background, and it also supports panning if the image is bigger than the control's bounds; and most classes from superwaba.ext.xplat.ui are now at the totalcross.ui package.
• MultiEdit has been improved in speed. It also supports justifying, by making justify=true.
• The Window class no longer support saving the background: now when a Window is moved, all the screens below them are repainted. This occurs very fast and is not noticiable by the user, because now there's only one offscreen Image representing the screen, which is updated when Control.updateScreen is called. This approach will also avoid spending a lot of memory to store the offscreen and backgrounds.
• Settings.setUIStyle was moved to MainWindow.setUIStyle.
• Method repaint just sets a flag in the parent window telling it to be repainted. We adopted this solution because in penless devices, after a focus change, all the screen was repainted, so keeping track of the invalidated area is just a waste of time. On the other hand, repaintNow repaints only the changed area.
• At each component repaint, a new Graphics object is created, so changing its properties will cause no harm to the other components. Graphics objects are no longer cached.
• Label now uses \n to split lines instead of |
• MessageBox now splits the text automatically if it finds that its bigger than the screen width.
• Added Column, Scatter, Line chart classes.
• Migrated Launcher to use the JDK 1.1 dispatch/listener event model. This fixes accentuation support when running as application.
• New Slider class: just like the ScrollBar, but without the edge buttons and with a nicer thumb.
• Improved the Whiteboard accuracy, bypassing all the other event stuff and redirecting the Pen events directly to it, using the new setGrabPenEvents method in the Window class.
• Now repaint is a static method of the Control class: all it does is set a flag that invalidates all windows to be repainted.
• onAdd was renamed to onAddAgain. To let onAddAgain and onRemove be called, you must explicitly set set callOnAddAgain and callOnRemove to true. We made this to improve startup speed and also because these two methods (specially the last one) are almost never used.
• onStart was renamed to initUI to enforce that its the place where user interface must be initialized.
• Label now supports justified text; pass FILL as the alignment. Note that it only makes sense for Labels with more than one line.
• Label.commonVGap is no longer visible. Now Label has the same preferrred height of Edit and Button in all ui styles.

Dispatch/Listener event model

Although not recommended, we added the dispatch/listener event model to TotalCross. It can be useful in some situations, but be aware that using onEvent makes your code smaller and faster. The added listener interfaces are:

• PenListener: penDown, penUp and penDrag
• KeyListener: actionkeyPressed, devicekeyPressed, keyPressed
• WindowListener: windowClosed
• FocusListener: focusIn and focusOut
• PressListener: controlPressed
• TimerListener: timerTriggered
• GridListener: gridSelected, gridTextChanged and gridCheckChanged

The dispatch/listener is called before onEvent, so setting the event's consumed flag will break the event chain.

Graphics, Palette and Color class changes

TotalCross has a Graphics engine written from scratch (like almost everything on it). Some important performance-tailored decisions were taken. No matter which bpp is used at the device, we now store the screen and images in a 24 bpp RGB array. All drawings are made into a single offscreen, which is then converted on the fly to the target screen bpp when the updateScreen method is called. Graphics.drawText now handles the tab character.

All Colors were changed to int, which represents a color in the 0xRRGGBB format. The Color class is now only a utility class for manipulating ints, with all methods static and receiving an rgb int as parameter. Removed setBackColor and setForeColor from the Graphics class: now you set them directly acessing the public foreColor and backColor members. A null color is represented with a -1 value.

The Palette class was removed, because now there are no palettes, since all images are true 24 bpp and only the offscreen is converted to the target screen bpp. If the screen is 8bpp, a variant of the 685 palette (6 bits for red, 8 for green and 5 for blue, plus 16 shades of gray) is used.

Font changes

In SuperWaba, Palm OS 160x160 used one font (the internal device's font), Palm OS 320x320 used another font (which was exactly 2 times bigger in width and height than the 160x160 one), Windows CE used the Arial 11 font, and Symbian the Arial 10 font.

Now all platforms will use the Tahoma font. A modified version of the TTF2PDBs (now called tc.tools.FontGenerator) program create a set of bitmap fonts ranging from size 6 to 22, normal and bold, which can be used by the programs. Each font is compressed to save space (the Tahoma takes only 160kb). Then, the following sizes are used as the NORMAL_SIZE depending on the screen resolution at runtime: 9 (for Palm OS 160x160), 11 (Symbian 176x189), 12 (Windows CE 240x320), 18 (Palm OS 320x320 or 320x480). The program can easily select another size and, by using a relative size like Font.getFont(false, Font.NORMAL_SIZE+2), the font will be correctly scaled among the different resolutions.

The Tahoma was chosen because it has the same height of the original Palm OS fonts, but it is a bit shorter, so, you get more space on screen. Its appearance at the device was also better than Arial. Note that, due to this font change, your screens may move a bit, so a general review of all screens of your application is recommended.

Another important change is the support for antialiased fonts (thanks to Pierre G. Richard, from Jaxo), which makes the font appear rounded and improves legibility. See a sample here (take a close look to the W letter in the left combobox).

FontMetrics.leading was removed. Unicode is still supported, but the unicode fonts must be generated by the user using the FontGenerator utility.

Image changes

When deploying, the Gif and Bmp images are automatically converted to 24 bpp PNG graphics format. The Image.createBmp method was changed to Image.createPng, which saves a compressed 24 bpp image instead of a device-specific palletized bmp as in SuperWaba.

The Image class now loads Png, Gif, Bmp (including 24 bpp) and JPeg at desktop, but only Png / JPeg at the device.

There are two new methods: getSmoothScaledInstance and smoothScaleBy, which uses the area averaging algorithm instead of the replication one used on the getScaledInstance/scaleBy methods. This allow a very important change: you can now create only one 320x320 image, and then smoothly resize it to the target resolution, instead of providing one image per resolution. This can greatly decrease the file size. One important thing is that these methods receive as parameter the background color where the buttons will be placed, so if the images have antialiasing on their borders, the conversion creates a really smooth image.

Here are parts of an example of how to create buttons for the several resolutions based on a 320x320 one:

       int original = 320, targets[] = {320, 240, 176, 160};
       double factor = (double)targets[i] / (double)original;
       img2 = img.smoothScaledBy(factor,factor, getBackColor());
       img2.transparentPixel = img.transparentPixel;
       Button btn = new Button(img2);
       btn.setBorder(Button.BORDER_NONE);
       add(btn,xx,i==0 ? TOP+5 : AFTER+5);

... which outputs these images.

Pouporri

• Now totalcross.util.zip.ZLib uses Java's Deflater/Inflater native libraries at desktop, and the standard parameters at device, thus, making it compatible with the other ZLibs. This improves performance at desktop.
• Reordered some event values. If you don't use ControlEvent.xxx, but use the values directly, you may get into troubles.
• It is not possible to implement finalizer with the new garbage collector, because there's no pass to unmark unreachable objects. To bypass this problem, we added the totalcross.sys.HasFinalizer interface, and the methods Vm.addFinalizer and Vm.removeFinalizer, which are called when the application QUITS. Here's a sumary of how it works, using as example the Socket class: when a socket is created, it registers itself with the Vm.addFinalizer. If the socket is closed, it removes itself using Vm.removeFinalizer; otherwise, if the vm quits before the user closes the socket, its finalizer (note that the name is finalizer, NOT finalize) method is invoked for that object, which then calls the socket.close method. Note that since the object is stored in a hashtable by the method addFinalizer, it will never be collected by the gc.
• Vm.alert(message) will display the message text in a modal window. Under JDK, its a modal frame outside the window's area; in all the other platforms, its a message box. Useful for debugging the application at the device. Note that the area to show the text can be very limited in some platforms.
• New class NetConnector: allows the creation of all network profiles and selection of one of them to use in the current running program.
• New class SocketSever: allows the vm to listen to a connection from a remote machine. Instead of keep polling the server, just sit and wait. Note that this depends on the cell phone's company to allow such connections.
• In SuperWaba, to check if a file exists, you could do a code like this:

         File f = new File(DeploySettings.currentDir + "/" + name, File.READ_ONLY); // or File.READ_WRITE
         return f.isOpen();

In TotalCross, this will FAIL. To make it work, use the other File constructor, which uses the File.DONT_OPEN mode, and also check if it exists:

         return new File(DeploySettings.currentDir + "/" + name).exists();

---

This wiki will be often updated. Stay tuned!