Using TypeScript LanguageService to build an JS/TypeScript IDE in Java

Yesterday I blogged about my endeavors into loading and using the TypeScript Language Service (V8 and Nashorn) and calling it from Java to get things like an outline, auto-completions, … .

Today I connected these headless pieces to my JavaFX-Editor-Framework. The result can be seen in the video below.

To get the TypeScript LanguageService feeling responsible not only for TypeScript files but also for JavaScript I used the 1.8 beta.

As you notice JS-Support is not yet really at a stage where it can replace eg Tern but I guess things are going to improve in future.

JavaScript Performance V8 vs Nashorn (for Typescript Language Service)

On the weekend I’ve worked on my API to interface with the Typescript language service from my Java code.

While the initial version I developed some months ago used the “tsserver” to communicate with the LanguageService I decided to rewrite that and to interface with the service directly (in memory or through an extra process).

For the in memory version I implemented 2 possible ways to load the JavaScript sources and call them

  • Nashorn
  • V8(with the help of j2v8)

I expected that Nashorn is slower than V8 already but after having implemented a small (none scientific) performance sample the numbers show that Nashorn is between 2 and 4 times slower than V8 (there’s only one call faster in Nashorn).

The sample code looks like this:

public static void main(String[] args) {
  try {
    System.err.println("V8");
    System.err.println("============");
    executeTests(timeit("Boostrap", () -> new V8Dispatcher()));
    System.err.println();
    System.err.println("Nashorn");
    System.err.println("============");
    executeTests(timeit("Nashorn", () -> new NashornDispatcher()));
  } catch (Throwable e) {
    e.printStackTrace();
  }
}

private static void executeTests(Dispatcher dispatcher) throws Exception {
  timeit("Project", () -> dispatcher.sendSingleValueRequest(
    "LanguageService", "createProject", String.class, "MyProject").get());

  timeit("File", () -> dispatcher.sendSingleValueRequest(
    "LanguageService", "addFile", String.class, "p_0", DispatcherPerformance.class.getResource("sample.ts")).get());

  timeit("File", () -> dispatcher.sendSingleValueRequest(
    "LanguageService", "addFile", String.class, "p_0", DispatcherPerformance.class.getResource("sample2.ts")).get());

  timeit("Outline", () -> dispatcher.sendMultiValueRequest(
    "LanguageService", "getNavigationBarItems", NavigationBarItemPojo.class, "p_0", "f_0").get());

  timeit("Outline", () -> dispatcher.sendMultiValueRequest(
    "LanguageService", "getNavigationBarItems", NavigationBarItemPojo.class, "p_0", "f_1").get());
}

Provides the following numbers:

V8
============
Boostrap : 386
Project : 72
File : 1
File : 0
Outline : 40
Outline : 10

Nashorn
============
Nashorn : 4061
Project : 45
File : 29
File : 2
Outline : 824
Outline : 39

The important numbers to compare are:

  • Bootstrap: ~400ms vs ~4000ms
  • 2nd Outline: ~10ms vs ~40ms

So performance indicates that the service should go with j2v8 but requiring that as hard dependency has the following disadvantages:

  • you need to ship different native binaries for each OS you want to run on
  • you need to ship v8 which might/or might not be a problem

So the strategy internally is that if j2v8 is available we’ll use v8, if not we fallback to the slower nashorn, a strategy I would recommend probably for your own projects as well.

If there are any Nashorn experts around who feel free to help me fix my implementation

Why I think @UIEventTopic / @EventTopic have been a bad idea

I’ve been working on a document since some time who holds my thoughts on the future direction of Eclipse 4 or better the Eclipse 4 Application Platform.

At first I’d like to mention that these as my personal views, not aligned with the e4 committers, so you’ll most likely find others who disagree.

It’s not my intention put e4 or any e4 committer down and if I’d blame someone it would be myself because I was part of the e4 effort since day 1.

The document is not yet ready but inspired by a tweet from Marcel Bruch

I’d like to take one of the points I’ll mention in this big vision/reflection document and explain why I think in retrospect providing those 2 annotations is wrong.

OSGi-Leakage

The first problem I have with those annotations is not an architectural one but caused by the bundle they are currently shipped in. We currently find them in “org.eclipse.e4.ui.di” and “org.eclipse.e4.core.di.extensions” who themselves require things like the Eclipse-DI-container, Equinox-container.

In my world of e4 components this is a deal breaker, as I want my code have no dependency on any of those bundles at compile time nor runtime but in general I think this is a minor problem and fixable. The architectural is the real blocker.

Event data is different to any other DI-Information

The real problem I have with @UIEventTopic and @EventTopic is the information they carry with them is totally different to any other data available in the DI-System (starting from OSGi-Services to Preference or IContextFunction derived values) because it’s only temporary. I think this characteristic makes @UIEventTopic/@EventTopic data not suited for DI.

The guess I have for those annotations being so popular is that they free you from the useage of IEventBroker (who leaks OSGi at an even larger scale by showing OSGi-Classes in its API, once more for no good reason) and the verbose code you need to write to subscribe (wild guess once more nobody was really worried about the OSGi-Class-Leakage).

@PostConstruct
void init(IEventBroker b) {
   b.subscribe( "my/event/Topic", new org.osgi.service.event.EventHandler() {
     public void handle(org.osgi.service.event.Event event) {
       handleEvent( (String)event.getProperty(IEventBroker.DATA) );
     }
   } );
}

private void handleEvent(String data) {
   // ...
}

In a Java8 world the verbosity can be reduced with the help of lamdas and method refs


import static my.sample.Util.extractEventData;

@PostConstruct
void init(IEventBroker b) {
  b.subscribe( "my/event/Topic", extractEventData(this::handleEvent));
}

private void handleEvent(String data) {
   // ...
}

// Util.java
public static <T> Consumer<Event> extractEventData( Consumer<T> dataConsumer) {
   return e -> (T)e.getProperty(IEventBroker.DATA);
}

The extractEventData could be provided by IEventBroker in a Java8 world BTW.

You loose typesafety for no good reason

IEventBroker and @UIEventTopic/@EventTopic trade IMHO typesafety for no good reason.

One can argue that this is the case for DI in general as well but you trade in this case typesafety against loose coupeling freeing your business code from dependencies on large framework (and yes I consider e4 and OSGi large frameworks) and providing you better testing support – so you get something in return.

This is not true for IEventBroker nor is it true for @UIEventTopic/@EventTopic, while IEventBroker could be fixed to provide a certain amount of typesafety (see e(fx)clipse EventBus) this will never be possible for @UIEventTopic/@EventTopic!

One could argue that on the receiver side you’ll have less framework bindings (in case one splits out the annotations from their existing owner bundle) but because the publisher requires the event bus anyways I don’t see any reason the receiver should not and the EventBus-Service is easy enough to be mocked in Unit-Tests.

And because it is not unlikely that many events are sent through the event system it puts pressure on the DI system which could have been avoid if you used the EventBus directly.

Major improvements to @Preference in e(fx)clipse 2.2.0

With 2.1.0 we introduced our custom @Preference annotation who is to be preferred over the one available as part of the Eclipse 4 Application Platform.

With the next nightly build there are even more reasons to use our annotation because you get even cooler features:

Support for default values

If there’s no value found in the preference store you most likely don’t want to retrieve a default value null (String), false (boolean), 0 (int & long & float & double) but a custom one. You can now define the default value in the annotation

class Component {
@Inject
@Preference(key="velocity",defaultValue="1000")
private int velocity;
}

Support for more value types

It is fairly likely that you want not only store/retrieve standard values like int, long, float, double and String but eg date or time.

class Component {
@Inject
@Preference(key="lastModified")
private java.time.Instant lastModified;
}

The support for “value types” is provided by a new service named ValueSerializer who supports the most common value types (eg values in java.time) and if you have your own types you contribute your own serializers.

Complex objects

While the above introduced support for “value types” works for things like date and time it is not unlikely that you have to remember non trivial objects like configurations the new support for complex values might get you excited

@XmlRootElement
public class Complex {
private String name;
private int value;
private List<Complex> children;

// ....
}

class Component {
@Inject
@Preference(key="complexData")
private Complex complexData;
}

Serialization and Deserialization is provided through the ObjectSerializer service

Developing a source code editor in JavaFX (on the Eclipse 4 Application Platform)

You’ve chosen e4 and JavaFX as the technologies to implement your cool rich client application. Congrats!

In the first iteration you’ve implemented all the form UIs you need in your application which you made flashy with the help of CSS, animations for perspective switches, lightweight dialogs as introduced in e(fx)clipse 2.0, … .

In the 2nd iteration your task now might be to implement some scripting support but for that you require an editor who at least supports lexical syntax highlighting. So you now have multiple choices:

  • Use a WebView and use an editor written in JavaScript like Orion
  • Use the StyledTextArea shipped with e(fx)clipse 2.0 and implement all the hard stuff like paritioning, tokenizing, …
  • Get e(fx)clipse 2.1 and let the IDE generate the editor for you

If you decided to go with the last option the following explains how you get that going by developing a e4 JavaFX application

application

like shown in this video

Get e(fx)clipse 2.1.0

As of this writing e(fx)clipse 2.1 has not been released so you need to grab the nightly builds eg by simply downloading our All-in-One build.

Set up a target platform

We have a self-contained target platform feature (org.eclipse.fx.code.target.feature) available from our runtime-p2 repository to get started super easy.

target_1

target_2

target_3

target_4

Warning: Make sure you uncheck “Include required software” because the target won’t resolve if you have that checked!

target_5

target_6

Setup the project

The project setup is done like you are used to for all e4 on JavaFX applications.

project_1

project_2

project_3

The wizard should have created:

  • at.bestsolution.sample.code.app: The main application module
  • at.bestsolution.sample.code.app.feature: The feature making up the main application module
  • at.bestsolution.sample.code.app.product: The product definition require for exporting
  • at.bestsolution.sample.code.app.releng: The release engineering project driving the build

Now we need to add some dependencies to your MANIFEST.MF:

  • org.eclipse.fx.core: Some Core APIs
  • org.eclipse.fx.code.editor: Core (=UI Toolkit independent) APIs for code editors
  • org.eclipse.fx.code.editor.fx: JavaFX dependent APIs for code editors
  • org.eclipse.text: Core APIs for text parsing, …
  • org.eclipse.fx.text: Core APIs for text parsing, highlighting, …
  • org.eclipse.fx.text.ui: JavaFX APIs for text parsing, highlighting, …
  • org.eclipse.fx.ui.controls: Additional controls for eg a File-System-Viewer
  • org.eclipse.osgi.services: OSGi-Service APIs we make use of when generateing DS-Services
  • org.eclipse.fx.core.di: Dependency Inject addons
  • org.eclipse.fx.code.editor.e4: code editor integration to e4
  • org.eclipse.fx.code.editor.fx.e4: JavaFX code editor integration to e4

For export reasons also add all those bundles to the feature.xml in at.bestsolution.sample.code.app.feature.

Generate editor infrastructure

Having everything configured now appropriately we start developing:

  • Create package at.bestsolution.sample.code.app.editor
  • Create a file named dart.ldef and copy the following content into it
    package at.bestsolution.sample.code
    
    dart {
    	partitioning {
    		partition __dftl_partition_content_type
    		partition __dart_singlelinedoc_comment
    		partition __dart_multilinedoc_comment
    		partition __dart_singleline_comment
    		partition __dart_multiline_comment
    		partition __dart_string
    		rule {
    			single_line __dart_string "'" => "'"
    			single_line __dart_string '"' => '"'
    			single_line __dart_singlelinedoc_comment '///' => ''
          		single_line __dart_singleline_comment '//' => ''
          		multi_line __dart_multilinedoc_comment '/**' => '*/'
          		multi_line  __dart_multiline_comment '/*' => '*/'
    		}
    	}
    	lexical_highlighting {
    		rule __dftl_partition_content_type whitespace javawhitespace {
    			default dart_default
    			dart_operator {
    				character [ ';', '.', '=', '/', '\\', '+', '-', '*', '<', '>', ':', '?', '!', ',', '|', '&', '^', '%', '~' ]
    			}
    			dart_bracket {
    				character [ '(', ')', '{', '}', '[', ']' ]
    			}
    			dart_keyword {
    				keywords [ 	  "break", "case", "catch", "class", "const", "continue", "default"
    							, "do", "else", "enum", "extends", "false", "final", "finally", "for"
    							,  "if", "in", "is", "new", "null", "rethrow", "return", "super"
    							, "switch", "this", "throw", "true", "try", "var", "void", "while"
    							, "with"  ]
    			}
    			dart_keyword_1 {
    				keywords [ 	  "abstract", "as", "assert", "deferred"
    							, "dynamic", "export", "external", "factory", "get"
    							, "implements", "import", "library", "operator", "part", "set", "static"
    							, "typedef" ]
    			}
    			dart_keyword_2 {
    				keywords [ "async", "async*", "await", "sync*", "yield", "yield*" ]
    			}
    			dart_builtin_types {
    				keywords [ "num", "String", "bool", "int", "double", "List", "Map" ]
    			}
    		}
    		rule __dart_singlelinedoc_comment {
    			default dart_doc
    			dart_doc_reference {
    				single_line "[" => "]"
    			}
    		}
    		rule __dart_multilinedoc_comment {
    			default dart_doc
    			dart_doc_reference {
    				single_line "[" => "]"
    			}
    		}
    		rule __dart_singleline_comment {
    			default dart_single_line_comment
    		}
    		rule __dart_multiline_comment {
    			default dart_multi_line_comment
    		}
    		rule __dart_string {
    			default dart_string
    			dart_string_inter {
    				single_line "${" => "}"
    				//TODO We need a $ => IDENTIFIER_CHAR rule
    			}
    		}
    	}
    	integration {
    		javafx {
    			java "at.bestsolution.sample.code.app.editor.generated"
    			e4 "at.bestsolution.sample.code.app.editor.generated"
    		}
    	}
    
    }
    
  • Xtext will prompt to add the Xtext nature to your project
    ldef_2. Choose “YES”
  • The sources are generated into the src-gen folder you should add that one to your build path
    add-source
  • It’s important to note that beside the files generated by the ldef-Language there are 2 files generated to your OSGi-INF-Folder by DS-Tooling from ca.ecliptical.pde.ds

I won’t explain the details of the dart.ldef-File because there’s already a blog post with a detailed description of the file.

The only part that is new is the integration section:

integration {
	javafx {
		java "at.bestsolution.sample.code.app.editor.generated"
		e4 "at.bestsolution.sample.code.app.editor.generated"
	}
}

who configures the code generator to:

  • Generate Java code for the partitioning and tokenizing
  • Generate e4 registration informations in terms of OSGi-Services

In contrast to the last blog where we’ve run our stuff in an NONE-OSGi/NONE-e4-world where we had to wire stuff ourselves this is not needed this time because the Eclipse DI container will take care of that!

Define a Filesystem-Viewer-Part

To browse the filesystem we need a viewer which might look like this:

package at.bestsolution.sample.code.app;

import java.net.URI;
import java.nio.file.Path;
import java.nio.file.Paths;

import javax.annotation.PostConstruct;
import javax.inject.Inject;
import javax.inject.Named;

import org.eclipse.e4.core.di.annotations.Optional;
import org.eclipse.e4.ui.di.PersistState;
import org.eclipse.fx.code.editor.services.TextEditorOpener;
import org.eclipse.fx.core.Memento;
import org.eclipse.fx.ui.controls.filesystem.FileItem;
import org.eclipse.fx.ui.controls.filesystem.ResourceEvent;
import org.eclipse.fx.ui.controls.filesystem.ResourceItem;
import org.eclipse.fx.ui.controls.filesystem.ResourceTreeView;

import javafx.collections.FXCollections;
import javafx.scene.layout.BorderPane;

public class ResourceViewerPart {
	@Inject
	TextEditorOpener opener;

	private Path rootDirectory;

	private ResourceTreeView viewer;

	@PostConstruct
	void init(BorderPane parent, Memento memento) {
		viewer = new ResourceTreeView();

		if( rootDirectory == null ) {
			String dir = memento.get("root-dir", null);
			if( dir != null ) {
				rootDirectory = Paths.get(URI.create(dir));
			}
		}

		if( rootDirectory != null ) {
			viewer.setRootDirectories(FXCollections.observableArrayList(ResourceItem.createObservedPath(rootDirectory)));
		}
		viewer.addEventHandler(ResourceEvent.openResourceEvent(), this::handleOpenResource);
		parent.setCenter(viewer);
	}

	@Inject
	@Optional
	public void setRootDirectory(@Named("rootDirectory") Path rootDirectory) {
		this.rootDirectory = rootDirectory;
		if( viewer != null ) {
			viewer.setRootDirectories(FXCollections.observableArrayList(ResourceItem.createObservedPath(rootDirectory)));
		}
	}

	private void handleOpenResource(ResourceEvent<ResourceItem> e) {
		e.getResourceItems()
			.stream()
			.filter( r -> r instanceof FileItem)
			.map( r -> (FileItem)r)
			.filter( r -> r.getName().endsWith(".dart"))
			.forEach(this::handle);
	}

	private void handle(FileItem item) {
		opener.openEditor(item.getUri());
	}

	@PersistState
	public void rememberState(Memento memento) {
		if( rootDirectory != null ) {
			memento.put("root-dir", rootDirectory.toFile().toURI().toString());
		}
	}
}

Define the application

e4 applications as you already know are not defined by code but with the help of the e4 application model which is stored by default in e4xmi-Files. The final model has to looks like this:

model

The important parts are:

  • DirtyStateTrackingAddon: Is a special addon who tracks the dirty state of the editor and should be added to the applications Addon section
  • Handler: We using a framework handler org.eclipse.fx.code.editor.e4.handlers.SaveFile
  • Window-Variables: We have 2 special variables defined at the window level (activeInput, rootDirectory)
    window-variables
  • PartStack-Tags: We tagged the Part Stack who is hosting the editors with editorContainer
    stack-tags
  • Resource Viewer Part: We register the resource viewer implementation from above to the part definition
  • Root Directory Handler: To set the root directory we have a handler who looks like this
    package at.bestsolution.sample.code.app.handler;
    
    import java.io.File;
    import java.nio.file.Path;
    import java.nio.file.Paths;
    
    import org.eclipse.e4.core.di.annotations.Execute;
    import org.eclipse.fx.core.di.ContextValue;
    
    import javafx.beans.property.Property;
    import javafx.stage.DirectoryChooser;
    import javafx.stage.Stage;
    
    public class SetRootDirectory {
    
    	@Execute
    	public void setRootDirectory(@ContextValue("rootDirectory") Property<Path> rootDirectory, Stage stage) {
    		DirectoryChooser chooser = new DirectoryChooser();
    		File directory = chooser.showDialog(stage);
    		if( directory != null ) {
    			rootDirectory.setValue(Paths.get(directory.getAbsolutePath()));
    		}
    	}
    }
    

Add the highlightings

The final step before we launch the application is that you need to set the styles inside your default.css

.styled-text-area .dart.dart_default {
    -styled-text-color: rgb(0, 0, 0);
}
 
.styled-text-area .dart.dart_operator {
    -styled-text-color: rgb(0, 0, 0);
}
 
.styled-text-area .dart.dart_bracket {
    -styled-text-color: rgb(0, 0, 0);
}
 
.styled-text-area .dart.dart_keyword {
    -styled-text-color: rgb(127, 0, 85);
    -fx-font-weight: bold;
}
 
.styled-text-area .dart.dart_keyword_1 {
    -styled-text-color: rgb(127, 0, 85);
    -fx-font-weight: bold;
}
 
.styled-text-area .dart.dart_keyword_2 {
    -styled-text-color: rgb(127, 0, 85);
    -fx-font-weight: bold;
}
 
.styled-text-area .dart.dart_single_line_comment {
    -styled-text-color: rgb(63, 127, 95);
}
 
.styled-text-area .dart.dart_multi_line_comment {
    -styled-text-color: rgb(63, 127, 95);
}
 
.styled-text-area .dart.dart_string {
    -styled-text-color: rgb(42, 0, 255);
}
 
.styled-text-area .dart.dart_string_inter {
    -styled-text-color: rgb(42, 0, 255);
    -fx-font-weight: bold;
}
 
.styled-text-area .dart.dart_builtin_types {
    -styled-text-color: #74a567;
    -fx-font-weight: bold;
}
 
.styled-text-area .dart.dart_doc {
    -styled-text-color: rgb(63, 95, 191);
}
 
.styled-text-area .dart.dart_doc_reference {
    -styled-text-color: rgb(63, 95, 191);
    -fx-font-weight: bold;
}

Finish

Now you can launch the application with the already generated Launch-Config. Afterwards use the Menu-Entry (“Select root folder …”) to set a root directory who contains dart-Files and double click on one of the files.

Code editors in general with JavaFX

Yesterday I did a short demo on how one can build code editors with JavaFX while the talk was at Xtextcon most information apply to any code editor you may want to develop with the runtime components developed as part of e(fx)clipse.

I’ve uploaded the slides to slideshare

The important point is that all this is plain Java! No OSGi involved so eg the Java-Sample I showed looks like this:

public class SampleJavaCode extends Application {
  private final static String[] LEGAL_CONTENT_TYPES = new String[] { 
    IJavaPartitions.JAVA_DOC,
    IJavaPartitions.JAVA_MULTI_LINE_COMMENT, 
    IJavaPartitions.JAVA_SINGLE_LINE_COMMENT,
    IJavaPartitions.JAVA_STRING, IJavaPartitions.JAVA_CHARACTER };

  private int count = 0;

  @Override
  public void start(Stage primaryStage) throws Exception {
    BorderPane container = new BorderPane();
    Document document = new Document();
    document.set(
      Util.getFileContent(getClass().getClassLoader().getResource("Sample_big.txt").toURI()));

    JavaSourceConfiguration configuration = new JavaSourceConfiguration();

    SourceViewer viewer = new SourceViewer();

    FastPartitioner partitioner = new FastPartitioner(new FastJavaPartitionScanner(), LEGAL_CONTENT_TYPES);
    document.setDocumentPartitioner(configuration.getConfiguredDocumentPartitioning(viewer), partitioner);
    partitioner.connect(document);

    viewer.configure(configuration);
    viewer.setDocument(document);
    container.setCenter(viewer.getTextWidget());

    Scene s = new Scene(container);
    s.getStylesheets().add(getClass().getResource("dark.css").toExternalForm());
    primaryStage.setScene(s);
    primaryStage.show();
  }

  public static void main(String[] args) {
    Application.launch(args);
  }
}

and looks like this:

bright.css:
blog_screen1

dark.css:
blog_screen2

All sources for the stuff I presented are available in the e(fx)clipse git repository – watch out for projects starting with “org.eclipse.fx.xtext”.