0.4.0 (Jan 19, 2021)

We are happy to announce the release of Scala Native 0.4.0!

Scala Native is an optimizing ahead-of-time compiler and lightweight managed runtime designed specifically for Scala. It is developed at the Scala Center and with the help from VirtusLab along with contributors from the community.

Check out the documentation at https://scala-native.readthedocs.io/

TL;DR

  • Not backward compatible with previous releases,

  • A unique nativeConfig setting replaces the set of nativeX settings,

  • The partial implementation of the JDK packages java.time and java.text were removed from core repo. Third-party libraries such as scala-java-time and scala-java-locales should be used instead,

  • CFuncPtr is now created by implicit conversion from ordinary scala.Function

  • Added Scala 2.12 and 2.13 support,

  • Added support for JUnit,

  • Additional C/C++ can be added to compilation pipeline,

  • Allowed for cross compilation using custom target triple

  • Allowed reflective instantiation by using @EnableReflectiveInstantiation annotation,

  • Added new Garbage Collector - Concurrent Mark and Parallel Sweep Garbage Collector, called Commix,

  • Various bug fixes

Breaking changes

Broken backward compatibility

Scala Native 0.4.0 breaks backward binary compatibility with previous releases of Scala Native. Libraries published using version 0.4.0-M2 or older must be republished for Scala Native 0.4.x.

Removal of java.time / java.text

This release removes the partial implementations of the java.time and java.text packages from Scala Native core. This will allow third-party libraries, like scala-java-time and scala-java-locales, to provide more complete versions thereof.

Using methods that directly or transitively need the removed classes will require an additional dependency on the appropriate third-party library. For example:

val str: String = "Hello Native"
str.toLowerCase()              // works as before
str.toLowerCase(Locale.French) // requires scala-java-locales to link

NativeConfig replaces setting keys

The nativeXyz setting keys are now deprecated in favor of a single nativeConfig setting, which can be used as follows:

// build.sbt
nativeConfig ~= {
  _.withMode(build.Mode.releaseFast)
   .withGC(build.GC.immix)
   .withLTO(build.LTO.full)
   .withOptimize(true)
   .withCompileOptions(Nil)
   .withLinkingOptions(Nil)
}

Old style settings keys are still supported, but they have lower priority than the new config and will be removed at some point in the future. In the following example resulting LTO setting would be set to thin

nativeConfig := nativeConfig.value.withLTO(build.LTO.thin)

nativeLTO := "none"

CFuncPtr changes

You no longer need to implement the CFuncPtrN trait which is now private for Scala Native implementation. Instead, you can use an implicit conversion method taking arbitrary scala.FunctionN and returning CFuncPtrN.

type Callback = CFuncPtr1[CInt,Unit]
def registerCallback(cFn: Callback): Unit = extern
def fn(n: CInt): Unit = ???

registerCallback(CFuncPtr1.fromScalaFunction(fn))
registerCallback(fn)

registerCallback { (n: CInt) => println("hello native") }

It’s now also possible to work with an arbitrary pointer and convert it to CFuncPtrN that can be called in your Scala code or to convert your function to any pointer if your native library needs this.

import scala.scalanative.unsafe.Ptr
val cFnPtr: CFuncPtr0[CInt]    = ???
val fnPtr: Ptr[Byte]           = CFuncPtr.toPtr(cFnPtr)
val fnFromPtr: CFuncPtr0[CInt] = CFuncPtr.fromPtr[CFuncPtr0[CInt]](fnPtr)

Other breaking changes:

  • Sbt 0.13.x is no longer supported - upgrade to 1.1.6 or newer.

  • The minimal version of Clang working with Scala Native is now 6.0

  • CSize is now unsigned numeral type

  • Usage of signed numeral types for methods expecting CSize was deprecated.

New features

Supported Scala versions

We added support for Scala 2.12.13 and 2.13.4, in addition to the existing support for 2.11.12.

JUnit Support

Scala Native now comes with JUnit support out of the box, this means that you can write tests in the same way you would do for a Scala/JVM or Scala.js project. To enable JUnit tests all you will need to do is to add the two following lines to your build.sbt.

addCompilerPlugin("org.scala-native" % "junit-plugin" % nativeVersion cross CrossVersion.full)
libraryDependencies += "org.scala-native" %%% "junit-runtime" % nativeVersion % "test"

Reflective instantiation

Since this release you are able to reflectively instantiate definitions marked with the @EnableReflectiveInstantation annotation, as well as its descendants. Annotated classes and modules, having a concrete implementation, can be accessed via the provided scalanative.reflect.Reflect API. If you have used Scala.js before, it may seem similar to you, as the new implementation uses exactly the same API.

Scala Native does not support full reflection support, although this feature might fix most of the issues that could occur in users code.

package x.y.z

@EnableReflectiveInstantation
trait ReflectiveFoo {
  val value: String = "foo"
}

object SingleFoo extends ReflectiveFoo

case class MultipleFoo(times: Int) extends ReflectiveFoo {
  override val value: String = super.value * times
}

for {
  cls  <- lookupInstantiatableClass("x.y.z.MultipleFoo")
  ctor <- cls.getConstructor(classOf[Int])
  obj  <- ctor.newInstance(5)
} yield obj // results in Some(new MultipleFoo(5))

for {
  cls <- lookupLoadableModule("x.y.z.SingleFoo")
  obj <- cls.loadModule()
} yield obj // results Some(SingleFoo)

Cross compilation

It is now possible to define a custom target for the compiler by providing an LLVM-style TargetTriple in your config. The default behavior is still to target the host architecture and operating system.

For example, if you’re working on Linux and would like to create an executable suitable for MacOS without changing your whole build, you can use the following sbt setting::

sbt 'set nativeConfig ~= {_.withTargetTriple("x86_64-apple-darwin<version>")}' myApp/nativeLink

We consider changing target triple as a feature for advanced users, and cannot promise it would currently work with any possible configuration yet. However, the number of supported architectures and operating systems would definitely grow in the future.

When using Linux / MacOS, you can check the target triple used in your environment with the command llvm-config --host-target.

Native sources in the build

With the 0.4.0 release you’re able to put your C/C++ sources in the resources/scala-native directory inside your project, so they will be linked and compiled inside the SN pipeline.

As an example you can use it to access macro-defined constants and functions or to pass structs from the stack to C functions.

// src/resources/scala-native/example.c
typedef int (*Callback0) (void);

const int EXAMPLE_CONSTANT = 42;

int exec(Callback0 f) {
 return f();
};


// src/main/example.scala
@extern
object example {
 def exec(cb: CFuncPtr0[CInt]): ExecResult = extern

 @name("EXAMPLE_CONSTANT")
 final val someConstant: Int = extern
}

Commix GC

This release also adds a new Garbage Collector - Commix, a parallel mark, and concurrent sweep GC, based on the well known Immix GC. It reduces GC pause times by utilizing additional processor cores during mark and sweep phases.

While the GC itself will use multiple threads, Scala Native still does not support multi-threading in the application code. Commix GC was written in C and uses pthread to work. In case your application needs concurrency support, you may try the experimental library scala-native-loop

Bugfixes

  • Failures during the build of multiple parallel projects using common jar were fixed,

  • Lowered overall memory usage when compiling and linking,

  • Value classes are now correctly handled in lambda functions,

  • The synchronized flag in now taken into account when generating methods,

  • Constructors are no longer treated are virtual methods, they’re always resolved statically,

  • Generic CFuncPtr can be passed as method arguments,

  • Binary operations with Nothing arguments will no longer break compilation,

  • Resolving of public method no longer can result in private method with the same name,

  • Instances of java.lang.Class are now cached and can be correctly tested using reference equality,

  • Triple-quoted CString’s are now correctly escaped,

  • Identifiers starting with digits are now correctly handled,

  • Fixed errors with too many open files after consecutive runs,

  • Fixed crashes when HOME env variable was not set,

  • Boehm GC installed using MacPorts is now supported,

  • Fixed segmentation fault when trying to access current, unlinked directory,

  • malloc will now throw OutOfMemoryError when it cannot allocate memory,

  • toCString & fromCString now correctly return null,

  • Fixed errors with not cleared errno when using POSIX readdir

  • Array operation now throw JVM-compilant ArrayIndexOutOfBoundsException,

  • Fix bug with BufferedInputStream.read() for values bigger then 0x7f,

  • Files.walk accepts non-directory files,

  • Improved IEEE754 specification compliance when parsing strings,

  • Fixed infinite loop in java.io.RandomAccesFile.readLine,

  • Added multiple missing javalib classes and methods

Contributors

Big thanks to everybody who contributed to this release or reported an issue!

$ git shortlog -sn --no-merges v0.4.0-M2..v0.4.0
    64	LeeTibbert
    58	Wojciech Mazur
    37	Eric K Richardson
    13	Kirill A. Korinsky
    10	Ergys Dona
     8	Lorenzo Gabriele
     4	Sébastien Doeraene
     3	Valdis Adamsons
     2	Denys Shabalin
     2	Ondra Pelech
     2	kerr
     1	Danny Lee
     1	Nadav Samet
     1	Richard Whaling
     1	jokade
Merged PRs 207
Closed issues 203
Contributors 15

Full Changelog

The most impacting merged pull requests:

Compiler

Sbt plugin

  • Fix #1849: Streamline clang version detection #2099 (ekrich)

  • Remove target triple discovery code #2033 (ekrich)

  • Fix #2024: Use a shared Scope in the sbt plugin for all parallel tasks #2039 (WojciechMazur)

  • Fix #1999: Clear errno before readdir in posixlib dirent.c #2000 (LeeTibbert)

  • Fix #1711: Ignore non-jar non-directory elements on the classpath #1987 (ekrich)

  • Fix #1970: Restrict native code to a specified subdirectory #1876 (ekrich)

  • Fix #1597: Introduce nativeConfig instead of multiple nativeX keys #1864 (WojciechMazur)

  • Import the testing infrastructure of Scala.js. #1869 (WojciechMazur)

  • Discover and use clang 11 if present, and drop clang < 5.0. #1874 (LeeTibbert)

  • Fix #657: Give libraries a way to include native C code to be compiled. #1637 (ekrich)

  • Fix “too many open files” after consecutive runs #1839 (errikos)

  • Drop support for sbt 0.13.x. #1712 (ekrich)

  • Support boehm installed from macports #2071 (catap)

  • Never use default path that doesn’t exists #2091 (catap)

  • Fix crash when HOME env variable is not set #1738 (lolgab)

Native library

Java standard library

  • Partial fix #1023: Port j.u.NavigableMap #1893 (LeeTibbert)

  • Support java.util.Date methods using java.time.Instant. #2088 (WojciechMazur)

  • Remove dummy java.time implementation from core repo. #2087 (WojciechMazur)

  • String to{Lower, Upper}Case handles Unicode special, non-locale dependent, casing rules #2098 (WojciechMazur)

  • Port localized String.to{Lower, Upper}Case from Scala.js #2095 (WojciechMazur)

  • Port optional Locale j.u.Formatter from Scala.js #2079 (WojciechMazur)

  • Implement j.u.Map default methods. #2061 (LeeTibbert)

  • Fix #2049: Use natural ordering for Arrays#sort with null comparator #2050 (LeeTibbert)

  • Fix #1993: Port ju.ConcurrentLinkedQueue from Scala.js #1994 (lolgab)

  • Fix #2044: Throw JVM-compliant ArrayIndexOutOfBoundsException for array ops #2047 (WojciechMazur)

  • Work around limitation for JDK12+ about j.l.constant.Constable #1941 (catap)

  • Port j.u.Objects#requireNonNull with Supplier argument #1975 (LeeTibbert)

  • Port Scala.js j.u.Objects parameter widening & later changes. #1953 (LeeTibbert)

  • Add j.l.Iterable.forEach #1934 (LeeTibbert)

  • Add the default methods of j.u.Iterator default methods #1937 (LeeTibbert)

  • Fix BufferedInputStream.read() for values bigger than 0x7f #1922 (catap)

  • Update java.util.Properties to match Scala.js changes #1892 (ekrich)

  • Provide more useful j.l.Thread#getStackTrace stub #1899 (LeeTibbert)

  • Fix #1780: Fix an ambiguous overload about java.nio.FileSystems.newFileSystem on JDK 13+. #1873 (WojciechMazur)

  • Fix #1871: Fix a corner case of defaults in ju.Properties.{stringP,p}ropertyNames. #1872 (ekrich)

  • Fix #1064: Implement java.util.Properties.{load,store}. #1653 (ekrich)

  • Fix #1758: Accept a non-directory file in Files.walk() #1838 (WojciechMazur)

  • Fix #1559: Improve spec compliance when parsing IEEE754 strings. #1703 (LeeTibbert)

  • Port/implement j.u.ArrayDeque #1696 (LeeTibbert)

  • Fix #1693: j.u.AbstractCollection#toString output now matches Scala JVM #1697 (LeeTibbert)

  • Fix #1683: Implement suppression and non-writable trace for Throwable #1688 (LeeTibbert)

  • Fix #1623: Fix an infinite loop in j.i.RandomAccessFile#readLine #2100 (LeeTibbert)

  • Fix scala-js#4088: Avoid an Int overflow in BigDecimal.toString(). #1837 (LeeTibbert)

  • Update uppercase lowercase to use UnicodeData.txt vs CaseFolding.txt #1611 (ekrich)