Build System
There are many ways to build prism, which means the build system is a bit more complicated than usual.
Requirements
-
It must work to build prism for all 6 uses-cases below.
-
It must be possible to build prism without needing ruby/rake/etc. Because once prism is the single parser in TruffleRuby, JRuby or CRuby there won’t be another Ruby parser around to parse such Ruby code. Most/every Ruby implementations want to avoid depending on another Ruby during the build process as that is very brittle.
-
It is desirable to compile prism with the same or very similar compiler flags for all use-cases (e.g. optimization level, warning flags, etc). Otherwise, there is the risk prism does not work correctly with those different compiler flags.
The main solution for the second point seems a Makefile, otherwise many of the usages would have to duplicate the logic to build prism.
General Design
-
Templates are generated by
templates/template.rb -
The
Makefilecompiles bothlibprism.aandlibprism.{so,dylib,dll}from thesrc/**/*.candinclude/**/*.hfiles -
The
Rakefile:compiletask ensures the above prerequisites are done, then callsmake, and usesRake::ExtensionTaskto compile the C extension (using itsextconf.rb)
This way there is minimal duplication, and each layer builds on the previous one and has its own responsibilities.
The static library exports no symbols, to avoid any conflict. The shared library exports some symbols, and this is fine since there should only be one libprism shared library loaded per process (i.e., at most one version of the prism gem loaded in a process, only the gem uses the shared library).
The various ways to build prism
Building from ruby/prism repository with bundle exec rake
rake calls make and then uses Rake::ExtensionTask to compile the C extension (see above).
Changes to the extension’s configuration options, passed after -- or through CONFIGURE_ARGS, rerun extconf.rb and rebuild the extension. For example, bundle exec rake compile -- --with-cflags=-O0 changes its compiler flags without requiring rake clobber.
Building the prism gem by gem install/bundle install
The gem contains the pre-generated templates.
When installing the gem on CRuby, extconf.rb is used and that compiles the C extension with mkmf, including both the extension files and the sources of prism itself.
When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead the extconf.rb runs make build/libprism.{so,dylib,dll}. There is Ruby code using FFI which uses libprism.{so,dylib,dll} to implement the same methods as the C extension, but using serialization instead of many native calls/accesses (JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
Building the prism gem from git, e.g. gem "prism", github: "ruby/prism"
The same as above, except the extconf.rb additionally runs first: * templates/template.rb to generate the templates
Because of course those files are not part of the git repository.
Building prism as part of CRuby
This script imports prism sources in CRuby.
The script generates the templates when importing.
prism’s Makefile is not used at all in CRuby. Instead, CRuby’s Makefile is used.
Building prism as part of TruffleRuby
This script imports prism sources in TruffleRuby. The script generates the templates when importing.
Then when mx build builds TruffleRuby and the prism mx project inside, it runs make.
Then the prism bindings mx project is built, which contains the bindings and links to libprism.a (to avoid exporting symbols, so no conflict when installing the prism gem).
Building prism as part of JRuby
TODO, similar to TruffleRuby.
Building prism for embedded system
For instance, you can build a static library libprism.a targeting the Arm Cortex-M0+ embedded system by the commands below:
-
templates/template.rb -
CFLAGS="-mcpu=cortex-m0plus" make static CC=arm-none-eabi-gcc
The build process internally looks up _POSIX_MAPPED_FILES and _WIN32 macros to determine whether the functions of the memory map are available on the target platform.
Building prism with custom memory allocator
If you need to use memory allocation functions implemented outside of the standard library, follow these steps:
-
Add
-D PRISM_XALLOCATORto the build options -
Additionally, include
-I [path/to/custom_allocator]where yourprism_xallocator.his located -
Link the implementation of
prism_xallocator.cthat contains functions declared inprism_xallocator.h
For further clarity, refer to include/prism/internal/allocator.h.
Building prism from source as a C library
All of the source files match src/**/*.c and all of the headers match include/**/*.h.
If you want to build prism as a shared library and link against it, you should compile with:
-
-fPIC -shared- Compile as a shared library -
-DPRISM_EXPORT_SYMBOLS- Export the symbols (by default nothing is exported) -
-Wl,-Bsymbolic(ELF platforms) - Bind the library’s internal calls to its own exported functions at link time, so that they cannot be redirected to another copy of libprism loaded in the same process
Flags
make respects the MAKEFLAGS environment variable. rake sets it to run one job per processor when it is not already set, so to choose a different number of jobs you can run:
MAKEFLAGS="-j4" bundle exec rake compile
Build variants
A build variant is chosen by an environment variable, which both the C extension (through ext/prism/extconf.rb) and libprism (through the Makefile) read. A variant is on when its variable is set, even to an empty value:
-
PRISM_BUILD_DEBUG- Defines thePRISM_BUILD_DEBUGbuild option below. -
PRISM_BUILD_MINIMAL- Defines thePRISM_BUILD_MINIMALbuild option below. -
PRISM_BUILD_NDEBUG- DefinesNDEBUG, which compiles out the assertions. -
PRISM_BUILD_ASAN- Compiles with the address sanitizer.rake test:asanpreloads the sanitizer runtime and runs the tests against this variant. -
PRISM_BUILD_UBSAN- Compiles with the undefined behavior sanitizer, which aborts on the first undefined behavior it detects.rake test:ubsanruns the tests against this variant.
For example, this builds the minimal variant and runs the tests against it:
PRISM_BUILD_MINIMAL=1 bundle exec rake
Each combination of variants builds the C extension in its own directory under tmp/, and libprism is rebuilt in build/ whenever its flags change, so switching between variants never needs rake clobber. The minimal and all-no-debug targets of the Makefile remain as shorthands for building libprism alone; all-no-debug also compiles with -O3.
Build options
-
PRISM_BUILD_DEBUG- Will cause all file reading to copy into its own allocation to allow easier tracking of reading off the end of the buffer. By default this is off. -
PRISM_BUILD_MINIMAL- Define all of thePRISM_EXCLUDE_*flags at once. -
PRISM_ENCODING_EXCLUDE_FULL- Will cause the library to exclude the full encoding API, and only include the minimal number of encodings to support parsing Ruby code without encoding comments. By default this is off. -
PRISM_EXPORT_SYMBOLS- Will cause the shared library to export symbols. By default this is off. -
PRISM_EXCLUDE_JSON- Will cause the library to exclude the JSON API. By default this is off. -
PRISM_EXCLUDE_PRETTYPRINT- Will cause the library to exclude the prettyprint API. By default this is off. -
PRISM_EXCLUDE_SERIALIZATION- Will cause the library to exclude the serialization API. By default this is off. -
PRISM_XALLOCATOR- Will cause the library to use the custom memory allocator. By default this is off.