34 KiB

Raw Permalink Blame History

Soong-Bazel equivalents

This doc aims to describe internal-facing implementation concepts. For external-facing, see https://android.googlesource.com/platform/build/bazel/+/refs/heads/master/docs/concepts.md.

[TOC]

Overview

Soong/Ninja	Bazel	Remarks
make phony goal, e.g. "dist", "sdk", "apps_only", "droidcore"	Top level `filegroup` rule target	Details
Ninja build target (phony)	(readable) alias to a file target
Ninja build target (non-phony)	File target
`ModuleFactory`	`RuleConfiguredTargetFactory`
`Module type` (e.g. `cc_library`)	Rule class (e.g. `cc_library`)
Module object instance	Target (instance of a rule)	Details
Module properties	Rule attributes	Details
Module name	Target label
Module variant	(Split) configured target
LoadHooks	macros (ish)
Top-down mutators on modules	Split configuration on targets	Allows building multiple "variants" of the same build artifact in the same build.
Bottom-up mutators on modules	Aspects on targets
Build statement (Ninja)	Action (result of ctx.actions.run)
Rule statement (Ninja)	ctx.actions.run() API
`out/soong/build.ninja` and `out/build-<target>.ninja`	Action graph (serialized)
Pool (ninja)	Thread pools / `ExecutorService`
Blueprint's Registration and Parse, `ResolveDependencies` phase	Loading phase
Blueprint's Generate and Write phases	Analysis Phase
Ninja execution	Execution phase
Blueprints/`Android.bp` files	`BUILD`/`BUILD.bazel` files
Namespaces	Packages	Most Soong modules are within the global namespace
Mutators	Configuration keys (ish)
Variation	Configuration value
Singleton	Aspect-ish
Target (system + vendor + product)	Platform
Bash scripts e.g. envsetup functions, `soong_ui.bash`)	Repository rule
Product and board configuration makefile and env variables	Configuration in Bazel (ish)	Details
Dependency Tags	Provider names

Remarks

Phony goals

Soong maintains the make terminology of goals to denote what should be built. All modules can be specified by name as a goal, in addition, phony goals are supported.

A Phony goal creates a Make-style phony rule, a rule with no commands that can depend on other phony rules or real files. Phony can be called on the same name multiple times to add additional dependencies. These are often used to build many targets at once. The default goal for Android's build system is droid. Some other common phony goals include: nothing (perform loading/analysis), docs, checkbuild, apps_only.

Some common phony goals are defined in build/make/core/main.mk The purpose is to help soong_ui to determine what top level files to build.

Module/Target

When a Module is instantiated by Blueprint (which calls the appropriate ModuleFactory), the property structs are populated by Blueprint.

Blueprint performs no additional operations on these properties, such that dependencies on other modules and references to source files are unresolved initially. Mutators then introspect the values of properties to specify dependencies between modules, which Blueprint resolves. Source files (including globs) and output paths for references to other modules are resolved during blueprint analysis via the various Path[s]ForModuleSrc[Excludes] functions within build/soong/android/paths.go.

For a Bazel target instance, the dependencies and source file references within attrs have been resolved by Bazel.

Bazel implementation to collect deps.

Properties/Attributes

Properties

Within Soong/Blueprint, properties are represented as Go structs, which can be nested, with no depth limit. Properties can be primitive or pointer types, but they must be one of these types: int64, string, bool, list.

These properties can be defined from various structs within the module type factory itself (via AddProperties) or from common helper functions such as:

InitAndroidModule: specifies name-related, common, and dist properties.
InitAndroidArchModule: adds host/device properies

Go comments for a property will be treated as documentation to describe the property. In some cases, these comments describe a default value for the property. However, the default value is not based on the comment or field definition but resolved somewhere within the module's mutators or build. These defaults are often determined using Blueprint proptools *Default functions. For example, cc modules have a property include_build_directory, which is described in the comments. The default value is resolved when compiler flags are being determined.

In general, these can be set in an Android.bp file. However, if the property is tagged with `blueprint:"mutated"`, it can only be set programmatically within Blueprint/Soong. Additionally, mutated tagged properties also support map and int types in addition to those mentioned above. These mutated properties are used to propagate data that gets set during mutations, which ensures that the information is copied successfully to module variants during mutation.

Soong supports additional property tags to provide additional functionality/information about a property:

`android:arch_variant`: This specifies that a property can be configured for different architectures, operating systems, targets, etc. The arch mutator, will merge target-specific properties into the correct variant for properties with this tag.

Note: if a nested property is arch-variant, all recursively nesting structs that can be specified in an Android.bp file must also be tagged as arch-variant.
`android:variant_prepend`: When merging properties for the arch variant, the arch-specific values should be prepended rather than appended to existing property values.
`android:path`: This specifies that this property will contain some combination of:
- module-relative paths
- references to other modules in the form:
  - ":<name>{.<tag>}", where {.<tag>} is optional to specify a non-default output file, specific to the module type
  - "<namespace>:<name>{.<tag>}""
Note: Dependencies to other modules for these properties will be automatically added by the pathdeps mutator.

Attributes

Similar to properties, attributes only support a few types. The difference is that Bazel attributes cannot be nested .

Some attributes are common across many/all rule classes, including (but not limited to) name, tag, visibility.

The definition of an attribute can contain settings, such as: its default value, whether it is mandatory ot have a value, and its documentation.

To specify a source file or reference to another module, use label or label_list attribute types (rather than regular string or string_list types). These support additional restrictions (as compared to string* types), such as:

whether files are supported
the providers that must be given by a dependency
whether the dependency should be executable
the configuration (host, target)
aspects

Unlike Soong, when accessing this attribute within the rule's implementation (at anlysis time), the label(s) will be resolved to the file or target they refer to.

Attributes do not need to specify whether they accept configurable attribute. However, the rule definition can specify the configuration or specify a configuration transition.

However, not all target definitions within a BUILD file are invoking a rule. Instead, they may invoke a Starlark macro, which is a load-time wrapper around rules. Arguments for a macro are not typed. If macros are used, their arguments would have to be wrangled into an attribute-compatible type.

LoadHooks

LoadHooks provide access to :

append/prepend additional properties to the module (AppendProperties/PrependProperties)
create a new module CreateModule

LoadHooks make it easier to extend existing module factories to always specify certain properties or to split a single Android.bp definition into multiple Module instances .

Build Statement (ninja)

Ninja build statements can be expanded from Ninja rules, which are like templates.

# rule
rule cattool
  depfile = out/test/depfile.d
  command = ${in} ${out}

# build statement
build out/test/output.txt: cattool test/cattool.sh test/one test/two

# build statement
build out/test/other_output.txt: cattool test/cattool.sh test/three test/four

Rules for Android.mk modules (out/build-<target>.ninja) and build statements are 1:1. That is every rule is only used once by a single build statement.

Soong (out/soong/build.ninja) rules are reused extensively in build statements (1:many). For example the Cp rule is a commonly used rule for creating build statements which copy files.

Ninja Rules in Soong

In Soong, Ninja rules can be defined in two ways:

Blueprint Generate & Write phase

ResolveDependencies Running a series of Mutators, to add dependencies, split modules with variations, etc
PrepareBuildActions:
1. Running Modules’ GenerateBuildActions to generate Ninja statements, which in turn calls each module's GenerateAndroidBuildActions.
2. Running Singletons to generate Ninja statements that generate docs, android.mk statements, etc

Soong namespaces

Module Namespaces can import other namespaces, and there’s a module name lookup algorithm which terminates in the global namespace.

Note: this is not widely used and most Soong modules are in the global namespace.

Bazel packages

Packages can nest subpackages recursively, but they are independent containers of Bazel targets. This means that Bazel target names only need to be unique within a package.

Mutators

blueprint invokes mutators are invoking in the order they are registered (e.g. top-down and bottom-up can be interleaved). Each mutator applys a single visitation to every module in the graph.

Mutators visiting module can parallelized, while maintaining their ordering, by calling .Parallel().

While top-down and bottom-up mutators differ in their purposes, the interface available to each contains many similarities. Both have access to: BaseModuleContext and BaseMutatorContext.

In addition to the registration order, Soong supports phase-based ordering of mutators:

Pre-Arch: mutators that need to run before arch-variation. For example, defaults are handled at this stage such properties from defaults are correctly propagated to arch-variants later.
(Hard-coded) archMutator splits a module into the appropriate target(s). Next, the arch- and OS-specific properties are merged into the appropriate variant.
Pre-Deps: mutators that can/need to run before deps have been resolved, for instance, creating variations that have an impact on dependency resolution.
(Hard-coded) depsMutator, which calls the DepsMutator function that must be part of a Soong Module's interface.
Post-Deps: mutators that need to run after deps have been resolved
Final-Deps like post-deps but variations cannot be created

Top-down Mutator

A top-down mutator is invoked on a module before its dependencies.

The general purpose is to propagate dependency info from a module to its dependencies.

Bottom-up Mutator

A bottom-up mutator is invoked on a module only after the mutator has been invoked on all its dependencies.

The general purpose of a bottom-up mutator is to split modules into variants.

Soong/Blueprint Variation

A tuple (name of mutator, variation / config value) passed to CreateVariations.

Configuration

Soong's config process encompasses both what should build and how it should build. This section focuses on the how aspect.

We do not cover how Soong's configuration will be implemented in Bazel, but the general capabilities of Bazel to configure builds.

Soong

Android users can configure their builds based on:

Specifying a target (via lunch, banchan, tapas, or Soong’s command line options)
Environment variables

Some environment variables or command line options are used directly to alter the build. However, specification of target product encompasses many aspects of both what and how things are built. This configuration is currently handled within Make but is in the process of being migrated to Starlark.

Soong invokes Kati to run in a "config" mode, also commonly known as "product config". This mode limits the scope of what .mk files are parsed. The product-specific handlers are largely in:

product_config.mk: this subset of functionality is also commonly referred to as "product config"
board_config.mk

However, these cover only a subset of config.mk. This ensures that all values have appropriate defaults and specify details necessary to the build. Some examples:

handling of version defaults
rbe setup
user-defined config: buildspec.mk is similar to .bazelrc file.
ensuring PRODUCT_SHIPPING_API_LEVEL is defaulted if not specified by the target.

Finally, Kati dumps variables to be consumed by Soong:

environment variables specifically requested by Soong
writes soong.variables, a JSON file

Throughout Soong, environment variables can be accessed to alter the build via the Config:

Soong loads the soong.variables config file, stored as productVariables. These variables are used in three ways:

Direct access from Config, for example: paths can be opted out of specific sanitizers
In limited cases, users can use these within their Android.bp file to control what is built or perform variable replacement. variableProperties limits which configuration variables can be specified within an Android.bp file and which properties they can apply to. The values specified within an Android.bp file, are merged/replaced by the VariableMutator, which appends performs string replacement if requested and merges the properties into the modules.
Through Soong Config Variables: which allow users to specify additional configuration variables that can be used within an Android.bp file for the module type and properties they request. Soong config variable structs are dynamically generated via reflection. In the factory, the properties to merge into the module instance are identified based on the config variable's type.

The product configuration also provides information about architecture and operating system, both for target(s) and host. This is used within the archMutator to split a module into the required variants and merge target-specific properties into the appropriate variant. Only properties which have been tagged with android:"arch_variant" can be specified within an Android.bp as arch/os/target-specific. For example:

type properties struct {
  // this property will be arch-variant
  Arch_variant_not_nested *string `android:"arch_variant"`

  Nested_with_arch_variant struct {
    // this property is arch-variant
    Arch_variant_nested *string `android:"arch_variant"`

    // this property is **not** arch-variant
    Not_arch_variant_nested *string
  } `android:"arch_variant"`

  Nested_no_arch_variant struct {
    // this property is **NOT** arch-variant
    No_arch_variant_nested_not_arch_variant *string `android:"arch_variant"`

    // this property is **not** arch-variant
    No_arch_variant_nested *string
  }
}

The arch/os/target-specific structs are dynamically generated based on the tags using reflection.

Bazel

Bazel documentation covers configurable builds fairly extensively, so this is a short overview that primarily links to existing Bazel documentation rather than repeating it here.

Configurable attributes, (aka select()) allows users to toggle values of build rule attributes on the command line.

Within a rule, the value of a select will have been resolved based on the configuration at analysis phase. However, within a macro (at loading phase, before analysis phase), a select() is an opaque type that cannot be inspected. This restricts what operations are possible on the arguments passed to a macro.

The conditions within a select statement are one of:

A config_setting is a collection of build settings, whether defined by Bazel, or user-defined.

User-defined build settings allow users to specify additional configuration, which optionally can be specified as a flag. In addition to specifying build settings within a config_setting, rules can depend directly on them.

In addition, Bazel supports platforms, which is a named collection of constraints. Both a target and host platform can be specified on the command line. More about platforms.

Communicating between modules/targets

Soong communication

There are many mechanisms to communicate between Soong modules. Because of this, it can be difficult to trace the information communicated between modules.

Dependency Tags

Dependency tags are the primary way to filter module dependencies by what purpose the dependency serves. For example, to filter for annotation processor plugins in the deps of a Java library module, use ctx.VisitDirectDeps and check the tags:

ctx.VisitDirectDeps(func(module android.Module) {
  tag := ctx.OtherModuleDependencyTag(module)
  if tag == pluginTag { patchPaths += ":" + strings.Split(ctx.OtherModuleDir(module), "/")[0] }
  }
)

At this point the module managing the dependency, may have enough information to cast it to a specific type or interface and perform more specific operations.

For instance, shared libraries and executables have special handling for static library dependencies: where the coverage files and source based ABI dump files are needed explicitly. Based on the dependency tag, the module is cast to a concrete type, like cc.Module, where internal fields are accessed and used to obtain the desired data.

Usage of dependency tags can be more evident when used between module types representing different langauges, as the functions must be exported in Go due to Soong's language-based package layout. For example, rust uses cc module's HasStubVariants.

Interfaces

A common mechanism for a module to communicate information about itself is to define or implement a Go interface.

Some interfaces are common throughout Soong:

SourceFileProducer, by implementing Srcs() Paths
OutputFileProducer by implementing OutputFiles(string) (Paths, error)
HostToolProvider by implementing HostToolPath() OptionalPath

SourceFileProducer and OutputFileProducer are used to resolve references to other modules via android:"path" references.

Modules may define additional interfaces. For example, genrule defines a SourceFileGenerator interface.

Providers

Soong has Bazel-inspired providers, but providers are not used in all cases yet.

Usages of providers are the easiest, simplest, and cleanest communication approach in Soong.

In the module providing information, these are specified via SetProvider and SetVariationProvider.

In the module retrieving information, HasProvider and Provider or OtherModuleHasProvider and OtherModuleProvider are used to test existence and retrieve a provider.

Bazel communication

Targets primarily communicate with each other via providers in Bazel rule implementations. All rules have access to any of the providers but rules will pick and choose which ones to access based on their needs. For example, all rules can access JavaInfo provider, which provides information about compile and rolled-up runtime jars for javac and java invocations downstream. However, the JavaInfo provider is only useful to java_* rules or rules that need jvm information.

Starlark rules

Providers are pieces of information exposed to other modules.

One such provider is DefaultInfo, which contains the default output files and runfiles.

Rule authors can also create custom providers or implement existing providers to communicate information specific to their rule logic. For instance, in Android Starlark cc_object rule implementation, we return a CcInfo provider and a custom CcObjectInfo provider.

Native rules

For implementation of native rules in Java, ruleContext.getPrerequisite is used to extract providers from dependencies.

`depset` construction

depset are used in conjunction with providers to accumulate data efficiently from transitive dependencies. used to accumulate data from transitive dependencies.

`exports`

Some target have an exports attribute by convention, like java_library.exports. This attribute is commonly used to propagate transitive dependencies to the dependent as though the dependent has a direct edge to the transitive dependencies.

34 KiB Raw Permalink Blame History Unescape Escape