FreeBSD Manual Pages
SYNTH(1) General Commands Manual SYNTH(1) NAME synth -- custom package repository builder for FreeBSD and DragonFly SYNOPSIS synth [help | configure | version | status | purge-distfiles] synth [rebuild-repository | prepare-system | upgrade-system] synth [everything | status-everything] synth [status | build | just-build | install | force | test] [file | port-origin ...] DESCRIPTION The synth program is an advanced concurrent ports building tool aimed at system administrators that prefer or require the building of packages from source rather than installing official binary packages. synth will build packages in a clean environment can exactly mirror the system that they are built on, it will create local package respositories and install pkg(8) repository configuration file that causes locally built packages to be used with the highest priority, all while allowing the system to fully upgraded with a single command. synth version When synth is executed with the version command, an unknown command, or no command at all, it will display a short block of text containing an identi- fication, summary, version number, copyright, usage, and a summary of valid commands. synth help The help command displays lists each valid command with a short synopsis of the command's function. It also elaborates the several commands expect one or more port origins to be provided, or a path to a file with the same in- formation. synth configure The configure command launches an interactive menu for providing the user with a clean method to configure synth. It works on the concept of pro- files, and the default profile "LiveSystem" is automatically installed with values that are determined by scanning the system. Most of the configurable items are directories that are used to construct the build environment, but they aren't limited to that. Ports directory If the PORTSDIR environment variable is set when the initial profile is created, it establishes the initial value. Otherwise, synth will extract the value of PORTSDIR from /usr/share/mk and use that. If neither source points to a valid path, /usr/dports and /usr/ports will be successfully tried. If those don't exist either, synth will exit with an error message. Packages directory This is the location of the local repository metadata and all the packages that are built by synth. It clearly must have lots of available disk space. The default value is /var/synth/live_packages but it should be altered if the system has a /var partition with limited space. Distfiles directory This indicates where all the ports source distri- bution files are stored, and the initial value is determined by scanning the existing configura- tion. The defaults are defined by ports and are /usr/ports/distfiles for (FreeBSD) and /usr/distfiles for (DragonFly). Most users will want to keep the initial value as that directory should already contain distfiles. Port options directory This is the directory where all the selected op- tions for ports are cached. The initial value comes from a system scan, so chances are it has the correct value. However, if the user would like a separate configuration area for port op- tions, they would create the new directory and set this value accordingly (but the user would have to ensure the new location is passed to the port when configuring port options later in that case). Build logs directory A log is produced for every port built (subse- quent builds of the same port will overwrite pre- vious logs). This item dictates where those logs are stored, which is /var/log/synth by default. If the /var partition is limited, the user will definitely want to change this location as the uncompressed logs can potentially consume giga- byte of space. Build base directory This is the mount point for all the builders. When tmpfs is used, this location requires very little space as it contains empty directories that serve as mount points. If tmpfs is not used, then this location should have access to a lot available diskspace as it will provide the work area and virtual /usr/local for each builder. The default location of /usr/obj/synth-live should be fine for most users (it generally has lots of space and tmpfs is the default build mode). System root directory Most users will use the base system as the basis for the build environment, and thus the default setting of / is correct. Advanced users may use DESTDIR with installworld to create an alterna- tive build base (e.g. i386 cross-build on amd64 architecture or a modified base that serves as a fallout test for an upcoming feature). Those would be the users that would modify this value. Compiler cache directory If ccache is properly installed on the system, set this value to the location of the cache. All the builders will leverage it. It is disabled by default. It can be disabled again by entering any invalid path as a new value, e.g. "none". Num. concurrent builders The represents the number of ports that can be simultaneously built on the system. The selected value is influenced by the number of physical cores and hyperthreads the system has, the amount of memory on the system, the amount of available swap, and if that swap is a solid-state drive or not. Generally memory is the limiting resource when tmpfs is used, so the default value for the number of builders is generally 75% of the number of CPUs of the system. The user is free to bal- ance jobs versus builders as well. Max. jobs per builder If memory is constrained, it's often a better performance tradeoff to reduce the number of builders and increase the number of jobs per builder. The default value varies depending on the system, but it will never exceed 5. Use tmpfs for work area If the initial scan shows there is at least 1.25 Gb per builder, this item will default to "true". Building on tmpfs is a big speed spot and it's recommended if the memory resources allow it. This item is directed at the work area for build- ing the port Use tmpfs for /usr/local Before a port can start building, all the build and library dependencies have to be installed into the localbase. Extracting, installing and later deinstalling is a lot of work for a disk, so using tmpfs here is another performance boost. It is recommended that users with adequate memory leave this enabled. Display using ncurses During the concurrent building, the default dis- play is constructed using ncurses. It shows a lot of good information and it should be used. The display will fall back to a text mode if the ter- minal doesn't support it. This setting forces text mode unconditionally, but few users will want or need to use this option. Fetch prebuilt packages When active, this option will scan the external repository for suitable prebuilt packages to sat- isfy build requirements. To be considered suit- able, the ABI, options, and dependencies of the remote package must all match requirements. If a package is located, it will be fetched and placed in the packages directory and later incorporated into the local repository. This option is not active by default, meaning that synth will nor- mally build everything from source and ignore ex- ternal repositories. To create alternative profiles, press the ">" key at the prompt. A new menu will appear that provides options to switch to another existing profile or create a new one that can be tailored and named. Any newly created profile becomes the new default profile, but this can be easily changed by repeat- ing the profile selection process. Pressing the Alphabetic key associated with each configuration item brings up a prompt to change the value. Directory entries must be valid paths ex- cept in the case of ccache. The boolean values are modified with a single keypress ("T" or "F") and the numeric items require positive integers. Af- ter changing the values, the old menu returns and altered values are marked with an asterisk. Simply entering the "Enter" key (carriage return) saves the changes while pressing the "Escape" key will discard them. If more than one profile exists, an option to delete profiles will appear on the menu. By pressing the "<" key, a new menu will appear that lists all the inactive profiles. Entering the number associated with one of those profiles will immediately delete the profile. The deletion cannot be undone. synth upgrade-system The upgrade-system command automates the full upgrade of the system upon which synth is installed. It is expected that the ports tree has been up- dated to the latest version before the command is executed. Using the ports tree as a reference, synth will query pkg(8) to determine what is currently installed and which software has newer versions available. It will rebuild those packages and any package that depends libraries they contain or that have a runtime dependency on the rebuild packages. In addition, any port that has changed with respect to its options, dependencies, or ABI will be rebuilt as well. After analysis, the concurrent builder will begin building the set of packages that have been identified as requiring a rebuild. When that phase is complete, the local repository will be rebuilt without wait- ing for confirmation. Finally, synth will command pkg(8) to upgrade the system using the local repository, effectively bringing it completely up to date. synth prepare-system The prepare-system command has the same functionality of the upgrade-system command with the exception that the packages will not be installed. Once the local repository is fully rebuilt, the command exits. synth rebuild-repository The rebuild-repository command will perform a full sanity check on all packages present in the packages directory and remove the ones which fail the check. The local Synth repository will be created (or recreated) using the remaining packages. This step is also done as part of several other commands, so it mainly exists for scripting purposes. synth status The status command with no arguments performs a dry-run of the upgrade-system command. It will not delete obsolete packages, nor rebuild the local repository. It will list all the ports that will be rebuild along with a total, and it also logs the same information to /var/synth/synth_status_results.txt since the full list is often longer than the terminal height. synth just-build [file | port-origin ...] The just-build command takes a list (one or more) of port origins (an ori- gin is the combination of a category and the port directory, e.g. lang/python27) and it will builds the packages of those ports if necessary. If analysis determines the packages are already up to date, nothing is done. This is the first of several commands that take arguments. The arguments are either an unlimited number of port origins, e.g. synth just-build editors/joe editors/nano editors/libreoffice or they are limited to one, a path to a file, e.g. synth just-build /tmp/build.list A file that is equivalent to the first example would contain: editors/joe editors/nano editors/libreoffice When the building task is complete, synth exits. synth build [file | port-origin ...] The build command differs from the just-build command by asking the user if they wish to rebuild the local repository when the building is complete. The answer is usually "N" (for "no") because rebuilding the repository can take a few minutes to complete, and it only makes sense to do when all the building is complete. Should the user opt to rebuild the repository, synth will then ask if the user wishes to install the packages that come from the ports list on the system. synth install [file | port-origin ...] The install command is similar to the build command except that it will not ask permission to rebuild the repository or install the resulting packages. It will just do it all in sequence. synth force [file | port-origin ...] The force command is similar to the build command except that any packages related to the ports list will be deleted first, regardless if they are up to date or not. This results in that every port on the given list will build. As with the build command, synth will request permission to rebuild the repository and possibly install the packages when the building phase is complete. synth test [file | port-origin ...] The test command will pre-delete any existing packages related to the ports list before rebuilding them with extra DEVELOPER_MODE tests and settings. When the building is complete, it will just exit without offering to re- build the repository. synth status [file | port-origin ...] This version of the status command provides a dry-run and reports on what would happen if the just-build command was executed with the same argu- ments. The output and logging is similar to the solo status command, but the build scope is limited to the given ports list. synth everything The everything command is not meant for most users. It will attempt to build every port in the ports tree, which can take several days depending on the power and resources of the building machine. Even downloading the tens of gigabytes worth of distfiles consumes significant time! synth status-everything The status-everything command is just a dry-run for the everything command. Since synth works through increment building (that is, it will not build a port if an up to date package already exists for it and all the packages dependencies are also in place) , the resultant build list can be much shorter than the full list of ports in the tree. Only those rare users that wish to build the entire set of ports would use this command. synth purge-distfiles This is a useful command that will first scan the entire ports tree to gather a complete list of distfiles used by the ports tree, and then it will scan the contents of the configured distfiles directory. All source distribution files that cannot be matched to at least one port in the tree will be removed. This can result in gigabytes of recovered disk space if significant time has passed since the distfiles directory was last cleaned. ENVIRONMENT PORTSDIR The PORTSDIR environment variable is only checked on the very first signficant command of synth (meaning everything except help and version) when the default profile is created. It is also checked when adding new profiles. The variable dictates the default location of the ports tree with a higher priority than the standard locations. SYNTHPROFILE When this environment variable is set to name of an existing synth profile, that profile will be loaded when synth is launch rather than the active profile. It is a temporary override; removing the variable will allow the default pro- file to load again. WHYFAIL When this environment variable is defined to any value, the sanity check routines will provide more detail on dependency and option check failures on a separate output line. ENTERAFTER This environment variable only applies with the test command, and then only when a single port is specified. If both of those conditions are satisfied, and the variable has the value of: - extract - patch - configure - build - stage - install - deinstall then an interactive build will be attempted. First, all the port's prerequisites are built normally and synth will return to text mode. If everything built successfully, the specified port will build up to and including the point specified by ENTERAFTER. At that point, control will be given to the user by providing them with a tcsh shell with the current direc- tory set at the root of the builder. When the user is done, they should terminate the shell with the exit command to sig- nal synth to clean up and terminate. This is a developer tool to troubleshoot port build problems and it is not intended for the average user. LOCK This environment variable only applies with the test command. When it is defined to any value, the builder's localbase is remounted read-only prior to the configure phase and restored to read/write permissions after building is complete. This is a diagnostic tool aimed at pinpointing file system viola- tions as a port is not supposed to write to localbase prior to staging. Note that the localbase tmpfs setting will con- sidered as False in this mode, so building may be noticeably slower than normal. SELFTEST This environment variable only applies with the test command. When it is defined, the "test" target is executed as part of the build cycle; this target is not executed under any other circumstances. FILES The following files may be used or created: synth.ini This is the configuration file. It is automatically gen- erated and manipulated using the interactive configura- tion menu. There is no need to touch this file other than to remove unwanted profiles as currently that is not pos- sible from the program itself. It is normally located at /usr/local/etc/synth/synth.ini <profile>-make.conf This is an optional, user-provided file. If it exists, the builder's /etc/make.conf will be appended with the contents of this file. For the default profile, the file would normally be located at /usr/local/etc/synth/LiveSystem-make.conf <profile>-environment This is an optional, user-provided file. It contains a list of name-value pairs joins with an equals sign (e.g. HTTP_PROXY=http://proxyserver:8888), one pair per line. If the file exists, these variables will be defined in the builder environment. For the default profile, the file would normally be located at /usr/local/etc/synth/LiveSystem-environment <profile>-private.key This is one of two files required to support the signing of the repository by a local RSA key. It is the output of the openssl genrsa command; see the EXAMPLE section for how to generate it. The file permissions are re- quired to be set at 400 as well. For the default pro- file, the file is normally located at /usr/local/etc/synth/LiveSystem-private.key <profile>-public.key This is the second of two files required to support the signing of the repository by a local RSA key. synth must confirm the public counterpart of the signing key is available because its path is required for the generated pkg repository configuration file. For the default pro- file, the file is normally located at /usr/local/etc/synth/LiveSystem-public.key <profile>-signing_command This is one of two files required to support the signing of the repository by an external signing server. It con- sists of a single line containing the command; see the EXAMPLE section. For the default profile, the file is normally located at /usr/local/etc/synth/LiveSystem-signing_command <profile>-fingerprint This is the second of two files required to support the signing of the repository by an external signing server. It consists of a single line containing the SHA 256 hash of the public key that is the counterpart of the key that signed the repository. For the default profile, the file is normally located at /usr/local/etc/synth/LiveSystem-fingerprint <profile>-index The introduction of flavors within FreeBSD ports de- stroyed the 1:1 relationship between port directories and packages. Now any port can generated any number of pack- ages. This requires scanning every port in the tree to determine the true origin range. This is a time consum- ing task, so Synth caches the scan results and reuses it as long as all ports files are older than the generated index. When Synth detects the index is out-of-date or if it doesn't exist, it is automatically generated. One in- dex exists for each profile, with the default profile in- dex normally located at /var/cache/synth/LiveSystem-index synth_status_results.txt Whenever a status command is run, a list of ports that would build with the actual command is produced at /var/synth/synth_status_results.txt synth_prefetch_list.txt Whenever a status command is run, and the option to use suitable prebuilt packages has been selected, a list of packages that will be fetched as a result will be written to /var/synth/synth_prefetch_list.txt index.html For every build run, Synth installs a web summary at the Report subdirectory of the build logs directory. For the default profile, the file would be located at /var/log/synth/Report/index.html. To remotely view the web report with a browser, a web server must be present and configured to expose the logs directory. The web re- port is dynamic and updated 10 times per minute. The complete history is available and searchable, with quick- filters on "Built", "Failed", "Ignored" and "Skipped" fields as well as on the "No." column which is useful to examine cascaded skips. The search filter can be quickly cleared by clicking on the "Total" field. port logs Every single attempt at building a port results in a build log. The name of the log is always in the form of [category]___[port].log so that subsequent builds will overwrite previous attempts. They are created in the di- rectory specified by the configuration. hook_run_start If this file exists, and it is executable (or it is a symbolic link that leads to an executable file) then it will be executed by synth at the start of the build. The same requirement exists for all hooks. This hook, and the next five hooks all define the following common environ- ment variables: PROFILE, DIR_PACKAGES, DIR_REPOSITORY, DIR_PORTS, DIR_OPTIONS, DIR_DISTFILES, DIR_LOGS, DIR_BUILDBASE. Uniquely, this hook also defines PORTS_QUEUED which is the number of ports that synth will attempt to build. This file must be colated with synth.ini, so that would be /usr/local/etc/synth/hook_run_start for most installa- tions. hook_run_end This hook is fired when the build completes. In addition to the common definitions, it defines PORTS_BUILT, PORTS_FAILED, PORTS_IGNORED, and PORTS_SKIPPED in the en- vironment, which represents the results of the build. hook_pkg_success This hook is fired whenever a builder successfully pack- ages a port. In addition to the common definitions, it defines RESULT=success, ORIGIN, and PKGNAME in the envi- ronment. hook_pkg_failure This hook is fired whenever a builder fails to package a port. In addition to the common definitions, it defines RESULT=failure, ORIGIN, and PKGNAME in the environment. hook_pkg_ignored This hook is fired prior to the start of a build when synth prescans the queue for ports that are to be ignored according to the ports tree. Once the build has started, this hook is never fired again. In addition to the common definitions, it defines RESULT=ignored, ORIGIN, and PKGNAME in the environment. hook_pkg_skipped This hook is fired whenever a package gets ignored or fails which in turns cascades into a number of ports get- ting removed from the build queue due to the dependency's failure. This may fire before the build starts due to ig- nored prescanning, or any time during the build. In addi- tion to the common definitions, it defines RESULT=skipped, ORIGIN, and PKGNAME in the environment. SEE ALSO pkg(8), pkg-repo(8) NOTES Graceful exit During development, the interrupt signal (SIGINT) was captured and handled. It turns out this handling extended to the builders, so any event resulting in a SIGINT in the build would lock up the builder permanently. There's no apparent workaround for this given the design, so pressing Control-C is no longer handled. It will stop the build, but it will leave everything in bad state. The next significant call to synth will attempt to clean that up though. If a user wants to stop the build, the best approach is to type the Control-Q combination. The program will shut down as soon as it can. During the building phase no new builds will start after the shutdown begins, but the active builders will continue until completion. Unprivileged users At this time, synth can only be run by the root user. Impulse indicator A common question is "What does the Impulse indicator on the display mean?" It is the package build rate (packages/hour) over the last 500 seconds. For the first roughly ten minutes of a bulk run, the "Impulse" rate and the "Pkg/Hour" rate will be identical. After that, the "Impulse" build rate will differ from the rate averaged over the entire elapsed time, often by a great amount. It's just a way to gauge the package production rate over the previous few minutes. Supported architectures Synth is written in Ada, and thus an Ada compiler or cross-compiler is re- quired to build it. For FreeBSD, that means synth is currently limited to amd64 and i386. It is feasible to port GNAT to the ARMv7 architecture as this has already been done (See lang/gnatdroid-armv7 port) but it's not a trivial task to do it. Other architectures would be challenging, although not impossible, to support. EXAMPLE To activate the signing of default profile's repository with a local RSA key, generate the two key files: % cd /usr/local/etc/synth % openssl genrsa -out LiveSystem-private.key 2048 % chmod 0400 LiveSystem-private.key % openssl rsa -in LiveSystem-private.key -out \ LiveSystem-public.key -pubout To activate the signing of default profile's repository with an external signing server, two files need to be created. It's assumed the signing server has already been configured as described by the pkg-repo(8) man page, and that the keys are located in the repo.pub and repo.key files. On the signing server which is 192.168.0.44 in this example: % sha256 -q repo.pub > LiveSystem-fingerprint % echo "/usr/bin/ssh 192.168.0.44 /usr/local/bin/sign.sh" \ > LiveSystem-signing_command Then move both files to /usr/local/etc/synth directory on the build server. The build server root user needs to have a passphrase-free SSH key access to the signing server in order for Synth to sign the repository. HISTORY synth was conceived long before coding began in December of 2015. It was designed to be system-agnostic from the beginning, and in particular sup- port for FreeBSD and DragonFly was added simulaneously. It was hoped that it would attract users of the older PortMaster and PortUpgrade ports man- agement tools by providing them with a superior and well-maintained alter- native. synth was written entirely by John Marino <marino@FreeBSD.org> and released under the Internet Software Consortium license. FreeBSD 13.2 January 27, 2021 SYNTH(1)
NAME | SYNOPSIS | DESCRIPTION | ENVIRONMENT | FILES | SEE ALSO | NOTES | EXAMPLE | HISTORY
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=synth&sektion=1&manpath=FreeBSD+13.2-RELEASE+and+Ports>