static-php-cli/docs/en/develop/source-module.md

Source module

The download source module of static-php-cli is a major module. It includes dependent libraries, external extensions, PHP source code download methods and file decompression methods. The download configuration file mainly involves the source.json and pkg.json file, which records the download method of all downloadable sources.

The main commands involved in the download function are bin/spc download and bin/spc extract. The download command is a downloader that downloads sources according to the configuration file, and the extract command is an extractor that extract sources from downloaded files.

Generally speaking, downloading sources may be slow because these sources come from various official websites, GitHub, and other different locations. At the same time, they also occupy a large space, so you can download the sources once and reuse them.

The configuration file of the downloader is source.json, which contains the download methods of all sources. You can add the source download methods you need, or modify the existing source download methods.

The download configuration structure of each source is as follows. The following is the source download configuration corresponding to the libevent extension:

{
  "libevent": {
    "type": "ghrel",
    "repo": "libevent/libevent",
    "match": "libevent.+\\.tar\\.gz",
    "provide-pre-built": true,
      "license": {
      "type": "file",
      "path": "LICENSE"
    }
  }
}

The most important field here is type. Currently, the types it supports are:

• url: Directly use URL to download, for example: https://download.libsodium.org/libsodium/releases/libsodium-1.0.18.tar.gz.
• ghrel: Use the GitHub Release API to download, download the artifacts uploaded from the latest version released by maintainers.
• ghtar: Use the GitHub Release API to download. Different from ghrel, ghtar is downloaded from the source code (tar.gz) in the latest Release of the project.
• ghtagtar: Use GitHub Release API to download. Compared with ghtar, ghtagtar can find the latest one from the tags list and download the source code in tar.gz format (because some projects only use tag release version).
• bitbuckettag: Download using BitBucket API, basically the same as ghtagtar, except this one applies to BitBucket.
• git: Clone the project directly from a Git address to download sources, applicable to any public Git repository.
• filelist: Use a crawler to crawl the Web download site that provides file index, and get the latest version of the file name and download it.
• custom: If none of the above download methods are satisfactory, you can write custom, create a new class under src/SPC/store/source/, extends CustomSourceBase, and write the download script yourself.

source.json Common parameters

Each source file in source.json has the following params:

• license: the open source license of the source code, see Open Source License section below
• type: must be one of the types mentioned above
• path (optional): release the source code to the specified directory instead of source/{name}
• provide-pre-built (optional): whether to provide precompiled binary files. If true, it will automatically try to download precompiled binary files when running bin/spc download

::: tip The path parameter in source.json can specify a relative or absolute path. When specified as a relative path, the path is based on source/. :::

Download type - url

URL type sources refer to downloading files directly from the URL.

The parameters included are:

• url: The download address of the file, such as https://example.com/file.tgz
• filename (optional): The file name saved to the local area. If not specified, the file name of the url will be used.

Example (download the imagick extension and extract it to the extension storage path of the php source code):

{
   "ext-imagick": {
     "type": "url",
     "url": "https://pecl.php.net/get/imagick",
     "path": "php-src/ext/imagick",
     "filename": "imagick.tgz",
     "license": {
       "type": "file",
       "path": "LICENSE"
     }
   }
}

Download type - ghrel

ghrel will download files from Assets uploaded in GitHub Release. First use the GitHub Release API to get the latest version, and then download the corresponding files according to the regular matching method.

The parameters included are:

• repo: GitHub repository name
• match: regular expression matching Assets files
• prefer-stable: Whether to download stable versions first (default is false)

Example (download the libsodium library, matching the libsodium-x.y.tar.gz file in Release):

{
   "libsodium": {
     "type": "ghrel",
     "repo": "jedisct1/libsodium",
     "match": "libsodium-\\d+(\\.\\d+)*\\.tar\\.gz",
     "license": {
       "type": "file",
       "path": "LICENSE"
     }
   }
}

Download type - ghtar

ghtar will download the file from the GitHub Release Tag. Unlike ghrel, ghtar will download the source code (tar.gz) from the latest Release of the project.

The parameters included are:

• repo: GitHub repository name
• prefer-stable: Whether to download stable versions first (default is false)

Example (brotli library):

{
   "brotli": {
     "type": "ghtar",
     "repo": "google/brotli",
     "license": {
       "type": "file",
       "path": "LICENSE"
     }
   }
}

Download type - ghtagtar

Use the GitHub Release API to download. Compared with ghtar, ghtagtar can find the latest one from the tags list and download the source code in tar.gz format (because some projects only use the tag version).

The parameters included are:

• repo: GitHub repository name
• prefer-stable: Whether to download stable versions first (default is false)

Example (gmp library):

{
   "gmp": {
     "type": "ghtagtar",
     "repo": "alisw/GMP",
     "license": {
       "type": "text",
       "text": "EXAMPLE LICENSE"
     }
   }
}

Download Type - bitbuckettag

Download using BitBucket API, basically the same as ghtagtar, except this one works with BitBucket.

The parameters included are:

• repo: BitBucket repository name

Download type - git

Clone the project directly from a Git address to download sources, applicable to any public Git repository.

The parameters included are:

• url: Git link (HTTPS only)

• rev: branch name

  {
  "imap": {
   "type": "git",
   "url": "https://github.com/static-php/imap.git",
   "rev": "master",
   "license": {
     "type": "file",
     "path": "LICENSE"
   }
  }
  }
  

Download type - filelist

Use a crawler to crawl a web download site that provides a file index and get the latest version of the file name and download it.

Note that this method is only applicable to static sites with page index functions such as mirror sites and GNU official websites.

The parameters included are:

• url: The URL of the page to crawl the latest version of the file
• regex: regular expression matching file names and download links

Example (download the libiconv library from the GNU official website):

{
   "libiconv": {
     "type": "filelist",
     "url": "https://ftp.gnu.org/gnu/libiconv/",
     "regex": "/href=\"(?<file>libiconv-(?<version>[^\"]+)\\.tar\\.gz)\"/",
     "license": {
       "type": "file",
       "path": "COPYING"
     }
   }
}

Download type - custom

If the above downloading methods are not satisfactory, you can write custom, create a new class under src/SPC/store/source/, extends CustomSourceBase, and write the download script yourself.

I won’t go into details here, you can look at src/SPC/store/source/PhpSource.php or src/SPC/store/source/PostgreSQLSource.php as examples.

pkg.json General parameters

pkg.json stores non-source-code files, such as precompiled tools musl-toolchain and UPX. It includes:

• type: The same type as source.json and different kinds of parameters.
• extract (optional): The path to decompress after downloading, the default is pkgroot/{pkg_name}.
• extract-files (optional): Extract only the specified files to the specified location after downloading.

It should be noted that pkg.json does not involve compilation, modification and distribution of source code, so there is no license open source license field. And you cannot use the extract and extract-files parameters at the same time.

Example (download nasm locally and extract only program files to PHP SDK):

{
   "nasm-x86_64-win": {
     "type": "url",
     "url": "https://www.nasm.us/pub/nasm/releasebuilds/2.16.01/win64/nasm-2.16.01-win64.zip",
     "extract-files": {
       "nasm-2.16.01/nasm.exe": "{php_sdk_path}/bin/nasm.exe",
       "nasm-2.16.01/ndisasm.exe": "{php_sdk_path}/bin/ndisasm.exe"
     }
   }
}

The key name in extract-files is the file in the source folder, and the key value is the storage path. The storage path can use the following variables:

• {php_sdk_path}: (Windows only) PHP SDK path
• {pkg_root_path}: pkgroot/
• {working_dir}: current working directory
• {download_path}: download directory
• {source_path}: source code decompression directory

When extract-files does not use variables and is a relative path, the directory of the relative path is {working_dir}.

Open source license

For source.json, each source file should contain an open source license. The license field stores the open source license information.

Each license contains the following parameters:

• type: file or text
• path: the license file in the source code directory (required when type is file)
• text: License text (required when type is text)

Example (yaml extension source code with LICENSE file):

{
   "yaml": {
     "type": "git",
     "path": "php-src/ext/yaml",
     "rev": "php7",
     "url": "https://github.com/php/pecl-file_formats-yaml",
     "license": {
       "type": "file",
       "path": "LICENSE"
     }
   }
}

When an open source project has multiple licenses, multiple files can be specified:

{
   "libuv": {
     "type": "ghtar",
     "repo": "libuv/libuv",
     "license": [
       {
         "type": "file",
         "path": "LICENSE"
       },
       {
         "type": "file",
         "path": "LICENSE-extra"
       }
     ]
   }
}

When the license of an open source project uses different files between versions, path can be used as an array to list the possible license files:

{
  "redis": {
    "type": "git",
    "path": "php-src/ext/redis",
    "rev": "release/6.0.2",
    "url": "https://github.com/phpredis/phpredis",
    "license": {
      "type": "file",
      "path": [
        "LICENSE",
        "COPYING"
      ]
    }
  }
}