netdata/packaging: Binary distributions - clean up .DEB package generation process (#6465)

* netdata/packaging: [ci skip] reinstate dev branch to help testing

* netdata/packaging: [ci skip] Yes, make the right package choice on json-c for debian based distros..

* netdata/packaging: [ci skip] two more packages that differentiate on debian/ubuntu...............

* netdata/packaging: [ci skip] Add more package dependencies around building

* netdata/packaging: [ci skip] Introduce changelog generation technique for .DEB

1) Remove contrib/debian/changelog from gitignore, we want it in
2) Add variables to monitor latest release version and latest release date from git
3) Add a templated contrib/debian/changelog, that will be the starting point for our changelog generation.
   Either on stable release or nightly releases, we will be generating the logs since the previous release, to manage the content.
   Note: We might eventually take the effort to produce one large changelog and append each time the extra stuff
4) Add the steps to prepare the source and the changelog for building and execute the build

At this stage things should still be broken, but we should have come closer to a ready-to-build-the-package environment per distro

* netdata/packaging: [ci skip] Extract to specified directory, otherwise we dump it on a privileged folder using an unprivileged user (builder)

Also, a fix on sed command, missed the s keyword on the string
replacement

* netdata/packaging: [ci skip] Should be in-place replacement, use -i instead of -e

* netdata/packaging: [ci skip] wrong parameter usage

* netdata/packaging: [ci skip] factor out the two build commands to run as a single bundle. refactor trigger deb build script respectively

* netdata/packaging: [ci skip] Be sure to copy over the build script to the build path

* netdata/packaging: [ci skip] OCD - unnecessary s added

* netdata/packaging: [ci skip] Changelog generation should take place on the repo side, then copy it over to the destination (If that doesnt work well, will just provide a static link for changelog

* netdata/packaging: [ci skip] missed the ignore branch option
	Also, a nit on the gbp command

* netdata/packaging: [ci skip] link to the original path, then create symlink on build folder
                             Also, fix --since option

* netdata/packaging: [ci skip] Run replacement of changelog fields on the host, not on the container

* netdata/packaging: [ci skip] fix contrib path, you are no longer reading from container path

* netdata/packaging: [ci skip] fix version

* netdata/packaging: [ci skip] add dep

* netdata/packaging: [ci skip] fix changelog template

* netdata/packaging: [ci skip] Fix debian package build, we were looping through forever. Exclude contrib all together

* netdata/packaging: [ci skip] Add logic to gather all package related content from the container folder and push it for publishing

* netdata/packaging: [ci skip] fixes and nits
1) use the branch for the beta deployment
2) change the cp pattern

* netdata/packaging:[ci skip] add more debug info

* netdata/packaging: [ci skip] Adjust LXC directory permissions, so that the rest of the workflow can pull through

* netdata/packaging: [ci skip] Using expressions within double quotes doesnt work, use quotes only on the variable to avoid syntax warnings from shellchecking

* netdata/packaging: [ci skip] Update build deps

* netdata/packaging: Add runtime deps (hopefully)

* netdata/packaging: [ci skip] remove old distro

* netdata/packaging: [ci skip] dont let that space there, breaks the syntax

* netdata/packaging: [ci skip] missed one

* netdata/packaging: [ci skip] yank yank yank

* netdata/packaging: [ci skip] You missed www over there

* netdata/packaging: [ci skip] Remove broken distros (no container images). We will revisit the way to support multiple distros in a different way, lxc failed us

* netdata/packaging: Pull in go.d plugin, also fix chmod command to just pass the folder since with the -R doesnt require anything else

* netdata/packaging: [ci skip] Get cups, freeipmi and nfacct on

* netdata/packaging: [ci skip] pull different package in

* netdata/packaging: [ci skip] Install dependencies on the build machine

* netdata/packaging: [ci skip] dont do post-clean, not on all versions

* netdata/packaging: [ci skip] move go install seperately

* netdata/packaging: [ci skip] Towards a more appropriate go.d plugin install

* netdata/packaging: [ci skip] fix permissions setting

* netdata/packaging: [ci skip] fix freeipmi package install

* netdata/packaging: [ci skip] Fix build arch setting on the build command, also remove pre-clean, it breaks on debian/jessie

* netdata/packaging: [ci skip] fix postinst actions, we use libexec

* netdata/packaging: adjust www and go.d permissions

* netdata/packaging: [ci skip] Handle the different build tool version cases, also take care some dependency issues on older version jessie

* netdata/packaging: [ci skip] adjust the control file based on the distro

* netdata/packaging: [ci skip] Move control file preparation prior to packaging the source

* netdata/packaging: [ci skip] Adjust versioning scheme for the nightlies

* netdata/packaging: [ci skip] Let RPM be built too for testing

* netdata/packaging: [ci skip] Factor out version calculation

* netdata/packaging: [ci skip] Attempt to modify version properly

* netdata/packaging: [ci skip] Update documentationg regarding .DEB files

* netdata/packaging: [ci skip] autogen not available everywhere, also fix json-c deps

* netdata/packaging: Remove development settings, also remove beta deployment as its not needed

.gitignore

@@ -139,8 +139,6 @@ cmake_install.cmake
 .DS_Store
 webcopylocal*
 
-contrib/debian/changelog
-
 # converted diagrams
 diagrams/*.png
 diagrams/*.svg

.travis.yml

@@ -24,6 +24,8 @@ install:
 - export NOTIF_CHANNEL="automation-beta"
 - if [ "${TRAVIS_REPO_SLUG}" = "netdata/netdata" ]; then export NOTIF_CHANNEL="automation"; fi;
 - export BUILD_VERSION="$(cat packaging/version | cut -d'-' -f1)"
+- export LATEST_RELEASE_VERSION="$(cat packaging/version | cut -d'-' -f1)"
+- export LATEST_RELEASE_DATE="$(git log -1 --format=%aD "${LATEST_RELEASE_VERSION}" | cat)"
 - if [[ "${TRAVIS_COMMIT_MESSAGE}" = *"[Build latest]"* ]]; then export BUILD_VERSION="$(cat packaging/version | cut -d'-' -f1,2 | sed -e 's/-/./g').latest"; fi;
 - export DEPLOY_REPO="netdata" # Default production packaging repository
 - if [[ "${TRAVIS_COMMIT_MESSAGE}" = *"[Build latest]"* ]]; then export DEPLOY_REPO="netdata-edge"; fi;
@@ -69,8 +71,6 @@ stages:
   if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package amd64 DEB Ubuntu\]|\[Package amd64 DEB\])/
 - name: "Package ubuntu/bionic"
   if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package amd64 DEB Ubuntu\]|\[Package amd64 DEB\])/
-- name: "Package ubuntu/artful"
-  if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package amd64 DEB Ubuntu\]|\[Package amd64 DEB\])/
 
   # Debian
 - name: "Package debian/buster"
@@ -79,8 +79,6 @@ stages:
   if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package amd64 DEB Debian\]|\[Package amd64 DEB\])/
 - name: "Package debian/jessie"
   if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package amd64 DEB Debian\]|\[Package amd64 DEB\])/
-- name: "Package debian/wheezy"
-  if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package amd64 DEB Debian\]|\[Package amd64 DEB\])/
 
   # Build RPM packages under special conditions
   # Enterprise linux (Covers CentOS, Redhat, Amazon linux)
@@ -129,7 +127,6 @@ stages:
     before_deploy:
     - .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale RPM found"
     deploy:
-      # Beta packages deployment
       - provider: packagecloud
         repository: "${DEPLOY_REPO}"
         username: "${PACKAGING_USER}"
@@ -142,19 +139,6 @@ stages:
           repo: ${TRAVIS_REPO_SLUG}
           branch: "master"
           condition: -d "${PACKAGES_DIRECTORY}"
-      # Production release packages deployment
-      - provider: packagecloud
-        repository: "netdata"
-        username: "netdata"
-        token: "${PKG_CLOUD_TOKEN}"
-        dist: "${BUILD_STRING}"
-        local_dir: "${PACKAGES_DIRECTORY}"
-        skip_cleanup: true
-        on:
-          # Only deploy on ${USER}/netdata, master branch, when packages directory is created
-          repo: "netdata/netdata"
-          branch: "master"
-          condition: -d "${PACKAGES_DIRECTORY}"
     after_deploy:
       - if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;
       - if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;
@@ -165,7 +149,7 @@ stages:
 - stage: &_DEB_TEMPLATE
     name: "Build & Publish DEB package"
     before_install:
-    - sudo apt-get install -y wget lxc lxc-templates
+    - sudo apt-get install -y wget lxc lxc-templates dh-make git-buildpackage build-essential libdistro-info-perl
     - source tests/installer/slack.sh
     before_script:
     - post_message "TRAVIS_MESSAGE" "Starting package preparation and publishing for ${BUILD_STRING}.${BUILD_ARCH}" "${NOTIF_CHANNEL}"
@@ -177,7 +161,8 @@ stages:
     - echo "packaging/version:" && cat packaging/version
     - echo "Creating LXC environment for the build" && sudo -E .travis/package_management/create_lxc_for_build.sh
     - echo "Building package in container" && sudo -E .travis/package_management/build_package_in_container.sh
-    - sudo chmod -R 755 "/var/lib/lxc"
+    - sudo chown -R root:travis "/var/lib/lxc"
+    - sudo chmod -R 750 "/var/lib/lxc"
     - echo "Preparing DEB packaging contents for upload" && sudo -E .travis/package_management/prepare_packages.sh
     git:
       depth: false
@@ -185,7 +170,6 @@ stages:
     before_deploy:
     - .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale DEB found"
     deploy:
-      # Beta packages deployment
       - provider: packagecloud
         repository: "${DEPLOY_REPO}"
         username: "${PACKAGING_USER}"
@@ -198,19 +182,6 @@ stages:
           repo: ${TRAVIS_REPO_SLUG}
           branch: "master"
           condition: -d "${PACKAGES_DIRECTORY}"
-      # Production release packages deployment
-      - provider: packagecloud
-        repository: "netdata"
-        username: "netdata"
-        token: "${PKG_CLOUD_TOKEN}"
-        dist: "${BUILD_STRING}"
-        local_dir: "${PACKAGES_DIRECTORY}"
-        skip_cleanup: true
-        on:
-          # Only deploy on ${USER}/netdata, master branch, when build-area directory is created
-          repo: "netdata/netdata"
-          branch: "master"
-          condition: -d "${PACKAGES_DIRECTORY}"
     after_deploy:
       - if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;
       - if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;
@@ -422,16 +393,6 @@ jobs:
 
 
 
-  - stage:
-    <<: *_DEB_TEMPLATE
-    stage: "Package ubuntu/artful"
-    env:
-    - BUILDER_NAME="builder" BUILD_DISTRO="ubuntu" BUILD_RELEASE="artful" BUILD_STRING="ubuntu/artful"
-    - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
-    - ALLOW_SOFT_FAILURE_HERE=true
-
-
-
   # Debian distros build
   - stage:
     <<: *_DEB_TEMPLATE
@@ -463,16 +424,6 @@ jobs:
 
 
 
-  - stage:
-    <<: *_DEB_TEMPLATE
-    stage: "Package debian/wheezy"
-    env:
-    - BUILDER_NAME="builder" BUILD_DISTRO="debian" BUILD_RELEASE="wheezy" BUILD_STRING="debian/wheezy"
-    - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
-    - ALLOW_SOFT_FAILURE_HERE=true
-
-
-
   # Enterprise linux builds (Centos, Redhat, Amazon linux (el/6))
   #
   - stage:

.travis/package_management/build.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+UNPACKAGED_NETDATA_PATH="$1"
+LATEST_RELEASE_VERSION="$2"
+
+if [ -z "${LATEST_RELEASE_VERSION}" ]; then
+	echo "Parameter 'LATEST_RELEASE_VERSION' not defined"
+	exit 1
+fi
+
+if [ -z "${UNPACKAGED_NETDATA_PATH}" ]; then
+	echo "Parameter 'UNPACKAGED_NETDATA_PATH' not defined"
+	exit 1
+fi
+
+echo "Running changelog generation mechanism since ${LATEST_RELEASE_VERSION}"
+
+echo "Entering ${UNPACKAGED_NETDATA_PATH}"
+cd "${UNPACKAGED_NETDATA_PATH}"
+
+echo "Linking debian -> contrib/debian"
+ln -sf contrib/debian debian
+
+echo "Executing dpkg-buildpackage"
+if dpkg-buildpackage --version 2> /dev/null | grep -q "1.18"; then
+	dpkg-buildpackage --post-clean --pre-clean --build=binary
+else
+	dpkg-buildpackage -b
+fi
+
+echo "DEB build script completed!"

.travis/package_management/common.py

@@ -8,6 +8,21 @@ import lxc
 import subprocess
 import os
 
+def fetch_version(orig_build_version):
+    tag = None
+    friendly_version = ""
+
+    # TODO: Checksum validations
+    if str(orig_build_version).count(".latest") == 1:
+        version_list=str(orig_build_version).replace('v', '').split('.')
+        friendly_version='.'.join(version_list[0:2]) + "." + version_list[3]
+    else:
+        friendly_version = orig_build_version.replace('v', '')
+        tag = friendly_version # Go to stable tag
+    print("Version set to %s from %s" % (friendly_version, orig_build_version))
+
+    return friendly_version, tag
+
 def replace_tag(tag_name, spec, new_tag_content):
     print("Fixing tag %s in %s" % (tag_name, spec))
 
@@ -51,24 +66,36 @@ def install_common_dependendencies(container):
         run_command(container, [os.environ["REPO_TOOL"], "clean", "-a"])
         run_command(container, [os.environ["REPO_TOOL"], "--no-gpg-checks", "update", "-y"])
         run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-glib-devel"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
 
     elif str(os.environ["REPO_TOOL"]).count("yum") == 1:
-        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])
         run_command(container, [os.environ["REPO_TOOL"], "clean", "all"])
         run_command(container, [os.environ["REPO_TOOL"], "update", "-y"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
+
         if os.environ["BUILD_STRING"].count("el/7") == 1 and os.environ["BUILD_ARCH"].count("i386") == 1:
             print ("Skipping epel-release install for %s-%s" % (os.environ["BUILD_STRING"], os.environ["BUILD_ARCH"]))
         else:
             run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "epel-release"])
+    elif str(os.environ["REPO_TOOL"]).count("apt-get") == 1:
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libipmimonitoring-dev"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libjson-c-dev"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libcups2-dev"])
     else:
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
         run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])
         run_command(container, [os.environ["REPO_TOOL"], "update", "-y"])
 
+    if os.environ["BUILD_STRING"].count("el/6") < 0:
+        run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "autogen"])
+
     run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "sudo"])
     run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "wget"])
     run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "bash"])
-    run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
-    run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
 
 def prepare_version_source(dest_archive, pkg_friendly_version, tag=None):
     print(".0 Preparing local implementation tarball for version %s" % pkg_friendly_version)

.travis/package_management/configure_deb_lxc_environment.py

@@ -39,6 +39,8 @@ print("Waiting for container connectivity to start configuration sequence")
 if not container.get_ips(timeout=30):
     raise Exception("Timeout while waiting for container")
 
+build_path = "/home/%s" % os.environ['BUILDER_NAME']
+
 # Run the required activities now
 # 1. Create the builder user
 print("1. Adding user %s" % os.environ['BUILDER_NAME'])
@@ -48,28 +50,43 @@ common.run_command(container, ["useradd", "-m", os.environ['BUILDER_NAME']])
 print("2. Installing package dependencies within LXC container")
 common.install_common_dependendencies(container)
 
+print("2.1 Install .DEB build support packages")
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dpkg-dev"])
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libdistro-info-perl"])
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-make"])
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-systemd"])
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-autoreconf"])
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "git-buildpackage"])
+
+print("2.2 Add more dependencies")
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libnetfilter-acct-dev"])
+common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libcups2-dev"])
+
 print ("3. Run install-required-packages scriptlet")
-common.run_command(container, ["wget", "-T", "15", "-O", "/home/%s/.install-required-packages.sh" % (os.environ['BUILDER_NAME']), "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])
-common.run_command(container, ["bash", "/home/%s/.install-required-packages.sh" % (os.environ['BUILDER_NAME']), "netdata", "--dont-wait", "--non-interactive"])
+common.run_command(container, ["wget", "-T", "15", "-O", "%s/.install-required-packages.sh" % build_path, "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])
+common.run_command(container, ["bash", "%s/.install-required-packages.sh" % build_path, "netdata", "--dont-wait", "--non-interactive"])
 
 friendly_version=""
 dest_archive=""
 download_url=""
 tag = None
-
-# TODO: Checksum validations
-if str(os.environ['BUILD_VERSION']).count(".latest") == 1:
-    version_list=str(os.environ['BUILD_VERSION']).replace('v', '').split('.')
-    friendly_version='.'.join(version_list[0:3]) + "." + version_list[3]
-else:
-    friendly_version = os.environ['BUILD_VERSION'].replace('v', '')
-    tag = friendly_version # Go to stable tag
+friendly_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])
 
 tar_file="%s/netdata-%s.tar.gz" % (os.path.dirname(dest_archive), friendly_version)
 
 print("5. I will be building version '%s' of netdata." % os.environ['BUILD_VERSION'])
-dest_archive="/home/%s/netdata-%s.tar.gz" % (os.environ['BUILDER_NAME'], friendly_version)
+dest_archive="%s/netdata-%s.tar.gz" % (build_path, friendly_version)
+
+if str(os.environ["BUILD_STRING"]).count("debian/jessie") == 1:
+    print("5.1 We are building for Jessie, adjusting control file")
+    common.run_command_in_host(['sudo', 'rm', 'contrib/debian/control'])
+    common.run_command_in_host(['sudo', 'cp', 'contrib/debian/control.jessie', 'contrib/debian/control'])
 
 common.prepare_version_source(dest_archive, friendly_version, tag=tag)
 
+print("6. Installing build.sh script to build path")
+common.run_command_in_host(['sudo', 'cp', '.travis/package_management/build.sh', "%s/%s/build.sh" % (os.environ['LXC_CONTAINER_ROOT'], build_path)])
+common.run_command_in_host(['sudo', 'chmod', '777', "%s/%s/build.sh" % (os.environ['LXC_CONTAINER_ROOT'], build_path)])
+common.run_command_in_host(['sudo', 'ln', '-sf', 'contrib/debian', 'debian'])
+
 print("Done!")

.travis/package_management/configure_rpm_lxc_environment.py

@@ -79,15 +79,7 @@ dest_archive=""
 download_url=""
 spec_file="/home/%s/rpmbuild/SPECS/netdata.spec" % os.environ['BUILDER_NAME']
 tag = None
-
-# TODO: Checksum validations
-if str(os.environ['BUILD_VERSION']).count(".latest") == 1:
-    version_list=str(os.environ['BUILD_VERSION']).replace('v', '').split('.')
-    rpm_friendly_version='.'.join(version_list[0:3]) + "." + version_list[3]
-else:
-    rpm_friendly_version = os.environ['BUILD_VERSION'].replace('v', '')
-    tag = rpm_friendly_version # Go to stable tag
-
+rpm_friendly_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])
 tar_file="%s/netdata-%s.tar.gz" % (os.path.dirname(dest_archive), rpm_friendly_version)
 
 print("5. I will be building version '%s' of netdata." % os.environ['BUILD_VERSION'])

.travis/package_management/prepare_packages.sh

@@ -27,29 +27,36 @@ for d in ${CREATED_CONTAINERS[@]}; do
 
 	# Pick up any RPMS from builder
 	RPM_BUILD_PATH="${LXC_ROOT}/${d}/rootfs/home/${BUILDER_NAME}/rpmbuild"
-	echo "Checking folder ${RPM_BUILD_PATH} for RPMS and SRPMS"
-
-	if [ -d "${RPM_BUILD_PATH}/RPMS" ]; then
-		echo "Copying any RPMS in '${RPM_BUILD_PATH}', copying over the following:"
-		ls -ltrR "${RPM_BUILD_PATH}/RPMS"
-		[[ -d "${RPM_BUILD_PATH}/RPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/x86_64/* "${PACKAGES_DIRECTORY}"
-		[[ -d "${RPM_BUILD_PATH}/RPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i386/* "${PACKAGES_DIRECTORY}"
-		[[ -d "${RPM_BUILD_PATH}/RPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i686/* "${PACKAGES_DIRECTORY}"
-	fi
-
-	if [ -d "${RPM_BUILD_PATH}/SRPMS" ]; then
-		echo "Copying any SRPMS in '${RPM_BUILD_PATH}', copying over the following:"
-		ls -ltrR "${RPM_BUILD_PATH}/SRPMS"
-		[[ -d "${RPM_BUILD_PATH}/SRPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/x86_64/* "${PACKAGES_DIRECTORY}"
-		[[ -d "${RPM_BUILD_PATH}/SRPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i386/* "${PACKAGES_DIRECTORY}"
-		[[ -d "${RPM_BUILD_PATH}/SRPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i686/* "${PACKAGES_DIRECTORY}"
+	if [ -d "${RPM_BUILD_PATH}" ]; then
+		echo "Checking folder ${RPM_BUILD_PATH} for RPMS and SRPMS"
+
+		if [ -d "${RPM_BUILD_PATH}/RPMS" ]; then
+			echo "Copying any RPMS in '${RPM_BUILD_PATH}', copying over the following:"
+			ls -ltrR "${RPM_BUILD_PATH}/RPMS"
+			[[ -d "${RPM_BUILD_PATH}/RPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/x86_64/* "${PACKAGES_DIRECTORY}"
+			[[ -d "${RPM_BUILD_PATH}/RPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i386/* "${PACKAGES_DIRECTORY}"
+			[[ -d "${RPM_BUILD_PATH}/RPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i686/* "${PACKAGES_DIRECTORY}"
+		fi
+
+		if [ -d "${RPM_BUILD_PATH}/SRPMS" ]; then
+			echo "Copying any SRPMS in '${RPM_BUILD_PATH}', copying over the following:"
+			ls -ltrR "${RPM_BUILD_PATH}/SRPMS"
+			[[ -d "${RPM_BUILD_PATH}/SRPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/x86_64/* "${PACKAGES_DIRECTORY}"
+			[[ -d "${RPM_BUILD_PATH}/SRPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i386/* "${PACKAGES_DIRECTORY}"
+			[[ -d "${RPM_BUILD_PATH}/SRPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i686/* "${PACKAGES_DIRECTORY}"
+		fi
+	else
+		DEB_BUILD_PATH="${LXC_ROOT}/${d}/rootfs/home/${BUILDER_NAME}"
+		echo "Checking folder ${DEB_BUILD_PATH} for DEB packages"
+		if [ -d "${DEB_BUILD_PATH}" ]; then
+			cp "${DEB_BUILD_PATH}"/netdata*.ddeb "${PACKAGES_DIRECTORY}" || echo "Could not copy any .ddeb files"
+			cp "${DEB_BUILD_PATH}"/netdata*.deb "${PACKAGES_DIRECTORY}" || echo "Could not copy any .deb files"
+			cp "${DEB_BUILD_PATH}"/netdata*.buildinfo "${PACKAGES_DIRECTORY}" || echo "Could not copy any .buildinfo files"
+			cp "${DEB_BUILD_PATH}"/netdata*.changes "${PACKAGES_DIRECTORY}" || echo "Could not copy any .changes files"
+		else
+			echo "Folder ${DEB_BUILD_PATH} does not exist or not a directory, nothing to do for package preparation"
+		fi
 	fi
-
-	# Pick up any DEBs from builder
-	DEB_BUILD_PATH="${d}/home/${BUILDER_NAME}/build-area"
-	echo "Checking folder ${DEB_BUILD_PATH} for DEB packages"
-	#TODO: During debian clean up we 'll fill this up
-
 done
 
 chmod -R 777 "${PACKAGES_DIRECTORY}"

.travis/package_management/trigger_deb_lxc_build.py

@@ -37,15 +37,37 @@ if not container.running or not container.state == "RUNNING":
 if not container.get_ips(timeout=30):
     raise Exception("Timeout while waiting for container")
 
+build_path = "/home/%s" % os.environ['BUILDER_NAME']
+
 print("Setting up EMAIL and DEBFULLNAME variables required by the build tools")
 os.environ["EMAIL"] = "bot@netdata.cloud"
 os.environ["DEBFULLNAME"] = "Netdata builder"
 
 # Run the build process on the container
-print("Starting DEB build process, running dh-make")
-new_version = os.environ["BUILD_VERSION"].replace('v', '')
+new_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])
+print("Starting DEB build process for version %s" % new_version)
+
+netdata_tarball = "%s/netdata-%s.tar.gz" % (build_path, new_version)
+unpacked_netdata = netdata_tarball.replace(".tar.gz", "")
+
+print("Extracting tarball %s" % netdata_tarball)
+common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "tar", "xf", netdata_tarball, "-C", build_path])
+
+print("Fixing changelog tags")
+changelog_in_host = "contrib/debian/changelog"
+common.run_command_in_host(['sed', '-i', 's/PREVIOUS_PACKAGE_VERSION/%s-1/g' % os.environ["LATEST_RELEASE_VERSION"].replace("v", ""), changelog_in_host])
+common.run_command_in_host(['sed', '-i', 's/PREVIOUS_PACKAGE_DATE/%s/g' % os.environ["LATEST_RELEASE_DATE"], changelog_in_host])
+
+print("Executing gbp dch command..")
+common.run_command_in_host(['gbp', 'dch', '--release', '--ignore-branch', '--spawn-editor=snapshot', '--since=%s' % os.environ["LATEST_RELEASE_VERSION"], '--new-version=%s' % new_version])
+
+print("Copying over changelog to the destination machine")
+common.run_command_in_host(['sudo', 'cp', 'debian/changelog', "%s/%s/netdata-%s/contrib/debian/" % (os.environ['LXC_CONTAINER_ROOT'], build_path, new_version)])
+
+print("Running debian build script since %s" % os.environ["LATEST_RELEASE_VERSION"])
+common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "%s/build.sh" % build_path, unpacked_netdata, new_version])
 
-print("Building the package")
-common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "dpkg-buildpackage", "--host-arch", "amd64", "--target-arch", "amd64", "--post-clean", "--pre-clean", "--build=binary", "--release-by=\"Netdata Builder\"", "--build-by=\"Netdata Builder\""])
+print("Listing contents on build path")
+common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "ls", "-ltr", build_path])
 
 print('Done!')

.travis/package_management/yank_stale_rpm.sh

@@ -21,7 +21,7 @@ fi
 
 PACKAGES_DIR="$1"
 DISTRO="$2"
-PACKAGES_LIST="$(ls -AR "${PACKAGES_DIR}" | grep '\.rpm')"
+PACKAGES_LIST="$(ls -AR "${PACKAGES_DIR}" | grep -e '\.rpm' -e '\.deb' -e '\.ddeb' )"
 
 if [ ! -d "${PACKAGES_DIR}" ] || [ -z "${PACKAGES_LIST}" ]; then
 	echo "Folder ${PACKAGES_DIR} does not seem to be a valid directory or is empty. No packages to check for yanking"

contrib/debian/changelog

@@ -0,0 +1,5 @@
+netdata (PREVIOUS_PACKAGE_VERSION) unstable; urgency=medium
+
+  * Initial Release
+
+-- Netdata Builder <bot@netdata.cloud> PREVIOUS_PACKAGE_DATE