Browse Source

Pipeline redesign master (#5885)

* netdata/packaging/ci: [WIP] Redesign packaging release

* netdata/packaging/ci: Add explicitly os definition, also remove sourcing of slack, its already done and rename before_install -> install

* netdata/packaging/ci: fix stray stage name

* netdata/packaging/ci: style

* netdata/packaging/ci: fix syntax for yaml

* netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)

* Revert "netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)"

This reverts commit ad5556a3ccf6c11ebc9026942af8013d4d0ea666.
wow travis - this is weird

* netdata/packaging/ci: a few more changes

* netdata/packaging/ci: This is totally experimental, will adjust after i test it

* netdata/packaging/ci: Reinstate original design - the problem i am trying to solve is more travis-related and fundamental on the way travis works

* netdata/packaging/ci: i wonder..

* Revert "netdata/packaging/ci: i wonder.."

This reverts commit a160df4fcac866375e8319bf200f47263b2bcaa3.

* netdata/packaging/ci: Fix broken items

* netdata/packaging/ci: docker images are built and published independently. As long as tagging has taken place earlier, during packaging

* netdata/packaging/ci: Following hints from https://github.com/travis-ci/travis-ci/issues/4681 and the mayhem triggered from this change
So we adding the compiler, to see if it will get travis to build the right matrix.
then linter broke on addons for some weird reason, so remove addons->apt->update too

* netdata/packaging/ci: Now do the rest of the changes, since with addition of compiler, things got worse

* netdata/packaging/ci: Ok, as expected the workaround from them did not work. Lets get creative though and assume they exclude what they added, so lets do that for os

* netdata/packaging/ci: test slack

* netdata/packaging/ci: Bring the modified slack script on this branch

* netdata/packaging/ci: Add a few more slack messages

* Fix empty stage occurrence

* netdata/packaging/ci: Another set of restructure changes, explained below

1) Publish release -> Publish for release (minor rename for clarity)
2) Add some more work on the make dist testing (temp folder for root and for now just validate the existence of the tarball)
3) Add some more slack notifications (that is still beta, will have to properly refine once we are done with changes of the flow)
4) Restructure the release process, finally
   a) relase.sh becomes draft_release.sh and changelog_and_tag_release.sh, basically splitting up the steps into meaningful independent blocks
   b) Adjust the pipeline, now we run 'changelog_and_tag_release.sh' to get the latest changelog and tag the release,
      then we run artifacts submission and draft release generation on github
   still more things to re-check on this step, it will become clearer once i get it up to speed to run on my local repos end to end

* netdata/packaging/ci: Fix wrong path usage

* netdata/packaging/ci: Update webhook URL

* netdata/packaging/ci: Update stage config as proposed by the fix from travis team

* netdata/packaging/ci: Another round of restructuring, after 1-1 comparison with old version

Briefly:
1) Publish for release stage will only run when there is a TAG defined (To be tested though, but thats the idea).
   The code previously had the releaser.sh script that did exactly that, if tagger script failed, it would stop process.
   Now we have distributed the processing in clear chunks and we pick up the process based on shared info that travis or the environment provides

2) Rename <<Create release (only on special commit msg)>> to a more appropriate name, we are not really creating release there its more complicated

3) Move the labeler execution to the packaging step, since publishing now not happening all the time. Previously we hacked this by exit 0 with the tagger
   and allow the labeler (which was the second part of the execution) to run.

4) before_install in the publishing stage is wrong. We only need this as before_deploy before deploy execution. That is where we use it, really.

5) Rename changelog_and_tag_release.sh to generate_changelog_and_tag_release.sh, just to be clearer. No problem with slightly longer file names

6) Do not let generate_changelog_and_tag_release.sh execute if GIT_TAG is not defined, just exit quietly with a message and a success code

7) Remove the tagger from the generate_changelog_and_tag_release.sh, its confusing to shove it in there. Pull it out and let it run on a clear step from the pipeline but within the same machine to make sure it shares environment with generate_changelog_and_tag_release.sh (So it has to be within the same job)

This is it mostly, keep in mind that the nightly and release parts are not tested yet.
After this commit, there will be more code that will allow me to execute all the flows on my pipeline and fully test and adjust the workflows as needed

* netdata/packaging/ci: Add more notifications, adjust the existing ones too

* netdata/packaging/ci: Add safeguards for beta, do a rename too

1) rename generate_changelog.sh -> generate_changelog_for_nightlies.sh, just to provide clarity on the usage.
   Note: we will simplify this later, this was a workaround that raised a bug on release after refactoring nightlies.
   We will have to clean this up properly once we merge the new pipeline flow

2) Add checks based on TRAVIS_REPO_SLUG to identify when we running on personal repo and abort mission critical actions like deploying or generating changelogs
   Note: The deployment step for now will simply fail, due to the lack of the necessary encryption keys on the environment.
   At a later iteration, we should have beta buckets on GCS and also beta repositories on Docker to properly test our publishing capability on our beta stack

* netdata/packaging/ci: Refactor README.md -- still half way there, will resume when i am done with testing

* netdata/packaging/ci: At this stage make the scripts harder to run, add checks to bypass tested execution.

* netdata/packaging/ci: Seems that adding the install tag within the stage, kind of overriding the outer install

pull out install items from packaging to the main install, no point to isolate two packages, just install all required all the time

* netdata/packaging/ci: tag obviously cant work at that step, moving it down to the jobs and i need to recheck if the syntax is applicable for travis too

* netdata/packaging/ci: So the jobs in a list dont share the environment, lets see if we do it otherwise. also remove the tag check, not work as expected

* netdata/packaging/ci: Add another way to check the RC condition

* netdata/packaging/ci: experimental - check conditions on message

* netdata/packaging/ci: Introduce an alternate master branch for the workflow testing
1. I created pipeline-redesign-master that will be the master branch for my use cases
2. Modified travis to trigger all production workflows on that branch, so that i can experiment freely with test commits to trigger process and verify execution flow

* remove spaces

* netdata/packaging/ci: comment out

* netdata/packaging/ci: wording nit

* netdata/packaging/ci: adjust the condition properly

* netdata/packaging/ci: test condition for triggering on specific keywords

* netdata/packaging/ci: try  different syntax

* netdata/packaging/ci: remove depth, causes problems as we dont get all required objects from repo (adds --depth 50 in clone)

* Missing equal sign in regex?

* netdata/packaging/ci: Update tagger script
1) More verbose messages
2) Update comments copyrights etc
3) Rename release_candidate to set_tag_release_candidate
4) Add curlies on the variables, also change the way we check for path

* netdata/packaging/ci: reinstate git depth

* netdata/packaging/ci: Remove git depth from packaging too

* netdata/packaging/ci: RTFM - was using wrong syntax for slack @here approach

* netdata/packaging/ci: Using install tag at the stage layer, overrides the global one for some reason. fix that

* netdata/packaging/ci: Dont use the pattern matching from travis, do it outside with a script

* netdata/packaging/ci: Do it the other way, script execution doesnt work there

* netdata/packaging/ci: Attempt to fix conditional on release, remove the faulty assumptions as provided by travis team

* netdata/packaging/ci: reintroduce the tag checking

* netdata/packaging/ci: you need to bring tag checking at the stage, as it is calculated during packaging step

* netdata/packaging/ci: you cant escape that one for the moment, merge them and we will rework it later again

* netdata/packaging/ci: dont forget git depth

* netdata/packaging/ci: [WIP] Redesign packaging release

* netdata/packaging/ci: Add explicitly os definition, also remove sourcing of slack, its already done and rename before_install -> install

* netdata/packaging/ci: fix stray stage name

* netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)

* Revert "netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)"


* netdata/packaging/ci: Reinstate original design - the problem i am trying to solve is more travis-related and fundamental on the way travis works

* netdata/packaging/ci: Fix broken items

* netdata/packaging/ci: docker images are built and published independently. As long as tagging has taken place earlier, during packaging

* netdata/packaging/ci: Following hints from https://github.com/travis-ci/travis-ci/issues/4681 and the mayhem triggered from this change
So we adding the compiler, to see if it will get travis to build the right matrix.
then linter broke on addons for some weird reason, so remove addons->apt->update too

* netdata/packaging/ci: Now do the rest of the changes, since with addition of compiler, things got worse

* netdata/packaging/ci: Ok, as expected the workaround from them did not work. Lets get creative though and assume they exclude what they added, so lets do that for os

* netdata/packaging/ci: test slack

* netdata/packaging/ci: Bring the modified slack script on this branch

* netdata/packaging/ci: Add a few more slack messages

* Fix empty stage occurrence

* netdata/packaging/ci: Another set of restructure changes, explained below

1) Publish release -> Publish for release (minor rename for clarity)
2) Add some more work on the make dist testing (temp folder for root and for now just validate the existence of the tarball)
3) Add some more slack notifications (that is still beta, will have to properly refine once we are done with changes of the flow)
4) Restructure the release process, finally
   a) relase.sh becomes draft_release.sh and changelog_and_tag_release.sh, basically splitting up the steps into meaningful independent blocks
   b) Adjust the pipeline, now we run 'changelog_and_tag_release.sh' to get the latest changelog and tag the release,
      then we run artifacts submission and draft release generation on github
   still more things to re-check on this step, it will become clearer once i get it up to speed to run on my local repos end to end

* netdata/packaging/ci: Fix wrong path usage

* netdata/packaging/ci: Update webhook URL

* netdata/packaging/ci: Update stage config as proposed by the fix from travis team

* netdata/packaging/ci: Another round of restructuring, after 1-1 comparison with old version

Briefly:
1) Publish for release stage will only run when there is a TAG defined (To be tested though, but thats the idea).
   The code previously had the releaser.sh script that did exactly that, if tagger script failed, it would stop process.
   Now we have distributed the processing in clear chunks and we pick up the process based on shared info that travis or the environment provides

2) Rename <<Create release (only on special commit msg)>> to a more appropriate name, we are not really creating release there its more complicated

3) Move the labeler execution to the packaging step, since publishing now not happening all the time. Previously we hacked this by exit 0 with the tagger
   and allow the labeler (which was the second part of the execution) to run.

4) before_install in the publishing stage is wrong. We only need this as before_deploy before deploy execution. That is where we use it, really.

5) Rename changelog_and_tag_release.sh to generate_changelog_and_tag_release.sh, just to be clearer. No problem with slightly longer file names

6) Do not let generate_changelog_and_tag_release.sh execute if GIT_TAG is not defined, just exit quietly with a message and a success code

7) Remove the tagger from the generate_changelog_and_tag_release.sh, its confusing to shove it in there. Pull it out and let it run on a clear step from the pipeline but within the same machine to make sure it shares environment with generate_changelog_and_tag_release.sh (So it has to be within the same job)

This is it mostly, keep in mind that the nightly and release parts are not tested yet.
After this commit, there will be more code that will allow me to execute all the flows on my pipeline and fully test and adjust the workflows as needed

* netdata/packaging/ci: Add more notifications, adjust the existing ones too

* netdata/packaging/ci: Add safeguards for beta, do a rename too

1) rename generate_changelog.sh -> generate_changelog_for_nightlies.sh, just to provide clarity on the usage.
   Note: we will simplify this later, this was a workaround that raised a bug on release after refactoring nightlies.
   We will have to clean this up properly once we merge the new pipeline flow

2) Add checks based on TRAVIS_REPO_SLUG to identify when we running on personal repo and abort mission critical actions like deploying or generating changelogs
   Note: The deployment step for now will simply fail, due to the lack of the necessary encryption keys on the environment.
   At a later iteration, we should have beta buckets on GCS and also beta repositories on Docker to properly test our publishing capability on our beta stack

* netdata/packaging/ci: Refactor README.md -- still half way there, will resume when i am done with testing

* netdata/packaging/ci: At this stage make the scripts harder to run, add checks to bypass tested execution.

* netdata/packaging/ci: Seems that adding the install tag within the stage, kind of overriding the outer install

pull out install items from packaging to the main install, no point to isolate two packages, just install all required all the time

* netdata/packaging/ci: tag obviously cant work at that step, moving it down to the jobs and i need to recheck if the syntax is applicable for travis too

* netdata/packaging/ci: So the jobs in a list dont share the environment, lets see if we do it otherwise. also remove the tag check, not work as expected

* netdata/packaging/ci: Add another way to check the RC condition

* netdata/packaging/ci: experimental - check conditions on message

* netdata/packaging/ci: Introduce an alternate master branch for the workflow testing
1. I created pipeline-redesign-master that will be the master branch for my use cases
2. Modified travis to trigger all production workflows on that branch, so that i can experiment freely with test commits to trigger process and verify execution flow

* remove spaces

* netdata/packaging/ci: comment out

* netdata/packaging/ci: wording nit

* netdata/packaging/ci: adjust the condition properly

* netdata/packaging/ci: test condition for triggering on specific keywords

* netdata/packaging/ci: try  different syntax

* netdata/packaging/ci: remove depth, causes problems as we dont get all required objects from repo (adds --depth 50 in clone)

* Missing equal sign in regex?

* netdata/packaging/ci: Update tagger script
1) More verbose messages
2) Update comments copyrights etc
3) Rename release_candidate to set_tag_release_candidate
4) Add curlies on the variables, also change the way we check for path

* netdata/packaging/ci: reinstate git depth

* netdata/packaging/ci: Remove git depth from packaging too

* netdata/packaging/ci: RTFM - was using wrong syntax for slack @here approach

* netdata/packaging/ci: Using install tag at the stage layer, overrides the global one for some reason. fix that

* netdata/packaging/ci: Dont use the pattern matching from travis, do it outside with a script

* netdata/packaging/ci: Do it the other way, script execution doesnt work there

* netdata/packaging/ci: Attempt to fix conditional on release, remove the faulty assumptions as provided by travis team

* netdata/packaging/ci: reintroduce the tag checking

* netdata/packaging/ci: you need to bring tag checking at the stage, as it is calculated during packaging step

* netdata/packaging/ci: you cant escape that one for the moment, merge them and we will rework it later again

* netdata/packaging/ci: dont forget git depth

* netdata/packaging/ci: bring back the dist checking

* netdata/packaging/ci: fix syntax

* netdata/packaging/ci: no depth

* netdata/packaging/ci: Introduce make dist validation tasks: Install from dist file and verify service is starting up

* netdata/packaging/ci: Add a couple more information for troubleshooting

* netdata/packaging/ci: Add extra information prior to executing crucial operations

* netdata/packaging/ci: Make rm error soft here

* netdata/packaging/ci: This is not needed here

* netdata/packaging/ci: Prepare for beta stage deployment process

- introduce a beta provider
- adjust key decryption process, different keys for beta, different for prod
- Enter conditions to deploy beta only on beta process and production only on production process
- Use travis variables, to allow multiple users have beta stages for deployment, if they want

* netdata/packaging/ci: add one more message

* netdata/packaging/ci: Bring back naming to the actual master branch

* netdata/packaging/ci: rename beta to dev

* netdata/packaging/ci: Make permanent restrictions for some of the stricts, so that we don't mess things up

* netdata/packaging/ci: Update README that was missed and also make sure git depth false is at the places needed

* netdata/packaging/ci: fix codacy complaints
Paul Emm. Katsoulakis 5 years ago
parent
commit
a8ba6234f6

+ 248 - 86
.travis.yml

@@ -1,113 +1,275 @@
-dist: trusty
 sudo: true
 language: c
 services:
 - docker
 
+# This is a hook to help us introduce "soft" errors on our process
+matrix:
+  allow_failures:
+    - env: ALLOW_SOFT_FAILURE_HERE=true
+
+# Install dependencies for all, once
+#
+install:
+- sudo apt-get install -y libcap2-bin zlib1g-dev uuid-dev fakeroot libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev gnupg python-pip
+- sudo apt install -y --only-upgrade docker-ce
+- sudo pip install git-semver
+- docker info
+- source tests/installer/slack.sh
+
+# Setup notification system
+#
+notifications:
+  webhooks: https://app.fossa.io/hooks/travisci
+
+# Define the stage sequence and conditionals
+#
 stages:
-- test
-- build
-- name: packaging
-  if: branch = master AND type != pull_request AND type != cron
-- name: nightlies
+# Mandatory runs, we always want these executed
+- name: Code quality, linting, syntax, code style
+- name: Build process
+- name: Artifacts validation
+
+  # Nightly operations
+- name: Nightly operations
   if: branch = master AND type = cron
-- name: Integrity testing
+- name: Nightly release
   if: branch = master AND type = cron
 
+  # Scheduled releases
+- name: Packaging for release
+  if: branch = master AND type != pull_request AND type != cron
+
+- name: Publish for release
+  if: branch = master AND type != pull_request AND type != cron AND commit_message =~ /(\[netdata release candidate\]|\[netdata major release\]|\[netdata minor release\]|\[netdata patch release\])/
+
+
+
+# Define stage implementation details
+#
 jobs:
   include:
-  - stage: test
-    name: C
-    install: sudo apt-get install -y libcap2-bin zlib1g-dev uuid-dev fakeroot libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev
-    script: fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto && $HOME/netdata/usr/sbin/netdata -W unittest
-    env: CFLAGS='-O1 -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1'
-  - name: dashboard.js
-    script: cp web/gui/dashboard.js /tmp/dashboard.js && ./build/build.sh && diff /tmp/dashboard.js web/gui/dashboard.js
-  - name: lint .sh.in files
+    # Do code quality, syntax checking and other pre-build activities
+  - stage: Code quality, linting, syntax, code style
+
+    name: Run shellchecking on BASH
     script: shellcheck --format=gcc $(find . -name '*.sh.in' -not -iwholename '*.git*')
-  - name: check checksums for kickstart files
+
+    # This falls under same stage defined earlier
+  - name: Run checksum checks on kickstart files
     script: ./tests/installer/checksums.sh
     env: LOCAL_ONLY="true"
-  - name: Kickstart Script execution integrity
-    script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" packaging/installer/kickstart.sh --dont-wait --dont-start-it --install /tmp
-  - name: Kickstart-static64 Script execution integrity
-    before_script: rm -rf /tmp/kickstart-static64-temp && mkdir -p /tmp/kickstart-static64-temp
-    script: docker run -it -v "${PWD}:/code:rw" -v "/tmp/kickstart-static64-temp:/opt" -w /code "netdata/os-test:centos7" packaging/installer/kickstart-static64.sh --dont-wait --dont-start-it
-    after_script: rm -rf /tmp/kickstart-static64-temp
-  - name: coverity
-    install: sudo apt-get install -y zlib1g-dev uuid-dev libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev
-    script: ./coverity-install.sh && ./coverity-scan.sh || echo "Coverity failed :("
-    if: type = cron
-
-  - stage: build
-# TODO: enable when travis OSX become stable. Probably after 12.01.2019
-#   name: OSX
-#   install: brew install fakeroot ossp-uuid
-#   script: fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it
-#   os: osx
-# - name: ubuntu 14.04 (not containerized)
-    name: ubuntu 14.04 (not containerized)
-    install: sudo apt-get install -y libcap2-bin zlib1g-dev uuid-dev fakeroot
-    script: fakeroot ./netdata-installer.sh --dont-wait --dont-start-it --install $HOME
-  - name: build container (alpine installation)
-    script: ./packaging/docker/build.sh
+
+    # This falls under same stage defined earlier
+  - name: Web Dashboard pre-generated file consistency checks (dashboard.js)
+    script: cp web/gui/dashboard.js /tmp/dashboard.js && ./build/build.sh && diff /tmp/dashboard.js web/gui/dashboard.js
+
+
+
+    # Ensure netdata code builds successfully
+  - stage: Build process
+
+    name: Standard netdata build
+    script: fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto
+    env: CFLAGS='-O1 -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1'
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> standard netdata build is failing (Still dont know which one, will improve soon)"
+
+  - name: Docker container build process (alpine installation)
+    script: packaging/docker/build.sh
     env: DEVEL="true"
-  - name: ubuntu 18.04 + lifecycle
+    after_failure: post_message "TRAVIS_MESSAGE" "Docker build process failed"
+
+  - name: Run 'make dist' validation
+    before_script: mkdir /tmp/netdata-makedist-test
+    script:
+    - echo "GIT Branch:" && git branch
+    - echo "Last commit:" && git log -1
+    - echo "GIT Describe:" && git describe
+    - echo "packaging/version:" && cat packaging/version
+    - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make clean || echo "Nothing to clean"
+    - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make distclean || echo "Nothing to distclean"
+    - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" autoreconf -ivf && ./configure --prefix=/netdata_install/usr --sysconfdir=/netdata_install/etc --localstatedir=/netdata_install/var --with-zlib --with-math --with-user=netdata CFLAGS=-O2
+    - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make dist
+    - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" ls -ltr ./netdata-$(git describe).tar.gz || ls -ltr ./netdata-$(cat packaging/version | tr -d '\n').tar.gz
+    - .travis/run_install_with_dist_file.sh
+    - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make distclean
+    git:
+      depth: false
+    after_script: rm -rf /tmp/netdata-makedist-test
+    after_failure: post_message "TRAVIS_MESSAGE" "'make dist' failed"
+
+
+
+  - stage: Artifacts validation
+
+    name: Unit Testing
+    script:
+    - fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto
+    - $HOME/netdata/usr/sbin/netdata -W unittest
+    env: CFLAGS='-O1 -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1'
+    after_failure: post_message "TRAVIS_MESSAGE" "Unit testing failed"
+
+  - name: Build/install on ubuntu 14.04 (not containerized)
+    script: fakeroot ./netdata-installer.sh --dont-wait --dont-start-it --install $HOME
+    after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on ubuntu 14.04"
+
+  - name: Build/Install for ubuntu 18.04 (not containerized)
+    script: fakeroot ./netdata-installer.sh --dont-wait --dont-start-it --install $HOME
+    after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on ubuntu 18.04"
+
+  - name: Run netdata lifecycle, on ubuntu 18.04 (Containerized)
     script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:ubuntu1804" bats --tap tests/lifecycle.bats
-  - name: CentOS 7
-    script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" ./netdata-installer.sh --dont-wait --dont-start-it --install /tmp
-  - name: CentOS 6
+    after_failure: post_message "TRAVIS_MESSAGE" "Netdata lifecycle test script failed on ubuntu 18.04"
+
+  - name: Build/install for CentOS 6 (Containerized)
     script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos6" ./netdata-installer.sh --dont-wait --dont-start-it --install /tmp
+    after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on CentOS 6"
 
-  - stage: packaging
-    name: Create release (only on special commit msg)
-    install:
-    - sudo apt-get install -y gnupg libcap2-bin zlib1g-dev uuid-dev fakeroot python-pip
-    - sudo apt install -y --only-upgrade docker-ce
-    - docker info
-    before_script: sudo pip install git-semver
-    script: ".travis/releaser.sh && .travis/labeler.sh" # labeler should be replaced with GitHub Actions when they hit GA
+  - name: Build/install for CentOS 7 (Containerized)
+    script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" ./netdata-installer.sh --dont-wait --dont-start-it --install /tmp
+    after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on CentOS 7"
+
+
+
+  - stage: Packaging for release
+
+    name: Generate changelog and TAG the release (only on special commit msg)
+    before_script: post_message "TRAVIS_MESSAGE" "Packaging step for release initiated"
+    script: .travis/generate_changelog_and_tag_release.sh
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> Packaging for release failed"
     git:
       depth: false
 
-  - stage: nightlies
-    name: Nightly build
-    before_install: openssl aes-256-cbc -K $encrypted_8daf19481253_key -iv $encrypted_8daf19481253_iv -in .travis/gcs-credentials.json.enc -out .travis/gcs-credentials.json -d
-    install:
-    - sudo apt-get install -y gnupg libcap2-bin zlib1g-dev uuid-dev fakeroot
-    - sudo apt install -y --only-upgrade docker-ce
-    - docker info
-    script: ".travis/nightlies.sh"
+  - name: Run labeler on github issues
+    script: .travis/labeler.sh # labeler should be replaced with GitHub Actions when they hit GA
+
+
+
+    # We only publish if a TAG has been set during packaging
+  - stage: Publish for release
+
+    name: Build & Publish docker images
+    before_script: post_message "TRAVIS_MESSAGE" "Publishing docker images"
+    script:
+    - echo "GIT Branch:" && git branch
+    - echo "Last commit:" && git log -1
+    - echo "GIT Describe:" && git describe
+    - echo "packaging/version:" && cat packaging/version
+    - packaging/docker/build.sh
+    - packaging/docker/publish.sh
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> Docker image publishing failed"
     git:
       depth: false
-    deploy:
-      provider: gcs
-      edge:
-        branch: gcs-ng
-      project_id: netdata-storage
-      credentials: .travis/gcs-credentials.json
-      bucket: "netdata-nightlies"
-      skip_cleanup: true
-      local_dir: "artifacts"
-      on:
-        # Only deploy on netdata/netdata, master branch, when artifacts directory is created
-        repo: netdata/netdata
-        branch: master
-        condition: -d "artifacts"
-    after_deploy: rm -f .travis/gcs-credentials.json
+    env: ALLOW_SOFT_FAILURE_HERE=true
+    # We don't run on release candidates
+    if: tag !~ /(-rc)/
+
+  - name: Generate release artifacts
+    before_script: post_message "TRAVIS_MESSAGE" "Starting artifacts generation for release"
+    script: .travis/create_artifacts.sh
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> Release artifacts generation failed"
+    git:
+     depth: false
+    # We don't run on release candidates
+    if: tag !~ /(-rc)/
+
+  - name: Create release draft
+    before_script: post_message "TRAVIS_MESSAGE" "Drafting release on github"
+    script:
+    - echo "GIT Branch:" && git branch
+    - echo "Last commit:" && git log -1
+    - echo "GIT Describe:" && git describe
+    - echo "packaging/version:" && cat packaging/version
+    - .travis/draft_release.sh
+    git:
+      depth: false
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> Draft release submission failed"
+    # We don't run on release candidates
+    if: tag !~ /(-rc)/
+
+
+
+    # This is the nightly pre-execution step (Jobs, preparatory steps for nightly, etc)
+  - stage: Nightly operations
+
+    name: Run coverity scan
+    # Just notify people that Nightly ops triggered, use the first step as a hook to do that
+    before_script: post_message "TRAVIS_MESSAGE" "Starting nightly operations"
+    script: ./coverity-install.sh && ./coverity-scan.sh || echo "Coverity failed :("
 
-  - stage: Integrity testing
-    name: Kickstart Files integrity testing
+  - name: Kickstart files integrity testing (extended)
     script: ./tests/installer/checksums.sh
-  - name: Kickstart Execution integrity
-    before_script: rm -rf /tmp/kickstart-static64-temp && mkdir -p /tmp/kickstart-static64-temp
+
+  - name: Run labeler on github issues
+    script: .travis/labeler.sh # labeler should be replaced with GitHub Actions when they hit GA
+
+    # This is generating the changelog for nightly release and publish it
+  - name: Generate nightly changelog
+    before_script: post_message "TRAVIS_MESSAGE" "Starting changelog generation for nightlies"
+    script: ".travis/nightlies.sh"
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> Nightly changelog generation failed"
+    git:
+      depth: false
+
+
+
+    # This is the nightly execution step
+    #
+  - stage: Nightly release
+
+    name: Build & Publish docker images
+    before_script: post_message "TRAVIS_MESSAGE" "Publishing docker images for nightlies"
     script:
-    - docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" packaging/installer/kickstart.sh --dont-wait --dont-start-it --install /tmp
-    - docker run -it -v "${PWD}:/code:rw" -v "/tmp/kickstart-static64-temp:/opt" -w /code "netdata/os-test:centos7" packaging/installer/kickstart-static64.sh --dont-wait --dont-start-it
-    after_script: rm -rf /tmp/kickstart-static64-temp
+    - docker info
+    - packaging/docker/build.sh
+    - packaging/docker/publish.sh
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> Nightly docker image publish failed"
+    env: ALLOW_SOFT_FAILURE_HERE=true
 
-notifications:
-  webhooks: https://app.fossa.io/hooks/travisci
-  slack:
-    rooms:
-      - secure: "NuW1p7s+WGLcyhEceeiLRSV1JgAc6N47zgdSsYoxrjSFRQHDfc8jensypDcEJwgs1K2Hcve9iKRaAddEHEw7AkS6rie9gFR5HmmbKXfW2GFMqOr6maYTFsvaECPqiWk1n9/XnRLsAi5kZ8HxH+a8ldb/eaVoFQesY1jPXgh11BM5DwvpXjEtwg0WGASsKiymvnXFS3KcC+sR7Lln2GX1a8vfCX2I3TEmOedKMlSHUy5JilGGC3AWA0SWS8tR8PUH0u3dHL5j0RNIr1RO3Yx24QgUpg/YpvKymnW/iIIEOq2vb2mBhhiKEQjJ1djUL4VSPzjIDpUzThVpKaHk3syOp6W9qZEHKhR/sqjc5Yk2XRjsw1cM0nS60gaCgxtKhEMKWcjtvWf04oJAVrmcUwcYXj0eA+jgRCZl5VhyufK/fUJavjOfsQGjwhdjxQfwDCw33W17ypJUt4GZngdb6jbIhEOcKHSLQDu1vuHTw82hJJkthkmR59PX30qJdl/MEGcfVLdN/fkCokjR/qwfmkNwQm+wYSKsK/Jq4RgBT0/oZwY3e8nkCq2ov7lBbDO3/0rzQKWZ9Uy//tnoCM3vGhDwGHQxsHshv7g6KwdhYTcmm7WWWIucfLupcjFUO1HbRuJ+7ZnvxRRwKiV+MGkFT2SNJkS8q1/jCu9KGbmktd0WUSE="
+  - name: Create nightly release artifacts, publish to GCS
+    before_script: post_message "TRAVIS_MESSAGE" "Starting artifacts generation for nightlies"
+    script:
+    - echo "GIT Branch:" && git branch
+    - echo "Last commit:" && git log -1
+    - echo "GIT Describe:" && git describe
+    - echo "packaging/version:" && cat packaging/version
+    - .travis/create_artifacts.sh
+    after_failure: post_message "TRAVIS_MESSAGE" "<!here> Nightly artifacts generation failed"
+    before_deploy:
+      echo "Preparing creds under ${TRAVIS_REPO_SLUG}";
+      if [ "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
+         openssl aes-256-cbc -K $encrypted_8daf19481253_key -iv $encrypted_8daf19481253_iv -in .travis/gcs-credentials.json.enc -out .travis/gcs-credentials.json -d;
+      else
+         echo "Beta deployment stage in progress";
+         openssl aes-256-cbc -K $encrypted_8daf19481253_key -iv $encrypted_8daf19481253_iv -in .travis/gcs-credentials.json.enc -out .travis/gcs-credentials.json -d;
+      fi;
+    deploy:
+      # Beta storage, used for testing purposes
+      - provider: gcs
+        edge:
+          branch: gcs-ng
+        project_id: netdata-storage
+        credentials: .travis/gcs-credentials.json
+        bucket: "netdata-dev-nightlies"
+        skip_cleanup: true
+        local_dir: "artifacts"
+        on:
+          # Only deploy on netdata/netdata, master branch, when artifacts directory is created
+          repo: ${TRAVIS_REPO_SLUG}
+          branch: master
+          condition: -d "artifacts"
+      # Production storage
+      - provider: gcs
+        edge:
+          branch: gcs-ng
+        project_id: netdata-storage
+        credentials: .travis/gcs-credentials.json
+        bucket: "netdata-nightlies"
+        skip_cleanup: true
+        local_dir: "artifacts"
+        on:
+          # Only deploy on netdata/netdata, master branch, when artifacts directory is created
+          repo: netdata/netdata
+          branch: master
+          condition: -d "artifacts"
+    after_deploy: rm -f .travis/gcs-credentials.json

+ 87 - 81
.travis/README.md

@@ -8,84 +8,90 @@
 - encrypted_8daf19481253_key - key needed by openssl to decrypt GCS credentials file
 - encrypted_8daf19481253_iv - IV needed by openssl to decrypt GCS credentials file
 - COVERITY_SCAN_TOKEN - Token to allow coverity test analysis uploads
-
-## Stages
-
-### Test
-
-Unit tests and coverage tests are executed here. Stage consists of 2 parallel jobs:
-  - C tests - executed every time
-  - dashboard.js - test if source files create the same file as it is in current repo
-  - coverity test - executed only when pipeline was triggered from cron
-
-### Build
-
-Stage is executed every time and consists of 5 parallel jobs which execute containerized and non-containerized
-installations of netdata. Jobs are run on following operating systems:
-  - OSX
-  - ubuntu 14.04
-  - ubuntu 16.04 (containerized)
-  - CentOS 6 (containerized)
-  - CentOS 7 (containerized)
-  - alpine (containerized)
-
-Images for system containers are stored on dockerhub and are created from Dockerfiles located in 
-[netdata/helper-images](https://github.com/netdata/helper-images) repository.
-
-### Packaging
-
-This stage is executed only on "master" brach and allows us to create a new tag just looking at git commit message.
-It executes one script called `releaser.sh` which is responsible for creating a release on GitHub by using
-[hub](https://github.com/github/hub). This script is also executing other scripts which can also be used in other
-CI jobs:
-  - `.travis/tagger.sh`
-  - `.travis/generate_changelog.sh`
-  - `packaging/docker/build.sh`
-  - `.travis/create_artifacts.sh`
-
-Alternatively new release can be also created by pushing new tag to master branch.
-Additionally this step is also executing `.travis/labeler.sh` which is a temporary workaround to automatically label 
-issues and PR. This script should be replaced with GitHub Actions when they are available to public.
-
-##### tagger.sh
-
-Script responsible to find out what will be the next tag based on a keyword in last commit message. Keywords are:
- - `[netdata patch release]` to bump patch number
- - `[netdata minor release]` to bump minor number
- - `[netdata major release]` to bump major number
- - `[netdata release candidate]` to create a new release candidate (appends or modifies suffix `-rcX` of previous tag)
-All keywords MUST be surrounded with square brackets.
-Tag is then stored in `GIT_TAG` variable.
-
-##### generate_changelog.sh
-
-Automatic changelog generator which updates our CHANGELOG.md file based on GitHub features (mostly labels and pull
-requests). Internally it uses
-[github-changelog-generator](https://github.com/github-changelog-generator/github-changelog-generator) and more
-information can be found on that project site.
-
-##### build.sh and create_artifacts.sh
-
-Scripts used to build new container images and provide release artifacts (tar.gz and makeself archives)
-
-### Nightlies
-
-##### Tarball and self-extractor build AND Nightly docker images
-
-As names might suggest those two jobs are responsible for nightly netdata package creation and are run every day (in
-cron). Combined they produce:
-  - docker images
-  - tar.gz archive (soon to be removed)
-  - self-extracting package
-
-This is achieved by running 2 scripts described earlier:
-  - `create_artifacts.sh`
-  - `build.sh`
-
-Artifacts are pushed to GCS and container images are stored in docker hub.
-
-##### Changelog generation
-
-This job is responsible for regenerating changelog every day by executing `generate_changelog.sh` script. This is done
-only once a day due to github rate limiter.
-
+- SLACK_USERNAME - This is required for the slack notifications triggered by travis pipeline
+- SLACK_CHANNEL - This is the channel that Travis will be posting messages
+- SLACK_NOTIFY_WEBHOOK_URL - This is the incoming URL webhook as provided by slack integration. Visit Apps integration in slack to generate the required hook
+- SLACK_BOT_NAME - This is the name your bot will appear with on slack
+
+## CI workflow details
+Our CI pipeline is designed to help us identify and mitigate risks at all stages of implementation.
+To accommodate this need, we used [Travis CI](http://www.travis-ci.com) as our CI/CD tool.
+Our main areas of concern are:
+1) Only push code that is working. That means fail fast so that we can improve before we reach the public
+
+2) Reduce the time to market to minimum, by streamlining the release process.
+   That means a lot of testing, a lot of consistency checks, a lot of validations
+
+3) Generated artifacts consistency. We should not allow broken software to reach the public.
+   When this happens, it's embarassing and we struggle to eliminate it.
+
+4) We are an innovative company, so we love to automate :)
+
+
+Having said that, here's a brief introduction to Netdata's improved CI/CD pipeline with Travis.
+Our CI/CD lifecycle contains three different execution entry points:
+1) A user opens a pull request to netdata/master: Travis will run a pipeline on the branch under that PR
+2) A merge or commit happens on netdata/master. This will trigger travis to run, but we have two distinct cases in this scenario:
+   a) A user merges a pull request to netdata/master: Travis will run on master, after the merge.
+   b) A user runs a commit/merge with a special keyword (mentioned later).
+      This triggers a release for either minor, major or release candidate versions, depending the keyword
+3) A scheduled job runs on master once per day: Travis will run on master at the scheduled interval
+
+To accommodate all three entry points our CI/CD workflow has a set of steps that run on all three entry points.
+Once all these steps are successfull, then our pipeline executes another subset of steps for entry points 2 and 3.
+In travis terms the "steps" are "Stages" and within each stage we execute a set of activities called "jobs" in travis.
+
+### Always run: Stages that running on all three execution entry points
+
+## Code quality, linting, syntax, code style
+At this early stage we iterate through a set of basic quality control checks:
+- Shell checking: Run linters for our various BASH scripts
+- Checksum validators: Run validators to ensure our installers and documentation are in sync
+- Dashboard validator: We provide a pre-generated dashboard.js script file that we need to make sure its up to date. We validate that.
+
+## Build process
+At this stage, basically, we build :-)
+We do a baseline check of our build artifacts to guarantee they are not broken
+Briefly our activities include:
+- Verify docker builds successfully
+- Run the standard netdata installer, to make sure we build & run properly
+- Do the same through 'make dist', as this is our stable channel for our kickstart files
+
+## Artifacts validation
+At this point we know our software is building, we need to go through the a set of checks, to guarantee
+that our product meets certain epxectations. At the current stage, we are focusing on basic capabilities
+like installing in different distributions, running the full lifecycle of install-run-update-install and so on.
+We are still working on enriching this with more and more use cases, to get us closer to achieving full stability of our software.
+Briefly we currently evaluate the following activities:
+- Basic software unit testing
+- Non containerized build and install on ubuntu 14.04
+- Non containerized build and install on ubuntu 18.04
+- Running the full netdata lifecycle (install, update, uninstall) on ubuntu 18.04
+- Build and install on CentOS 6
+- Build and install on CentOS 7
+(More to come)
+
+### Nightly operations: Stages that run daily under cronjob
+The nightly stages are related to the daily nightly activities, that produce our daily latest releases.
+We also maintain a couple of cronjobs that run during the night to provide us with deeper insights,
+like for example coverity scanning or extended kickstart checksum checks
+
+## Nightly operations
+At this stage we run scheduled jobs and execute the nightly changelog generator, coverity scans,
+labeler for our issues and extended kickstart files checksum validations.
+
+## Nightly release
+During this stage we are building and publishing latest docker images, prepare the nightly artifacts
+and deploy them (the artifacts) to our google cloud service provider.
+
+
+### Publishing
+Publishing is responsible for executing the major/minor/patch releases and is separated
+in two stages: packaging preparation process and publishing.
+
+## Packaging for release
+During packaging we are preparing the release changelog information and run the labeler.
+
+## Publish for release
+The publishing stage is the most complex part in publishing. This is the stage were we generate and publish docker images,
+prepare the release artifacts and get ready with the release draft.

+ 5 - 0
.travis/create_artifacts.sh

@@ -8,6 +8,11 @@ if [ ! -f .gitignore ]; then
 	exit 1
 fi
 
+if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
+	echo "Beta mode on ${TRAVIS_REPO_SLUG}, not running anything here"
+	exit 0
+fi;
+
 # Everything from this directory will be uploaded to GCS
 mkdir -p artifacts
 BASENAME="netdata-$(git describe)"

+ 60 - 0
.travis/draft_release.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+#
+# Draft release generator.
+# This utility is responsible for submitting a draft release to github repo
+# It is agnostic of other processes, when executed it will draft a release,
+# based on the most recent reachable tag.
+#
+# Requirements:
+#   - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo
+#   - artifacts directory in place
+#     - The directory is created by create_artifacts.sh mechanism
+#     - The artifacts need to be created with the same tag, obviously
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author: Pavlos Emm. Katsoulakis <paul@netdata.cloud>
+
+set -e
+
+if [ ! -f .gitignore ]; then
+	echo "Run as ./travis/$(basename "$0") from top level directory of git repository"
+	exit 1
+fi
+
+if [[ $(git describe) =~ -rc* ]]; then
+	echo "This is a release candidate tag, we do not generate a release draft"
+	exit 0
+fi
+
+# Load the tag, if any
+GIT_TAG=$(git describe)
+
+if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
+	echo "Beta mode on ${TRAVIS_REPO_SLUG}, i was about to run for release (${GIT_TAG}), but i am emulating, so bye"
+	exit 0
+fi;
+
+echo "---- CREATING RELEASE DRAFT WITH ASSETS -----"
+# Download hub
+HUB_VERSION=${HUB_VERSION:-"2.5.1"}
+wget "https://github.com/github/hub/releases/download/v${HUB_VERSION}/hub-linux-amd64-${HUB_VERSION}.tgz" -O "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"
+tar -C /tmp -xvf "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"
+export PATH=$PATH:"/tmp/hub-linux-amd64-${HUB_VERSION}/bin"
+
+# Create a release draft
+if [ -z ${GIT_TAG+x} ]; then
+	echo "Variable GIT_TAG is not set. Something went terribly wrong! Exiting."
+	exit 1
+fi
+if [ "${GIT_TAG}" != "$(git tag --points-at)" ]; then
+	echo "ERROR! Current commit is not tagged. Stopping release creation."
+	exit 1
+fi
+until hub release create --draft \
+		-a "artifacts/netdata-${GIT_TAG}.tar.gz" \
+		-a "artifacts/netdata-${GIT_TAG}.gz.run" \
+		-a "artifacts/sha256sums.txt" \
+		-m "${GIT_TAG}" "${GIT_TAG}"; do
+	sleep 5
+done

+ 69 - 0
.travis/generate_changelog_and_tag_release.sh

@@ -0,0 +1,69 @@
+#!/bin/bash
+#
+# Script to automatically do a couple of things:
+#   - generate a new tag according to semver (https://semver.org/)
+#   - generate CHANGELOG.md by using https://github.com/skywinder/github-changelog-generator
+#
+# Tags are generated by searching for a keyword in last commit message. Keywords are:
+#  - [patch] or [fix] to bump patch number
+#  - [minor], [feature] or [feat] to bump minor number
+#  - [major] or [breaking change] to bump major number
+# All keywords MUST be surrounded with square braces.
+#
+# Script uses git mechanisms for locking, so it can be used in parallel builds
+#
+# Requirements:
+#   - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo
+#   - docker
+#
+# This is a modified version of:
+# https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author: Pavlos Emm. Katsoulakis <paul@netdata.cloud>
+# Author: Pawel Krupa (@paulfantom)
+set -e
+
+if [ ! -f .gitignore ]; then
+	echo "Run as ./travis/$(basename "$0") from top level directory of git repository"
+	exit 1
+fi
+
+echo "--- Executing Tagging facility to determine TAG ---"
+.travis/tagger.sh
+
+echo "--- Changelog generator and tagger script starting ---"
+# If tagger script hasn't produced a TAG, there is nothing to do so bail out happy
+if [ -z "${GIT_TAG}" ]; then
+	echo "GIT_TAG is empty, nothing to do for now (Value: $GIT_TAG)"
+	exit 0
+fi
+
+if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
+	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing to do on the changelog generator and tagging script for (${GIT_TAG}), bye"
+	exit 0
+fi
+
+echo "--- Initialize git configuration ---"
+export GIT_MAIL="bot@netdata.cloud"
+export GIT_USER="netdatabot"
+git config user.email "${GIT_MAIL}"
+git config user.name "${GIT_USER}"
+git checkout master
+git pull
+
+echo "---- UPDATE VERSION FILE ----"
+echo "$GIT_TAG" >packaging/version
+git add packaging/version
+
+echo "---- GENERATE CHANGELOG -----"
+./.travis/generate_changelog_for_release.sh
+git add CHANGELOG.md
+
+echo "---- COMMIT AND PUSH CHANGES ----"
+git commit -m "[ci skip] release $GIT_TAG"
+git tag "$GIT_TAG" -a -m "Automatic tag generation for travis build no. $TRAVIS_BUILD_NUMBER"
+git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')"
+git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')" --tags
+# After those operations output of command `git describe` should be identical with a value of GIT_TAG

+ 6 - 1
.travis/generate_changelog.sh → .travis/generate_changelog_for_nightlies.sh

@@ -31,8 +31,13 @@ if [ -z ${GIT_TAG+x} ]; then
 else
 	OPTS="--future-release ${GIT_TAG}"
 fi
-
 echo "We got $COMMITS_SINCE_RELEASE changes since $LAST_TAG, re-generating changelog"
+
+if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
+	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do here"
+	exit 0
+fi
+
 git config user.email "${GIT_MAIL}"
 git config user.name "${GIT_USER}"
 git checkout master

+ 5 - 2
.travis/generate_changelog_for_release.sh

@@ -11,13 +11,17 @@ ORGANIZATION=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $1}')
 PROJECT=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $2}')
 GIT_MAIL=${GIT_MAIL:-"bot@netdata.cloud"}
 GIT_USER=${GIT_USER:-"netdatabot"}
-
 if [ -z ${GIT_TAG+x} ]; then
 	OPTS=""
 else
 	OPTS="--future-release ${GIT_TAG}"
 fi
 
+if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
+	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do"
+	exit 0
+fi
+
 echo "--- Creating changelog ---"
 git checkout master
 git pull
@@ -30,4 +34,3 @@ docker run -it -v "$(pwd)":/project markmandel/github-changelog-generator:latest
 	--unreleased-label "**Next release**" \
 	--exclude-labels "stale,duplicate,question,invalid,wontfix,discussion,no changelog" \
 	--no-compare-link ${OPTS}
-

+ 10 - 14
.travis/nightlies.sh

@@ -1,11 +1,9 @@
 #!/bin/bash
 #
-# This is the nightlies orchastration script
-# It runs the following activities in order:
-# 1) Generate changelog
-# 2) Build docker images
-# 3) Publish docker images
-# 4) Generate the rest of the artifacts (Source code .tar.gz file and makeself binary generation)
+# This is the nightly changelog generation script
+# It is responsible for two major activities:
+# 1) Update packaging/version with the current nightly version
+# 2) Generate the changelog for the mentioned version
 #
 # Copyright: SPDX-License-Identifier: GPL-3.0-or-later
 #
@@ -34,14 +32,12 @@ if [ "${COMMITS_SINCE_RELEASE}" == "${PREVIOUS_NIGHTLY_COUNT}" ]; then
 	exit 0
 fi
 
-echo "--- Running Changelog generation ---"
-.travis/generate_changelog.sh "${LAST_TAG}" "${COMMITS_SINCE_RELEASE}" || echo "Changelog generation has failed, this is a soft error, process continues"
-
-echo "--- Build && publish docker images ---"
-# Do not fail artifacts creation if docker fails. We will be restructuring this on a follow up PR
-packaging/docker/build.sh && packaging/docker/publish.sh || echo "Failed to build and publish docker images"
+if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
+	echo "Beta mode -- nothing to do for ${TRAVIS_REPO_SLUG} on the nightlies script, bye"
+	exit 0
+fi
 
-echo "--- Build artifacts ---"
-.travis/create_artifacts.sh
+echo "--- Running Changelog generation ---"
+.travis/generate_changelog_for_nightlies.sh "${LAST_TAG}" "${COMMITS_SINCE_RELEASE}" || echo "Changelog generation has failed, this is a soft error, process continues"
 
 exit "${FAIL}"

+ 0 - 96
.travis/releaser.sh

@@ -1,96 +0,0 @@
-#!/bin/bash
-# SPDX-License-Identifier: MIT
-# Copyright (C) 2018 Pawel Krupa (@paulfantom) - All Rights Reserved
-# Permission to copy and modify is granted under the MIT license
-#
-# Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh
-#
-# Script to automatically do a couple of things:
-#   - generate a new tag according to semver (https://semver.org/)
-#   - generate CHANGELOG.md by using https://github.com/skywinder/github-changelog-generator
-#   - create draft of GitHub releases by using https://github.com/github/hub
-#
-# Tags are generated by searching for a keyword in last commit message. Keywords are:
-#  - [patch] or [fix] to bump patch number
-#  - [minor], [feature] or [feat] to bump minor number
-#  - [major] or [breaking change] to bump major number
-# All keywords MUST be surrounded with square braces.
-#
-# Script uses git mechanisms for locking, so it can be used in parallel builds
-#
-# Requirements:
-#   - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo
-#   - docker
-
-set -e
-
-if [ ! -f .gitignore ]; then
-	echo "Run as ./travis/$(basename "$0") from top level directory of git repository"
-	exit 1
-fi
-
-export GIT_MAIL="bot@netdata.cloud"
-export GIT_USER="netdatabot"
-echo "--- Initialize git configuration ---"
-git config user.email "${GIT_MAIL}"
-git config user.name "${GIT_USER}"
-git checkout master
-git pull
-
-echo "---- FIGURING OUT TAGS ----"
-# tagger.sh is sourced since we need environment variables it sets
-#shellcheck source=/dev/null
-source .travis/tagger.sh || exit 0
-# variable GIT_TAG is produced by tagger.sh script
-
-echo "---- UPDATE VERSION FILE ----"
-echo "$GIT_TAG" >packaging/version
-git add packaging/version
-
-echo "---- GENERATE CHANGELOG -----"
-./.travis/generate_changelog_for_release.sh
-git add CHANGELOG.md
-
-echo "---- COMMIT AND PUSH CHANGES ----"
-git commit -m "[ci skip] release $GIT_TAG"
-git tag "$GIT_TAG" -a -m "Automatic tag generation for travis build no. $TRAVIS_BUILD_NUMBER"
-git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')"
-git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')" --tags
-# After those operations output of command `git describe` should be identical with a value of GIT_TAG
-
-if [[ $(git describe) =~ -rc* ]]; then
-	echo "This is a release candidate tag, exiting without artifact building"
-	exit 0
-fi
-
-echo "---- CREATING TAGGED DOCKER CONTAINERS ----"
-./packaging/docker/build.sh
-./packaging/docker/publish.sh
-
-echo "---- CREATING RELEASE ARTIFACTS -----"
-# Artifacts are stored in `artifacts/` directory
-./.travis/create_artifacts.sh
-
-echo "---- CREATING RELEASE DRAFT WITH ASSETS -----"
-# Download hub
-HUB_VERSION=${HUB_VERSION:-"2.5.1"}
-wget "https://github.com/github/hub/releases/download/v${HUB_VERSION}/hub-linux-amd64-${HUB_VERSION}.tgz" -O "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"
-tar -C /tmp -xvf "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"
-export PATH=$PATH:"/tmp/hub-linux-amd64-${HUB_VERSION}/bin"
-
-# Create a release draft
-if [ -z ${GIT_TAG+x} ]; then
-	echo "Variable GIT_TAG is not set. Something went terribly wrong! Exiting."
-	exit 1
-fi
-if [ "${GIT_TAG}" != "$(git tag --points-at)" ]; then
-	echo "ERROR! Current commit is not tagged. Stopping release creation."
-	exit 1
-fi
-until hub release create --draft \
-		-a "artifacts/netdata-${GIT_TAG}.tar.gz" \
-		-a "artifacts/netdata-${GIT_TAG}.gz.run" \
-		-a "artifacts/sha256sums.txt" \
-		-m "${GIT_TAG}" "${GIT_TAG}"; do
-	sleep 5
-done

+ 48 - 0
.travis/run_install_with_dist_file.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+#
+# This script is evaluating netdata installation with the source from make dist
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author  : Pavlos Emm. Katsoulakis <paul@netdata.cloud)
+#
+set -e
+
+# If we are not in netdata git repo, at the top level directory, fail
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
+CWD=$(git rev-parse --show-cdup || echo "")
+if [ -n "${CWD}" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
+    echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"
+    echo "Changelog generation process aborted"
+    exit 1
+fi
+
+echo "Initiating dist archive contents validation"
+DIST_FILE_FROM_GIT="netdata-$(git describe).tar.gz"
+DIST_FILE_FROM_FILE="netdata-$(tr -d '\n' < packaging/version).tar.bgz"
+if [ -f ${DIST_FILE_FROM_GIT} ]; then
+	DIST_FILE="${DIST_FILE_FROM_GIT}"
+elif [ -f ${DIST_FILE_FROM_FILE} ]; then
+	DIST_FILE="${DIST_FILE_FROM_FILE}"
+else
+	echo "I could not find netdata distfile. Nor ${DIST_FILE_FROM_GIT} or ${DIST_FILE_FROM_FILE} exist"
+	exit 1
+fi
+
+echo "Opening dist archive ${DIST_FILE}"
+tar -xovf "${DIST_FILE}"
+NETDATA_DIST_FOLDER=$(echo ${DIST_FILE} | cut -d. -f1,2,3)
+if [ ! -d ${NETDATA_DIST_FOLDER} ]; then
+	echo "I could not locate folder ${NETDATA_DIST_FOLDER}, something went wrong failing the test"
+	exit 1
+fi
+
+echo "Entering ${NETDATA_DIST_FOLDER} and starting docker compilation"
+cd ${NETDATA_DIST_FOLDER}
+docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" /bin/bash -c "./netdata-installer.sh --dont-wait --install /tmp && echo \"Validating netdata instance is running\" && wget -O'-' 'http://127.0.0.1:19999/api/v1/info' | grep version"
+
+echo "Installation completed with no errors! Removing temporary folders"
+
+# TODO: Travis give me a permission denied on some files here, i made it a soft error until i figure out what is wrong
+cd -
+rm -rf ${NETDATA_DIST_FOLDER} || echo "I could not remove temporary directory, make sure you delete ${NETDATA_DIST_FOLDER} by yourself if this wasn't run over ephemeral storage"

Some files were not shown because too many files changed in this diff